Skip to content

Add: skill-optimizer — Static LLM tool-calling benchmark and doc optimizer#7

Open
damienen wants to merge 1 commit into
Vvkmnn:mainfrom
damienen:add-skill-optimizer
Open

Add: skill-optimizer — Static LLM tool-calling benchmark and doc optimizer#7
damienen wants to merge 1 commit into
Vvkmnn:mainfrom
damienen:add-skill-optimizer

Conversation

@damienen

Copy link
Copy Markdown

skill-optimizer benchmarks whether LLMs call the right SDK, CLI, or MCP tools from your guidance docs (SKILL.md), using static action + argument matching. It also runs an iterative optimizer that rewrites your docs until every configured model meets a per-model score floor — producing CI-ready PASS/FAIL verdicts.

Fits under "Core Frameworks" alongside Promptfoo and Inspect AI as a static, offline evaluation harness with a focus on tool-calling correctness rather than answer quality.

Repo: https://github.com/fastxyz/skill-optimizer

Checklist:

  • Entry is in scope for this list (static LLM evaluation harness)
  • Alphabetically placed (S, after Ragas, before TruLens)
  • No duplicates
  • Badge appears before the link, matching existing format

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant