toolfit 0.1.0__tar.gz

toolfit-0.1.0/.env.example

@@ -0,0 +1,11 @@
+# Copy this file to .env and fill in your own keys. .env is gitignored — never commit real keys.
+
+# Required for `toolfit eval` / `scripts/run_spike.py` — get one at https://platform.claude.com/settings/keys
+ANTHROPIC_API_KEY=
+
+# Optional — only needed if `--model` names an OpenAI model (gpt-*, o1-*, o3-*, o4-*)
+OPENAI_API_KEY=
+
+# Optional — only needed if `--model` names an OpenRouter vendor/model (e.g. qwen/qwen-2.5-72b-instruct),
+# or for scripts/openrouter_check.py (OpenRouter compatibility check)
+OPENROUTER_API_KEY=

toolfit-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,23 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: uv sync --extra dev
+      - run: uv run pytest -q
+      - run: uv build
+      # The free path must keep working against the toy server with no key present.
+      - run: uv run toolfit scan examples/toy_server.py

toolfit-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,20 @@
+name: publish
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  pypi:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write   # PyPI trusted publishing (OIDC) — no token stored anywhere
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync --extra dev
+      - run: uv run pytest -q
+      - run: uv build
+      - run: uv publish --trusted-publishing always

toolfit-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+.superpowers/
+.claude/worktrees/
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+.env
+dist/

toolfit-0.1.0/CHANGELOG.md

@@ -0,0 +1,15 @@
+# Changelog
+
+## 0.1.0 (2026-09-05)
+
+First release.
+
+- `toolfit scan <server>`: free static lint over `tools/list` — missing, too-short, duplicated, and self-declared-deprecated descriptions. No model calls.
+- `toolfit eval <server>`: inverted task generation (sample schema-valid arguments first, then ask a generator to write the request), structural grading with canonicalisation, confusion matrix, per-tool pass rates with Wilson 95% intervals, leakage and solvability guardrails reported as warnings.
+- `--mutate 'tool:new description'`: paired re-run of that tool's own tasks against a catalog with one description patched; exact one-sided McNemar p-value; one Bonferroni correction across everything re-measured in the run.
+- `--fix` / `--fix-tool NAME`: propose a rewrite per failing tool (rewriter sees the real parameters and neighbour descriptions), re-measure, report accepted and rejected alike; `toolfit-fixes.json` with description text only.
+- `--badge`: SVG coloured by pass rate with model, generator, seeds, and task-suite hash embedded. `--strict` / `--strict-threshold`: exit codes for CI.
+- Servers as a `.py` script (`uv run`), any command line (`npx -y …`), or an `http(s)://` URL; subprocess inherits the environment.
+- Adapters for Anthropic, OpenAI, and OpenRouter, inferred from `--model`; retry with backoff on 429/5xx for every model call.
+- Composite GitHub Action (`action.yml`); repo CI on 3.10/3.13; PyPI publish on `v*` tags via trusted publishing.
+- Evidence: toy server and three public servers evaluated end-to-end (`docs/examples/`), twenty public servers scanned (`docs/corpus.md`).

toolfit-0.1.0/CLAUDE.md

@@ -0,0 +1,19 @@
+
+## Skill routing
+
+When the user's request matches an available skill, invoke it via the Skill tool. When in doubt, invoke the skill.
+
+Key routing rules:
+- Product ideas/brainstorming → invoke /office-hours
+- Strategy/scope → invoke /plan-ceo-review
+- Architecture → invoke /plan-eng-review
+- Design system/plan review → invoke /design-consultation or /plan-design-review
+- Full review pipeline → invoke /autoplan
+- Bugs/errors → invoke /investigate
+- QA/testing site behavior → invoke /qa or /qa-only
+- Code review/diff check → invoke /review
+- Visual polish → invoke /design-review
+- Ship/deploy/PR → invoke /ship or /land-and-deploy
+- Save progress → invoke /context-save
+- Resume context → invoke /context-restore
+- Author a backlog-ready spec/issue → invoke /spec

toolfit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sreshta Talluri
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

toolfit-0.1.0/PKG-INFO

@@ -0,0 +1,244 @@
+Metadata-Version: 2.5
+Name: toolfit
+Version: 0.1.0
+Summary: Finds the specific places your MCP server confuses models, and proves a fix with a before-and-after eval.
+Project-URL: Repository, https://github.com/sreshtalluri/toolfit
+Author: Sreshta Talluri
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,eval,llm,mcp,model-context-protocol,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: anthropic>=0.40
+Requires-Dist: mcp<3,>=2.0
+Requires-Dist: openai>=1.50
+Requires-Dist: pydantic>=2.0
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# toolfit
+
+toolfit finds the specific places your MCP server confuses models, rewrites the tool
+descriptions to fix them, and proves the fix with a before-and-after eval.
+
+```
+uvx toolfit scan "npx -y @modelcontextprotocol/server-github"     # free, static, seconds
+uvx toolfit eval examples/toy_server.py --seeds 10 --fix --badge   # model-graded, minutes, your key
+```
+
+Every existing MCP grader stops at a number and a list of complaints. toolfit measures which
+tool pairs a model actually confuses, proposes a description rewrite for each failing tool,
+re-runs the same tasks against the rewrite, and reports the delta with a p-value — accepted
+*and* rejected. The rejected ones are what make the accepted ones believable.
+
+## What a run looks like
+
+`toolfit eval examples/toy_server.py --seeds 10 --fix --badge`, model under test `claude-sonnet-5`
+(full output: [`docs/examples/toy-server/`](docs/examples/toy-server/)). The toy server has two
+deliberately confusable pairs: `create_task`/`update_task` share the description *"Add a new task."*,
+and `list_tasks`/`count_tasks` share *"Get tasks by status."*
+
+```
+## Confusion Matrix
+
+| Intended \ Called | count_tasks | create_reminder | create_task | list_tasks | update_task |
+|---|---|---|---|---|---|
+| count_tasks       | 0  | 0  | 0  | 10 | 0  |
+| create_reminder   | 0  | 10 | 0  | 0  | 0  |
+| create_task       | 0  | 0  | 10 | 0  | 0  |
+| list_tasks        | 0  | 0  | 0  | 10 | 0  |
+| update_task       | 0  | 0  | 0  | 0  | 10 |
+
+## Pass Rates
+- count_tasks: 0/10 (0%), 95% CI [0%, 28%]
+- create_reminder: 8/10 (80%), 95% CI [49%, 94%]
+- create_task: 9/10 (90%), 95% CI [60%, 98%]
+- list_tasks: 10/10 (100%), 95% CI [72%, 100%]
+- update_task: 10/10 (100%), 95% CI [72%, 100%]
+
+## Proposed Fixes
+
+### create_task — REJECTED
+- Before: 'Add a new task.'
+- After:  'Creates a brand-new task by specifying its title and priority, distinct from
+           update_task (which modifies existing tasks) or create_reminder (...)'
+- Pass rate: 9/10 → 7/10, p-value 1.0000
+- Reason: rejected: made things worse
+
+### count_tasks — REJECTED
+- Before: 'Get tasks by status.'
+- After:  'Return the number of tasks matching a given status, rather than the tasks
+           themselves, by accepting a required status argument.'
+- Pass rate: 0/10 → 0/10, p-value 1.0000
+- Reason: rejected: no change
+```
+
+![toolfit: 74%](docs/examples/toy-server/toolfit-badge.svg)
+
+Three things this run shows, none of them flattering, all of them the point:
+
+- **The matrix names the real problem.** `count_tasks` is called *zero* times out of ten; every
+  request meant for it goes to `list_tasks`. The solvability check flagged the identical
+  `create_task`/`update_task` descriptions on 14 of 20 tasks — yet Sonnet 5 still picked the right
+  one every time, because it reads `task_id` in the schema. Descriptions aren't the whole story,
+  and toolfit measures what the model does, not what the text says.
+- **The fix loop refused to claim a win.** Three rewrites were proposed and measured; one made
+  things worse, two changed nothing. They are printed anyway. Every failure here is at the
+  argument level or in the generator's "count vs. list" phrasing — things a description rewrite
+  can't fix — and the numbers say so instead of the tool saying so.
+- **Every number carries its uncertainty.** `9/10` and `7/10` overlap almost entirely at n=10;
+  the exact test gives p=1.0 for a change in the wrong direction. Nothing gets reported as a
+  finding because it looked good once.
+
+## On real servers
+
+Same command against three public servers, model under test `claude-sonnet-5`, 10 seeds per tool
+(full reports in [`docs/examples/`](docs/examples/)):
+
+| Server | Tools | Pass | What the matrix showed |
+|---|---|---|---|
+| `@modelcontextprotocol/server-memory` | 9 | **90/90** | Perfect diagonal. Nine crisp descriptions, zero warnings, nothing to fix. |
+| `mcp-server-git` | 12 | **112/120** | `git_commit` 4/10: five requests went to `git_add` first. The rewrite that stressed "commits *staged* changes" measured worse, 4→2, and was rejected. |
+| `@modelcontextprotocol/server-filesystem` | 14 | **77/140** | `read_file` 0/10 — its description says *DEPRECATED, use read_text_file* and the model obeys (`scan` now flags this). Thirty-plus requests across seven tools went to `list_allowed_directories` first. |
+
+The git and filesystem results share a cause that a description can't fix: **the model takes a
+correct precondition step** — stage before commit, check allowed directories before touching a
+path — and a single-step eval scores it as the wrong tool. That column in the confusion matrix is
+the finding; it tells you which tools need their precondition stated ("paths are validated for
+you") or a multi-step harness. toolfit does not paper over it by grading the first call leniently.
+
+Twelve rewrites were proposed for the filesystem server. Five improved the number (`8→10`,
+`6→8`); none were accepted, because one Bonferroni correction across twelve proposals at n=10
+sets α=0.004 and the report says so. Run `--fix-tool list_directory --fix-tool move_file
+--seeds 20` on the tools the matrix names, not `--fix` on the whole catalog at once.
+
+## Two commands, two budgets
+
+| | `scan` | `eval` |
+|---|---|---|
+| What | Static lint over `tools/list`: missing, too-short, and duplicated descriptions | Live model behaviour: which tool it calls, with which arguments, for a request that should lead to each tool |
+| Cost | Free — no model calls, no key | Your API key: roughly `tools × seeds × 3` calls, plus `seeds` per proposed fix |
+| Time | Under a second after the server starts | Minutes |
+| Output | Findings list (never a letter grade) | Confusion matrix, per-tool pass rates with 95% CIs, mutation/fix verdicts, optional badge and `toolfit-fixes.json` |
+
+They are deliberately separate. `scan` is the zero-config front door; on mature servers it finds
+little ([1 finding across 166 tools on 15 public servers](docs/corpus.md)) because the bug it
+catches is the copy-paste class. The confusion is what `eval` is for.
+
+## Install
+
+```
+uvx toolfit --help          # zero-install, one-off
+pipx install toolfit        # persistent / CI
+```
+
+Python 3.10+. Talks to servers over the MCP protocol via the official `mcp` SDK, so the
+server can be in any language.
+
+## Pointing it at a server
+
+```
+toolfit scan path/to/server.py                                  # run via `uv run`
+toolfit scan "npx -y @modelcontextprotocol/server-filesystem ."   # any command line
+toolfit scan https://your-host/mcp                              # Streamable HTTP
+```
+
+The subprocess gets your environment (minus toolfit's own `*_API_KEY`s, which a third-party server
+binary has no business seeing), so servers that read a token from `GITHUB_TOKEN` or
+`STRIPE_SECRET_KEY` work unchanged. toolfit never calls a tool — it only ever asks for the
+catalog (`tools/list`), then asks a model what it *would* call. Nothing touches your backend.
+
+## How the measurement works
+
+The credibility problem with LLM-generated evals is circularity: if one model writes the task
+and also decides the expected answer, you are measuring agreement with that model. toolfit
+inverts it:
+
+1. Sample a concrete, schema-valid argument set for one tool (`gen/schema_sampler.py`),
+   honouring `required`, enums, formats, and nullables.
+2. Ask a generator model to write the sentence a user would type that leads to *exactly those
+   arguments*, without naming the tool. Ground truth is the sampled tuple, not the model's opinion.
+3. Send that sentence plus the whole catalog to the model under test.
+4. Grade structurally (`grade/grader.py`): right tool, and arguments equal after canonicalising
+   dates, case, whitespace, and array order. No LLM judge, ever.
+
+Guardrails: a second pass checks each task for tool-name leakage and for solvability against
+the catalog; both are reported as warnings, never silently dropped. Tools whose schema the
+sampler can't handle are excluded and listed — a partial number is never printed as a complete one.
+
+**Mutation testing** is the same grader run twice. `--mutate 'tool:new description'` re-runs a
+tool's *own* base tasks against a catalog where only that description is patched (protocol-level;
+your source is never touched) and compares pass rates on paired trials. `--fix` does the same with
+a proposed rewrite for every failing tool.
+
+**Significance** is an exact one-sided McNemar test on the discordant pairs, Bonferroni-corrected
+across everything re-measured in one run. It is deterministic and honest at small n: with 5 seeds
+the smallest attainable p-value is 1/32, so `--seeds 10` is the practical floor for a verdict —
+the CLI warns if you go lower. Every rate carries `n` and a Wilson 95% interval.
+
+## CI
+
+```yaml
+- uses: sreshtalluri/toolfit@main
+  with:
+    server: "npx -y @modelcontextprotocol/server-github"
+    eval: true                      # omit for the free scan only
+    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+`scan --strict` exits 1 on any finding; `eval --strict` exits 1 if any tool's pass rate is below
+`--strict-threshold` (default 0.9). Tools excluded by a schema warning are named on stderr and
+*not* counted — they don't fail the gate, but you'll see them.
+
+`--badge` writes `toolfit-badge.svg` with the pass rate (or the before→after delta of a single
+mutation or accepted fix), coloured by rate, with the model, generator, seed count, and a hash of
+the task suite embedded so the number is never separable from what produced it.
+
+## Models
+
+`--model` picks the model under test and the provider is inferred from the name: `claude*` →
+Anthropic, `gpt*`/`o*` → OpenAI, `vendor/model` → OpenRouter. Keys come from `ANTHROPIC_API_KEY`,
+`OPENAI_API_KEY`, `OPENROUTER_API_KEY`. Task generation and fix proposals always use Anthropic, so
+that key is required for `eval` regardless of `--model`.
+
+**Data handling.** The tool catalog and the generated task text are sent to whichever provider you
+configure, with your key. toolfit itself stores nothing and phones nowhere. If your server is
+internal, that is the one place its schema leaves your machine.
+
+## Landscape
+
+| Tool | What it does | Where it stops |
+|---|---|---|
+| mcpgrade | Static lint + single-step eval, A–F grade | No re-measured fix |
+| mcpx | ESLint-style schema lint, CI gate | Never runs a model |
+| MCProbe | Usability rules + input fuzzing | Fuzzing, not task performance |
+| lastmile-ai/mcp-eval | Assertion framework with LLM judges | You write the tests |
+| MCPJam Inspector | Hosted evals with cross-model comparison | GUI product, not a CLI |
+| MCP-Atlas / MCP-Bench | Benchmarks ranking *models* | Not pointable at your server |
+
+toolfit is not a model leaderboard and does not test for prompt injection. One server per run.
+
+## Development
+
+```
+uv sync --extra dev
+uv run pytest -q
+uv run toolfit scan examples/toy_server.py
+```
+
+`examples/toy_server.py` has two deliberately confusable tool pairs — `create_task`/`update_task`
+share a description, `list_tasks`/`count_tasks` share a vague one — so the fix loop has something
+real to find. Design history and the methodology decisions behind every number are in
+[`docs/designs/toolfit-v0-scope.md`](docs/designs/toolfit-v0-scope.md).
+
+MIT.

toolfit-0.1.0/README.md

@@ -0,0 +1,216 @@
+# toolfit
+
+toolfit finds the specific places your MCP server confuses models, rewrites the tool
+descriptions to fix them, and proves the fix with a before-and-after eval.
+
+```
+uvx toolfit scan "npx -y @modelcontextprotocol/server-github"     # free, static, seconds
+uvx toolfit eval examples/toy_server.py --seeds 10 --fix --badge   # model-graded, minutes, your key
+```
+
+Every existing MCP grader stops at a number and a list of complaints. toolfit measures which
+tool pairs a model actually confuses, proposes a description rewrite for each failing tool,
+re-runs the same tasks against the rewrite, and reports the delta with a p-value — accepted
+*and* rejected. The rejected ones are what make the accepted ones believable.
+
+## What a run looks like
+
+`toolfit eval examples/toy_server.py --seeds 10 --fix --badge`, model under test `claude-sonnet-5`
+(full output: [`docs/examples/toy-server/`](docs/examples/toy-server/)). The toy server has two
+deliberately confusable pairs: `create_task`/`update_task` share the description *"Add a new task."*,
+and `list_tasks`/`count_tasks` share *"Get tasks by status."*
+
+```
+## Confusion Matrix
+
+| Intended \ Called | count_tasks | create_reminder | create_task | list_tasks | update_task |
+|---|---|---|---|---|---|
+| count_tasks       | 0  | 0  | 0  | 10 | 0  |
+| create_reminder   | 0  | 10 | 0  | 0  | 0  |
+| create_task       | 0  | 0  | 10 | 0  | 0  |
+| list_tasks        | 0  | 0  | 0  | 10 | 0  |
+| update_task       | 0  | 0  | 0  | 0  | 10 |
+
+## Pass Rates
+- count_tasks: 0/10 (0%), 95% CI [0%, 28%]
+- create_reminder: 8/10 (80%), 95% CI [49%, 94%]
+- create_task: 9/10 (90%), 95% CI [60%, 98%]
+- list_tasks: 10/10 (100%), 95% CI [72%, 100%]
+- update_task: 10/10 (100%), 95% CI [72%, 100%]
+
+## Proposed Fixes
+
+### create_task — REJECTED
+- Before: 'Add a new task.'
+- After:  'Creates a brand-new task by specifying its title and priority, distinct from
+           update_task (which modifies existing tasks) or create_reminder (...)'
+- Pass rate: 9/10 → 7/10, p-value 1.0000
+- Reason: rejected: made things worse
+
+### count_tasks — REJECTED
+- Before: 'Get tasks by status.'
+- After:  'Return the number of tasks matching a given status, rather than the tasks
+           themselves, by accepting a required status argument.'
+- Pass rate: 0/10 → 0/10, p-value 1.0000
+- Reason: rejected: no change
+```
+
+![toolfit: 74%](docs/examples/toy-server/toolfit-badge.svg)
+
+Three things this run shows, none of them flattering, all of them the point:
+
+- **The matrix names the real problem.** `count_tasks` is called *zero* times out of ten; every
+  request meant for it goes to `list_tasks`. The solvability check flagged the identical
+  `create_task`/`update_task` descriptions on 14 of 20 tasks — yet Sonnet 5 still picked the right
+  one every time, because it reads `task_id` in the schema. Descriptions aren't the whole story,
+  and toolfit measures what the model does, not what the text says.
+- **The fix loop refused to claim a win.** Three rewrites were proposed and measured; one made
+  things worse, two changed nothing. They are printed anyway. Every failure here is at the
+  argument level or in the generator's "count vs. list" phrasing — things a description rewrite
+  can't fix — and the numbers say so instead of the tool saying so.
+- **Every number carries its uncertainty.** `9/10` and `7/10` overlap almost entirely at n=10;
+  the exact test gives p=1.0 for a change in the wrong direction. Nothing gets reported as a
+  finding because it looked good once.
+
+## On real servers
+
+Same command against three public servers, model under test `claude-sonnet-5`, 10 seeds per tool
+(full reports in [`docs/examples/`](docs/examples/)):
+
+| Server | Tools | Pass | What the matrix showed |
+|---|---|---|---|
+| `@modelcontextprotocol/server-memory` | 9 | **90/90** | Perfect diagonal. Nine crisp descriptions, zero warnings, nothing to fix. |
+| `mcp-server-git` | 12 | **112/120** | `git_commit` 4/10: five requests went to `git_add` first. The rewrite that stressed "commits *staged* changes" measured worse, 4→2, and was rejected. |
+| `@modelcontextprotocol/server-filesystem` | 14 | **77/140** | `read_file` 0/10 — its description says *DEPRECATED, use read_text_file* and the model obeys (`scan` now flags this). Thirty-plus requests across seven tools went to `list_allowed_directories` first. |
+
+The git and filesystem results share a cause that a description can't fix: **the model takes a
+correct precondition step** — stage before commit, check allowed directories before touching a
+path — and a single-step eval scores it as the wrong tool. That column in the confusion matrix is
+the finding; it tells you which tools need their precondition stated ("paths are validated for
+you") or a multi-step harness. toolfit does not paper over it by grading the first call leniently.
+
+Twelve rewrites were proposed for the filesystem server. Five improved the number (`8→10`,
+`6→8`); none were accepted, because one Bonferroni correction across twelve proposals at n=10
+sets α=0.004 and the report says so. Run `--fix-tool list_directory --fix-tool move_file
+--seeds 20` on the tools the matrix names, not `--fix` on the whole catalog at once.
+
+## Two commands, two budgets
+
+| | `scan` | `eval` |
+|---|---|---|
+| What | Static lint over `tools/list`: missing, too-short, and duplicated descriptions | Live model behaviour: which tool it calls, with which arguments, for a request that should lead to each tool |
+| Cost | Free — no model calls, no key | Your API key: roughly `tools × seeds × 3` calls, plus `seeds` per proposed fix |
+| Time | Under a second after the server starts | Minutes |
+| Output | Findings list (never a letter grade) | Confusion matrix, per-tool pass rates with 95% CIs, mutation/fix verdicts, optional badge and `toolfit-fixes.json` |
+
+They are deliberately separate. `scan` is the zero-config front door; on mature servers it finds
+little ([1 finding across 166 tools on 15 public servers](docs/corpus.md)) because the bug it
+catches is the copy-paste class. The confusion is what `eval` is for.
+
+## Install
+
+```
+uvx toolfit --help          # zero-install, one-off
+pipx install toolfit        # persistent / CI
+```
+
+Python 3.10+. Talks to servers over the MCP protocol via the official `mcp` SDK, so the
+server can be in any language.
+
+## Pointing it at a server
+
+```
+toolfit scan path/to/server.py                                  # run via `uv run`
+toolfit scan "npx -y @modelcontextprotocol/server-filesystem ."   # any command line
+toolfit scan https://your-host/mcp                              # Streamable HTTP
+```
+
+The subprocess gets your environment (minus toolfit's own `*_API_KEY`s, which a third-party server
+binary has no business seeing), so servers that read a token from `GITHUB_TOKEN` or
+`STRIPE_SECRET_KEY` work unchanged. toolfit never calls a tool — it only ever asks for the
+catalog (`tools/list`), then asks a model what it *would* call. Nothing touches your backend.
+
+## How the measurement works
+
+The credibility problem with LLM-generated evals is circularity: if one model writes the task
+and also decides the expected answer, you are measuring agreement with that model. toolfit
+inverts it:
+
+1. Sample a concrete, schema-valid argument set for one tool (`gen/schema_sampler.py`),
+   honouring `required`, enums, formats, and nullables.
+2. Ask a generator model to write the sentence a user would type that leads to *exactly those
+   arguments*, without naming the tool. Ground truth is the sampled tuple, not the model's opinion.
+3. Send that sentence plus the whole catalog to the model under test.
+4. Grade structurally (`grade/grader.py`): right tool, and arguments equal after canonicalising
+   dates, case, whitespace, and array order. No LLM judge, ever.
+
+Guardrails: a second pass checks each task for tool-name leakage and for solvability against
+the catalog; both are reported as warnings, never silently dropped. Tools whose schema the
+sampler can't handle are excluded and listed — a partial number is never printed as a complete one.
+
+**Mutation testing** is the same grader run twice. `--mutate 'tool:new description'` re-runs a
+tool's *own* base tasks against a catalog where only that description is patched (protocol-level;
+your source is never touched) and compares pass rates on paired trials. `--fix` does the same with
+a proposed rewrite for every failing tool.
+
+**Significance** is an exact one-sided McNemar test on the discordant pairs, Bonferroni-corrected
+across everything re-measured in one run. It is deterministic and honest at small n: with 5 seeds
+the smallest attainable p-value is 1/32, so `--seeds 10` is the practical floor for a verdict —
+the CLI warns if you go lower. Every rate carries `n` and a Wilson 95% interval.
+
+## CI
+
+```yaml
+- uses: sreshtalluri/toolfit@main
+  with:
+    server: "npx -y @modelcontextprotocol/server-github"
+    eval: true                      # omit for the free scan only
+    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+`scan --strict` exits 1 on any finding; `eval --strict` exits 1 if any tool's pass rate is below
+`--strict-threshold` (default 0.9). Tools excluded by a schema warning are named on stderr and
+*not* counted — they don't fail the gate, but you'll see them.
+
+`--badge` writes `toolfit-badge.svg` with the pass rate (or the before→after delta of a single
+mutation or accepted fix), coloured by rate, with the model, generator, seed count, and a hash of
+the task suite embedded so the number is never separable from what produced it.
+
+## Models
+
+`--model` picks the model under test and the provider is inferred from the name: `claude*` →
+Anthropic, `gpt*`/`o*` → OpenAI, `vendor/model` → OpenRouter. Keys come from `ANTHROPIC_API_KEY`,
+`OPENAI_API_KEY`, `OPENROUTER_API_KEY`. Task generation and fix proposals always use Anthropic, so
+that key is required for `eval` regardless of `--model`.
+
+**Data handling.** The tool catalog and the generated task text are sent to whichever provider you
+configure, with your key. toolfit itself stores nothing and phones nowhere. If your server is
+internal, that is the one place its schema leaves your machine.
+
+## Landscape
+
+| Tool | What it does | Where it stops |
+|---|---|---|
+| mcpgrade | Static lint + single-step eval, A–F grade | No re-measured fix |
+| mcpx | ESLint-style schema lint, CI gate | Never runs a model |
+| MCProbe | Usability rules + input fuzzing | Fuzzing, not task performance |
+| lastmile-ai/mcp-eval | Assertion framework with LLM judges | You write the tests |
+| MCPJam Inspector | Hosted evals with cross-model comparison | GUI product, not a CLI |
+| MCP-Atlas / MCP-Bench | Benchmarks ranking *models* | Not pointable at your server |
+
+toolfit is not a model leaderboard and does not test for prompt injection. One server per run.
+
+## Development
+
+```
+uv sync --extra dev
+uv run pytest -q
+uv run toolfit scan examples/toy_server.py
+```
+
+`examples/toy_server.py` has two deliberately confusable tool pairs — `create_task`/`update_task`
+share a description, `list_tasks`/`count_tasks` share a vague one — so the fix loop has something
+real to find. Design history and the methodology decisions behind every number are in
+[`docs/designs/toolfit-v0-scope.md`](docs/designs/toolfit-v0-scope.md).
+
+MIT.

toolfit-0.1.0/TODOS.md

@@ -0,0 +1,16 @@
+# TODOS
+
+## eval
+
+- **Multi-step grading (source doc M5).** On real servers the dominant off-diagonal mass is precondition calls (`git_add` before `git_commit`, `list_allowed_directories` before any path op) — correct behaviour a single-step grader scores as confusion. Grade the sequence, or let a tool declare its preconditions. **Priority:** P1
+- **Separate `(error)` column.** max_tokens truncation, malformed tool-call JSON, and empty `choices` are tallied with genuine no-calls. Add `error` to `ToolCall` and a column so a flaky provider doesn't read as a confusing catalog. **Priority:** P2
+- **Non-Anthropic model under test on the corpus.** `OPENROUTER_API_KEY` was empty during the scenario runs; a small model is far likelier to confuse the toy pairs and give `--fix` something to accept. **Priority:** P2
+- **`--strict` on schema-excluded tools.** Currently warn-only (decided 2026-09-05). Revisit if a real CI user is surprised by a green gate on an unevaluated server. **Priority:** P3
+
+## gen
+
+- **Constraints the schema can't express.** `read_text_file` accepts `head` OR `tail`, not both; the sampler draws both 25% of the time and the solvability check rejects the task. Consider honouring `oneOf`/`not` at the object level, or a per-server task-rejection budget. **Priority:** P3
+
+## Completed
+
+- v0.1.0 (2026-09-05): scan, eval, mutation testing, fix loop, badge, strict, generic launch, Action, corpus, three real-server scenarios.