mispick 0.1.0__tar.gz

mispick-0.1.0/.github/ISSUE_TEMPLATE/benchmark-rerun.md

@@ -0,0 +1,27 @@
+---
+name: Request a benchmark re-run
+about: Ask for a server to be re-measured, or point out a result that looks wrong
+title: 'Re-run: <server name>'
+labels: benchmark
+---
+
+**Server**
+<!-- The registry name as it appears on the leaderboard, e.g. io.github.owner/repo -->
+
+**Why**
+<!-- Pick whichever applies:
+ - the descriptions changed since the snapshot was captured
+ - the snapshot is missing tools, or has tools that no longer exist
+ - the result looks wrong for a reason you can name
+ - the server was skipped but does start without credentials
+-->
+
+**Anything we should know about running it**
+<!-- e.g. it needs a flag, or it only exposes its full tool list after some setup -->
+
+---
+
+A reminder on what the benchmark is: one model, one seed, one day, reading only
+`initialize` and `tools/list`. A low score is a finding about how that model reads a tool
+surface, not a judgement of the server. If you think the framing on your server's page is
+unfair, say so here and we will fix the wording.

mispick-0.1.0/.github/workflows/benchmark.yml

@@ -0,0 +1,76 @@
+name: benchmark
+
+on:
+  workflow_dispatch:
+    inputs:
+      limit:
+        description: 'How many servers to pick from the registry'
+        default: '30'
+      refresh-snapshots:
+        description: 'Re-capture tool lists from the registry (otherwise use the committed ones)'
+        type: boolean
+        default: false
+  schedule:
+    # Weekly, not nightly: tool surfaces do not change that fast, and every run starts other
+    # people's servers.
+    - cron: '17 4 * * 1'
+
+permissions:
+  contents: read
+
+concurrency:
+  group: benchmark
+  cancel-in-progress: false
+
+jobs:
+  benchmark:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v10.2.0
+        with:
+          python-version: '3.12'
+          enable-cache: true
+      - run: uv sync
+
+      - name: Pick servers
+        if: ${{ inputs.refresh-snapshots }}
+        run: uv run python benchmark/pick.py --limit "${{ inputs.limit || '30' }}"
+
+      - name: Capture tool lists
+        if: ${{ inputs.refresh-snapshots }}
+        run: uv run python benchmark/capture.py
+
+      - name: Install Ollama
+        run: |
+          curl -fsSL https://ollama.com/install.sh | sh
+          ollama serve &
+          for _ in $(seq 1 30); do
+            curl -sf http://localhost:11434/api/tags >/dev/null && break
+            sleep 1
+          done
+          ollama pull qwen3.5:4b
+
+      - name: Measure
+        run: |
+          uv run python benchmark/measure.py \
+            --snapshots benchmark/snapshots \
+            --out benchmark/results \
+            --model ollama/qwen3.5:4b \
+            --queries 8 --runs 3 --temperature 0 --seed 1 --concurrency 4
+
+      - name: Build the site
+        run: uv run python benchmark/build_site.py
+
+      - uses: actions/upload-artifact@v7
+        with:
+          name: benchmark
+          path: |
+            benchmark/results
+            benchmark/site
+
+      # Publishing is a separate, manually reviewed step: the site names other people's
+      # projects, so a bad run must not go live on a schedule.
+      - uses: actions/upload-pages-artifact@v5
+        with:
+          path: benchmark/site

mispick-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,72 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+        python: ['3.11', '3.12', '3.13']
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v10.2.0
+        with:
+          python-version: ${{ matrix.python }}
+          enable-cache: true
+      - run: uv sync --all-extras
+      - name: Lint
+        run: uv run ruff check src tests
+      - name: Types
+        run: uv run mypy src
+      - name: Tests
+        run: uv run pytest -q
+      # The suite must pass with no network and no model backend. This is the check that
+      # keeps that true.
+      - name: Tests are offline
+        run: uv run pytest -q
+        env:
+          OPENAI_API_KEY: ''
+          ANTHROPIC_API_KEY: ''
+          # Point outbound HTTP at a dead port, so a test that reaches the network fails
+          # here instead of passing quietly because the runner happens to be online.
+          # NO_PROXY='*' would do the opposite: it permits direct connections.
+          HTTP_PROXY: 'http://127.0.0.1:9'
+          HTTPS_PROXY: 'http://127.0.0.1:9'
+
+  workflows:
+    # Two failure modes that arrive with no job and no log, so they are checked here:
+    # a workflow file GitHub rejects outright (e.g. two env keys differing only in case),
+    # and a `uses:` pin whose tag does not exist, which fails job setup.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: docker://rhysd/actionlint:latest
+        with:
+          args: -color
+      - name: Every action pin resolves
+        run: python3 scripts/check_action_pins.py
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+  action:
+    # Proves action.yml at least parses and skips cleanly with no model key.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: ./
+        with:
+          snapshot: tests/fixtures/confusing_tools.json
+          comment: 'false'
+      - name: The run should have skipped, not failed
+        run: |
+          test -f mispick-comment.md
+          grep -q "skipped" mispick-comment.md

mispick-0.1.0/.github/workflows/pages.yml

@@ -0,0 +1,41 @@
+name: pages
+
+# Deliberately manual. The leaderboard names other people's projects, so a run is reviewed
+# before it is published - a scheduled deploy could put a bad measurement in front of them.
+on:
+  workflow_dispatch:
+    inputs:
+      run-id:
+        description: 'The benchmark workflow run whose site artifact should be published'
+        required: true
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+jobs:
+  deploy:
+    environment:
+      name: github-pages
+      url: ${{ steps.deploy.outputs.page_url }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - name: Download the reviewed site
+        uses: actions/download-artifact@v8
+        with:
+          name: benchmark
+          run-id: ${{ inputs.run-id }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          path: downloaded
+      - uses: actions/configure-pages@v6
+      - uses: actions/upload-pages-artifact@v5
+        with:
+          path: downloaded/site
+      - id: deploy
+        uses: actions/deploy-pages@v5

mispick-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,89 @@
+name: release
+
+# Publishes to PyPI via Trusted Publishing - no API token is stored in this repo.
+# Set up once at https://pypi.org/manage/account/publishing/ with:
+#   owner: msc35   repository: mispick   workflow: release.yml   environment: pypi
+on:
+  push:
+    tags: ['v*']
+  workflow_dispatch:
+    inputs:
+      target:
+        description: 'Where to publish'
+        type: choice
+        options: [testpypi, pypi]
+        default: testpypi
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v10.2.0
+        with:
+          python-version: '3.12'
+
+      # Never ship a release that does not pass its own tests.
+      - run: uv sync --all-extras
+      - run: uv run ruff check .
+      - run: uv run mypy src
+      - run: uv run pytest -q
+
+      - name: Check the tag matches the version
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: |
+          version=$(uv run python -c 'import mispick; print(mispick.__version__)')
+          tag="${GITHUB_REF_NAME#v}"
+          if [ "$version" != "$tag" ]; then
+            echo "::error::tag $tag does not match mispick.__version__ $version"
+            exit 1
+          fi
+          echo "tag $tag matches version $version"
+
+      - run: uv build
+
+      # Prove the built wheel actually runs before it is published, rather than after.
+      - name: Smoke-test the wheel
+        run: |
+          uvx --from ./dist/*.whl mispick version
+          uvx --from ./dist/*.whl mispick run \
+            --snapshot tests/fixtures/confusing_tools.json --model mock -n 2 -k 1 \
+            --cache-dir "$(mktemp -d)"
+
+      - uses: actions/upload-artifact@v7
+        with:
+          name: dist
+          path: dist/
+
+  testpypi:
+    needs: build
+    if: github.event_name == 'workflow_dispatch' && inputs.target == 'testpypi'
+    runs-on: ubuntu-latest
+    environment: testpypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v8
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://test.pypi.org/legacy/
+
+  pypi:
+    needs: build
+    if: startsWith(github.ref, 'refs/tags/v') || (github.event_name == 'workflow_dispatch' && inputs.target == 'pypi')
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v8
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

mispick-0.1.0/.gitignore

@@ -0,0 +1,18 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.toolconf/
+*-badge.svg
+.DS_Store
+benchmark/site/
+benchmark/results/
+mispick-report.*
+mispick-comment.md
+mispick-badge.svg

mispick-0.1.0/CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+
+## 0.1.0 — unreleased
+
+First release. Not yet published to PyPI.
+
+- Load tools from a stdio server, a Streamable HTTP server, a captured snapshot, or every server
+  in a `claude_desktop_config.json` / `.mcp.json` at once. Only `initialize` and `tools/list` are
+  ever sent; `tools/call` is unreachable by construction and by test.
+- Generate a test set per tool (straightforward, paraphrased, hard negatives against the nearest
+  neighbour, plus "no tool fits" cases), cached to an editable `.mispick/queries.yaml` keyed on a
+  per-tool fingerprint.
+- Measure selection accuracy, per-tool precision and recall, a confusion matrix with `(none)` and
+  `(phantom)` rows, argument validity against `inputSchema`, stability across K runs, over-
+  triggering, and a tool-list token estimate. Every rate carries a Wilson 95% interval.
+- `mispick fix`: rewrite the worst confused pairs, re-run the same queries, and accept a rewrite
+  only when an exact McNemar test agrees. Rejected rewrites are shown.
+- Cross-server analysis: name collisions plus measured leakage between servers.
+- Reports: terminal, JSON (versioned schema), Markdown for PR comments, a self-contained HTML page,
+  and an SVG badge. `mispick report` re-renders a saved run; `mispick compare` diffs two runs.
+- Backends: Ollama (default, local, free), OpenAI, Anthropic, and a deterministic mock.
+- GitHub Action with a sticky PR comment, base-branch comparison and `--fail-under`.
+- Benchmark scripts and a static leaderboard generator.

mispick-0.1.0/CLAUDE.md

@@ -0,0 +1,29 @@
+# CLAUDE.md
+
+Read `SPEC.md` fully before doing anything. It is the source of truth.
+
+## How to work on this repo
+- Work milestone by milestone (SPEC section 10). Start with **M0 research**. Do not write product code until `docs/research.md` exists and the user has reviewed it.
+- After each milestone: run tests, update README, commit, and stop to summarize what changed and what's next.
+- If research shows a competitor already does the four differentiators in SPEC section 2, stop and report. Don't build a duplicate.
+- Ask before adding any dependency not listed in SPEC section 5.
+
+## Non-negotiables
+- **Never call `tools/call` on any MCP server.** Only `initialize` and `tools/list`. There's a test for this. Keep it passing.
+- Default model backend is local Ollama. Tests use a mocked backend and must pass offline.
+- Never modify the user's MCP server source. Fix mode only suggests.
+- Reports always state model, N, K, and date.
+
+## Commands
+- `uv sync --all-extras` — install. **Use `--all-extras`**: plain `uv sync` leaves the
+  `anthropic` extra out, and `[[tool.mypy.overrides]]` ignores its imports when missing, so
+  `mypy` silently skips `models/anthropic.py`. That hid a backend that raised `TypeError` on
+  every call.
+- `uv run pytest` — tests
+- `uv run ruff check . && uv run mypy src` — lint and types
+- `uv run mispick run --snapshot tests/fixtures/confusing_tools.json` — quick manual check
+
+## Style
+- Python 3.11+, type hints everywhere, pydantic models for data.
+- Small modules, small functions. No frameworks beyond SPEC section 5.
+- Clear error messages that tell the user what to do next.

mispick-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,60 @@
+# Contributing
+
+```bash
+uv sync --all-extras          # --all-extras matters, see below
+uv run pytest
+uv run ruff check . && uv run mypy src
+```
+
+**Always `--all-extras`.** Plain `uv sync` leaves the optional `anthropic` dependency out, and
+mypy is configured to ignore its imports when absent — so `models/anthropic.py` goes unchecked
+and nothing tells you. That is exactly how a backend that passed `temperature` to an API which
+accepts no `temperature`, and would have raised `TypeError` on every call, survived a green
+local run. CI uses `--all-extras` and caught it on the first push.
+
+Workflows are linted too; `ruff` does not read them:
+
+```bash
+actionlint                              # brew install actionlint
+python3 scripts/check_action_pins.py    # every `uses:` tag actually exists
+```
+
+Both guard against failures that arrive with **no job and no log**, which are miserable to debug:
+
+- A workflow with two `env` keys differing only in case is rejected by GitHub wholesale, giving a
+  zero-second run and the message "likely a workflow file issue".
+- A `uses:` pin whose tag does not exist fails *job setup*, before any step runs. Note that not
+  every repository publishes moving major tags — `actions/*` do, but `astral-sh/setup-uv` and
+  `marocchino/sticky-pull-request-comment` publish only exact versions, so reading the latest
+  release name is not enough. `release.yml` and `pages.yml` only run at release time, so a stale
+  pin there would surface at the worst possible moment.
+
+## Things the tests enforce, so a reviewer does not have to
+
+**mispick never calls a tool.** `tests/test_never_calls_tools.py` fails the build if any module
+under `src/mispick/` so much as references `call_tool` or the string `tools/call`, if anything
+returned by `sources/` exposes a client, or if a tool load succeeds with `Client.call_tool`
+monkeypatched to raise. This is the promise that makes mispick safe to point at a destructive
+server. If you need to change it, you need a different tool.
+
+**The suite runs offline.** No network, no Ollama, no API keys. Use `--model mock`. CI re-runs the
+suite with the key environment variables blanked to keep this true.
+
+**Every report states the model, N, K and the date**, and carries the caveat that results depend on
+the model. There are tests for this in every format, including one that renders the terminal report
+at 52 columns to make sure the date is not truncated.
+
+**Unmeasured rates are `null`, not `0`.** A K=1 run has no stability; reporting 0% would be a lie.
+
+**Benchmark wording stays neutral.** "Most confused pair", never "worst server". Tested.
+
+## Style
+
+Python 3.11+, type hints everywhere, pydantic for data, small modules. Error messages say what to
+do next, not just what went wrong. `ruff` and `mypy --strict` clean.
+
+## Adding a model backend
+
+Implement `mispick.models.base.Backend` (`choose` and `generate`), register it in
+`models/registry.py`, and set `supports_seed` honestly — reports use it to decide whether to claim
+a run is reproducible.

mispick-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 mispick contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mispick-0.1.0/PKG-INFO

@@ -0,0 +1,217 @@
+Metadata-Version: 2.5
+Name: mispick
+Version: 0.1.0
+Summary: Find out which of your MCP tools the model mixes up - measured, not guessed.
+Project-URL: Homepage, https://github.com/msc35/mispick
+Project-URL: Repository, https://github.com/msc35/mispick
+License: MIT
+License-File: LICENSE
+Keywords: confusion-matrix,evaluation,llm,mcp,model-context-protocol,tools
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: jsonschema>=4.20.0
+Requires-Dist: mcp>=2.2.0
+Requires-Dist: openai>=1.60.0
+Requires-Dist: pydantic>=2.12.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.9.4
+Requires-Dist: typer>=0.16.0
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.40.0; extra == 'anthropic'
+Description-Content-Type: text/markdown
+
+# mispick
+
+**Find out which of your MCP tools the model mixes up — measured, not guessed.**
+
+<p align="center">
+  <img src="docs/media/demo.svg" alt="mispick's terminal report: a confusion matrix showing search_docs and search_issues being mistaken for each other, per-tool precision and recall with confidence intervals, and a 0-100 score." width="900">
+</p>
+
+```bash
+uvx mispick run --cmd "python -m my_server"
+```
+
+Point it at any MCP server. It generates realistic user requests, asks a model to pick a tool for
+each one, and shows you a **confusion matrix**: which tools get chosen for which intent. Then it
+proposes description rewrites for the confused pairs and **re-runs to prove the fix helped**.
+
+It never calls a tool. Only `initialize` and `tools/list`, so it is safe to point at a server that
+deletes things.
+
+---
+
+## Why this rather than a static linter
+
+`mcp-lint`, `mcpolish`, `oh-my-mcp`, `mcp-conform` and `mcp-cve-lint` check tool definitions
+against rules, and they are good at it — run one. But a description can pass every rule and still
+be indistinguishable from the tool next to it. Rule-based linters predict problems; this measures
+them, by putting your tool list in front of a real model and counting what it picks. Two other
+projects do measure selection — [`whichtool`](https://github.com/mattagame/whichtool) (local
+models, but no rewrites, one server per run) and [`toolfit`](https://pypi.org/project/toolfit/)
+(rewrites, but cloud-only, one server per run). Neither looks across servers. The full comparison,
+with what each one actually does, is in [docs/research.md](docs/research.md).
+
+## How it works
+
+1. Connect to your server and read `tools/list` (never `tools/call`).
+2. For each tool, generate N realistic requests — plus **hard negatives** written to sit as close
+   as possible to its nearest neighbour, and some requests no tool should answer.
+3. Ask the model to choose a tool for each request, K times, showing it the real tool definitions
+   a client would send.
+4. Score it: accuracy, per-tool precision and recall, a confusion matrix, argument validity,
+   stability across the K runs — every rate with a Wilson 95% interval.
+5. Propose rewrites for the worst pairs, re-run the same requests against them, and keep only the
+   ones that measurably won.
+
+## Install
+
+> **Not on PyPI yet.** `mispick` has not been released, so the commands below do not work
+> today. Until then, run it from a checkout:
+>
+> ```bash
+> git clone https://github.com/msc35/mispick && cd mispick
+> uv sync --all-extras
+> uv run mispick --help
+> ```
+
+```bash
+uvx mispick --help          # no install
+uv tool install mispick     # or keep it
+pip install mispick
+```
+
+The default backend is a local model through [Ollama](https://ollama.com), so it costs nothing and
+needs no account:
+
+```bash
+ollama pull qwen3.5:4b
+```
+
+Cloud models are optional: `--model openai/gpt-4.1-mini` or
+`--model anthropic/claude-haiku-4-5-20251001`, with the matching key in the environment.
+
+## Use
+
+```bash
+# measure a local server, a remote one, or a captured snapshot
+mispick run --cmd "python -m my_server"
+mispick run --url https://example.com/mcp
+mispick run --snapshot tools.json
+
+# capture a snapshot so later runs are offline and reproducible
+mispick snapshot --cmd "python -m my_server" -o tools.json
+
+# every server in your config at once, and what they do to each other
+mispick run --config ~/Library/Application\ Support/Claude/claude_desktop_config.json
+
+# rewrites for the worst pairs, each one re-tested
+mispick fix --cmd "python -m my_server" --patch fix.diff
+
+# reports and CI
+mispick run --snapshot tools.json --format html --out report.html --badge badge.svg
+mispick run --snapshot tools.json --fail-under 80          # exit 1 below the threshold
+mispick compare base.json head.json --max-drop 5           # exit 1 on a regression
+```
+
+Generated requests are cached in `.mispick/queries.yaml`. **Read that file and edit it** — it is
+your test set, and it only regenerates for a tool whose name, description or schema changed.
+Commit it.
+
+### Proven fixes
+
+`mispick fix` rewrites both descriptions in a confused pair, then re-runs the *same* requests
+against the rewrite and reports before/after with an exact McNemar test — the trials are paired,
+so an unpaired test would be the wrong one. Rewrites that fail are shown too.
+
+That re-test is the whole point. Hasan et al. rewrote tool descriptions with an LLM across 231
+tasks and found success rose by a median 5.85 points — but **regressed in 16.67% of cases**
+([arXiv 2602.14878](https://arxiv.org/abs/2602.14878)). An unmeasured rewrite is a one-in-six
+chance of making things worse. mispick also caps rewrite length, so a rewrite cannot win by
+crowding out its neighbours, and it never edits your source — it prints a suggestion.
+
+**Expect rejections, especially on a small local model.** In our own runs against
+`ollama/qwen3.5:4b`, two of three proposed rewrites measurably made selection *worse* — one of them
+took a tool from 78% to 58% accuracy, breaking 14 trials and fixing none, and it replicated when
+re-run with three times the data. Its stated reasoning read perfectly sensibly. The third improved
+by 10 points but over only two changed trials, which is not enough to call, so mispick reported it
+as inconclusive rather than banking it. A 4B model is good enough to *find* confusion and often not
+good enough to *write its way out of it*; fix mode is more productive with a larger model. Either
+way the verdicts are the product — a run where nothing is accepted has told you something true.
+
+### Across servers
+
+Point `--config` at a whole `claude_desktop_config.json` and mispick loads every server at once,
+because that is how a client sees them. It reports which bare tool names collide across servers
+and — the part that matters — how often a request for one server's tool is answered by another
+server's. On the two-server fixture, 17% of trials cross servers and `workspace:search_docs` falls
+to 0% recall: every request for it is answered by `wiki:search_docs`. The MCP spec names this
+problem and warns that `serverInfo.name` is not unique across servers, so mispick keys on your
+config label instead.
+
+### In CI
+
+```yaml
+- uses: actions/checkout@v4
+- uses: msc35/mispick@v0.1.0
+  with:
+    snapshot: tools.json
+    model: anthropic/claude-haiku-4-5-20251001
+    fail-under: '80'
+    compare-base: 'true'
+  env:
+    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+It posts the report as a sticky PR comment, and with `compare-base` the comment reads
+`84 → 91 (▲7)` and names the confusions that appeared or disappeared. With no model key it skips
+with an explanation instead of failing — a check that goes red over a missing secret is a check
+people learn to ignore. See [examples/github-action](examples/github-action/).
+
+## Benchmark
+
+`benchmark/` runs the pipeline over servers from the official MCP registry that start without
+credentials, and builds a static leaderboard. Snapshots are committed so results are reproducible,
+only `initialize` and `tools/list` are ever sent, and the page lists the servers it could *not*
+measure so the coverage is not overstated. See [benchmark/README.md](benchmark/README.md).
+
+## Limitations
+
+Read these before quoting a number.
+
+- **Results depend on the model.** A score is one model's reading of one tool surface on one day,
+  not a property of your server. Every report says so, and states the model, N, K and the date.
+- **Small local models are noisy.** That is why every rate carries a Wilson 95% interval and why
+  `--runs` exists. With K=3 the intervals are wide; read them, not the point estimate.
+- **The requests are generated, so they are not your traffic.** They are a stand-in for it. Edit
+  `.mispick/queries.yaml` to make them yours.
+- **It measures one selection turn.** Not multi-step agent behaviour, not tool output quality, not
+  recovery after a bad call. Arguments are checked against `inputSchema` and no further.
+- **A low score can be correct.** Two tools may genuinely overlap. mispick tells you the model
+  cannot separate them; whether they *should* be separable is your call.
+- **`mispick fix` often accepts nothing**, and that is working as intended. See the note above.
+- **Token counts are estimates**, and lower bounds — a real tokenizer and the provider's framing
+  both add to them.
+- **Reasoning models** spend heavily on hidden thinking. mispick turns thinking off by default for
+  Ollama (measured: 89s → 2.2s per call on `qwen3.5:4b`, same answers); `--think` restores it.
+
+## Development
+
+```bash
+uv sync
+uv run pytest                         # offline; no model or network needed
+uv run ruff check . && uv run mypy src
+uv run mispick run --snapshot tests/fixtures/confusing_tools.json --model mock
+```
+
+`--model mock` is a deterministic offline backend. The test suite uses it, so `pytest` passes with
+no Ollama, no keys and no network.
+
+## License
+
+MIT. See [LICENSE](LICENSE).