toolmenu 0.27.0 → 0.27.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -5,7 +5,7 @@
5
5
  <h1 align="center">toolmenu</h1>
6
6
 
7
7
  <p align="center">
8
- <b>Tests your MCP server the way agents and clients meet it: the menu they read, the calls they make, the results and sign-in they get back, and every release after. Problems caught before your users hit them.</b>
8
+ <b>Tests your MCP server the way agents and clients meet it: the menu they read, the calls they make, the results and sign-in they get back, and every release after.</b>
9
9
  </p>
10
10
 
11
11
  <p align="center">
@@ -15,42 +15,64 @@
15
15
  [![license: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/niksa90/toolmenu/blob/main/LICENSE)
16
16
  [![MCP: 2026-07-28 and 2025](https://img.shields.io/badge/MCP-2026--07--28%20%C2%B7%202025-6E56CF)](https://modelcontextprotocol.io)
17
17
  [![toolmenu](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Fniksa90%2Ftoolmenu%2Fmain%2Fdocs%2Ftoolmenu-badge.json)](https://github.com/niksa90/toolmenu)
18
- [![toolmenu MCP server on Glama](https://glama.ai/mcp/servers/niksa90/toolmenu/badges/score.svg)](https://glama.ai/mcp/servers/niksa90/toolmenu)
19
- [![toolmenu MCP on mcprush](https://mcprush.com/badge/toolmenu-mcp/score.svg)](https://mcprush.com/to/toolmenu-mcp?ref=badge)
20
- [![Listed on mcpservers.org](https://mcpservers.org/badge.svg)](https://mcpservers.org/servers/niksa90/toolmenu)
18
+ [![toolmenu MCP server on Glama](https://glama.ai/mcp/servers/niksa90/toolmenu/badges/score.svg)](https://glama.ai/mcp/servers/niksa90/toolmenu)
19
+ [![toolmenu MCP on mcprush](https://mcprush.com/badge/toolmenu-mcp/score.svg)](https://mcprush.com/to/toolmenu-mcp?ref=badge)
20
+ [![Listed on mcpservers.org](https://mcpservers.org/badge.svg)](https://mcpservers.org/servers/niksa90/toolmenu)
21
+
22
+ ## License
23
+
24
+ MIT
25
+
21
26
 
22
27
  </p>
23
28
 
24
- Your MCP server's tool list is the menu every agent reads before it does anything.
25
- toolmenu starts there and follows the agent in. It **connects** as a client does (stdio
26
- or HTTP, every protocol version from 2024-11-05 to 2026-07-28), **reads** the menu,
27
- **calls** the tools (read-only by default; every tool, writes included, in a sealed
28
- sandbox with `--call-everything`), **checks** each result against the spec and the tool's
29
- own `outputSchema`, **walks** the OAuth discovery a client follows before it logs in, and
30
- **compares** the menu across steps, connections, processes, clients and releases.
29
+ An MCP server can pass its own tests and still fail in the client. The client's SDK
30
+ rejects a tool list the server's own test client accepted. A default built from the
31
+ clock makes every listing different, so no prompt cache ever matches. A patch release
32
+ makes a parameter required. None of it shows until someone's agent has fewer tools, or
33
+ a bigger bill, and no error to explain it.
31
34
 
32
- It isn't only a linter: a linter grades the text of one snapshot, and much of what
33
- breaks agents only shows when you call the server, reconnect, or ship the next version.
34
- No LLM is involved, so the same server gives the same answer on every run and it fits
35
- in CI. The same checks run [every week on ~3,200 public servers](#tested-on-real-servers).
35
+ toolmenu does what a client does, in order, and checks each step against the spec, the
36
+ tool's own schemas and each client's documented limits:
37
+
38
+ ![What a client does with an MCP server, and what toolmenu checks at each step: sign-in, the connection, the menu, tool calls, the menu during a conversation, and the next release](https://raw.githubusercontent.com/niksa90/toolmenu/main/docs/overview.svg)
39
+
40
+ No LLM is involved, so the same server gives the same findings on every run and it fits
41
+ in CI. It doesn't tell you whether an agent picks the right tool; that takes evals with a
42
+ model ([what it can't catch](#what-it-cant-catch)).
36
43
 
37
44
  ```sh
38
45
  npx toolmenu snapshot -- node dist/server.js # or: npx toolmenu snapshot https://example.com/mcp
39
46
  npx toolmenu init -- node dist/server.js # CI: a baseline, a workflow, a comment on every PR
40
47
  ```
41
48
 
49
+ Needs Node 22 or later. Your server can be in any language: toolmenu starts it (stdio)
50
+ or connects to it (HTTP).
51
+
42
52
  ![toolmenu session output: tools inserted mid-list, a description edited, each pinned to the step that caused it](https://raw.githubusercontent.com/niksa90/toolmenu/main/docs/demo.svg)
43
53
 
44
- > **Status: 0.27.** Spec in [docs/SPEC.md](https://github.com/niksa90/toolmenu/blob/main/docs/SPEC.md),
45
- > every rule in [docs/RULES.md](docs/RULES.md), protocol versions in [docs/PROTOCOLS.md](docs/PROTOCOLS.md)
46
- > (written by a test that runs each one), changes in [CHANGELOG.md](CHANGELOG.md).
54
+ **Contents:** [What it catches](#what-it-catches) · [A finding](#what-a-finding-looks-like) ·
55
+ [In CI](#in-ci-one-pr-comment) · [Commands](#commands) · [Tested on real servers](#tested-on-real-servers) ·
56
+ [What it can't catch](#what-it-cant-catch) · [Docs](#docs)
57
+
58
+ ## What it catches
59
+
60
+ Each row is something a run on real servers found
61
+ ([docs/FINDINGS.md](https://github.com/niksa90/toolmenu/blob/main/docs/FINDINGS.md) has the versions and how to reproduce each).
47
62
 
48
- **Contents:** [A finding](#what-a-finding-looks-like) ·
49
- [What it catches](#what-it-catches) · [Why "when it changes" matters](#why-when-it-changes-matters) ·
50
- [Tested on real servers](#tested-on-real-servers) · [Start here](#start-here) ·
51
- [CI](#in-ci-one-pr-comment) · [`session`](#session-the-menu-changing-while-the-agent-works) ·
52
- [`diff`](#diff-compare-releases) · [`review`](#accepting-a-new-baseline-review) · [`snapshot` rules](#snapshot-rules) ·
53
- [What it can't catch](#what-toolmenu-cant-catch) · [Options](#options)
63
+ | | What goes wrong | Seen on |
64
+ |---|---|---|
65
+ | **Lists and results a client rejects** | The official SDK client validates `tools/list` and every result. One bad schema and it drops the whole list; an answer that breaks the tool's own `outputSchema` becomes an error | 12 of 1,089 registry servers: 8 lose every tool, 4 lose answers |
66
+ | **A menu that never matches twice** | A default built from the clock, an order from a hash set, a container ID in a description. Every listing or every restart is a new menu, so cached prompts miss | PayPal (`list_transactions` dates), mcp-atlassian, Desktop Commander |
67
+ | **A menu that changes mid-conversation** | Most clients put tools first in the prompt, so a tool added by an "unlock" call means the conversation after it is processed again. 2026-07-28 rules per-connection menus out | One unlock re-processed 40–57K cached tokens in the author's own logs |
68
+ | **Releases that break callers** | Removed tools, new required parameters, narrowed types, a tool that became destructive, with the version bump the change needs | The official filesystem server made `move_file` destructive in a calendar release; Exa removed tools in patch releases |
69
+ | **Sign-in a strict client can't finish** | OAuth discovery that differs from the RFCs: a `resource` that isn't the server's URL, metadata that isn't there, PKCE not advertised | 12 of 79 vendor endpoints that answer 401 name a parent or slash-variant `resource` |
70
+ | **Tools one client can't use** | More tools than the client sends, names its API refuses or rewrites, descriptions cut before the instruction that matters | 6 servers in Docker's catalog list more than VS Code's 128 a request |
71
+ | **Tokens spent on nothing** | Unused `$defs`, the same schema block repeated across tools, growth per release | Notion's server: the same 9 unused `$defs` in all 24 tools, 72% of its menu |
72
+
73
+ It also has heuristics for what agents misread (a parameter called just `id`, writes
74
+ with no annotations, two tools fighting over one word). Those are marked as guesses and
75
+ are never errors. Every rule is in [docs/RULES.md](https://github.com/niksa90/toolmenu/blob/main/docs/RULES.md).
54
76
 
55
77
  ## What a finding looks like
56
78
 
@@ -89,250 +111,12 @@ something toolmenu saw happen (above: "looks like it unlocks tools"). It says wh
89
111
  seen, not what it means. No LLM is involved anywhere: the same server gives the same
90
112
  findings on every run.
91
113
 
92
- ## What it catches
93
-
94
- | | |
95
- |---|---|
96
- | **A menu that changes mid-session** | `session` calls your tools and watches the list. A tool added mid-conversation can rewrite the whole cached prompt: one unlock cost 40–57K tokens of cache in my logs. |
97
- | **Every tool, called** | `session --auto` calls the tools marked read-only; `--call-everything` calls all of them, writes included, with made-up input in a container with no network and no credentials. A call that crashes, hangs or breaks the protocol is reported; a readable tool error is what the spec asks for. |
98
- | **Results clients reject** | Results are checked against the tool's `outputSchema` (a MUST since 2025-06-18) and, on 2026-07-28, for `resultType`, the way the official SDK client checks them before your agent ever sees the answer. |
99
- | **Sign-in, before logging in** | `auth check` walks OAuth discovery as a spec-following client does: the 401, protected resource metadata (RFC 9728), authorization server metadata (RFC 8414 or OpenID Connect), PKCE and registration. |
100
- | **Breaking releases** | `diff` compares two menus: removed tools, new required parameters, narrowed types, token growth. It also says whether your version bump is big enough. |
101
- | **The 2026-07-28 protocol** | The tool set must not vary per connection, and results need `resultType`. Elicitation, multi round-trip input and `server/discover` are checked too, as are the 2025 handshakes. |
102
- | **MCP Apps, Tasks, Server Cards** | It reads `ui://` resources as a host does and checks who may see each tool. It catches tasks sent to clients that can't poll them, and Server Cards that contradict the live server. |
103
- | **Your client's view** | `--client` with Claude Code, the Claude API, OpenAI, Codex, VS Code (Copilot Chat), Copilot cloud agent and code review, Gemini CLI, the Gemini API or Antigravity shows what each loads, cuts, can find, refuses, rewrites or leaves out, from each one's own docs. |
104
- | **What agents misread** | It flags vague `id`s, IDs no tool returns, instructions buried late in descriptions, and writes without annotations or marked read-only. |
105
- | **Words routing to the wrong tool** | `routes.yml` pins which tool each keyword must (and must not) reach, the way a keyword or BM25 tool search sees the menu. A description edit that lets `get_audit_trail` win "audit" fails the check, with no model and no eval budget. [More](#snapshot-rules) |
106
- | **Menus that drift** | It catches defaults built from the clock or a hash order, menus that differ per process, per connection or offline, and listings that depend on who asks. |
107
-
108
- ## Why "when it changes" matters
109
-
110
- The MCP linters I found grade one snapshot of the menu: tool names, descriptions,
111
- schemas. toolmenu also checks **when** the menu changes:
112
-
113
- - **Mid-session.** Most clients put the tool list at the start of the prompt, so any
114
- change to it, even a tool added at the end, means everything after it is processed
115
- again. More on this, with a benchmark of the setups, in
116
- [this write-up](https://niksa.me/agent-tools-design-lessons). `session` pins each
117
- change to the step that caused it, and says where in the list it happened.
118
- - **Per connection.** 2026-07-28 says the tool set **must not** vary per connection or
119
- as a side effect of other requests (the 2025 versions allow it). Per-session
120
- "unlock" designs, like the one I built, are ruled out. `session` tells you whether
121
- yours is one of them. It also checks the menu clients will actually show: a broken
122
- `x-mcp-header` or a `$ref` they can't resolve hides a tool or fails every call to
123
- it. When a tool stops to ask the user something, `session` answers "no" and says
124
- whether the tool can take it.
125
- - **Between releases.** `diff` finds breaking changes and token growth, and checks the
126
- version bump. `history` does it across a package's published releases. It installs
127
- old versions with today's dependencies, so it records what they serve now
128
- (FINDINGS F4).
129
-
130
- ## Tested on real servers
131
-
132
- Every rule change is run against real MCP servers before it ships, and every finding
133
- that changes on one of them is read by hand; a change has to say why in the
134
- [changelog](CHANGELOG.md). The corpus is pinned to exact versions, so a rerun checks the
135
- same code ([bench/README.md](bench/README.md) has the commands):
136
-
137
- | Corpus | Servers | Checked with |
138
- |---|---|---|
139
- | The reference server, `@modelcontextprotocol/server-everything` 2.1.0 (2026-07-28) | 1 | `snapshot --lists`, `session --auto`, and a scenario that answers its elicitation demos. **No false findings**: a change that makes one appear is wrong until shown otherwise |
140
- | Recent stdio servers from the MCP registry that start without credentials | 86 | `snapshot` |
141
- | The servers across the stdio lists (registry, popular npm, PyPI) that answer the 2026-07-28 handshake | 39 | `session --auto` |
142
- | Named servers run in a container: GitHub, Atlassian, Sentry, Notion, Supabase, Playwright, MongoDB… ([`bench/servers.txt`](bench/servers.txt)) | 22 | `snapshot`, `session --init`, `session` |
143
- | Hosted (Streamable HTTP) servers from the registry that need no headers, sampled with a fixed seed | 200 | `snapshot`, read-only `session --auto --max-calls 10` |
144
- | Hosted servers behind OAuth: Sentry, GitHub, Atlassian (F14) | 3 | `auth login`, then `snapshot` |
145
- | **Weekly:** Docker's MCP catalog, every local server at the digest the catalog pins, with a dummy value for each setting it declares (F20) | 237 | `snapshot`, `session --auto`, and the sealed pass: 4,117 of the 4,237 tools listed called |
146
- | **Weekly:** stdio servers from the registry, npm and PyPI (F17, F18); each week rechecks up to 300 new releases and newly listed servers | 1,090 at baseline | `snapshot`, `session --auto`, and a sealed `--call-everything` pass: 15,550 of 16,518 tools called, every one of the rest accounted for |
147
- | The servers big vendors publish to run locally: GitHub, AWS Labs, Microsoft, PayPal, Stripe, Atlassian, Notion, HashiCorp, Grafana… as npm, PyPI and container images, with dummy credentials (F19) | 98 | `snapshot`, `session --auto`, and the sealed pass: 2,213 of the 2,292 tools listed called |
148
- | **Weekly:** hosted servers, 109 vendor endpoints (GitHub, Google, Microsoft, Atlassian, Stripe…) and a seeded 2,000 of the registry's | 2,109 | `snapshot`, read-only `session --auto`, and `auth check` on each that answers 401 |
149
-
150
- What those runs found is in [docs/FINDINGS.md](https://github.com/niksa90/toolmenu/blob/main/docs/FINDINGS.md),
151
- including what toolmenu itself got wrong and how it was fixed. A few:
152
-
153
- - **mcp-atlassian serves a different menu every time it starts**: a default built from a
154
- Python set, in a new order each process, so every restart misses the cache (F12).
155
- - **Atlassian Rovo served 4 tools to any `Authorization` header**, valid or not, so a
156
- client whose token expired gets fewer tools instead of a reason to log in again (`auth/invalid-accepted`).
157
- - **The official filesystem server has reported `0.2.0` for 19 releases**, so a
158
- server's self-reported version can't be used for semver checks (F5).
159
- - **Notion's server ships the same 9 unused `$defs` in all 24 tools**: 72% of its menu.
160
- - **12 of 1,090 stdio servers fail in the official SDK client today** (8 lose every tool,
161
- 4 lose answers), and 22 of 2,000 hosted ones send a 2026-07-28 tool list that client
162
- rejects (F18).
163
- - **PayPal's server builds a new menu on every call**: `list_transactions` defaults its dates to
164
- the moment the list is built, so no cached prompt matches the next one (F19).
165
- - **Microsoft Sentinel's sign-in can't pass a client that follows MCP 2025-11-25 to the
166
- letter**: Entra ID supports PKCE but its OpenID Connect metadata doesn't say so, and the
167
- spec tells clients to refuse (F18).
168
-
169
- The weekly run compares each week with the last: menus that changed, servers that fixed
170
- or gained a finding, and the trend ([bench/results/weekly](bench/results/weekly)).
171
-
172
- ## Start here
173
-
174
- Needs Node 22 or later. The GitHub Action brings its own if the runner's Node is older,
175
- for toolmenu only: your server and the job's later steps keep the job's Node.
176
- Your server can be in any language: toolmenu starts it (stdio) or connects to it (HTTP).
177
-
178
- One command sets up CI: it snapshots your server into `menu.json` (the baseline) and
179
- writes `.github/workflows/toolmenu.yml` for your project (Node, Python, Go or Rust).
180
- Commit both, open a pull request, and toolmenu comments on it.
181
-
182
- ```sh
183
- npx toolmenu init -- node dist/server.js # add --with-session to also run session --auto
184
- ```
185
-
186
- Every value you pass with `--env` or `--header` goes into the workflow as
187
- `${{ secrets.… }}`, never as text (a name like `DATABASE_URL` doesn't say it holds a
188
- password); `init` says which secrets to add. It never overwrites a file.
189
-
190
- Run it anywhere in the repository. `.github/workflows/toolmenu.yml` goes to the git
191
- root, and the server starts in CI from the folder you ran `init` in, which is also
192
- where `menu.json` is saved. That's how a monorepo package works: run `init` in
193
- `packages/server`. Paths to files in the repository are made relative, so they work on
194
- the runner. `init` lists anything that won't work there (a file outside the
195
- repository, a `localhost` URL, a virtualenv the runner doesn't have), each with what
196
- to do.
197
-
198
- By hand:
199
-
200
- ```sh
201
- npx toolmenu snapshot -- node dist/server.js # the menu, its findings, menu.json
202
- npx toolmenu diff menu.json new-menu.json # what changed between two releases
203
- npx toolmenu review menu.json new-menu.json # should the new menu replace the baseline?
204
- npx toolmenu session --auto -- node dist/server.js # call read-only tools, watch the menu
205
- ```
206
-
207
- HTTP servers work the same way: `npx toolmenu snapshot https://example.com/mcp --header "Authorization: Bearer $TOKEN"`.
208
- Servers behind OAuth (most hosted ones: GitHub, Atlassian, Sentry…): log in once, in a
209
- browser, and `snapshot` and `session` use that login from then on, refreshing it as
210
- needed:
211
-
212
- ```sh
213
- npx toolmenu auth login https://mcp.example.com/mcp
214
- npx toolmenu snapshot https://mcp.example.com/mcp
215
- ```
216
-
217
- Before logging in, `toolmenu auth check https://mcp.example.com/mcp` walks the server's
218
- OAuth discovery the way a client that follows the spec does: the 401 and its
219
- `WWW-Authenticate`, the protected resource metadata (and whether its `resource` is the
220
- server's URL), the authorization server's metadata, PKCE and registration. No login, no
221
- tokens: only the requests a client makes first.
222
-
223
- Logins are stored per server in `~/.config/toolmenu/auth` (readable only by you);
224
- `auth list` and `auth logout <url>` manage them. A server that doesn't allow
225
- dynamic registration needs a pre-registered app: `--client-id` (and
226
- `--client-secret`). Over SSH or in a container, `auth login --no-browser` prints the
227
- login URL instead of opening it: open it on any device, sign in, and paste back the
228
- address the browser ends on (a `localhost` page that doesn't load is expected). In
229
- CI, where nobody can sign in, pass a token as a header.
230
- Both protocol generations are supported: 2026-07-28 (`server/discover`, stateless)
231
- and the 2025 `initialize` handshake, through the official TypeScript SDK.
232
-
233
- How to read a finding is [above](#what-a-finding-looks-like). `toolmenu explain <rule>`
234
- says what a rule checks, its default level and how to change it (every rule is in
235
- [docs/RULES.md](docs/RULES.md)). Every command has its own help: `toolmenu history --help`.
236
-
237
- Behind a proxy (`HTTPS_PROXY`), stdio servers get the proxy and CA settings
238
- (`HTTPS_PROXY`, `NO_PROXY`, `NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE`, `REQUESTS_CA_BUNDLE`…)
239
- along with the minimal environment, and `NODE_USE_ENV_PROXY=1` so a Node server's
240
- `fetch` uses the proxy. Without them a server that calls an API can't reach it, and
241
- fails as if it were broken. A proxy URL with a password in it
242
- (`https://user:pass@proxy:8080`) reaches the server that way too: give a server you
243
- don't trust a proxy without one, or `--env HTTPS_PROXY=` to leave it out.
244
-
245
- ### Every server you use: `scan`
246
-
247
- ```sh
248
- npx toolmenu scan ~/Library/Application\ Support/Claude/claude_desktop_config.json --allow-env LINEAR_API_KEY
249
- npx toolmenu scan .mcp.json --format markdown
250
- ```
251
-
252
- `scan` runs `snapshot` on every server in an MCP client config (Claude Desktop, Claude
253
- Code's `.mcp.json`, Cursor, VS Code: the `mcpServers` or `servers` object) or in a
254
- text file with one server per line (a URL or a command, optionally `name<TAB>` first),
255
- four at a time (`--concurrency`), and prints them side by side, then each server's
256
- findings, like this:
257
-
258
- ```
259
- server tools tokens findings
260
- github 42 ~18K 1 error · 2 warn
261
- filesystem 11 ~2.1K no findings
262
- linear – – not checked: LINEAR_API_KEY is not set (the config refers to it)
263
- ```
264
-
265
- `scan` starts every command the list names, so scan only lists you'd run yourself (not
266
- a `.mcp.json` from a repository you haven't read). A config's `${VAR}` is filled only
267
- from the variables you name with `--allow-env` (comma-separated, or `'*'` for all): the
268
- config decides where each value goes, a URL or a header included, so one from someone
269
- else could otherwise send your environment to a server of theirs. An entry that uses
270
- a variable you didn't allow isn't checked, and says which `--allow-env` it needs.
271
-
272
- `${VAR}`, `${VAR:-default}` and `${env:VAR}` in the config are filled from the
273
- environment. A server whose variable isn't set isn't started with an empty
274
- credential: it's a row that says so. A server that couldn't be started or read
275
- makes the exit code 2, because it wasn't checked; one marked `disabled` is listed and
276
- left alone. URLs are printed without their query, which can hold a key. Nothing is
277
- written unless `--out-dir` names a folder for each server's menu file.
278
-
279
- ## For the agent building your server: `mcp`
280
-
281
- `toolmenu mcp` is toolmenu as an MCP server (stdio), so Claude Code, Cursor or any
282
- other client can check the server it's working on while it works. In Claude Code, as a
283
- plugin (the server, and skills for checking a change, checking a release, and setting up CI):
284
-
285
- ```
286
- /plugin marketplace add niksa90/toolmenu
287
- /plugin install toolmenu@toolmenu
288
- ```
289
-
290
- Or just the server, in any client: `claude mcp add toolmenu -- npx -y toolmenu mcp`, or
291
- `{"command": "npx", "args": ["-y", "toolmenu", "mcp"]}` in an `mcpServers` config.
292
-
293
- | Tool | Runs |
294
- |---|---|
295
- | `snapshot_menu` | `snapshot`: findings and token cost; writes the menu file only with `save_as` |
296
- | `diff_menus` | `diff`: breaking changes, the token change and the version bump between two menu files |
297
- | `check_session` | `session --auto`: calls the read-only tools and reports each menu change |
298
- | `explain_rule` | `explain`: what a rule checks and how to change its level |
299
-
300
- Each tool runs the CLI in a child process and returns its text report, so options,
301
- `toolmenu.config.json` and stored OAuth logins work as they do on the command line.
302
- Relative paths resolve from the folder `toolmenu mcp` runs in, or a call's `cwd`.
303
- `snapshot_menu` and `check_session` start the command you give them, as the CLI does;
304
- clients ask before each tool call unless you allow it. toolmenu's own menu passes its
305
- checks with no findings, on both protocols and in a session (the test suite checks).
306
-
307
- ## As a library
308
-
309
- Everything the CLI does is exported from the `toolmenu` package, with types, for test
310
- suites and scripts of your own:
311
-
312
- ```js
313
- import { snapshot, diffMenus, loadMenu, session, loadScenario } from 'toolmenu';
314
-
315
- // The menu and its findings, as `toolmenu snapshot` gives them.
316
- const { menu, findings } = await snapshot({ kind: 'stdio', command: 'node', args: ['dist/server.js'] });
317
- // or { kind: 'http', url: 'https://example.com/mcp', headers: { Authorization: `Bearer ${token}` } }
318
-
319
- // What changed since the committed baseline, and the version bump it needs.
320
- const diff = diffMenus(await loadMenu('menu.json'), menu);
321
- console.log(diff.suggestedBump, diff.tokens);
322
- if (diff.findings.some((f) => f.severity === 'error')) process.exitCode = 1;
323
- ```
324
-
325
- `snapshot`, `diffMenus`, `session` and `history` take the same options as their
326
- commands (`SnapshotOptions`, `DiffOptions`, `SessionOptions`, `HistoryOptions`) and
327
- return the findings the reports print: each `Finding` has its `rule`, `severity`,
328
- `message`, `tool` and the `fix` shown as `→ Next:`. Lower down, `compareMenus(before,
329
- after)` returns each tool's change (`ToolChange[]`) and `runRules` runs the menu rules
330
- on a menu you built. The `menu.json` format is the `Menu` type (`toolmenu: 1`);
331
- a later format will bump that number. Library functions don't print or exit: that's
332
- the CLI's job.
333
-
334
114
  ## In CI: one PR comment
335
115
 
116
+ `npx toolmenu init -- <your server command>` snapshots the server into `menu.json` (the
117
+ baseline) and writes this workflow for your project (Node, Python, Go or Rust). Commit
118
+ both. Credentials you pass go into the workflow as `${{ secrets.… }}`, never as text.
119
+
336
120
  ```yaml
337
121
  # .github/workflows/toolmenu.yml
338
122
  on: pull_request
@@ -347,643 +131,127 @@ jobs:
347
131
  - uses: actions/setup-node@v7
348
132
  with: { node-version: 22 }
349
133
  - run: npm ci && npm run build
350
- - uses: niksa90/toolmenu@v0.27.0
134
+ - uses: niksa90/toolmenu@v0.27.1
351
135
  with:
352
136
  command: node dist/server.js
353
137
  baseline: menu.json # your committed snapshot
354
138
  scenario: scenario.yml # optional: run a session too
355
139
  ```
356
140
 
357
- It snapshots the PR's build, diffs it against the baseline as it is on the PR's base
358
- branch (or at the previous tag, on a tag push), runs the scenario, and
359
- posts **one comment that updates on every push**: first the verdict (can this menu
360
- replace the baseline as it is?), then breaking changes, the token change
361
- ("this PR adds ~1,240 tokens to every conversation"), the suggested version bump,
362
- and anything `session` caught. The same report goes to the job summary.
363
-
364
- | Input | Default | |
365
- |---|---|---|
366
- | `command` or `url` | | How to start the server over stdio, or its Streamable HTTP endpoint |
367
- | `headers` | | HTTP headers for a `url` server, one `Name: value` per line. Use a secret for keys: `x-api-key: ${{ secrets.MCP_API_KEY }}` |
368
- | `env` | | Environment variables for a `command` server, one `KEY=value` per line |
369
- | `baseline` | `menu.json` | The committed snapshot to diff against |
370
- | `scenario` | | A scenario to run with `session`, or `auto` to build the steps from the menu |
371
- | `catalog` | `false` | `true` also diffs the operations behind a search tool (`snapshot --catalog`) |
372
- | `baseline-from` | `snapshot` | `session` diffs every tool the scenario saw (the union menu), for servers whose tools appear after an unlock |
373
- | `release` | `auto` | Release versions for the bump check. `auto` reads `package.json`, `pyproject.toml` or `Cargo.toml` on the base branch and the head, or compares the tag with the previous one on a tag push. Set `"1.4.0..1.5.0"` yourself, or `off` |
374
- | `fail-on` | `error` | Fail the job on findings at or above this level |
375
- | `gate` | `findings` | `review` passes or fails the job on the [review verdict](#accepting-a-new-baseline-review) instead of `diff`'s findings: a breaking change the release bump allows passes, new findings at `fail-on` fail ([more](https://github.com/niksa90/toolmenu/blob/main/docs/github-action.md#the-verdict-and-gate-review)) |
376
- | `comment` | `true` | Post and update the PR comment |
377
- | `sarif` | | A folder to also write the findings to as SARIF, for code scanning (below) |
378
- | `version` | the Action's own | toolmenu version from npm: by default the one matching the Action tag you pinned (`latest` for the newest). The Action installs toolmenu from npm, so it needs a published version |
379
-
380
- ### In the Security tab: code scanning
381
-
382
- With `sarif`, findings also go to GitHub code scanning: alerts in the Security tab,
383
- each on the line of its tool in your `menu.json`, and a code scanning check on the pull
384
- request. GitHub shows an alert inline in the pull request only when that line changed in
385
- it, so most stay in the Security tab and the check. Run the workflow on pushes to your
386
- default branch too, so GitHub has a base to compare a pull request's alerts with. Code scanning is free for public repositories; private ones need GitHub Advanced
387
- Security.
388
-
389
- ```yaml
390
- on:
391
- pull_request:
392
- push:
393
- branches: [main] # the base GitHub compares a pull request's alerts with
394
- permissions:
395
- contents: read
396
- pull-requests: write
397
- security-events: write # to upload SARIF
398
- steps:
399
- # … checkout, build, as above
400
- - uses: niksa90/toolmenu@v0.27.0
401
- with:
402
- command: node dist/server.js
403
- sarif: toolmenu-sarif
404
- - uses: github/codeql-action/upload-sarif@v4
405
- if: always() # findings fail the toolmenu step; upload them anyway
406
- with:
407
- sarif_file: toolmenu-sarif
408
- ```
409
-
410
- Outside the Action: `toolmenu snapshot --sarif toolmenu.sarif -- node dist/server.js`
411
- writes the file beside the usual report (`diff` and `session` take `--sarif` too), or
412
- `--format sarif` prints it instead.
413
-
414
- ### Other CI systems: JUnit test reports
415
-
416
- GitLab, Jenkins, Azure DevOps, CircleCI and Buildkite show JUnit XML as test results.
417
- `--junit <path>` writes the findings of `snapshot`, `diff` or `session` as JUnit beside
418
- the usual report (`--format junit` prints it instead). New in 0.25.2. Each finding is a test case named
419
- after its tool, in a class named after its rule (`toolmenu.diff.diff/removed`), so the
420
- CI system tracks it across runs. Findings at or above `--fail-on` are failures, the rest
421
- pass with their text, so the test report agrees with the exit code. A clean run is one
422
- passing case.
423
-
424
- ```yaml
425
- # .gitlab-ci.yml
426
- toolmenu:
427
- image: node:22
428
- script:
429
- - npm ci && npm run build
430
- # Exit 1 means findings: keep going so both reports are written, then fail.
431
- - npx toolmenu snapshot --out new-menu.json --junit snapshot.xml -- node dist/server.js || code=$?
432
- - npx toolmenu diff menu.json new-menu.json --junit diff.xml || code=$?
433
- - exit ${code:-0}
434
- artifacts:
435
- when: always
436
- reports:
437
- junit: [snapshot.xml, diff.xml]
438
- ```
439
-
440
- ### A report to keep: HTML
441
-
442
- `--html <path>` writes the run of `snapshot`, `diff` or `session` as one HTML page
443
- beside the usual report: the numbers that matter at the top (errors, warnings, tools,
444
- tokens; for `diff`, breaking changes and the suggested bump), the menu as a table in
445
- server order with each tool's share of the tokens, and the findings with their next
446
- steps. `diff` shows each tool before and after, new and removed tools marked. `session`
447
- has a row per step with the menu after it, tools inserted, appended or edited marked,
448
- and links from a step to its findings.
449
-
450
- The page has no script and loads nothing, so it opens offline: keep it as a CI artifact
451
- (GitLab `artifacts: paths`, `actions/upload-artifact`) or attach it to an issue for a
452
- server's maintainer. It follows the viewer's light or dark setting.
141
+ Every pull request then gets one comment that updates on each push. It leads with a
142
+ verdict (can this menu replace the baseline as it is?), then the breaking changes, the
143
+ token change, the version bump the change needs, and anything `session` caught:
453
144
 
454
- ```sh
455
- npx toolmenu diff menu.json new-menu.json --html diff.html
456
- ```
457
-
458
- ### A badge for your README
459
-
460
- ```sh
461
- npx toolmenu snapshot --badge toolmenu-badge.json -- node dist/server.js
462
- ```
463
-
464
- writes a [shields.io endpoint](https://shields.io/badges/endpoint-badge) file (tools,
465
- tokens, and errors or warnings, in red, yellow or green) and prints the README line
466
- for it. Commit it with `menu.json`: it changes when the baseline does, so nothing
467
- has to commit from CI. The badge at the top of this README is toolmenu's own MCP
468
- server (`toolmenu mcp`), checked by itself.
469
-
470
- Server behind auth, or PRs from forks and Dependabot? See [docs/github-action.md](https://github.com/niksa90/toolmenu/blob/main/docs/github-action.md).
471
- Server that unlocks tools on request? Snapshot the full menu for the baseline and run the session on the default one: [servers that unlock tools](https://github.com/niksa90/toolmenu/blob/main/docs/github-action.md#servers-that-unlock-tools).
472
- PRs without secrets are skipped, and a skipped check still passes; the `skipped` output tells you.
473
-
474
- ## `session`: the menu changing while the agent works
475
-
476
- `session` runs a scripted session (the scenario *is* the agent), lists the menu
477
- after every step, and reports each change with the step that caused it.
478
-
479
- ```
480
- step the scenario (the agent) the menu the client sees toolmenu
481
- ──── ─────────────────────────────── ─────────────────────────────── ─────────────────────────────
482
- 1 list search_forms, get_form, … baseline: 5 tools
483
- 2 call search_forms (unchanged) no change
484
- 3 call unlock_toolset audits search_forms, list_team_audits, +2 tools at position 1:
485
- get_team_audit, get_form, … session/mid-insert (error)
486
- fresh process: not there →
487
- session/side-effect (warn)
488
- ```
489
-
490
- Each step's menu is compared with the one before it, so a change is pinned to the call
491
- that caused it, with where in the list it starts and what it costs the prompt cache.
492
- toolmenu then starts a second process (or opens a second connection) to see whether
493
- the change is global, or only this session's, which 2026-07-28 rules out.
494
-
495
- ```yaml
496
- # scenario.yml
497
- allow_writes: false # refuse to call tools not marked readOnlyHint
498
- steps:
499
- - list
500
- - call: search_forms
501
- args: { query: onboarding }
502
- - call: unlock_toolset
503
- args: { toolset: audits }
504
- - wait_for: tools_list_changed # optional; timeout_ms defaults to 5000
505
- ```
506
-
507
- `--init` writes a starter scenario from the live menu: it calls every read-only tool
508
- that needs no arguments, then repeats one. If a tool looks like it unlocks more tools
509
- (a `domains` or `toolset` parameter with an enum, "unlock" or "capabilities" in its
510
- name, a description that says it enables or loads tools), the starter unlocks
511
- **every** value, listing after each, then repeats the first, so the session sees every
512
- tool behind the unlock. The values come from the parameter's enum or, without one,
513
- from the server's own read-only listing (`list_toolsets`). Lookups that only describe
514
- toolsets (`list_toolsets`, `get_toolset_tools`) aren't unlocks, and one not marked
515
- read-only is suggested, never called. Anything it can't fill in is left as a
516
- commented-out step. An unlock nothing in the menu names, like joining a room that
517
- brings more tools, needs a step you add yourself. A scenario is only as good as its
518
- steps, so the starter is a floor, not a ceiling.
519
-
520
- On github-mcp-server 1.0.5 with `--dynamic-toolsets` (3 tools at the start, 19
521
- toolsets), unlocking every toolset reaches 107 tools: all 81 that `--toolsets all`
522
- serves, and 23 issue and pull-request tools that only the dynamic mode has.
523
-
524
- | Rule | Default | Catches |
525
- |---|---|---|
526
- | `session/mid-insert`, `session/reorder`, `session/remove`, `session/edit` | error | The menu changing mid-session anywhere but the end, with where the change starts and which value changed. Any change to the tool list invalidates the cached prompt: the whole tool list (its size, estimated) and the conversation after it are processed again |
527
- | `session/append` | warn | Tools added at the end of the list. Still a cache miss for the conversation when your client sends tools at the start of the prompt (Claude's Messages API does); cache-safe only when the client adds new tools after the cached content, as tool search (deferred loading) does |
528
- | `session/connection-local` | error on 2026-07-28 (HTTP) | A change only this connection sees. 2026-07-28: the tool set MUST NOT vary "per-connection or as a side effect of other requests on the connection" |
529
- | `session/side-effect` | warn on 2026-07-28 (stdio) | The same, where stdio can't tell per-connection from global |
530
- | `menu/process-variance`, `menu/connection-variance` | error | Checked before the first step, as in `snapshot` (below) |
531
- | `session/unannounced` | warn | The menu changed without `notifications/tools/list_changed`, although the server declared `listChanged` |
532
- | `session/untested` | warn, error if no call got through | Calls that failed before reaching the tool: authentication, something missing on this machine (no Chrome), the network. One finding for the run, not one per step, so an expired CI secret doesn't pass a run that tested nothing |
533
- | `session/refused`, `session/step-failed` | error | A write the scenario didn't allow, or a call that failed for another reason |
534
- | `session/tool-error` | warn | A tool that answered with an error for another reason (`isError`): the run tested less than it looks. The same error at several steps is one finding that names them |
535
- | `session/known-variance` | info | A mid-session change only in values that already vary from one tools/list to the next: same cause as the finding it names, not counted again |
536
- | `session/assumed-read-only` | info | Tools called on your word (`--assume-read-only`, `assume_read_only`) although the server doesn't mark them readOnlyHint |
537
- | `session/unlock-coverage` | warn | An unlock whose enum lists its values, called with only some of them: the tools behind the rest were never seen, so a baseline from the session misses them. With `--union-out` or `--auto`, an unlock never called at all too, with why `--auto` skipped it |
538
- | `session/nothing-called` | warn | `--auto` called no tools (all open-world, missing values, or not read-only): the run only listed the menu |
539
- | `session/session-lost` | error | The server ended the session after toolmenu opened a second connection with the same credentials (servers with one session per client; 2025-era servers only). The run stops there; rerun with `--processes 1` |
540
- | `session/handle-error` | warn | 2026-07-28: `--auto` called a read-only tool with a handle that can't exist (`toolmenu-unknown-session`), and it answered with a protocol error instead of a tool error the model can read |
541
- | `session/error-not-flagged` | warn | A tool answered with an HTTP error from the API behind it (`{"status":401,"object":"error",…}`) as a plain result, not marked `isError`, so a client that reads `isError` takes the call for a success (Notion's server). An auth one also counts toward `session/untested`. Tools that take a URL are left out: their answer is the page's (heuristic) |
542
- | `session/server-unresponsive` | error | The server stopped answering after a step: even `tools/list` timed out. The run stops there with this one finding, instead of every later step waiting out its timeout (a server that polls a device-code sign-in inside a tool call) |
543
- | `session/handle-roundtrip` | warn | With `--create-handles`: the tool that creates a session, basket or cart was called, and the tools that take it back refused the handle it had just returned (info when its result had no handle to find) |
544
- | `session/output-schema` | error | A tool that declares an `outputSchema` answered with no `structuredContent`, or one that doesn't match it: from 2025-06-18 a **MUST**, and the official SDK client raises an error instead of the answer |
545
- | `session/rate-limited` | error | The server kept refusing requests for being too many (a 429) after toolmenu waited about a minute, or a tool not marked `readOnlyHint` said it was rate-limited (it isn't called again). The run stops there, with the steps it didn't check; give the run its own server instance or raise the limit for it |
546
- | `session/scope-unchecked` | info | A second server process or connection couldn't start mid-session (a server that holds a file or a port), so whether a change was global wasn't checked. Said once for the run |
547
-
548
- **No scenario? `session --auto`** builds the steps from the menu: every read-only tool
549
- whose required arguments the schema itself gives (a `default`, `examples`, an `enum`,
550
- a type or format), cheapest first, then every value of each read-only unlock (outside
551
- `--max-calls`, and the first value twice: a second identical unlock should change
552
- nothing), then the first call again. It never guesses an ID: a tool that needs
553
- `owner`, `repo` or an issue key is skipped, and the report names the flag that fills
554
- it: `--value repo_path=/src/app` fills that parameter on every tool that requires it,
555
- `--value get_issue.issue_key=ABC-1` on one tool, `--values-file values.yml` takes a map
556
- of either. Servers that don't mark their tools `readOnlyHint` (DeepWiki, every
557
- 2024-11-05 server) get nothing called unless you vouch for them by name:
558
- `--assume-read-only read_wiki_structure,ask_wiki_question`. Only exact names are
559
- accepted: a pattern would also cover tools a later release adds. A tool marked or
560
- named as a write is never called, and the report says which tools ran on your word.
561
- With `--create-handles` on 2026-07-28, `--auto` also calls the tool that creates a
562
- session, basket or cart (`start_session`; a write, so only with the flag, and never
563
- one marked `destructiveHint`; not locks, leases or checkouts) and gives the tools that
564
- take it back the handle it returned, beside the made-up one.
565
- `--save-scenario auto.yml` writes the steps, with the skipped tools commented out, for
566
- you to fill in. Tools marked `openWorldHint: true` (web
567
- search, fetch, scraping, which can cost API credits) are called only with
568
- `--open-world`, except unlocks: changing the menu is what a session watches, and an
569
- unlock spends no search credits, so it runs even on a server that marks every tool
570
- open-world. A run that ends up calling nothing is `session/nothing-called` (warn),
571
- with why each tool was left out, not a clean pass. On the 22 servers in `bench/`,
572
- `--auto` reaches 60 of 262 read-only tools, 84 with `--open-world`.
573
-
574
- **`session` calls tools for real.** Without `allow_writes: true` it refuses any tool
575
- not marked `readOnlyHint: true`. `--plan` prints the steps without connecting.
576
-
577
- **`--call-everything`** (with `--auto`, stdio only) calls **every** tool once, writes and
578
- open-world ones included, with a made-up value (`toolmenu-test`, `1`, a one-item list)
579
- where the schema gives none. It's for a server in a sandbox with nothing to lose: a
580
- container with no credentials and no network, as `bench/scripts/batch-full.sh` runs
581
- them with `SEALED=1`. Every tool's answer is then checked: results against their own
582
- `outputSchema`, errors flagged as errors, calls that hang. It's refused over HTTP,
583
- because a hosted server's tools act on someone's real account.
584
-
585
- ## `diff`: compare releases
586
-
587
- ```sh
588
- npx toolmenu diff menu.json new-menu.json
589
- ```
590
-
591
- ```
592
- $ npx toolmenu diff --release 2026.1.14..2026.8.31 fs-2026.1.14.json fs-2026.8.31.json
593
- toolmenu diff secure-filesystem-server 2026.1.14 → 2026.8.31
594
- changes 1 breaking (1 tool) · 0 minor · 3 notice (13 tools)
595
- version suggested bump: major · not checked (calendar version)
596
- tokens ~1,640 → ~1,664 (+24, estimate): this release adds ~24 tokens to every conversation that loads the menu
597
- tools 14 → 14
598
-
599
- ERROR diff/safety-hint
600
- `move_file` was marked additive-only and is now destructive (destructiveHint false → true; unset means destructive). Clients that auto-approve additive tools may now ask first.
601
- → Next: If the tool still never deletes or overwrites, set destructiveHint: false; if it does now, say so in the release notes.
602
- INFO diff/annotations
603
- Annotations changed the same way (openWorldHint (unset) → false) in 13 of 14 tools: read_file, read_text_file, read_media_file, read_multiple_files, write_file, edit_file, … (+7).
604
- …
605
- ```
606
-
607
- That's two real releases of the official filesystem server. Its calendar versions
608
- promise nothing about compatibility, so the bump isn't judged, but the breaking change
609
- is still an error. Changes are classified
610
- the way OpenAPI breaking-change checkers do it:
145
+ <p align="center">
146
+ <img src="https://raw.githubusercontent.com/niksa90/toolmenu/main/docs/review-comment.png" width="760" alt="toolmenu's review of two real releases of the official filesystem server, as markdown: don't accept as is, 1 breaking change (move_file is now destructive), suggested bump major, +24 tokens">
147
+ </p>
611
148
 
612
- | Class | Changes | Rules |
613
- |---|---|---|
614
- | **breaking** (error) | tool removed or renamed, parameter removed, new required parameter, narrower or different type, enum narrowed, unlisted properties now rejected (`additionalProperties: false` added), a tool becoming less safe (no longer read-only, or additive → destructive); with `--lists` on both sides, a prompt removed or newly requiring an argument, a resource template removed | `diff/tool-removed`, `diff/tool-renamed`, `diff/param-removed`, `diff/param-required`, `diff/param-type`, `diff/enum-narrowed`, `diff/properties-closed`, `diff/safety-hint`, `diff/prompt-removed`, `diff/prompt-argument-required`, `diff/resource-template-removed` |
615
- | **minor** (info) | tool added, new optional parameter, parameter now optional, type or enum widened, unlisted properties now accepted (`additionalProperties: false` removed), a prompt or resource template added | `diff/tool-added`, `diff/param-added`, `diff/param-relaxed`, `diff/type-widened`, `diff/enum-widened`, `diff/properties-opened`, `diff/prompt-added`, `diff/resource-template-added` |
616
- | **notice** (info) | optional parameter dropped (extra properties still allowed), description changed (with the text diff: it changes what the agent does), output schema, annotations, other fields, order, a schema restructured but accepting the same input, only the declared `$schema` dialect changed, a prompt's description or optional arguments, resources added or gone (data, so no bump) | `diff/param-dropped`, `diff/description`, `diff/schema-other`, `diff/annotations`, `diff/other`, `diff/order`, `diff/schema-equivalent`, `diff/schema-dialect`, `diff/prompt-changed`, `diff/resources` |
617
-
618
- The same rules apply inside parameters: fields of an object parameter, array items,
619
- and the options of an `anyOf`/`oneOf` union (zod's unions, discriminated unions and
620
- `.nullable()`) are compared at every depth, with their path (`gen.body.text`,
621
- `search.filters[].field`, `gen.block(kind="image").url`). Union options are paired by
622
- the value of a discriminator they all fix; without one (a plain `z.union` of objects),
623
- identical options first, so a reorder is no change, then by type and shared property
624
- names. An option gone is breaking, a new one widens. The same change reached through
625
- one shared definition at several places is one finding that lists them (`places` in
626
- `--json`). Local `$ref`s (`#/$defs/…`, `#/definitions/…`) are expanded first,
627
- recursive ones one level deep, so a schema is compared by what it accepts, not how
628
- it's spelled: moving a repeated block into `$defs` is one `diff/schema-equivalent`
629
- notice with its token change ("accepts the same input: ~899 tokens fewer"), and a
630
- breaking change inside `$defs` is breaking. A schema change no rule classifies is
631
- never silent: it's a `diff/schema-other` "review it".
632
-
633
- On mongodb-mcp-server 2.1.2 → 3.0.0 this finds `aggregate.pipeline[]…$vectorSearch.numCandidates`
634
- and `.limit` narrowed from `number` to `integer` inside a union, where the whole
635
- `pipeline` used to be one "review it".
636
-
637
- Different spellings of the same schema are one schema (`additionalProperties` absent,
638
- `true` or `{}`; zod 4's safe-integer bounds), and a change every tool shares is said
639
- once for the menu: a server moving from zod 3 to zod 4 reads as its real changes (a
640
- field that became required, the patterns zod 4 adds to `email` and `uuid`) plus one
641
- `diff/properties-opened` line ("27 tools now accept properties they don't list … most
642
- likely a schema generator upgrade"), not a "review it" per object.
643
-
644
- The same change in many tools is reported once, with the tools it touches (`pageId`
645
- is new and required in 25 of 29 tools, on chrome-devtools-mcp 1.8.0); counts are per
646
- change. When a tool loses a parameter and gains a required one with a close name,
647
- `diff/param-renamed` (warn, unsure) points out the likely rename; both changes stay
648
- breaking. Every breaking finding's next step names the version to release when the
649
- bump is too small.
650
-
651
- It also reports the token change per tool and suggests a semver bump. To check the
652
- bump, pass the **release** versions: `--release 1.4.0..1.5.0` (npm, a git tag).
653
- `diff/version-bump` then warns when the release's bump is smaller. Under 1.0.0 it
654
- reads versions as npm's caret does (`^0.2.3` accepts any 0.2.x): breaking changes need
655
- a minor bump, new features a patch, and under 0.1.0 anything goes.
656
- `diff/version-backwards` warns when the version goes down. Calendar versions and prereleases aren't judged. The version in the snapshot is what the
657
- server reports (`serverInfo.version`), which is often not the release: the filesystem
658
- server has said `0.2.0` for 19 releases (FINDINGS F5). It's shown, and only checked
659
- with `--server-version-is-release`. `diff` also enforces an optional `tokenBudget`
660
- (`diff/token-budget`). Order changes are only a notice here: between releases, prompt caches rebuild. Order
661
- matters *within* a session.
662
-
663
- ### Accepting a new baseline: `review`
664
-
665
- `diff` says what changed. Before you commit a refreshed `menu.json`, `review` answers
666
- the question that follows: should it replace the baseline?
149
+ That's `toolmenu review --format markdown` on two real releases of the official
150
+ filesystem server. The same findings go to GitHub code scanning (SARIF), to GitLab,
151
+ Jenkins and others as JUnit, or into one offline HTML page:
667
152
 
668
- ```sh
669
- npx toolmenu snapshot --out new-menu.json -- node dist/server.js
670
- npx toolmenu review --release 1.4.0..1.5.0 menu.json new-menu.json
671
- ```
153
+ <p align="center">
154
+ <img src="https://raw.githubusercontent.com/niksa90/toolmenu/main/docs/report.png" width="760" alt="toolmenu's HTML report of a session: 3 errors, 2 warnings, the menu after each step with inserted tools marked, and each finding with its next step">
155
+ </p>
672
156
 
673
- ```
674
- toolmenu review new-menu.json as the new menu.json
675
- verdict ✗ don't accept as is: 1 breaking change, and the release's minor bump is too small (release 2.0.0).
676
- changes 1 breaking (1 tool) · 0 minor · 3 notice (13 tools)
677
- findings 0 new · 0 gone · 1 changed · 1 unchanged
678
- ```
157
+ Inputs, servers behind auth, forks and `gate: review` are in
158
+ [docs/github-action.md](https://github.com/niksa90/toolmenu/blob/main/docs/github-action.md); SARIF, JUnit, HTML and the badge in
159
+ [docs/ci.md](https://github.com/niksa90/toolmenu/blob/main/docs/ci.md).
679
160
 
680
- It runs `diff`, and runs the menu rules on both files, so it also says which findings
681
- the new menu **adds**, which it **clears**, and which **changed** (a summary that now
682
- names another tool counts as worse). The verdict is one of:
161
+ ## Commands
683
162
 
684
- | Verdict | Exit | When |
163
+ | Command | What it does | |
685
164
  |---|---|---|
686
- | `unchanged` | 0 | The same menu and the same findings |
687
- | `safe` | 0 | Nothing breaks callers, and no finding at or above `--fail-on` is new or worse |
688
- | `breaking-released` | 0 | Breaking changes, and `--release` shows a bump that allows them |
689
- | `breaking` | 1 | Breaking changes, and no release that allows them: none given, too small, or not checkable (calendar versions) |
690
- | `review` | 1 | Nothing breaks, but findings at or above `--fail-on` are new or worse, or `diff`'s own checks fail (`tokenBudget`) |
691
-
692
- It reads only the two files, so rules that need the live server (a second process,
693
- the raw `tools/list`) aren't part of it; `snapshot` runs those. `--format markdown`
694
- gives the same as a PR comment, `--json` for scripts (`verdict`, `accept`, and the
695
- findings by kind).
696
-
697
- ### Operations behind a search tool (`--catalog`)
698
-
699
- Big servers increasingly keep most operations out of the menu: a small fixed menu, a
700
- search tool, and an execute tool (Sentry's `search_sentry_tools` + `execute_sentry_tool`,
701
- Atlassian's `discover` + `executeRead`). A breaking change to one of those operations
702
- never shows in the menu. `snapshot --catalog` asks the search tool a fixed set of
703
- queries and keeps the operations it returns in `menu.json`; `diff` then compares
704
- operations found in both snapshots with the same rules as tools.
165
+ | `snapshot` | Connects, lists the menu, checks it (the spec, a second process, your client's limits, token cost) and writes `menu.json` | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/snapshot.md) |
166
+ | `session` | Plays a scenario, or builds one with `--auto`: calls tools, checks every result, and lists the menu after each step. Read-only tools unless you allow writes; `--call-everything` calls every tool, for a server in a sandbox | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/session.md) |
167
+ | `diff` | Compares two menu files: breaking, minor, notice; token change; the semver bump | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/diff.md) |
168
+ | `review` | Should the new menu replace the baseline? One verdict and an exit code | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/diff.md#accepting-a-new-baseline-review) |
169
+ | `auth check` | Walks an HTTP server's OAuth discovery as a spec-following client does, without logging in | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/usage.md#http-servers-and-oauth) |
170
+ | `auth login` | Signs in to a server behind OAuth once, in a browser; `snapshot` and `session` then use that login | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/usage.md#http-servers-and-oauth) |
171
+ | `scan` | `snapshot` on every server in a client config (Claude Desktop, `.mcp.json`, Cursor, VS Code), side by side | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/snapshot.md#every-server-you-use-scan) |
172
+ | `history` | Installs a package's published versions one by one and diffs them. Runs third-party code: use a container | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/diff.md#history-a-packages-releases-researched) |
173
+ | `init` | Writes the baseline and the CI workflow | [above](#in-ci-one-pr-comment) |
174
+ | `mcp` | toolmenu as an MCP server and a Claude Code plugin, so the agent building your server can check it while it works | [docs](https://github.com/niksa90/toolmenu/blob/main/docs/usage.md#for-the-agent-building-your-server-mcp) |
175
+ | `explain <rule>` | What a rule checks, its default level and how to change it | [RULES](https://github.com/niksa90/toolmenu/blob/main/docs/RULES.md) |
176
+
177
+ `--client <name>` checks the menu for one client, from that client's own docs: Claude
178
+ Code, the Claude API, OpenAI, Codex, VS Code (Copilot Chat), Copilot cloud agent and code
179
+ review, Gemini CLI, the Gemini API or Antigravity ([docs](https://github.com/niksa90/toolmenu/blob/main/docs/clients.md)). A client
180
+ that doesn't document a limit doesn't get one: Cursor's and Windsurf's aren't checked.
181
+
182
+ `session` calls tools for real. Without `allow_writes: true` it refuses any tool not
183
+ marked `readOnlyHint: true`, and `--call-everything` is refused over HTTP, because a
184
+ hosted server's tools act on someone's real account.
185
+
186
+ Everything is also exported as a library, with types
187
+ ([docs](https://github.com/niksa90/toolmenu/blob/main/docs/usage.md#as-a-library)). All protocol versions from 2024-11-05 to
188
+ 2026-07-28 are supported, through the official TypeScript SDK; the table in
189
+ [docs/PROTOCOLS.md](https://github.com/niksa90/toolmenu/blob/main/docs/PROTOCOLS.md) is written by a test that runs each one.
705
190
 
706
- ```sh
707
- npx toolmenu snapshot --catalog -- node dist/server.js
708
- ```
709
-
710
- A search returns only its top matches, so toolmenu crawls the catalog by its own
711
- words: it starts from the search tool's own example phrases and the nouns in your tool
712
- names, then searches for every operation it finds or sees mentioned, until 20 queries
713
- in a row turn up nothing new (at most 200). On Atlassian's hosted server that finds all
714
- 154 operations behind its 21 tools; on Sentry's, 65 behind 9, the same on every run.
715
- Queries are paced and a rate limit is waited out. An operation not found this time is a
716
- notice (`diff/catalog-missing`: "not returned by the same queries"), never "removed";
717
- one found only now is `diff/catalog-added`, and catalogs read with different queries,
718
- or partly (a query failed), are `diff/catalog-queries`. To steer it, set
719
- `"catalog": { "queries": [...], "maxQueries": 200, "crawl": true }` in
720
- `toolmenu.config.json` (FINDINGS F15).
721
-
722
- A crawl through search is a lower bound, not a proof: an operation that nothing in the
723
- catalog names, and whose words no query uses, stays unfound. On `@sentry/mcp-server`
724
- 0.42 (stdio) it finds 63 of the 64 operations in the package's catalog; `whoami` comes
725
- back only when asked for by name. Add queries like that to `catalog.queries`.
726
-
727
- **Command routers.** Some servers put their operations behind router tools instead:
728
- one tool per area that takes a `command` and its `parameters`, and lists its commands
729
- when called with `learn: true` (Azure's MCP server serves 71 such tools in front of
730
- ~410 operations). `--catalog` detects them (a `command` parameter, an arguments
731
- object, a boolean listing flag, and a description that says "router" or "sub
732
- commands") and calls each once in its listing mode, never with a command, so nothing
733
- runs. Routers whose command is required aren't called. Detection missed one? Name
734
- them: `"catalog": { "routers": ["keyvault", "storage"] }`. A router lists everything
735
- it has, so unlike a search the catalog is complete except for routers that failed
736
- (`catalog/failed` says which, and why).
737
-
738
- ## `history`: a package's releases, researched
739
-
740
- ```sh
741
- npx toolmenu history @modelcontextprotocol/server-filesystem --versions 19 --arg /tmp --csv fs.csv
742
- ```
743
-
744
- It installs each published version into a temp directory (install scripts off),
745
- snapshots it, and diffs it against the previous version that worked. For every
746
- version it records the protocol, the menu, the findings and the **dependency
747
- versions that actually got installed**: an old version installed today resolves its
748
- dependencies today, and can serve a different menu than it shipped with.
749
-
750
- It's best effort: packages that need env vars, arguments or auth are recorded as
751
- failed, with a reason, never skipped. **It installs and runs third-party code. Run it
752
- in a container.**
753
-
754
- ## Where the tokens go
755
-
756
- `snapshot` also shows what the menu's tokens are spent on: the biggest tools (split
757
- into description and schema), large enums, and waste you can remove:
758
-
759
- - **Repeated blocks**: the same piece of schema at several places, inside one tool (a
760
- `$defs` entry could hold it once: "one ~230-token block ×5 in
761
- content_generateMessageHtml") or across tools (paid once per tool: mongodb-mcp-server
762
- 3.0.4 repeats a ~2,200-token `pipeline` schema in 3 tools). Only the largest repeated
763
- block is named, not the pieces inside it.
764
- - **Unused `$defs`**: definitions nothing in the tool refers to. Notion's server 2.5.2
765
- ships the same 9 in all 24 tools, 72% of its menu.
766
-
767
- Every tool's estimate is in the menu file, for the ones the summary leaves out:
768
- `jq -r '.tools[] | "\(.tokens)\t\(.name)"' menu.json | sort -rn`.
769
-
770
- ## `snapshot` rules
191
+ ## Tested on real servers
771
192
 
772
- Every rule comes from a failure I actually hit building 115 tools for one MCP
773
- server, or from the spec itself. If a rule can't point to one, it doesn't ship.
193
+ Rules are checked against public MCP servers before they ship, and a rule that changes
194
+ a finding on one has to say why in the [changelog](CHANGELOG.md). The run on 2026-11-09
195
+ is the baseline for a weekly one ([`bench/scripts/weekly.sh`](bench/scripts/weekly.sh)),
196
+ which compares each week with the last:
774
197
 
775
- | Rule | Default | Catches |
198
+ | Panel | Servers | What ran |
776
199
  |---|---|---|
777
- | `menu/nondeterministic` | error | Two identical `tools/list` calls returning different menus (order, descriptions, schemas, annotations), with the value that differs |
778
- | `menu/process-variance` | error | A second server process, started the same way, serves a different menu, so every restart misses the cache. Seen on mcp-atlassian: a default built from a Python set, in a new order every start (FINDINGS F12). toolmenu runs the main process with `PYTHONHASHSEED=0`, so a Python server's saved menu is reproducible, and the second with another seed. `--processes 1` turns it off |
779
- | `menu/connection-variance` | error (warn on 2025 protocols) | The same over HTTP: a second connection with the same credentials gets a different menu. Retried once before reporting |
780
- | `spec/schema` | error | A `tools/list` result that fails the official schema for its protocol version |
781
- | `spec/tools-capability` | error | The server doesn't declare the `tools` capability, so clients (the official SDK included) never ask for its tools |
782
- | `menu/duplicate-name` | error | Two tools with the same name: only one can be called, and some clients reject the list |
783
- | `naming/route` | error | A `routes.yml` expectation broke: a keyword now matches the wrong tool at least as well as the right one |
784
- | `description/buried` | warn | Instructions to the agent ("use X instead", "don't retry", "never guess one") past the point where your client cuts descriptions: in Claude Code 16,384 characters for a tool loaded through tool search (its default) and 4,096 for one that loads up front, or your `descriptionLimit`. From a real failure: a client cut at 280 characters, and the line that decided routing was at 1,222 |
785
- | `description/cut` | info | Descriptions longer than the client sends, with nothing that reads as an instruction past the cut, as one summary that shows where each one is cut. The model gets a prefix that can read as complete |
786
- | `description/late-instruction` | info | With the default cut: instructions after character 280, which a client that cuts descriptions short would hide, as one summary. Set `descriptionLimit` and `description/buried` checks your real cut instead. (Before 0.13 this was part of `description/cut`; that id's setting still applies.) |
787
- | `naming/vague-id` | warn | Parameters called just `id` that don't say *which* thing, as one summary with a suggested name |
788
- | `ids/authored` | info | An ID the agent must supply that no tool appears to return, so the agent may invent it, one finding per kind of ID (heuristic: IDs from a URL or another tool's text output look the same) |
789
- | `ids/handle-lifetime` | info | 2026-07-28: a tool that creates a handle another tool takes back (`start_session` → `session_id`) doesn't say how long it lasts (heuristic) |
790
- | `write/unannotated` | warn | Tools named like writes (`delete_`, `send_`…) with no annotations, as one summary grouped by the annotation each name suggests |
791
- | `naming/shared-word` | info | Nouns that name different things across tools (`list_team_audits` vs `get_audit_trail`), as one summary |
792
- | `write/read-only-mismatch` | warn | A tool marked `readOnlyHint: true` whose name says it writes (`create_my_account`): clients act on the hint, and `--auto` doesn't call it on the hint alone (heuristic) |
793
- | `write/no-dry-run` | info | Destructive tools with no dry-run parameter or preview tool, as one summary |
794
- | `spec/discover` | info | The server only speaks the 2025 protocol, so 2026-07-28 rules are skipped |
795
- | `spec/deprecated`, `spec/cache-hints` | info | 2026-07-28: deprecated features still advertised; `ttlMs: 0`, or `cacheScope: "public"` on a server that took credentials |
796
- | `lists/variance` | warn | With `--lists`: `prompts/list`, `resources/list` or `resources/templates/list` differs between two calls, or from a second process or connection (2026-07-28: lists no longer vary per connection) |
797
- | `lists/schema` | error | With `--lists`: a list result that doesn't match the official schema (2026-07-28 requires `ttlMs` and `cacheScope` on each), or a declared list that fails to answer (warn; info for Method not found, or a setup failure such as missing credentials) |
798
- | `naming/client-name` | warn | With `--client`: tool names the client's API refuses: `^[a-zA-Z0-9_-]{1,128}$` for the Claude API, `^[a-zA-Z0-9_-]+$` up to 128 for OpenAI's functions. MCP allows names such as `files.read` that neither takes as they are |
799
- | `menu/auth-variance` | info | With `--as` (or `identities` in the config): the tool set differs by identity, what each adds, lacks or defines differently against the first, and its tokens. Allowed by the spec; a baseline of one identity's menu doesn't cover the others. An identity that can't be listed is said here too |
800
- | `spec/cache-scope-public` | warn | 2026-07-28, with `--as`: the tool set differs by identity, and `tools/list` says `cacheScope: "public"`, which lets shared caches serve one caller's list to another |
801
- | `menu/capability-dependent` | info | Tools a server lists only to a client that supports elicitation, or only to one that doesn't. `snapshot` lists as a client with elicitation, so the menu file holds those tools (github-mcp-server 1.14 lists `delete_repository` only to a client that can ask the user to confirm; before 0.22 a baseline missed it), and once more without |
802
- | `menu/network-dependent` | warn | With `--offline-check` (stdio): the menu listed once more with the network cut off (the proxy variables pointed at a closed port, loopback left alone) differs: inite-visibility serves 3 tools offline and 10 online, so a baseline made with the network doesn't match a machine without it. A server that doesn't start offline is info. A server that ignores the proxy variables still reaches the network where the machine has it directly; then nothing is reported, which isn't a pass |
803
- | `auth/invalid-accepted` | warn | HTTP with credentials: the menu listed once more with an invalid credential in the same scheme. A server that serves a menu to it, not 401, gives a client whose token expired no reason to log in again; the agent just has fewer tools (Atlassian Rovo served 4 tools to any `Authorization` header). `--processes 1` turns it off |
804
-
805
- The full list, `diff` and `session` rules included, is [docs/RULES.md](docs/RULES.md).
806
-
807
- The naming and ID rules are heuristics, tuned on real servers (FINDINGS F7 and
808
- F10) and a private 115-tool one. They're the extras; the change checks are the point.
809
-
810
- **Pin routing with `routes.yml`.** Two tools fighting over one word needs a regression
811
- test, not a rename you'll forget:
200
+ | Registry stdio servers (npm), pinned versions | 1,089 | `snapshot`, `session --auto`, and every tool called in a container with no network and no credentials: 15,550 of 16,518 tools |
201
+ | Docker's MCP catalog, at the digests it pins | 237 | The same: 4,117 of 4,237 tools called (41 servers didn't list tools) |
202
+ | Servers vendors publish to run locally (GitHub, AWS Labs, Microsoft, PayPal, Stripe, Notion, HashiCorp, Grafana…) | 98 | The same, with dummy credentials: 2,213 of 2,292 tools called (20 didn't list tools) |
203
+ | Hosted servers: 109 vendor endpoints and a seeded 2,000 from the registry | 2,109 | `snapshot`, a read-only `session --auto` of at most 10 calls, `auth check` on each that answers 401 |
812
204
 
813
- ```yaml
814
- audit:
815
- must_match: [list_team_audits]
816
- must_not_match: [get_audit_trail]
817
- audit trail:
818
- must_not_match: [get_store_scores] # on its own: must not match at all
819
- ```
820
-
821
- Matching is keyword matching, the way a keyword or BM25 tool search sees the menu: a
822
- tool matches if every word of the route is in its name or description (the exact
823
- phrase scores higher). It can't read "not", and neither can those searches: a
824
- disclaimer like "not the audit trail" puts "audit trail" into exactly the tool you
825
- meant to steer away from, and toolmenu says so.
826
-
827
- **Config** (optional `toolmenu.config.json`):
828
-
829
- ```json
830
- {
831
- "rules": { "naming/shared-word": "off", "ids/authored": "error" },
832
- "ignore": ["debug_*"],
833
- "routes": "routes.yml",
834
- "tokenBudget": 4000,
835
- "descriptionLimit": "claude-code",
836
- "fullDescriptions": ["search_capabilities", "lookup_*"],
837
- "offlineCheck": true
838
- }
839
- ```
205
+ Read those numbers for what they are. Most servers pass most checks: about 1 in 90 of
206
+ the registry's stdio servers fails in the official SDK client. A sealed container with
207
+ made-up input shows whether a tool crashes, hangs or breaks its own schema, not whether
208
+ it does its job: more than half of the calls there came back as a readable tool error,
209
+ the right answer to made-up input or a dummy key. Hosted servers got a few read-only calls each.
840
210
 
841
- `offlineCheck` turns on `--offline-check` for every `snapshot` of a stdio server.
842
-
843
- `descriptionLimit` is where your client cuts tool descriptions: a number, or a client
844
- whose cut has been reported: `"claude-code"` (the default: Claude Code's own docs,
845
- [code.claude.com/docs/en/mcp](https://code.claude.com/docs/en/mcp), read 2026-11-02:
846
- 16,384 characters for a tool loaded through tool search, which is on by default, and
847
- 4,096 for a tool that loads up front, marked `_meta["anthropic/alwaysLoad"]: true`;
848
- the cut was 2,048 before v2.1.296), `"claude-code-upfront"` (4,096 for every tool:
849
- Claude Code with tool search off) or
850
- `"amazon-q"` (10,024, reported in
851
- [makenotion/notion-mcp-server#145](https://github.com/makenotion/notion-mcp-server/issues/145)).
852
- Clients that send descriptions in full don't need a setting beyond that. If your own
853
- client cuts shorter, as mine did at 280, set its number. `fullDescriptions` lists the
854
- tools it sends uncut, so the description rules skip them.
855
-
856
- ### Checking for one client (`--client`)
857
-
858
- What an agent sees depends on its client. `snapshot --client <name>` (or `"client"` in the
859
- config) checks the menu against what that client's own docs say, and every value
860
- toolmenu uses names its source and the date it was read (`src/clients.ts`). What a
861
- client doesn't document is left out, not guessed.
862
-
863
- | `--client` | Up front | Search | Description cut | Names |
864
- |---|---|---|---|---|
865
- | `claude-code` | tool names; full definitions for tools marked `_meta["anthropic/alwaysLoad"]` | reads a description up to 4,096 characters | 16,384 through search, 4,096 up front | Claude API's |
866
- | `claude-api-search` | nothing from tools you defer (keep the 3–5 used most loaded) | name, description, argument names and descriptions; 5 results by default | none documented | `^[a-zA-Z0-9_-]{1,128}$` |
867
- | `openai` | each deferred function's name and description (its schema loads on demand); a deferred MCP server shows only its own name and description | how many results, and which fields, isn't documented, so `search/findable` doesn't run | none documented | `^[a-zA-Z0-9_-]+$`, up to 128 |
868
- | `codex` | not documented | not documented | none documented | OpenAI's (it sends tools to the Responses API) |
869
- | `antigravity` | each tool's name; the agent reads a tool's schema before calling it, unless the user marks it `"eager"` | not a search: the agent reads the schema file | none documented | not documented |
870
- | `gemini-cli` | not documented | not documented | none documented | rewrites: characters outside letters, digits, `_ - . :` become `_`, names over 63 characters are cut in the middle |
871
- | `gemini-api` | not documented | not documented | none documented | letters, digits, `_ - . :`, up to 128 |
872
-
873
- Three more check what a client limits rather than what it loads:
874
-
875
- | `--client` | What it checks |
876
- |---|---|
877
- | `vscode` | VS Code (GitHub Copilot Chat) sends at most 128 tools a request: a server over it alone is `menu/client-limit` |
878
- | `copilot-cloud-agent` | uses MCP tools only, not prompts or resources (`menu/client-features`, with `--lists`) |
879
- | `copilot-code-review` | the same, and only tools marked `readOnlyHint: true`: the rest are listed |
211
+ The reference server, `@modelcontextprotocol/server-everything`, gets no false finding,
212
+ and a clean fixture server must get none from any command; the test suite fails if one
213
+ appears. What the runs found, including what toolmenu itself got wrong and how it was
214
+ fixed, is in [docs/FINDINGS.md](https://github.com/niksa90/toolmenu/blob/main/docs/FINDINGS.md). The corpora, the scripts and the
215
+ datasets are in [bench/](bench/README.md).
880
216
 
881
- Read, but nothing in them toolmenu can check yet: Copilot CLI defers tools behind a
882
- search at about 30 (it searches names, descriptions and parameters; how many results
883
- isn't said); Copilot in JetBrains, Visual Studio, Eclipse and Xcode documents no tool
884
- limit; JetBrains AI Assistant and Junie document none either (Junie CLI times out any
885
- request after 60 s, startup and `tools/list` included). Not here, because their docs
886
- don't state it: Cursor's tool limit (forum posts say 40), Windsurf's (100 in older docs,
887
- not in the current page), ChatGPT's. A value comes in when the client documents it.
888
-
889
- The snapshot adds a line such as `as Claude Code: up front ~524 tokens · on demand ~344
890
- per tool a search loads (largest: update_contacts, ~2,093)`, and `--json` a `client`
891
- object. With a client that defers tools, `search/findable` runs at any menu size.
892
-
893
- ## What toolmenu can't catch
217
+ ## What it can't catch
894
218
 
895
219
  toolmenu sees what the server sends. Plenty of agent failures happen elsewhere:
896
220
 
897
- - **Routing to a feature with no tools.** If the agent reaches for something your
898
- menu doesn't have, no menu check can see the gap.
899
- - **What the client does with the menu.** Truncating descriptions, sorting or
900
- filtering tools, merging several servers. The description rules use Claude Code's
901
- reported cut unless you set yours (`descriptionLimit`); they can't see your client.
902
- - **Your code and your logs.** A capability check that's answered with one yes/no
903
- for two different behaviours, or a log line that reads the same whether something
904
- happened or not. Those live in the client or the server's code.
905
- - **Whether the agent actually picks the right tool.** That needs evals with a model.
906
- toolmenu stays deterministic on purpose.
907
-
908
- ## Options
909
-
910
- ```
911
- --out <path> where to write the menu (default: menu.json)
912
- --no-write don't write the menu file
913
- --routes <path> routes.yml
914
- --config <path> config file (default: toolmenu.config.json, if present)
915
- --format <fmt> text (default), json, github (annotations), markdown, sarif
916
- (code scanning) or junit (CI test reports)
917
- --junit <path> snapshot, diff, session: also write the findings as JUnit XML
918
- --html <path> snapshot, diff, session: also write the report as one HTML page
919
- --fail-on <level> error (default), warn or info
920
- --header <k: v> HTTP header, repeatable
921
- --env <K=V> env var for a stdio server, repeatable (it only gets a
922
- minimal environment otherwise)
923
- --timeout <ms> per-request timeout (default: 30000)
924
- --protocol <p> auto (default), legacy (2025 handshake only) or modern
925
- (2026-07-28 only; an error if the server lacks it)
926
- --processes <n> server processes or connections to compare (default: 2);
927
- 1 opens no second one, session's scope check included
928
- --lists <kinds> snapshot: also read and check prompts, resources (comma-separated);
929
- saved in the menu file beside the tools
930
- --client <name> snapshot: check the menu for one client, from its own docs:
931
- claude-code, claude-api-search, openai, codex, vscode,
932
- copilot-cloud-agent, copilot-code-review, gemini-cli,
933
- gemini-api or antigravity
934
- --as <name>:<cred> snapshot: list the menu as this identity too, repeatable:
935
- "reader:Authorization: Bearer …" (HTTP) or "admin:ROLE=admin"
936
- (stdio). The first is the menu saved; the others are compared
937
- --offline-check snapshot (stdio): also list the menu with the network cut off
938
- --concurrency <n> scan: servers checked at a time (default: 4)
939
- --out-dir <dir> scan: write each server's menu file there, as <name>.json
940
- --allow-env <names> scan: variables a config's ${VAR} may be filled from (default:
941
- none; '*' for all)
942
- ```
943
-
944
- Exit codes: `0` clean · `1` findings at or above `--fail-on` · `2` couldn't connect
945
- (for `scan`: a server wasn't checked) or bad usage. `toolmenu <command> --help` lists that command's own options.
946
-
947
- Credentials on the command line (`--header`, `--env`, `--client-secret`) can be read by
948
- every user of the machine while toolmenu runs (`ps`). To keep them off it, put the
949
- options in `TOOLMENU_ARGS`, a JSON array of toolmenu options and their values; they go
950
- before the server command's `--`. A process's environment is readable only by its own
951
- user (and root), so this keeps them from other users, not from other processes of
952
- yours. jq builds the array whatever the token holds:
953
-
954
- ```sh
955
- TOOLMENU_ARGS=$(jq -cn --arg t "$TOKEN" '["--header", "Authorization: Bearer " + $t]') npx toolmenu snapshot https://example.com/mcp
956
- ```
957
-
958
- The GitHub Action passes its `headers` and `env` this way (to toolmenu 0.26.0 and
959
- later; an older `version` gets them as arguments, with a warning).
960
-
961
- ## Limitations
962
-
963
- - **Server order isn't final client order.** Clients merge tools from several servers
964
- and may sort or filter them. A stable server menu is necessary for cache hits, not
965
- sufficient. toolmenu measures what the server controls.
966
- - **Token counts are estimates** (`o200k_base`) of what the model reads: each tool's
967
- name, description and input schema. Output schemas, annotations, icons and `_meta`
968
- stay with the client and aren't counted. Vendors tokenize, cache and bill
969
- differently. `diff` recounts both menus with the toolmenu that runs it, so a
970
- baseline from an older version still compares fairly; a number quoted elsewhere (a
971
- commit message, a dashboard) should say which toolmenu counted it (0.7.1 changed the
972
- count for menus with icons by up to 5×, FINDINGS F7).
973
- - **Heuristic rules say so** in their messages.
974
- - Not a security scanner or a full conformance suite. Other tools do those well.
221
+ - **Whether the agent picks the right tool.** That needs evals with a model. toolmenu
222
+ stays deterministic on purpose; `routes.yml` pins keyword routing, which is as far as
223
+ it goes.
224
+ - **What the client does with the menu.** Clients merge several servers, sort, filter
225
+ and cut. A stable server menu is necessary for cache hits, not sufficient. The
226
+ `--client` profiles cover what each client documents, and nothing it doesn't.
227
+ - **A feature with no tools.** If the agent reaches for something your menu doesn't
228
+ have, no menu check sees the gap.
229
+ - **Whether a tool does its job.** A call that answers is checked against the schema it
230
+ declares, not against what it should have done.
231
+ - **Security.** It isn't a security scanner or a full conformance suite.
232
+
233
+ Token counts are estimates (`o200k_base`) of what the model reads: each tool's name,
234
+ description and input schema. Vendors tokenize, cache and bill differently.
235
+
236
+ ## Docs
237
+
238
+ - [snapshot](https://github.com/niksa90/toolmenu/blob/main/docs/snapshot.md): rules, where the tokens go, `routes.yml`, the config file, `scan`
239
+ - [session](https://github.com/niksa90/toolmenu/blob/main/docs/session.md): scenarios, `--init`, `--auto`, `--call-everything`, every session rule
240
+ - [diff, review, catalogs, history](https://github.com/niksa90/toolmenu/blob/main/docs/diff.md): how changes are classified, the verdicts, operations behind a search tool
241
+ - [clients](https://github.com/niksa90/toolmenu/blob/main/docs/clients.md): what each `--client` profile checks, and its sources
242
+ - [usage](https://github.com/niksa90/toolmenu/blob/main/docs/usage.md): HTTP and OAuth, proxies, the MCP server and plugin, the library, every option
243
+ - [the GitHub Action](https://github.com/niksa90/toolmenu/blob/main/docs/github-action.md) and [other CI output](https://github.com/niksa90/toolmenu/blob/main/docs/ci.md)
244
+ - [RULES](https://github.com/niksa90/toolmenu/blob/main/docs/RULES.md), [PROTOCOLS](https://github.com/niksa90/toolmenu/blob/main/docs/PROTOCOLS.md), [FINDINGS](https://github.com/niksa90/toolmenu/blob/main/docs/FINDINGS.md), [SPEC](https://github.com/niksa90/toolmenu/blob/main/docs/SPEC.md), [CHANGELOG](CHANGELOG.md)
975
245
 
976
246
  ## About
977
247
 
978
- Built by [Nikša Perović](https://niksa.me) while running a 115-tool MCP server and watching
979
- agents trip over its menu. Every rule started as one of those failures, and the
980
- research behind them is in [docs/FINDINGS.md](https://github.com/niksa90/toolmenu/blob/main/docs/FINDINGS.md). The story of building it, and where it was wrong, is [on niksa.me](https://niksa.me/toolmenu).
248
+ Built by [Nikša Perović](https://niksa.me) while running a 115-tool MCP server and
249
+ watching agents trip over its menu. It started as a check for menus that change at the
250
+ wrong time ([the story, and where it was wrong](https://niksa.me/toolmenu)) and grew
251
+ into a client-side test for the whole server as real servers kept failing in other
252
+ places.
981
253
 
982
254
  Found a false positive, or a failure toolmenu should catch?
983
255
  [Open an issue](https://github.com/niksa90/toolmenu/issues) with the menu (a
984
- `menu.json` is enough) and what you expected. Changes are listed in
985
- [CHANGELOG.md](https://github.com/niksa90/toolmenu/blob/main/CHANGELOG.md).
986
-
987
- ## License
256
+ `menu.json` is enough) and what you expected.
988
257
 
989
- MIT