toolmenu 0.11.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +77 -13
- package/dist/auth.js +8 -1
- package/dist/auto.d.ts +1 -1
- package/dist/auto.js +24 -1
- package/dist/breakdown.d.ts +33 -6
- package/dist/breakdown.js +129 -16
- package/dist/catalog.js +3 -3
- package/dist/cli.js +39 -11
- package/dist/compare.js +20 -5
- package/dist/diff.d.ts +13 -1
- package/dist/diff.js +671 -72
- package/dist/failures.d.ts +46 -0
- package/dist/failures.js +86 -0
- package/dist/init.d.ts +8 -3
- package/dist/init.js +14 -7
- package/dist/probe.d.ts +15 -3
- package/dist/probe.js +35 -11
- package/dist/rules/description.js +4 -1
- package/dist/rules/determinism.js +2 -1
- package/dist/rules/index.js +3 -1
- package/dist/rules/spec.d.ts +2 -0
- package/dist/rules/spec.js +29 -0
- package/dist/semver.js +1 -1
- package/dist/session.d.ts +42 -3
- package/dist/session.js +354 -40
- package/dist/snapshot.js +8 -5
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -29,7 +29,7 @@ No LLM anywhere: the same inputs give the same answer, so it can sit in CI.
|
|
|
29
29
|
(`history` installs old versions with today's dependencies, so it records what they
|
|
30
30
|
serve now, which can differ from what they shipped with: FINDINGS F4.)
|
|
31
31
|
|
|
32
|
-
> **Status: 0.
|
|
32
|
+
> **Status: 0.12, early.** Spec in [docs/SPEC.md](https://github.com/niksa90/toolmenu/blob/main/docs/SPEC.md). What it has found on
|
|
33
33
|
> real servers: [docs/FINDINGS.md](https://github.com/niksa90/toolmenu/blob/main/docs/FINDINGS.md).
|
|
34
34
|
|
|
35
35
|
## Start here
|
|
@@ -46,8 +46,9 @@ Commit both, open a pull request, and toolmenu comments on it.
|
|
|
46
46
|
npx toolmenu init -- node dist/server.js # add --with-session to also run session --auto
|
|
47
47
|
```
|
|
48
48
|
|
|
49
|
-
|
|
50
|
-
|
|
49
|
+
Every value you pass with `--env` or `--header` goes into the workflow as
|
|
50
|
+
`${{ secrets.… }}`, never as text (a name like `DATABASE_URL` doesn't say it holds a
|
|
51
|
+
password); `init` says which secrets to add. It never overwrites a file.
|
|
51
52
|
|
|
52
53
|
By hand:
|
|
53
54
|
|
|
@@ -90,7 +91,7 @@ jobs:
|
|
|
90
91
|
- uses: actions/setup-node@v7
|
|
91
92
|
with: { node-version: 22 }
|
|
92
93
|
- run: npm ci && npm run build
|
|
93
|
-
- uses: niksa90/toolmenu@v0.
|
|
94
|
+
- uses: niksa90/toolmenu@v0.12.0
|
|
94
95
|
with:
|
|
95
96
|
command: node dist/server.js
|
|
96
97
|
baseline: menu.json # your committed snapshot
|
|
@@ -141,9 +142,19 @@ steps:
|
|
|
141
142
|
`--init` writes a starter scenario from the live menu: it calls every read-only tool
|
|
142
143
|
that needs no arguments, then repeats one. If a tool looks like it unlocks more tools
|
|
143
144
|
(a `domains` or `toolset` parameter with an enum, "unlock" or "capabilities" in its
|
|
144
|
-
name
|
|
145
|
-
|
|
146
|
-
|
|
145
|
+
name, a description that says it enables or loads tools), the starter unlocks
|
|
146
|
+
**every** value, listing after each, then repeats the first, so the session sees every
|
|
147
|
+
tool behind the unlock. The values come from the parameter's enum or, without one,
|
|
148
|
+
from the server's own read-only listing (`list_toolsets`). Lookups that only describe
|
|
149
|
+
toolsets (`list_toolsets`, `get_toolset_tools`) aren't unlocks, and one not marked
|
|
150
|
+
read-only is suggested, never called. Anything it can't fill in is left as a
|
|
151
|
+
commented-out step. An unlock nothing in the menu names, like joining a room that
|
|
152
|
+
brings more tools, needs a step you add yourself. A scenario is only as good as its
|
|
153
|
+
steps, so the starter is a floor, not a ceiling.
|
|
154
|
+
|
|
155
|
+
On github-mcp-server 1.0.5 with `--dynamic-toolsets` (3 tools at the start, 19
|
|
156
|
+
toolsets), unlocking every toolset reaches 107 tools: all 81 that `--toolsets all`
|
|
157
|
+
serves, and 23 issue and pull-request tools that only the dynamic mode has.
|
|
147
158
|
|
|
148
159
|
| Rule | Default | Catches |
|
|
149
160
|
|---|---|---|
|
|
@@ -156,16 +167,26 @@ as its steps, so the starter is a floor, not a ceiling.
|
|
|
156
167
|
| `session/untested` | warn, error if no call got through | Calls that failed before reaching the tool: authentication, something missing on this machine (no Chrome), the network. One finding for the run, not one per step, so an expired CI secret doesn't pass a run that tested nothing |
|
|
157
168
|
| `session/refused`, `session/step-failed` | error | A write the scenario didn't allow, or a call that failed for another reason |
|
|
158
169
|
| `session/tool-error` | warn | A tool that answered with an error for another reason (`isError`): the run tested less than it looks |
|
|
170
|
+
| `session/unlock-coverage` | warn | An unlock whose enum lists its values, called with only some of them: the tools behind the rest were never seen, so a baseline from the session misses them. With `--union-out` or `--auto`, an unlock never called at all too, with why `--auto` skipped it |
|
|
171
|
+
| `session/nothing-called` | warn | `--auto` called no tools (all open-world, missing values, or not read-only): the run only listed the menu |
|
|
172
|
+
| `session/session-lost` | error | The server ended the session after toolmenu opened a second connection with the same credentials (servers with one session per client). The run stops there; rerun with `--processes 1` |
|
|
173
|
+
| `session/rate-limited` | error | The server kept refusing requests for being too many (a 429) after toolmenu waited about a minute, or a tool not marked `readOnlyHint` said it was rate-limited (it isn't called again). The run stops there, with the steps it didn't check; give the run its own server instance or raise the limit for it |
|
|
174
|
+
| `session/scope-unchecked` | info | A second server process or connection couldn't start mid-session (a server that holds a file or a port), so whether a change was global wasn't checked. Said once for the run |
|
|
159
175
|
|
|
160
176
|
**No scenario? `session --auto`** builds the steps from the menu: every read-only tool
|
|
161
177
|
whose required arguments the schema itself gives (a `default`, `examples`, an `enum`,
|
|
162
|
-
a type or format), cheapest first, then
|
|
178
|
+
a type or format), cheapest first, then every value of each read-only unlock (outside
|
|
179
|
+
`--max-calls`, and the first value twice: a second identical unlock should change
|
|
180
|
+
nothing), then the first call again. It never guesses an ID:
|
|
163
181
|
a tool that needs `owner`, `repo` or an issue key is skipped, and the report says
|
|
164
182
|
which values were missing (`--save-scenario auto.yml` writes the steps, with the
|
|
165
183
|
skipped tools commented out, to fill in). Tools marked `openWorldHint: true` (web
|
|
166
184
|
search, fetch, scraping, which can cost API credits) are called only with
|
|
167
|
-
`--open-world
|
|
168
|
-
|
|
185
|
+
`--open-world`, except unlocks: changing the menu is what a session watches, and an
|
|
186
|
+
unlock spends no search credits, so it runs even on a server that marks every tool
|
|
187
|
+
open-world. A run that ends up calling nothing is `session/nothing-called` (warn),
|
|
188
|
+
with why each tool was left out, not a clean pass. On the 22 servers in `bench/`,
|
|
189
|
+
`--auto` reaches 60 of 262 read-only tools, 84 with `--open-world`.
|
|
169
190
|
|
|
170
191
|
**`session` calls tools for real.** Without `allow_writes: true` it refuses any tool
|
|
171
192
|
not marked `readOnlyHint: true`. `--plan` prints the steps without connecting.
|
|
@@ -196,7 +217,26 @@ the way OpenAPI breaking-change checkers do it:
|
|
|
196
217
|
|---|---|---|
|
|
197
218
|
| **breaking** (error) | tool removed or renamed, parameter removed, new required parameter, narrower or different type, enum narrowed, a tool becoming less safe (no longer read-only, or additive → destructive) | `diff/tool-removed`, `diff/tool-renamed`, `diff/param-removed`, `diff/param-required`, `diff/param-type`, `diff/enum-narrowed`, `diff/safety-hint` |
|
|
198
219
|
| **minor** (info) | tool added, new optional parameter, parameter now optional, type or enum widened | `diff/tool-added`, `diff/param-added`, `diff/param-relaxed`, `diff/type-widened`, `diff/enum-widened` |
|
|
199
|
-
| **notice** (info) | optional parameter dropped (extra properties still allowed), description changed (with the text diff: it changes what the agent does), output schema, annotations, other fields, order | `diff/param-dropped`, `diff/description`, `diff/schema-other`, `diff/annotations`, `diff/other`, `diff/order` |
|
|
220
|
+
| **notice** (info) | optional parameter dropped (extra properties still allowed), description changed (with the text diff: it changes what the agent does), output schema, annotations, other fields, order, a schema restructured but accepting the same input | `diff/param-dropped`, `diff/description`, `diff/schema-other`, `diff/annotations`, `diff/other`, `diff/order`, `diff/schema-equivalent` |
|
|
221
|
+
|
|
222
|
+
The same rules apply inside parameters: fields of an object parameter, array items,
|
|
223
|
+
and the options of an `anyOf`/`oneOf` union (zod's unions, discriminated unions and
|
|
224
|
+
`.nullable()`) are compared at every depth, with their path (`gen.body.text`,
|
|
225
|
+
`search.filters[].field`, `gen.block(kind="image").url`). Union options are paired by
|
|
226
|
+
the value of a discriminator they all fix; without one (a plain `z.union` of objects),
|
|
227
|
+
identical options first, so a reorder is no change, then by type and shared property
|
|
228
|
+
names. An option gone is breaking, a new one widens. The same change reached through
|
|
229
|
+
one shared definition at several places is one finding that lists them (`places` in
|
|
230
|
+
`--json`). Local `$ref`s (`#/$defs/…`, `#/definitions/…`) are expanded first,
|
|
231
|
+
recursive ones one level deep, so a schema is compared by what it accepts, not how
|
|
232
|
+
it's spelled: moving a repeated block into `$defs` is one `diff/schema-equivalent`
|
|
233
|
+
notice with its token change ("accepts the same input: ~899 tokens fewer"), and a
|
|
234
|
+
breaking change inside `$defs` is breaking. A schema change no rule classifies is
|
|
235
|
+
never silent: it's a `diff/schema-other` "review it".
|
|
236
|
+
|
|
237
|
+
On mongodb-mcp-server 2.1.2 → 3.0.0 this finds `aggregate.pipeline[]…$vectorSearch.numCandidates`
|
|
238
|
+
and `.limit` narrowed from `number` to `integer` inside a union, where the whole
|
|
239
|
+
`pipeline` used to be one "review it".
|
|
200
240
|
|
|
201
241
|
It also reports the token change per tool and suggests a semver bump. To check the
|
|
202
242
|
bump, pass the **release** versions: `--release 1.4.0..1.5.0` (npm, a git tag).
|
|
@@ -232,6 +272,11 @@ notice ("not returned by the same queries"), never "removed". To steer it, set
|
|
|
232
272
|
`"catalog": { "queries": [...], "maxQueries": 200, "crawl": true }` in
|
|
233
273
|
`toolmenu.config.json` (FINDINGS F15).
|
|
234
274
|
|
|
275
|
+
A crawl through search is a lower bound, not a proof: an operation that nothing in the
|
|
276
|
+
catalog names, and whose words no query uses, stays unfound. On `@sentry/mcp-server`
|
|
277
|
+
0.42 (stdio) it finds 63 of the 64 operations in the package's catalog; `whoami` comes
|
|
278
|
+
back only when asked for by name. Add queries like that to `catalog.queries`.
|
|
279
|
+
|
|
235
280
|
## `history`: a package's releases, researched
|
|
236
281
|
|
|
237
282
|
```sh
|
|
@@ -248,6 +293,19 @@ It's best effort: packages that need env vars, arguments or auth are recorded as
|
|
|
248
293
|
failed, with a reason, never skipped. **It installs and runs third-party code. Run it
|
|
249
294
|
in a container.**
|
|
250
295
|
|
|
296
|
+
## Where the tokens go
|
|
297
|
+
|
|
298
|
+
`snapshot` also shows what the menu's tokens are spent on: the biggest tools (split
|
|
299
|
+
into description and schema), large enums, and waste you can remove:
|
|
300
|
+
|
|
301
|
+
- **Repeated blocks**: the same piece of schema at several places, inside one tool (a
|
|
302
|
+
`$defs` entry could hold it once: "one ~230-token block ×5 in
|
|
303
|
+
content_generateMessageHtml") or across tools (paid once per tool: mongodb-mcp-server
|
|
304
|
+
3.0.4 repeats a ~2,200-token `pipeline` schema in 3 tools). Only the largest repeated
|
|
305
|
+
block is named, not the pieces inside it.
|
|
306
|
+
- **Unused `$defs`**: definitions nothing in the tool refers to. Notion's server 2.5.2
|
|
307
|
+
ships the same 9 in all 24 tools, 72% of its menu.
|
|
308
|
+
|
|
251
309
|
## `snapshot` rules
|
|
252
310
|
|
|
253
311
|
Every rule comes from a failure I actually hit building 115 tools for one MCP
|
|
@@ -259,6 +317,8 @@ server, or from the spec itself. If a rule can't point to one, it doesn't ship.
|
|
|
259
317
|
| `menu/process-variance` | error | A second server process, started the same way, serves a different menu, so every restart misses the cache. Seen on mcp-atlassian: a default built from a Python set, in a new order every start (FINDINGS F12). toolmenu runs the main process with `PYTHONHASHSEED=0`, so a Python server's saved menu is reproducible, and the second with another seed. `--processes 1` turns it off |
|
|
260
318
|
| `menu/connection-variance` | error (warn on 2025 protocols) | The same over HTTP: a second connection with the same credentials gets a different menu. Retried once before reporting |
|
|
261
319
|
| `spec/schema` | error | A `tools/list` result that fails the official schema for its protocol version |
|
|
320
|
+
| `spec/tools-capability` | error | The server doesn't declare the `tools` capability, so clients (the official SDK included) never ask for its tools |
|
|
321
|
+
| `menu/duplicate-name` | error | Two tools with the same name: only one can be called, and some clients reject the list |
|
|
262
322
|
| `naming/route` | error | A `routes.yml` expectation broke: a keyword now matches the wrong tool at least as well as the right one |
|
|
263
323
|
| `description/buried` | warn | Instructions to the agent ("use X instead", "don't retry", "never guess one") past the point where your client cuts descriptions: 2,048 characters in Claude Code, or your `descriptionLimit`. From a real failure: a client cut at 280 characters, and the line that decided routing was at 1,222 |
|
|
264
324
|
| `description/cut` | info | Descriptions longer than the client sends, with nothing that reads as an instruction past the cut, as one summary. The model gets a prefix that can read as complete |
|
|
@@ -342,7 +402,8 @@ toolmenu sees what the server sends. Plenty of agent failures happen elsewhere:
|
|
|
342
402
|
--env <K=V> env var for a stdio server, repeatable (it only gets a
|
|
343
403
|
minimal environment otherwise)
|
|
344
404
|
--timeout <ms> per-request timeout (default: 30000)
|
|
345
|
-
--processes <n> server processes or connections to compare (default: 2)
|
|
405
|
+
--processes <n> server processes or connections to compare (default: 2);
|
|
406
|
+
1 opens no second one, session's scope check included
|
|
346
407
|
```
|
|
347
408
|
|
|
348
409
|
Exit codes: `0` clean · `1` findings at or above `--fail-on` · `2` couldn't connect
|
|
@@ -356,7 +417,10 @@ or bad usage. `toolmenu --help` lists the `session` and `history` options.
|
|
|
356
417
|
- **Token counts are estimates** (`o200k_base`) of what the model reads: each tool's
|
|
357
418
|
name, description and input schema. Output schemas, annotations, icons and `_meta`
|
|
358
419
|
stay with the client and aren't counted. Vendors tokenize, cache and bill
|
|
359
|
-
differently.
|
|
420
|
+
differently. `diff` recounts both menus with the toolmenu that runs it, so a
|
|
421
|
+
baseline from an older version still compares fairly; a number quoted elsewhere (a
|
|
422
|
+
commit message, a dashboard) should say which toolmenu counted it (0.7.1 changed the
|
|
423
|
+
count for menus with icons by up to 5×, FINDINGS F7).
|
|
360
424
|
- **Heuristic rules say so** in their messages.
|
|
361
425
|
- Not a security scanner or a full conformance suite. Other tools do those well.
|
|
362
426
|
|
package/dist/auth.js
CHANGED
|
@@ -294,7 +294,14 @@ function openInBrowser(url) {
|
|
|
294
294
|
if (scope)
|
|
295
295
|
process.stderr.write(`Asking for the server's default scopes: ${scope} (narrower: --scope; toolmenu never calls a write tool on its own)\n`);
|
|
296
296
|
process.stderr.write(`Opening your browser to log in. If it doesn't open, visit:\n ${url.href}\n`);
|
|
297
|
-
|
|
297
|
+
// Not `cmd /c start` on Windows: cmd reads the URL's `&` as a command separator,
|
|
298
|
+
// cutting the URL, and the URL comes from the server's metadata. rundll32 opens
|
|
299
|
+
// it in the default browser without a shell.
|
|
300
|
+
const [cmd, args] = process.platform === 'darwin'
|
|
301
|
+
? ['open', [url.href]]
|
|
302
|
+
: process.platform === 'win32'
|
|
303
|
+
? ['rundll32', ['url.dll,FileProtocolHandler', url.href]]
|
|
304
|
+
: ['xdg-open', [url.href]];
|
|
298
305
|
try {
|
|
299
306
|
spawn(cmd, args, { detached: true, stdio: 'ignore' }).on('error', () => { }).unref();
|
|
300
307
|
}
|
package/dist/auto.d.ts
CHANGED
package/dist/auto.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { synthesizeArgs } from './args.js';
|
|
2
|
+
import { MAX_UNLOCKS, unlockers } from './session.js';
|
|
2
3
|
import { COLLECTION_VERBS, LOOKUP_VERBS, verbOf } from './words.js';
|
|
3
4
|
/**
|
|
4
5
|
* A scenario built from the menu, no hand-written steps: every read-only tool
|
|
@@ -11,7 +12,21 @@ export function autoScenario(menu, options = {}) {
|
|
|
11
12
|
const maxCalls = options.maxCalls ?? 20;
|
|
12
13
|
const skipped = [];
|
|
13
14
|
const candidates = [];
|
|
15
|
+
// Read-only unlocks with every value in an enum: each value, outside the call
|
|
16
|
+
// budget, so the tools behind every toolset are seen (GitHub: 19 toolsets).
|
|
17
|
+
// Called even when marked openWorldHint, without --open-world: an unlock changes
|
|
18
|
+
// the menu, which is what a session watches, and spends no search credits. A
|
|
19
|
+
// server whose every tool calls an outside service marks its unlock that way too,
|
|
20
|
+
// and --auto then tested nothing.
|
|
21
|
+
const unlocks = unlockers(menu.tools).filter((u) => {
|
|
22
|
+
const a = u.tool.annotations ?? {};
|
|
23
|
+
const required = u.tool.inputSchema?.required ?? [];
|
|
24
|
+
return u.param && u.values.length && a.readOnlyHint === true && required.every((p) => p === u.param);
|
|
25
|
+
});
|
|
26
|
+
const unlocking = new Set(unlocks.map((u) => u.tool.name));
|
|
14
27
|
for (const tool of menu.tools) {
|
|
28
|
+
if (unlocking.has(tool.name))
|
|
29
|
+
continue;
|
|
15
30
|
const a = tool.annotations ?? {};
|
|
16
31
|
if (a.readOnlyHint !== true) {
|
|
17
32
|
skipped.push({ tool: tool.name, reason: 'not read-only' });
|
|
@@ -40,9 +55,17 @@ export function autoScenario(menu, options = {}) {
|
|
|
40
55
|
const steps = [{ kind: 'list' }];
|
|
41
56
|
for (const c of chosen)
|
|
42
57
|
steps.push({ kind: 'call', tool: c.tool.name, args: c.args });
|
|
58
|
+
for (const u of unlocks) {
|
|
59
|
+
const array = u.tool.inputSchema?.properties?.[u.param]?.type === 'array';
|
|
60
|
+
const values = u.values.slice(0, MAX_UNLOCKS);
|
|
61
|
+
for (const v of values)
|
|
62
|
+
steps.push({ kind: 'call', tool: u.tool.name, args: { [u.param]: array ? [v] : v } });
|
|
63
|
+
// The same unlock again should change nothing, as in the --init starter.
|
|
64
|
+
steps.push({ kind: 'call', tool: u.tool.name, args: { [u.param]: array ? [values[0]] : values[0] } });
|
|
65
|
+
}
|
|
43
66
|
if (chosen.length)
|
|
44
67
|
steps.push({ kind: 'call', tool: chosen[0].tool.name, args: chosen[0].args });
|
|
45
|
-
return { scenario: { allowWrites: false, steps }, called: chosen.map((c) => c.tool.name), skipped };
|
|
68
|
+
return { scenario: { allowWrites: false, steps }, called: [...chosen.map((c) => c.tool.name), ...unlocks.map((u) => u.tool.name)], skipped };
|
|
46
69
|
}
|
|
47
70
|
function rank(tool, args) {
|
|
48
71
|
if (Object.keys(args).length === 0)
|
package/dist/breakdown.d.ts
CHANGED
|
@@ -16,13 +16,40 @@ export interface Breakdown {
|
|
|
16
16
|
values: number;
|
|
17
17
|
tokens: number;
|
|
18
18
|
}[];
|
|
19
|
-
/**
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
19
|
+
/**
|
|
20
|
+
* A block of schema, byte for byte the same content, at several places: inside
|
|
21
|
+
* one tool (a `$defs` entry could hold it once) or across tools (paid once per
|
|
22
|
+
* tool: a tool's schema can't refer to another's). The largest repeated block
|
|
23
|
+
* only, not the pieces inside it.
|
|
24
|
+
*/
|
|
25
|
+
repeated: Repeated[];
|
|
26
|
+
/**
|
|
27
|
+
* `$defs`/`definitions` entries that nothing in their tool refers to, followed
|
|
28
|
+
* transitively: sent with every conversation, read by no one. Notion's server
|
|
29
|
+
* 2.5.2 ships the same 9 in all 24 tools, 72% of its menu.
|
|
30
|
+
*/
|
|
31
|
+
unusedDefs: {
|
|
23
32
|
tools: number;
|
|
24
|
-
|
|
25
|
-
|
|
33
|
+
tokens: number;
|
|
34
|
+
examples: string[];
|
|
35
|
+
};
|
|
36
|
+
}
|
|
37
|
+
export interface Repeated {
|
|
38
|
+
/** The property it sits under (the first place it's found), for a label. */
|
|
39
|
+
param: string;
|
|
40
|
+
tokens: number;
|
|
41
|
+
/** Places it appears, in all. */
|
|
42
|
+
count: number;
|
|
43
|
+
tools: number;
|
|
44
|
+
/** tokens × count. */
|
|
45
|
+
total: number;
|
|
46
|
+
/** The most copies inside one tool, and that tool. */
|
|
47
|
+
within: number;
|
|
48
|
+
withinTool: string;
|
|
49
|
+
/** Tokens a `$defs` entry could save inside the tools that repeat it (estimate). */
|
|
50
|
+
saving: number;
|
|
51
|
+
/** Up to three places, as `tool.path`. */
|
|
52
|
+
where: string[];
|
|
26
53
|
}
|
|
27
54
|
/** Where a menu's tokens go. Estimates, like every count in toolmenu. */
|
|
28
55
|
export declare function breakdown(menu: Menu): Breakdown;
|
package/dist/breakdown.js
CHANGED
|
@@ -2,8 +2,12 @@ import { canonical } from './compare.js';
|
|
|
2
2
|
import { countTokens } from './menu.js';
|
|
3
3
|
const TOP = 5;
|
|
4
4
|
const BIG_ENUM = 20;
|
|
5
|
-
const REPEATED_MIN_TOKENS =
|
|
5
|
+
const REPEATED_MIN_TOKENS = 25;
|
|
6
|
+
const SCHEMA_KEYWORDS = new Set(['anyOf', 'oneOf', 'allOf', 'not', 'items', 'additionalProperties', 'prefixItems', 'then', 'else', 'if', '$defs', 'definitions', 'patternProperties']);
|
|
7
|
+
/** Across tools only, a block has to be in this many to count (the same small parameter in two tools is normal). */
|
|
6
8
|
const REPEATED_MIN_TOOLS = 3;
|
|
9
|
+
/** What a `{"$ref": "#/$defs/Block"}` costs in place of a copy (estimate). */
|
|
10
|
+
const REF_TOKENS = countTokens(JSON.stringify({ $ref: '#/$defs/Block' }));
|
|
7
11
|
/** Where a menu's tokens go. Estimates, like every count in toolmenu. */
|
|
8
12
|
export function breakdown(menu) {
|
|
9
13
|
const total = menu.totalTokens || 1;
|
|
@@ -18,25 +22,121 @@ export function breakdown(menu) {
|
|
|
18
22
|
schema: t.inputSchema ? countTokens(JSON.stringify(t.inputSchema)) : 0,
|
|
19
23
|
}));
|
|
20
24
|
const enums = [];
|
|
21
|
-
const seen = new Map();
|
|
22
25
|
for (const tool of menu.tools) {
|
|
23
26
|
for (const [param, schema] of Object.entries(tool.inputSchema?.properties ?? {})) {
|
|
24
27
|
const values = enumValues(schema);
|
|
25
28
|
if (values.length >= BIG_ENUM)
|
|
26
29
|
enums.push({ tool: tool.name, param, values: values.length, tokens: countTokens(JSON.stringify(values)) });
|
|
27
|
-
const key = `${param}\u0000${canonical(schema)}`;
|
|
28
|
-
const entry = seen.get(key) ?? { param, tokens: countTokens(JSON.stringify({ [param]: schema })), tools: new Set() };
|
|
29
|
-
entry.tools.add(tool.name);
|
|
30
|
-
seen.set(key, entry);
|
|
31
30
|
}
|
|
32
31
|
}
|
|
33
|
-
const repeated = [...seen.values()]
|
|
34
|
-
.filter((e) => e.tokens >= REPEATED_MIN_TOKENS && e.tools.size >= REPEATED_MIN_TOOLS)
|
|
35
|
-
.map((e) => ({ param: e.param, tokens: e.tokens, tools: e.tools.size, total: e.tokens * e.tools.size }))
|
|
36
|
-
.sort((a, b) => b.total - a.total)
|
|
37
|
-
.slice(0, 5);
|
|
38
32
|
enums.sort((a, b) => b.tokens - a.tokens);
|
|
39
|
-
return { total: menu.totalTokens, top, enums: enums.slice(0, 5), repeated };
|
|
33
|
+
return { total: menu.totalTokens, top, enums: enums.slice(0, 5), repeated: repeatedBlocks(menu), unusedDefs: unusedDefs(menu) };
|
|
34
|
+
}
|
|
35
|
+
function unusedDefs(menu) {
|
|
36
|
+
let tokens = 0;
|
|
37
|
+
const tools = [];
|
|
38
|
+
for (const tool of menu.tools) {
|
|
39
|
+
const schema = (tool.inputSchema ?? {});
|
|
40
|
+
const pools = ['$defs', 'definitions'].filter((k) => schema[k] && typeof schema[k] === 'object');
|
|
41
|
+
if (!pools.length)
|
|
42
|
+
continue;
|
|
43
|
+
const { $defs: _d, definitions: _df, ...rest } = schema;
|
|
44
|
+
const used = new Set();
|
|
45
|
+
const queue = [JSON.stringify(rest)];
|
|
46
|
+
while (queue.length) {
|
|
47
|
+
for (const m of queue.pop().matchAll(/"\$ref"\s*:\s*"#\/(\$defs|definitions)\/([^"/]+)/g)) {
|
|
48
|
+
const id = `${m[1]}/${decodeURIComponent(m[2]).replace(/~1/g, '/').replace(/~0/g, '~')}`;
|
|
49
|
+
if (used.has(id))
|
|
50
|
+
continue;
|
|
51
|
+
used.add(id);
|
|
52
|
+
const [pool, name] = [m[1], id.slice(m[1].length + 1)];
|
|
53
|
+
queue.push(JSON.stringify(schema[pool][name] ?? {}));
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
let unused = 0;
|
|
57
|
+
for (const pool of pools) {
|
|
58
|
+
for (const [name, def] of Object.entries(schema[pool])) {
|
|
59
|
+
if (!used.has(`${pool}/${name}`))
|
|
60
|
+
unused += countTokens(JSON.stringify({ [name]: def }));
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
if (unused > 0) {
|
|
64
|
+
tokens += unused;
|
|
65
|
+
tools.push(tool.name);
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
return { tools: tools.length, tokens, examples: tools.slice(0, 3) };
|
|
69
|
+
}
|
|
70
|
+
/** Every sub-schema of every tool's inputSchema, fingerprinted, and the ones that repeat. */
|
|
71
|
+
function repeatedBlocks(menu) {
|
|
72
|
+
const places = new Map();
|
|
73
|
+
const tokens = new Map();
|
|
74
|
+
/** `map`: a name → schema map (properties, $defs), not a schema itself. */
|
|
75
|
+
const visit = (tool, node, path, map, ancestors) => {
|
|
76
|
+
if (!node || typeof node !== 'object')
|
|
77
|
+
return;
|
|
78
|
+
if (Array.isArray(node)) {
|
|
79
|
+
node.forEach((x, i) => visit(tool, x, `${path}[${i}]`, false, ancestors));
|
|
80
|
+
return;
|
|
81
|
+
}
|
|
82
|
+
let inner = ancestors;
|
|
83
|
+
if (path !== '' && !map) {
|
|
84
|
+
const key = canonical(node);
|
|
85
|
+
if (!tokens.has(key))
|
|
86
|
+
tokens.set(key, countTokens(JSON.stringify(node)));
|
|
87
|
+
places.set(key, [...(places.get(key) ?? []), { tool, path, ancestors }]);
|
|
88
|
+
inner = [...ancestors, key];
|
|
89
|
+
}
|
|
90
|
+
for (const [k, v] of Object.entries(node)) {
|
|
91
|
+
if (map)
|
|
92
|
+
visit(tool, v, `${path}.${k}`, false, inner);
|
|
93
|
+
else if (k === 'properties')
|
|
94
|
+
visit(tool, v, path, true, inner);
|
|
95
|
+
else if (k === '$defs' || k === 'definitions' || k === 'patternProperties')
|
|
96
|
+
visit(tool, v, `${path}.${k}`, true, inner);
|
|
97
|
+
else if (k === 'items' && !Array.isArray(v))
|
|
98
|
+
visit(tool, v, `${path}[]`, false, inner);
|
|
99
|
+
else
|
|
100
|
+
visit(tool, v, `${path}.${k}`, false, inner);
|
|
101
|
+
}
|
|
102
|
+
};
|
|
103
|
+
for (const tool of menu.tools)
|
|
104
|
+
visit(tool.name, tool.inputSchema, '', false, []);
|
|
105
|
+
const counts = new Map([...places].map(([k, occ]) => [k, occ.length]));
|
|
106
|
+
const out = [];
|
|
107
|
+
for (const [key, occ] of places) {
|
|
108
|
+
const size = tokens.get(key);
|
|
109
|
+
if (occ.length < 2 || size < REPEATED_MIN_TOKENS)
|
|
110
|
+
continue;
|
|
111
|
+
// Inside a bigger block that repeats as often: that one is the finding.
|
|
112
|
+
if (occ.every((o) => o.ancestors.some((a) => (tokens.get(a) ?? 0) >= REPEATED_MIN_TOKENS && (counts.get(a) ?? 0) >= occ.length)))
|
|
113
|
+
continue;
|
|
114
|
+
const perTool = new Map();
|
|
115
|
+
for (const o of occ)
|
|
116
|
+
perTool.set(o.tool, (perTool.get(o.tool) ?? 0) + 1);
|
|
117
|
+
const [withinTool, within] = [...perTool].sort((a, b) => b[1] - a[1])[0];
|
|
118
|
+
if (within < 2 && perTool.size < REPEATED_MIN_TOOLS)
|
|
119
|
+
continue;
|
|
120
|
+
const saving = [...perTool.values()].filter((k) => k > 1).reduce((s, k) => s + Math.max(0, (k - 1) * size - k * REF_TOKENS), 0);
|
|
121
|
+
// The nearest property name: `.freshness.anyOf[1]` is freshness.
|
|
122
|
+
const label = occ[0].path.split(/[.[\]]+/).filter((s) => s && !/^\d+$/.test(s) && !SCHEMA_KEYWORDS.has(s)).pop() ?? occ[0].path;
|
|
123
|
+
out.push({
|
|
124
|
+
param: label,
|
|
125
|
+
tokens: size,
|
|
126
|
+
count: occ.length,
|
|
127
|
+
tools: perTool.size,
|
|
128
|
+
total: size * occ.length,
|
|
129
|
+
within,
|
|
130
|
+
withinTool,
|
|
131
|
+
saving,
|
|
132
|
+
where: occ.slice(0, 3).map((o) => `${o.tool}${o.path}`),
|
|
133
|
+
waste: size * (occ.length - 1),
|
|
134
|
+
});
|
|
135
|
+
}
|
|
136
|
+
return out
|
|
137
|
+
.sort((a, b) => b.waste - a.waste)
|
|
138
|
+
.slice(0, 5)
|
|
139
|
+
.map(({ waste: _w, ...r }) => r);
|
|
40
140
|
}
|
|
41
141
|
function enumValues(schema) {
|
|
42
142
|
if (Array.isArray(schema.enum))
|
|
@@ -47,17 +147,30 @@ function enumValues(schema) {
|
|
|
47
147
|
}
|
|
48
148
|
/** The breakdown as report lines, or none for a menu too small to need one. */
|
|
49
149
|
export function breakdownLines(b, tools) {
|
|
50
|
-
if (tools < 5)
|
|
51
|
-
return [];
|
|
52
150
|
const n = (x) => `~${x.toLocaleString('en-US')}`;
|
|
151
|
+
const repeated = b.repeated.map((r) => {
|
|
152
|
+
const inside = r.within > 1 ? `${r.tools === 1 ? '' : ', '}×${r.within} inside ${r.withinTool}${r.saving > 0 ? `: a $defs entry there could save ${n(r.saving)}` : ''}` : '';
|
|
153
|
+
if (r.tools === 1)
|
|
154
|
+
return ` repeated: one ${n(r.tokens)}-token block ×${r.count} in ${r.withinTool} (${r.where.map((w) => w.slice(r.withinTool.length + 1)).join(', ')}${r.count > r.where.length ? ', …' : ''})${r.saving > 0 ? `: a $defs entry could save ${n(r.saving)}` : ''}`;
|
|
155
|
+
return ` repeated: ${r.param} (${n(r.tokens)}) in ${r.tools} tools, ${n(r.total)} in all${inside}`;
|
|
156
|
+
});
|
|
53
157
|
const pct = (x) => `${Math.round(x * 100)}%`;
|
|
158
|
+
const u = b.unusedDefs;
|
|
159
|
+
const unused = u.tokens
|
|
160
|
+
? [` unused $defs: ${n(u.tokens)} tokens${b.total ? ` (${pct(u.tokens / b.total)} of the menu)` : ''} in ${u.tools} tool${u.tools === 1 ? '' : 's'} (${u.examples.join(', ')}${u.tools > u.examples.length ? ', …' : ''}): definitions nothing in the tool refers to`]
|
|
161
|
+
: [];
|
|
162
|
+
// A small menu gets no breakdown, but waste inside one tool is worth a line anyway.
|
|
163
|
+
if (tools < 5) {
|
|
164
|
+
const within = [...unused, ...repeated.filter((_, i) => b.repeated[i].within > 1)];
|
|
165
|
+
return within.length ? ['Where the tokens go (estimate):', ...within] : [];
|
|
166
|
+
}
|
|
54
167
|
const topShare = b.top.reduce((s, t) => s + t.share, 0);
|
|
55
168
|
const lines = [`Where the tokens go (estimate): the top ${b.top.length} tools are ${pct(topShare)} of the menu`];
|
|
56
169
|
for (const t of b.top)
|
|
57
170
|
lines.push(` ${pct(t.share).padStart(4)} ${t.name} ${n(t.tokens)} (description ${n(t.description)}, schema ${n(t.schema)})`);
|
|
171
|
+
lines.push(...unused);
|
|
58
172
|
for (const e of b.enums)
|
|
59
173
|
lines.push(` enum: ${e.tool}.${e.param} has ${e.values} values, ${n(e.tokens)} tokens`);
|
|
60
|
-
|
|
61
|
-
lines.push(` repeated: ${r.param} (${n(r.tokens)}) in ${r.tools} tools, ${n(r.total)} in all`);
|
|
174
|
+
lines.push(...repeated);
|
|
62
175
|
return lines;
|
|
63
176
|
}
|
package/dist/catalog.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { toolTokens } from './menu.js';
|
|
2
|
+
import { RATE_LIMITED, tooMany } from './failures.js';
|
|
2
3
|
import { nouns, singular } from './words.js';
|
|
3
4
|
const QUERY_PARAM = /^(q|query|search|keywords?|text)$/i;
|
|
4
5
|
const SEARCH_NAME = /(search|find|list|discover)[\w-]*(tools?|operations?|actions?|capabilit)|^discover$|(tools?|operations?)[\w-]*(search|discover)/i;
|
|
@@ -69,7 +70,7 @@ export async function readCatalog(conn, tools, options = {}) {
|
|
|
69
70
|
try {
|
|
70
71
|
const result = await conn.client.callTool({ name: tool.name, arguments: { [queryParam]: query, ...limit } }, { timeout: options.timeoutMs ?? 30_000 });
|
|
71
72
|
const text = result.isError ? resultText(result) : '';
|
|
72
|
-
if (result.isError &&
|
|
73
|
+
if (result.isError && RATE_LIMITED.test(text) && attempt < 3) {
|
|
73
74
|
await sleep(2000 * 2 ** attempt);
|
|
74
75
|
continue;
|
|
75
76
|
}
|
|
@@ -92,7 +93,7 @@ export async function readCatalog(conn, tools, options = {}) {
|
|
|
92
93
|
}
|
|
93
94
|
catch (error) {
|
|
94
95
|
const message = error instanceof Error ? error.message.split('\n')[0] : String(error);
|
|
95
|
-
if (
|
|
96
|
+
if (tooMany(error) && attempt < 3) {
|
|
96
97
|
await sleep(2000 * 2 ** attempt);
|
|
97
98
|
continue;
|
|
98
99
|
}
|
|
@@ -121,7 +122,6 @@ function describedExamples(tool) {
|
|
|
121
122
|
const text = [tool.description ?? '', ...Object.values(tool.inputSchema?.properties ?? {}).map((p) => String(p.description ?? ''))].join(' ');
|
|
122
123
|
return [...text.matchAll(/["“]([a-z][a-z0-9 ,'-]{5,60})["”]/g)].map((m) => m[1]).filter((q) => q.includes(' '));
|
|
123
124
|
}
|
|
124
|
-
const RATE_LIMIT = /rate.?limit|too many requests|\b429\b/i;
|
|
125
125
|
function sleep(ms) {
|
|
126
126
|
return new Promise((resolve) => setTimeout(resolve, ms));
|
|
127
127
|
}
|
package/dist/cli.js
CHANGED
|
@@ -7,13 +7,14 @@ import { diffMenus } from './diff.js';
|
|
|
7
7
|
import { loadMenu } from './menu.js';
|
|
8
8
|
import { history, historyCsv } from './history.js';
|
|
9
9
|
import { counts, formatDiff, formatHistory, formatPlan, formatSession, formatSnapshot } from './report.js';
|
|
10
|
-
import { loadScenario, session, starterScenario } from './session.js';
|
|
11
|
-
import {
|
|
10
|
+
import { loadScenario, session, starterScenario, unlockListers, valuesFromListing } from './session.js';
|
|
11
|
+
import { listTools } from './connect.js';
|
|
12
|
+
import { patiently, tooMany } from './failures.js';
|
|
12
13
|
import { buildMenu } from './menu.js';
|
|
13
14
|
import { existsSync } from 'node:fs';
|
|
14
15
|
import { loadRoutes } from './routes.js';
|
|
15
16
|
import { snapshot } from './snapshot.js';
|
|
16
|
-
import { MAIN_SEED, probeMenu, seeded } from './probe.js';
|
|
17
|
+
import { connectPatiently, MAIN_SEED, probeMenu, refusedForTooMany, seeded } from './probe.js';
|
|
17
18
|
import { autoScenario, scenarioYaml } from './auto.js';
|
|
18
19
|
import { authDir, listLogins, login, logout } from './auth.js';
|
|
19
20
|
import { detectProject, secretsNeeded, workflowYaml } from './init.js';
|
|
@@ -58,7 +59,8 @@ Options:
|
|
|
58
59
|
--catalog also read the operations behind a search tool (search and execute:
|
|
59
60
|
discover, search_*_tools) and keep them in the menu file for diff
|
|
60
61
|
--processes <n> server processes (stdio) or connections (HTTP) to compare,
|
|
61
|
-
the main one included (default: 2; 1
|
|
62
|
+
the main one included (default: 2; 1 opens no second one, for
|
|
63
|
+
session's scope check either: for servers with one session per client)
|
|
62
64
|
-h, --help show this help
|
|
63
65
|
-v, --version show the version
|
|
64
66
|
|
|
@@ -238,12 +240,29 @@ export async function main(argv) {
|
|
|
238
240
|
if (existsSync(path))
|
|
239
241
|
throw new UsageError(`${path} already exists. Pick another path with --scenario.`);
|
|
240
242
|
const initTarget = parseTarget(rest, command, values.header ?? [], values.env ?? [], values['no-auth']);
|
|
241
|
-
const conn = await
|
|
243
|
+
const conn = await connectPatiently(seeded(initTarget, MAIN_SEED), { timeoutMs });
|
|
242
244
|
try {
|
|
243
|
-
const list = await listTools(conn, { timeoutMs });
|
|
245
|
+
const list = await patiently(() => listTools(conn, { timeoutMs }), { error: tooMany });
|
|
244
246
|
const menu = buildMenu(list.tools, { name: conn.server.name, version: conn.server.version, protocolVersion: conn.protocolVersion });
|
|
245
|
-
|
|
246
|
-
|
|
247
|
+
// An unlock without an enum: its values come from the server's own read-only
|
|
248
|
+
// listing (list_toolsets), so the scenario can unlock every one.
|
|
249
|
+
const unlockValues = {};
|
|
250
|
+
const read = [];
|
|
251
|
+
for (const { unlock, lister } of unlockListers(menu.tools)) {
|
|
252
|
+
try {
|
|
253
|
+
const result = await conn.client.callTool({ name: lister, arguments: {} }, { timeout: timeoutMs });
|
|
254
|
+
const found = result.isError ? [] : valuesFromListing(result);
|
|
255
|
+
if (found.length) {
|
|
256
|
+
unlockValues[unlock] = found;
|
|
257
|
+
read.push(`${found.length} values for ${unlock} from ${lister}`);
|
|
258
|
+
}
|
|
259
|
+
}
|
|
260
|
+
catch {
|
|
261
|
+
// Left as TODO in the scenario.
|
|
262
|
+
}
|
|
263
|
+
}
|
|
264
|
+
await writeFile(path, starterScenario(menu, { values: unlockValues }));
|
|
265
|
+
process.stdout.write(`wrote ${path}: a starter scenario for ${menu.tools.length} tools${read.length ? ` (read ${read.join('; ')})` : ''}. Fill in the TODOs, then run toolmenu session --scenario ${path}\n`);
|
|
247
266
|
}
|
|
248
267
|
finally {
|
|
249
268
|
await conn.close().catch(() => { });
|
|
@@ -289,7 +308,10 @@ export async function main(argv) {
|
|
|
289
308
|
if (savePath && existsSync(savePath))
|
|
290
309
|
throw new UsageError(`${savePath} already exists. Pick another path for --save-scenario.`);
|
|
291
310
|
const autoTarget = parseTarget(rest, command, values.header ?? [], values.env ?? [], values['no-auth']);
|
|
292
|
-
const
|
|
311
|
+
const waited = { ms: 0 };
|
|
312
|
+
const menu = await probeMenu(seeded(autoTarget, MAIN_SEED), timeoutMs, undefined, waited).catch((error) => {
|
|
313
|
+
throw refusedForTooMany(error, waited.ms);
|
|
314
|
+
});
|
|
293
315
|
const plan = autoScenario(menu, { openWorld: values['open-world'], maxCalls });
|
|
294
316
|
if (savePath)
|
|
295
317
|
await writeFile(savePath, scenarioYaml(plan, menu.server.name));
|
|
@@ -297,7 +319,7 @@ export async function main(argv) {
|
|
|
297
319
|
process.stdout.write(formatPlan(plan.scenario, 'auto') + '\n');
|
|
298
320
|
return 0;
|
|
299
321
|
}
|
|
300
|
-
const result = await session(autoTarget, plan.scenario, { timeoutMs, processes, rules: config.rules, ignore: config.ignore, scenarioName: 'auto' });
|
|
322
|
+
const result = await session(autoTarget, plan.scenario, { timeoutMs, processes, rules: config.rules, ignore: config.ignore, scenarioName: 'auto', unionOut: !!values['union-out'], auto: { called: plan.called, skipped: plan.skipped } });
|
|
301
323
|
result.auto = { called: plan.called, skipped: plan.skipped };
|
|
302
324
|
if (values['union-out'])
|
|
303
325
|
await writeFile(values['union-out'], JSON.stringify(result.union, null, 2) + '\n');
|
|
@@ -321,7 +343,7 @@ export async function main(argv) {
|
|
|
321
343
|
return 0;
|
|
322
344
|
}
|
|
323
345
|
const sessionTarget = parseTarget(rest, command, values.header ?? [], values.env ?? [], values['no-auth']);
|
|
324
|
-
const result = await session(sessionTarget, scenario, { timeoutMs, processes, rules: config.rules, ignore: config.ignore, scenarioName: values.scenario });
|
|
346
|
+
const result = await session(sessionTarget, scenario, { timeoutMs, processes, rules: config.rules, ignore: config.ignore, scenarioName: values.scenario, unionOut: !!values['union-out'] });
|
|
325
347
|
if (values['union-out'])
|
|
326
348
|
await writeFile(values['union-out'], JSON.stringify(result.union, null, 2) + '\n');
|
|
327
349
|
const output = formatSession(result, format);
|
|
@@ -405,6 +427,12 @@ function exitWhenFlushed(code) {
|
|
|
405
427
|
}
|
|
406
428
|
process.stdout.write('', () => process.stderr.write('', () => process.exit(code)));
|
|
407
429
|
}
|
|
430
|
+
// stdout carries the report alone (--format json is parsed by CI). Libraries log
|
|
431
|
+
// with console: the MCP SDK prints a console.debug line to stdout when a server
|
|
432
|
+
// has no tools capability. Their lines go to stderr.
|
|
433
|
+
for (const method of ['log', 'info', 'debug']) {
|
|
434
|
+
console[method] = (...args) => console.error(...args);
|
|
435
|
+
}
|
|
408
436
|
main(process.argv.slice(2)).then(exitWhenFlushed, (error) => {
|
|
409
437
|
const message = error instanceof Error ? error.message : String(error);
|
|
410
438
|
process.stderr.write(`toolmenu: ${message}\n`);
|