toolgz 0.2.3 → 0.2.5

package/README.md

@@ -101,6 +101,55 @@ Recompute any figure with
 `npx tsx bench/analyze-multi.ts --sweep=2026-07-25T19-19` against the raw per-run
 records in [`bench/results/`](bench/results).
 
+### How it scales, measured on real tools
+
+The table above is a synthetic 100-tool fixture. This is the real one: **149 tools
+harvested from 14 live MCP servers**, scaled by replicating the corpus, measured with
+Anthropic's `count_tokens` and confirmed against live API calls.
+
+| Tools | Uncompressed | Level 3 | Reclaimed |
+|---:|---:|---:|---:|
+| 149 (the real corpus) | 68,536 | **3,022** | **95.6%** |
+| 435 | 199,822 | **8,690** | **95.7%** |
+| 800 | 368,826 | **16,006** | **95.7%** |
+| 1,200 | 552,795 | **23,880** | **95.7%** |
+
+**The ratio does not decay with scale.** Both the uncompressed block and the map grow
+linearly, so 95.7% holds from 149 tools to 1,200. Real MCP tools measure ~460 tokens
+each — 17% heavier than the ~393 in the published academic benchmark, so a real
+catalogue hits limits sooner than synthetic ones suggest.
+
+Which gives a practical ceiling per context window:
+
+| Context window | Tools that fit uncompressed | With level 3 |
+|---|---:|---:|
+| 8K (small/local models) | **17** | ~409 |
+| 32K | 71 | ~1,638 |
+| 200K (typical frontier cap) | **434** | ~10,000 |
+
+That 200K row is the one to note: **most deployments cap at 200k, making ~434 real
+tools a hard ceiling.** An independent study ([Sakizli 2026](https://github.com/SKZL-AI/tscg))
+measured the same threshold at ~494 tools using tools 17% lighter than ours — two
+separate measurements agreeing within about 15%.
+
+### What we could not test
+
+**We have not demonstrated that compression improves accuracy, and we do not claim it.**
+
+The published study above finds a *binary enablement* effect: at 8K with 28 tools,
+uncompressed schemas overflow the window and exact-match accuracy collapses to 2.6%,
+while compression restores it (+20.5pp average). At 32K, where both fit, four of five
+models show ≤1pp difference — the effect is **budget-driven, not intrinsic**.
+
+We tried to reproduce it and could not, for an honest reason: every provider we test
+against has a window far larger than our corpus needs. Uncompressed requests ran
+successfully at 149, 435 and **800 tools (368,826 tokens)** on `claude-opus-5`, picking
+the correct tool each time. Reaching overflow on a 1M window would take ~2,173 tools.
+
+So the enablement regime — where this stops being an optimisation and becomes a
+prerequisite — lives on **small-context models we do not currently test**. If you run
+local models at 8K–32K, that study is more relevant to you than our benchmarks are.
+
 ### What about cost?
 
 **Cost is not the claim, and we deliberately do not lead with it.** Prompt caching
@@ -234,6 +283,19 @@ trip. A test asserts that file matches the code, so it cannot drift.
 
 ---
 
+### `signature`: the level-3 style with no lookups
+
+Worth calling out because it trades differently from the others. `signature` puts the full
+parameter list in the map, so the model never needs a `q()` lookup — **measured 0.0 lookups
+on all four providers**, against 0.1–0.5 for the default.
+
+That makes it the fastest and cheapest option on OpenAI (4.0s vs 5.6s; median cost $0.0106
+vs $0.0129) at the price of a larger cached map. But it is **not** universally better: on
+grok-4.5 it was slower (7.4s vs 4.6s), dearer, and produced the one malformed argument in
+that arm. Reach for it if lookups are your bottleneck and you are not on xAI.
+
+---
+
 ## Optional: let the library pick the map style for your model
 
 Level 3 has several map styles. Which one is cheapest turns out to depend on the
@@ -779,6 +841,12 @@ Returns:
 | `resolve(name, args)` | `→ Resolution` | translate a model call back |
 | `codeFor(name)` | `→ string` | real name → level-3 code; throws below level 3 |
 | `stats` | `CompressStats` | `level`, `mapStyle`, `requestedMapStyle`, `fallbackReason`, `toolCount`, `wireToolCount`, `originalChars`, `compressedChars`, `savedPct` |
+
+> **`savedPct` counts characters, not tokens — and it overstates.** It is derived from
+> `countSchemaTokensApprox`, which is just `JSON.stringify(x).length`. On our real corpus
+> it reports ~97.8% where `count_tokens` measures **95.6%**. Use it as a cheap sanity
+> signal, never as a published figure; for anything you quote, measure with your
+> provider's token counter.
 | `encodeCallForTest(name, args)` | `→ {name, args}` | build the raw call a model would emit; test aid |
 
 ### `recommendLevel(tools, namespaceOf?) → Recommendation`
@@ -815,6 +883,34 @@ Methodology and repo conventions: [../AGENTS.md](../AGENTS.md).
 
 ## Providers
 
+### Why `forGemini` returns one tool where the others return two
+
+It isn't sending less. Gemini's API nests *all* function declarations inside a single
+tool object — `[{ functionDeclarations: [...] }]` — where Anthropic and OpenAI take a
+flat array of tools. At level 3 the two dispatcher tools (`t` and `q`) are both present;
+they are just both inside that one wrapper. Count `tools[0].functionDeclarations.length`,
+not `tools.length`.
+
+### `forGemini` repairs three schema forms Gemini rejects
+
+Gemini rejects the **whole request** if any single declaration is invalid, so one
+non-conforming tool anywhere in your catalogue breaks every call. Three forms occur in
+real MCP servers, and the adapter repairs all three:
+
+| Form | Found in the real corpus | Repair |
+|---|---|---|
+| array with no `items` | 7 of 149 tools | adds `items: {}` |
+| `enum` on a non-string type | `{type:"number", enum:[1,2,3]}` | drops the `enum`, keeps the type |
+| union type `["string","array"]` | 1 tool | takes the first type |
+
+**The dropped `enum` is not a lost constraint.** `validateArgs` checks arguments against
+your *original* schema before dispatch at every level, so an out-of-range value is still
+caught — the check moves from provider-side to library-side. We deliberately do not coerce
+a numeric enum into a string enum: Gemini would accept it, and the model would then send
+`"1"` where your API wants `1`, turning a caught error into silent bad data.
+
+All 149 tools in the committed corpus pass Gemini at levels 0–3.
+
 ```ts
 import { forAnthropic, forOpenAI, forOpenAIResponses, forGemini } from "toolgz";
 ```
@@ -859,7 +955,7 @@ and it does not get deleted.
 ## Development
 
 ```bash
-npm test        # 252 tests, offline, no cost
+npm test        # 268 tests, offline, no cost
 npm run build   # tsc → dist/ with .d.ts
 
 npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants   # costs money

package/dist/providers/index.js

@@ -95,6 +95,50 @@ export function forGemini(c) {
                         ? clean(v)
                         : v;
         }
+        // Gemini rejects an array without `items`, and rejects the WHOLE request:
+        //   400 GenerateContentRequest.tools[0].function_declarations[4]
+        //       .parameters.properties[shipments].items: missing field
+        //
+        // Real MCP servers ship these — 7 of the 149 tools in bench/fixtures have an
+        // untyped array parameter (analyze_consolidation.shipments,
+        // compute_route.intermediates, optimize_waypoints.waypoints among them). One such
+        // tool anywhere in the catalogue used to fail every Gemini call.
+        //
+        // `items: {}` is what Gemini accepts and is the only honest repair: the source
+        // schema does not say what the items are, so we do not invent a type. Guessing
+        // `{type:"string"}` would also be accepted and would make the model send strings
+        // where the real API may want objects — a silent wrong-data bug in place of a loud
+        // rejection. Verified empirically: omitted is rejected; `{}`, `{type:"string"}` and
+        // `{type:"object"}` are all accepted.
+        // JSON Schema permits a union type (`"type": ["string","array"]`, as
+        // send_email_with_attachments.cc uses). Gemini requires a single type string and
+        // rejects the whole request otherwise. Take the first — deterministic, and the
+        // prompt-cache prefix must be byte-stable — and drop `items` if the pick is not an
+        // array, where it would be meaningless.
+        //
+        // Narrowing is safe because the ORIGINAL schema still accepts the narrowed form, so
+        // a value the model sends against it remains valid; validateArgs checks against
+        // that original before dispatch.
+        if (Array.isArray(out.type)) {
+            out.type = out.type[0];
+            if (out.type !== "array")
+                delete out.items;
+        }
+        if (out.type === "array" && out.items === undefined)
+            out.items = {};
+        // Gemini accepts `enum` only on strings, and rejects the whole request otherwise:
+        //   400 Invalid value at tools[0].function_declarations[30].parameters.properties[2]
+        // Real MCP servers ship numeric enums — deep_research.depth is
+        // {type:"number", enum:[1,2,3]}. Verified: string+enum accepted, number+enum and
+        // integer+enum rejected, `oneOf` accepted (so it needs no repair).
+        //
+        // We drop the enum and keep the type. The constraint is not lost: validateArgs
+        // checks arguments against the ORIGINAL schema before dispatch at every level, so
+        // an out-of-range value is still caught — the check moves from provider-side to
+        // library-side. Coercing to a string enum instead would make the model send "1"
+        // where the API wants 1, which is a silent type error rather than a caught one.
+        if (out.enum && out.type && out.type !== "string")
+            delete out.enum;
         return out;
     };
     const functionDeclarations = c.tools

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toolgz",
-  "version": "0.2.3",
+  "version": "0.2.5",
   "description": "Compress LLM tool definitions to reclaim context window. Measured across Anthropic, OpenAI, Google and xAI.",
   "main": "dist/index.js",
   "scripts": {