toolgz 0.2.4 → 0.2.6

package/README.md

@@ -2,7 +2,7 @@
 
 <p><strong>Your agent spends 30–70k tokens of context on tool definitions before the user types a word. toolgz gets up to ~85% of it back.</strong></p>
 
-<p><em>On a large tool set, at level 3. The safe default (level 1) reclaims 13–32% and gives up nothing — <code>recommendLevel()</code> picks for you.</em></p>
+<p><em>On a large tool set, at level 3. The safe default (level 1) reclaims 13–39% and gives up nothing — <code>recommendLevel()</code> picks for you.</em></p>
 
 <p>
 <a href="#measured-results">420-run cross-provider sweep</a> ·
@@ -39,7 +39,7 @@ const { tools, system } = forAnthropic(c);    // send these instead
 ```
 
 `compress(myTools)` with no `level` gives you **level 1** — safe, native tool calling,
-provider schema enforcement intact, and 13–32% smaller. The 71–85% figures below are
+provider schema enforcement intact, and 13–39% smaller (13–32% on the synthetic benchmark, 39% on the real 149-tool corpus). The 71–85% figures below are
 **level 3**, which is what `recommendLevel` returns once a tool set is big enough to
 amortise the dispatcher. Ask for it explicitly with `{ level: 3 }` if you prefer.
 
@@ -84,7 +84,7 @@ claim about real deployments.
 </picture>
 
 **All figures below are level 3** (`minified-plus`, the shipped default map style).
-Level 1 on the same sweep saves 13–32%; level 2 is dominated by level 3.
+Level 1 on the same sweep saves 13–32%; on the real 149-tool corpus it saves 39%; level 2 is dominated by level 3.
 
 | Provider | Model | Tool block | Prompt tokens | Latency | Tasks |
 |---|---|---:|---:|---:|:-:|
@@ -283,6 +283,19 @@ trip. A test asserts that file matches the code, so it cannot drift.
 
 ---
 
+### `signature`: the level-3 style with no lookups
+
+Worth calling out because it trades differently from the others. `signature` puts the full
+parameter list in the map, so the model never needs a `q()` lookup — **measured 0.0 lookups
+on all four providers**, against 0.1–0.5 for the default.
+
+That makes it the fastest and cheapest option on OpenAI (4.0s vs 5.6s; median cost $0.0106
+vs $0.0129) at the price of a larger cached map. But it is **not** universally better: on
+grok-4.5 it was slower (7.4s vs 4.6s), dearer, and produced the one malformed argument in
+that arm. Reach for it if lookups are your bottleneck and you are not on xAI.
+
+---
+
 ## Optional: let the library pick the map style for your model
 
 Level 3 has several map styles. Which one is cheapest turns out to depend on the
@@ -828,6 +841,12 @@ Returns:
 | `resolve(name, args)` | `→ Resolution` | translate a model call back |
 | `codeFor(name)` | `→ string` | real name → level-3 code; throws below level 3 |
 | `stats` | `CompressStats` | `level`, `mapStyle`, `requestedMapStyle`, `fallbackReason`, `toolCount`, `wireToolCount`, `originalChars`, `compressedChars`, `savedPct` |
+
+> **`savedPct` counts characters, not tokens — and it overstates.** It is derived from
+> `countSchemaTokensApprox`, which is just `JSON.stringify(x).length`. On our real corpus
+> it reports ~97.8% where `count_tokens` measures **95.6%**. Use it as a cheap sanity
+> signal, never as a published figure; for anything you quote, measure with your
+> provider's token counter.
 | `encodeCallForTest(name, args)` | `→ {name, args}` | build the raw call a model would emit; test aid |
 
 ### `recommendLevel(tools, namespaceOf?) → Recommendation`
@@ -864,6 +883,34 @@ Methodology and repo conventions: [../AGENTS.md](../AGENTS.md).
 
 ## Providers
 
+### Why `forGemini` returns one tool where the others return two
+
+It isn't sending less. Gemini's API nests *all* function declarations inside a single
+tool object — `[{ functionDeclarations: [...] }]` — where Anthropic and OpenAI take a
+flat array of tools. At level 3 the two dispatcher tools (`t` and `q`) are both present;
+they are just both inside that one wrapper. Count `tools[0].functionDeclarations.length`,
+not `tools.length`.
+
+### `forGemini` repairs three schema forms Gemini rejects
+
+Gemini rejects the **whole request** if any single declaration is invalid, so one
+non-conforming tool anywhere in your catalogue breaks every call. Three forms occur in
+real MCP servers, and the adapter repairs all three:
+
+| Form | Found in the real corpus | Repair |
+|---|---|---|
+| array with no `items` | 7 of 149 tools | adds `items: {}` |
+| `enum` on a non-string type | `{type:"number", enum:[1,2,3]}` | drops the `enum`, keeps the type |
+| union type `["string","array"]` | 1 tool | takes the first type |
+
+**The dropped `enum` is not a lost constraint.** `validateArgs` checks arguments against
+your *original* schema before dispatch at every level, so an out-of-range value is still
+caught — the check moves from provider-side to library-side. We deliberately do not coerce
+a numeric enum into a string enum: Gemini would accept it, and the model would then send
+`"1"` where your API wants `1`, turning a caught error into silent bad data.
+
+All 149 tools in the committed corpus pass Gemini at levels 0–3.
+
 ```ts
 import { forAnthropic, forOpenAI, forOpenAIResponses, forGemini } from "toolgz";
 ```
@@ -908,7 +955,7 @@ and it does not get deleted.
 ## Development
 
 ```bash
-npm test        # 252 tests, offline, no cost
+npm test        # 272 tests, offline, no cost
 npm run build   # tsc → dist/ with .d.ts
 
 npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants   # costs money

package/dist/providers/index.js

@@ -95,6 +95,50 @@ export function forGemini(c) {
                         ? clean(v)
                         : v;
         }
+        // Gemini rejects an array without `items`, and rejects the WHOLE request:
+        //   400 GenerateContentRequest.tools[0].function_declarations[4]
+        //       .parameters.properties[shipments].items: missing field
+        //
+        // Real MCP servers ship these — 7 of the 149 tools in bench/fixtures have an
+        // untyped array parameter (analyze_consolidation.shipments,
+        // compute_route.intermediates, optimize_waypoints.waypoints among them). One such
+        // tool anywhere in the catalogue used to fail every Gemini call.
+        //
+        // `items: {}` is what Gemini accepts and is the only honest repair: the source
+        // schema does not say what the items are, so we do not invent a type. Guessing
+        // `{type:"string"}` would also be accepted and would make the model send strings
+        // where the real API may want objects — a silent wrong-data bug in place of a loud
+        // rejection. Verified empirically: omitted is rejected; `{}`, `{type:"string"}` and
+        // `{type:"object"}` are all accepted.
+        // JSON Schema permits a union type (`"type": ["string","array"]`, as
+        // send_email_with_attachments.cc uses). Gemini requires a single type string and
+        // rejects the whole request otherwise. Take the first — deterministic, and the
+        // prompt-cache prefix must be byte-stable — and drop `items` if the pick is not an
+        // array, where it would be meaningless.
+        //
+        // Narrowing is safe because the ORIGINAL schema still accepts the narrowed form, so
+        // a value the model sends against it remains valid; validateArgs checks against
+        // that original before dispatch.
+        if (Array.isArray(out.type)) {
+            out.type = out.type[0];
+            if (out.type !== "array")
+                delete out.items;
+        }
+        if (out.type === "array" && out.items === undefined)
+            out.items = {};
+        // Gemini accepts `enum` only on strings, and rejects the whole request otherwise:
+        //   400 Invalid value at tools[0].function_declarations[30].parameters.properties[2]
+        // Real MCP servers ship numeric enums — deep_research.depth is
+        // {type:"number", enum:[1,2,3]}. Verified: string+enum accepted, number+enum and
+        // integer+enum rejected, `oneOf` accepted (so it needs no repair).
+        //
+        // We drop the enum and keep the type. The constraint is not lost: validateArgs
+        // checks arguments against the ORIGINAL schema before dispatch at every level, so
+        // an out-of-range value is still caught — the check moves from provider-side to
+        // library-side. Coercing to a string enum instead would make the model send "1"
+        // where the API wants 1, which is a silent type error rather than a caught one.
+        if (out.enum && out.type && out.type !== "string")
+            delete out.enum;
         return out;
     };
     const functionDeclarations = c.tools

package/dist/recommend.js

@@ -1,4 +1,5 @@
 import { defaultNamespaceOf } from "./render/index.js";
+import { compress } from "./index.js";
 /**
  * Pick a level for a given tool set.
  *
@@ -34,23 +35,39 @@ export function recommendLevel(tools, namespaceOf = defaultNamespaceOf) {
     const namespaceCount = namespaces.size;
     const opsPerNamespace = namespaceCount ? toolCount / namespaceCount : 0;
     const base = { toolCount, namespaceCount, opsPerNamespace };
-    if (toolCount < 15) {
+    // Size the level-1 block by asking the library what it would actually emit, rather
+    // than reconstructing it — an earlier version rebuilt the schema by hand, kept the
+    // full descriptions where level 1 emits a signature line, and overestimated by ~34%.
+    //
+    // `countSchemaTokensApprox` counts characters. Measured against `count_tokens` on
+    // real MCP tools the ratio is a stable ~2.1 chars/token (1.98 at 10 tools, 2.15 at
+    // 149), so dividing gives a fair local estimate with no API call.
+    const CHARS_PER_TOKEN = 2.1;
+    const l1Tokens = Math.round(compress(tools, { level: 1 }).stats.compressedChars / CHARS_PER_TOKEN);
+    // The old heuristic gated level 3 on `opsPerNamespace >= 4`. That is a LEVEL-2
+    // question: level 2 pays dispatcher overhead per namespace, so its shape matters
+    // there. Level 3 uses one flat dispatcher and does not care about namespaces at all.
+    //
+    // The consequence was concrete: on the 149-tool real corpus (63 namespaces, 2.4 ops
+    // each) it recommended level 1 at 41,648 tokens when level 3 measures 2,980 — leaving
+    // ~38,700 tokens on the table. Measured on real tools, level 3 is smaller at EVERY
+    // count tested, down to 5 tools (635 vs 1,178).
+    //
+    // So size never argues for level 1. What argues for level 1 is that it keeps the
+    // provider's own schema enforcement, which levels 2-3 give up. That is worth more
+    // than a saving you would not notice — hence an absolute threshold on the block,
+    // not a shape test.
+    const THRESHOLD_TOKENS = 4000;
+    if (l1Tokens < THRESHOLD_TOKENS) {
         return {
             ...base,
             level: 1,
-            reason: `Only ${toolCount} tools — flattening schemas captures nearly all the available saving, and namespace collapse would add dispatcher overhead for little return.`,
-        };
-    }
-    if (opsPerNamespace < 4) {
-        return {
-            ...base,
-            level: 1,
-            reason: `${toolCount} tools spread across ${namespaceCount} namespaces (${opsPerNamespace.toFixed(1)} ops each). The compound-dispatcher overhead is paid per namespace, so a set this sparse stays smaller at level 1.`,
+            reason: `The tool block is only ~${l1Tokens.toLocaleString()} tokens at level 1. Level 3 would be smaller, but not by enough to be worth giving up the provider's own argument validation, which level 1 keeps. Reach for level 3 when the block is large enough that reclaiming it changes what fits in your context.`,
         };
     }
     return {
         ...base,
         level: 3,
-        reason: `${toolCount} tools across ${namespaceCount} namespaces (${opsPerNamespace.toFixed(1)} ops each) — deep enough that a single dispatcher plus a cached code map beats per-tool definitions. Measured at ~82% fewer prompt tokens with no accuracy penalty across Opus 5, Sonnet 5 and Haiku 4.5, at the cost of roughly 0.6 extra turns and 1.7 lookup calls per task. Keep argument validation on — on weaker models malformed arguments rise and validation is what catches them. If your workload is latency-critical rather than context-critical, drop to level 1.`,
+        reason: `${toolCount} tools take ~${l1Tokens.toLocaleString()} tokens at level 1; level 3 replaces them with two dispatcher tools plus a cached map. Measured on 149 real MCP tools: 41,648 tokens at level 1 against 2,980 at level 3, and 60/60 tasks completed across four frontier providers with zero hallucinated names. The cost is roughly 0.3-1.7 lookup calls per task — so about half an extra turn — plus the loss of provider-side argument checking — keep \`validate\` on, which is what catches malformed arguments instead. If latency matters more than context, stay at level 1.`,
     };
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toolgz",
-  "version": "0.2.4",
+  "version": "0.2.6",
   "description": "Compress LLM tool definitions to reclaim context window. Measured across Anthropic, OpenAI, Google and xAI.",
   "main": "dist/index.js",
   "scripts": {