toolgz 0.2.5 → 0.2.6

package/README.md

@@ -2,7 +2,7 @@
 
 <p><strong>Your agent spends 30–70k tokens of context on tool definitions before the user types a word. toolgz gets up to ~85% of it back.</strong></p>
 
-<p><em>On a large tool set, at level 3. The safe default (level 1) reclaims 13–32% and gives up nothing — <code>recommendLevel()</code> picks for you.</em></p>
+<p><em>On a large tool set, at level 3. The safe default (level 1) reclaims 13–39% and gives up nothing — <code>recommendLevel()</code> picks for you.</em></p>
 
 <p>
 <a href="#measured-results">420-run cross-provider sweep</a> ·
@@ -39,7 +39,7 @@ const { tools, system } = forAnthropic(c);    // send these instead
 ```
 
 `compress(myTools)` with no `level` gives you **level 1** — safe, native tool calling,
-provider schema enforcement intact, and 13–32% smaller. The 71–85% figures below are
+provider schema enforcement intact, and 13–39% smaller (13–32% on the synthetic benchmark, 39% on the real 149-tool corpus). The 71–85% figures below are
 **level 3**, which is what `recommendLevel` returns once a tool set is big enough to
 amortise the dispatcher. Ask for it explicitly with `{ level: 3 }` if you prefer.
 
@@ -84,7 +84,7 @@ claim about real deployments.
 </picture>
 
 **All figures below are level 3** (`minified-plus`, the shipped default map style).
-Level 1 on the same sweep saves 13–32%; level 2 is dominated by level 3.
+Level 1 on the same sweep saves 13–32%; on the real 149-tool corpus it saves 39%; level 2 is dominated by level 3.
 
 | Provider | Model | Tool block | Prompt tokens | Latency | Tasks |
 |---|---|---:|---:|---:|:-:|
@@ -955,7 +955,7 @@ and it does not get deleted.
 ## Development
 
 ```bash
-npm test        # 268 tests, offline, no cost
+npm test        # 272 tests, offline, no cost
 npm run build   # tsc → dist/ with .d.ts
 
 npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants   # costs money

package/dist/recommend.js

@@ -1,4 +1,5 @@
 import { defaultNamespaceOf } from "./render/index.js";
+import { compress } from "./index.js";
 /**
  * Pick a level for a given tool set.
  *
@@ -34,23 +35,39 @@ export function recommendLevel(tools, namespaceOf = defaultNamespaceOf) {
     const namespaceCount = namespaces.size;
     const opsPerNamespace = namespaceCount ? toolCount / namespaceCount : 0;
     const base = { toolCount, namespaceCount, opsPerNamespace };
-    if (toolCount < 15) {
+    // Size the level-1 block by asking the library what it would actually emit, rather
+    // than reconstructing it — an earlier version rebuilt the schema by hand, kept the
+    // full descriptions where level 1 emits a signature line, and overestimated by ~34%.
+    //
+    // `countSchemaTokensApprox` counts characters. Measured against `count_tokens` on
+    // real MCP tools the ratio is a stable ~2.1 chars/token (1.98 at 10 tools, 2.15 at
+    // 149), so dividing gives a fair local estimate with no API call.
+    const CHARS_PER_TOKEN = 2.1;
+    const l1Tokens = Math.round(compress(tools, { level: 1 }).stats.compressedChars / CHARS_PER_TOKEN);
+    // The old heuristic gated level 3 on `opsPerNamespace >= 4`. That is a LEVEL-2
+    // question: level 2 pays dispatcher overhead per namespace, so its shape matters
+    // there. Level 3 uses one flat dispatcher and does not care about namespaces at all.
+    //
+    // The consequence was concrete: on the 149-tool real corpus (63 namespaces, 2.4 ops
+    // each) it recommended level 1 at 41,648 tokens when level 3 measures 2,980 — leaving
+    // ~38,700 tokens on the table. Measured on real tools, level 3 is smaller at EVERY
+    // count tested, down to 5 tools (635 vs 1,178).
+    //
+    // So size never argues for level 1. What argues for level 1 is that it keeps the
+    // provider's own schema enforcement, which levels 2-3 give up. That is worth more
+    // than a saving you would not notice — hence an absolute threshold on the block,
+    // not a shape test.
+    const THRESHOLD_TOKENS = 4000;
+    if (l1Tokens < THRESHOLD_TOKENS) {
         return {
             ...base,
             level: 1,
-            reason: `Only ${toolCount} tools — flattening schemas captures nearly all the available saving, and namespace collapse would add dispatcher overhead for little return.`,
-        };
-    }
-    if (opsPerNamespace < 4) {
-        return {
-            ...base,
-            level: 1,
-            reason: `${toolCount} tools spread across ${namespaceCount} namespaces (${opsPerNamespace.toFixed(1)} ops each). The compound-dispatcher overhead is paid per namespace, so a set this sparse stays smaller at level 1.`,
+            reason: `The tool block is only ~${l1Tokens.toLocaleString()} tokens at level 1. Level 3 would be smaller, but not by enough to be worth giving up the provider's own argument validation, which level 1 keeps. Reach for level 3 when the block is large enough that reclaiming it changes what fits in your context.`,
         };
     }
     return {
         ...base,
         level: 3,
-        reason: `${toolCount} tools across ${namespaceCount} namespaces (${opsPerNamespace.toFixed(1)} ops each) — deep enough that a single dispatcher plus a cached code map beats per-tool definitions. Measured at ~82% fewer prompt tokens with no accuracy penalty across Opus 5, Sonnet 5 and Haiku 4.5, at the cost of roughly 0.6 extra turns and 1.7 lookup calls per task. Keep argument validation on — on weaker models malformed arguments rise and validation is what catches them. If your workload is latency-critical rather than context-critical, drop to level 1.`,
+        reason: `${toolCount} tools take ~${l1Tokens.toLocaleString()} tokens at level 1; level 3 replaces them with two dispatcher tools plus a cached map. Measured on 149 real MCP tools: 41,648 tokens at level 1 against 2,980 at level 3, and 60/60 tasks completed across four frontier providers with zero hallucinated names. The cost is roughly 0.3-1.7 lookup calls per task — so about half an extra turn — plus the loss of provider-side argument checking — keep \`validate\` on, which is what catches malformed arguments instead. If latency matters more than context, stay at level 1.`,
     };
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toolgz",
-  "version": "0.2.5",
+  "version": "0.2.6",
   "description": "Compress LLM tool definitions to reclaim context window. Measured across Anthropic, OpenAI, Google and xAI.",
   "main": "dist/index.js",
   "scripts": {