toolgz 0.2.5 → 0.2.7

package/README.md

@@ -2,7 +2,7 @@
 
 <p><strong>Your agent spends 30–70k tokens of context on tool definitions before the user types a word. toolgz gets up to ~85% of it back.</strong></p>
 
-<p><em>On a large tool set, at level 3. The safe default (level 1) reclaims 13–32% and gives up nothing — <code>recommendLevel()</code> picks for you.</em></p>
+<p><em>On a large tool set, at level 3. The safe default (level 1) reclaims 13–39% and gives up nothing — <code>recommendLevel()</code> picks for you.</em></p>
 
 <p>
 <a href="#measured-results">420-run cross-provider sweep</a> ·
@@ -39,7 +39,7 @@ const { tools, system } = forAnthropic(c);    // send these instead
 ```
 
 `compress(myTools)` with no `level` gives you **level 1** — safe, native tool calling,
-provider schema enforcement intact, and 13–32% smaller. The 71–85% figures below are
+provider schema enforcement intact, and 13–39% smaller (13–32% on the synthetic benchmark, 39% on the real 149-tool corpus). The 71–85% figures below are
 **level 3**, which is what `recommendLevel` returns once a tool set is big enough to
 amortise the dispatcher. Ask for it explicitly with `{ level: 3 }` if you prefer.
 
@@ -84,7 +84,7 @@ claim about real deployments.
 </picture>
 
 **All figures below are level 3** (`minified-plus`, the shipped default map style).
-Level 1 on the same sweep saves 13–32%; level 2 is dominated by level 3.
+Level 1 on the same sweep saves 13–32%; on the real 149-tool corpus it saves 39%; level 2 is dominated by level 3.
 
 | Provider | Model | Tool block | Prompt tokens | Latency | Tasks |
 |---|---|---:|---:|---:|:-:|
@@ -767,8 +767,8 @@ console.log(c.stats);
 //   originalChars: 61461, compressedChars: 2211, savedPct: 96.4 }
 ```
 
-`savedPct` is a character-count proxy, not a token count — useful for a quick
-sanity check, not for billing.
+`savedPct` estimates the token saving (calibrated against `count_tokens`, ~1–2pp
+accurate). `originalChars` and `compressedChars` give the raw character counts.
 
 **I need the real token numbers.**
 Count them with the provider's own endpoint. Never use `tiktoken` for Claude —
@@ -842,11 +842,16 @@ Returns:
 | `codeFor(name)` | `→ string` | real name → level-3 code; throws below level 3 |
 | `stats` | `CompressStats` | `level`, `mapStyle`, `requestedMapStyle`, `fallbackReason`, `toolCount`, `wireToolCount`, `originalChars`, `compressedChars`, `savedPct` |
 
-> **`savedPct` counts characters, not tokens — and it overstates.** It is derived from
-> `countSchemaTokensApprox`, which is just `JSON.stringify(x).length`. On our real corpus
-> it reports ~97.8% where `count_tokens` measures **95.6%**. Use it as a cheap sanity
-> signal, never as a published figure; for anything you quote, measure with your
-> provider's token counter.
+> **`savedPct` estimates tokens, and is accurate to a point or two.** It used to be a raw
+> character ratio, which overstated by 7.6 points at level 1. Characters per token differs
+> between the two sides — uncompressed JSON is punctuation-dense, a signature line reads
+> more like prose — so it now divides each side by a ratio calibrated against
+> `count_tokens` on two unrelated corpora. Current error on the real corpus: **+1.5pp at
+> level 1, −0.3pp at level 2, +0.1pp at level 3.**
+>
+> It is still a local estimate with no API call. For a figure you publish, measure with
+> your provider's token counter; `originalChars` and `compressedChars` are also on `stats`
+> if you want the raw counts.
 | `encodeCallForTest(name, args)` | `→ {name, args}` | build the raw call a model would emit; test aid |
 
 ### `recommendLevel(tools, namespaceOf?) → Recommendation`
@@ -955,7 +960,7 @@ and it does not get deleted.
 ## Development
 
 ```bash
-npm test        # 268 tests, offline, no cost
+npm test        # 277 tests, offline, no cost
 npm run build   # tsc → dist/ with .d.ts
 
 npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants   # costs money

package/dist/index.js

@@ -160,6 +160,27 @@ export function compress(input, options = {}) {
     };
     const finish = (wire, systemPreamble, cachePreamble, resolve, encode) => {
         const compressedChars = countSchemaTokensApprox(wire) + systemPreamble.length;
+        // `savedPct` is meant to answer "how much of my tool block did I get back", which is
+        // a TOKEN question. Deriving it from a raw character ratio answered a different one
+        // and overstated: on the real corpus it reported 46.8% at level 1 where count_tokens
+        // measures 39.2% — 7.6 points high, and the field was documented as "do not publish
+        // this" rather than fixed.
+        //
+        // Characters per token is not constant across the two sides. Uncompressed JSON is
+        // punctuation-dense; a signature line or a map row reads more like prose. Measured
+        // against count_tokens on two unrelated corpora (149 real MCP tools and a 100-tool
+        // synthetic fixture) the ratios are stable per level:
+        //
+        //   uncompressed  2.39 / 2.45      level 2  2.20 / 1.99
+        //   level 1       2.15 / 2.19      level 3  1.92 / 1.91
+        //
+        // Dividing by these brings the error to ~1.5 points at level 1 and ~0.1 at level 3.
+        // Still an estimate — it is a local calculation with no API call — but now an
+        // estimate of the right quantity.
+        const CHARS_PER_TOKEN = { 0: 2.42, 1: 2.17, 2: 2.1, 3: 1.92 };
+        const estTokens = (chars, at) => chars / CHARS_PER_TOKEN[at];
+        const originalTokens = estTokens(originalChars, 0);
+        const compressedTokens = estTokens(compressedChars, level);
         return {
             tools: wire,
             systemPreamble,
@@ -183,7 +204,7 @@ export function compress(input, options = {}) {
                 compressedChars,
                 savedPct: originalChars === 0
                     ? 0
-                    : Math.round((1 - compressedChars / originalChars) * 1000) / 10,
+                    : Math.round((1 - compressedTokens / originalTokens) * 1000) / 10,
             },
         };
     };

package/dist/recommend.js

@@ -1,4 +1,5 @@
 import { defaultNamespaceOf } from "./render/index.js";
+import { compress } from "./index.js";
 /**
  * Pick a level for a given tool set.
  *
@@ -34,23 +35,39 @@ export function recommendLevel(tools, namespaceOf = defaultNamespaceOf) {
     const namespaceCount = namespaces.size;
     const opsPerNamespace = namespaceCount ? toolCount / namespaceCount : 0;
     const base = { toolCount, namespaceCount, opsPerNamespace };
-    if (toolCount < 15) {
+    // Size the level-1 block by asking the library what it would actually emit, rather
+    // than reconstructing it — an earlier version rebuilt the schema by hand, kept the
+    // full descriptions where level 1 emits a signature line, and overestimated by ~34%.
+    //
+    // `countSchemaTokensApprox` counts characters. Measured against `count_tokens` on
+    // real MCP tools the ratio is a stable ~2.1 chars/token (1.98 at 10 tools, 2.15 at
+    // 149), so dividing gives a fair local estimate with no API call.
+    const CHARS_PER_TOKEN = 2.1;
+    const l1Tokens = Math.round(compress(tools, { level: 1 }).stats.compressedChars / CHARS_PER_TOKEN);
+    // The old heuristic gated level 3 on `opsPerNamespace >= 4`. That is a LEVEL-2
+    // question: level 2 pays dispatcher overhead per namespace, so its shape matters
+    // there. Level 3 uses one flat dispatcher and does not care about namespaces at all.
+    //
+    // The consequence was concrete: on the 149-tool real corpus (63 namespaces, 2.4 ops
+    // each) it recommended level 1 at 41,648 tokens when level 3 measures 2,980 — leaving
+    // ~38,700 tokens on the table. Measured on real tools, level 3 is smaller at EVERY
+    // count tested, down to 5 tools (635 vs 1,178).
+    //
+    // So size never argues for level 1. What argues for level 1 is that it keeps the
+    // provider's own schema enforcement, which levels 2-3 give up. That is worth more
+    // than a saving you would not notice — hence an absolute threshold on the block,
+    // not a shape test.
+    const THRESHOLD_TOKENS = 4000;
+    if (l1Tokens < THRESHOLD_TOKENS) {
         return {
             ...base,
             level: 1,
-            reason: `Only ${toolCount} tools — flattening schemas captures nearly all the available saving, and namespace collapse would add dispatcher overhead for little return.`,
-        };
-    }
-    if (opsPerNamespace < 4) {
-        return {
-            ...base,
-            level: 1,
-            reason: `${toolCount} tools spread across ${namespaceCount} namespaces (${opsPerNamespace.toFixed(1)} ops each). The compound-dispatcher overhead is paid per namespace, so a set this sparse stays smaller at level 1.`,
+            reason: `The tool block is only ~${l1Tokens.toLocaleString()} tokens at level 1. Level 3 would be smaller, but not by enough to be worth giving up the provider's own argument validation, which level 1 keeps. Reach for level 3 when the block is large enough that reclaiming it changes what fits in your context.`,
         };
     }
     return {
         ...base,
         level: 3,
-        reason: `${toolCount} tools across ${namespaceCount} namespaces (${opsPerNamespace.toFixed(1)} ops each) — deep enough that a single dispatcher plus a cached code map beats per-tool definitions. Measured at ~82% fewer prompt tokens with no accuracy penalty across Opus 5, Sonnet 5 and Haiku 4.5, at the cost of roughly 0.6 extra turns and 1.7 lookup calls per task. Keep argument validation on — on weaker models malformed arguments rise and validation is what catches them. If your workload is latency-critical rather than context-critical, drop to level 1.`,
+        reason: `${toolCount} tools take ~${l1Tokens.toLocaleString()} tokens at level 1; level 3 replaces them with two dispatcher tools plus a cached map. Measured on 149 real MCP tools: 41,648 tokens at level 1 against 2,980 at level 3, and 60/60 tasks completed across four frontier providers with zero hallucinated names. The cost is roughly 0.3-1.7 lookup calls per task — so about half an extra turn — plus the loss of provider-side argument checking — keep \`validate\` on, which is what catches malformed arguments instead. If latency matters more than context, stay at level 1.`,
     };
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toolgz",
-  "version": "0.2.5",
+  "version": "0.2.7",
   "description": "Compress LLM tool definitions to reclaim context window. Measured across Anthropic, OpenAI, Google and xAI.",
   "main": "dist/index.js",
   "scripts": {