toolgz 0.2.6 → 0.2.7

package/README.md

@@ -767,8 +767,8 @@ console.log(c.stats);
 //   originalChars: 61461, compressedChars: 2211, savedPct: 96.4 }
 ```
 
-`savedPct` is a character-count proxy, not a token count — useful for a quick
-sanity check, not for billing.
+`savedPct` estimates the token saving (calibrated against `count_tokens`, ~1–2pp
+accurate). `originalChars` and `compressedChars` give the raw character counts.
 
 **I need the real token numbers.**
 Count them with the provider's own endpoint. Never use `tiktoken` for Claude —
@@ -842,11 +842,16 @@ Returns:
 | `codeFor(name)` | `→ string` | real name → level-3 code; throws below level 3 |
 | `stats` | `CompressStats` | `level`, `mapStyle`, `requestedMapStyle`, `fallbackReason`, `toolCount`, `wireToolCount`, `originalChars`, `compressedChars`, `savedPct` |
 
-> **`savedPct` counts characters, not tokens — and it overstates.** It is derived from
-> `countSchemaTokensApprox`, which is just `JSON.stringify(x).length`. On our real corpus
-> it reports ~97.8% where `count_tokens` measures **95.6%**. Use it as a cheap sanity
-> signal, never as a published figure; for anything you quote, measure with your
-> provider's token counter.
+> **`savedPct` estimates tokens, and is accurate to a point or two.** It used to be a raw
+> character ratio, which overstated by 7.6 points at level 1. Characters per token differs
+> between the two sides — uncompressed JSON is punctuation-dense, a signature line reads
+> more like prose — so it now divides each side by a ratio calibrated against
+> `count_tokens` on two unrelated corpora. Current error on the real corpus: **+1.5pp at
+> level 1, −0.3pp at level 2, +0.1pp at level 3.**
+>
+> It is still a local estimate with no API call. For a figure you publish, measure with
+> your provider's token counter; `originalChars` and `compressedChars` are also on `stats`
+> if you want the raw counts.
 | `encodeCallForTest(name, args)` | `→ {name, args}` | build the raw call a model would emit; test aid |
 
 ### `recommendLevel(tools, namespaceOf?) → Recommendation`
@@ -955,7 +960,7 @@ and it does not get deleted.
 ## Development
 
 ```bash
-npm test        # 272 tests, offline, no cost
+npm test        # 277 tests, offline, no cost
 npm run build   # tsc → dist/ with .d.ts
 
 npx tsx bench/harness/run-multi.ts --provider=all --reps=3 --variants   # costs money

package/dist/index.js

@@ -160,6 +160,27 @@ export function compress(input, options = {}) {
     };
     const finish = (wire, systemPreamble, cachePreamble, resolve, encode) => {
         const compressedChars = countSchemaTokensApprox(wire) + systemPreamble.length;
+        // `savedPct` is meant to answer "how much of my tool block did I get back", which is
+        // a TOKEN question. Deriving it from a raw character ratio answered a different one
+        // and overstated: on the real corpus it reported 46.8% at level 1 where count_tokens
+        // measures 39.2% — 7.6 points high, and the field was documented as "do not publish
+        // this" rather than fixed.
+        //
+        // Characters per token is not constant across the two sides. Uncompressed JSON is
+        // punctuation-dense; a signature line or a map row reads more like prose. Measured
+        // against count_tokens on two unrelated corpora (149 real MCP tools and a 100-tool
+        // synthetic fixture) the ratios are stable per level:
+        //
+        //   uncompressed  2.39 / 2.45      level 2  2.20 / 1.99
+        //   level 1       2.15 / 2.19      level 3  1.92 / 1.91
+        //
+        // Dividing by these brings the error to ~1.5 points at level 1 and ~0.1 at level 3.
+        // Still an estimate — it is a local calculation with no API call — but now an
+        // estimate of the right quantity.
+        const CHARS_PER_TOKEN = { 0: 2.42, 1: 2.17, 2: 2.1, 3: 1.92 };
+        const estTokens = (chars, at) => chars / CHARS_PER_TOKEN[at];
+        const originalTokens = estTokens(originalChars, 0);
+        const compressedTokens = estTokens(compressedChars, level);
         return {
             tools: wire,
             systemPreamble,
@@ -183,7 +204,7 @@ export function compress(input, options = {}) {
                 compressedChars,
                 savedPct: originalChars === 0
                     ? 0
-                    : Math.round((1 - compressedChars / originalChars) * 1000) / 10,
+                    : Math.round((1 - compressedTokens / originalTokens) * 1000) / 10,
             },
         };
     };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "toolgz",
-  "version": "0.2.6",
+  "version": "0.2.7",
   "description": "Compress LLM tool definitions to reclaim context window. Measured across Anthropic, OpenAI, Google and xAI.",
   "main": "dist/index.js",
   "scripts": {