tokentrimmer 0.1.1 → 0.2.0

package/README.md

@@ -37,8 +37,9 @@ for over and over. TokenTrimmer rewrites tool results into a shorter form as eac
 - **You can see what it does.** A local dashboard shows tokens before and after for every request, and dry-run
   mode measures savings without changing anything.
 
-What to expect: on real Claude Code sessions, safe mode saved about 3% of input tokens with no loss in answer
-accuracy, and the lossy modes saved 10–27% at a measurable accuracy cost (see [Real sessions](#real-sessions)).
+What to expect: on real Claude Code sessions, safe mode saved about 7% of input tokens (5% of the bill with prompt
+caching) with no loss in answer accuracy, and the lossy modes saved 14–30% at a measurable accuracy cost (see
+[Real sessions](#real-sessions)).
 Savings depend heavily on how much repetitive tool output your sessions have; run dry-run mode first to see
 your own number.
 
@@ -147,8 +148,8 @@ is experimental.
 | Log lines sharing a template: templates listed once, each line written as `~N|var|var`, original order kept | safe | yes |
 | Tables nested inside a larger JSON object keep a JSON form (`{"_table":{cols,rows}}`) | safe | yes |
 | grep-style `path/file:line:text` output: each directory written once as an `@ dir/` header | safe | yes |
-| Identical tool results sent again later become a one-line reference | safe | yes |
-| A run of 4+ lines already shown verbatim in an earlier tool result becomes `[tokentrimmer: lines A-B of the tool result in message M repeated here]` | safe | yes |
+| Identical tool results sent again later become `[same as a tool result in message M]` | safe | yes |
+| A line or run of lines the model already has verbatim becomes a short reference: `[same as message M lines A-B]` (an earlier tool result, as sent) or `[same as lines A-B above]` (earlier in the same result). Only used when the run is at least 60 characters and twice as long as the reference | safe | yes |
 | Drop null / empty JSON fields and all-empty table columns | balanced | no |
 | Collapse runs of similar log lines (first and last kept, count shown) | balanced | no |
 | Cap huge outputs: head and tail kept, middle omitted and labelled | aggressive | no |
@@ -260,18 +261,20 @@ message list. Reproduce with `compare/export.js` then `compare/run_headroom.py`.
 |---|---|---|---|
 | JSON API response, 150 records | 74% | 78% | 72% |
 | Mixed-shape JSON events | 73% | 74% | 73% |
-| grep results, 500 hits | 20% | 20% | 20% |
+| grep results, 500 hits | 21% | 21% | 20% |
 | Server log, retry storm | 61% | 98% | 10% |
 | JavaScript source file | 6% | 6% | 0% |
-| Python source with comments | 0% | 0% | 0% |
-| Verbose prose | 8% | 8% | 0% |
+| Python source with comments | 66% | 66% | 0% |
+| Verbose prose | 64% | 64% | 0% |
 | Agent conversation, 5 tool results | 59% | 65% | 17% |
-| Total | 44% | 50% | 20% |
+| Total | 51% | 57% | 20% |
 
 How to read this:
 - Safe mode is the like-for-like column: Headroom's output on JSON, logs and grep kept every
   row. Safe mode is verified lossless by decoder round-trip tests (`src/formats.js`).
 - Balanced and aggressive win by omitting content and are not comparable.
+- The Python and prose payloads are one template repeated 60 times, which line references collapse. Real files
+  rarely repeat like that, so those two rows overstate what to expect.
 - The counter is characters divided by four. Real tokenizers may count digits, punctuation and
   CSV-style rows differently, so real savings can be lower or higher. Use dry run: the
   dashboard's "billed input" column is the real figure.
@@ -306,18 +309,23 @@ tokens counted with tiktoken o200k_base, Headroom 0.42.0 in library mode. Saving
 
 | | Final request | Cache-weighted cost, whole session |
 |---|---|---|
-| TokenTrimmer safe | 3.3% | 2.7% |
-| TokenTrimmer aggressive | 10.8% | 9.2% |
-| TokenTrimmer balanced + `TOKENTRIMMER_STALE_AFTER=10` | 27.4% | 14.7% |
-| TokenTrimmer balanced + stale 10 + `TOKENTRIMMER_RETRIEVE=1` | 26.5% | 13.6% (before retrieval round trips) |
-| Headroom library mode (default, coding, general, agent-90 profiles alike) | 1.0% | 1.0% |
+| TokenTrimmer safe | 6.9% | 5.3% |
+| TokenTrimmer aggressive | 13.9% | 11.6% |
+| TokenTrimmer balanced + `TOKENTRIMMER_STALE_AFTER=10` | 29.4% | 16.4% |
+| TokenTrimmer balanced + stale 10 + `TOKENTRIMMER_RETRIEVE=1` | 28.7% | 15.8% (before retrieval round trips) |
+| Headroom library mode, default and coding profiles | 1.2% | 1.3% |
+| Headroom library mode, agent-90 profile | 5.6% | not measured |
+
+Most of safe mode's saving on these sessions comes from line references: browser page trees and command output
+repeat the same lines within a result and across results, and a 12-token reference replaces each run.
 
 Cache-weighted cost replays each session request by request, priced as prompt caching bills it (0.1x for the prefix
 shared with the previous request, 1.25x for the rest). Rewriting old messages re-bills everything after them, which is
 why stale mode's saving roughly halves there.
 
 Accuracy: 140 questions over 8 of the sessions, each quoting a line from an old tool result and asking for the line
-after it (exact match), answered by Claude Opus 5.5 through `claude -p` with each compressed conversation:
+after it (exact match), answered by Claude Opus 5.5 through `claude -p` with each compressed conversation. Safe mode
+was measured on the current version; the other TokenTrimmer rows on v0.1.1, which used fewer, longer references:
 
 | | Correct | Answered UNKNOWN | Wrong guess |
 |---|---|---|---|
@@ -328,7 +336,9 @@ after it (exact match), answered by Claude Opus 5.5 through `claude -p` with eac
 | TokenTrimmer balanced + stale 10 + retrieval | 99% | 0 | 1 |
 | Headroom default | 95% | 3 | 4 |
 
-Safe mode matched the original, including answers that had to be found by following a repeated-block reference.
+Safe mode matched the original. Because few of those 140 answers sit behind a reference, 38 more questions were
+added whose answer, in the safe output, is hidden behind a `[same as …]` reference (both kinds); the model answered
+38/38, the same as on the original conversation.
 Stale mode alone loses most old detail. With retrieval the model read back what it needed and recovered almost
 all of it, but every retrieval is an extra round trip that re-reads the context: in this test, where most
 questions targeted cut content, stale + retrieval cost about 1.7x as much as sending the original. It pays off
@@ -349,16 +359,19 @@ send to the API. Tokens include any retrieval tool a proxy adds.
 | Headroom, `cache` (default) | 1.5% | 0.2% | 104 ms |
 | Headroom, `--lossless` | 1.6% | 0.9% | 66 ms |
 | Headroom, `token` | 5.8% | **-11.4%** (costs more than the original) | 914 ms |
-| TokenTrimmer, safe (default) | 3.7% | 2.6% | 12 ms |
-| TokenTrimmer, aggressive | 11.2% | 9.2% | 13 ms |
-| TokenTrimmer, balanced + stale 10 | 27.8% | 14.6% | 13 ms |
-| TokenTrimmer, balanced + stale 10 + retrieval | 26.9% | 13.5% | 15 ms |
+| TokenTrimmer, safe (default) | 7.2% | 5.3% | 9 ms |
+| TokenTrimmer, aggressive | 14.3% | 11.6% | 9 ms |
+| TokenTrimmer, balanced + stale 10 | 29.8% | 16.4% | 9 ms |
+| TokenTrimmer, balanced + stale 10 + retrieval | 28.8% | 15.3% | 10 ms |
+
+Safe mode is lossless (every request verified) and still smaller than Headroom's `token` mode, which rewrites
+content.
 
 Headroom's `token` mode re-compresses earlier turns as the conversation grows, so the cached prefix keeps
 changing and most of each request is billed at the full rate. Headroom's `cache` mode avoids that but compresses
 little of these sessions. Latency is the proxy's own time per request on an Apple Silicon laptop, with a
 local upstream that answers instantly. The slowest single request took 17 s through Headroom `token` mode and
-0.4 s through TokenTrimmer. Where proxies add `cache_control` breakpoints they are ignored in the cost
+0.13 s through TokenTrimmer. Where proxies add `cache_control` breakpoints they are ignored in the cost
 column, because they mark the cache rather than being part of it.
 
 Not measured here: Headroom's own retrieval (CCR) under a real model. In these runs its proxies added at most

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tokentrimmer",
-  "version": "0.1.1",
+  "version": "0.2.0",
   "description": "Cut Claude Code, Anthropic and OpenAI token usage and API cost: a local proxy that losslessly compresses tool output (JSON, logs, grep) before it reaches the API. Prompt-cache safe, zero dependencies.",
   "keywords": [
     "claude",

package/src/formats.js

@@ -275,36 +275,57 @@ function decodeLogTemplates(text) {
 
 // ---------- repeated line blocks ----------
 
-// A run of lines already shown verbatim in an earlier tool result is replaced by one reference line.
-// `message` is the index in the request's messages array; `result` (1-based) is only written when that
-// message holds more than one tool-result text. Lossless: decodeBlockRefs restores the exact text.
-const BLOCK_REF = /^\[tokentrimmer: lines (\d+)-(\d+) of (?:tool result (\d+)|the tool result) in message (\d+) repeated here\]$/;
+// A run of lines the model already has, verbatim, in an earlier tool result is replaced by one short
+// reference line, e.g. "[same as message 175 lines 16-19]". `message` is the index in the request's
+// messages array; `result` (1-based) is only written when that message holds more than one tool-result
+// text. Line numbers count lines of that result *as sent*, which is what the model reads. Lossless:
+// decodeBlockRefs against the sent request restores the exact text.
+const BLOCK_REF = /^\[same as message (\d+)(?: result (\d+))? lines (\d+)-(\d+)\]$/;
 
 function blockRef(from, to, message, result) {
-  const where = result ? `tool result ${result}` : 'the tool result';
-  return `[tokentrimmer: lines ${from}-${to} of ${where} in message ${message} repeated here]`;
+  return `[same as message ${message}${result ? ` result ${result}` : ''} lines ${from}-${to}]`;
 }
 
+// A run repeating lines earlier in the same tool result: "[same as lines 3-5 above]", counting lines of
+// this result as sent.
+const LOCAL_REF = /^\[same as lines (\d+)-(\d+) above\]$/;
+const localRef = (from, to) => `[same as lines ${from}-${to} above]`;
+const isRefLine = (l) => BLOCK_REF.test(l) || LOCAL_REF.test(l);
+
+// An identical repeated tool result becomes one line naming the message that already holds it.
+const IDENTICAL_REF = /^\[same as a tool result in message (\d+)\]$/;
+const identicalRef = (message) => `[same as a tool result in message ${message}]`;
+
 function hasBlockRefs(text) {
-  return text.split('\n').some((l) => BLOCK_REF.test(l));
+  return text.split('\n').some(isRefLine);
 }
 
-// getText(message, result) returns the referenced original text (result is 1-based, defaults to 1).
+// getText(message, result) returns the referenced text as sent (result is 1-based, defaults to 1).
 function decodeBlockRefs(text, getText) {
   const out = [];
-  for (const l of text.split('\n')) {
+  const sent = text.split('\n');
+  sent.forEach((l, at) => {
+    const local = LOCAL_REF.exec(l);
+    if (local) {
+      const from = Number(local[1]), to = Number(local[2]);
+      if (from < 1 || to < from || to > at) throw new Error(`reference to lines not above it: ${l}`);
+      const lines = sent.slice(from - 1, to);
+      if (lines.some(isRefLine)) throw new Error(`reference to a reference line: ${l}`);
+      out.push(...lines);
+      return;
+    }
     const m = BLOCK_REF.exec(l);
     if (!m) {
       out.push(l);
-      continue;
+      return;
     }
-    const src = getText(Number(m[4]), m[3] ? Number(m[3]) : 1);
+    const src = getText(Number(m[1]), m[2] ? Number(m[2]) : 1);
     if (typeof src !== 'string') throw new Error(`block reference to a missing tool result: ${l}`);
     const lines = src.split('\n');
-    const from = Number(m[1]), to = Number(m[2]);
+    const from = Number(m[3]), to = Number(m[4]);
     if (from < 1 || to < from || to > lines.length) throw new Error(`block reference out of range: ${l}`);
     out.push(...lines.slice(from - 1, to));
-  }
+  });
   return out.join('\n');
 }
 
@@ -333,6 +354,6 @@ function splitRetrieveNote(text) {
 module.exports = {
   encCell, splitCells, parseCell, encodeTable, encodeGroups, decodeTables, decodeJSONOutput, expand,
   templateLogs, decodeLogTemplates, ungroupGrep, STRIPPED_ROW,
-  BLOCK_REF, blockRef, hasBlockRefs, decodeBlockRefs,
+  BLOCK_REF, blockRef, LOCAL_REF, localRef, isRefLine, hasBlockRefs, decodeBlockRefs, IDENTICAL_REF, identicalRef,
   RETRIEVE_TOOL, retrieveId, retrieveNote, splitRetrieveNote,
 };

package/src/messages.js

@@ -11,10 +11,13 @@ const formats = require('./formats');
 
 const DEDUPE_MIN_CHARS = 400;
 const LOSSY = new Set(['json-drop-empty', 'log-collapse', 'cap-large-output']);
-// Repeated line blocks: a run must span at least BLOCK_WINDOW lines and BLOCK_MIN_CHARS characters
-// to be worth a reference line.
-const BLOCK_WINDOW = 4;
-const BLOCK_MIN_CHARS = 100; // tuned on real sessions via compare/real (opts.blockWindow / opts.blockMinChars override)
+// Repeated line blocks: a run must span at least BLOCK_WINDOW lines and BLOCK_MIN_CHARS characters, and be
+// BLOCK_RATIO times longer than the reference replacing it. Tuned on real sessions (compare/real): with
+// short references even a single long repeated line pays off.
+const BLOCK_WINDOW = 1;
+const BLOCK_MIN_CHARS = 60; // tuned on real sessions via compare/real (opts.blockWindow / opts.blockMinChars override)
+const MAX_SITES = 16; // earlier positions remembered per repeated window
+const BLOCK_RATIO = 2; // the run must be at least this many times longer than its reference line
 
 // Tools whose output is file content the model may later quote back exactly (edit tools match
 // text verbatim). Their results are never rewritten, but identical repeats are still deduplicated.
@@ -40,6 +43,7 @@ function makeContext(opts) {
     // Retrieval: id -> original text of every tool result that lost content (opts.retrieve).
     store: new Map(),
     blockWindow: opts.blockWindow || BLOCK_WINDOW, blockMinChars: opts.blockMinChars ?? BLOCK_MIN_CHARS,
+    blockRatio: opts.blockRatio ?? BLOCK_RATIO,
   };
 }
 
@@ -64,43 +68,83 @@ function note(ctx, list) {
 
 const windowKey = (lines, i, w) => crypto.createHash('sha1').update(lines.slice(i, i + w).join('\n')).digest('base64');
 
-// Remember a tool result that is sent unchanged, so later results can point at its lines.
-function indexSource(ctx, text, msgIndex, k) {
-  const lines = text.split('\n');
+// Remember a tool result as it is sent, so later results can point at its lines. Line numbers are
+// those of the sent text (what the model reads). Lines that are themselves references are never
+// pointed at, so a reference always resolves to text the model has verbatim.
+function indexSource(ctx, sent, msgIndex, k) {
+  const lines = sent.split('\n');
   if (lines.length < ctx.blockWindow) return;
   const id = `${msgIndex}:${k}`;
   ctx.sources.set(id, lines);
+  const isRef = lines.map(formats.isRefLine);
+  let refsAhead = 0; // reference lines inside the current window
+  for (let i = 0; i < ctx.blockWindow - 1 && i < lines.length; i++) refsAhead += isRef[i];
   for (let i = 0; i + ctx.blockWindow <= lines.length; i++) {
-    const key = windowKey(lines, i, ctx.blockWindow);
-    if (!ctx.blocks.has(key)) ctx.blocks.set(key, { id, msgIndex, k, line: i });
+    refsAhead += isRef[i + ctx.blockWindow - 1];
+    if (!refsAhead) {
+      const key = windowKey(lines, i, ctx.blockWindow);
+      const at = ctx.blocks.get(key);
+      const hit = { id, msgIndex, k, line: i };
+      if (!at) ctx.blocks.set(key, [hit]);
+      else if (at.length < MAX_SITES) at.push(hit);
+    }
+    refsAhead -= isRef[i];
   }
 }
 
-// Replace runs of lines already sent verbatim earlier with one reference line each. Returns null
-// when nothing qualifies. The first earlier occurrence is always used, so output is deterministic.
+// Replace runs of lines the model already has verbatim with one reference line each: to an earlier tool
+// result ("[same as message M lines A-B]") or to lines above in this one ("[same as lines A-B above]").
+// Returns null when nothing qualifies. Sites are tried in send order and only a strictly longer run wins,
+// so the output is deterministic and depends only on this text and earlier messages.
 function dedupeBlocks(text, ctx) {
-  if (!ctx.blocks.size || formats.hasBlockRefs(text)) return null;
+  const w = ctx.blockWindow;
   const lines = text.split('\n');
-  if (lines.length < ctx.blockWindow) return null;
-  const out = [];
+  if (lines.length < w + 1 && !ctx.blocks.size) return null;
+  const out = []; // lines as sent
+  const local = new Map(); // window key -> sites in `out` (verbatim lines only)
+  let verbatimRun = 0; // consecutive verbatim lines at the end of `out`
   let replaced = false;
+  const run = (src, from, i) => {
+    let n = 0;
+    let chars = 0;
+    while (i + n < lines.length && from + n < src.length && lines[i + n] === src[from + n] && !formats.isRefLine(src[from + n])) {
+      chars += lines[i + n++].length + 1;
+    }
+    return [n, chars];
+  };
   let i = 0;
   while (i < lines.length) {
-    const hit = i + ctx.blockWindow <= lines.length && ctx.blocks.get(windowKey(lines, i, ctx.blockWindow));
-    if (hit) {
-      const src = ctx.sources.get(hit.id);
-      let n = 0;
-      let chars = 0;
-      while (i + n < lines.length && hit.line + n < src.length && lines[i + n] === src[hit.line + n]) chars += lines[i + n++].length + 1;
-      if (n >= ctx.blockWindow && chars >= ctx.blockMinChars) {
-        const result = ctx.resultCounts.get(hit.msgIndex) > 1 ? hit.k : 0;
-        out.push(formats.blockRef(hit.line + 1, hit.line + n, hit.msgIndex, result));
-        replaced = true;
-        i += n;
-        continue;
+    let best = null;
+    if (i + w <= lines.length) {
+      const key = windowKey(lines, i, w);
+      for (const site of ctx.blocks.get(key) || []) {
+        const [n, chars] = run(ctx.sources.get(site.id), site.line, i);
+        if (!best || chars > best.chars) {
+          const ref = formats.blockRef(site.line + 1, site.line + n, site.msgIndex, ctx.resultCounts.get(site.msgIndex) > 1 ? site.k : 0);
+          best = { n, chars, ref };
+        }
+      }
+      for (const at of local.get(key) || []) {
+        const [n, chars] = run(out, at, i);
+        if (!best || chars > best.chars) best = { n, chars, ref: formats.localRef(at + 1, at + n) };
       }
     }
+    // Only worth it when the run is clearly longer than the reference that replaces it.
+    if (best && best.n >= w && best.chars >= ctx.blockMinChars && best.chars >= best.ref.length * ctx.blockRatio) {
+      out.push(best.ref);
+      verbatimRun = 0;
+      replaced = true;
+      i += best.n;
+      continue;
+    }
     out.push(lines[i++]);
+    if (++verbatimRun >= w) {
+      const at = out.length - w;
+      const key = windowKey(out, at, w);
+      const sites = local.get(key);
+      if (!sites) local.set(key, [at]);
+      else if (sites.length < MAX_SITES) sites.push(at);
+    }
   }
   return replaced ? out.join('\n') : null;
 }
@@ -133,28 +177,45 @@ function shrinkToolText(text, ctx, msgIndex, excluded, k, stale) {
     if (ctx.seen.has(hash)) {
       note(ctx, ['dedupe-identical']);
       ctx.fieldsChanged++;
-      return `[tokentrimmer: identical to the tool result in message ${ctx.seen.get(hash)}]`;
+      return formats.identicalRef(ctx.seen.get(hash));
     }
     ctx.seen.set(hash, msgIndex);
   }
+  const sent = shrinkUnique(text, ctx, excluded, stale);
+  indexSource(ctx, sent, msgIndex, k);
+  return sent;
+}
+
+function shrinkUnique(text, ctx, excluded, stale) {
   if (excluded) {
     // File content stays verbatim (edit tools match it exactly), and so can be referenced later.
     ctx.skipped++;
-    indexSource(ctx, text, msgIndex, k);
     return text;
   }
   const r = compressText(text, (stale && staleOptions(ctx.opts)) || ctx.opts);
-  const blocks = dedupeBlocks(text, ctx);
+  // References can replace lines of the raw text or of its compressed form; keep the shortest.
+  // Never when the original already has lines that look like references: they would be ambiguous.
+  const canRef = !formats.hasBlockRefs(text);
+  let blocks = canRef ? dedupeBlocks(text, ctx) : null;
+  let onCompressed = false;
+  if (canRef && r.techniques.length) {
+    const b2 = dedupeBlocks(r.text, ctx);
+    if (b2 !== null && (blocks === null || b2.length < blocks.length)) [blocks, onCompressed] = [b2, true];
+  }
   if (blocks !== null && blocks.length < r.text.length) {
-    note(ctx, ['dedupe-block']);
+    note(ctx, onCompressed ? [...r.techniques, 'dedupe-block'] : ['dedupe-block']);
     ctx.fieldsChanged++;
+    if (onCompressed && ctx.opts.retrieve && r.techniques.some((t) => LOSSY.has(t))) {
+      const id = formats.retrieveId(text);
+      ctx.store.set(id, text);
+      note(ctx, ['retrievable']);
+      return blocks + formats.retrieveNote(id);
+    }
     return blocks;
   }
   if (r.techniques.length) {
     note(ctx, r.techniques);
     ctx.fieldsChanged++;
-  } else {
-    indexSource(ctx, text, msgIndex, k);
   }
   if (ctx.opts.retrieve && r.techniques.some((t) => LOSSY.has(t))) {
     // Content was dropped: keep the original for the retrieval tool and tell the model how to get it.

package/src/verify.js

@@ -14,7 +14,7 @@ const formats = require('./formats');
 const { toolTexts, staleOptions } = require('./messages');
 
 const LOSSY = new Set(['json-drop-empty', 'log-collapse', 'cap-large-output']);
-const DEDUPE_REF = /^\[tokentrimmer: identical to the tool result in message (\d+)\]$/;
+const DEDUPE_REF = formats.IDENTICAL_REF;
 
 function containsString(node, str) {
   if (typeof node === 'string') return node === str;
@@ -43,15 +43,17 @@ function classify(orig, comp, originalBody, compressedBody, opts) {
       : { status: 'FAILED', techniques: ['dedupe-identical'], reason: 'referenced message does not contain this text' };
   }
   if (formats.hasBlockRefs(comp) && !formats.hasBlockRefs(orig)) {
-    // Decode against the original (nothing lost) and against the compressed body (the referenced
-    // lines are really in front of the model, unchanged).
+    // References point at lines as sent, so decode against the compressed body: an exact match proves
+    // both that nothing was lost and that the referenced lines really are in front of the model.
     const t = ['dedupe-block'];
     try {
-      for (const body of [originalBody, compressedBody]) {
-        const decoded = formats.decodeBlockRefs(comp, (m, k) => toolTexts(body.messages[m])[k - 1]);
-        if (decoded !== orig) return { status: 'FAILED', techniques: t, reason: 'referenced lines differ from the original text' };
-      }
-      return { status: 'exact', techniques: t };
+      const decoded = formats.decodeBlockRefs(comp, (m, k) => toolTexts(compressedBody.messages[m])[k - 1]);
+      if (decoded === orig) return { status: 'exact', techniques: t };
+      // References inside a compressed form: what they expand to must be what the compressor produces.
+      if (formats.hasBlockRefs(decoded)) return { status: 'FAILED', techniques: t, reason: 'a reference points at another reference' };
+      const inner = classify(orig, decoded, originalBody, compressedBody, opts);
+      if (inner.status === 'FAILED') return { ...inner, reason: `after expanding references: ${inner.reason}` };
+      return { ...inner, techniques: [...inner.techniques, ...t] };
     } catch (e) {
       return { status: 'FAILED', techniques: t, reason: `decode error: ${e.message}` };
     }