tanuki-context 0.2.1 → 0.2.3

package/README.md

@@ -22,21 +22,25 @@ cleaned up and drawn. A vision model reads it directly:
 
 ![a rendered page: dense 5x8-pixel text, 312 columns of system log](https://raw.githubusercontent.com/Osyna/tanuki-context/main/docs/example-page.png)
 
-*Four pages like this replace 51,200 tokens of raw text with 5,320 —
-a 90% cut. Zoom in: it's all still there, errors verbatim.*
+*Four pages like this replace 51,200 tokens of raw text with 5,264 —
+a 90% cut. Zoom in: it's all still there, errors verbatim. (Hostnames,
+IPs, MACs, and network names were rewritten to placeholders before
+rendering; the structure and repetition are untouched.)*
 
 ## What you save
 
-Measured on that same 200 KB slice of a real system journal (1,370 lines
-from tailscaled, NetworkManager, the kernel, and friends — mixed on purpose;
-a plain `journalctl` tail is so repetitive the result looks fake):
+Measured on that same 200 KB slice of a real system journal (1,434 lines
+from tailscaled, NetworkManager, the kernel, and friends — mixed on
+purpose; a plain `journalctl` tail is so repetitive the result looks fake.
+Identifiers were rewritten before measuring; the repetition shape and
+every error line are untouched):
 
 | how the log enters the conversation | tokens |     saved |
 | ----------------------------------- | -----: | --------: |
 | pasted as raw text                  | 51,200 |         0 |
 | drawn as image pages                | 10,752 |  **-79%** |
-| noise removed first, then drawn     |  5,320 |  **-90%** |
-| plus codebook and tiny font         |  2,352 |  **-95%** |
+| noise removed first, then drawn     |  5,264 |  **-90%** |
+| plus codebook and tiny font         |  2,576 |  **-95%** |
 
 Every row is one command on your own file:
 
@@ -81,8 +85,8 @@ call replaces a whole round of trial and error:
 ```
 npx tanuki-context estimate journal.log 0
 # { "imageTokens": 10752, "verdict": "PIPELINE cheaper",
-#   "recommend": { "distill": true, "codebook": true, "imageTokens": 3920,
-#                  "pages": 3, "tinyImageTokens": 2352 }, ... }
+#   "recommend": { "distill": true, "codebook": true, "imageTokens": 4256,
+#                  "pages": 3, "tinyImageTokens": 2576 }, ... }
 ```
 
 **Implicit (proxy) — for clients you can't modify.** A small local relay.
@@ -160,8 +164,8 @@ input (11 lines):                          distilled (deterministic):
 ```
 
 The ERROR line survived untouched, the count is exact, and the same input
-always produces the same output. On the 200 KB journal: 1,371 lines to 621,
-half the characters gone, all 318 error/warn lines kept verbatim.
+always produces the same output. On the 200 KB journal: 1,434 lines to 641,
+half the characters gone, all 310 error/warn lines kept verbatim.
 
 **Stage 1 — the ladder.** Five text-compression levels, 0 (off) to 4 (gist
 only). From level 2 up, an exact-recall guard keeps code, identifiers,
@@ -225,6 +229,49 @@ harness that holds the two engines to byte-identical JSON and
 pixel-identical PNGs on every knob combination, including distilled
 renders and a full MCP session with error paths.
 
+## How it compares
+
+Five approaches to the same problem, measured on the same 200 KB journal
+(51,200 tokens raw) on the same machine. They are not five competitors —
+they are different answers to "does the model need to see this at all?",
+and several of them compose.
+
+| approach | what enters context | tokens | what the model can still do |
+| --- | --- | ---: | --- |
+| raw text (baseline) | everything, as text | 51,200 | everything, quotable |
+| caveman-style compression (tanuki ladder L4, stays text) | reworded text | 51,198 (-0%) | on a log: nothing changes — the exact-recall guard protects every line, and unguarded caveman would corrupt them. On prose it manages -6% (DESIGN.md). Wrong tool for logs |
+| ponytail strategy (send only the queried slice: `distill` + `query`) | matched + error lines, as text | 22,110 (-57%) | greppable, quotable text of everything important; the unmatched rest is gone |
+| [pxpipe](https://github.com/teamchong/pxpipe) (its own `export` CLI on this corpus) | 8 PNG pages + prompt + factsheet | ~11,965 (-77%) | read everything back; 96 precision-critical strings ride verbatim in the factsheet |
+| tanuki, distill + render | 4 PNG pages | 5,264 (-90%) | read everything that survives distill: repeats collapsed with exact counts, all 310 error/warn lines verbatim |
+| tanuki, + codebook + tiny font | 2 PNG pages | 2,576 (-95%) | same content, lossy glyphs (99.7% read-back) — opt-in |
+| [context-mode](https://www.npmjs.com/package/context-mode) (sandbox pass over the file) | one analysis result | ~270 per question (-99.5%) | only the answer it asked for; the file never enters context at all |
+
+Reading that honestly:
+
+- **context-mode wins whenever a question suffices.** If the model never
+  needs to *see* the log, ~270 tokens per query is unbeatable. The moment it
+  needs the material in front of it (debugging an unknown, quoting context,
+  "read all of this and tell me what's weird"), retrieval stops being the
+  same product. The two compose: keep the file in context-mode's sandbox,
+  render it with tanuki the day the model actually has to read it.
+- **pxpipe and tanuki are the same engine**, so the gap between -77% and
+  -90% here is not the imaging — it is distill (pxpipe does not de-noise
+  logs) plus the sidecar text its export flow pastes alongside. On prose,
+  where distill does nothing, the two land in the same place. pxpipe's
+  factsheet is a fidelity feature tanuki lacks: guaranteed-verbatim strings
+  next to the pages.
+- **Caveman-style prompt compression is the weakest option for logs**, and
+  tanuki's own level 4 proves it: the guard that makes it safe also makes
+  it a no-op on structured content. It exists for wordy prose.
+- **The ponytail row is a strategy, not a tool**: send only what is needed.
+  It beats nothing on this corpus except raw text, but it composes — the
+  22,110-token slice drawn as pages is 4,704 tokens, and `estimate` will
+  happily price that route for you.
+
+Every row is reproducible: rows 1-3 and 5-6 are single `tanuki-context`
+commands, the pxpipe row is `pxpipe export --stdin < your.log` on their
+CLI, the context-mode row is one `ctx_execute_file` call in its sandbox.
+
 ## Limits, plainly
 
 - **Rendering is exact; reading is a model skill.** Pages are pixel-faithful

package/dist/agent.js

@@ -65,7 +65,7 @@ class ZJ{value;constructor(J){this.value=J}}function nJ(){let J=process.env.TANU
 //!      tanuki-context estimate <file> [level] [--distill]
 //!      tanuki-context render <file> [level] [outdir]
 //!      tanuki-context proxy [--port N] [--upstream URL] [knobs]   (implicit mode)
-var cJ="0.2.1",NJ=6;function JZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function C(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof ZJ){let z=J.value;return Number.isFinite(z)&&Number.isInteger(z)?z.toFixed(1):String(z)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let q="[";for(let X=0;X<J.length;X++){if(X>0)q+=",";q+=C(J[X],!1)}return q+"]"}let z=Z+"  ",K=`[
+var cJ="0.2.3",NJ=6;function JZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function C(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof ZJ){let z=J.value;return Number.isFinite(z)&&Number.isInteger(z)?z.toFixed(1):String(z)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let q="[";for(let X=0;X<J.length;X++){if(X>0)q+=",";q+=C(J[X],!1)}return q+"]"}let z=Z+"  ",K=`[
 `;for(let q=0;q<J.length;q++){if(q>0)K+=`,
 `;K+=z+C(J[q],!0,z)}return K+`
 `+Z+"]"}let V=J,G=Object.keys(V).sort(JZ);if(G.length===0)return"{}";if(!Q){let z="{";for(let K=0;K<G.length;K++){if(K>0)z+=",";z+=JSON.stringify(G[K])+":"+C(V[G[K]],!1)}return z+"}"}let Y=Z+"  ",H=`{

package/dist/cli.js

@@ -71,7 +71,7 @@ var c={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,min
 //!      tanuki-context estimate <file> [level] [--distill]
 //!      tanuki-context render <file> [level] [outdir]
 //!      tanuki-context proxy [--port N] [--upstream URL] [knobs]   (implicit mode)
-var NZ="0.2.1",TJ=6;function jZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function E(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof e){let H=J.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let B="[";for(let X=0;X<J.length;X++){if(X>0)B+=",";B+=E(J[X],!1)}return B+"]"}let H=Z+"  ",K=`[
+var NZ="0.2.3",TJ=6;function jZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function E(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof e){let H=J.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let B="[";for(let X=0;X<J.length;X++){if(X>0)B+=",";B+=E(J[X],!1)}return B+"]"}let H=Z+"  ",K=`[
 `;for(let B=0;B<J.length;B++){if(B>0)K+=`,
 `;K+=H+E(J[B],!0,H)}return K+`
 `+Z+"]"}let V=J,z=Object.keys(V).sort(jZ);if(z.length===0)return"{}";if(!Q){let H="{";for(let K=0;K<z.length;K++){if(K>0)H+=",";H+=JSON.stringify(z[K])+":"+E(V[z[K]],!1)}return H+"}"}let Y=Z+"  ",G=`{

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tanuki-context",
-  "version": "0.2.1",
+  "version": "0.2.3",
   "description": "Token-cutting context pipeline as an MCP server: distill logs, compress text, render dense PNG pages (pxpipe imaging engine). Zero dependencies.",
   "license": "MIT",
   "type": "module",