tanuki-context 0.2.0 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/README.md +256 -196
  2. package/dist/agent.js +1 -1
  3. package/dist/cli.js +2 -2
  4. package/package.json +1 -1
package/README.md CHANGED
@@ -1,36 +1,48 @@
1
1
  # tanuki-context
2
2
 
3
3
  [![npm](https://img.shields.io/npm/v/tanuki-context)](https://www.npmjs.com/package/tanuki-context)
4
- Zero dependencies. 0.98 MB tarball. Runs anywhere Node >= 18 runs. MIT.
4
+ Zero dependencies. 0.98 MB install. Runs anywhere Node >= 18 runs. MIT.
5
5
 
6
- A model pays about 1 token per 4 characters to read text. A PNG page is
7
- billed by pixel dimensions (Anthropic's 28x28-pixel patch grid), and a dense
8
- page fits 3+ characters per token. [pxpipe](https://github.com/teamchong/pxpipe)
9
- measured ~3.1 chars per image-token on real Claude Code traffic and built a
10
- transparent proxy on that gap. tanuki-context is the same imaging engine with
11
- a different contract: the model calls a tool when it wants the cut, sees
12
- exactly what got imaged, and nothing is rewritten behind its back.
6
+ **AI models charge you for every word they read. tanuki-context turns the
7
+ bulky parts of a conversation — logs, command output, long documents — into
8
+ compact images the same model reads for a fraction of the price.**
13
9
 
14
- It takes bulky context (logs, docs, command output), optionally distills and
15
- compresses it, then renders it as dense PNG pages. You get the numbers before
16
- committing: `estimate` prices both routes and says which one wins.
10
+ That sounds like a trick, but it is just how the pricing works. Text costs
11
+ about 1 token per 4 characters. An image costs a fixed amount set by its
12
+ size in pixels, no matter how much text is drawn inside it. So if you pack
13
+ 28,000 characters of text into one dense 1568x728 image, the model reads it
14
+ for 1,456 tokens instead of about 7,000. [pxpipe](https://github.com/teamchong/pxpipe)
15
+ discovered how far that gap stretches on real traffic (~3.1 characters per
16
+ image token, against ~1 for text) and built a proxy on it. tanuki-context is
17
+ the same imaging engine, packaged so the model itself decides when to use it
18
+ — plus a relay mode for when you can't change anything.
17
19
 
18
- ## Measured
20
+ This is a real page, straight out of the pipeline — 200 KB of system log,
21
+ cleaned up and drawn. A vision model reads it directly:
19
22
 
20
- A 200 KB slice of this machine's system journal: 1,370 lines from tailscaled,
21
- NetworkManager, the kernel, wpa_supplicant, and friends. Not a synthetic
22
- fixture, and not cherry-picked to repeat itself (a plain `journalctl` tail is
23
- so repetitive that distill flattens it to 5 lines, which looks fake even
24
- though it isn't).
23
+ ![a rendered page: dense 5x8-pixel text, 312 columns of system log](https://raw.githubusercontent.com/Osyna/tanuki-context/main/docs/example-page.png)
25
24
 
26
- | how it enters context | tokens | saved |
27
- | -------------------------------- | -----: | --------: |
28
- | pasted as raw text | 51,200 | 0 |
29
- | rendered as pages (level 0) | 10,752 | **-79%** |
30
- | distilled first, then rendered | 5,320 | **-90%** |
31
- | distilled + codebook + tiny font | 2,352 | **-95%** |
25
+ *Four pages like this replace 51,200 tokens of raw text with 5,264 —
26
+ a 90% cut. Zoom in: it's all still there, errors verbatim. (Hostnames,
27
+ IPs, MACs, and network names were rewritten to placeholders before
28
+ rendering; the structure and repetition are untouched.)*
32
29
 
33
- Reproduce on your own log; every row is one command:
30
+ ## What you save
31
+
32
+ Measured on that same 200 KB slice of a real system journal (1,434 lines
33
+ from tailscaled, NetworkManager, the kernel, and friends — mixed on
34
+ purpose; a plain `journalctl` tail is so repetitive the result looks fake.
35
+ Identifiers were rewritten before measuring; the repetition shape and
36
+ every error line are untouched):
37
+
38
+ | how the log enters the conversation | tokens | saved |
39
+ | ----------------------------------- | -----: | --------: |
40
+ | pasted as raw text | 51,200 | 0 |
41
+ | drawn as image pages | 10,752 | **-79%** |
42
+ | noise removed first, then drawn | 5,264 | **-90%** |
43
+ | plus codebook and tiny font | 2,576 | **-95%** |
44
+
45
+ Every row is one command on your own file:
34
46
 
35
47
  ```
36
48
  npx tanuki-context estimate your.log 0
@@ -38,76 +50,84 @@ npx tanuki-context estimate your.log 0 --distill
38
50
  npx tanuki-context estimate your.log 0 --distill --codebook --font tiny
39
51
  ```
40
52
 
41
- Savings depend on the content. Repetitive logs crush well. Dense unique text
42
- still gets the flat ~3-4x from imaging alone. Small inputs technically win
43
- too, but by so little it is rarely worth the modality switch; see
44
- [Limits](#limits-plainly).
53
+ `estimate` is instant, never renders anything, and tells you honestly when
54
+ plain text would be cheaper.
45
55
 
46
- ## Quick start
56
+ ## Try it in 30 seconds
57
+
58
+ Claude Code:
47
59
 
48
60
  ```
49
- npx tanuki-context
61
+ claude mcp add tanuki-context -- npx -y tanuki-context
50
62
  ```
51
63
 
52
- That is the MCP stdio server. Register it in any client:
64
+ Any other MCP client:
53
65
 
54
66
  ```json
55
67
  { "mcpServers": { "tanuki-context": { "command": "npx", "args": ["-y", "tanuki-context"] } } }
56
68
  ```
57
69
 
58
- Or poke at the CLI first. Real output, same 200 KB journal slice:
70
+ No AI client at all, just curious what it would save you:
59
71
 
60
72
  ```
61
- npx tanuki-context estimate journal.log 0
62
- # { "imageTokens": 10752, "verdict": "PIPELINE cheaper",
63
- # "recommend": { "distill": true, "codebook": true, "imageTokens": 3920,
64
- # "pages": 3, "tinyImageTokens": 2352 }, ... }
65
- npx tanuki-context render journal.log 2 ./pages --codebook
66
- # ./pages/page0.png ...
73
+ npx tanuki-context estimate some-big-file.log 0
67
74
  ```
68
75
 
69
- ## Clients
76
+ ## The two modes
70
77
 
71
- **OMP (oh-my-pi)** - `~/.omp/agent/mcp.json`, or `.omp/mcp.json` in the project:
78
+ **Explicit (MCP tools) — the default and the recommendation.** Your AI gets
79
+ five tools. It calls `tanuki_estimate` on bulky text, reads the verdict, and
80
+ renders only when the pipeline wins. The model stays in charge and can see
81
+ exactly what happened to every byte. Since 0.2.0 the estimate answer includes
82
+ a `recommend` field — the cheapest safe settings, already priced — so one
83
+ call replaces a whole round of trial and error:
72
84
 
73
- ```json
74
- { "mcpServers": { "tanuki-context": { "command": "npx", "args": ["-y", "tanuki-context"] } } }
75
85
  ```
76
-
77
- **jcode** - `~/.jcode/mcp.json`, or `.jcode/mcp.json` in the project. jcode
78
- speaks stdio only; `"shared": true` reuses one server across sessions:
79
-
80
- ```json
81
- { "mcpServers": { "tanuki-context": { "command": "npx", "args": ["-y", "tanuki-context"], "shared": true } } }
86
+ npx tanuki-context estimate journal.log 0
87
+ # { "imageTokens": 10752, "verdict": "PIPELINE cheaper",
88
+ # "recommend": { "distill": true, "codebook": true, "imageTokens": 4256,
89
+ # "pages": 3, "tinyImageTokens": 2576 }, ... }
82
90
  ```
83
91
 
84
- **pi** - pi has no MCP layer, so this package doubles as a pi extension (the
85
- `"pi"` manifest field in package.json):
92
+ **Implicit (proxy) — for clients you can't modify.** A small local relay.
93
+ Point your client's Anthropic URL at it and oversized blocks get swapped for
94
+ image pages on the way out, under strict rules (listed below — the short
95
+ version is: it never touches your prompt, your tools, or your latest
96
+ message, and it never rewrites anything unless the numbers clearly win):
86
97
 
87
98
  ```
88
- pi install npm:tanuki-context
99
+ npx tanuki-context proxy # listens on 127.0.0.1:8484
100
+ export ANTHROPIC_BASE_URL=http://127.0.0.1:8484
89
101
  ```
90
102
 
91
- The five `tanuki_*` tools register natively; the extension spawns this
92
- package's own stdio server per session.
103
+ Both modes run the same engine and log their savings to the same file, which
104
+ `tanuki_stats` summarizes.
93
105
 
94
- **Claude Code**:
106
+ ## What people actually use it for
95
107
 
96
- ```
97
- claude mcp add tanuki-context -- npx -y tanuki-context
98
- ```
108
+ - **Pasting a build log into a chat.** 200 KB of CI output becomes a few
109
+ pages; the error lines survive verbatim, the 4,000 repeats don't.
110
+ - **Agents that read big files.** An agent reviewing code calls `estimate`
111
+ first, renders when it wins, and keeps thousands of tokens of headroom.
112
+ - **Long sessions near the context limit.** Old bulky turns can be imaged by
113
+ the proxy while recent ones stay text.
114
+ - **Fleets of agents.** One in-process server shared by a whole team via the
115
+ Agent SDK glue (below), with a canned instruction block that teaches every
116
+ agent the estimate-first habit.
99
117
 
100
- **Rust instead of Node?** Same pipeline, same numbers, one 5.7 MB static
101
- binary:
118
+ ## The five tools
102
119
 
103
- ```
104
- cargo install --git https://github.com/Osyna/tanuki-context --branch rust
105
- ```
120
+ | tool | what it does | when to reach for it |
121
+ | ----------------- | ----------------------------------------------------------- | ---------------------------------------- |
122
+ | `tanuki_estimate` | prices both routes, returns `recommend` | always first; it's free and instant |
123
+ | `tanuki_render` | the full pipeline, returns PNG pages + a breakdown | when estimate says the pipeline wins |
124
+ | `tanuki_distill` | de-noises a log, output stays text | when you want greppable text, not images |
125
+ | `tanuki_compress` | text-only compression, five levels | prose you'll paraphrase anyway |
126
+ | `tanuki_stats` | totals from the session's savings log | end-of-session accounting |
106
127
 
107
- Then `"command": "tanuki-context", "args": []` in any snippet above, or
108
- `TANUKI_BIN=~/.cargo/bin/tanuki-context` for the pi extension. The two
109
- engines are parity-tested against each other on every change; see
110
- [Two engines](#two-engines) below.
128
+ ---
129
+
130
+ Everything below is for readers who want the machinery.
111
131
 
112
132
  ## How it works
113
133
 
@@ -121,132 +141,182 @@ flowchart LR
121
141
  E --> F["PNG pages<br/>tokens = 28-px patches"]
122
142
  ```
123
143
 
124
- Three stages. The first two are optional.
125
-
126
- **Stage 0, distill.** Built for logs. Repeated lines and repeated multi-line
127
- blocks collapse to one exemplar plus an exact `xN` count. Near-duplicates
128
- that differ only in timestamps, ids, or numbers fold into a template, also
129
- counted. Error, warning, and fail lines are never touched. A `query` regex
130
- returns only the slice you care about, with context. On the journal slice
131
- above: 1,371 lines to 621, half the characters gone, all 318 error/warn
132
- lines kept verbatim.
133
-
134
- **Stage 1, the ladder.** Five text-compression levels, 0 (off) to 4 (gist
135
- only). From level 2 up an exact-recall guard keeps code, identifiers, hashes,
136
- paths, and indented lines verbatim, so loss is confined to prose. Honest
137
- consequence: on technical files there is little prose to cut. This repo's
138
- README loses 10% of its characters at level 2 and 14% at level 4; DESIGN.md
139
- loses 6% at level 4. The ladder earns its keep on wordy prose (meeting notes,
140
- chat transcripts, tickets); on code-heavy content it mostly just guards.
141
- Level 1 (whitespace cleanup) is lossless and always safe.
142
-
143
- **Stage 2, imaging.** The renderer packs text into 312-column pages of 5x8
144
- antialiased cells (1568x728 px max; short pages get width-trimmed) and
145
- encodes grayscale PNGs. This is where the big cut comes from, and it is flat:
146
- raw text pays ~1 token per 4 chars, a full page carries ~28,000 chars for
147
- 1,456 image tokens. Full Unicode, 92,812 codepoints: CJK, Cyrillic, emoji,
148
- astral planes. Unassigned codepoints render as readable `[U+HEX]` escapes.
149
- Nothing disappears silently.
150
-
151
- ## Density knobs
152
-
153
- Measured by `reference/methods-report.mjs` on three corpora; regenerate the
154
- table with `bun reference/methods-report.mjs`. Percentages are against the
155
- baseline renderer on the same content.
156
-
157
- | knob | what it does | code | prose | log |
158
- | ------------------- | ----------------------------------------------------------------------------- | ---: | ----: | ---: |
159
- | `pack` (default on) | single-cell tabs, `⇥N` indent runs, width-trimmed pages; byte-exact | -5% | 0% | 0% |
160
- | `codebook` | repeated tokens and path prefixes become 1-cell sigils plus a `·legend·` line | -9% | 0% | -38% |
161
- | `font: "tiny"` | glyphs box-filtered into 4x6 cells; opt-in, see limits below | -36% | -38% | -40% |
162
- | all three stacked | | **-45%** | **-38%** | **-62%** |
163
-
164
- The three corpora: `src/main.ts` (code), `DESIGN.md` (prose), a path-heavy
165
- synthetic log. Knobs compose but do not add up linearly; `codebook` needs
166
- repetition to bite, `pack` needs indentation.
167
-
168
- ## Tools (MCP, stdio)
169
-
170
- | tool | arguments | returns |
171
- | ----------------- | --------------------------------------------------------- | ------------------------------------------------ |
172
- | `tanuki_render` | `text, level?, distill?, query?, pack?, font?, codebook?` | PNG page blocks + savings breakdown |
173
- | `tanuki_estimate` | same as render | page geometry, token math, and a `recommend` field: the cheapest safe knob set, priced |
174
- | `tanuki_distill` | `text, query?` | stage 0 alone; output stays greppable text |
175
- | `tanuki_compress` | `text, level` | stage 1 alone |
176
- | `tanuki_stats` | none | savings summary from the session event log |
177
-
178
- `estimate` walks the safe knob ladder server-side (plain, codebook, distill,
179
- both, at level 0) and returns the first rung that holds as `recommend`, with
180
- the tiny-font price alongside. Probing those combos by hand costs about 590
181
- tokens of tool chatter; the walk is one call.
182
-
183
- `estimate` never touches pixel data. Call it on everything; render only when
184
- the verdict says the pipeline wins.
144
+ **Stage 0 — distill.** Built for logs. Repeated lines and repeated
145
+ multi-line blocks collapse to one example plus an exact count.
146
+ Near-duplicates that differ only in timestamps, ids, or numbers fold into a
147
+ template, also counted. Errors and warnings are never touched. Here is the
148
+ whole idea in eleven lines — real output:
149
+
150
+ ```
151
+ input (11 lines): distilled (deterministic):
152
+
153
+ 12:00:01 INFO sync started 12:00:01 INFO sync started
154
+ 12:00:02 INFO uploaded photo_001.jpg ok 12:00:02 INFO uploaded photo_001.jpg ok
155
+ 12:00:02 INFO uploaded photo_002.jpg ok 12:00:02 INFO uploaded photo_002.jpg ok
156
+ 12:00:03 INFO uploaded photo_003.jpg ok 12:00:03 INFO uploaded photo_003.jpg ok
157
+ 12:00:03 INFO uploaded photo_004.jpg ok 12:00:04 ERROR upload failed photo_006.jpg: connection reset
158
+ 12:00:04 INFO uploaded photo_005.jpg ok 12:00:06 INFO sync finished: 8 ok, 1 failed
159
+ 12:00:04 ERROR upload failed photo_006.jpg: connection reset
160
+ 12:00:05 INFO uploaded photo_007.jpg ok ── 5 repeated lines suppressed (0 exact ×N,
161
+ 12:00:05 INFO uploaded photo_008.jpg ok 5 same-template; first occurrences kept above) ──
162
+ 12:00:06 INFO uploaded photo_009.jpg ok ×8 (template) 12:00:02 INFO uploaded photo_001.jpg ok
163
+ 12:00:06 INFO sync finished: 8 ok, 1 failed
164
+ ```
165
+
166
+ The ERROR line survived untouched, the count is exact, and the same input
167
+ always produces the same output. On the 200 KB journal: 1,434 lines to 641,
168
+ half the characters gone, all 310 error/warn lines kept verbatim.
169
+
170
+ **Stage 1 — the ladder.** Five text-compression levels, 0 (off) to 4 (gist
171
+ only). From level 2 up, an exact-recall guard keeps code, identifiers,
172
+ hashes, paths, and indented lines verbatim, so loss is confined to prose.
173
+ Honest consequence: technical files barely shrink, because the guard
174
+ protects most of them. This repo's README loses 10% of its characters at
175
+ level 2 and 14% at level 4. The ladder earns its keep on wordy prose;
176
+ on code it mostly just guards. Level 1 (whitespace cleanup) is lossless.
177
+
178
+ **Stage 2 — imaging.** Text is packed into 312-column pages of 5x8
179
+ antialiased glyphs (1568x728 px max; short pages are width-trimmed) and
180
+ encoded as grayscale PNGs. Anthropic bills images by 28x28-pixel patches:
181
+ a full page is 56x26 = 1,456 tokens carrying up to 28,080 characters —
182
+ about 19 characters per token, where text gets 4. Full Unicode: 92,812
183
+ codepoints, CJK to emoji. Unassigned codepoints become readable `[U+HEX]`
184
+ escapes. Nothing disappears silently.
185
+
186
+ ## The techniques, named
187
+
188
+ | technique | what it does | why it saves |
189
+ | --- | --- | --- |
190
+ | pixel pricing | pages billed by 28-px patch grid, not content | the core ~4x gap text never gets |
191
+ | distill | dedupe with exact counts, errors verbatim | noisy logs are mostly repetition |
192
+ | exact-recall guard | levels 2-4 can't touch code/ids/paths | loss stays confined to prose |
193
+ | pack | single-cell tabs, `⇥N` indent runs, width-trimmed pages | -5% on code, byte-exact |
194
+ | codebook | repeated tokens/paths become 1-cell sigils + a `·legend·` line | -38% on path-heavy logs, reversible |
195
+ | tiny font | glyphs box-filtered to 4x6 cells | -36 to -40% more, opt-in (lossy read-back) |
196
+ | `recommend` | the estimate call walks all safe knob combos server-side | saves ~590 tokens of tool-call probing |
197
+ | proxy dedupe | byte-identical repeats become a one-line pointer | ~1,400 tokens per repeated 30 KB block |
198
+ | append-stable pages | appending text never changes earlier pages | prompt caching keeps pricing them at cache rates |
199
+
200
+ ## Benchmarks
201
+
202
+ Density knobs, measured by `reference/methods-report.mjs` (regenerate it
203
+ yourself with `bun reference/methods-report.mjs`; percentages vs the
204
+ baseline renderer on the same content):
205
+
206
+ | knob | code | prose | log |
207
+ | ------------------- | ---: | ----: | ---: |
208
+ | `pack` (default on) | -5% | 0% | 0% |
209
+ | `codebook` | -9% | 0% | -38% |
210
+ | `font: "tiny"` | -36% | -38% | -40% |
211
+ | all three stacked | **-45%** | **-38%** | **-62%** |
212
+
213
+ Corpora: `src/main.ts` (code), `DESIGN.md` (prose), a path-heavy synthetic
214
+ log. Knobs need something to bite on: codebook needs repetition, pack needs
215
+ indentation.
216
+
217
+ Footprint, measured 2026-07-26 on a Ryzen 7 9700X (median of 5 cold starts;
218
+ the reference server is the original node MCP wrapping pxpipe's library):
219
+
220
+ | metric | node | bun | rust | node ref |
221
+ | ----------------------- | ------- | ------- | ------- | -------- |
222
+ | spawn to first response | 35 ms | 27 ms | 0.4 ms | 158 ms |
223
+ | idle server RSS | 87 MB | 50 MB | 3.8 MB | 177 MB |
224
+ | distill a 12 MB log | 0.42 s | 0.31 s | 0.28 s | - |
225
+ | install | 0.98 MB tarball, zero deps | same | 5.7 MB static binary | node_modules tree |
226
+
227
+ Correctness: 39 TypeScript tests, 23 Rust tests, and a 92-check parity
228
+ harness that holds the two engines to byte-identical JSON and
229
+ pixel-identical PNGs on every knob combination, including distilled
230
+ renders and a full MCP session with error paths.
185
231
 
186
232
  ## Limits, plainly
187
233
 
188
234
  - **Rendering is exact; reading is a model skill.** Pages are pixel-faithful
189
- to pxpipe's production renderer, and pxpipe has receipts on how well models
190
- read them: near-perfect on arithmetic, gist, and state, but 13/15 on exact
191
- 12-char hex strings for claude-fable-5, and misses are silent. Their
192
- numbers: [pxpipe benchmarks](https://github.com/teamchong/pxpipe#benchmark-results-and-receipts).
235
+ to pxpipe's production renderer, and pxpipe has receipts on how well
236
+ models read them: near-perfect on arithmetic, gist, and state, but 13/15
237
+ on exact 12-char hex strings for claude-fable-5 — and misses are silent.
238
+ Their numbers: [pxpipe benchmarks](https://github.com/teamchong/pxpipe#benchmark-results-and-receipts).
193
239
  Keep byte-exact-critical values (secrets, hashes you must retype) in text.
194
240
  - **`font: "tiny"` trades legibility for tokens.** We measured 99.7%
195
- character read-back, and `M`/`H` is the confusable pair. Fine for logs and
196
- prose; skip it when the model must transcribe identifiers exactly.
197
- - **Levels 2-4 reword prose.** The guard keeps code and identifiers verbatim,
198
- but prose comes back paraphrase-shaped. Level 4 is gist only. Do not use it
199
- for anything you may need to quote.
241
+ character read-back; `M`/`H` is the confusable pair. Fine for logs, skip
242
+ it when the model must transcribe identifiers exactly.
243
+ - **Levels 2-4 reword prose.** Level 4 is gist only. Never use it for
244
+ anything you may need to quote.
200
245
  - **Tiny wins are not worth taking.** Width-trimming makes even small pages
201
246
  cheap on paper (12 short lines: 56 image tokens vs 115 as text), but a
202
- 50-token win buys you an image the model has to read back instead of text
203
- it can quote. The proxy gate defaults to at least 25% and 300 tokens saved
204
- for exactly this reason; in tool mode, just leave small things alone.
247
+ 50-token win buys you an image instead of quotable text. The proxy gate
248
+ requires at least 25% and 300 tokens saved for exactly this reason.
249
+
250
+ ## Proxy mode, the fine print
251
+
252
+ We left the proxy model early in this project because rewriting the system
253
+ prompt into a user turn reads exactly like a prompt injection — an agent
254
+ once flagged it as an attack, and it was right to. The proxy came back only
255
+ with rules that remove that failure:
205
256
 
206
- ## Implicit mode (proxy)
257
+ - the system prompt and tool definitions are never touched
258
+ - nothing moves between roles or positions; a block is replaced in place by
259
+ a visible `[tanuki-context: ...]` marker plus its pages
260
+ - the latest message is never imaged (you may need to quote it)
261
+ - blocks carrying `cache_control` are never touched
262
+ - anything that doesn't clearly win forwards byte-for-byte
207
263
 
208
- If you cannot touch the client, tanuki also runs as a local middlebox, the
209
- way pxpipe deploys. We left the proxy model early on because rewriting the
210
- system prompt into a user turn reads exactly like a prompt injection (an
211
- agent once flagged it as an attack, and it was right to). The middlebox
212
- came back with rules that remove that exact failure:
264
+ One reuse rule on top: when the same oversized block appears twice in one
265
+ request (agents re-read files constantly), the second byte-identical copy
266
+ becomes a one-line pointer to the pages above. Exact repeats only.
213
267
 
268
+ Responses stream through untouched. Savings land in
269
+ `~/.pxpipe/events.jsonl`, with the baseline named: what was billed plus
270
+ what the imaged blocks would have cost as text.
271
+
272
+ Knobs: `--port N` `--upstream URL` `--level 0-4` `--distill` `--codebook`
273
+ `--font tiny` `--min-chars N` `--ratio X` `--min-save N` `--max-pages N`.
274
+ Defaults are conservative: level 0, nothing lossy on.
275
+
276
+ ## Clients
277
+
278
+ **OMP (oh-my-pi)** — `~/.omp/agent/mcp.json`, or `.omp/mcp.json` in the project:
279
+
280
+ ```json
281
+ { "mcpServers": { "tanuki-context": { "command": "npx", "args": ["-y", "tanuki-context"] } } }
214
282
  ```
215
- npx tanuki-context proxy # listens on 127.0.0.1:8484
216
- export ANTHROPIC_BASE_URL=http://127.0.0.1:8484
283
+
284
+ **jcode** — `~/.jcode/mcp.json`, or `.jcode/mcp.json` in the project. jcode
285
+ speaks stdio only; `"shared": true` reuses one server across sessions:
286
+
287
+ ```json
288
+ { "mcpServers": { "tanuki-context": { "command": "npx", "args": ["-y", "tanuki-context"], "shared": true } } }
217
289
  ```
218
290
 
219
- Oversized text blocks in user messages and tool results are replaced in
220
- place by a short visible marker plus PNG blocks, only when `estimate` says
221
- imaging wins by a clear margin (default: at least 25% and 300 tokens). What
222
- it never does:
291
+ **pi** — pi has no MCP layer, so this package doubles as a pi extension (the
292
+ `"pi"` manifest field in package.json):
223
293
 
224
- - touch the system prompt or tool definitions
225
- - move content between roles or positions
226
- - image the latest message (you may need to quote it)
227
- - rewrite blocks carrying `cache_control` (that would break their cache)
228
- - rewrite anything when text is cheaper; those requests forward byte-for-byte
294
+ ```
295
+ pi install npm:tanuki-context
296
+ ```
229
297
 
230
- One thing it deliberately reuses: when the same oversized block appears
231
- twice in one request (agents re-read files constantly), the second
232
- byte-identical copy becomes a one-line pointer to the pages above instead
233
- of the same pages again. Exact repeats only; a one-byte difference images
234
- independently.
298
+ **Claude Code**:
235
299
 
236
- Responses stream through untouched. Savings land in `~/.pxpipe/events.jsonl`
237
- (the same file `tanuki_stats` reads), with the baseline named: what Anthropic
238
- billed, plus what the imaged blocks would have cost as text.
300
+ ```
301
+ claude mcp add tanuki-context -- npx -y tanuki-context
302
+ ```
239
303
 
240
- Knobs: `--port N` `--upstream URL` `--level 0-4` `--distill` `--codebook`
241
- `--font tiny` `--min-chars N` `--ratio X` `--min-save N` `--max-pages N`.
242
- Defaults are conservative: level 0, nothing lossy on.
304
+ **Rust instead of Node?** Same pipeline, same numbers, one 5.7 MB static
305
+ binary:
306
+
307
+ ```
308
+ cargo install --git https://github.com/Osyna/tanuki-context --branch rust
309
+ ```
310
+
311
+ Then `"command": "tanuki-context", "args": []` in any snippet above, or
312
+ `TANUKI_BIN=~/.cargo/bin/tanuki-context` for the pi extension.
243
313
 
244
314
  ## Claude Agent SDK
245
315
 
246
316
  `tanuki-context/agent` wires the pipeline into agents built on the
247
317
  [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-typescript).
248
318
 
249
- External (subprocess per session, zero extra dependencies):
319
+ One agent (subprocess per session, zero extra dependencies):
250
320
 
251
321
  ```ts
252
322
  import { query } from "@anthropic-ai/claude-agent-sdk";
@@ -257,7 +327,7 @@ for await (const msg of query({ prompt: task, options: withTanuki({ model: "clau
257
327
  }
258
328
  ```
259
329
 
260
- In-process (one server instance shared by every agent in the process):
330
+ A team of agents (one in-process server shared by all of them):
261
331
 
262
332
  ```ts
263
333
  import { tanukiSdkServer, tanukiAllowedTools, TANUKI_INSTRUCTIONS } from "tanuki-context/agent";
@@ -271,11 +341,11 @@ const options = {
271
341
  // hand the same `options` to every agent in the team
272
342
  ```
273
343
 
274
- `withTanuki(options?)` merges into existing options without clobbering other
275
- servers or tools. `TANUKI_INSTRUCTIONS` is a canned prompt block that teaches
276
- agents the estimate-first workflow and the page decode grammar. The core
277
- package stays zero-dependency: the SDK and zod are optional peers, touched
278
- only inside `tanukiSdkServer()`.
344
+ `withTanuki(options?)` merges into your existing options without clobbering
345
+ other servers or tools. `TANUKI_INSTRUCTIONS` teaches agents the
346
+ estimate-first workflow, the `recommend` shortcut, and the page decode
347
+ grammar. The core package stays zero-dependency: the SDK and zod are
348
+ optional peers, touched only inside `tanukiSdkServer()`.
279
349
 
280
350
  Python SDK, plain dict config:
281
351
 
@@ -289,40 +359,30 @@ options = ClaudeAgentOptions(
289
359
  )
290
360
  ```
291
361
 
292
- ## CLI
362
+ ## CLI reference
293
363
 
294
364
  ```
295
365
  npx tanuki-context # MCP stdio server (default)
296
- npx tanuki-context proxy [--port 8484] [--upstream URL] [knobs] # implicit middlebox
366
+ npx tanuki-context proxy [--port 8484] [--upstream URL] [knobs] # implicit relay
297
367
  npx tanuki-context distill <file> [query] # stats JSON to stdout
298
368
  npx tanuki-context estimate <file> [level] [--distill] [--no-pack] [--font tiny] [--codebook]
299
- npx tanuki-context render <file> [level] [outdir] [--no-pack] [--font tiny] [--codebook]
369
+ npx tanuki-context render <file> [level] [outdir] [--distill] [--no-pack] [--font tiny] [--codebook]
300
370
  npx tanuki-context bench <file> <distill|pipeline> [level] [runs] # in-process timing
301
371
  ```
302
372
 
303
- ## Footprint
304
-
305
- Measured 2026-07-26 on a Ryzen 7 9700X, Linux. Spawn time is the median of 5
306
- cold starts to the first MCP response; the reference server is the original
307
- node MCP that wraps pxpipe's library (kept in `reference/node-mcp/`).
308
-
309
- | metric | node | bun | rust | node ref (pxpipe lib) |
310
- | ----------------------- | ------- | ------- | ------- | --------------------- |
311
- | spawn to first response | 35 ms | 27 ms | 0.4 ms | 158 ms |
312
- | idle server RSS | 87 MB | 50 MB | 3.8 MB | 177 MB |
313
- | distill a 12 MB log | 0.42 s | 0.31 s | 0.28 s | - |
314
- | install | 0.98 MB tarball, zero deps | same | 5.7 MB static binary | node_modules tree |
373
+ The example page above is one command:
374
+ `npx tanuki-context render your.log 0 ./pages --distill`.
315
375
 
316
376
  ## Two engines
317
377
 
318
378
  `main` is this TypeScript package. The [`rust` branch](../../tree/rust) is
319
379
  the same pipeline in Rust: same patch-grid token model, same escapes, same
320
380
  glyph atlas, same proxy rules. `reference/parity-ts.mjs` holds them to
321
- byte-identical JSON and pixel-identical PNGs on every knob combination, plus
322
- a full MCP session including error paths. When one engine changes, the
323
- other follows, or the parity harness fails loudly. The npm packaging and
324
- the Agent SDK / pi glue are TypeScript-only; everything the model sees is
325
- identical.
381
+ byte-identical JSON and pixel-identical PNGs — 92 checks, including
382
+ distilled renders (a check that exists because it once caught the Rust
383
+ engine emitting its dedupe summary in nondeterministic order; fixed, tested,
384
+ and now permanently covered). The npm packaging and the Agent SDK / pi glue
385
+ are TypeScript-only; everything the model sees is identical.
326
386
 
327
387
  The imaging engine is a remake of [pxpipe](https://github.com/teamchong/pxpipe):
328
388
  page geometry and glyphs come from pxpipe's own generated atlas (Spleen 5x8
@@ -344,7 +404,7 @@ unassigned codepoints.
344
404
  | `src/ladder.ts` | stage 1: levels 0-4 with the exact-recall guard |
345
405
  | `src/codebook.ts` | repeated tokens and path prefixes to sigils plus `·legend·` (opt-in) |
346
406
  | `src/render.ts` | stage 2: reflow, pack, wrap, page split, AA blit, tiny 4x6 font |
347
- | `src/proxy.ts` | implicit mode: local Anthropic middlebox, in-place block imaging |
407
+ | `src/proxy.ts` | implicit mode: local Anthropic middlebox, in-place block imaging, dedupe |
348
408
  | `src/atlas.ts` | glyph atlas (92,812 codepoints): metadata eager, pixels inflated lazily |
349
409
  | `src/png.ts` | minimal grayscale PNG encoder (`node:zlib`, filter-0 rows) |
350
410
  | `src/stats.ts` | event log summary |
@@ -363,7 +423,7 @@ Node-compatible files:
363
423
  ```
364
424
  bun run build # dist/cli.js + dist/agent.js + dist/pi.js
365
425
  bun test # 39 tests
366
- bun run parity # TS vs rust binary, byte/pixel-exact (needs TANUKI_BIN)
426
+ bun run parity # TS vs rust binary, 92 checks (needs TANUKI_BIN)
367
427
  ```
368
428
 
369
429
  Regenerating glyphs after a pxpipe atlas rebuild (needs a pxpipe checkout
package/dist/agent.js CHANGED
@@ -65,7 +65,7 @@ class ZJ{value;constructor(J){this.value=J}}function nJ(){let J=process.env.TANU
65
65
  //! tanuki-context estimate <file> [level] [--distill]
66
66
  //! tanuki-context render <file> [level] [outdir]
67
67
  //! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
68
- var cJ="0.2.0",NJ=6;function JZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function C(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof ZJ){let z=J.value;return Number.isFinite(z)&&Number.isInteger(z)?z.toFixed(1):String(z)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let q="[";for(let X=0;X<J.length;X++){if(X>0)q+=",";q+=C(J[X],!1)}return q+"]"}let z=Z+" ",K=`[
68
+ var cJ="0.2.2",NJ=6;function JZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function C(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof ZJ){let z=J.value;return Number.isFinite(z)&&Number.isInteger(z)?z.toFixed(1):String(z)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let q="[";for(let X=0;X<J.length;X++){if(X>0)q+=",";q+=C(J[X],!1)}return q+"]"}let z=Z+" ",K=`[
69
69
  `;for(let q=0;q<J.length;q++){if(q>0)K+=`,
70
70
  `;K+=z+C(J[q],!0,z)}return K+`
71
71
  `+Z+"]"}let V=J,G=Object.keys(V).sort(JZ);if(G.length===0)return"{}";if(!Q){let z="{";for(let K=0;K<G.length;K++){if(K>0)z+=",";z+=JSON.stringify(G[K])+":"+C(V[G[K]],!1)}return z+"}"}let Y=Z+" ",H=`{
package/dist/cli.js CHANGED
@@ -71,7 +71,7 @@ var c={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,min
71
71
  //! tanuki-context estimate <file> [level] [--distill]
72
72
  //! tanuki-context render <file> [level] [outdir]
73
73
  //! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
74
- var NZ="0.2.0",TJ=6;function jZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function E(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof e){let H=J.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let B="[";for(let X=0;X<J.length;X++){if(X>0)B+=",";B+=E(J[X],!1)}return B+"]"}let H=Z+" ",K=`[
74
+ var NZ="0.2.2",TJ=6;function jZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function E(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof e){let H=J.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let B="[";for(let X=0;X<J.length;X++){if(X>0)B+=",";B+=E(J[X],!1)}return B+"]"}let H=Z+" ",K=`[
75
75
  `;for(let B=0;B<J.length;B++){if(B>0)K+=`,
76
76
  `;K+=H+E(J[B],!0,H)}return K+`
77
77
  `+Z+"]"}let V=J,z=Object.keys(V).sort(jZ);if(z.length===0)return"{}";if(!Q){let H="{";for(let K=0;K<z.length;K++){if(K>0)H+=",";H+=JSON.stringify(z[K])+":"+E(V[z[K]],!1)}return H+"}"}let Y=Z+" ",G=`{
@@ -83,7 +83,7 @@ vs ~${H} text-tokens raw = TOTAL -${QJ(H,z)}%`,Z.protectedLines>0)X+=` · ${Z.pr
83
83
  `)}function EZ(){let J=Buffer.alloc(0);S.stdin.on("data",(Q)=>{let Z=J.length>0?Buffer.concat([J,Q]):Q,$=0,V;while((V=Z.indexOf(10,$))!==-1)QQ(Z.toString("utf8",$,V)),$=V+1;J=Z.subarray($)}),S.stdin.on("end",()=>{if(J.length>0)QQ(J.toString("utf8")),J=Buffer.alloc(0)})}function k(J){S.stderr.write(J+`
84
84
  `),S.exit(101)}function UJ(J){try{return FZ(J,"utf8")}catch{k("read file")}}function XJ(J,Q){if(!/^\+?[0-9]+$/.test(J))return null;let Z=BigInt(J.startsWith("+")?J.slice(1):J);return Z>Q?null:Number(Z)}var ZQ=255n,$Q=0xffffffffffffffffn;function YQ(){let J=S.argv.slice(1);switch(J[1]){case"distill":{let Q=J[2]??k("usage: tanuki-context distill <file> [query]"),Z=UJ(Q),$=b(Z,J[3]??null,2);S.stdout.write(E($.stats,!1)+`
85
85
  `);break}case"estimate":{let Q=J[2]??k("usage: tanuki-context estimate <file> [level] [--distill] [--no-pack] [--font tiny] [--codebook]"),Z=UJ(Q),$=J.slice(3).filter((H)=>!H.startsWith("--")),V=$.length>0?XJ($[0],$Q)??0:0,z=J.indexOf("--font"),Y=z!==-1&&J[z+1]!==void 0?J[z+1]:"normal",G=GQ({text:Z,level:V,distill:J.includes("--distill"),pack:!J.includes("--no-pack"),font:Y,codebook:J.includes("--codebook")});S.stdout.write(E(G,!1)+`
86
- `);break}case"render":{let Q=J[2]??k("usage: tanuki-context render <file> [level] [outdir] [--no-pack] [--font tiny] [--codebook]"),Z=UJ(Q),$=J.slice(3).filter((O)=>!O.startsWith("--")),V=$.length>0?XJ($[0],ZQ)??0:0,z=!J.includes("--no-pack"),Y=J.includes("--codebook"),G=J.indexOf("--font"),H=g(G!==-1&&J[G+1]!==void 0?J[G+1]:"normal"),K=BJ(Z,V,!1,null,Y),B=p(K.compressed,!0,z,H),X=B.tokens;S.stdout.write(E({pages:B.pages.length,imageTokens:X,dropped:B.dropped,rawTextTokens:JJ(h(Z))},!1)+`
86
+ `);break}case"render":{let Q=J[2]??k("usage: tanuki-context render <file> [level] [outdir] [--distill] [--no-pack] [--font tiny] [--codebook]"),Z=UJ(Q),$=J.slice(3).filter((O)=>!O.startsWith("--")),V=$.length>0?XJ($[0],ZQ)??0:0,z=!J.includes("--no-pack"),Y=J.includes("--codebook"),G=J.indexOf("--font"),H=g(G!==-1&&J[G+1]!==void 0?J[G+1]:"normal"),K=BJ(Z,V,J.includes("--distill"),null,Y),B=p(K.compressed,!0,z,H),X=B.tokens;S.stdout.write(E({pages:B.pages.length,imageTokens:X,dropped:B.dropped,rawTextTokens:JJ(h(Z))},!1)+`
87
87
  `);let U=$[1];if(U!==void 0){try{MZ(U,{recursive:!0})}catch{k("mkdir")}for(let O=0;O<B.pages.length;O++)try{WZ(`${U}/page${O}.png`,B.pages[O].png)}catch{k("write png")}}break}case"bench":{let Q=J[2]??k("usage: tanuki-context bench <file> <op> [level] [runs] [--distill]"),Z=J[3]??"pipeline",$=J[4]!==void 0?XJ(J[4],ZQ)??0:0,V=J[5]!==void 0?XJ(J[5],$Q)??3:3,z=J.includes("--distill"),Y=UJ(Q),G=[],H=null;for(let K=0;K<=V;K++){let B=performance.now();if(Z==="distill")H=b(Y,null,2).stats;else{let X=BJ(Y,$,z,null,!1),U=p(X.compressed,!0,!1,"normal");H={pages:U.pages.length,imageTokens:U.tokens,stage1Chars:h(X.compressed),dropped:U.dropped}}if(K>0)G.push(performance.now()-B)}if(G.sort((K,B)=>K-B),G.length===0)k("index out of bounds: the len is 0 but the index is 0");S.stdout.write(E({medianMs:new e(G[Math.floor(G.length/2)]),runs:V,result:H},!1)+`
88
88
  `);break}case"proxy":{let Q=(V,z)=>{let Y=J.indexOf(V);if(Y===-1||J[Y+1]===void 0)return z;let G=Number(J[Y+1]);return Number.isFinite(G)?G:z},Z=J.indexOf("--upstream"),$=J.indexOf("--font");JQ({port:Q("--port",8484),upstream:Z!==-1&&J[Z+1]!==void 0?J[Z+1]:S.env.TANUKI_UPSTREAM??"https://api.anthropic.com",level:Q("--level",c.level),distill:J.includes("--distill"),codebook:J.includes("--codebook"),font:g($!==-1&&J[$+1]!==void 0?J[$+1]:"normal"),minChars:Q("--min-chars",c.minChars),ratio:Q("--ratio",c.ratio),minSave:Q("--min-save",c.minSave),maxPages:Q("--max-pages",c.maxPages)});break}case"serve":case void 0:EZ();break;default:S.stderr.write(`unknown command: ${J[1]}
89
89
  usage: tanuki-context [serve|proxy|distill|estimate|render] ...
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tanuki-context",
3
- "version": "0.2.0",
3
+ "version": "0.2.2",
4
4
  "description": "Token-cutting context pipeline as an MCP server: distill logs, compress text, render dense PNG pages (pxpipe imaging engine). Zero dependencies.",
5
5
  "license": "MIT",
6
6
  "type": "module",