tanuki-context 0.3.0 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +164 -11
- package/dist/agent.js +25 -25
- package/dist/cli.js +45 -41
- package/dist/pi.js +1 -1
- package/package.json +2 -1
- package/skills/tanuki-context/SKILL.md +60 -0
package/README.md
CHANGED
|
@@ -73,20 +73,79 @@ No AI client at all, just curious what it would save you:
|
|
|
73
73
|
npx tanuki-context estimate some-big-file.log 0
|
|
74
74
|
```
|
|
75
75
|
|
|
76
|
-
##
|
|
76
|
+
## Installing it: two ways, same tools
|
|
77
|
+
|
|
78
|
+
**npm (the default).** `npx -y tanuki-context` downloads a 0.98 MB tarball
|
|
79
|
+
with zero dependencies and starts the MCP server on stdio. That is the
|
|
80
|
+
whole install. Your client spawns one short-lived process per session
|
|
81
|
+
(35 ms to first response under node, 27 under bun); the model calls the
|
|
82
|
+
seven `tanuki_*` tools; pages come back inline as PNG blocks. On disk it
|
|
83
|
+
touches exactly two places, both yours to delete: `~/.tanuki/stash` for
|
|
84
|
+
stashed text and `~/.pxpipe/events.jsonl` for the savings log.
|
|
85
|
+
|
|
86
|
+
**cargo (the static binary).** No node at all:
|
|
87
|
+
|
|
88
|
+
```
|
|
89
|
+
cargo install --git https://github.com/Osyna/tanuki-context --branch rust
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
One 5.7 MB binary, same seven tools, same numbers — a 103-check parity
|
|
93
|
+
harness holds the two engines to byte-identical JSON and pixel-identical
|
|
94
|
+
PNGs on every release. It spawns in 0.4 ms and idles at 3.8 MB RSS, which
|
|
95
|
+
matters when every session forks its own server. Point any client config
|
|
96
|
+
at `"command": "tanuki-context"` instead of the npx line and nothing else
|
|
97
|
+
changes.
|
|
98
|
+
|
|
99
|
+
## Should you use it at all?
|
|
100
|
+
|
|
101
|
+
Reach for tanuki when:
|
|
102
|
+
|
|
103
|
+
- **bulky text is about to enter context** — logs, build output, command
|
|
104
|
+
results, long docs, anything past roughly 2,000 tokens;
|
|
105
|
+
- **an agent keeps re-reading big files** — the proxy dedupes exact
|
|
106
|
+
repeats, and stash turns re-reads into cheap fetches;
|
|
107
|
+
- **a long session is running out of context** — pages carry ~4x more
|
|
108
|
+
characters per token, and the proxy can image old bulky turns in place;
|
|
109
|
+
- **you cannot change the client** — the proxy mode needs only a base-URL
|
|
110
|
+
export.
|
|
111
|
+
|
|
112
|
+
Leave it alone when:
|
|
113
|
+
|
|
114
|
+
- **your model cannot read images.** Hard requirement. Everything here
|
|
115
|
+
assumes a vision-capable reader (any current Claude model qualifies).
|
|
116
|
+
- **the exact bytes must survive a round trip** — secrets, hashes, code
|
|
117
|
+
the model is about to edit. Rendering is exact, but model read-back of
|
|
118
|
+
dense random strings is not guaranteed (pxpipe measured 13/15 on
|
|
119
|
+
12-char hex). Keep those in text.
|
|
120
|
+
- **the content is small.** A 500-token snippet is not worth a modality
|
|
121
|
+
switch even when the math technically favors it; `estimate` and the
|
|
122
|
+
proxy gate both say so.
|
|
123
|
+
- **your bill is output-dominated.** tanuki cuts input tokens only. If
|
|
124
|
+
most of your spend is the model's own output, fix that first.
|
|
125
|
+
- **you are not on Anthropic pricing.** Verdicts use Anthropic's
|
|
126
|
+
28-px patch grid. OpenAI and Google price images differently (tiles,
|
|
127
|
+
fixed per-image rates), so every verdict needs re-deriving before you
|
|
128
|
+
trust it there.
|
|
129
|
+
- **the bulk is already prompt-cached.** Cache reads cost a tenth of
|
|
130
|
+
fresh input; imaging content that was riding the cache can be a net
|
|
131
|
+
loss. The proxy never touches `cache_control` blocks for this reason.
|
|
132
|
+
|
|
133
|
+
## The three ways to run it
|
|
77
134
|
|
|
78
135
|
**Explicit (MCP tools) — the default and the recommendation.** Your AI gets
|
|
79
|
-
|
|
80
|
-
renders only when the pipeline wins. The model stays in charge and can
|
|
81
|
-
exactly what happened to every byte.
|
|
82
|
-
|
|
83
|
-
|
|
136
|
+
seven tools. It calls `tanuki_estimate` on bulky text, reads the verdict,
|
|
137
|
+
and renders only when the pipeline wins. The model stays in charge and can
|
|
138
|
+
see exactly what happened to every byte. The estimate answer prices the
|
|
139
|
+
knobs for you — reversible ones as the headline, the lossy-but-counted
|
|
140
|
+
distill route separately, because "cheapest" and "safe" are different
|
|
141
|
+
claims:
|
|
84
142
|
|
|
85
143
|
```
|
|
86
144
|
npx tanuki-context estimate journal.log 0
|
|
87
145
|
# { "imageTokens": 10752, "verdict": "PIPELINE cheaper",
|
|
88
|
-
# "recommend": { "
|
|
89
|
-
# "
|
|
146
|
+
# "recommend": { "codebook": true, "imageTokens": 8624, "pages": 6,
|
|
147
|
+
# "tinyImageTokens": 5208,
|
|
148
|
+
# "withDistill": { "codebook": true, "imageTokens": 4256 } }, ... }
|
|
90
149
|
```
|
|
91
150
|
|
|
92
151
|
**Implicit (proxy) — for clients you can't modify.** A small local relay.
|
|
@@ -100,7 +159,27 @@ npx tanuki-context proxy # listens on 127.0.0.1:8484
|
|
|
100
159
|
export ANTHROPIC_BASE_URL=http://127.0.0.1:8484
|
|
101
160
|
```
|
|
102
161
|
|
|
103
|
-
|
|
162
|
+
**The `run` wrapper — for the tokens that burn in your terminal.**
|
|
163
|
+
[rtk](https://github.com/rtk-ai/rtk) proved that the cheapest place to cut
|
|
164
|
+
agent tokens is command output, before it ever reaches the model, and does
|
|
165
|
+
it with per-command parsers for a hundred-plus tools. tanuki adopts the
|
|
166
|
+
shape with its own generic machinery: wrap any command, get the distilled
|
|
167
|
+
output inline (repeats counted, errors verbatim, progress bars collapsed to
|
|
168
|
+
their final frame), and — when the output is huge — the full capture parked
|
|
169
|
+
in the stash. Measured, verbatim:
|
|
170
|
+
|
|
171
|
+
```
|
|
172
|
+
tanuki-context run -- npm install --loglevel silly typescript
|
|
173
|
+
# [tanuki run] exit 0 · 137 -> 54 lines · 70% of chars removed
|
|
174
|
+
# ... errors verbatim, ×N counts, the stash pointer when output is big ...
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Exit codes pass through untouched, so agents and scripts can wrap blindly.
|
|
178
|
+
If you want rtk's hand-tuned per-command output shapes, use rtk itself —
|
|
179
|
+
the two stack fine (rtk for the commands it knows, `run` for everything
|
|
180
|
+
else plus the stash escape hatch).
|
|
181
|
+
|
|
182
|
+
All modes run the same engine and log their savings to the same file, which
|
|
104
183
|
`tanuki_stats` summarizes.
|
|
105
184
|
|
|
106
185
|
## What people actually use it for
|
|
@@ -238,6 +317,56 @@ escapes. Nothing disappears silently.
|
|
|
238
317
|
| proxy dedupe | byte-identical repeats become a one-line pointer | ~1,400 tokens per repeated 30 KB block |
|
|
239
318
|
| append-stable pages | appending text never changes earlier pages | prompt caching keeps pricing them at cache rates |
|
|
240
319
|
| stash + fetch | park text on disk, return a distill map + content-address id | retrieval economics with awareness and imaged slices |
|
|
320
|
+
| `run` wrapper | wrap any command; distilled output inline, full capture stashed | -70% of chars on chatty commands, before tokenization |
|
|
321
|
+
| progress-frame collapse | `\r` spinner frames reduce to what the terminal showed | build/download logs stop paying per frame |
|
|
322
|
+
|
|
323
|
+
## What wins where
|
|
324
|
+
|
|
325
|
+
Six kinds of content, measured 2026-07-26 on this machine. All corpora are
|
|
326
|
+
real: a resampled system journal, a fresh `cargo build -v` of a crate with
|
|
327
|
+
three dependencies, an `npm install --loglevel silly`, this repo's own
|
|
328
|
+
TypeScript sources and docs, and a `journalctl -o json` slice. Identifiers
|
|
329
|
+
were rewritten to placeholders first; repetition and structure untouched.
|
|
330
|
+
Numbers are tokens; "reversible" = pack + codebook only (byte-exact or
|
|
331
|
+
legend-decodable), "distill route" = repeats collapsed with exact counts
|
|
332
|
+
(honest for logs, wrong for code), "tiny" = the 4x6 font on top of the
|
|
333
|
+
reversible pick (99.7% glyph read-back).
|
|
334
|
+
|
|
335
|
+
| corpus | raw text | caveman (L4 text) | pxpipe export | tanuki reversible | tanuki distill route | tanuki tiny |
|
|
336
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
337
|
+
| system journal, 200 KB | 51,200 | 51,198 | 11,966 | 8,624 | **4,256** | 5,208 |
|
|
338
|
+
| cargo build -v, 21 KB | 5,413 | 5,413 | 4,402 | **784** | 784 | 448 |
|
|
339
|
+
| npm install silly, 12 KB | 3,131 | 3,128 | 1,922 | 504 | **224** | 336 |
|
|
340
|
+
| TypeScript source, 111 KB | 28,349 | 28,024 | 53,554 † | **5,656** | 4,592 ‡ | 3,416 |
|
|
341
|
+
| markdown docs, 51 KB | 12,909 | 11,797 | 14,139 † | **2,688** | 2,632 ‡ | 1,624 |
|
|
342
|
+
| journalctl JSON, 200 KB | 51,200 | 51,200 | 12,129 | **10,696** | 7,840 | 6,384 |
|
|
343
|
+
|
|
344
|
+
† pxpipe's own CLI prints the warning here ("70.5% more expensive" on the
|
|
345
|
+
source corpus) and its proxy would refuse to image these — its export flow
|
|
346
|
+
preserves layout, so deeply indented content renders sparse. The gap is
|
|
347
|
+
tanuki's `pack` step collapsing indentation, not the engine (it is the same
|
|
348
|
+
engine).
|
|
349
|
+
‡ cheaper than the reversible route, but distill collapses similar-looking
|
|
350
|
+
lines — fine for logs, not for code or docs you want intact. That is why
|
|
351
|
+
`recommend` prices it separately instead of calling it safe.
|
|
352
|
+
|
|
353
|
+
The same table as advice:
|
|
354
|
+
|
|
355
|
+
| your content | use | expect |
|
|
356
|
+
| --- | --- | --- |
|
|
357
|
+
| noisy logs (journal, CI, install) | `distill: true`, add `codebook` for path-heavy ones | **-90 to -93%** |
|
|
358
|
+
| build/tool output, live | `tanuki-context run -- <cmd>` | -70% of chars before tokens even enter |
|
|
359
|
+
| source code | reversible knobs only (`pack` is on by default) | **-80%**, byte-exact |
|
|
360
|
+
| docs / prose | reversible knobs; tiny font if read-back risk is fine | -79 to -87% |
|
|
361
|
+
| structured JSON | reversible + codebook | -79% |
|
|
362
|
+
| "I'll only need two slices of this" | `tanuki_stash` + `tanuki_fetch` | ~300-token map, slices from ~112 |
|
|
363
|
+
| "one narrow answer, never the file" | [context-mode](https://www.npmjs.com/package/context-mode)-style retrieval, or a stash you never fetch from | ~270/question |
|
|
364
|
+
| wordy prose that must stay text | ladder levels 2-4 (caveman) | -9% on real docs; it is the weakest tool here |
|
|
365
|
+
|
|
366
|
+
Caveman-style text compression losing every matchup it enters is our own
|
|
367
|
+
level 4 — we ship it, and the guard that makes it safe on code is exactly
|
|
368
|
+
what makes it a no-op there. Text-side rewording just does not compete
|
|
369
|
+
with pixel pricing.
|
|
241
370
|
|
|
242
371
|
## Benchmarks
|
|
243
372
|
|
|
@@ -266,7 +395,7 @@ the reference server is the original node MCP wrapping pxpipe's library):
|
|
|
266
395
|
| distill a 12 MB log | 0.42 s | 0.31 s | 0.28 s | - |
|
|
267
396
|
| install | 0.98 MB tarball, zero deps | same | 5.7 MB static binary | node_modules tree |
|
|
268
397
|
|
|
269
|
-
Correctness:
|
|
398
|
+
Correctness: 48 TypeScript tests, 36 Rust tests, and a 103-check parity
|
|
270
399
|
harness that holds the two engines to byte-identical JSON and
|
|
271
400
|
pixel-identical PNGs on every knob combination, including distilled
|
|
272
401
|
renders and a full MCP session with error paths.
|
|
@@ -389,6 +518,20 @@ pi install npm:tanuki-context
|
|
|
389
518
|
claude mcp add tanuki-context -- npx -y tanuki-context
|
|
390
519
|
```
|
|
391
520
|
|
|
521
|
+
The package also ships a **skill** — a short instruction file that teaches
|
|
522
|
+
the model the whole workflow (estimate first, read `recommend`, stash big
|
|
523
|
+
references, the page decode grammar, the do-nots) without you prompting
|
|
524
|
+
any of it. Install it once and the tools get used correctly on their own:
|
|
525
|
+
|
|
526
|
+
```
|
|
527
|
+
npm i -g tanuki-context # or any install that gives you the package on disk
|
|
528
|
+
cp -r "$(npm root -g)/tanuki-context/skills/tanuki-context" ~/.claude/skills/
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
From a checkout it is just `cp -r skills/tanuki-context ~/.claude/skills/`.
|
|
532
|
+
The same file works for any skill-aware harness (OMP picks it up from the
|
|
533
|
+
same directory).
|
|
534
|
+
|
|
392
535
|
**Rust instead of Node?** Same pipeline, same numbers, one 5.7 MB static
|
|
393
536
|
binary:
|
|
394
537
|
|
|
@@ -458,6 +601,7 @@ npx tanuki-context render <file> [level] [outdir] [--distill] [--no-pack] [--fon
|
|
|
458
601
|
npx tanuki-context bench <file> <distill|pipeline> [level] [runs] # in-process timing
|
|
459
602
|
npx tanuki-context stash <file> # park text, print the map + id
|
|
460
603
|
npx tanuki-context fetch <id> [outdir] [--query re] [--lines a-b]
|
|
604
|
+
npx tanuki-context run [--query re] -- <command> [args...] # rtk-style wrapper
|
|
461
605
|
```
|
|
462
606
|
|
|
463
607
|
The example page above is one command:
|
|
@@ -512,7 +656,7 @@ Node-compatible files:
|
|
|
512
656
|
|
|
513
657
|
```
|
|
514
658
|
bun run build # dist/cli.js + dist/agent.js + dist/pi.js
|
|
515
|
-
bun test #
|
|
659
|
+
bun test # 48 tests
|
|
516
660
|
bun run parity # TS vs rust binary, 103 checks (needs TANUKI_BIN)
|
|
517
661
|
```
|
|
518
662
|
|
|
@@ -535,6 +679,15 @@ PXPIPE_DIST=~/Projects/pxpipe/dist node tools/gen-glyphs.mjs
|
|
|
535
679
|
are the read-back evidence this README leans on. If you want the
|
|
536
680
|
whole-bill transparent proxy with per-model profiles and eval receipts,
|
|
537
681
|
use pxpipe itself; the two compose fine.
|
|
682
|
+
- [rtk](https://github.com/rtk-ai/rtk) is where the `run` wrapper shape
|
|
683
|
+
comes from: cut command output before the model reads it. rtk does it
|
|
684
|
+
with hand-tuned parsers for 100+ commands; tanuki's `run` uses its
|
|
685
|
+
generic distill plus the stash escape hatch. Different tools, same
|
|
686
|
+
instinct, and they stack.
|
|
687
|
+
- [context-mode](https://www.npmjs.com/package/context-mode) is where the
|
|
688
|
+
stash pattern comes from: content parked outside the window, queried on
|
|
689
|
+
demand. `tanuki_stash`/`tanuki_fetch` is that idea with a map up front
|
|
690
|
+
and imaged slices on the way back.
|
|
538
691
|
- The bitmap fonts inside the atlas are
|
|
539
692
|
[Spleen](https://github.com/fcambus/spleen) 5x8 by Frederic Cambus and
|
|
540
693
|
[GNU Unifont](https://unifoundry.com/unifont/) (BMP coverage plus
|
package/dist/agent.js
CHANGED
|
@@ -1,19 +1,19 @@
|
|
|
1
|
-
import{createRequire as KQ}from"node:module";var EJ=KQ(import.meta.url);import{existsSync as
|
|
2
|
-
·legend· `;for(let
|
|
3
|
-
`),Y=G.length,H=Y,z=Array(Y),K=Array(Y),
|
|
4
|
-
`),AJ=CJ.byteLength(J),_J=CJ.byteLength(PJ),zJ=(1-_J/AJ)*100,YQ=J.length===0?0:zJ<0?-Math.round(-zJ):Math.round(zJ),HQ=k.map(([
|
|
5
|
-
`),G=Array(V.length);for(let H=0;H<V.length;H++){let z=V[H],K=z.length;while(K>0){let
|
|
1
|
+
import{createRequire as KQ}from"node:module";var EJ=KQ(import.meta.url);import{existsSync as OZ}from"node:fs";import{fileURLToPath as UZ}from"node:url";function BQ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function KJ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}Q++}return Q}function XQ(J,Q){let Z=0,$=0;while(Z<J.length&&$<Q.length){let V=J.codePointAt(Z),G=Q.codePointAt($);if(V!==G)return V-G;Z+=V>65535?2:1,$+=G>65535?2:1}return J.length-Z-(Q.length-$)}function t(J){let Q=new Map,Z=(X)=>{Q.set(X,(Q.get(X)??0)+1)},$=(X)=>{if(KJ(X)>=12)Z(X);if(X.includes("/")){let q=X.split("/"),D="";for(let M=0;M<q.length;M++){if(M>0)D+="/";if(D+=q[M],M>=2){let R=D+"/";if(KJ(R)>=12)Z(R)}}}},V=-1;for(let X=0;X<J.length;){let q=J.codePointAt(X),D=q>65535?2:1;if(BQ(q)){if(V>=0)$(J.slice(V,X)),V=-1}else if(V<0)V=X;X+=D}if(V>=0)$(J.slice(V));let G=[];for(let[X,q]of Q)if(q>=3){let D=KJ(X);G.push({k:X,c:q,len:D,saved:(D-1)*q})}G.sort((X,q)=>q.saved-X.saved||XQ(X.k,q.k));let Y=[];for(let X of"§¤¢£¥µ¶ª°±¬×÷ØÞßæðøþ¡¿")if(!J.includes(X))Y.push(X);let H=[],z=[];for(let{k:X,c:q,len:D}of G){if(H.length>=Y.length)break;if((D-1)*q<=D+3)continue;let M=!1;for(let P=0;P<z.length;P++){let T=z[P];if(T.startsWith(X)||X.startsWith(T)){M=!0;break}}if(M)continue;let R=Y[H.length];z.push(X),H.push({sig:R,val:X,len:D})}if(H.length===0)return{text:J,entries:0};let K=H.map((X,q)=>q);K.sort((X,q)=>H[q].len-H[X].len);let O=J;for(let X=0;X<K.length;X++){let{sig:q,val:D}=H[K[X]];O=O.replaceAll(D,q)}let B=`
|
|
2
|
+
·legend· `;for(let X=0;X<H.length;X++)B+=H[X].sig+"="+H[X].val+" ";return O+=B.slice(0,-1),{text:O,entries:H.length}}import{Buffer as CJ}from"node:buffer";var OQ=/\x1b\[[0-9;]*[A-Za-z]/g,BJ=/\b([0-9A-Za-z_]*(error|exception)s?|err|warn(ing)?s?|fail(s|ed|ure|ures)?|panic(s|ked)?|fatal|critical|traceback|denied|refused|timeouts?|timed.?out|assert(s|ed|ion|ions)?|segfault(s|ed)?)\b/i,UQ=/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/g,DQ=/\b[0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?\b/g,qQ=/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,MQ=/\b[0-9a-f]{7,64}\b/gi,FQ=/\b[0-9]+(\.[0-9]+)?[ \t\u00a0]?(ms|us|µs|ns|s|m|h|%|[KMGT]i?B)?\b/gi,WQ=/[0-9a-f]/i,jQ=3,NQ=8,XJ=2,SJ=40;function IQ(J){if(!WQ.test(J))return J;let Q=J.replace(UQ,"<ts>");return Q=Q.replace(DQ,"<time>"),Q=Q.replace(qQ,"<uuid>"),Q=Q.replace(MQ,"<hex>"),Q.replace(FQ,"<n>")}function m(J){if(J===32)return!0;if(J<9)return!1;if(J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function p(J){let Q=0,Z=J.length;while(Q<Z&&m(J.charCodeAt(Q)))Q++;while(Z>Q&&m(J.charCodeAt(Z-1)))Z--;return Q===0&&Z===J.length?J:J.slice(Q,Z)}function RQ(J,Q){let Z=0,$=J.length;while(Z<$&&m(J.charCodeAt(Z)))Z++;while($>Z&&m(J.charCodeAt($-1)))$--;let V=0;for(let G=Z;G<$&&V<Q;V++){let Y=J.charCodeAt(G);G+=Y>=55296&&Y<=56319&&G+1<$?2:1}return V<Q}function PQ(J){let Q=0,Z=J.length;while(Q<Z&&m(J.charCodeAt(Q)))Q++;return J.charCodeAt(Q)===91&&J.charCodeAt(Q+1)===215}function AQ(J){let Q=[],Z=-1,$=!0,V=!1,G=!1,Y=!0;for(let H=0;H<J.length;){let z=J.codePointAt(H),K=z>65535?2:1,O=m(z);if(Y){if(Y=!1,O)Q.push("<v>")}if(O){if(Z>=0)Q.push($?J.slice(Z,H):"<v>"),Z=-1;G=!0}else{if(V=!0,G=!1,Z<0)Z=H,$=!0;if($){let B=z|32;if(B<97||B>122||z>127)$=!1}}H+=K}if(Z>=0)Q.push($?J.slice(Z):"<v>");if(V&&G)Q.push("<v>");return Q.join(" ")}function n(J,Q){let Z=0,$=0,V=J.length;while(Z<V&&$<Q){let G=J.charCodeAt(Z);Z+=G>=55296&&G<=56319&&Z+1<V?2:1,$++}return Z>=V?J:J.slice(0,Z)}var _Q=/[.*+?^${}()|[\]\\]/g;function b(J,Q=null,Z=2){let $=Z,G=J.replace(OQ,"").split(`
|
|
3
|
+
`).map((U)=>{let F=U.endsWith("\r")?U.slice(0,-1):U,W=F.lastIndexOf("\r");return W===-1?F:F.slice(W+1)}),Y=G.length,H=Y,z=Array(Y),K=Array(Y),O=0;for(let U=0;U<Y;U++){z[U]=IQ(G[U]);let F=BJ.test(G[U]);if(K[U]=F,F)O++}let B=[],X=[],q=[],D=0,M=0;while(M<Y){let U=0,F=0;for(let W=1;W<=NQ&&M+2*W<=Y;W++){if(K[M+W-1])break;let N=1;J:for(;;){let j=M+N*W;if(j+W>Y)break;for(let _=0;_<W;_++)if(K[j+_]||z[j+_]!==z[M+_])break J;N++}if(N>=jQ&&W*(N-1)>U*(F>0?F-1:0))U=W,F=N}if(U>0){for(let W=M;W<M+U;W++)B.push(G[W]),X.push(K[W]),q.push(z[W]);B.push(U===1?` [×${F} similar]`:` [×${F} similar ${U}-line blocks]`),X.push(!1),q.push(null),D++,M+=U*F}else B.push(G[M]),X.push(K[M]),q.push(z[M]),M++}let R=new Map,P=new Map,T=XJ+1,E=[],A=[],y=0,w=0;for(let U=0;U<B.length;U++){let F=B[U],W=X[U];if(W||PQ(F)||RQ(F,4)){E.push(F),A.push(W);continue}let N=q[U],j=R.get(N);if(j!==void 0){if(j.count++,j.count<=XJ)E.push(F),A.push(!1);else y++;continue}let _=AQ(N);R.set(N,{count:1,exemplar:F});let l=P.get(_);if(l!==void 0)if(l.count++,l.count<=T)E.push(F),A.push(!1);else w++;else P.set(_,{count:1,exemplar:F}),E.push(F),A.push(!1)}let k=[];for(let U of R.values())if(U.count>XJ)k.push([U.count,"exact",U.exemplar]);for(let U of P.values())if(U.count>T)k.push([U.count,"template",U.exemplar]);if(k.sort((U,F)=>F[0]-U[0]),k.length>SJ)k.length=SJ;if(y+w>0){let U=`── ${y+w} repeated lines suppressed (${y} exact ×N, ${w} same-template; first occurrences kept above) ──`;E.push(U),A.push(BJ.test(U));for(let[F,W,N]of k){let _=` ×${F}${W==="template"?" (template)":""} ${n(p(N),160)}`;E.push(_),A.push(BJ.test(_))}}let f;if(Q!=null){let U;try{U=new RegExp(Q,"i")}catch{U=new RegExp(Q.replace(_Q,"\\$&"),"i")}let F=E.length,W=new Uint8Array(F);for(let j=0;j<F;j++)if(A[j]||U.test(E[j])){let _=j>$?j-$:0,l=Math.min(j+$,F-1);W.fill(1,_,l+1)}f=[];let N=0;for(let j=0;j<F;j++)if(W[j]){if(N>0)f.push(`… ${N} lines omitted`),N=0;f.push(E[j])}else N++;if(N>0)f.push(`… ${N} lines omitted`)}else f=E;let PJ=f.join(`
|
|
4
|
+
`),AJ=CJ.byteLength(J),_J=CJ.byteLength(PJ),zJ=(1-_J/AJ)*100,YQ=J.length===0?0:zJ<0?-Math.round(-zJ):Math.round(zJ),HQ=k.map(([U,F,W])=>({count:U,exemplar:n(p(W),160),kind:F})),zQ={collapsedRuns:D,importantKept:O,origChars:AJ,origLines:H,outChars:_J,outLines:f.length,query:Q??null,savedPct:YQ,suppressedLines:y,templateSuppressed:w,topRepeats:HQ};return{distilled:PJ,stats:zQ}}var s=[["none","none","passthrough (baseline)"],["whitespace","lossless","trailing whitespace + blank-line runs collapsed; safe for code"],["prose","light","L1 + prose lines: collapse spaces, cut redundant filler phrases (code/IDs protected)"],["dense","medium","L2 + prose: drop articles & intensifiers"],["caveman","heavy","L3 + prose: telegraphic — drop function words; gist only, NOT verbatim"]];var C=(J)=>new RegExp(`(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:${J})(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])`,"giu"),OJ=[[C("in order to"),"to"],[C("due to the fact that"),"because"],[C("at this point in time"),"now"],[C("in the event that"),"if"],[C("for the purpose of"),"for"],[C("with regard to"),"about"],[C("a large number of"),"many"],[C("it is important to note that"),""],[C("please note that"),""],[C("as a matter of fact"),""],[C("in terms of"),"for"],[C("the fact that"),"that"]],EQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:the|an|a)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),CQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:very|really|just|actually|basically|simply|quite|rather|essentially|literally)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),SQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:is|are|was|were|am|be|been|being|do|does|did|have|has|had|will|would|shall|should|can|could|may|might|of|to|in|on|at|for|with|that|this|these|those|it|its|there|here)(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]*","giu"),LJ=/ {2,}/g,LQ=new RegExp("[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+([.,;:!?])","gu"),TQ=/\n{3,}/g;function UJ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function yQ(J){let Q=0,Z=J.length;while(Q<Z){let $=J.codePointAt(Q);if(!UJ($))break;Q+=$>65535?2:1}while(Z>Q){let $=J.charCodeAt(Z-1);if(!UJ($))break;Z--}return Q===0&&Z===J.length?J:J.slice(Q,Z)}function wQ(J){if(J.length===0)return!1;let Q=J.charCodeAt(0);if(Q===32||Q===9)return!0;let Z=0,$=0,V=0,G=!1;for(let Y=0;Y<J.length;){let H=J.codePointAt(Y);if(Y+=H>65535?2:1,Z++,UJ(H)){V=0;continue}if(V++,V>=24)G=!0;if(H>=48&&H<=57||H>=65&&H<=90||H>=97&&H<=122)continue;switch(H){case 46:case 44:case 59:case 58:case 39:case 34:case 33:case 63:case 40:case 41:case 45:continue}$++}if($/Z>0.3)return!0;return G}function bQ(J,Q){let Z=J.replace(LJ," ");for(let V=0;V<OJ.length;V++)Z=Z.replace(OJ[V][0],OJ[V][1]);if(Q>=3)Z=Z.replace(EQ,""),Z=Z.replace(CQ,"");if(Q>=4)Z=Z.replace(SQ,"");Z=Z.replace(LJ," "),Z=yQ(Z.replace(LQ,"$1"));let $=Z.charCodeAt(0);if($>=97&&$<=122)Z=String.fromCharCode($-32)+Z.slice(1);return Z}function e(J,Q){let Z=Math.min(Q,4);if(Z===0)return{compressed:J,protectedLines:0,level:Z};let $=0,V=J.split(`
|
|
5
|
+
`),G=Array(V.length);for(let H=0;H<V.length;H++){let z=V[H],K=z.length;while(K>0){let B=z.charCodeAt(K-1);if(B!==32&&B!==9)break;K--}let O=K===z.length?z:z.slice(0,K);if(Z===1){G[H]=O;continue}if(wQ(O)){$++,G[H]=O;continue}G[H]=bQ(O,Z)}return{compressed:G.join(`
|
|
6
6
|
`).replace(TQ,`
|
|
7
7
|
|
|
8
|
-
`),protectedLines:$,level:Z}}import{readFileSync as
|
|
8
|
+
`),protectedLines:$,level:Z}}import{readFileSync as qJ}from"node:fs";import{inflateSync as vQ}from"node:zlib";//! Full-BMP glyph atlas (Spleen 5x8 for ASCII/code + Unifont fallback),
|
|
9
9
|
//! extracted from pxpipe's generated gray atlas by `tools/gen-glyphs.mjs`.
|
|
10
10
|
//!
|
|
11
11
|
//! Codepoints + wide flags load eagerly (~175 KB, needed for wrap math);
|
|
12
12
|
//! coverage pixels stay zlib-packed on disk and decompress lazily on the
|
|
13
13
|
//! first blit, so `estimate` never pays for them.
|
|
14
|
-
var v=5,c=8,h=
|
|
14
|
+
var v=5,c=8,h=qJ(new URL("../assets/glyphs.cps",import.meta.url)),JJ=qJ(new URL("../assets/glyphs.wide",import.meta.url)),TJ=(()=>{let J=h.byteLength>>>2;if((h.byteOffset&3)===0)return new Uint32Array(h.buffer,h.byteOffset,J);return new Uint32Array(h.buffer.slice(h.byteOffset,h.byteOffset+(J<<2)))})(),xQ=(()=>{let J=new Uint32Array(JJ.length),Q=0;for(let Z=0;Z<JJ.length;Z++)J[Z]=Q,Q+=(JJ[Z]===1?2*v:v)*c;return J})(),DJ=null;function o(J){let Q=0,Z=TJ.length-1;while(Q<=Z){let $=Q+Z>>>1,V=TJ[$];if(V<J)Q=$+1;else if(V>J)Z=$-1;else return $}return-1}function u(J){return JJ[J]===1}function MJ(J){if(DJ===null)DJ=vQ(qJ(new URL("../assets/glyphs.pix.z",import.meta.url)));let Q=xQ[J],Z=u(J)?2*v:v;return DJ.subarray(Q,Q+Z*c)}function yJ(J,Q,Z){let $=MJ(J),V=u(J)?2*v:v;return kQ($,V,c,Q,Z)}function kQ(J,Q,Z,$,V){let G=new Uint8Array($*V),Y=Q/$,H=Z/V;for(let z=0;z<V;z++){let K=z*H,O=(z+1)*H;for(let B=0;B<$;B++){let X=B*Y,q=(B+1)*Y,D=0,M=0,R=Math.min(Math.ceil(O),Z);for(let P=Math.floor(K);P<R;P++){let T=Math.min(P+1,O)-Math.max(P,K);if(T<=0)continue;let E=Math.min(Math.ceil(q),Q);for(let A=Math.floor(X);A<E;A++){let y=Math.min(A+1,q)-Math.max(A,X);if(y<=0)continue;let w=y*T;D+=w*J[P*Q+A],M+=w}}G[z*$+B]=M>0?Math.min(255,Math.max(0,Math.round(D/M))):0}}return G}import{deflateSync as fQ}from"node:zlib";//! Minimal grayscale PNG encoder (bit depth 8, color type 0), mirroring
|
|
15
15
|
//! pxpipe's png.ts: IHDR + one IDAT (zlib) + IEND, filter byte 0 per row.
|
|
16
|
-
var hQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $=0;$<8;$++)Z=(Z&1)!==0?3988292384^Z>>>1:Z>>>1;J[Q]=Z>>>0}return J})();function FJ(J,Q,Z,$){let V=$.length;J[Q]=V>>>24&255,J[Q+1]=V>>>16&255,J[Q+2]=V>>>8&255,J[Q+3]=V&255;let G=Q+4;for(let z=0;z<4;z++)J[G+z]=Z.charCodeAt(z);J.set($,G+4);let Y=4294967295;for(let z=G;z<G+4+V;z++)Y=hQ[(Y^J[z])&255]^Y>>>8;Y=~Y>>>0;let H=G+4+V;return J[H]=Y>>>24&255,J[H+1]=Y>>>16&255,J[H+2]=Y>>>8&255,J[H+3]=Y&255,H+4}function wJ(J,Q,Z){let $=new Uint8Array(Z*(Q+1));for(let z=0,K=0,
|
|
16
|
+
var hQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $=0;$<8;$++)Z=(Z&1)!==0?3988292384^Z>>>1:Z>>>1;J[Q]=Z>>>0}return J})();function FJ(J,Q,Z,$){let V=$.length;J[Q]=V>>>24&255,J[Q+1]=V>>>16&255,J[Q+2]=V>>>8&255,J[Q+3]=V&255;let G=Q+4;for(let z=0;z<4;z++)J[G+z]=Z.charCodeAt(z);J.set($,G+4);let Y=4294967295;for(let z=G;z<G+4+V;z++)Y=hQ[(Y^J[z])&255]^Y>>>8;Y=~Y>>>0;let H=G+4+V;return J[H]=Y>>>24&255,J[H+1]=Y>>>16&255,J[H+2]=Y>>>8&255,J[H+3]=Y&255,H+4}function wJ(J,Q,Z){let $=new Uint8Array(Z*(Q+1));for(let z=0,K=0,O=0;z<Z;z++)$[K++]=0,$.set(J.subarray(O,O+Q),K),K+=Q,O+=Q;let V=fQ($,{level:6}),G=new Uint8Array(33+(12+V.length)+12);G[0]=137,G[1]=80,G[2]=78,G[3]=71,G[4]=13,G[5]=10,G[6]=26,G[7]=10;let Y=new Uint8Array(13);Y[0]=Q>>>24&255,Y[1]=Q>>>16&255,Y[2]=Q>>>8&255,Y[3]=Q&255,Y[4]=Z>>>24&255,Y[5]=Z>>>16&255,Y[6]=Z>>>8&255,Y[7]=Z&255,Y[8]=8,Y[9]=0;let H=FJ(G,8,"IHDR",Y);return H=FJ(G,H,"IDAT",V),FJ(G,H,"IEND",new Uint8Array(0)),G}//! Stage 2: the `pxpipe` imaging engine, ported from pxpipe's render.ts
|
|
17
17
|
//! production dense path (bare 5x8 AA cell, 312 cols, 1568x728 pages).
|
|
18
18
|
//! Glyphs cover BMP (pxpipe's atlas: Spleen for ASCII/code, Unifont fallback)
|
|
19
19
|
//! PLUS the astral planes (unifont_upper, incl. emoji) — beyond pxpipe, which
|
|
@@ -28,19 +28,19 @@ var hQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $
|
|
|
28
28
|
//! * `font` — `Tiny` renders the same atlas box-filtered into a 4x6 cell
|
|
29
29
|
//! (390 cols x 120 rows/page), ~40% fewer image-tokens; opt-in,
|
|
30
30
|
//! transcription-accuracy gated.
|
|
31
|
-
var i=4,QJ=4,mQ=1568,uQ=728,ZJ="↵",gQ="⏎",$J="→",dQ="⇢",WJ="⇥",lQ="⇨",pQ=9647,bJ=4,nQ=3,vJ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",cQ=/\n{4,}/g;function g(J){return J.length===4&&J.replace(/[A-Z]/g,(Q)=>Q.toLowerCase())==="tiny"?"tiny":"normal"}function hJ(J){let Q=J==="tiny"?4:v,Z=J==="tiny"?6:c,$=Math.floor((mQ-2*i)/Q),V=Math.floor((uQ-2*QJ)/Z);return{cw:Q,ch:Z,cols:$,maxLines:V,maxChars:$*V}}var oQ=(()=>{let J=new Uint8Array(128);for(let Q=0;Q<128;Q++){let Z=o(Q);J[Q]=Z>=0&&u(Z)?2:1}return J})();function
|
|
31
|
+
var i=4,QJ=4,mQ=1568,uQ=728,ZJ="↵",gQ="⏎",$J="→",dQ="⇢",WJ="⇥",lQ="⇨",pQ=9647,bJ=4,nQ=3,vJ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",cQ=/\n{4,}/g;function g(J){return J.length===4&&J.replace(/[A-Z]/g,(Q)=>Q.toLowerCase())==="tiny"?"tiny":"normal"}function hJ(J){let Q=J==="tiny"?4:v,Z=J==="tiny"?6:c,$=Math.floor((mQ-2*i)/Q),V=Math.floor((uQ-2*QJ)/Z);return{cw:Q,ch:Z,cols:$,maxLines:V,maxChars:$*V}}var oQ=(()=>{let J=new Uint8Array(128);for(let Q=0;Q<128;Q++){let Z=o(Q);J[Q]=Z>=0&&u(Z)?2:1}return J})();function jJ(J){if(J<128)return oQ[J];let Q=o(J);return Q>=0&&u(Q)?2:1}function iQ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<56320&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<57344)Z++}Q++}return Q}function NJ(J){let Q=J.split(`
|
|
32
32
|
`);for(let Z=0;Z<Q.length;Z++){let $=Q[Z],V=$.length;while(V>0){let G=$.charCodeAt(V-1);if(G!==32&&G!==9)break;V--}if(V!==$.length)Q[Z]=$.slice(0,V)}return Q.join(`
|
|
33
33
|
`).replace(cQ,`
|
|
34
34
|
|
|
35
35
|
|
|
36
|
-
`)}function mJ(J){if(!J.includes("\t"))return J;let Q="",Z=0;for(let $ of J)if($==="\t"){let V=bJ-Z%bJ;Q+=$J;for(let G=1;G<V;G++)Q+=" ";Z+=V}else Q+=$,Z+=
|
|
37
|
-
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=mJ(Q[Z]);return Q.join(ZJ)}function sQ(J){let Q=
|
|
38
|
-
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=
|
|
39
|
-
`)){let G=mJ(V);if(G.length===0){Z.push("");continue}let Y="",H=0;for(let z of G){let K=
|
|
36
|
+
`)}function mJ(J){if(!J.includes("\t"))return J;let Q="",Z=0;for(let $ of J)if($==="\t"){let V=bJ-Z%bJ;Q+=$J;for(let G=1;G<V;G++)Q+=" ";Z+=V}else Q+=$,Z+=jJ($.codePointAt(0));return Q}function aQ(J){let Q=J.includes("\t")?J.replaceAll("\t",$J):J,Z=0;while(Z<Q.length&&Q.charCodeAt(Z)===32)Z++;if(Z>=nQ&&Z<vJ.length)return WJ+vJ[Z]+Q.slice(Z);return Q}function uJ(J){return J.includes(ZJ)?J.replaceAll(ZJ,gQ):J}function rQ(J){let Q=uJ(J);if(Q.includes($J))Q=Q.replaceAll($J,dQ);if(Q.includes(WJ))Q=Q.replaceAll(WJ,lQ);return Q}function tQ(J){let Q=NJ(J).split(`
|
|
37
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=mJ(Q[Z]);return Q.join(ZJ)}function sQ(J){let Q=NJ(J).split(`
|
|
38
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=aQ(Q[Z]);return Q.join(ZJ)}function eQ(J,Q){let Z=[],$=NJ(J);for(let V of $.split(`
|
|
39
|
+
`)){let G=mJ(V);if(G.length===0){Z.push("");continue}let Y="",H=0;for(let z of G){let K=jJ(z.codePointAt(0));if(H+K>Q)Z.push(Y),Y=z,H=K;else Y+=z,H+=K}if(Y.length!==0)Z.push(Y)}return Z}function JZ(J,Q,Z){let $=[],V=[],G=0;for(let Y of J){let H=iQ(Y),z=H+(V.length!==0?1:0);if(V.length!==0&&(V.length>=Q||G+z>Z))$.push(V),V=[],G=0;G+=H+(V.length!==0?1:0),V.push(Y)}if(V.length!==0)$.push(V);return $}function gJ(J){if(J<32)return!0;if(J>=127&&J<=159)return!0;if(J>=768&&J<=879)return!0;if(J===8203||J===8204||J===8205||J===8288||J===65279)return!0;if(J>=65024&&J<=65039)return!0;if(J>=917760&&J<=917999)return!0;return!1}function QZ(J){let Q=null,Z=0;for(let $ of J){let V=$.codePointAt(0);if(o(V)<0&&!gJ(V)){if(Q===null)Q=J.slice(0,Z);Q+=`[U+${V.toString(16).toUpperCase()}]`}else if(Q!==null)Q+=$;Z+=$.length}return Q??J}function dJ(J,Q,Z,$){let V=Q?Z?sQ(rQ(J)):tQ(uJ(J)):J;return JZ(eQ(QZ(V),$.cols),$.maxLines,$.maxChars)}function lJ(J,Q,Z){if(!Z)return 2*i+Q.cols*Q.cw;let $=0;for(let Y of J){let H=0;for(let z of Y)H+=jJ(z.codePointAt(0));if(H>Q.cols)H=Q.cols;if(H>$)$=H}let V=2*i+$*Q.cw,G=2*i+Q.cw;return V>G?V:G}var xJ=new Map;function kJ(J,Q,Z,$,V,G,Y){let H=o(V);if(H<0)return 0;let z=u(H),K=z?2*G.cw:G.cw,O;if(Y==="normal")O=MJ(H);else{let B=xJ.get(H);if(B===void 0)B=yJ(H,K,G.ch),xJ.set(H,B);O=B}for(let B=0;B<G.ch;B++){let X=($+B)*Q+Z,q=B*K;for(let D=0;D<K;D++){let M=O[q+D];if(M>0){let R=X+D;if(M>J[R])J[R]=M}}}return z?2:1}function ZZ(J,Q,Z,$){let V=lJ(J,Q,Z),G=2*QJ+J.length*Q.ch,Y=new Uint8Array(V*G),H=0;for(let z=0;z<J.length;z++){let K=QJ+z*Q.ch,O=0;for(let B of J[z]){if(O>=Q.cols)break;let X=i+O*Q.cw,q=kJ(Y,V,X,K,B.codePointAt(0),Q,$);if(q===0){if(q=1,gJ(B.codePointAt(0)));else if(H++,B!==" ")kJ(Y,V,X,K,pQ,Q,$)}O+=q}}for(let z=0;z<Y.length;z++)Y[z]=255-Y[z];return{png:wJ(Y,V,G),width:V,height:G,dropped:H}}function VJ(J,Q,Z,$){let V=hJ($),G=dJ(J,Q,Z,V).map((K)=>ZZ(K,V,Z,$)),Y=0,H=0,z=0;for(let K of G)Y+=K.width*K.height,H+=pJ(K.width,K.height),z+=K.dropped;return{pages:G,pixels:Y,tokens:H,dropped:z}}function GJ(J,Q,Z,$){let V=hJ($),G=dJ(J,Q,Z,V),Y=0,H=0;for(let z of G){let K=lJ(z,V,Z),O=2*QJ+z.length*V.ch;Y+=K*O,H+=pJ(K,O)}return{pages:G.length,pixels:Y,tokens:H}}var fJ=28;function pJ(J,Q){return Math.ceil(J/fJ)*Math.ceil(Q/fJ)}import{readFileSync as $Z}from"node:fs";import{join as VZ}from"node:path";//! pxpipe measurement-log summary (~/.pxpipe/events.jsonl), same math as the
|
|
40
40
|
//! node MCP: actual = every way input bytes get billed (input + cache reads +
|
|
41
41
|
//! cache creates) — ignoring cache_read would fake the savings.
|
|
42
42
|
class YJ{value;constructor(J){this.value=J}}function nJ(){let J=process.env.TANUKI_EVENTS;if(J!==void 0)return J;let Q=process.env.HOME??"";return VZ(Q,".pxpipe","events.jsonl")}function d(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:0}function GZ(J){return J<0?-Math.round(-J):Math.round(J)}function IJ(){let J=nJ(),Q;try{Q=$Z(J,"utf8")}catch{return{available:!1,note:`no ${J} yet`}}let Z=0,$=0,V=0,G=0,Y=0,H=0;for(let K of Q.split(`
|
|
43
|
-
`)){if(K.trim().length===0)continue;let
|
|
43
|
+
`)){if(K.trim().length===0)continue;let O;try{O=JSON.parse(K)}catch{continue}Z+=1;let B=O!==null&&typeof O==="object"&&!Array.isArray(O)?O:{};if(B.compressed===!0)$+=1,V+=d(B.orig_chars),G+=d(B.image_count);Y+=d(B.baseline_tokens),H+=d(B.input_tokens)+d(B.cache_read_tokens)+d(B.cache_create_tokens)}let z=Y>0&&H>0?new YJ(GZ((1-H/Y)*1000)/10):null;return{available:!0,requests:Z,compressedRequests:$,imagedChars:V,imagesEmitted:G,baselineTokens:Y,actualInputTokens:H,estInputSavedPct:z}}//! Implicit mode: a local Anthropic middlebox, the pxpipe deployment shape
|
|
44
44
|
//! without pxpipe's structural flaw. Rules that keep it injection-shaped-free:
|
|
45
45
|
//!
|
|
46
46
|
//! 1. The system prompt and tool definitions are NEVER touched.
|
|
@@ -67,7 +67,7 @@ import{createHash as YZ}from"node:crypto";import{mkdirSync as HZ,readFileSync as
|
|
|
67
67
|
//! clearly win. Contract is byte-identical with the Rust engine.
|
|
68
68
|
function oJ(){let J=cJ.env.TANUKI_STASH;if(J!==void 0&&J!=="")return J;return`${cJ.env.HOME??""}/.tanuki/stash`}function iJ(J){let Q=YZ("sha256").update(J,"utf8").digest("hex").slice(0,12),Z=oJ();HZ(Z,{recursive:!0}),KZ(`${Z}/${Q}`,J);let $=Buffer.byteLength(J,"utf8"),V=J.split(`
|
|
69
69
|
`),G=b(J,null,2).stats,Y=[`stashed ${Q} · ${$} bytes · ${V.length} lines`,`distill map: ${G.origLines} -> ${G.outLines} lines · ${G.savedPct}% of chars removable · ${G.importantKept} error/warn lines`];if(G.topRepeats.length>0){Y.push("top repeats:");for(let z of G.topRepeats.slice(0,5)){let K=z.kind==="template"?" (template)":"";Y.push(` ×${z.count}${K} ${z.exemplar}`)}}let H="";for(let z=V.length-1;z>=0;z--)if(V[z]!==""){H=V[z];break}return Y.push(`first: ${n(p(V[0]),160)}`),Y.push(`last: ${n(p(H),160)}`),Y.push(`fetch: tanuki_fetch {"id":"${Q}","query":"<regex>"} or {"id":"${Q}","lines":"a-b"}`),{id:Q,overview:Y.join(`
|
|
70
|
-
`)}}function
|
|
70
|
+
`)}}function aJ(J,Q,Z){if(Q===null===(Z===null))throw Error("give exactly one of query or lines");let $;try{$=zZ(`${oJ()}/${J}`,"utf8")}catch{throw Error(`unknown stash id: ${J}`)}if(Z!==null){let V=/^(\d+)-(\d+)$/.exec(Z);if(V===null)throw Error("bad lines range");let G=$.split(`
|
|
71
71
|
`),Y=Math.max(1,Number(V[1])),H=Math.min(G.length,Number(V[2]));if(Number(V[1])>Number(V[2]))throw Error("bad lines range");return G.slice(Y-1,H).join(`
|
|
72
72
|
`)}return b($,Q,2).distilled}//! tanuki-context — token-cutting context pipeline.
|
|
73
73
|
//! pipeline: text -> distill (stage 0, logs) -> ladder level 0-4 (stage 1)
|
|
@@ -78,15 +78,15 @@ function oJ(){let J=cJ.env.TANUKI_STASH;if(J!==void 0&&J!=="")return J;return`${
|
|
|
78
78
|
//! tanuki-context estimate <file> [level] [--distill]
|
|
79
79
|
//! tanuki-context render <file> [level] [outdir]
|
|
80
80
|
//! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
|
|
81
|
-
var
|
|
82
|
-
`;for(let
|
|
83
|
-
`;K+=z+L(J[
|
|
84
|
-
`+Z+"]"}let V=J,G=Object.keys(V).sort(
|
|
81
|
+
var rJ="0.4.1",RJ=6;function BZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function L(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof YJ){let z=J.value;return Number.isFinite(z)&&Number.isInteger(z)?z.toFixed(1):String(z)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let O="[";for(let B=0;B<J.length;B++){if(B>0)O+=",";O+=L(J[B],!1)}return O+"]"}let z=Z+" ",K=`[
|
|
82
|
+
`;for(let O=0;O<J.length;O++){if(O>0)K+=`,
|
|
83
|
+
`;K+=z+L(J[O],!0,z)}return K+`
|
|
84
|
+
`+Z+"]"}let V=J,G=Object.keys(V).sort(BZ);if(G.length===0)return"{}";if(!Q){let z="{";for(let K=0;K<G.length;K++){if(K>0)z+=",";z+=JSON.stringify(G[K])+":"+L(V[G[K]],!1)}return z+"}"}let Y=Z+" ",H=`{
|
|
85
85
|
`;for(let z=0;z<G.length;z++){if(z>0)H+=`,
|
|
86
86
|
`;H+=Y+JSON.stringify(G[z])+": "+L(V[G[z]],!0,Y)}return H+`
|
|
87
|
-
`+Z+"}"}function I(J,Q){return J!==null&&typeof J==="object"&&!Array.isArray(J)?J[Q]:void 0}function S(J){return typeof J==="string"?J:null}function HJ(J){return typeof J==="boolean"?J:null}function tJ(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:null}function x(J){let Q=0;for(let Z=0;Z<J.length;Z++){Q++;let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}}return Q}function sJ(J,Q,Z,$,V){let G,Y=null;if(Z||$!==null){let K=b(J,$,2);G=K.distilled,Y=K.stats}else G=J;let H=0;if(V){let K=t(G);G=K.text,H=K.entries}let z=e(G,Q);return{stage0:Y,compressed:z.compressed,protectedLines:z.protectedLines,level:z.level,cbEntries:H}}function
|
|
88
|
-
`}if(
|
|
89
|
-
vs ~${z} text-tokens raw = TOTAL -${
|
|
87
|
+
`+Z+"}"}function I(J,Q){return J!==null&&typeof J==="object"&&!Array.isArray(J)?J[Q]:void 0}function S(J){return typeof J==="string"?J:null}function HJ(J){return typeof J==="boolean"?J:null}function tJ(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:null}function x(J){let Q=0;for(let Z=0;Z<J.length;Z++){Q++;let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}}return Q}function sJ(J,Q,Z,$,V){let G,Y=null;if(Z||$!==null){let K=b(J,$,2);G=K.distilled,Y=K.stats}else G=J;let H=0;if(V){let K=t(G);G=K.text,H=K.entries}let z=e(G,Q);return{stage0:Y,compressed:z.compressed,protectedLines:z.protectedLines,level:z.level,cbEntries:H}}function a(J){return Math.round(J/4)}function r(J,Q){if(J===0)return 0;let Z=(1-Q/J)*100;return Z<0?-Math.round(-Z):Math.round(Z)}function eJ(J){return{text:S(I(J,"text"))??"",level:(tJ(I(J,"level"))??0)%256,distill:HJ(I(J,"distill"))??!1,query:S(I(J,"query")),reflow:HJ(I(J,"reflow"))??!0,pack:HJ(I(J,"pack"))??!0,font:S(I(J,"font"))??"normal",codebook:HJ(I(J,"codebook"))??!1}}function XZ(J){let Q=(G)=>{let Y={codebook:!1,tokens:1/0,pages:0,text:G};for(let H of[!1,!0]){let z=H?t(G).text:G,K=GJ(z,!0,!0,g("normal"));if(K.tokens<Y.tokens)Y={codebook:H,tokens:K.tokens,pages:K.pages,text:z}}return Y},Z=Q(J),$=Q(b(J,null,2).distilled),V=GJ(Z.text,!0,!0,g("tiny"));return{codebook:Z.codebook,imageTokens:Z.tokens,pages:Z.pages,tinyImageTokens:V.tokens,withDistill:{codebook:$.codebook,imageTokens:$.tokens}}}function JQ(J){let Q=eJ(J),Z=sJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=g(Q.font),V=GJ(Z.compressed,Q.reflow,Q.pack,$),G=V.tokens,Y=x(Q.text),H=x(Z.compressed),z=a(Y),[K,O]=s[Z.level];return{engine:"pxpipe",level:`${Z.level} ${K}`,loss:O,distill:Z.stage0,origChars:Y,stage1Chars:H,stage1SavedPct:r(Y,H),pages:V.pages,imageTokens:G,rawTextTokens:z,totalSavedPct:r(z,G),protectedLines:Z.protectedLines,pack:Q.pack,font:$==="tiny"?"tiny":"normal",codebook:Q.codebook?Z.cbEntries:!1,verdict:G<z?"PIPELINE cheaper":"TEXT cheaper",recommend:XZ(Q.text)}}function QQ(J){let Q=eJ(J),Z=sJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=g(Q.font),V=VJ(Z.compressed,Q.reflow,Q.pack,$),G=V.tokens,Y=x(Q.text),H=x(Z.compressed),z=a(Y),[K,O]=s[Z.level],B="";if(Z.stage0!==null){let D=Z.stage0;B+=`distill: ${D.origLines} -> ${D.outLines} lines (-${D.savedPct}% chars, ${D.collapsedRuns} runs, ${D.suppressedLines} exact + ${D.templateSuppressed} template suppressed, ${D.importantKept} error/warn kept)
|
|
88
|
+
`}if(B+=`L${Z.level} ${K} (${O}): ${Y} chars -> ${H} chars (stage1 -${r(Y,H)}%) -> ${V.pages.length} page(s), ~${G} image-tokens
|
|
89
|
+
vs ~${z} text-tokens raw = TOTAL -${r(z,G)}%`,Z.protectedLines>0)B+=` · ${Z.protectedLines} lines kept verbatim`;if(V.dropped>0)B+=` · ${V.dropped} unmapped glyphs -> ▯`;if(Z.cbEntries>0)B+=` · codebook: ${Z.cbEntries} sigils (see ·legend·)`;if($==="tiny")B+=" · font: tiny 4x6";if(Q.pack)B+=" · packed (⇥N indent, → tab)";if(Q.reflow)B+=" · ↵ = newline · engine: pxpipe";let X=[{type:"text",text:B}],q=Math.min(V.pages.length,RJ);for(let D=0;D<q;D++){let M=V.pages[D].png;X.push({type:"image",data:Buffer.from(M.buffer,M.byteOffset,M.byteLength).toString("base64"),mimeType:"image/png"})}if(V.pages.length>RJ)X.push({type:"text",text:`(+${V.pages.length-RJ} more page(s))`});return X}function ZQ(J){let Q=S(I(J,"text"))??"",Z=b(Q,S(I(J,"query")),2);return[{type:"text",text:L(Z.stats,!0)},{type:"text",text:Z.distilled}]}function $Q(J){let Q=S(I(J,"text"))??"",Z=(tJ(I(J,"level"))??1)%256,$=e(Q,Z),[V,G,Y]=s[$.level],H=x(Q),z=x($.compressed),K=a(H),O=a(z),B={level:`${$.level} ${V}`,loss:G,note:Y,origChars:H,outChars:z,approxOrigTokens:K,approxOutTokens:O,savedPct:r(K,O),protectedLines:$.protectedLines};return[{type:"text",text:L(B,!0)},{type:"text",text:$.compressed}]}function VQ(J){let Q=S(I(J,"text"))??"";return[{type:"text",text:iJ(Q).overview}]}function GQ(J){let Q=S(I(J,"id"))??"",Z=aJ(Q,S(I(J,"query"))??null,S(I(J,"lines"))??null),$=a(x(Z)),V=VJ(Z,!0,!0,g("normal"));if(!(V.tokens<=$*0.75&&$-V.tokens>=300&&V.pages.length<=6))return[{type:"text",text:Z}];let H=[{type:"text",text:`[tanuki-context stash ${Q}: slice of ${x(Z)} chars imaged as ${V.pages.length} PNG page(s), ~${V.tokens} vs ~${$} text tokens. ↵=newline →=tab ⇥N=indent]`}];for(let z of V.pages){let K=z.png;H.push({type:"image",data:Buffer.from(K.buffer,K.byteOffset,K.byteLength).toString("base64"),mimeType:"image/png"})}return H}//! Claude Agent SDK integration (`tanuki-context/agent`).
|
|
90
90
|
//!
|
|
91
91
|
//! Two flavours, both one-liners from an agent's point of view:
|
|
92
92
|
//!
|
|
@@ -103,8 +103,8 @@ vs ~${z} text-tokens raw = TOTAL -${a(z,G)}%`,Z.protectedLines>0)X+=` · ${Z.pro
|
|
|
103
103
|
//! The core package stays zero-dependency: the SDK and zod are touched only
|
|
104
104
|
//! inside `tanukiSdkServer()` via dynamic import, and both are already present
|
|
105
105
|
//! in any Agent SDK project (zod is the SDK's own peer dependency).
|
|
106
|
-
var
|
|
107
|
-
Workflow: call tanuki_estimate first (instant, exact, never renders pixels). Its "recommend" field
|
|
106
|
+
var DZ=["tanuki_render","tanuki_estimate","tanuki_distill","tanuki_compress","tanuki_stats","tanuki_stash","tanuki_fetch"],qZ=`tanuki-context turns bulky text (logs, command output, docs) into dense PNG pages that cost a fraction of the text tokens.
|
|
107
|
+
Workflow: call tanuki_estimate first (instant, exact, never renders pixels). Its "recommend" field prices the reversible route; for logs, recommend.withDistill prices the distill route - do not probe combos by hand. If the verdict says "PIPELINE cheaper", call tanuki_render with the recommended knobs and use the returned pages instead of pasting the text.
|
|
108
108
|
For logs, pass distill:true (repeats collapse, error/warn lines stay verbatim; add query:"regex" to slice). For prose you will not quote verbatim, level 2-3 shrinks it further. codebook:true helps path-heavy logs. Never image content you must quote byte-exact at level 4 or font tiny.
|
|
109
109
|
For huge references you will only consult occasionally: tanuki_stash parks the text outside context for a few hundred tokens of map; tanuki_fetch pulls slices later, auto-imaged when pages win.
|
|
110
|
-
Pages decode as: ↵ = newline, → = tab, ⇥N = N leading spaces, a trailing ·legend· line maps sigils back to full tokens.`;function MZ(){let J=new URL("./cli.js",import.meta.url);if(J.protocol==="file:"){let Q=
|
|
110
|
+
Pages decode as: ↵ = newline, → = tab, ⇥N = N leading spaces, a trailing ·legend· line maps sigils back to full tokens.`;function MZ(){let J=new URL("./cli.js",import.meta.url);if(J.protocol==="file:"){let Q=UZ(J);if(OZ(Q))return{type:"stdio",command:process.execPath,args:[Q]}}return{type:"stdio",command:"npx",args:["-y","tanuki-context"]}}function FZ(J="tanuki"){return DZ.map((Q)=>`mcp__${J}__${Q}`)}function H$(J,Q={}){let Z=J??{},$=Q.key??"tanuki";return{...Z,mcpServers:{...Z.mcpServers??{},[$]:Q.server??MZ()},allowedTools:[...Z.allowedTools??[],...FZ($)]}}function WZ(J){let Q=J.string().describe("the bulky text to process"),Z=J.number().int().min(0).max(4).optional().describe("ladder level 0-4 (default 0)"),$={text:Q,level:Z,distill:J.boolean().optional().describe("stage 0 log distiller"),query:J.string().optional().describe("distill: keep matching lines +context"),reflow:J.boolean().optional().describe("pack short lines into full rows (default true)"),pack:J.boolean().optional().describe("indent RLE + width trim, lossless (default true)"),font:J.enum(["normal","tiny"]).optional().describe("tiny = 4x6 cells, ~40% fewer tokens, gated"),codebook:J.boolean().optional().describe("repeated tokens/paths -> sigils + legend")},V=(Y)=>({content:Y}),G=async(Y)=>{try{return V(Y())}catch(H){return{content:[{type:"text",text:`tanuki-context error: ${H instanceof Error?H.message:String(H)}`}],isError:!0}}};return[{name:"tanuki_render",description:"Render text through the pipeline (optional distill/level/codebook) into dense PNG pages. Call after tanuki_estimate says PIPELINE cheaper.",inputSchema:$,handler:(Y)=>G(()=>QQ(Y))},{name:"tanuki_estimate",description:"Exact page/token math for the same arguments as tanuki_render, without touching pixels. Instant; call this first.",inputSchema:$,handler:(Y)=>G(()=>[{type:"text",text:L(JQ(Y),!0)}])},{name:"tanuki_distill",description:"Stage 0 alone: collapse repeated log lines/blocks and template near-dupes; error/warn lines kept verbatim. Output stays greppable text.",inputSchema:{text:Q,query:$.query},handler:(Y)=>G(()=>ZQ(Y))},{name:"tanuki_compress",description:"Stage 1 alone: graded text compression, levels 0-4, code/paths/hashes protected from level 2 up.",inputSchema:{text:Q,level:Z},handler:(Y)=>G(()=>$Q(Y))},{name:"tanuki_stats",description:"Session savings summary from the events log (honest denominator: input + cache reads + cache creates).",inputSchema:{},handler:()=>G(()=>[{type:"text",text:L(IJ(),!0)}])},{name:"tanuki_stash",description:"Park bulky text outside the context window; returns a compact map (distill stats, top repeats, id). Retrieval pattern, tanuki pricing on the way back.",inputSchema:{text:Q},handler:(Y)=>G(()=>VQ(Y))},{name:"tanuki_fetch",description:"Pull a slice of stashed text by id + query regex or lines 'a-b'. Big slices return as dense PNG pages automatically.",inputSchema:{id:$.query,query:$.query,lines:$.query},handler:(Y)=>G(()=>GQ(Y))}]}async function z$(){let J,Q;try{J=await import("@anthropic-ai/claude-agent-sdk");let $=await import("zod");Q=$.z??$}catch($){throw Error(`tanukiSdkServer() needs the host project's @anthropic-ai/claude-agent-sdk and zod (npm i @anthropic-ai/claude-agent-sdk zod). Import failed: ${$ instanceof Error?$.message:String($)}`)}let Z=WZ(Q).map(($)=>J.tool($.name,$.description,$.inputSchema,(V)=>$.handler(V)));return J.createSdkMcpServer({name:"tanuki-context",version:rJ,instructions:qZ,tools:Z})}export{H$ as withTanuki,WZ as tanukiSdkToolSpecs,z$ as tanukiSdkServer,MZ as tanukiMcpServer,FZ as tanukiAllowedTools,DZ as TANUKI_TOOL_NAMES,qZ as TANUKI_INSTRUCTIONS};
|
package/dist/cli.js
CHANGED
|
@@ -1,20 +1,20 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import{readFileSync as
|
|
3
|
-
·legend· `;for(let
|
|
4
|
-
`),G=z.length,Y=G,H=Array(G),K=Array(G),
|
|
5
|
-
`),
|
|
6
|
-
`),z=Array(V.length);for(let Y=0;Y<V.length;Y++){let H=V[Y],K=H.length;while(K>0){let
|
|
7
|
-
`).replace(
|
|
2
|
+
import{spawnSync as SZ}from"node:child_process";import{readFileSync as CZ,mkdirSync as GQ,writeFileSync as YQ}from"node:fs";import R from"node:process";function WQ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function FJ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}Q++}return Q}function jQ(J,Q){let Z=0,$=0;while(Z<J.length&&$<Q.length){let V=J.codePointAt(Z),z=Q.codePointAt($);if(V!==z)return V-z;Z+=V>65535?2:1,$+=z>65535?2:1}return J.length-Z-(Q.length-$)}function a(J){let Q=new Map,Z=(O)=>{Q.set(O,(Q.get(O)??0)+1)},$=(O)=>{if(FJ(O)>=12)Z(O);if(O.includes("/")){let X=O.split("/"),D="";for(let M=0;M<X.length;M++){if(M>0)D+="/";if(D+=X[M],M>=2){let E=D+"/";if(FJ(E)>=12)Z(E)}}}},V=-1;for(let O=0;O<J.length;){let X=J.codePointAt(O),D=X>65535?2:1;if(WQ(X)){if(V>=0)$(J.slice(V,O)),V=-1}else if(V<0)V=O;O+=D}if(V>=0)$(J.slice(V));let z=[];for(let[O,X]of Q)if(X>=3){let D=FJ(O);z.push({k:O,c:X,len:D,saved:(D-1)*X})}z.sort((O,X)=>X.saved-O.saved||jQ(O.k,X.k));let G=[];for(let O of"§¤¢£¥µ¶ª°±¬×÷ØÞßæðøþ¡¿")if(!J.includes(O))G.push(O);let Y=[],H=[];for(let{k:O,c:X,len:D}of z){if(Y.length>=G.length)break;if((D-1)*X<=D+3)continue;let M=!1;for(let S=0;S<H.length;S++){let k=H[S];if(k.startsWith(O)||O.startsWith(k)){M=!0;break}}if(M)continue;let E=G[Y.length];H.push(O),Y.push({sig:E,val:O,len:D})}if(Y.length===0)return{text:J,entries:0};let K=Y.map((O,X)=>X);K.sort((O,X)=>Y[X].len-Y[O].len);let B=J;for(let O=0;O<K.length;O++){let{sig:X,val:D}=Y[K[O]];B=B.replaceAll(D,X)}let U=`
|
|
3
|
+
·legend· `;for(let O=0;O<Y.length;O++)U+=Y[O].sig+"="+Y[O].val+" ";return B+=U.slice(0,-1),{text:B,entries:Y.length}}import{Buffer as fJ}from"node:buffer";var NQ=/\x1b\[[0-9;]*[A-Za-z]/g,WJ=/\b([0-9A-Za-z_]*(error|exception)s?|err|warn(ing)?s?|fail(s|ed|ure|ures)?|panic(s|ked)?|fatal|critical|traceback|denied|refused|timeouts?|timed.?out|assert(s|ed|ion|ions)?|segfault(s|ed)?)\b/i,IQ=/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/g,RQ=/\b[0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?\b/g,AQ=/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,PQ=/\b[0-9a-f]{7,64}\b/gi,_Q=/\b[0-9]+(\.[0-9]+)?[ \t\u00a0]?(ms|us|µs|ns|s|m|h|%|[KMGT]i?B)?\b/gi,EQ=/[0-9a-f]/i,SQ=3,CQ=8,jJ=2,hJ=40;function TQ(J){if(!EQ.test(J))return J;let Q=J.replace(IQ,"<ts>");return Q=Q.replace(RQ,"<time>"),Q=Q.replace(AQ,"<uuid>"),Q=Q.replace(PQ,"<hex>"),Q.replace(_Q,"<n>")}function p(J){if(J===32)return!0;if(J<9)return!1;if(J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function r(J){let Q=0,Z=J.length;while(Q<Z&&p(J.charCodeAt(Q)))Q++;while(Z>Q&&p(J.charCodeAt(Z-1)))Z--;return Q===0&&Z===J.length?J:J.slice(Q,Z)}function LQ(J,Q){let Z=0,$=J.length;while(Z<$&&p(J.charCodeAt(Z)))Z++;while($>Z&&p(J.charCodeAt($-1)))$--;let V=0;for(let z=Z;z<$&&V<Q;V++){let G=J.charCodeAt(z);z+=G>=55296&&G<=56319&&z+1<$?2:1}return V<Q}function yQ(J){let Q=0,Z=J.length;while(Q<Z&&p(J.charCodeAt(Q)))Q++;return J.charCodeAt(Q)===91&&J.charCodeAt(Q+1)===215}function wQ(J){let Q=[],Z=-1,$=!0,V=!1,z=!1,G=!0;for(let Y=0;Y<J.length;){let H=J.codePointAt(Y),K=H>65535?2:1,B=p(H);if(G){if(G=!1,B)Q.push("<v>")}if(B){if(Z>=0)Q.push($?J.slice(Z,Y):"<v>"),Z=-1;z=!0}else{if(V=!0,z=!1,Z<0)Z=Y,$=!0;if($){let U=H|32;if(U<97||U>122||H>127)$=!1}}Y+=K}if(Z>=0)Q.push($?J.slice(Z):"<v>");if(V&&z)Q.push("<v>");return Q.join(" ")}function t(J,Q){let Z=0,$=0,V=J.length;while(Z<V&&$<Q){let z=J.charCodeAt(Z);Z+=z>=55296&&z<=56319&&Z+1<V?2:1,$++}return Z>=V?J:J.slice(0,Z)}var bQ=/[.*+?^${}()|[\]\\]/g;function w(J,Q=null,Z=2){let $=Z,z=J.replace(NQ,"").split(`
|
|
4
|
+
`).map((q)=>{let F=q.endsWith("\r")?q.slice(0,-1):q,W=F.lastIndexOf("\r");return W===-1?F:F.slice(W+1)}),G=z.length,Y=G,H=Array(G),K=Array(G),B=0;for(let q=0;q<G;q++){H[q]=TQ(z[q]);let F=WJ.test(z[q]);if(K[q]=F,F)B++}let U=[],O=[],X=[],D=0,M=0;while(M<G){let q=0,F=0;for(let W=1;W<=CQ&&M+2*W<=G;W++){if(K[M+W-1])break;let I=1;J:for(;;){let N=M+I*W;if(N+W>G)break;for(let T=0;T<W;T++)if(K[N+T]||H[N+T]!==H[M+T])break J;I++}if(I>=SQ&&W*(I-1)>q*(F>0?F-1:0))q=W,F=I}if(q>0){for(let W=M;W<M+q;W++)U.push(z[W]),O.push(K[W]),X.push(H[W]);U.push(q===1?` [×${F} similar]`:` [×${F} similar ${q}-line blocks]`),O.push(!1),X.push(null),D++,M+=q*F}else U.push(z[M]),O.push(K[M]),X.push(H[M]),M++}let E=new Map,S=new Map,k=jJ+1,y=[],C=[],x=0,f=0;for(let q=0;q<U.length;q++){let F=U[q],W=O[q];if(W||yQ(F)||LQ(F,4)){y.push(F),C.push(W);continue}let I=X[q],N=E.get(I);if(N!==void 0){if(N.count++,N.count<=jJ)y.push(F),C.push(!1);else x++;continue}let T=wQ(I);E.set(I,{count:1,exemplar:F});let i=S.get(T);if(i!==void 0)if(i.count++,i.count<=k)y.push(F),C.push(!1);else f++;else S.set(T,{count:1,exemplar:F}),y.push(F),C.push(!1)}let u=[];for(let q of E.values())if(q.count>jJ)u.push([q.count,"exact",q.exemplar]);for(let q of S.values())if(q.count>k)u.push([q.count,"template",q.exemplar]);if(u.sort((q,F)=>F[0]-q[0]),u.length>hJ)u.length=hJ;if(x+f>0){let q=`── ${x+f} repeated lines suppressed (${x} exact ×N, ${f} same-template; first occurrences kept above) ──`;y.push(q),C.push(WJ.test(q));for(let[F,W,I]of u){let T=` ×${F}${W==="template"?" (template)":""} ${t(r(I),160)}`;y.push(T),C.push(WJ.test(T))}}let g;if(Q!=null){let q;try{q=new RegExp(Q,"i")}catch{q=new RegExp(Q.replace(bQ,"\\$&"),"i")}let F=y.length,W=new Uint8Array(F);for(let N=0;N<F;N++)if(C[N]||q.test(y[N])){let T=N>$?N-$:0,i=Math.min(N+$,F-1);W.fill(1,T,i+1)}g=[];let I=0;for(let N=0;N<F;N++)if(W[N]){if(I>0)g.push(`… ${I} lines omitted`),I=0;g.push(y[N])}else I++;if(I>0)g.push(`… ${I} lines omitted`)}else g=y;let vJ=g.join(`
|
|
5
|
+
`),kJ=fJ.byteLength(J),xJ=fJ.byteLength(vJ),MJ=(1-xJ/kJ)*100,qQ=J.length===0?0:MJ<0?-Math.round(-MJ):Math.round(MJ),MQ=u.map(([q,F,W])=>({count:q,exemplar:t(r(W),160),kind:F})),FQ={collapsedRuns:D,importantKept:B,origChars:kJ,origLines:Y,outChars:xJ,outLines:g.length,query:Q??null,savedPct:qQ,suppressedLines:x,templateSuppressed:f,topRepeats:MQ};return{distilled:vJ,stats:FQ}}var GJ=[["none","none","passthrough (baseline)"],["whitespace","lossless","trailing whitespace + blank-line runs collapsed; safe for code"],["prose","light","L1 + prose lines: collapse spaces, cut redundant filler phrases (code/IDs protected)"],["dense","medium","L2 + prose: drop articles & intensifiers"],["caveman","heavy","L3 + prose: telegraphic — drop function words; gist only, NOT verbatim"]];var b=(J)=>new RegExp(`(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:${J})(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])`,"giu"),NJ=[[b("in order to"),"to"],[b("due to the fact that"),"because"],[b("at this point in time"),"now"],[b("in the event that"),"if"],[b("for the purpose of"),"for"],[b("with regard to"),"about"],[b("a large number of"),"many"],[b("it is important to note that"),""],[b("please note that"),""],[b("as a matter of fact"),""],[b("in terms of"),"for"],[b("the fact that"),"that"]],vQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:the|an|a)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),kQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:very|really|just|actually|basically|simply|quite|rather|essentially|literally)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),xQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:is|are|was|were|am|be|been|being|do|does|did|have|has|had|will|would|shall|should|can|could|may|might|of|to|in|on|at|for|with|that|this|these|those|it|its|there|here)(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]*","giu"),mJ=/ {2,}/g,fQ=new RegExp("[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+([.,;:!?])","gu"),hQ=/\n{3,}/g;function IJ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function mQ(J){let Q=0,Z=J.length;while(Q<Z){let $=J.codePointAt(Q);if(!IJ($))break;Q+=$>65535?2:1}while(Z>Q){let $=J.charCodeAt(Z-1);if(!IJ($))break;Z--}return Q===0&&Z===J.length?J:J.slice(Q,Z)}function uQ(J){if(J.length===0)return!1;let Q=J.charCodeAt(0);if(Q===32||Q===9)return!0;let Z=0,$=0,V=0,z=!1;for(let G=0;G<J.length;){let Y=J.codePointAt(G);if(G+=Y>65535?2:1,Z++,IJ(Y)){V=0;continue}if(V++,V>=24)z=!0;if(Y>=48&&Y<=57||Y>=65&&Y<=90||Y>=97&&Y<=122)continue;switch(Y){case 46:case 44:case 59:case 58:case 39:case 34:case 33:case 63:case 40:case 41:case 45:continue}$++}if($/Z>0.3)return!0;return z}function gQ(J,Q){let Z=J.replace(mJ," ");for(let V=0;V<NJ.length;V++)Z=Z.replace(NJ[V][0],NJ[V][1]);if(Q>=3)Z=Z.replace(vQ,""),Z=Z.replace(kQ,"");if(Q>=4)Z=Z.replace(xQ,"");Z=Z.replace(mJ," "),Z=mQ(Z.replace(fQ,"$1"));let $=Z.charCodeAt(0);if($>=97&&$<=122)Z=String.fromCharCode($-32)+Z.slice(1);return Z}function s(J,Q){let Z=Math.min(Q,4);if(Z===0)return{compressed:J,protectedLines:0,level:Z};let $=0,V=J.split(`
|
|
6
|
+
`),z=Array(V.length);for(let Y=0;Y<V.length;Y++){let H=V[Y],K=H.length;while(K>0){let U=H.charCodeAt(K-1);if(U!==32&&U!==9)break;K--}let B=K===H.length?H:H.slice(0,K);if(Z===1){z[Y]=B;continue}if(uQ(B)){$++,z[Y]=B;continue}z[Y]=gQ(B,Z)}return{compressed:z.join(`
|
|
7
|
+
`).replace(hQ,`
|
|
8
8
|
|
|
9
|
-
`),protectedLines:$,level:Z}}import{appendFileSync as
|
|
9
|
+
`),protectedLines:$,level:Z}}import{appendFileSync as DZ,mkdirSync as qZ}from"node:fs";import{dirname as MZ}from"node:path";import ZQ from"node:http";import FZ from"node:https";import{URL as WZ}from"node:url";import{readFileSync as AJ}from"node:fs";import{inflateSync as dQ}from"node:zlib";//! Full-BMP glyph atlas (Spleen 5x8 for ASCII/code + Unifont fallback),
|
|
10
10
|
//! extracted from pxpipe's generated gray atlas by `tools/gen-glyphs.mjs`.
|
|
11
11
|
//!
|
|
12
12
|
//! Codepoints + wide flags load eagerly (~175 KB, needed for wrap math);
|
|
13
13
|
//! coverage pixels stay zlib-packed on disk and decompress lazily on the
|
|
14
14
|
//! first blit, so `estimate` never pays for them.
|
|
15
|
-
var h=5,e=8,d=
|
|
15
|
+
var h=5,e=8,d=AJ(new URL("../assets/glyphs.cps",import.meta.url)),YJ=AJ(new URL("../assets/glyphs.wide",import.meta.url)),uJ=(()=>{let J=d.byteLength>>>2;if((d.byteOffset&3)===0)return new Uint32Array(d.buffer,d.byteOffset,J);return new Uint32Array(d.buffer.slice(d.byteOffset,d.byteOffset+(J<<2)))})(),lQ=(()=>{let J=new Uint32Array(YJ.length),Q=0;for(let Z=0;Z<YJ.length;Z++)J[Z]=Q,Q+=(YJ[Z]===1?2*h:h)*e;return J})(),RJ=null;function JJ(J){let Q=0,Z=uJ.length-1;while(Q<=Z){let $=Q+Z>>>1,V=uJ[$];if(V<J)Q=$+1;else if(V>J)Z=$-1;else return $}return-1}function n(J){return YJ[J]===1}function PJ(J){if(RJ===null)RJ=dQ(AJ(new URL("../assets/glyphs.pix.z",import.meta.url)));let Q=lQ[J],Z=n(J)?2*h:h;return RJ.subarray(Q,Q+Z*e)}function gJ(J,Q,Z){let $=PJ(J),V=n(J)?2*h:h;return pQ($,V,e,Q,Z)}function pQ(J,Q,Z,$,V){let z=new Uint8Array($*V),G=Q/$,Y=Z/V;for(let H=0;H<V;H++){let K=H*Y,B=(H+1)*Y;for(let U=0;U<$;U++){let O=U*G,X=(U+1)*G,D=0,M=0,E=Math.min(Math.ceil(B),Z);for(let S=Math.floor(K);S<E;S++){let k=Math.min(S+1,B)-Math.max(S,K);if(k<=0)continue;let y=Math.min(Math.ceil(X),Q);for(let C=Math.floor(O);C<y;C++){let x=Math.min(C+1,X)-Math.max(C,O);if(x<=0)continue;let f=x*k;D+=f*J[S*Q+C],M+=f}}z[H*$+U]=M>0?Math.min(255,Math.max(0,Math.round(D/M))):0}}return z}import{deflateSync as nQ}from"node:zlib";//! Minimal grayscale PNG encoder (bit depth 8, color type 0), mirroring
|
|
16
16
|
//! pxpipe's png.ts: IHDR + one IDAT (zlib) + IEND, filter byte 0 per row.
|
|
17
|
-
var
|
|
17
|
+
var cQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $=0;$<8;$++)Z=(Z&1)!==0?3988292384^Z>>>1:Z>>>1;J[Q]=Z>>>0}return J})();function _J(J,Q,Z,$){let V=$.length;J[Q]=V>>>24&255,J[Q+1]=V>>>16&255,J[Q+2]=V>>>8&255,J[Q+3]=V&255;let z=Q+4;for(let H=0;H<4;H++)J[z+H]=Z.charCodeAt(H);J.set($,z+4);let G=4294967295;for(let H=z;H<z+4+V;H++)G=cQ[(G^J[H])&255]^G>>>8;G=~G>>>0;let Y=z+4+V;return J[Y]=G>>>24&255,J[Y+1]=G>>>16&255,J[Y+2]=G>>>8&255,J[Y+3]=G&255,Y+4}function dJ(J,Q,Z){let $=new Uint8Array(Z*(Q+1));for(let H=0,K=0,B=0;H<Z;H++)$[K++]=0,$.set(J.subarray(B,B+Q),K),K+=Q,B+=Q;let V=nQ($,{level:6}),z=new Uint8Array(33+(12+V.length)+12);z[0]=137,z[1]=80,z[2]=78,z[3]=71,z[4]=13,z[5]=10,z[6]=26,z[7]=10;let G=new Uint8Array(13);G[0]=Q>>>24&255,G[1]=Q>>>16&255,G[2]=Q>>>8&255,G[3]=Q&255,G[4]=Z>>>24&255,G[5]=Z>>>16&255,G[6]=Z>>>8&255,G[7]=Z&255,G[8]=8,G[9]=0;let Y=_J(z,8,"IHDR",G);return Y=_J(z,Y,"IDAT",V),_J(z,Y,"IEND",new Uint8Array(0)),z}//! Stage 2: the `pxpipe` imaging engine, ported from pxpipe's render.ts
|
|
18
18
|
//! production dense path (bare 5x8 AA cell, 312 cols, 1568x728 pages).
|
|
19
19
|
//! Glyphs cover BMP (pxpipe's atlas: Spleen for ASCII/code, Unifont fallback)
|
|
20
20
|
//! PLUS the astral planes (unifont_upper, incl. emoji) — beyond pxpipe, which
|
|
@@ -29,19 +29,19 @@ var nQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $
|
|
|
29
29
|
//! * `font` — `Tiny` renders the same atlas box-filtered into a 4x6 cell
|
|
30
30
|
//! (390 cols x 120 rows/page), ~40% fewer image-tokens; opt-in,
|
|
31
31
|
//! transcription-accuracy gated.
|
|
32
|
-
var QJ=4,
|
|
32
|
+
var QJ=4,HJ=4,oQ=1568,iQ=728,KJ="↵",aQ="⏎",BJ="→",rQ="⇢",EJ="⇥",tQ="⇨",sQ=9647,lJ=4,eQ=3,pJ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",JZ=/\n{4,}/g;function v(J){return J.length===4&&J.replace(/[A-Z]/g,(Q)=>Q.toLowerCase())==="tiny"?"tiny":"normal"}function iJ(J){let Q=J==="tiny"?4:h,Z=J==="tiny"?6:e,$=Math.floor((oQ-2*QJ)/Q),V=Math.floor((iQ-2*HJ)/Z);return{cw:Q,ch:Z,cols:$,maxLines:V,maxChars:$*V}}var QZ=(()=>{let J=new Uint8Array(128);for(let Q=0;Q<128;Q++){let Z=JJ(Q);J[Q]=Z>=0&&n(Z)?2:1}return J})();function SJ(J){if(J<128)return QZ[J];let Q=JJ(J);return Q>=0&&n(Q)?2:1}function ZZ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<56320&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<57344)Z++}Q++}return Q}function CJ(J){let Q=J.split(`
|
|
33
33
|
`);for(let Z=0;Z<Q.length;Z++){let $=Q[Z],V=$.length;while(V>0){let z=$.charCodeAt(V-1);if(z!==32&&z!==9)break;V--}if(V!==$.length)Q[Z]=$.slice(0,V)}return Q.join(`
|
|
34
|
-
`).replace(
|
|
34
|
+
`).replace(JZ,`
|
|
35
35
|
|
|
36
36
|
|
|
37
|
-
`)}function
|
|
38
|
-
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=
|
|
39
|
-
`);for(let Z=0;Z<Q.length;Z++)Q[Z]
|
|
40
|
-
`)){let z=
|
|
37
|
+
`)}function aJ(J){if(!J.includes("\t"))return J;let Q="",Z=0;for(let $ of J)if($==="\t"){let V=lJ-Z%lJ;Q+=BJ;for(let z=1;z<V;z++)Q+=" ";Z+=V}else Q+=$,Z+=SJ($.codePointAt(0));return Q}function $Z(J){let Q=J.includes("\t")?J.replaceAll("\t",BJ):J,Z=0;while(Z<Q.length&&Q.charCodeAt(Z)===32)Z++;if(Z>=eQ&&Z<pJ.length)return EJ+pJ[Z]+Q.slice(Z);return Q}function rJ(J){return J.includes(KJ)?J.replaceAll(KJ,aQ):J}function VZ(J){let Q=rJ(J);if(Q.includes(BJ))Q=Q.replaceAll(BJ,rQ);if(Q.includes(EJ))Q=Q.replaceAll(EJ,tQ);return Q}function zZ(J){let Q=CJ(J).split(`
|
|
38
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=aJ(Q[Z]);return Q.join(KJ)}function GZ(J){let Q=CJ(J).split(`
|
|
39
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=$Z(Q[Z]);return Q.join(KJ)}function YZ(J,Q){let Z=[],$=CJ(J);for(let V of $.split(`
|
|
40
|
+
`)){let z=aJ(V);if(z.length===0){Z.push("");continue}let G="",Y=0;for(let H of z){let K=SJ(H.codePointAt(0));if(Y+K>Q)Z.push(G),G=H,Y=K;else G+=H,Y+=K}if(G.length!==0)Z.push(G)}return Z}function HZ(J,Q,Z){let $=[],V=[],z=0;for(let G of J){let Y=ZZ(G),H=Y+(V.length!==0?1:0);if(V.length!==0&&(V.length>=Q||z+H>Z))$.push(V),V=[],z=0;z+=Y+(V.length!==0?1:0),V.push(G)}if(V.length!==0)$.push(V);return $}function tJ(J){if(J<32)return!0;if(J>=127&&J<=159)return!0;if(J>=768&&J<=879)return!0;if(J===8203||J===8204||J===8205||J===8288||J===65279)return!0;if(J>=65024&&J<=65039)return!0;if(J>=917760&&J<=917999)return!0;return!1}function KZ(J){let Q=null,Z=0;for(let $ of J){let V=$.codePointAt(0);if(JJ(V)<0&&!tJ(V)){if(Q===null)Q=J.slice(0,Z);Q+=`[U+${V.toString(16).toUpperCase()}]`}else if(Q!==null)Q+=$;Z+=$.length}return Q??J}function sJ(J,Q,Z,$){let V=Q?Z?GZ(VZ(J)):zZ(rJ(J)):J;return HZ(YZ(KZ(V),$.cols),$.maxLines,$.maxChars)}function eJ(J,Q,Z){if(!Z)return 2*QJ+Q.cols*Q.cw;let $=0;for(let G of J){let Y=0;for(let H of G)Y+=SJ(H.codePointAt(0));if(Y>Q.cols)Y=Q.cols;if(Y>$)$=Y}let V=2*QJ+$*Q.cw,z=2*QJ+Q.cw;return V>z?V:z}var nJ=new Map;function cJ(J,Q,Z,$,V,z,G){let Y=JJ(V);if(Y<0)return 0;let H=n(Y),K=H?2*z.cw:z.cw,B;if(G==="normal")B=PJ(Y);else{let U=nJ.get(Y);if(U===void 0)U=gJ(Y,K,z.ch),nJ.set(Y,U);B=U}for(let U=0;U<z.ch;U++){let O=($+U)*Q+Z,X=U*K;for(let D=0;D<K;D++){let M=B[X+D];if(M>0){let E=O+D;if(M>J[E])J[E]=M}}}return H?2:1}function BZ(J,Q,Z,$){let V=eJ(J,Q,Z),z=2*HJ+J.length*Q.ch,G=new Uint8Array(V*z),Y=0;for(let H=0;H<J.length;H++){let K=HJ+H*Q.ch,B=0;for(let U of J[H]){if(B>=Q.cols)break;let O=QJ+B*Q.cw,X=cJ(G,V,O,K,U.codePointAt(0),Q,$);if(X===0){if(X=1,tJ(U.codePointAt(0)));else if(Y++,U!==" ")cJ(G,V,O,K,sQ,Q,$)}B+=X}}for(let H=0;H<G.length;H++)G[H]=255-G[H];return{png:dJ(G,V,z),width:V,height:z,dropped:Y}}function m(J,Q,Z,$){let V=iJ($),z=sJ(J,Q,Z,V).map((K)=>BZ(K,V,Z,$)),G=0,Y=0,H=0;for(let K of z)G+=K.width*K.height,Y+=JQ(K.width,K.height),H+=K.dropped;return{pages:z,pixels:G,tokens:Y,dropped:H}}function OJ(J,Q,Z,$){let V=iJ($),z=sJ(J,Q,Z,V),G=0,Y=0;for(let H of z){let K=eJ(H,V,Z),B=2*HJ+H.length*V.ch;G+=K*B,Y+=JQ(K,B)}return{pages:z.length,pixels:G,tokens:Y}}var oJ=28;function JQ(J,Q){return Math.ceil(J/oJ)*Math.ceil(Q/oJ)}import{readFileSync as OZ}from"node:fs";import{join as UZ}from"node:path";//! pxpipe measurement-log summary (~/.pxpipe/events.jsonl), same math as the
|
|
41
41
|
//! node MCP: actual = every way input bytes get billed (input + cache reads +
|
|
42
42
|
//! cache creates) — ignoring cache_read would fake the savings.
|
|
43
|
-
class ZJ{value;constructor(J){this.value=J}}function
|
|
44
|
-
`)){if(K.trim().length===0)continue;let
|
|
43
|
+
class ZJ{value;constructor(J){this.value=J}}function TJ(){let J=process.env.TANUKI_EVENTS;if(J!==void 0)return J;let Q=process.env.HOME??"";return UZ(Q,".pxpipe","events.jsonl")}function c(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:0}function XZ(J){return J<0?-Math.round(-J):Math.round(J)}function QQ(){let J=TJ(),Q;try{Q=OZ(J,"utf8")}catch{return{available:!1,note:`no ${J} yet`}}let Z=0,$=0,V=0,z=0,G=0,Y=0;for(let K of Q.split(`
|
|
44
|
+
`)){if(K.trim().length===0)continue;let B;try{B=JSON.parse(K)}catch{continue}Z+=1;let U=B!==null&&typeof B==="object"&&!Array.isArray(B)?B:{};if(U.compressed===!0)$+=1,V+=c(U.orig_chars),z+=c(U.image_count);G+=c(U.baseline_tokens),Y+=c(U.input_tokens)+c(U.cache_read_tokens)+c(U.cache_create_tokens)}let H=G>0&&Y>0?new ZJ(XZ((1-Y/G)*1000)/10):null;return{available:!0,requests:Z,compressedRequests:$,imagedChars:V,imagesEmitted:z,baselineTokens:G,actualInputTokens:Y,estInputSavedPct:H}}//! Implicit mode: a local Anthropic middlebox, the pxpipe deployment shape
|
|
45
45
|
//! without pxpipe's structural flaw. Rules that keep it injection-shaped-free:
|
|
46
46
|
//!
|
|
47
47
|
//! 1. The system prompt and tool definitions are NEVER touched.
|
|
@@ -57,12 +57,12 @@ class ZJ{value;constructor(J){this.value=J}}function CJ(){let J=process.env.TANU
|
|
|
57
57
|
//!
|
|
58
58
|
//! Responses stream through untouched; usage is scraped from the stream for
|
|
59
59
|
//! the ~/.pxpipe/events.jsonl savings log (same format tanuki_stats reads).
|
|
60
|
-
var o={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,minSave:300,maxPages:20};function
|
|
61
|
-
`)}catch{}}function
|
|
60
|
+
var o={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,minSave:300,maxPages:20};function LJ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($<55296||$>56319)Q++}return Q}function jZ(J,Q){let Z=LJ(J);if(Z<Q.minChars)return null;let $=J;if(Q.distill)$=w($,null,2).distilled;let V=0;if(Q.codebook){let K=a($);$=K.text,V=K.entries}if(Q.level>0)$=s($,Q.level).compressed;let z=Math.round(Z/4),G=m($,!0,!0,Q.font);if(G.pages.length>Q.maxPages)return null;if(G.tokens>z*Q.ratio||z-G.tokens<Q.minSave)return null;let H=[{type:"text",text:`[tanuki-context: ${Z} chars imaged in place as ${G.pages.length} PNG page(s), ~${G.tokens} vs ~${z} text tokens. ↵=newline →=tab ⇥N=indent`+(V>0?`; ·legend· line maps ${V} sigils`:"")+"]"}];for(let K of G.pages)H.push({type:"image",source:{type:"base64",media_type:"image/png",data:Buffer.from(K.png.buffer,K.png.byteOffset,K.png.byteLength).toString("base64")}});return{blocks:H,origChars:Z,pages:G.pages.length,savedTokens:z-G.tokens}}function UJ(J){return J!==null&&typeof J==="object"&&!Array.isArray(J)}function NZ(J,Q){let Z;try{Z=JSON.parse(J)}catch{return null}if(!UJ(Z)||!Array.isArray(Z.messages))return null;let $=0,V=0,z=0,G=0,Y=new Map,H=(K)=>{let B=Y.get(K),U;if(B!==void 0){let O=LJ(K),X=`[tanuki-context: ${O} chars, byte-identical to a block imaged above (${B} PNG page(s)); not repeated]`;U={blocks:[{type:"text",text:X}],origChars:O,pages:0,savedTokens:Math.round(O/4)-Math.round(LJ(X)/4)}}else if(U=jZ(K,Q),U)Y.set(K,U.pages);if(U)$++,V+=U.origChars,z+=U.pages,G+=U.savedTokens;return U};for(let K=0;K<Z.messages.length-1;K++){let B=Z.messages[K];if(!UJ(B)||B.role!=="user")continue;if(typeof B.content==="string"){let O=H(B.content);if(O)B.content=O.blocks;continue}if(!Array.isArray(B.content))continue;let U=[];for(let O of B.content){if(!UJ(O)||O.cache_control!==void 0){U.push(O);continue}if(O.type==="text"&&typeof O.text==="string"){let X=H(O.text);if(X)U.push(...X.blocks);else U.push(O);continue}if(O.type==="tool_result"){if(typeof O.content==="string"){let X=H(O.content);if(X)O.content=X.blocks}else if(Array.isArray(O.content)){let X=[];for(let D of O.content){if(UJ(D)&&D.type==="text"&&typeof D.text==="string"&&D.cache_control===void 0){let M=H(D.text);if(M){X.push(...M.blocks);continue}}X.push(D)}O.content=X}}U.push(O)}B.content=U}if($===0)return null;return{body:JSON.stringify(Z),imagedBlocks:$,origChars:V,imageCount:z,savedTokens:G}}function IZ(J){let Q=(Z)=>{let $=Z.exec(J);return $?Number($[1]):0};return{input:Q(/"input_tokens"\s*:\s*(\d+)/),cacheRead:Q(/"cache_read_input_tokens"\s*:\s*(\d+)/),cacheCreate:Q(/"cache_creation_input_tokens"\s*:\s*(\d+)/)}}function RZ(J){try{let Q=TJ();qZ(MZ(Q),{recursive:!0}),DZ(Q,JSON.stringify(J)+`
|
|
61
|
+
`)}catch{}}function $Q(J){let Q=new WZ(J.upstream),Z=Q.protocol==="https:"?FZ:ZQ,$=ZQ.createServer((V,z)=>{let G=[];V.on("data",(Y)=>G.push(Y)),V.on("end",()=>{let Y=Buffer.concat(G),H=V.method==="POST"&&(V.url??"").startsWith("/v1/messages")&&!(V.url??"").includes("count_tokens")&&V.headers["content-encoding"]===void 0,K=null;if(H){if(K=NZ(Y.toString("utf8"),J),K)Y=Buffer.from(K.body,"utf8")}let B={...V.headers};if(delete B.host,delete B.connection,B["content-length"]=String(Y.length),H)delete B["accept-encoding"];let U=Z.request({protocol:Q.protocol,hostname:Q.hostname,port:Q.port||(Q.protocol==="https:"?443:80),path:V.url,method:V.method,headers:B},(O)=>{if(z.writeHead(O.statusCode??502,O.headers),H){let X=[];O.on("data",(D)=>{X.push(D),z.write(D)}),O.on("end",()=>{z.end();let D=IZ(Buffer.concat(X).toString("utf8")),M=D.input+D.cacheRead+D.cacheCreate;RZ({ts:Date.now(),tool:"proxy",compressed:K!==null,orig_chars:K?.origChars??0,image_count:K?.imageCount??0,baseline_tokens:M+(K?.savedTokens??0),input_tokens:D.input,cache_read_tokens:D.cacheRead,cache_create_tokens:D.cacheCreate})})}else O.pipe(z)});U.on("error",(O)=>{z.writeHead(502,{"content-type":"application/json"}),z.end(JSON.stringify({type:"error",error:{type:"api_error",message:`tanuki proxy: upstream unreachable (${O.message})`}}))}),U.end(Y)})});return $.listen(J.port,"127.0.0.1",()=>{let V=$.address(),z=V!==null&&typeof V==="object"?V.port:J.port,G=`level=${J.level} distill=${J.distill} codebook=${J.codebook} font=${J.font} minChars=${J.minChars} ratio=${J.ratio} minSave=${J.minSave}`;process.stderr.write(`tanuki-context proxy on http://127.0.0.1:${z} -> ${J.upstream}
|
|
62
62
|
${G}
|
|
63
63
|
`+` rules: system prompt & tools untouched · in-place blocks only · latest message never imaged · cache_control skipped · identical blocks imaged once
|
|
64
64
|
`+` point your client at it: export ANTHROPIC_BASE_URL=http://127.0.0.1:${z}
|
|
65
|
-
`)}),$}import{createHash as
|
|
65
|
+
`)}),$}import{createHash as AZ}from"node:crypto";import{mkdirSync as PZ,readFileSync as _Z,writeFileSync as EZ}from"node:fs";import VQ from"node:process";//! Stash mode: context-mode's shape (content parked outside the context
|
|
66
66
|
//! window, queried on demand) fused with tanuki's pricing (big answers come
|
|
67
67
|
//! back as dense pages).
|
|
68
68
|
//!
|
|
@@ -71,9 +71,9 @@ var o={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,min
|
|
|
71
71
|
//! repeats, first/last lines, the id). fetch = pull a slice by regex query
|
|
72
72
|
//! (distill-powered) or line range; the caller images it only when pages
|
|
73
73
|
//! clearly win. Contract is byte-identical with the Rust engine.
|
|
74
|
-
function
|
|
74
|
+
function zQ(){let J=VQ.env.TANUKI_STASH;if(J!==void 0&&J!=="")return J;return`${VQ.env.HOME??""}/.tanuki/stash`}function $J(J){let Q=AZ("sha256").update(J,"utf8").digest("hex").slice(0,12),Z=zQ();PZ(Z,{recursive:!0}),EZ(`${Z}/${Q}`,J);let $=Buffer.byteLength(J,"utf8"),V=J.split(`
|
|
75
75
|
`),z=w(J,null,2).stats,G=[`stashed ${Q} · ${$} bytes · ${V.length} lines`,`distill map: ${z.origLines} -> ${z.outLines} lines · ${z.savedPct}% of chars removable · ${z.importantKept} error/warn lines`];if(z.topRepeats.length>0){G.push("top repeats:");for(let H of z.topRepeats.slice(0,5)){let K=H.kind==="template"?" (template)":"";G.push(` ×${H.count}${K} ${H.exemplar}`)}}let Y="";for(let H=V.length-1;H>=0;H--)if(V[H]!==""){Y=V[H];break}return G.push(`first: ${t(r(V[0]),160)}`),G.push(`last: ${t(r(Y),160)}`),G.push(`fetch: tanuki_fetch {"id":"${Q}","query":"<regex>"} or {"id":"${Q}","lines":"a-b"}`),{id:Q,overview:G.join(`
|
|
76
|
-
`)}}function yJ(J,Q,Z){if(Q===null===(Z===null))throw Error("give exactly one of query or lines");let $;try{$=
|
|
76
|
+
`)}}function yJ(J,Q,Z){if(Q===null===(Z===null))throw Error("give exactly one of query or lines");let $;try{$=_Z(`${zQ()}/${J}`,"utf8")}catch{throw Error(`unknown stash id: ${J}`)}if(Z!==null){let V=/^(\d+)-(\d+)$/.exec(Z);if(V===null)throw Error("bad lines range");let z=$.split(`
|
|
77
77
|
`),G=Math.max(1,Number(V[1])),Y=Math.min(z.length,Number(V[2]));if(Number(V[1])>Number(V[2]))throw Error("bad lines range");return z.slice(G-1,Y).join(`
|
|
78
78
|
`)}return w($,Q,2).distilled}//! tanuki-context — token-cutting context pipeline.
|
|
79
79
|
//! pipeline: text -> distill (stage 0, logs) -> ladder level 0-4 (stage 1)
|
|
@@ -84,26 +84,30 @@ function VQ(){let J=$Q.env.TANUKI_STASH;if(J!==void 0&&J!=="")return J;return`${
|
|
|
84
84
|
//! tanuki-context estimate <file> [level] [--distill]
|
|
85
85
|
//! tanuki-context render <file> [level] [outdir]
|
|
86
86
|
//! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
|
|
87
|
-
var
|
|
88
|
-
`;for(let
|
|
89
|
-
`;K+=H+
|
|
90
|
-
`+Z+"]"}let V=J,z=Object.keys(V).sort(
|
|
87
|
+
var TZ="0.4.1",wJ=6,bJ=8000;function LZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function P(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof ZJ){let H=J.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let B="[";for(let U=0;U<J.length;U++){if(U>0)B+=",";B+=P(J[U],!1)}return B+"]"}let H=Z+" ",K=`[
|
|
88
|
+
`;for(let B=0;B<J.length;B++){if(B>0)K+=`,
|
|
89
|
+
`;K+=H+P(J[B],!0,H)}return K+`
|
|
90
|
+
`+Z+"]"}let V=J,z=Object.keys(V).sort(LZ);if(z.length===0)return"{}";if(!Q){let H="{";for(let K=0;K<z.length;K++){if(K>0)H+=",";H+=JSON.stringify(z[K])+":"+P(V[z[K]],!1)}return H+"}"}let G=Z+" ",Y=`{
|
|
91
91
|
`;for(let H=0;H<z.length;H++){if(H>0)Y+=`,
|
|
92
|
-
`;Y+=G+JSON.stringify(z[H])+": "+
|
|
93
|
-
`+Z+"}"}function
|
|
94
|
-
`}if(
|
|
95
|
-
vs ~${H} text-tokens raw = TOTAL -${
|
|
96
|
-
`)}function
|
|
97
|
-
`),R.exit(101)}function
|
|
98
|
-
`);break}case"estimate":{let Q=J[2]??
|
|
99
|
-
`);break}case"render":{let Q=J[2]??
|
|
100
|
-
`);let
|
|
101
|
-
`);break}case"proxy":{let Q=(V,z)=>{let G=J.indexOf(V);if(G===-1||J[G+1]===void 0)return z;let Y=Number(J[G+1]);return Number.isFinite(Y)?Y:z},Z=J.indexOf("--upstream"),$=J.indexOf("--font")
|
|
102
|
-
`);break}case"fetch":{let Q=J[2]??
|
|
92
|
+
`;Y+=G+JSON.stringify(z[H])+": "+P(V[z[H]],!0,G)}return Y+`
|
|
93
|
+
`+Z+"}"}function j(J,Q){return J!==null&&typeof J==="object"&&!Array.isArray(J)?J[Q]:void 0}function L(J){return typeof J==="string"?J:null}function XJ(J){return typeof J==="boolean"?J:null}function OQ(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:null}function _(J){let Q=0;for(let Z=0;Z<J.length;Z++){Q++;let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}}return Q}function qJ(J,Q,Z,$,V){let z,G=null;if(Z||$!==null){let K=w(J,$,2);z=K.distilled,G=K.stats}else z=J;let Y=0;if(V){let K=a(z);z=K.text,Y=K.entries}let H=s(z,Q);return{stage0:G,compressed:H.compressed,protectedLines:H.protectedLines,level:H.level,cbEntries:Y}}function l(J){return Math.round(J/4)}function zJ(J,Q){if(J===0)return 0;let Z=(1-Q/J)*100;return Z<0?-Math.round(-Z):Math.round(Z)}function UQ(J){return{text:L(j(J,"text"))??"",level:(OQ(j(J,"level"))??0)%256,distill:XJ(j(J,"distill"))??!1,query:L(j(J,"query")),reflow:XJ(j(J,"reflow"))??!0,pack:XJ(j(J,"pack"))??!0,font:L(j(J,"font"))??"normal",codebook:XJ(j(J,"codebook"))??!1}}function yZ(J){let Q=(z)=>{let G={codebook:!1,tokens:1/0,pages:0,text:z};for(let Y of[!1,!0]){let H=Y?a(z).text:z,K=OJ(H,!0,!0,v("normal"));if(K.tokens<G.tokens)G={codebook:Y,tokens:K.tokens,pages:K.pages,text:H}}return G},Z=Q(J),$=Q(w(J,null,2).distilled),V=OJ(Z.text,!0,!0,v("tiny"));return{codebook:Z.codebook,imageTokens:Z.tokens,pages:Z.pages,tinyImageTokens:V.tokens,withDistill:{codebook:$.codebook,imageTokens:$.tokens}}}function XQ(J){let Q=UQ(J),Z=qJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=v(Q.font),V=OJ(Z.compressed,Q.reflow,Q.pack,$),z=V.tokens,G=_(Q.text),Y=_(Z.compressed),H=l(G),[K,B]=GJ[Z.level];return{engine:"pxpipe",level:`${Z.level} ${K}`,loss:B,distill:Z.stage0,origChars:G,stage1Chars:Y,stage1SavedPct:zJ(G,Y),pages:V.pages,imageTokens:z,rawTextTokens:H,totalSavedPct:zJ(H,z),protectedLines:Z.protectedLines,pack:Q.pack,font:$==="tiny"?"tiny":"normal",codebook:Q.codebook?Z.cbEntries:!1,verdict:z<H?"PIPELINE cheaper":"TEXT cheaper",recommend:yZ(Q.text)}}function wZ(J){let Q=UQ(J),Z=qJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=v(Q.font),V=m(Z.compressed,Q.reflow,Q.pack,$),z=V.tokens,G=_(Q.text),Y=_(Z.compressed),H=l(G),[K,B]=GJ[Z.level],U="";if(Z.stage0!==null){let D=Z.stage0;U+=`distill: ${D.origLines} -> ${D.outLines} lines (-${D.savedPct}% chars, ${D.collapsedRuns} runs, ${D.suppressedLines} exact + ${D.templateSuppressed} template suppressed, ${D.importantKept} error/warn kept)
|
|
94
|
+
`}if(U+=`L${Z.level} ${K} (${B}): ${G} chars -> ${Y} chars (stage1 -${zJ(G,Y)}%) -> ${V.pages.length} page(s), ~${z} image-tokens
|
|
95
|
+
vs ~${H} text-tokens raw = TOTAL -${zJ(H,z)}%`,Z.protectedLines>0)U+=` · ${Z.protectedLines} lines kept verbatim`;if(V.dropped>0)U+=` · ${V.dropped} unmapped glyphs -> ▯`;if(Z.cbEntries>0)U+=` · codebook: ${Z.cbEntries} sigils (see ·legend·)`;if($==="tiny")U+=" · font: tiny 4x6";if(Q.pack)U+=" · packed (⇥N indent, → tab)";if(Q.reflow)U+=" · ↵ = newline · engine: pxpipe";let O=[{type:"text",text:U}],X=Math.min(V.pages.length,wJ);for(let D=0;D<X;D++){let M=V.pages[D].png;O.push({type:"image",data:Buffer.from(M.buffer,M.byteOffset,M.byteLength).toString("base64"),mimeType:"image/png"})}if(V.pages.length>wJ)O.push({type:"text",text:`(+${V.pages.length-wJ} more page(s))`});return O}function bZ(J){let Q=L(j(J,"text"))??"",Z=w(Q,L(j(J,"query")),2);return[{type:"text",text:P(Z.stats,!0)},{type:"text",text:Z.distilled}]}function vZ(J){let Q=L(j(J,"text"))??"",Z=(OQ(j(J,"level"))??1)%256,$=s(Q,Z),[V,z,G]=GJ[$.level],Y=_(Q),H=_($.compressed),K=l(Y),B=l(H),U={level:`${$.level} ${V}`,loss:z,note:G,origChars:Y,outChars:H,approxOrigTokens:K,approxOutTokens:B,savedPct:zJ(K,B),protectedLines:$.protectedLines};return[{type:"text",text:P(U,!0)},{type:"text",text:$.compressed}]}function kZ(J){let Q=L(j(J,"text"))??"";return[{type:"text",text:$J(Q).overview}]}function xZ(J){let Q=L(j(J,"id"))??"",Z=yJ(Q,L(j(J,"query"))??null,L(j(J,"lines"))??null),$=l(_(Z)),V=m(Z,!0,!0,v("normal"));if(!(V.tokens<=$*0.75&&$-V.tokens>=300&&V.pages.length<=6))return[{type:"text",text:Z}];let Y=[{type:"text",text:`[tanuki-context stash ${Q}: slice of ${_(Z)} chars imaged as ${V.pages.length} PNG page(s), ~${V.tokens} vs ~${$} text tokens. ↵=newline →=tab ⇥N=indent]`}];for(let H of V.pages){let K=H.png;Y.push({type:"image",data:Buffer.from(K.buffer,K.byteOffset,K.byteLength).toString("base64"),mimeType:"image/png"})}return Y}function fZ(){let J={type:"string"},Q={type:"integer",minimum:0,maximum:4};return{tools:[{name:"tanuki_render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow (single-cell tabs, ⇥N indent runs, width-trimmed pages). font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",inputSchema:{type:"object",properties:{text:J,level:Q,distill:{type:"boolean"},query:{type:"string"},reflow:{type:"boolean"},pack:{type:"boolean"},font:{type:"string",enum:["normal","tiny"]},codebook:{type:"boolean"}},required:["text"]}},{name:"tanuki_estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field prices the reversible knobs (pack/codebook, level 0) and, separately under 'withDistill', the lossy-but-counted log route - one call replaces manual knob probing.",inputSchema:{type:"object",properties:{text:J,level:Q,distill:{type:"boolean"},query:{type:"string"},reflow:{type:"boolean"},pack:{type:"boolean"},font:{type:"string",enum:["normal","tiny"]},codebook:{type:"boolean"}},required:["text"]}},{name:"tanuki_distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes (exact + same-template) with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. Deterministic, order-preserving.",inputSchema:{type:"object",properties:{text:J,query:{type:"string"}},required:["text"]}},{name:"tanuki_compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",inputSchema:{type:"object",properties:{text:J,level:Q},required:["text"]}},{name:"tanuki_stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates).",inputSchema:{type:"object",properties:{}}},{name:"tanuki_stash",description:"Park bulky text outside the context window (content-addressed file under TANUKI_STASH or ~/.tanuki/stash) and get back a compact map: distill stats, top repeats, first/last lines, and the stash id. Pay a few hundred tokens now, fetch slices later - the retrieval pattern, with tanuki pricing on the way back.",inputSchema:{type:"object",properties:{text:J},required:["text"]}},{name:"tanuki_fetch",description:"Pull a slice of stashed text by id: query (regex, distill-powered: matches + error/warn lines + context) or lines 'a-b'. Big slices come back as dense PNG pages automatically when they clearly win (>=25% and >=300 tokens cheaper, <=6 pages); small ones stay text.",inputSchema:{type:"object",properties:{id:{type:"string"},query:{type:"string"},lines:{type:"string"}},required:["id"]}}]}}function hZ(J){let Q=L(j(J,"name"))??"",Z=j(J,"arguments"),$;switch(Q){case"tanuki_render":$=wZ(Z);break;case"tanuki_estimate":$=[{type:"text",text:P(XQ(Z),!0)}];break;case"tanuki_distill":$=bZ(Z);break;case"tanuki_compress":$=vZ(Z);break;case"tanuki_stats":$=[{type:"text",text:P(QQ(),!0)}];break;case"tanuki_stash":$=kZ(Z);break;case"tanuki_fetch":try{$=xZ(Z)}catch(V){return{ok:!1,error:V instanceof Error?V.message:String(V)}}break;default:return{ok:!1,error:`unknown tool: ${Q}`}}return{ok:!0,value:{content:$}}}function HQ(J){let Q=J.endsWith("\r")?J.slice(0,-1):J;if(Q.trim().length===0)return;let Z;try{Z=JSON.parse(Q)}catch{return}let $=Z!==null&&typeof Z==="object"&&!Array.isArray(Z)&&Object.prototype.hasOwnProperty.call(Z,"id"),V=$?Z.id??null:null,z=L(j(Z,"method")),G;switch(z){case"initialize":{let Y=L(j(j(Z,"params"),"protocolVersion"))??"2025-06-18";G={jsonrpc:"2.0",id:V,result:{protocolVersion:Y,capabilities:{tools:{}},serverInfo:{name:"tanuki-context",version:TZ}}};break}case"notifications/initialized":case"notifications/cancelled":G=void 0;break;case"ping":G={jsonrpc:"2.0",id:V,result:{}};break;case"tools/list":G={jsonrpc:"2.0",id:V,result:fZ()};break;case"tools/call":{let Y=hZ(j(Z,"params"));G=Y.ok?{jsonrpc:"2.0",id:V,result:Y.value}:{jsonrpc:"2.0",id:V,error:{code:-32602,message:Y.error}};break}default:G=$?{jsonrpc:"2.0",id:V,error:{code:-32601,message:"method not found"}}:void 0}if(G!==void 0)R.stdout.write(P(G,!1)+`
|
|
96
|
+
`)}function mZ(){let J=Buffer.alloc(0);R.stdin.on("data",(Q)=>{let Z=J.length>0?Buffer.concat([J,Q]):Q,$=0,V;while((V=Z.indexOf(10,$))!==-1)HQ(Z.toString("utf8",$,V)),$=V+1;J=Z.subarray($)}),R.stdin.on("end",()=>{if(J.length>0)HQ(J.toString("utf8")),J=Buffer.alloc(0)})}function A(J){R.stderr.write(J+`
|
|
97
|
+
`),R.exit(101)}function VJ(J){try{return CZ(J,"utf8")}catch{A("read file")}}function DJ(J,Q){if(!/^\+?[0-9]+$/.test(J))return null;let Z=BigInt(J.startsWith("+")?J.slice(1):J);return Z>Q?null:Number(Z)}var KQ=255n,BQ=0xffffffffffffffffn;function DQ(){let J=R.argv.slice(1);switch(J[1]){case"distill":{let Q=J[2]??A("usage: tanuki-context distill <file> [query]"),Z=VJ(Q),$=w(Z,J[3]??null,2);R.stdout.write(P($.stats,!1)+`
|
|
98
|
+
`);break}case"estimate":{let Q=J[2]??A("usage: tanuki-context estimate <file> [level] [--distill] [--no-pack] [--font tiny] [--codebook]"),Z=VJ(Q),$=J.slice(3).filter((H)=>!H.startsWith("--")),V=$.length>0?DJ($[0],BQ)??0:0,z=J.indexOf("--font"),G=z!==-1&&J[z+1]!==void 0?J[z+1]:"normal",Y=XQ({text:Z,level:V,distill:J.includes("--distill"),pack:!J.includes("--no-pack"),font:G,codebook:J.includes("--codebook")});R.stdout.write(P(Y,!1)+`
|
|
99
|
+
`);break}case"render":{let Q=J[2]??A("usage: tanuki-context render <file> [level] [outdir] [--distill] [--no-pack] [--font tiny] [--codebook]"),Z=VJ(Q),$=J.slice(3).filter((X)=>!X.startsWith("--")),V=$.length>0?DJ($[0],KQ)??0:0,z=!J.includes("--no-pack"),G=J.includes("--codebook"),Y=J.indexOf("--font"),H=v(Y!==-1&&J[Y+1]!==void 0?J[Y+1]:"normal"),K=qJ(Z,V,J.includes("--distill"),null,G),B=m(K.compressed,!0,z,H),U=B.tokens;R.stdout.write(P({pages:B.pages.length,imageTokens:U,dropped:B.dropped,rawTextTokens:l(_(Z))},!1)+`
|
|
100
|
+
`);let O=$[1];if(O!==void 0){try{GQ(O,{recursive:!0})}catch{A("mkdir")}for(let X=0;X<B.pages.length;X++)try{YQ(`${O}/page${X}.png`,B.pages[X].png)}catch{A("write png")}}break}case"bench":{let Q=J[2]??A("usage: tanuki-context bench <file> <op> [level] [runs] [--distill]"),Z=J[3]??"pipeline",$=J[4]!==void 0?DJ(J[4],KQ)??0:0,V=J[5]!==void 0?DJ(J[5],BQ)??3:3,z=J.includes("--distill"),G=VJ(Q),Y=[],H=null;for(let K=0;K<=V;K++){let B=performance.now();if(Z==="distill")H=w(G,null,2).stats;else{let U=qJ(G,$,z,null,!1),O=m(U.compressed,!0,!1,"normal");H={pages:O.pages.length,imageTokens:O.tokens,stage1Chars:_(U.compressed),dropped:O.dropped}}if(K>0)Y.push(performance.now()-B)}if(Y.sort((K,B)=>K-B),Y.length===0)A("index out of bounds: the len is 0 but the index is 0");R.stdout.write(P({medianMs:new ZJ(Y[Math.floor(Y.length/2)]),runs:V,result:H},!1)+`
|
|
101
|
+
`);break}case"proxy":{let Q=(V,z)=>{let G=J.indexOf(V);if(G===-1||J[G+1]===void 0)return z;let Y=Number(J[G+1]);return Number.isFinite(Y)?Y:z},Z=J.indexOf("--upstream"),$=J.indexOf("--font");$Q({port:Q("--port",8484),upstream:Z!==-1&&J[Z+1]!==void 0?J[Z+1]:R.env.TANUKI_UPSTREAM??"https://api.anthropic.com",level:Q("--level",o.level),distill:J.includes("--distill"),codebook:J.includes("--codebook"),font:v($!==-1&&J[$+1]!==void 0?J[$+1]:"normal"),minChars:Q("--min-chars",o.minChars),ratio:Q("--ratio",o.ratio),minSave:Q("--min-save",o.minSave),maxPages:Q("--max-pages",o.maxPages)});break}case"stash":{let Q=J[2]??A("usage: tanuki-context stash <file>"),Z=$J(VJ(Q));R.stdout.write(Z.overview+`
|
|
102
|
+
`);break}case"fetch":{let Q=J[2]??A("usage: tanuki-context fetch <id> [outdir] [--query re] [--lines a-b]"),Z=J.indexOf("--query"),$=J.indexOf("--lines"),V="";try{V=yJ(Q,Z!==-1?J[Z+1]??null:null,$!==-1?J[$+1]??null:null)}catch(B){A(B instanceof Error?B.message:String(B))}let z=l(_(V)),G=m(V,!0,!0,v("normal"));if(!(G.tokens<=z*0.75&&z-G.tokens>=300&&G.pages.length<=6)){R.stdout.write(P({mode:"text"},!1)+`
|
|
103
103
|
`+V+`
|
|
104
|
-
`);break}R.stdout.write(
|
|
105
|
-
`);let H=new Set([Z+1,$+1]),K;for(let
|
|
106
|
-
|
|
104
|
+
`);break}R.stdout.write(P({imageTokens:G.tokens,mode:"pages",pages:G.pages.length,rawTextTokens:z},!1)+`
|
|
105
|
+
`);let H=new Set([Z+1,$+1]),K;for(let B=3;B<J.length;B++)if(!J[B].startsWith("--")&&!H.has(B)){K=J[B];break}if(K!==void 0){try{GQ(K,{recursive:!0})}catch{A("mkdir")}for(let B=0;B<G.pages.length;B++)try{YQ(`${K}/page${B}.png`,G.pages[B].png)}catch{A("write png")}}break}case"run":{let Q=J.indexOf("--"),Z=Q!==-1?J.slice(Q+1):[];if(Z.length===0)A("usage: tanuki-context run [--query re] -- <command> [args...]");let $=J.indexOf("--query"),V=$!==-1&&$<Q?J[$+1]??null:null,z=SZ(Z[0],Z.slice(1),{encoding:"utf8",maxBuffer:268435456});if(z.error!==void 0)A(`spawn failed: ${z.error.message}`);let G=(z.stdout??"")+((z.stderr??"")!==""?`
|
|
106
|
+
--- stderr ---
|
|
107
|
+
${z.stderr}`:""),Y=z.status??0,H=w(G,V,2),K=H.stats,B=[`[tanuki run] exit ${Y} · ${K.origLines} -> ${K.outLines} lines · ${K.savedPct}% of chars removed`];if(_(H.distilled)<=bJ||_(G)<=bJ){if(B.push(H.distilled),_(G)>bJ){let U=$J(G);B.push(`full output stashed: tanuki-context fetch ${U.id} [--query re] [--lines a-b]`)}}else{let U=$J(G);B.push(U.overview)}R.stdout.write(B.join(`
|
|
108
|
+
`)+`
|
|
109
|
+
`),R.exit(Y)}case"serve":case void 0:mZ();break;default:R.stderr.write(`unknown command: ${J[1]}
|
|
110
|
+
usage: tanuki-context [serve|proxy|distill|estimate|render|bench|stash|fetch|run] ...
|
|
107
111
|
`),R.exit(1)}}//! Bin entry: keeps src/main.ts importable as a library (src/agent.ts) without
|
|
108
112
|
//! starting the MCP server as an import side effect.
|
|
109
|
-
|
|
113
|
+
DQ();
|
package/dist/pi.js
CHANGED
|
@@ -11,5 +11,5 @@ import{spawn as W}from"node:child_process";import{fileURLToPath as X}from"node:u
|
|
|
11
11
|
//! ToolResult content shape, so results pass through untouched.
|
|
12
12
|
function Y(){let F=process.env.TANUKI_BIN;if(F)return{cmd:F,args:[]};return{cmd:process.execPath,args:[X(new URL("./cli.js",import.meta.url))]}}class Q{proc;buf="";nextId=1;pending=new Map;ready;constructor(){let{cmd:F,args:B}=Y();this.proc=W(F,B,{stdio:["pipe","pipe","inherit"]}),this.proc.stdout.setEncoding("utf8"),this.proc.stdout.on("data",(G)=>{this.buf+=G;let z;while((z=this.buf.indexOf(`
|
|
13
13
|
`))!==-1){let H=this.buf.slice(0,z).trim();if(this.buf=this.buf.slice(z+1),!H)continue;let D;try{D=JSON.parse(H)}catch{continue}let J=D.id!==void 0?this.pending.get(D.id):void 0;if(!J)continue;if(this.pending.delete(D.id),D.error)J.reject(Error(D.error.message??"tanuki-context error"));else J.resolve(D.result)}}),this.proc.on("exit",(G)=>{let z=Error(`tanuki-context server exited (code ${G})`);for(let H of this.pending.values())H.reject(z);this.pending.clear()}),this.ready=this.request("initialize",{})}request(F,B){let G=this.nextId++,{promise:z,resolve:H,reject:D}=Promise.withResolvers();if(!this.proc.stdin?.writable)return D(Error("tanuki-context server is gone")),z;return this.pending.set(G,{resolve:H,reject:D}),this.proc.stdin.write(JSON.stringify({jsonrpc:"2.0",id:G,method:F,params:B})+`
|
|
14
|
-
`),z}async call(F,B){return await this.ready,this.request("tools/call",{name:F,arguments:B})}kill(){this.proc.kill()}get alive(){return this.proc.exitCode===null&&!this.proc.killed}}var K=q.String({description:"The text to process"}),V=q.Optional(q.Integer({minimum:0,maximum:4,description:"0 raw · 1 whitespace · 2 prose · 3 dense · 4 caveman"})),N=q.Object({text:K,level:V,distill:q.Optional(q.Boolean()),query:q.Optional(q.String()),reflow:q.Optional(q.Boolean()),pack:q.Optional(q.Boolean()),font:q.Optional(q.String({enum:["normal","tiny"]})),codebook:q.Optional(q.Boolean())}),Z=[{name:"tanuki_render",label:"Tanuki Render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow. font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",parameters:N,snippet:"Render bulky text/logs as dense PNG pages that cost a fraction of the tokens"},{name:"tanuki_estimate",label:"Tanuki Estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field
|
|
14
|
+
`),z}async call(F,B){return await this.ready,this.request("tools/call",{name:F,arguments:B})}kill(){this.proc.kill()}get alive(){return this.proc.exitCode===null&&!this.proc.killed}}var K=q.String({description:"The text to process"}),V=q.Optional(q.Integer({minimum:0,maximum:4,description:"0 raw · 1 whitespace · 2 prose · 3 dense · 4 caveman"})),N=q.Object({text:K,level:V,distill:q.Optional(q.Boolean()),query:q.Optional(q.String()),reflow:q.Optional(q.Boolean()),pack:q.Optional(q.Boolean()),font:q.Optional(q.String({enum:["normal","tiny"]})),codebook:q.Optional(q.Boolean())}),Z=[{name:"tanuki_render",label:"Tanuki Render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow. font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",parameters:N,snippet:"Render bulky text/logs as dense PNG pages that cost a fraction of the tokens"},{name:"tanuki_estimate",label:"Tanuki Estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field prices the reversible knobs (pack/codebook, level 0) and, separately under 'withDistill', the lossy-but-counted log route - one call replaces manual knob probing.",parameters:N,snippet:"Instant token verdict: would imaging this text beat sending it as text?"},{name:"tanuki_distill",label:"Tanuki Distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. Deterministic, order-preserving.",parameters:q.Object({text:K,query:q.Optional(q.String())}),snippet:"Deterministically dedupe noisy logs, keeping every error line verbatim"},{name:"tanuki_compress",label:"Tanuki Compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",parameters:q.Object({text:K,level:V}),snippet:"Graded text compression (lossless whitespace up to gist-only)"},{name:"tanuki_stats",label:"Tanuki Stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates).",parameters:q.Object({}),snippet:"Session savings summary from the tanuki/pxpipe event log"},{name:"tanuki_stash",label:"Tanuki Stash",description:"Park bulky text outside the context window (content-addressed file under TANUKI_STASH or ~/.tanuki/stash) and get back a compact map: distill stats, top repeats, first/last lines, and the stash id. Pay a few hundred tokens now, fetch slices later - the retrieval pattern, with tanuki pricing on the way back.",parameters:q.Object({text:K}),snippet:"Park huge text on disk for a few hundred tokens of map; fetch slices later"},{name:"tanuki_fetch",label:"Tanuki Fetch",description:"Pull a slice of stashed text by id: query (regex, distill-powered: matches + error/warn lines + context) or lines 'a-b'. Big slices come back as dense PNG pages automatically when they clearly win (>=25% and >=300 tokens cheaper, <=6 pages); small ones stay text.",parameters:q.Object({id:q.String(),query:q.Optional(q.String()),lines:q.Optional(q.String())}),snippet:"Pull a stashed slice; big answers arrive as cheap dense pages"}];function $(F){let B=null,G=()=>{if(!B||!B.alive)B=new Q;return B};F.on("session_shutdown",async()=>{B?.kill(),B=null});for(let z of Z)F.registerTool({name:z.name,label:z.label,description:z.description,promptSnippet:z.snippet,parameters:z.parameters,async execute(H,D){let J=await G().call(z.name,D??{}),M=(J.content??[]).map((I)=>I.type==="image"?{type:"image",data:I.data??"",mimeType:I.mimeType??"image/png"}:{type:"text",text:I.text??""});if(J.isError)throw Error(M.map((I)=>("text"in I)?I.text:"").join(`
|
|
15
15
|
`)||z.name+" failed");return{content:M,details:{}}}})}export{$ as default};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tanuki-context",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.1",
|
|
4
4
|
"description": "Token-cutting context pipeline as an MCP server: distill logs, compress text, render dense PNG pages (pxpipe imaging engine). Zero dependencies.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
"files": [
|
|
18
18
|
"dist",
|
|
19
19
|
"assets",
|
|
20
|
+
"skills",
|
|
20
21
|
"README.md",
|
|
21
22
|
"LICENSE"
|
|
22
23
|
],
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: tanuki-context
|
|
3
|
+
version: 0.4.1
|
|
4
|
+
description: |
|
|
5
|
+
Cut input-token cost by rendering bulky text (logs, command output, long
|
|
6
|
+
docs) as dense PNG pages the model reads at a fraction of the price, or by
|
|
7
|
+
parking it outside context and fetching slices. Use when pasting or reading
|
|
8
|
+
anything over ~2,000 tokens of logs, build output, or documents; when a
|
|
9
|
+
session is close to its context limit; or before re-reading a large file.
|
|
10
|
+
Requires the tanuki-context MCP server (tanuki_* tools) and a
|
|
11
|
+
vision-capable model.
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# tanuki-context: pay pixels, not tokens
|
|
15
|
+
|
|
16
|
+
Text costs ~1 token per 4 characters. A dense PNG page costs a fixed 1,456
|
|
17
|
+
tokens and carries up to 28,080 characters. The tanuki_* tools exploit that
|
|
18
|
+
gap deterministically: nothing is summarized by a model, errors stay
|
|
19
|
+
verbatim, and every drop is counted.
|
|
20
|
+
|
|
21
|
+
## Workflow
|
|
22
|
+
|
|
23
|
+
1. **Estimate first, always.** `tanuki_estimate { text }` is instant and
|
|
24
|
+
never renders pixels. Read two things from the answer:
|
|
25
|
+
- `recommend` - the cheapest reversible knob set (pack/codebook), priced.
|
|
26
|
+
- `recommend.withDistill` - the log route (repeats collapsed with exact
|
|
27
|
+
counts). Cheaper, and honest for logs; do not use it on source code.
|
|
28
|
+
2. **Act on the verdict.**
|
|
29
|
+
- `"PIPELINE cheaper"` and you need the content in front of you ->
|
|
30
|
+
`tanuki_render` with the recommended knobs. Use the returned pages
|
|
31
|
+
instead of pasting the text.
|
|
32
|
+
- You only need parts of it, now or later -> `tanuki_stash { text }`
|
|
33
|
+
(returns a ~300-token map + id), then `tanuki_fetch { id, query }` or
|
|
34
|
+
`{ id, lines: "a-b" }`. Big slices arrive as pages automatically.
|
|
35
|
+
- `"TEXT cheaper"` -> just use the text. Small inputs are not worth an
|
|
36
|
+
image even when the math technically favors one.
|
|
37
|
+
3. **For noisy logs**, add `distill: true` (and `query: "regex"` for a
|
|
38
|
+
slice). Error/warn/fail lines survive verbatim; repeats become one
|
|
39
|
+
exemplar plus an exact xN count.
|
|
40
|
+
4. **Shell commands with chatty output**: run them as
|
|
41
|
+
`tanuki-context run -- <cmd>` instead of reading the firehose. Exit code
|
|
42
|
+
passes through; the full capture is stashed and fetchable.
|
|
43
|
+
|
|
44
|
+
## Reading the pages
|
|
45
|
+
|
|
46
|
+
`↵` = original newline · `→` = tab · `⇥N` = N leading spaces · a trailing
|
|
47
|
+
`·legend·` line maps codebook sigils back to full tokens · `[U+XXXX]` = a
|
|
48
|
+
codepoint the atlas has no glyph for · `[×N similar]` = N near-identical
|
|
49
|
+
lines collapsed here.
|
|
50
|
+
|
|
51
|
+
## Do not
|
|
52
|
+
|
|
53
|
+
- Do not image content whose exact bytes you must retype later (secrets,
|
|
54
|
+
hashes, code you are about to edit). Rendering is exact; model read-back
|
|
55
|
+
of dense hex is not guaranteed.
|
|
56
|
+
- Do not use `font: "tiny"` or ladder levels 2-4 on anything you may need
|
|
57
|
+
to quote. Tiny is a 99.7%-read-back trade; levels 2-4 reword prose.
|
|
58
|
+
- Do not use `withDistill` numbers on source code or docs you want intact -
|
|
59
|
+
distill collapses similar-looking lines.
|
|
60
|
+
- Do not probe knob combinations by hand; `recommend` already priced them.
|