tanuki-context 0.4.1 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -20
- package/dist/agent.js +70 -37
- package/dist/cli.js +83 -50
- package/dist/pi.js +1 -1
- package/package.json +1 -1
- package/skills/tanuki-context/SKILL.md +21 -3
package/README.md
CHANGED
|
@@ -73,6 +73,21 @@ No AI client at all, just curious what it would save you:
|
|
|
73
73
|
npx tanuki-context estimate some-big-file.log 0
|
|
74
74
|
```
|
|
75
75
|
|
|
76
|
+
Price the decision in real dollars — a cache-read token costs ~0.1× a fresh
|
|
77
|
+
one, so `--cached` flips the verdict when the text would ride the prompt cache:
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
npx tanuki-context estimate some-big-file.log 0 --model claude-opus-4 --cached
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Structured JSON (NDJSON, API dumps)? The `--table` knob states every key
|
|
84
|
+
once — on a 200 KB `journalctl -o json` slice it turns 51,182 raw tokens
|
|
85
|
+
into 5,320 with `--codebook`:
|
|
86
|
+
|
|
87
|
+
```
|
|
88
|
+
npx tanuki-context estimate journal.ndjson 0 --table --codebook
|
|
89
|
+
```
|
|
90
|
+
|
|
76
91
|
## Installing it: two ways, same tools
|
|
77
92
|
|
|
78
93
|
**npm (the default).** `npx -y tanuki-context` downloads a 0.98 MB tarball
|
|
@@ -89,7 +104,7 @@ stashed text and `~/.pxpipe/events.jsonl` for the savings log.
|
|
|
89
104
|
cargo install --git https://github.com/Osyna/tanuki-context --branch rust
|
|
90
105
|
```
|
|
91
106
|
|
|
92
|
-
One 5.7 MB binary, same seven tools, same numbers — a
|
|
107
|
+
One 5.7 MB binary, same seven tools, same numbers — a 161-check parity
|
|
93
108
|
harness holds the two engines to byte-identical JSON and pixel-identical
|
|
94
109
|
PNGs on every release. It spawns in 0.4 ms and idles at 3.8 MB RSS, which
|
|
95
110
|
matters when every session forks its own server. Point any client config
|
|
@@ -121,14 +136,20 @@ Leave it alone when:
|
|
|
121
136
|
switch even when the math technically favors it; `estimate` and the
|
|
122
137
|
proxy gate both say so.
|
|
123
138
|
- **your bill is output-dominated.** tanuki cuts input tokens only. If
|
|
124
|
-
most of your spend is the model's own output, fix that first
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
-
|
|
130
|
-
|
|
131
|
-
|
|
139
|
+
most of your spend is the model's own output, fix that first —
|
|
140
|
+
`tanuki_stats` now reports the output share of the bill, so you can see
|
|
141
|
+
when this applies instead of guessing.
|
|
142
|
+
- **you are not on Anthropic pricing.** Pass `model` to `tanuki_estimate`
|
|
143
|
+
and the `cost` verdict counts image tokens with that provider's own rule —
|
|
144
|
+
OpenAI 512-px high-detail tiles (85 + 170/tile), Gemini 768-px tiles
|
|
145
|
+
(258/tile, flagged `~approximate`; their usage field is authoritative) —
|
|
146
|
+
and prices text at that provider's cache-read ratio. Dollars stay labeled
|
|
147
|
+
list prices, overridable via `TANUKI_RATES`.
|
|
148
|
+
- **the bulk is already prompt-cached.** Cache reads cost a tenth of fresh
|
|
149
|
+
input, so imaging content that was riding the cache is usually a net loss.
|
|
150
|
+
Pass `cached:true` to `tanuki_estimate` and the `cost` verdict computes it —
|
|
151
|
+
it flips to "TEXT cheaper" exactly when the cache makes text the cheaper byte.
|
|
152
|
+
The proxy never touches `cache_control` blocks for the same reason.
|
|
132
153
|
|
|
133
154
|
## The three ways to run it
|
|
134
155
|
|
|
@@ -198,9 +219,9 @@ All modes run the same engine and log their savings to the same file, which
|
|
|
198
219
|
|
|
199
220
|
| tool | what it does | when to reach for it |
|
|
200
221
|
| ----------------- | ----------------------------------------------------------- | ---------------------------------------- |
|
|
201
|
-
| `tanuki_estimate` | prices
|
|
222
|
+
| `tanuki_estimate` | prices every route — knobs, `table`, provider-real `cost` — returns `recommend` | always first; it's free and instant |
|
|
202
223
|
| `tanuki_render` | the full pipeline, returns PNG pages + a breakdown | when estimate says the pipeline wins |
|
|
203
|
-
| `tanuki_distill` | de-noises a log, output stays text
|
|
224
|
+
| `tanuki_distill` | de-noises a log (`table:true` for NDJSON), output stays text | when you want greppable text, not images |
|
|
204
225
|
| `tanuki_compress` | text-only compression, five levels | prose you'll paraphrase anyway |
|
|
205
226
|
| `tanuki_stash` | parks text on disk, returns a ~300-token map + an id | huge references you'll consult, not read |
|
|
206
227
|
| `tanuki_fetch` | pulls a slice by regex or line range; auto-imaged when big | after a stash, when you actually need it |
|
|
@@ -319,6 +340,9 @@ escapes. Nothing disappears silently.
|
|
|
319
340
|
| stash + fetch | park text on disk, return a distill map + content-address id | retrieval economics with awareness and imaged slices |
|
|
320
341
|
| `run` wrapper | wrap any command; distilled output inline, full capture stashed | -70% of chars on chatty commands, before tokenization |
|
|
321
342
|
| progress-frame collapse | `\r` spinner frames reduce to what the terminal showed | build/download logs stop paying per frame |
|
|
343
|
+
| `table` | whole-JSON input becomes a `·cols·` header + tab-joined JSON cells | keys stated once: -33% on journald NDJSON before imaging even starts |
|
|
344
|
+
| provider-real cost | `estimate {model}` counts image tokens by that provider's tile rule and prices its cache ratio | verdicts stop being Anthropic-only guesses |
|
|
345
|
+
| output share | the proxy logs `output_tokens`; `tanuki_stats` reports the share | names the part of the bill no input-side tool can cut |
|
|
322
346
|
|
|
323
347
|
## What wins where
|
|
324
348
|
|
|
@@ -358,7 +382,7 @@ The same table as advice:
|
|
|
358
382
|
| build/tool output, live | `tanuki-context run -- <cmd>` | -70% of chars before tokens even enter |
|
|
359
383
|
| source code | reversible knobs only (`pack` is on by default) | **-80%**, byte-exact |
|
|
360
384
|
| docs / prose | reversible knobs; tiny font if read-back risk is fine | -79 to -87% |
|
|
361
|
-
| structured JSON |
|
|
385
|
+
| structured JSON (arrays, NDJSON) | `table: true` + codebook — `recommend` probes it unprompted | **-86 to -90%** reversible |
|
|
362
386
|
| "I'll only need two slices of this" | `tanuki_stash` + `tanuki_fetch` | ~300-token map, slices from ~112 |
|
|
363
387
|
| "one narrow answer, never the file" | [context-mode](https://www.npmjs.com/package/context-mode)-style retrieval, or a stash you never fetch from | ~270/question |
|
|
364
388
|
| wordy prose that must stay text | ladder levels 2-4 (caveman) | -9% on real docs; it is the weakest tool here |
|
|
@@ -368,6 +392,30 @@ level 4 — we ship it, and the guard that makes it safe on code is exactly
|
|
|
368
392
|
what makes it a no-op there. Text-side rewording just does not compete
|
|
369
393
|
with pixel pricing.
|
|
370
394
|
|
|
395
|
+
### Structured JSON: the table knob
|
|
396
|
+
|
|
397
|
+
An array of objects or an NDJSON stream (`journalctl -o json`, docker
|
|
398
|
+
events, API dumps) repeats every key on every row. `table: true` states the
|
|
399
|
+
keys once — a `·cols·` header line, then one row per line, cells as
|
|
400
|
+
tab-separated compact JSON. Value-lossless by construction (an absent key is
|
|
401
|
+
an empty cell; a JSON cell is never empty), decodable without the model, and
|
|
402
|
+
the size gate leaves tiny or mixed inputs alone. Uniform rows then collapse
|
|
403
|
+
harder under distill and codebook.
|
|
404
|
+
|
|
405
|
+
Fresh `journalctl -o json` slice, 200 KB (51,182 raw text tokens), measured
|
|
406
|
+
2026-07-26 on this machine:
|
|
407
|
+
|
|
408
|
+
| stack | image tokens | vs raw text |
|
|
409
|
+
| --- | ---: | ---: |
|
|
410
|
+
| imaging alone (reversible) | 10,752 | -79% |
|
|
411
|
+
| + `table` | 7,280 | -86% |
|
|
412
|
+
| + `table` + `codebook` | 5,320 | **-90%** |
|
|
413
|
+
| + `table` + `codebook` + `distill` | 4,424 | -91% |
|
|
414
|
+
| + tiny font on top | 3,192 | -94% |
|
|
415
|
+
|
|
416
|
+
You do not have to know the knob exists: `recommend` prices the table route
|
|
417
|
+
on every estimate call and reports `table: true` when it wins.
|
|
418
|
+
|
|
371
419
|
## Benchmarks
|
|
372
420
|
|
|
373
421
|
Density knobs, measured by `reference/methods-report.mjs` (regenerate it
|
|
@@ -395,7 +443,7 @@ the reference server is the original node MCP wrapping pxpipe's library):
|
|
|
395
443
|
| distill a 12 MB log | 0.42 s | 0.31 s | 0.28 s | - |
|
|
396
444
|
| install | 0.98 MB tarball, zero deps | same | 5.7 MB static binary | node_modules tree |
|
|
397
445
|
|
|
398
|
-
Correctness:
|
|
446
|
+
Correctness: 61 TypeScript tests, 48 Rust tests, and a 161-check parity
|
|
399
447
|
harness that holds the two engines to byte-identical JSON and
|
|
400
448
|
pixel-identical PNGs on every knob combination, including distilled
|
|
401
449
|
renders and a full MCP session with error paths.
|
|
@@ -486,9 +534,9 @@ Responses stream through untouched. Savings land in
|
|
|
486
534
|
`~/.pxpipe/events.jsonl`, with the baseline named: what was billed plus
|
|
487
535
|
what the imaged blocks would have cost as text.
|
|
488
536
|
|
|
489
|
-
Knobs: `--port N` `--upstream URL` `--level 0-4` `--distill` `--
|
|
490
|
-
`--font tiny` `--min-chars N` `--ratio X` `--min-save N`
|
|
491
|
-
Defaults are conservative: level 0, nothing lossy on.
|
|
537
|
+
Knobs: `--port N` `--upstream URL` `--level 0-4` `--distill` `--table`
|
|
538
|
+
`--codebook` `--font tiny` `--min-chars N` `--ratio X` `--min-save N`
|
|
539
|
+
`--max-pages N`. Defaults are conservative: level 0, nothing lossy on.
|
|
492
540
|
|
|
493
541
|
## Clients
|
|
494
542
|
|
|
@@ -595,9 +643,9 @@ options = ClaudeAgentOptions(
|
|
|
595
643
|
```
|
|
596
644
|
npx tanuki-context # MCP stdio server (default)
|
|
597
645
|
npx tanuki-context proxy [--port 8484] [--upstream URL] [knobs] # implicit relay
|
|
598
|
-
npx tanuki-context distill <file> [query] # stats JSON to stdout
|
|
599
|
-
npx tanuki-context estimate <file> [level] [--distill] [--no-pack] [--font tiny] [--codebook]
|
|
600
|
-
npx tanuki-context render <file> [level] [outdir] [--distill] [--no-pack] [--font tiny] [--codebook]
|
|
646
|
+
npx tanuki-context distill <file> [query] [--table] # stats JSON to stdout
|
|
647
|
+
npx tanuki-context estimate <file> [level] [--distill] [--table] [--no-pack] [--font tiny] [--codebook] [--model <id>] [--cached]
|
|
648
|
+
npx tanuki-context render <file> [level] [outdir] [--distill] [--table] [--no-pack] [--font tiny] [--codebook]
|
|
601
649
|
npx tanuki-context bench <file> <distill|pipeline> [level] [runs] # in-process timing
|
|
602
650
|
npx tanuki-context stash <file> # park text, print the map + id
|
|
603
651
|
npx tanuki-context fetch <id> [outdir] [--query re] [--lines a-b]
|
|
@@ -657,7 +705,7 @@ Node-compatible files:
|
|
|
657
705
|
```
|
|
658
706
|
bun run build # dist/cli.js + dist/agent.js + dist/pi.js
|
|
659
707
|
bun test # 48 tests
|
|
660
|
-
bun run parity # TS vs rust binary,
|
|
708
|
+
bun run parity # TS vs rust binary, 161 checks (needs TANUKI_BIN)
|
|
661
709
|
```
|
|
662
710
|
|
|
663
711
|
Regenerating glyphs after a pxpipe atlas rebuild (needs a pxpipe checkout
|
package/dist/agent.js
CHANGED
|
@@ -1,19 +1,51 @@
|
|
|
1
|
-
import{createRequire as
|
|
2
|
-
·legend· `;for(let
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
1
|
+
import{createRequire as WZ}from"node:module";var TQ=WZ(import.meta.url);import{existsSync as A$}from"node:fs";import{fileURLToPath as _$}from"node:url";function FZ(Q){if(Q===32||Q>=9&&Q<=13)return!0;if(Q<133)return!1;return Q===133||Q===160||Q===5760||Q>=8192&&Q<=8202||Q===8232||Q===8233||Q===8239||Q===8287||Q===12288}function BQ(Q){let Z=0;for(let $=0;$<Q.length;$++){let J=Q.charCodeAt($);if(J>=55296&&J<=56319&&$+1<Q.length){let G=Q.charCodeAt($+1);if(G>=56320&&G<=57343)$++}Z++}return Z}function OQ(Q,Z){let $=0,J=0;while($<Q.length&&J<Z.length){let G=Q.codePointAt($),V=Z.codePointAt(J);if(G!==V)return G-V;$+=G>65535?2:1,J+=V>65535?2:1}return Q.length-$-(Z.length-J)}function QQ(Q){let Z=new Map,$=(B)=>{Z.set(B,(Z.get(B)??0)+1)},J=(B)=>{if(BQ(B)>=12)$(B);if(B.includes("/")){let D=B.split("/"),M="";for(let U=0;U<D.length;U++){if(U>0)M+="/";if(M+=D[U],U>=2){let I=M+"/";if(BQ(I)>=12)$(I)}}}},G=-1;for(let B=0;B<Q.length;){let D=Q.codePointAt(B),M=D>65535?2:1;if(FZ(D)){if(G>=0)J(Q.slice(G,B)),G=-1}else if(G<0)G=B;B+=M}if(G>=0)J(Q.slice(G));let V=[];for(let[B,D]of Z)if(D>=3){let M=BQ(B);V.push({k:B,c:D,len:M,saved:(M-1)*D})}V.sort((B,D)=>D.saved-B.saved||OQ(B.k,D.k));let Y=[];for(let B of"§¤¢£¥µ¶ª°±¬×÷ØÞßæðøþ¡¿")if(!Q.includes(B))Y.push(B);let K=[],H=[];for(let{k:B,c:D,len:M}of V){if(K.length>=Y.length)break;if((M-1)*D<=M+3)continue;let U=!1;for(let R=0;R<H.length;R++){let T=H[R];if(T.startsWith(B)||B.startsWith(T)){U=!0;break}}if(U)continue;let I=Y[K.length];H.push(B),K.push({sig:I,val:B,len:M})}if(K.length===0)return{text:Q,entries:0};let X=K.map((B,D)=>D);X.sort((B,D)=>K[D].len-K[B].len);let O=Q;for(let B=0;B<X.length;B++){let{sig:D,val:M}=K[X[B]];O=O.replaceAll(M,D)}let z=`
|
|
2
|
+
·legend· `;for(let B=0;B<K.length;B++)z+=K[B].sig+"="+K[B].val+" ";return O+=z.slice(0,-1),{text:O,entries:K.length}}//! Situation-aware cost model — the "codeburn calculation." tanuki's verdict
|
|
3
|
+
//! compares token COUNTS, which equals real cost only when both sides bill at
|
|
4
|
+
//! the same per-token rate. They do not: on Anthropic a cache-read costs ~0.1×
|
|
5
|
+
//! a fresh input token, while image (visual) tokens bill AT the input rate. So
|
|
6
|
+
//! the cheapest technique depends on the SITUATION — is the text already cached
|
|
7
|
+
//! this turn, and which provider prices it. Only the RATIOS drive the verdict;
|
|
8
|
+
//! absolute $/Mtok (list prices, overridable via TANUKI_RATES) drive the
|
|
9
|
+
//! optional dollar figure. Image-token COUNTS are provider-correct when page
|
|
10
|
+
//! dims are supplied: Anthropic 28px patches, OpenAI 512px high-detail tiles
|
|
11
|
+
//! (85 + 170/tile), Gemini 768px tiles (258/tile, ~approximate — their crop
|
|
12
|
+
//! rule has undocumented edges; the API usage field is authoritative).
|
|
13
|
+
function jZ(Q){return Q<0?-Math.round(-Q):Math.round(Q)}function yQ(Q){return Math.round(Q*1e6)/1e6}var NZ="2026-07",ZQ={opus:{input:15,output:75,cacheReadMult:0.1,imageMult:1,family:"anthropic"},sonnet:{input:3,output:15,cacheReadMult:0.1,imageMult:1,family:"anthropic"},haiku:{input:1,output:5,cacheReadMult:0.1,imageMult:1,family:"anthropic"},gpt:{input:1.25,output:10,cacheReadMult:0.5,imageMult:1,family:"openai"},gemini:{input:1.25,output:10,cacheReadMult:0.25,imageMult:1,family:"gemini"},default:{input:3,output:15,cacheReadMult:0.1,imageMult:1,family:"anthropic"}};function IZ(){let Q=process.env.TANUKI_RATES;if(Q===void 0||Q.length===0)return ZQ;try{let Z=JSON.parse(Q),$={...ZQ};for(let[J,G]of Object.entries(Z))$[J]={...$[J]??ZQ.default,...G};return $}catch{return ZQ}}function PZ(Q){let Z=IZ(),$=(Q??"").toLowerCase();for(let J of["opus","sonnet","haiku","gpt","gemini"])if($.includes(J)&&Z[J]!==void 0)return{key:J,rate:Z[J]};return{key:"default",rate:Z.default}}function RZ(Q,Z){let $=0;for(let[J,G]of Q)if(Z==="openai"){let V=J,Y=G,K=Math.min(1,2048/Math.max(V,Y));V*=K,Y*=K;let H=Math.min(1,768/Math.min(V,Y));V=Math.ceil(V*H),Y=Math.ceil(Y*H),$+=85+170*(Math.ceil(V/512)*Math.ceil(Y/512))}else $+=J<=384&&G<=384?258:258*(Math.ceil(J/768)*Math.ceil(G/768));return $}function wQ(Q,Z,$,J){let{key:G,rate:V}=PZ($.model),Y=$.cached===!0,K=V.family!=="anthropic"&&J!==void 0?RZ(J.dims,V.family):Z,H=V.input/1e6,X=H*(Y?V.cacheReadMult:1),O=H*V.imageMult,z=Q*X,B=K*O,D=O>0?Math.floor(Q*X/O):1/0,M=B<z?"PIPELINE":"TEXT",U=z>0?jZ((1-B/z)*100):0,I=[];if(V.family==="openai")I.push(J!==void 0?"image tokens counted with OpenAI's high-detail tile rule (85 + 170 per 512px tile)":"no page dims supplied; image count falls back to Anthropic's 28px patch grid — approximate for openai");else if(V.family==="gemini")I.push(J!==void 0?"~approximate: Gemini's documented 768px-tile rule (258/tile); the API usage field is authoritative":"no page dims supplied; image count falls back to Anthropic's 28px patch grid — approximate for gemini");if(Y)I.push(`text priced at cache-read rate (${V.cacheReadMult}× input); imaging already-cached content usually loses`);return{model:G,cached:Y,ratesAsOf:NZ,imageTokens:K,textUsd:yQ(z),imageUsd:yQ(B),cheaper:M,savedPct:U,breakevenImageTokens:D,note:I.length>0?I.join("; "):void 0}}//! Columnar codec for structured JSON — the one domain where retrieval-store
|
|
14
|
+
//! compressors (Headroom's SmartCrusher) beat tanuki's line tools, done
|
|
15
|
+
//! tanuki-style instead: deterministic, no model in the loop, decode grammar
|
|
16
|
+
//! documented. An array of objects (or NDJSON — journalctl -o json, docker
|
|
17
|
+
//! events, API dumps) repeats every key on every row; state the keys ONCE:
|
|
18
|
+
//!
|
|
19
|
+
//! ·cols·<TAB>"k1"<TAB>"k2"...
|
|
20
|
+
//! cell<TAB>cell... one line per row, cells = compact JSON
|
|
21
|
+
//!
|
|
22
|
+
//! An absent key is an empty cell (a JSON cell is never empty, so this is
|
|
23
|
+
//! unambiguous). Compact JSON escapes every control char, so a raw tab can
|
|
24
|
+
//! never appear inside a cell. Round-trip contract: same VALUES in canonical
|
|
25
|
+
//! layout — whitespace and per-object key order are serialization, not
|
|
26
|
+
//! content (nested objects re-print with sorted keys, exactly serde_json's
|
|
27
|
+
//! default Value ordering, which the custom serializer in main.ts already
|
|
28
|
+
//! mimics engine-wide).
|
|
29
|
+
//!
|
|
30
|
+
//! ponytail: whole-input tables only — mixed prose+JSON stays text; add
|
|
31
|
+
//! block detection if a real corpus ever demands it.
|
|
32
|
+
function bQ(Q){let Z=0;for(let $=0;$<Q.length;$++){let J=Q.charCodeAt($);if(J<55296||J>56319)Z++}return Z}function qQ(Q){if(Q===null||typeof Q==="boolean")return JSON.stringify(Q);if(typeof Q==="number"){if(Number.isSafeInteger(Q))return String(Q);return String(Q).replace("e+","e")}if(typeof Q==="string")return JSON.stringify(Q);if(Array.isArray(Q))return`[${Q.map(qQ).join(",")}]`;let Z=Q;return`{${Object.keys(Z).sort().map((J)=>`${JSON.stringify(J)}:${qQ(Z[J])}`).join(",")}}`}function AZ(Q){let Z=(J)=>J!==null&&typeof J==="object"&&!Array.isArray(J);try{let J=JSON.parse(Q);if(Array.isArray(J)){if(J.length>=2&&J.every(Z))return J;return null}}catch{}let $=[];for(let J of Q.split(`
|
|
33
|
+
`)){if(J.trim().length===0)continue;try{let G=JSON.parse(J);if(!Z(G))return null;$.push(G)}catch{return null}}return $.length>=2?$:null}function n(Q){let Z=AZ(Q);if(Z===null)return null;let $=new Set;for(let Y of Z)for(let K of Object.keys(Y))$.add(K);let J=[...$].sort(OQ);if(J.length===0)return null;let G=["·cols·"+"\t"+J.map((Y)=>JSON.stringify(Y)).join("\t")];for(let Y of Z)G.push(J.map((K)=>(K in Y)?qQ(Y[K]):"").join("\t"));let V=G.join(`
|
|
34
|
+
`);if(bQ(V)>=bQ(Q))return null;return{text:V,rows:Z.length,cols:J.length}}import{Buffer as fQ}from"node:buffer";var _Z=/\x1b\[[0-9;]*[A-Za-z]/g,DQ=/\b([0-9A-Za-z_]*(error|exception)s?|err|warn(ing)?s?|fail(s|ed|ure|ures)?|panic(s|ked)?|fatal|critical|traceback|denied|refused|timeouts?|timed.?out|assert(s|ed|ion|ions)?|segfault(s|ed)?)\b/i,CZ=/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/g,EZ=/\b[0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?\b/g,SZ=/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,LZ=/\b[0-9a-f]{7,64}\b/gi,TZ=/\b[0-9]+(\.[0-9]+)?[ \t\u00a0]?(ms|us|µs|ns|s|m|h|%|[KMGT]i?B)?\b/gi,yZ=/[0-9a-f]/i,wZ=3,bZ=8,MQ=2,vQ=40;function fZ(Q){if(!yZ.test(Q))return Q;let Z=Q.replace(CZ,"<ts>");return Z=Z.replace(EZ,"<time>"),Z=Z.replace(SZ,"<uuid>"),Z=Z.replace(LZ,"<hex>"),Z.replace(TZ,"<n>")}function g(Q){if(Q===32)return!0;if(Q<9)return!1;if(Q<=13)return!0;if(Q<133)return!1;return Q===133||Q===160||Q===5760||Q>=8192&&Q<=8202||Q===8232||Q===8233||Q===8239||Q===8287||Q===12288}function c(Q){let Z=0,$=Q.length;while(Z<$&&g(Q.charCodeAt(Z)))Z++;while($>Z&&g(Q.charCodeAt($-1)))$--;return Z===0&&$===Q.length?Q:Q.slice(Z,$)}function vZ(Q,Z){let $=0,J=Q.length;while($<J&&g(Q.charCodeAt($)))$++;while(J>$&&g(Q.charCodeAt(J-1)))J--;let G=0;for(let V=$;V<J&&G<Z;G++){let Y=Q.charCodeAt(V);V+=Y>=55296&&Y<=56319&&V+1<J?2:1}return G<Z}function xZ(Q){let Z=0,$=Q.length;while(Z<$&&g(Q.charCodeAt(Z)))Z++;return Q.charCodeAt(Z)===91&&Q.charCodeAt(Z+1)===215}function hZ(Q){let Z=[],$=-1,J=!0,G=!1,V=!1,Y=!0;for(let K=0;K<Q.length;){let H=Q.codePointAt(K),X=H>65535?2:1,O=g(H);if(Y){if(Y=!1,O)Z.push("<v>")}if(O){if($>=0)Z.push(J?Q.slice($,K):"<v>"),$=-1;V=!0}else{if(G=!0,V=!1,$<0)$=K,J=!0;if(J){let z=H|32;if(z<97||z>122||H>127)J=!1}}K+=X}if($>=0)Z.push(J?Q.slice($):"<v>");if(G&&V)Z.push("<v>");return Z.join(" ")}function o(Q,Z){let $=0,J=0,G=Q.length;while($<G&&J<Z){let V=Q.charCodeAt($);$+=V>=55296&&V<=56319&&$+1<G?2:1,J++}return $>=G?Q:Q.slice(0,$)}var kZ=/[.*+?^${}()|[\]\\]/g;function b(Q,Z=null,$=2){let J=$,V=Q.replace(_Z,"").split(`
|
|
35
|
+
`).map((q)=>{let W=q.endsWith("\r")?q.slice(0,-1):q,F=W.lastIndexOf("\r");return F===-1?W:W.slice(F+1)}),Y=V.length,K=Y,H=Array(Y),X=Array(Y),O=0;for(let q=0;q<Y;q++){H[q]=fZ(V[q]);let W=DQ.test(V[q]);if(X[q]=W,W)O++}let z=[],B=[],D=[],M=0,U=0;while(U<Y){let q=0,W=0;for(let F=1;F<=bZ&&U+2*F<=Y;F++){if(X[U+F-1])break;let P=1;Q:for(;;){let j=U+P*F;if(j+F>Y)break;for(let _=0;_<F;_++)if(X[j+_]||H[j+_]!==H[U+_])break Q;P++}if(P>=wZ&&F*(P-1)>q*(W>0?W-1:0))q=F,W=P}if(q>0){for(let F=U;F<U+q;F++)z.push(V[F]),B.push(X[F]),D.push(H[F]);z.push(q===1?` [×${W} similar]`:` [×${W} similar ${q}-line blocks]`),B.push(!1),D.push(null),M++,U+=q*W}else z.push(V[U]),B.push(X[U]),D.push(H[U]),U++}let I=new Map,R=new Map,T=MQ+1,C=[],A=[],y=0,w=0;for(let q=0;q<z.length;q++){let W=z[q],F=B[q];if(F||xZ(W)||vZ(W,4)){C.push(W),A.push(F);continue}let P=D[q],j=I.get(P);if(j!==void 0){if(j.count++,j.count<=MQ)C.push(W),A.push(!1);else y++;continue}let _=hZ(P);I.set(P,{count:1,exemplar:W});let l=R.get(_);if(l!==void 0)if(l.count++,l.count<=T)C.push(W),A.push(!1);else w++;else R.set(_,{count:1,exemplar:W}),C.push(W),A.push(!1)}let x=[];for(let q of I.values())if(q.count>MQ)x.push([q.count,"exact",q.exemplar]);for(let q of R.values())if(q.count>T)x.push([q.count,"template",q.exemplar]);if(x.sort((q,W)=>W[0]-q[0]),x.length>vQ)x.length=vQ;if(y+w>0){let q=`── ${y+w} repeated lines suppressed (${y} exact ×N, ${w} same-template; first occurrences kept above) ──`;C.push(q),A.push(DQ.test(q));for(let[W,F,P]of x){let _=` ×${W}${F==="template"?" (template)":""} ${o(c(P),160)}`;C.push(_),A.push(DQ.test(_))}}let h;if(Z!=null){let q;try{q=new RegExp(Z,"i")}catch{q=new RegExp(Z.replace(kZ,"\\$&"),"i")}let W=C.length,F=new Uint8Array(W);for(let j=0;j<W;j++)if(A[j]||q.test(C[j])){let _=j>J?j-J:0,l=Math.min(j+J,W-1);F.fill(1,_,l+1)}h=[];let P=0;for(let j=0;j<W;j++)if(F[j]){if(P>0)h.push(`… ${P} lines omitted`),P=0;h.push(C[j])}else P++;if(P>0)h.push(`… ${P} lines omitted`)}else h=C;let EQ=h.join(`
|
|
36
|
+
`),SQ=fQ.byteLength(Q),LQ=fQ.byteLength(EQ),XQ=(1-LQ/SQ)*100,DZ=Q.length===0?0:XQ<0?-Math.round(-XQ):Math.round(XQ),MZ=x.map(([q,W,F])=>({count:q,exemplar:o(c(F),160),kind:W})),UZ={collapsedRuns:M,importantKept:O,origChars:SQ,origLines:K,outChars:LQ,outLines:h.length,query:Z??null,savedPct:DZ,suppressedLines:y,templateSuppressed:w,topRepeats:MZ};return{distilled:EQ,stats:UZ}}var $Q=[["none","none","passthrough (baseline)"],["whitespace","lossless","trailing whitespace + blank-line runs collapsed; safe for code"],["prose","light","L1 + prose lines: collapse spaces, cut redundant filler phrases (code/IDs protected)"],["dense","medium","L2 + prose: drop articles & intensifiers"],["caveman","heavy","L3 + prose: telegraphic — drop function words; gist only, NOT verbatim"]];var E=(Q)=>new RegExp(`(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:${Q})(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])`,"giu"),UQ=[[E("in order to"),"to"],[E("due to the fact that"),"because"],[E("at this point in time"),"now"],[E("in the event that"),"if"],[E("for the purpose of"),"for"],[E("with regard to"),"about"],[E("a large number of"),"many"],[E("it is important to note that"),""],[E("please note that"),""],[E("as a matter of fact"),""],[E("in terms of"),"for"],[E("the fact that"),"that"]],mZ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:the|an|a)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),uZ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:very|really|just|actually|basically|simply|quite|rather|essentially|literally)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),gZ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:is|are|was|were|am|be|been|being|do|does|did|have|has|had|will|would|shall|should|can|could|may|might|of|to|in|on|at|for|with|that|this|these|those|it|its|there|here)(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]*","giu"),xQ=/ {2,}/g,dZ=new RegExp("[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+([.,;:!?])","gu"),pZ=/\n{3,}/g;function WQ(Q){if(Q===32||Q>=9&&Q<=13)return!0;if(Q<133)return!1;return Q===133||Q===160||Q===5760||Q>=8192&&Q<=8202||Q===8232||Q===8233||Q===8239||Q===8287||Q===12288}function lZ(Q){let Z=0,$=Q.length;while(Z<$){let J=Q.codePointAt(Z);if(!WQ(J))break;Z+=J>65535?2:1}while($>Z){let J=Q.charCodeAt($-1);if(!WQ(J))break;$--}return Z===0&&$===Q.length?Q:Q.slice(Z,$)}function nZ(Q){if(Q.length===0)return!1;let Z=Q.charCodeAt(0);if(Z===32||Z===9)return!0;let $=0,J=0,G=0,V=!1;for(let Y=0;Y<Q.length;){let K=Q.codePointAt(Y);if(Y+=K>65535?2:1,$++,WQ(K)){G=0;continue}if(G++,G>=24)V=!0;if(K>=48&&K<=57||K>=65&&K<=90||K>=97&&K<=122)continue;switch(K){case 46:case 44:case 59:case 58:case 39:case 34:case 33:case 63:case 40:case 41:case 45:continue}J++}if(J/$>0.3)return!0;return V}function cZ(Q,Z){let $=Q.replace(xQ," ");for(let G=0;G<UQ.length;G++)$=$.replace(UQ[G][0],UQ[G][1]);if(Z>=3)$=$.replace(mZ,""),$=$.replace(uZ,"");if(Z>=4)$=$.replace(gZ,"");$=$.replace(xQ," "),$=lZ($.replace(dZ,"$1"));let J=$.charCodeAt(0);if(J>=97&&J<=122)$=String.fromCharCode(J-32)+$.slice(1);return $}function JQ(Q,Z){let $=Math.min(Z,4);if($===0)return{compressed:Q,protectedLines:0,level:$};let J=0,G=Q.split(`
|
|
37
|
+
`),V=Array(G.length);for(let K=0;K<G.length;K++){let H=G[K],X=H.length;while(X>0){let z=H.charCodeAt(X-1);if(z!==32&&z!==9)break;X--}let O=X===H.length?H:H.slice(0,X);if($===1){V[K]=O;continue}if(nZ(O)){J++,V[K]=O;continue}V[K]=cZ(O,$)}return{compressed:V.join(`
|
|
38
|
+
`).replace(pZ,`
|
|
7
39
|
|
|
8
|
-
`),protectedLines
|
|
40
|
+
`),protectedLines:J,level:$}}import{readFileSync as jQ}from"node:fs";import{inflateSync as oZ}from"node:zlib";//! Full-BMP glyph atlas (Spleen 5x8 for ASCII/code + Unifont fallback),
|
|
9
41
|
//! extracted from pxpipe's generated gray atlas by `tools/gen-glyphs.mjs`.
|
|
10
42
|
//!
|
|
11
43
|
//! Codepoints + wide flags load eagerly (~175 KB, needed for wrap math);
|
|
12
44
|
//! coverage pixels stay zlib-packed on disk and decompress lazily on the
|
|
13
45
|
//! first blit, so `estimate` never pays for them.
|
|
14
|
-
var
|
|
46
|
+
var f=5,i=8,k=jQ(new URL("../assets/glyphs.cps",import.meta.url)),GQ=jQ(new URL("../assets/glyphs.wide",import.meta.url)),hQ=(()=>{let Q=k.byteLength>>>2;if((k.byteOffset&3)===0)return new Uint32Array(k.buffer,k.byteOffset,Q);return new Uint32Array(k.buffer.slice(k.byteOffset,k.byteOffset+(Q<<2)))})(),iZ=(()=>{let Q=new Uint32Array(GQ.length),Z=0;for(let $=0;$<GQ.length;$++)Q[$]=Z,Z+=(GQ[$]===1?2*f:f)*i;return Q})(),FQ=null;function r(Q){let Z=0,$=hQ.length-1;while(Z<=$){let J=Z+$>>>1,G=hQ[J];if(G<Q)Z=J+1;else if(G>Q)$=J-1;else return J}return-1}function d(Q){return GQ[Q]===1}function NQ(Q){if(FQ===null)FQ=oZ(jQ(new URL("../assets/glyphs.pix.z",import.meta.url)));let Z=iZ[Q],$=d(Q)?2*f:f;return FQ.subarray(Z,Z+$*i)}function kQ(Q,Z,$){let J=NQ(Q),G=d(Q)?2*f:f;return rZ(J,G,i,Z,$)}function rZ(Q,Z,$,J,G){let V=new Uint8Array(J*G),Y=Z/J,K=$/G;for(let H=0;H<G;H++){let X=H*K,O=(H+1)*K;for(let z=0;z<J;z++){let B=z*Y,D=(z+1)*Y,M=0,U=0,I=Math.min(Math.ceil(O),$);for(let R=Math.floor(X);R<I;R++){let T=Math.min(R+1,O)-Math.max(R,X);if(T<=0)continue;let C=Math.min(Math.ceil(D),Z);for(let A=Math.floor(B);A<C;A++){let y=Math.min(A+1,D)-Math.max(A,B);if(y<=0)continue;let w=y*T;M+=w*Q[R*Z+A],U+=w}}V[H*J+z]=U>0?Math.min(255,Math.max(0,Math.round(M/U))):0}}return V}import{deflateSync as aZ}from"node:zlib";//! Minimal grayscale PNG encoder (bit depth 8, color type 0), mirroring
|
|
15
47
|
//! pxpipe's png.ts: IHDR + one IDAT (zlib) + IEND, filter byte 0 per row.
|
|
16
|
-
var
|
|
48
|
+
var tZ=(()=>{let Q=new Uint32Array(256);for(let Z=0;Z<256;Z++){let $=Z;for(let J=0;J<8;J++)$=($&1)!==0?3988292384^$>>>1:$>>>1;Q[Z]=$>>>0}return Q})();function IQ(Q,Z,$,J){let G=J.length;Q[Z]=G>>>24&255,Q[Z+1]=G>>>16&255,Q[Z+2]=G>>>8&255,Q[Z+3]=G&255;let V=Z+4;for(let H=0;H<4;H++)Q[V+H]=$.charCodeAt(H);Q.set(J,V+4);let Y=4294967295;for(let H=V;H<V+4+G;H++)Y=tZ[(Y^Q[H])&255]^Y>>>8;Y=~Y>>>0;let K=V+4+G;return Q[K]=Y>>>24&255,Q[K+1]=Y>>>16&255,Q[K+2]=Y>>>8&255,Q[K+3]=Y&255,K+4}function mQ(Q,Z,$){let J=new Uint8Array($*(Z+1));for(let H=0,X=0,O=0;H<$;H++)J[X++]=0,J.set(Q.subarray(O,O+Z),X),X+=Z,O+=Z;let G=aZ(J,{level:6}),V=new Uint8Array(33+(12+G.length)+12);V[0]=137,V[1]=80,V[2]=78,V[3]=71,V[4]=13,V[5]=10,V[6]=26,V[7]=10;let Y=new Uint8Array(13);Y[0]=Z>>>24&255,Y[1]=Z>>>16&255,Y[2]=Z>>>8&255,Y[3]=Z&255,Y[4]=$>>>24&255,Y[5]=$>>>16&255,Y[6]=$>>>8&255,Y[7]=$&255,Y[8]=8,Y[9]=0;let K=IQ(V,8,"IHDR",Y);return K=IQ(V,K,"IDAT",G),IQ(V,K,"IEND",new Uint8Array(0)),V}//! Stage 2: the `pxpipe` imaging engine, ported from pxpipe's render.ts
|
|
17
49
|
//! production dense path (bare 5x8 AA cell, 312 cols, 1568x728 pages).
|
|
18
50
|
//! Glyphs cover BMP (pxpipe's atlas: Spleen for ASCII/code, Unifont fallback)
|
|
19
51
|
//! PLUS the astral planes (unifont_upper, incl. emoji) — beyond pxpipe, which
|
|
@@ -28,19 +60,19 @@ var hQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $
|
|
|
28
60
|
//! * `font` — `Tiny` renders the same atlas box-filtered into a 4x6 cell
|
|
29
61
|
//! (390 cols x 120 rows/page), ~40% fewer image-tokens; opt-in,
|
|
30
62
|
//! transcription-accuracy gated.
|
|
31
|
-
var
|
|
32
|
-
`);for(let
|
|
33
|
-
`).replace(
|
|
63
|
+
var a=4,VQ=4,sZ=1568,eZ=728,YQ="↵",Q$="⏎",HQ="→",Z$="⇢",PQ="⇥",$$="⇨",J$=9647,uQ=4,G$=3,gQ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",V$=/\n{4,}/g;function p(Q){return Q.length===4&&Q.replace(/[A-Z]/g,(Z)=>Z.toLowerCase())==="tiny"?"tiny":"normal"}function nQ(Q){let Z=Q==="tiny"?4:f,$=Q==="tiny"?6:i,J=Math.floor((sZ-2*a)/Z),G=Math.floor((eZ-2*VQ)/$);return{cw:Z,ch:$,cols:J,maxLines:G,maxChars:J*G}}var Y$=(()=>{let Q=new Uint8Array(128);for(let Z=0;Z<128;Z++){let $=r(Z);Q[Z]=$>=0&&d($)?2:1}return Q})();function RQ(Q){if(Q<128)return Y$[Q];let Z=r(Q);return Z>=0&&d(Z)?2:1}function H$(Q){let Z=0;for(let $=0;$<Q.length;$++){let J=Q.charCodeAt($);if(J>=55296&&J<56320&&$+1<Q.length){let G=Q.charCodeAt($+1);if(G>=56320&&G<57344)$++}Z++}return Z}function AQ(Q){let Z=Q.split(`
|
|
64
|
+
`);for(let $=0;$<Z.length;$++){let J=Z[$],G=J.length;while(G>0){let V=J.charCodeAt(G-1);if(V!==32&&V!==9)break;G--}if(G!==J.length)Z[$]=J.slice(0,G)}return Z.join(`
|
|
65
|
+
`).replace(V$,`
|
|
34
66
|
|
|
35
67
|
|
|
36
|
-
`)}function
|
|
37
|
-
`);for(let
|
|
38
|
-
`);for(let
|
|
39
|
-
`)){let
|
|
68
|
+
`)}function cQ(Q){if(!Q.includes("\t"))return Q;let Z="",$=0;for(let J of Q)if(J==="\t"){let G=uQ-$%uQ;Z+=HQ;for(let V=1;V<G;V++)Z+=" ";$+=G}else Z+=J,$+=RQ(J.codePointAt(0));return Z}function K$(Q){let Z=Q.includes("\t")?Q.replaceAll("\t",HQ):Q,$=0;while($<Z.length&&Z.charCodeAt($)===32)$++;if($>=G$&&$<gQ.length)return PQ+gQ[$]+Z.slice($);return Z}function oQ(Q){return Q.includes(YQ)?Q.replaceAll(YQ,Q$):Q}function z$(Q){let Z=oQ(Q);if(Z.includes(HQ))Z=Z.replaceAll(HQ,Z$);if(Z.includes(PQ))Z=Z.replaceAll(PQ,$$);return Z}function X$(Q){let Z=AQ(Q).split(`
|
|
69
|
+
`);for(let $=0;$<Z.length;$++)Z[$]=cQ(Z[$]);return Z.join(YQ)}function B$(Q){let Z=AQ(Q).split(`
|
|
70
|
+
`);for(let $=0;$<Z.length;$++)Z[$]=K$(Z[$]);return Z.join(YQ)}function O$(Q,Z){let $=[],J=AQ(Q);for(let G of J.split(`
|
|
71
|
+
`)){let V=cQ(G);if(V.length===0){$.push("");continue}let Y="",K=0;for(let H of V){let X=RQ(H.codePointAt(0));if(K+X>Z)$.push(Y),Y=H,K=X;else Y+=H,K+=X}if(Y.length!==0)$.push(Y)}return $}function q$(Q,Z,$){let J=[],G=[],V=0;for(let Y of Q){let K=H$(Y),H=K+(G.length!==0?1:0);if(G.length!==0&&(G.length>=Z||V+H>$))J.push(G),G=[],V=0;V+=K+(G.length!==0?1:0),G.push(Y)}if(G.length!==0)J.push(G);return J}function iQ(Q){if(Q<32)return!0;if(Q>=127&&Q<=159)return!0;if(Q>=768&&Q<=879)return!0;if(Q===8203||Q===8204||Q===8205||Q===8288||Q===65279)return!0;if(Q>=65024&&Q<=65039)return!0;if(Q>=917760&&Q<=917999)return!0;return!1}function D$(Q){let Z=null,$=0;for(let J of Q){let G=J.codePointAt(0);if(r(G)<0&&!iQ(G)){if(Z===null)Z=Q.slice(0,$);Z+=`[U+${G.toString(16).toUpperCase()}]`}else if(Z!==null)Z+=J;$+=J.length}return Z??Q}function rQ(Q,Z,$,J){let G=Z?$?B$(z$(Q)):X$(oQ(Q)):Q;return q$(O$(D$(G),J.cols),J.maxLines,J.maxChars)}function aQ(Q,Z,$){if(!$)return 2*a+Z.cols*Z.cw;let J=0;for(let Y of Q){let K=0;for(let H of Y)K+=RQ(H.codePointAt(0));if(K>Z.cols)K=Z.cols;if(K>J)J=K}let G=2*a+J*Z.cw,V=2*a+Z.cw;return G>V?G:V}var dQ=new Map;function pQ(Q,Z,$,J,G,V,Y){let K=r(G);if(K<0)return 0;let H=d(K),X=H?2*V.cw:V.cw,O;if(Y==="normal")O=NQ(K);else{let z=dQ.get(K);if(z===void 0)z=kQ(K,X,V.ch),dQ.set(K,z);O=z}for(let z=0;z<V.ch;z++){let B=(J+z)*Z+$,D=z*X;for(let M=0;M<X;M++){let U=O[D+M];if(U>0){let I=B+M;if(U>Q[I])Q[I]=U}}}return H?2:1}function M$(Q,Z,$,J){let G=aQ(Q,Z,$),V=2*VQ+Q.length*Z.ch,Y=new Uint8Array(G*V),K=0;for(let H=0;H<Q.length;H++){let X=VQ+H*Z.ch,O=0;for(let z of Q[H]){if(O>=Z.cols)break;let B=a+O*Z.cw,D=pQ(Y,G,B,X,z.codePointAt(0),Z,J);if(D===0){if(D=1,iQ(z.codePointAt(0)));else if(K++,z!==" ")pQ(Y,G,B,X,J$,Z,J)}O+=D}}for(let H=0;H<Y.length;H++)Y[H]=255-Y[H];return{png:mQ(Y,G,V),width:G,height:V,dropped:K}}function KQ(Q,Z,$,J){let G=nQ(J),V=rQ(Q,Z,$,G).map((X)=>M$(X,G,$,J)),Y=0,K=0,H=0;for(let X of V)Y+=X.width*X.height,K+=tQ(X.width,X.height),H+=X.dropped;return{pages:V,pixels:Y,tokens:K,dropped:H}}function zQ(Q,Z,$,J){let G=nQ(J),V=rQ(Q,Z,$,G),Y=0,K=0,H=[];for(let X of V){let O=aQ(X,G,$),z=2*VQ+X.length*G.ch;H.push([O,z]),Y+=O*z,K+=tQ(O,z)}return{pages:V.length,pixels:Y,tokens:K,dims:H}}var lQ=28;function tQ(Q,Z){return Math.ceil(Q/lQ)*Math.ceil(Z/lQ)}import{readFileSync as U$}from"node:fs";import{join as W$}from"node:path";//! pxpipe measurement-log summary (~/.pxpipe/events.jsonl), same math as the
|
|
40
72
|
//! node MCP: actual = every way input bytes get billed (input + cache reads +
|
|
41
73
|
//! cache creates) — ignoring cache_read would fake the savings.
|
|
42
|
-
class
|
|
43
|
-
`)){if(
|
|
74
|
+
class t{value;constructor(Q){this.value=Q}}function eQ(){let Q=process.env.TANUKI_EVENTS;if(Q!==void 0)return Q;let Z=process.env.HOME??"";return W$(Z,".pxpipe","events.jsonl")}function m(Q){return typeof Q==="number"&&Number.isSafeInteger(Q)&&Q>=0?Q:0}function sQ(Q){return Q<0?-Math.round(-Q):Math.round(Q)}function _Q(){let Q=eQ(),Z;try{Z=U$(Q,"utf8")}catch{return{available:!1,note:`no ${Q} yet`}}let $=0,J=0,G=0,V=0,Y=0,K=0,H=0;for(let O of Z.split(`
|
|
75
|
+
`)){if(O.trim().length===0)continue;let z;try{z=JSON.parse(O)}catch{continue}$+=1;let B=z!==null&&typeof z==="object"&&!Array.isArray(z)?z:{};if(B.compressed===!0)J+=1,G+=m(B.orig_chars),V+=m(B.image_count);Y+=m(B.baseline_tokens),K+=m(B.input_tokens)+m(B.cache_read_tokens)+m(B.cache_create_tokens),H+=m(B.output_tokens)}let X=Y>0&&K>0?new t(sQ((1-K/Y)*1000)/10):null;return{available:!0,requests:$,compressedRequests:J,imagedChars:G,imagesEmitted:V,baselineTokens:Y,actualInputTokens:K,outputTokens:H,outputSharePct:H>0?new t(sQ(H/(K+H)*1000)/10):null,estInputSavedPct:X}}//! Implicit mode: a local Anthropic middlebox, the pxpipe deployment shape
|
|
44
76
|
//! without pxpipe's structural flaw. Rules that keep it injection-shaped-free:
|
|
45
77
|
//!
|
|
46
78
|
//! 1. The system prompt and tool definitions are NEVER touched.
|
|
@@ -56,7 +88,7 @@ class YJ{value;constructor(J){this.value=J}}function nJ(){let J=process.env.TANU
|
|
|
56
88
|
//!
|
|
57
89
|
//! Responses stream through untouched; usage is scraped from the stream for
|
|
58
90
|
//! the ~/.pxpipe/events.jsonl savings log (same format tanuki_stats reads).
|
|
59
|
-
import{createHash as
|
|
91
|
+
import{createHash as F$}from"node:crypto";import{mkdirSync as j$,readFileSync as N$,writeFileSync as I$}from"node:fs";import QZ from"node:process";//! Stash mode: context-mode's shape (content parked outside the context
|
|
60
92
|
//! window, queried on demand) fused with tanuki's pricing (big answers come
|
|
61
93
|
//! back as dense pages).
|
|
62
94
|
//!
|
|
@@ -65,11 +97,11 @@ import{createHash as YZ}from"node:crypto";import{mkdirSync as HZ,readFileSync as
|
|
|
65
97
|
//! repeats, first/last lines, the id). fetch = pull a slice by regex query
|
|
66
98
|
//! (distill-powered) or line range; the caller images it only when pages
|
|
67
99
|
//! clearly win. Contract is byte-identical with the Rust engine.
|
|
68
|
-
function
|
|
69
|
-
`),
|
|
70
|
-
`)}}function
|
|
71
|
-
`),Y=Math.max(1,Number(
|
|
72
|
-
`)}return b(
|
|
100
|
+
function ZZ(){let Q=QZ.env.TANUKI_STASH;if(Q!==void 0&&Q!=="")return Q;return`${QZ.env.HOME??""}/.tanuki/stash`}function $Z(Q){let Z=F$("sha256").update(Q,"utf8").digest("hex").slice(0,12),$=ZZ();j$($,{recursive:!0}),I$(`${$}/${Z}`,Q);let J=Buffer.byteLength(Q,"utf8"),G=Q.split(`
|
|
101
|
+
`),V=b(Q,null,2).stats,Y=[`stashed ${Z} · ${J} bytes · ${G.length} lines`,`distill map: ${V.origLines} -> ${V.outLines} lines · ${V.savedPct}% of chars removable · ${V.importantKept} error/warn lines`];if(V.topRepeats.length>0){Y.push("top repeats:");for(let H of V.topRepeats.slice(0,5)){let X=H.kind==="template"?" (template)":"";Y.push(` ×${H.count}${X} ${H.exemplar}`)}}let K="";for(let H=G.length-1;H>=0;H--)if(G[H]!==""){K=G[H];break}return Y.push(`first: ${o(c(G[0]),160)}`),Y.push(`last: ${o(c(K),160)}`),Y.push(`fetch: tanuki_fetch {"id":"${Z}","query":"<regex>"} or {"id":"${Z}","lines":"a-b"}`),{id:Z,overview:Y.join(`
|
|
102
|
+
`)}}function JZ(Q,Z,$){if(Z===null===($===null))throw Error("give exactly one of query or lines");let J;try{J=N$(`${ZZ()}/${Q}`,"utf8")}catch{throw Error(`unknown stash id: ${Q}`)}if($!==null){let G=/^(\d+)-(\d+)$/.exec($);if(G===null)throw Error("bad lines range");let V=J.split(`
|
|
103
|
+
`),Y=Math.max(1,Number(G[1])),K=Math.min(V.length,Number(G[2]));if(Number(G[1])>Number(G[2]))throw Error("bad lines range");return V.slice(Y-1,K).join(`
|
|
104
|
+
`)}return b(J,Z,2).distilled}//! tanuki-context — token-cutting context pipeline.
|
|
73
105
|
//! pipeline: text -> distill (stage 0, logs) -> ladder level 0-4 (stage 1)
|
|
74
106
|
//! -> pxpipe imaging (stage 2, name kept from the original mechanic)
|
|
75
107
|
//!
|
|
@@ -78,15 +110,16 @@ function oJ(){let J=cJ.env.TANUKI_STASH;if(J!==void 0&&J!=="")return J;return`${
|
|
|
78
110
|
//! tanuki-context estimate <file> [level] [--distill]
|
|
79
111
|
//! tanuki-context render <file> [level] [outdir]
|
|
80
112
|
//! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
|
|
81
|
-
var
|
|
82
|
-
`;for(let O=0;O<
|
|
83
|
-
`;
|
|
84
|
-
|
|
85
|
-
`;for(let
|
|
86
|
-
`;
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
113
|
+
var GZ="0.6.0",CQ=6;function P$(Q,Z){let $=Math.min(Q.length,Z.length),J=0;while(J<$&&Q.charCodeAt(J)===Z.charCodeAt(J))J++;if(J>=$)return Q.length-Z.length;return Q.codePointAt(J)-Z.codePointAt(J)}function L(Q,Z,$=""){if(Q===null||Q===void 0)return"null";if(Q instanceof t){let H=Q.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let J=typeof Q;if(J==="string")return JSON.stringify(Q);if(J==="number"||J==="boolean")return String(Q);if(Array.isArray(Q)){if(Q.length===0)return"[]";if(!Z){let O="[";for(let z=0;z<Q.length;z++){if(z>0)O+=",";O+=L(Q[z],!1)}return O+"]"}let H=$+" ",X=`[
|
|
114
|
+
`;for(let O=0;O<Q.length;O++){if(O>0)X+=`,
|
|
115
|
+
`;X+=H+L(Q[O],!0,H)}return X+`
|
|
116
|
+
`+$+"]"}let G=Q,V=Object.keys(G).sort(P$);if(V.length===0)return"{}";if(!Z){let H="{";for(let X=0;X<V.length;X++){if(X>0)H+=",";H+=JSON.stringify(V[X])+":"+L(G[V[X]],!1)}return H+"}"}let Y=$+" ",K=`{
|
|
117
|
+
`;for(let H=0;H<V.length;H++){if(H>0)K+=`,
|
|
118
|
+
`;K+=Y+JSON.stringify(V[H])+": "+L(G[V[H]],!0,Y)}return K+`
|
|
119
|
+
`+$+"}"}function N(Q,Z){return Q!==null&&typeof Q==="object"&&!Array.isArray(Q)?Q[Z]:void 0}function S(Q){return typeof Q==="string"?Q:null}function u(Q){return typeof Q==="boolean"?Q:null}function VZ(Q){return typeof Q==="number"&&Number.isSafeInteger(Q)&&Q>=0?Q:null}function v(Q){let Z=0;for(let $=0;$<Q.length;$++){Z++;let J=Q.charCodeAt($);if(J>=55296&&J<=56319&&$+1<Q.length){let G=Q.charCodeAt($+1);if(G>=56320&&G<=57343)$++}}return Z}function YZ(Q,Z,$,J,G,V){let Y=Q,K=null;if(V){let z=n(Y);if(z!==null)Y=z.text,K={rows:z.rows,cols:z.cols}}let H=null;if($||J!==null){let z=b(Y,J,2);Y=z.distilled,H=z.stats}let X=0;if(G){let z=QQ(Y);Y=z.text,X=z.entries}let O=JQ(Y,Z);return{stage0:H,compressed:O.compressed,protectedLines:O.protectedLines,level:O.level,cbEntries:X,table:K}}function s(Q){return Math.round(Q/4)}function e(Q,Z){if(Q===0)return 0;let $=(1-Z/Q)*100;return $<0?-Math.round(-$):Math.round($)}function HZ(Q){return{text:S(N(Q,"text"))??"",level:(VZ(N(Q,"level"))??0)%256,distill:u(N(Q,"distill"))??!1,query:S(N(Q,"query")),reflow:u(N(Q,"reflow"))??!0,pack:u(N(Q,"pack"))??!0,font:S(N(Q,"font"))??"normal",codebook:u(N(Q,"codebook"))??!1,table:u(N(Q,"table"))??!1}}function R$(Q){let Z=(H)=>{let X={codebook:!1,tokens:1/0,pages:0,text:H};for(let O of[!1,!0]){let z=O?QQ(H).text:H,B=zQ(z,!0,!0,p("normal"));if(B.tokens<X.tokens)X={codebook:O,tokens:B.tokens,pages:B.pages,text:z}}return X},$=Z(Q),J=!1,G=n(Q);if(G!==null){let H=Z(G.text);if(H.tokens<$.tokens)$=H,J=!0}let V=J&&G!==null?G.text:Q,Y=Z(b(V,null,2).distilled),K=zQ($.text,!0,!0,p("tiny"));return{codebook:$.codebook,imageTokens:$.tokens,pages:$.pages,table:J,tinyImageTokens:K.tokens,withDistill:{codebook:Y.codebook,imageTokens:Y.tokens}}}function KZ(Q){let Z=HZ(Q),$=YZ(Z.text,Z.level,Z.distill,Z.query,Z.codebook,Z.table),J=p(Z.font),G=zQ($.compressed,Z.reflow,Z.pack,J),V=G.tokens,Y=v(Z.text),K=v($.compressed),H=s(Y),[X,O]=$Q[$.level],z=S(N(Q,"model")),B=u(N(Q,"cached"))??!1,D={engine:"pxpipe",level:`${$.level} ${X}`,loss:O,distill:$.stage0,origChars:Y,stage1Chars:K,stage1SavedPct:e(Y,K),pages:G.pages,imageTokens:V,rawTextTokens:H,totalSavedPct:e(H,V),protectedLines:$.protectedLines,pack:Z.pack,font:J==="tiny"?"tiny":"normal",codebook:Z.codebook?$.cbEntries:!1,table:$.table!==null?$.table:!1,verdict:V<H?"PIPELINE cheaper":"TEXT cheaper",recommend:R$(Z.text)};if(z!==null||B)D.cost=wQ(H,V,{model:z,cached:B},{dims:G.dims});return D}function zZ(Q){let Z=HZ(Q),$=YZ(Z.text,Z.level,Z.distill,Z.query,Z.codebook,Z.table),J=p(Z.font),G=KQ($.compressed,Z.reflow,Z.pack,J),V=G.tokens,Y=v(Z.text),K=v($.compressed),H=s(Y),[X,O]=$Q[$.level],z="";if($.table!==null)z+=`table: ${$.table.rows} rows x ${$.table.cols} cols, keys stated once
|
|
120
|
+
`;if($.stage0!==null){let M=$.stage0;z+=`distill: ${M.origLines} -> ${M.outLines} lines (-${M.savedPct}% chars, ${M.collapsedRuns} runs, ${M.suppressedLines} exact + ${M.templateSuppressed} template suppressed, ${M.importantKept} error/warn kept)
|
|
121
|
+
`}if(z+=`L${$.level} ${X} (${O}): ${Y} chars -> ${K} chars (stage1 -${e(Y,K)}%) -> ${G.pages.length} page(s), ~${V} image-tokens
|
|
122
|
+
vs ~${H} text-tokens raw = TOTAL -${e(H,V)}%`,$.protectedLines>0)z+=` · ${$.protectedLines} lines kept verbatim`;if(G.dropped>0)z+=` · ${G.dropped} unmapped glyphs -> ▯`;if($.cbEntries>0)z+=` · codebook: ${$.cbEntries} sigils (see ·legend·)`;if(J==="tiny")z+=" · font: tiny 4x6";if(Z.pack)z+=" · packed (⇥N indent, → tab)";if(Z.reflow)z+=" · ↵ = newline · engine: pxpipe";let B=[{type:"text",text:z}],D=Math.min(G.pages.length,CQ);for(let M=0;M<D;M++){let U=G.pages[M].png;B.push({type:"image",data:Buffer.from(U.buffer,U.byteOffset,U.byteLength).toString("base64"),mimeType:"image/png"})}if(G.pages.length>CQ)B.push({type:"text",text:`(+${G.pages.length-CQ} more page(s))`});return B}function XZ(Q){let $=S(N(Q,"text"))??"",J=null;if(u(N(Q,"table"))??!1){let Y=n($);if(Y!==null)$=Y.text,J={rows:Y.rows,cols:Y.cols}}let G=b($,S(N(Q,"query")),2),V=J!==null?{...G.stats,table:J}:G.stats;return[{type:"text",text:L(V,!0)},{type:"text",text:G.distilled}]}function BZ(Q){let Z=S(N(Q,"text"))??"",$=(VZ(N(Q,"level"))??1)%256,J=JQ(Z,$),[G,V,Y]=$Q[J.level],K=v(Z),H=v(J.compressed),X=s(K),O=s(H),z={level:`${J.level} ${G}`,loss:V,note:Y,origChars:K,outChars:H,approxOrigTokens:X,approxOutTokens:O,savedPct:e(X,O),protectedLines:J.protectedLines};return[{type:"text",text:L(z,!0)},{type:"text",text:J.compressed}]}function OZ(Q){let Z=S(N(Q,"text"))??"";return[{type:"text",text:$Z(Z).overview}]}function qZ(Q){let Z=S(N(Q,"id"))??"",$=JZ(Z,S(N(Q,"query"))??null,S(N(Q,"lines"))??null),J=s(v($)),G=KQ($,!0,!0,p("normal"));if(!(G.tokens<=J*0.75&&J-G.tokens>=300&&G.pages.length<=6))return[{type:"text",text:$}];let K=[{type:"text",text:`[tanuki-context stash ${Z}: slice of ${v($)} chars imaged as ${G.pages.length} PNG page(s), ~${G.tokens} vs ~${J} text tokens. ↵=newline →=tab ⇥N=indent]`}];for(let H of G.pages){let X=H.png;K.push({type:"image",data:Buffer.from(X.buffer,X.byteOffset,X.byteLength).toString("base64"),mimeType:"image/png"})}return K}//! Claude Agent SDK integration (`tanuki-context/agent`).
|
|
90
123
|
//!
|
|
91
124
|
//! Two flavours, both one-liners from an agent's point of view:
|
|
92
125
|
//!
|
|
@@ -103,8 +136,8 @@ vs ~${z} text-tokens raw = TOTAL -${r(z,G)}%`,Z.protectedLines>0)B+=` · ${Z.pro
|
|
|
103
136
|
//! The core package stays zero-dependency: the SDK and zod are touched only
|
|
104
137
|
//! inside `tanukiSdkServer()` via dynamic import, and both are already present
|
|
105
138
|
//! in any Agent SDK project (zod is the SDK's own peer dependency).
|
|
106
|
-
var
|
|
107
|
-
Workflow: call tanuki_estimate first (instant, exact, never renders pixels). Its "recommend" field prices the reversible route; for logs, recommend.withDistill prices the distill route - do not probe combos by hand. If the verdict says "PIPELINE cheaper", call tanuki_render with the recommended knobs and use the returned pages instead of pasting the text.
|
|
108
|
-
For logs, pass distill:true (repeats collapse, error/warn lines stay verbatim; add query:"regex" to slice). For prose you will not quote verbatim, level 2-3 shrinks it further. codebook:true helps path-heavy logs. Never image content you must quote byte-exact at level 4 or font tiny.
|
|
139
|
+
var C$=["tanuki_render","tanuki_estimate","tanuki_distill","tanuki_compress","tanuki_stats","tanuki_stash","tanuki_fetch"],E$=`tanuki-context turns bulky text (logs, command output, docs) into dense PNG pages that cost a fraction of the text tokens.
|
|
140
|
+
Workflow: call tanuki_estimate first (instant, exact, never renders pixels). Its "recommend" field prices the reversible route (including table for whole-JSON input); for logs, recommend.withDistill prices the distill route - do not probe combos by hand. If the verdict says "PIPELINE cheaper", call tanuki_render with the recommended knobs and use the returned pages instead of pasting the text. Pass model:"<your model>" and cached:true when the text is already in the prompt cache - the returned "cost" field prices it in real dollars with provider-correct image counting (Anthropic/OpenAI/Gemini), and cached content usually should NOT be imaged (a cache-read token is ~0.1x a fresh one).
|
|
141
|
+
For logs, pass distill:true (repeats collapse, error/warn lines stay verbatim; add query:"regex" to slice). For whole-JSON input (arrays of objects, NDJSON), table:true states keys once - value-lossless, and identical rows then collapse harder under distill. For prose you will not quote verbatim, level 2-3 shrinks it further. codebook:true helps path-heavy logs. Never image content you must quote byte-exact at level 4 or font tiny.
|
|
109
142
|
For huge references you will only consult occasionally: tanuki_stash parks the text outside context for a few hundred tokens of map; tanuki_fetch pulls slices later, auto-imaged when pages win.
|
|
110
|
-
Pages decode as: ↵ = newline, → = tab, ⇥N = N leading spaces, a trailing ·legend· line maps sigils back to full tokens.`;function
|
|
143
|
+
Pages decode as: ↵ = newline, → = tab, ⇥N = N leading spaces, a trailing ·legend· line maps sigils back to full tokens.`;function S$(){let Q=new URL("./cli.js",import.meta.url);if(Q.protocol==="file:"){let Z=_$(Q);if(A$(Z))return{type:"stdio",command:process.execPath,args:[Z]}}return{type:"stdio",command:"npx",args:["-y","tanuki-context"]}}function L$(Q="tanuki"){return C$.map((Z)=>`mcp__${Q}__${Z}`)}function _J(Q,Z={}){let $=Q??{},J=Z.key??"tanuki";return{...$,mcpServers:{...$.mcpServers??{},[J]:Z.server??S$()},allowedTools:[...$.allowedTools??[],...L$(J)]}}function T$(Q){let Z=Q.string().describe("the bulky text to process"),$=Q.number().int().min(0).max(4).optional().describe("ladder level 0-4 (default 0)"),J={text:Z,level:$,distill:Q.boolean().optional().describe("stage 0 log distiller"),query:Q.string().optional().describe("distill: keep matching lines +context"),reflow:Q.boolean().optional().describe("pack short lines into full rows (default true)"),pack:Q.boolean().optional().describe("indent RLE + width trim, lossless (default true)"),font:Q.enum(["normal","tiny"]).optional().describe("tiny = 4x6 cells, ~40% fewer tokens, gated"),codebook:Q.boolean().optional().describe("repeated tokens/paths -> sigils + legend"),table:Q.boolean().optional().describe("columnar-encode whole-JSON input (keys stated once, value-lossless)"),model:Q.string().optional().describe("estimate: price the decision for this model (opus/sonnet/haiku/gpt/gemini)"),cached:Q.boolean().optional().describe("estimate: text already prompt-cached this turn? imaging it usually loses")},G=(Y)=>({content:Y}),V=async(Y)=>{try{return G(Y())}catch(K){return{content:[{type:"text",text:`tanuki-context error: ${K instanceof Error?K.message:String(K)}`}],isError:!0}}};return[{name:"tanuki_render",description:"Render text through the pipeline (optional distill/level/codebook) into dense PNG pages. Call after tanuki_estimate says PIPELINE cheaper.",inputSchema:J,handler:(Y)=>V(()=>zZ(Y))},{name:"tanuki_estimate",description:"Exact page/token math for the same arguments as tanuki_render, without touching pixels. Pass model and/or cached:true for a real-dollar 'cost' verdict (cached content usually should not be imaged). Instant; call this first.",inputSchema:J,handler:(Y)=>V(()=>[{type:"text",text:L(KZ(Y),!0)}])},{name:"tanuki_distill",description:"Stage 0 alone: collapse repeated log lines/blocks and template near-dupes; error/warn lines kept verbatim. Output stays greppable text.",inputSchema:{text:Z,query:J.query},handler:(Y)=>V(()=>XZ(Y))},{name:"tanuki_compress",description:"Stage 1 alone: graded text compression, levels 0-4, code/paths/hashes protected from level 2 up.",inputSchema:{text:Z,level:$},handler:(Y)=>V(()=>BZ(Y))},{name:"tanuki_stats",description:"Session savings summary from the events log (honest denominator: input + cache reads + cache creates).",inputSchema:{},handler:()=>V(()=>[{type:"text",text:L(_Q(),!0)}])},{name:"tanuki_stash",description:"Park bulky text outside the context window; returns a compact map (distill stats, top repeats, id). Retrieval pattern, tanuki pricing on the way back.",inputSchema:{text:Z},handler:(Y)=>V(()=>OZ(Y))},{name:"tanuki_fetch",description:"Pull a slice of stashed text by id + query regex or lines 'a-b'. Big slices return as dense PNG pages automatically.",inputSchema:{id:J.query,query:J.query,lines:J.query},handler:(Y)=>V(()=>qZ(Y))}]}async function CJ(){let Q,Z;try{Q=await import("@anthropic-ai/claude-agent-sdk");let J=await import("zod");Z=J.z??J}catch(J){throw Error(`tanukiSdkServer() needs the host project's @anthropic-ai/claude-agent-sdk and zod (npm i @anthropic-ai/claude-agent-sdk zod). Import failed: ${J instanceof Error?J.message:String(J)}`)}let $=T$(Z).map((J)=>Q.tool(J.name,J.description,J.inputSchema,(G)=>J.handler(G)));return Q.createSdkMcpServer({name:"tanuki-context",version:GZ,instructions:E$,tools:$})}export{_J as withTanuki,T$ as tanukiSdkToolSpecs,CJ as tanukiSdkServer,S$ as tanukiMcpServer,L$ as tanukiAllowedTools,C$ as TANUKI_TOOL_NAMES,E$ as TANUKI_INSTRUCTIONS};
|
package/dist/cli.js
CHANGED
|
@@ -1,20 +1,52 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import{spawnSync as
|
|
3
|
-
·legend· `;for(let
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
2
|
+
import{spawnSync as m$}from"node:child_process";import{readFileSync as u$,mkdirSync as UZ,writeFileSync as qZ}from"node:fs";import P from"node:process";function SZ(Q){if(Q===32||Q>=9&&Q<=13)return!0;if(Q<133)return!1;return Q===133||Q===160||Q===5760||Q>=8192&&Q<=8202||Q===8232||Q===8233||Q===8239||Q===8287||Q===12288}function NQ(Q){let Z=0;for(let $=0;$<Q.length;$++){let J=Q.charCodeAt($);if(J>=55296&&J<=56319&&$+1<Q.length){let z=Q.charCodeAt($+1);if(z>=56320&&z<=57343)$++}Z++}return Z}function jQ(Q,Z){let $=0,J=0;while($<Q.length&&J<Z.length){let z=Q.codePointAt($),V=Z.codePointAt(J);if(z!==V)return z-V;$+=z>65535?2:1,J+=V>65535?2:1}return Q.length-$-(Z.length-J)}function t(Q){let Z=new Map,$=(X)=>{Z.set(X,(Z.get(X)??0)+1)},J=(X)=>{if(NQ(X)>=12)$(X);if(X.includes("/")){let D=X.split("/"),U="";for(let M=0;M<D.length;M++){if(M>0)U+="/";if(U+=D[M],M>=2){let I=U+"/";if(NQ(I)>=12)$(I)}}}},z=-1;for(let X=0;X<Q.length;){let D=Q.codePointAt(X),U=D>65535?2:1;if(SZ(D)){if(z>=0)J(Q.slice(z,X)),z=-1}else if(z<0)z=X;X+=U}if(z>=0)J(Q.slice(z));let V=[];for(let[X,D]of Z)if(D>=3){let U=NQ(X);V.push({k:X,c:D,len:U,saved:(U-1)*D})}V.sort((X,D)=>D.saved-X.saved||jQ(X.k,D.k));let G=[];for(let X of"§¤¢£¥µ¶ª°±¬×÷ØÞßæðøþ¡¿")if(!Q.includes(X))G.push(X);let Y=[],H=[];for(let{k:X,c:D,len:U}of V){if(Y.length>=G.length)break;if((U-1)*D<=U+3)continue;let M=!1;for(let C=0;C<H.length;C++){let v=H[C];if(v.startsWith(X)||X.startsWith(v)){M=!0;break}}if(M)continue;let I=G[Y.length];H.push(X),Y.push({sig:I,val:X,len:U})}if(Y.length===0)return{text:Q,entries:0};let K=Y.map((X,D)=>D);K.sort((X,D)=>Y[D].len-Y[X].len);let O=Q;for(let X=0;X<K.length;X++){let{sig:D,val:U}=Y[K[X]];O=O.replaceAll(U,D)}let B=`
|
|
3
|
+
·legend· `;for(let X=0;X<Y.length;X++)B+=Y[X].sig+"="+Y[X].val+" ";return O+=B.slice(0,-1),{text:O,entries:Y.length}}//! Situation-aware cost model — the "codeburn calculation." tanuki's verdict
|
|
4
|
+
//! compares token COUNTS, which equals real cost only when both sides bill at
|
|
5
|
+
//! the same per-token rate. They do not: on Anthropic a cache-read costs ~0.1×
|
|
6
|
+
//! a fresh input token, while image (visual) tokens bill AT the input rate. So
|
|
7
|
+
//! the cheapest technique depends on the SITUATION — is the text already cached
|
|
8
|
+
//! this turn, and which provider prices it. Only the RATIOS drive the verdict;
|
|
9
|
+
//! absolute $/Mtok (list prices, overridable via TANUKI_RATES) drive the
|
|
10
|
+
//! optional dollar figure. Image-token COUNTS are provider-correct when page
|
|
11
|
+
//! dims are supplied: Anthropic 28px patches, OpenAI 512px high-detail tiles
|
|
12
|
+
//! (85 + 170/tile), Gemini 768px tiles (258/tile, ~approximate — their crop
|
|
13
|
+
//! rule has undocumented edges; the API usage field is authoritative).
|
|
14
|
+
function EZ(Q){return Q<0?-Math.round(-Q):Math.round(Q)}function gQ(Q){return Math.round(Q*1e6)/1e6}var CZ="2026-07",HQ={opus:{input:15,output:75,cacheReadMult:0.1,imageMult:1,family:"anthropic"},sonnet:{input:3,output:15,cacheReadMult:0.1,imageMult:1,family:"anthropic"},haiku:{input:1,output:5,cacheReadMult:0.1,imageMult:1,family:"anthropic"},gpt:{input:1.25,output:10,cacheReadMult:0.5,imageMult:1,family:"openai"},gemini:{input:1.25,output:10,cacheReadMult:0.25,imageMult:1,family:"gemini"},default:{input:3,output:15,cacheReadMult:0.1,imageMult:1,family:"anthropic"}};function LZ(){let Q=process.env.TANUKI_RATES;if(Q===void 0||Q.length===0)return HQ;try{let Z=JSON.parse(Q),$={...HQ};for(let[J,z]of Object.entries(Z))$[J]={...$[J]??HQ.default,...z};return $}catch{return HQ}}function TZ(Q){let Z=LZ(),$=(Q??"").toLowerCase();for(let J of["opus","sonnet","haiku","gpt","gemini"])if($.includes(J)&&Z[J]!==void 0)return{key:J,rate:Z[J]};return{key:"default",rate:Z.default}}function yZ(Q,Z){let $=0;for(let[J,z]of Q)if(Z==="openai"){let V=J,G=z,Y=Math.min(1,2048/Math.max(V,G));V*=Y,G*=Y;let H=Math.min(1,768/Math.min(V,G));V=Math.ceil(V*H),G=Math.ceil(G*H),$+=85+170*(Math.ceil(V/512)*Math.ceil(G/512))}else $+=J<=384&&z<=384?258:258*(Math.ceil(J/768)*Math.ceil(z/768));return $}function dQ(Q,Z,$,J){let{key:z,rate:V}=TZ($.model),G=$.cached===!0,Y=V.family!=="anthropic"&&J!==void 0?yZ(J.dims,V.family):Z,H=V.input/1e6,K=H*(G?V.cacheReadMult:1),O=H*V.imageMult,B=Q*K,X=Y*O,D=O>0?Math.floor(Q*K/O):1/0,U=X<B?"PIPELINE":"TEXT",M=B>0?EZ((1-X/B)*100):0,I=[];if(V.family==="openai")I.push(J!==void 0?"image tokens counted with OpenAI's high-detail tile rule (85 + 170 per 512px tile)":"no page dims supplied; image count falls back to Anthropic's 28px patch grid — approximate for openai");else if(V.family==="gemini")I.push(J!==void 0?"~approximate: Gemini's documented 768px-tile rule (258/tile); the API usage field is authoritative":"no page dims supplied; image count falls back to Anthropic's 28px patch grid — approximate for gemini");if(G)I.push(`text priced at cache-read rate (${V.cacheReadMult}× input); imaging already-cached content usually loses`);return{model:z,cached:G,ratesAsOf:CZ,imageTokens:Y,textUsd:gQ(B),imageUsd:gQ(X),cheaper:U,savedPct:M,breakevenImageTokens:D,note:I.length>0?I.join("; "):void 0}}//! Columnar codec for structured JSON — the one domain where retrieval-store
|
|
15
|
+
//! compressors (Headroom's SmartCrusher) beat tanuki's line tools, done
|
|
16
|
+
//! tanuki-style instead: deterministic, no model in the loop, decode grammar
|
|
17
|
+
//! documented. An array of objects (or NDJSON — journalctl -o json, docker
|
|
18
|
+
//! events, API dumps) repeats every key on every row; state the keys ONCE:
|
|
19
|
+
//!
|
|
20
|
+
//! ·cols·<TAB>"k1"<TAB>"k2"...
|
|
21
|
+
//! cell<TAB>cell... one line per row, cells = compact JSON
|
|
22
|
+
//!
|
|
23
|
+
//! An absent key is an empty cell (a JSON cell is never empty, so this is
|
|
24
|
+
//! unambiguous). Compact JSON escapes every control char, so a raw tab can
|
|
25
|
+
//! never appear inside a cell. Round-trip contract: same VALUES in canonical
|
|
26
|
+
//! layout — whitespace and per-object key order are serialization, not
|
|
27
|
+
//! content (nested objects re-print with sorted keys, exactly serde_json's
|
|
28
|
+
//! default Value ordering, which the custom serializer in main.ts already
|
|
29
|
+
//! mimics engine-wide).
|
|
30
|
+
//!
|
|
31
|
+
//! ponytail: whole-input tables only — mixed prose+JSON stays text; add
|
|
32
|
+
//! block detection if a real corpus ever demands it.
|
|
33
|
+
function pQ(Q){let Z=0;for(let $=0;$<Q.length;$++){let J=Q.charCodeAt($);if(J<55296||J>56319)Z++}return Z}function IQ(Q){if(Q===null||typeof Q==="boolean")return JSON.stringify(Q);if(typeof Q==="number"){if(Number.isSafeInteger(Q))return String(Q);return String(Q).replace("e+","e")}if(typeof Q==="string")return JSON.stringify(Q);if(Array.isArray(Q))return`[${Q.map(IQ).join(",")}]`;let Z=Q;return`{${Object.keys(Z).sort().map((J)=>`${JSON.stringify(J)}:${IQ(Z[J])}`).join(",")}}`}function wZ(Q){let Z=(J)=>J!==null&&typeof J==="object"&&!Array.isArray(J);try{let J=JSON.parse(Q);if(Array.isArray(J)){if(J.length>=2&&J.every(Z))return J;return null}}catch{}let $=[];for(let J of Q.split(`
|
|
34
|
+
`)){if(J.trim().length===0)continue;try{let z=JSON.parse(J);if(!Z(z))return null;$.push(z)}catch{return null}}return $.length>=2?$:null}function d(Q){let Z=wZ(Q);if(Z===null)return null;let $=new Set;for(let G of Z)for(let Y of Object.keys(G))$.add(Y);let J=[...$].sort(jQ);if(J.length===0)return null;let z=["·cols·"+"\t"+J.map((G)=>JSON.stringify(G)).join("\t")];for(let G of Z)z.push(J.map((Y)=>(Y in G)?IQ(G[Y]):"").join("\t"));let V=z.join(`
|
|
35
|
+
`);if(pQ(V)>=pQ(Q))return null;return{text:V,rows:Z.length,cols:J.length}}import{Buffer as lQ}from"node:buffer";var bZ=/\x1b\[[0-9;]*[A-Za-z]/g,RQ=/\b([0-9A-Za-z_]*(error|exception)s?|err|warn(ing)?s?|fail(s|ed|ure|ures)?|panic(s|ked)?|fatal|critical|traceback|denied|refused|timeouts?|timed.?out|assert(s|ed|ion|ions)?|segfault(s|ed)?)\b/i,fZ=/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/g,vZ=/\b[0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?\b/g,xZ=/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,kZ=/\b[0-9a-f]{7,64}\b/gi,hZ=/\b[0-9]+(\.[0-9]+)?[ \t\u00a0]?(ms|us|µs|ns|s|m|h|%|[KMGT]i?B)?\b/gi,mZ=/[0-9a-f]/i,uZ=3,gZ=8,PQ=2,nQ=40;function dZ(Q){if(!mZ.test(Q))return Q;let Z=Q.replace(fZ,"<ts>");return Z=Z.replace(vZ,"<time>"),Z=Z.replace(xZ,"<uuid>"),Z=Z.replace(kZ,"<hex>"),Z.replace(hZ,"<n>")}function o(Q){if(Q===32)return!0;if(Q<9)return!1;if(Q<=13)return!0;if(Q<133)return!1;return Q===133||Q===160||Q===5760||Q>=8192&&Q<=8202||Q===8232||Q===8233||Q===8239||Q===8287||Q===12288}function e(Q){let Z=0,$=Q.length;while(Z<$&&o(Q.charCodeAt(Z)))Z++;while($>Z&&o(Q.charCodeAt($-1)))$--;return Z===0&&$===Q.length?Q:Q.slice(Z,$)}function pZ(Q,Z){let $=0,J=Q.length;while($<J&&o(Q.charCodeAt($)))$++;while(J>$&&o(Q.charCodeAt(J-1)))J--;let z=0;for(let V=$;V<J&&z<Z;z++){let G=Q.charCodeAt(V);V+=G>=55296&&G<=56319&&V+1<J?2:1}return z<Z}function lZ(Q){let Z=0,$=Q.length;while(Z<$&&o(Q.charCodeAt(Z)))Z++;return Q.charCodeAt(Z)===91&&Q.charCodeAt(Z+1)===215}function nZ(Q){let Z=[],$=-1,J=!0,z=!1,V=!1,G=!0;for(let Y=0;Y<Q.length;){let H=Q.codePointAt(Y),K=H>65535?2:1,O=o(H);if(G){if(G=!1,O)Z.push("<v>")}if(O){if($>=0)Z.push(J?Q.slice($,Y):"<v>"),$=-1;V=!0}else{if(z=!0,V=!1,$<0)$=Y,J=!0;if(J){let B=H|32;if(B<97||B>122||H>127)J=!1}}Y+=K}if($>=0)Z.push(J?Q.slice($):"<v>");if(z&&V)Z.push("<v>");return Z.join(" ")}function QQ(Q,Z){let $=0,J=0,z=Q.length;while($<z&&J<Z){let V=Q.charCodeAt($);$+=V>=55296&&V<=56319&&$+1<z?2:1,J++}return $>=z?Q:Q.slice(0,$)}var cZ=/[.*+?^${}()|[\]\\]/g;function w(Q,Z=null,$=2){let J=$,V=Q.replace(bZ,"").split(`
|
|
36
|
+
`).map((q)=>{let W=q.endsWith("\r")?q.slice(0,-1):q,F=W.lastIndexOf("\r");return F===-1?W:W.slice(F+1)}),G=V.length,Y=G,H=Array(G),K=Array(G),O=0;for(let q=0;q<G;q++){H[q]=dZ(V[q]);let W=RQ.test(V[q]);if(K[q]=W,W)O++}let B=[],X=[],D=[],U=0,M=0;while(M<G){let q=0,W=0;for(let F=1;F<=gZ&&M+2*F<=G;F++){if(K[M+F-1])break;let R=1;Q:for(;;){let j=M+R*F;if(j+F>G)break;for(let T=0;T<F;T++)if(K[j+T]||H[j+T]!==H[M+T])break Q;R++}if(R>=uZ&&F*(R-1)>q*(W>0?W-1:0))q=F,W=R}if(q>0){for(let F=M;F<M+q;F++)B.push(V[F]),X.push(K[F]),D.push(H[F]);B.push(q===1?` [×${W} similar]`:` [×${W} similar ${q}-line blocks]`),X.push(!1),D.push(null),U++,M+=q*W}else B.push(V[M]),X.push(K[M]),D.push(H[M]),M++}let I=new Map,C=new Map,v=PQ+1,y=[],L=[],x=0,k=0;for(let q=0;q<B.length;q++){let W=B[q],F=X[q];if(F||lZ(W)||pZ(W,4)){y.push(W),L.push(F);continue}let R=D[q],j=I.get(R);if(j!==void 0){if(j.count++,j.count<=PQ)y.push(W),L.push(!1);else x++;continue}let T=nZ(R);I.set(R,{count:1,exemplar:W});let s=C.get(T);if(s!==void 0)if(s.count++,s.count<=v)y.push(W),L.push(!1);else k++;else C.set(T,{count:1,exemplar:W}),y.push(W),L.push(!1)}let u=[];for(let q of I.values())if(q.count>PQ)u.push([q.count,"exact",q.exemplar]);for(let q of C.values())if(q.count>v)u.push([q.count,"template",q.exemplar]);if(u.sort((q,W)=>W[0]-q[0]),u.length>nQ)u.length=nQ;if(x+k>0){let q=`── ${x+k} repeated lines suppressed (${x} exact ×N, ${k} same-template; first occurrences kept above) ──`;y.push(q),L.push(RQ.test(q));for(let[W,F,R]of u){let T=` ×${W}${F==="template"?" (template)":""} ${QQ(e(R),160)}`;y.push(T),L.push(RQ.test(T))}}let g;if(Z!=null){let q;try{q=new RegExp(Z,"i")}catch{q=new RegExp(Z.replace(cZ,"\\$&"),"i")}let W=y.length,F=new Uint8Array(W);for(let j=0;j<W;j++)if(L[j]||q.test(y[j])){let T=j>J?j-J:0,s=Math.min(j+J,W-1);F.fill(1,T,s+1)}g=[];let R=0;for(let j=0;j<W;j++)if(F[j]){if(R>0)g.push(`… ${R} lines omitted`),R=0;g.push(y[j])}else R++;if(R>0)g.push(`… ${R} lines omitted`)}else g=y;let hQ=g.join(`
|
|
37
|
+
`),mQ=lQ.byteLength(Q),uQ=lQ.byteLength(hQ),FQ=(1-uQ/mQ)*100,PZ=Q.length===0?0:FQ<0?-Math.round(-FQ):Math.round(FQ),AZ=u.map(([q,W,F])=>({count:q,exemplar:QQ(e(F),160),kind:W})),_Z={collapsedRuns:U,importantKept:O,origChars:mQ,origLines:Y,outChars:uQ,outLines:g.length,query:Z??null,savedPct:PZ,suppressedLines:x,templateSuppressed:k,topRepeats:AZ};return{distilled:hQ,stats:_Z}}var KQ=[["none","none","passthrough (baseline)"],["whitespace","lossless","trailing whitespace + blank-line runs collapsed; safe for code"],["prose","light","L1 + prose lines: collapse spaces, cut redundant filler phrases (code/IDs protected)"],["dense","medium","L2 + prose: drop articles & intensifiers"],["caveman","heavy","L3 + prose: telegraphic — drop function words; gist only, NOT verbatim"]];var b=(Q)=>new RegExp(`(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:${Q})(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])`,"giu"),AQ=[[b("in order to"),"to"],[b("due to the fact that"),"because"],[b("at this point in time"),"now"],[b("in the event that"),"if"],[b("for the purpose of"),"for"],[b("with regard to"),"about"],[b("a large number of"),"many"],[b("it is important to note that"),""],[b("please note that"),""],[b("as a matter of fact"),""],[b("in terms of"),"for"],[b("the fact that"),"that"]],oZ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:the|an|a)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),iZ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:very|really|just|actually|basically|simply|quite|rather|essentially|literally)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),rZ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:is|are|was|were|am|be|been|being|do|does|did|have|has|had|will|would|shall|should|can|could|may|might|of|to|in|on|at|for|with|that|this|these|those|it|its|there|here)(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]*","giu"),cQ=/ {2,}/g,aZ=new RegExp("[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+([.,;:!?])","gu"),sZ=/\n{3,}/g;function _Q(Q){if(Q===32||Q>=9&&Q<=13)return!0;if(Q<133)return!1;return Q===133||Q===160||Q===5760||Q>=8192&&Q<=8202||Q===8232||Q===8233||Q===8239||Q===8287||Q===12288}function tZ(Q){let Z=0,$=Q.length;while(Z<$){let J=Q.codePointAt(Z);if(!_Q(J))break;Z+=J>65535?2:1}while($>Z){let J=Q.charCodeAt($-1);if(!_Q(J))break;$--}return Z===0&&$===Q.length?Q:Q.slice(Z,$)}function eZ(Q){if(Q.length===0)return!1;let Z=Q.charCodeAt(0);if(Z===32||Z===9)return!0;let $=0,J=0,z=0,V=!1;for(let G=0;G<Q.length;){let Y=Q.codePointAt(G);if(G+=Y>65535?2:1,$++,_Q(Y)){z=0;continue}if(z++,z>=24)V=!0;if(Y>=48&&Y<=57||Y>=65&&Y<=90||Y>=97&&Y<=122)continue;switch(Y){case 46:case 44:case 59:case 58:case 39:case 34:case 33:case 63:case 40:case 41:case 45:continue}J++}if(J/$>0.3)return!0;return V}function Q$(Q,Z){let $=Q.replace(cQ," ");for(let z=0;z<AQ.length;z++)$=$.replace(AQ[z][0],AQ[z][1]);if(Z>=3)$=$.replace(oZ,""),$=$.replace(iZ,"");if(Z>=4)$=$.replace(rZ,"");$=$.replace(cQ," "),$=tZ($.replace(aZ,"$1"));let J=$.charCodeAt(0);if(J>=97&&J<=122)$=String.fromCharCode(J-32)+$.slice(1);return $}function ZQ(Q,Z){let $=Math.min(Z,4);if($===0)return{compressed:Q,protectedLines:0,level:$};let J=0,z=Q.split(`
|
|
38
|
+
`),V=Array(z.length);for(let Y=0;Y<z.length;Y++){let H=z[Y],K=H.length;while(K>0){let B=H.charCodeAt(K-1);if(B!==32&&B!==9)break;K--}let O=K===H.length?H:H.slice(0,K);if($===1){V[Y]=O;continue}if(eZ(O)){J++,V[Y]=O;continue}V[Y]=Q$(O,$)}return{compressed:V.join(`
|
|
39
|
+
`).replace(sZ,`
|
|
8
40
|
|
|
9
|
-
`),protectedLines
|
|
41
|
+
`),protectedLines:J,level:$}}import{appendFileSync as S$,mkdirSync as E$}from"node:fs";import{dirname as C$}from"node:path";import XZ from"node:http";import L$ from"node:https";import{URL as T$}from"node:url";import{readFileSync as EQ}from"node:fs";import{inflateSync as Z$}from"node:zlib";//! Full-BMP glyph atlas (Spleen 5x8 for ASCII/code + Unifont fallback),
|
|
10
42
|
//! extracted from pxpipe's generated gray atlas by `tools/gen-glyphs.mjs`.
|
|
11
43
|
//!
|
|
12
44
|
//! Codepoints + wide flags load eagerly (~175 KB, needed for wrap math);
|
|
13
45
|
//! coverage pixels stay zlib-packed on disk and decompress lazily on the
|
|
14
46
|
//! first blit, so `estimate` never pays for them.
|
|
15
|
-
var h=5
|
|
47
|
+
var h=5,$Q=8,p=EQ(new URL("../assets/glyphs.cps",import.meta.url)),XQ=EQ(new URL("../assets/glyphs.wide",import.meta.url)),oQ=(()=>{let Q=p.byteLength>>>2;if((p.byteOffset&3)===0)return new Uint32Array(p.buffer,p.byteOffset,Q);return new Uint32Array(p.buffer.slice(p.byteOffset,p.byteOffset+(Q<<2)))})(),$$=(()=>{let Q=new Uint32Array(XQ.length),Z=0;for(let $=0;$<XQ.length;$++)Q[$]=Z,Z+=(XQ[$]===1?2*h:h)*$Q;return Q})(),SQ=null;function JQ(Q){let Z=0,$=oQ.length-1;while(Z<=$){let J=Z+$>>>1,z=oQ[J];if(z<Q)Z=J+1;else if(z>Q)$=J-1;else return J}return-1}function i(Q){return XQ[Q]===1}function CQ(Q){if(SQ===null)SQ=Z$(EQ(new URL("../assets/glyphs.pix.z",import.meta.url)));let Z=$$[Q],$=i(Q)?2*h:h;return SQ.subarray(Z,Z+$*$Q)}function iQ(Q,Z,$){let J=CQ(Q),z=i(Q)?2*h:h;return J$(J,z,$Q,Z,$)}function J$(Q,Z,$,J,z){let V=new Uint8Array(J*z),G=Z/J,Y=$/z;for(let H=0;H<z;H++){let K=H*Y,O=(H+1)*Y;for(let B=0;B<J;B++){let X=B*G,D=(B+1)*G,U=0,M=0,I=Math.min(Math.ceil(O),$);for(let C=Math.floor(K);C<I;C++){let v=Math.min(C+1,O)-Math.max(C,K);if(v<=0)continue;let y=Math.min(Math.ceil(D),Z);for(let L=Math.floor(X);L<y;L++){let x=Math.min(L+1,D)-Math.max(L,X);if(x<=0)continue;let k=x*v;U+=k*Q[C*Z+L],M+=k}}V[H*J+B]=M>0?Math.min(255,Math.max(0,Math.round(U/M))):0}}return V}import{deflateSync as z$}from"node:zlib";//! Minimal grayscale PNG encoder (bit depth 8, color type 0), mirroring
|
|
16
48
|
//! pxpipe's png.ts: IHDR + one IDAT (zlib) + IEND, filter byte 0 per row.
|
|
17
|
-
var
|
|
49
|
+
var G$=(()=>{let Q=new Uint32Array(256);for(let Z=0;Z<256;Z++){let $=Z;for(let J=0;J<8;J++)$=($&1)!==0?3988292384^$>>>1:$>>>1;Q[Z]=$>>>0}return Q})();function LQ(Q,Z,$,J){let z=J.length;Q[Z]=z>>>24&255,Q[Z+1]=z>>>16&255,Q[Z+2]=z>>>8&255,Q[Z+3]=z&255;let V=Z+4;for(let H=0;H<4;H++)Q[V+H]=$.charCodeAt(H);Q.set(J,V+4);let G=4294967295;for(let H=V;H<V+4+z;H++)G=G$[(G^Q[H])&255]^G>>>8;G=~G>>>0;let Y=V+4+z;return Q[Y]=G>>>24&255,Q[Y+1]=G>>>16&255,Q[Y+2]=G>>>8&255,Q[Y+3]=G&255,Y+4}function rQ(Q,Z,$){let J=new Uint8Array($*(Z+1));for(let H=0,K=0,O=0;H<$;H++)J[K++]=0,J.set(Q.subarray(O,O+Z),K),K+=Z,O+=Z;let z=z$(J,{level:6}),V=new Uint8Array(33+(12+z.length)+12);V[0]=137,V[1]=80,V[2]=78,V[3]=71,V[4]=13,V[5]=10,V[6]=26,V[7]=10;let G=new Uint8Array(13);G[0]=Z>>>24&255,G[1]=Z>>>16&255,G[2]=Z>>>8&255,G[3]=Z&255,G[4]=$>>>24&255,G[5]=$>>>16&255,G[6]=$>>>8&255,G[7]=$&255,G[8]=8,G[9]=0;let Y=LQ(V,8,"IHDR",G);return Y=LQ(V,Y,"IDAT",z),LQ(V,Y,"IEND",new Uint8Array(0)),V}//! Stage 2: the `pxpipe` imaging engine, ported from pxpipe's render.ts
|
|
18
50
|
//! production dense path (bare 5x8 AA cell, 312 cols, 1568x728 pages).
|
|
19
51
|
//! Glyphs cover BMP (pxpipe's atlas: Spleen for ASCII/code, Unifont fallback)
|
|
20
52
|
//! PLUS the astral planes (unifont_upper, incl. emoji) — beyond pxpipe, which
|
|
@@ -29,19 +61,19 @@ var cQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $
|
|
|
29
61
|
//! * `font` — `Tiny` renders the same atlas box-filtered into a 4x6 cell
|
|
30
62
|
//! (390 cols x 120 rows/page), ~40% fewer image-tokens; opt-in,
|
|
31
63
|
//! transcription-accuracy gated.
|
|
32
|
-
var
|
|
33
|
-
`);for(let
|
|
34
|
-
`).replace(
|
|
64
|
+
var zQ=4,OQ=4,V$=1568,Y$=728,BQ="↵",H$="⏎",DQ="→",K$="⇢",TQ="⇥",X$="⇨",O$=9647,aQ=4,B$=3,sQ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",D$=/\n{4,}/g;function f(Q){return Q.length===4&&Q.replace(/[A-Z]/g,(Z)=>Z.toLowerCase())==="tiny"?"tiny":"normal"}function ZZ(Q){let Z=Q==="tiny"?4:h,$=Q==="tiny"?6:$Q,J=Math.floor((V$-2*zQ)/Z),z=Math.floor((Y$-2*OQ)/$);return{cw:Z,ch:$,cols:J,maxLines:z,maxChars:J*z}}var U$=(()=>{let Q=new Uint8Array(128);for(let Z=0;Z<128;Z++){let $=JQ(Z);Q[Z]=$>=0&&i($)?2:1}return Q})();function yQ(Q){if(Q<128)return U$[Q];let Z=JQ(Q);return Z>=0&&i(Z)?2:1}function q$(Q){let Z=0;for(let $=0;$<Q.length;$++){let J=Q.charCodeAt($);if(J>=55296&&J<56320&&$+1<Q.length){let z=Q.charCodeAt($+1);if(z>=56320&&z<57344)$++}Z++}return Z}function wQ(Q){let Z=Q.split(`
|
|
65
|
+
`);for(let $=0;$<Z.length;$++){let J=Z[$],z=J.length;while(z>0){let V=J.charCodeAt(z-1);if(V!==32&&V!==9)break;z--}if(z!==J.length)Z[$]=J.slice(0,z)}return Z.join(`
|
|
66
|
+
`).replace(D$,`
|
|
35
67
|
|
|
36
68
|
|
|
37
|
-
`)}function
|
|
38
|
-
`);for(let
|
|
39
|
-
`);for(let
|
|
40
|
-
`)){let z
|
|
69
|
+
`)}function $Z(Q){if(!Q.includes("\t"))return Q;let Z="",$=0;for(let J of Q)if(J==="\t"){let z=aQ-$%aQ;Z+=DQ;for(let V=1;V<z;V++)Z+=" ";$+=z}else Z+=J,$+=yQ(J.codePointAt(0));return Z}function M$(Q){let Z=Q.includes("\t")?Q.replaceAll("\t",DQ):Q,$=0;while($<Z.length&&Z.charCodeAt($)===32)$++;if($>=B$&&$<sQ.length)return TQ+sQ[$]+Z.slice($);return Z}function JZ(Q){return Q.includes(BQ)?Q.replaceAll(BQ,H$):Q}function W$(Q){let Z=JZ(Q);if(Z.includes(DQ))Z=Z.replaceAll(DQ,K$);if(Z.includes(TQ))Z=Z.replaceAll(TQ,X$);return Z}function F$(Q){let Z=wQ(Q).split(`
|
|
70
|
+
`);for(let $=0;$<Z.length;$++)Z[$]=$Z(Z[$]);return Z.join(BQ)}function N$(Q){let Z=wQ(Q).split(`
|
|
71
|
+
`);for(let $=0;$<Z.length;$++)Z[$]=M$(Z[$]);return Z.join(BQ)}function j$(Q,Z){let $=[],J=wQ(Q);for(let z of J.split(`
|
|
72
|
+
`)){let V=$Z(z);if(V.length===0){$.push("");continue}let G="",Y=0;for(let H of V){let K=yQ(H.codePointAt(0));if(Y+K>Z)$.push(G),G=H,Y=K;else G+=H,Y+=K}if(G.length!==0)$.push(G)}return $}function I$(Q,Z,$){let J=[],z=[],V=0;for(let G of Q){let Y=q$(G),H=Y+(z.length!==0?1:0);if(z.length!==0&&(z.length>=Z||V+H>$))J.push(z),z=[],V=0;V+=Y+(z.length!==0?1:0),z.push(G)}if(z.length!==0)J.push(z);return J}function zZ(Q){if(Q<32)return!0;if(Q>=127&&Q<=159)return!0;if(Q>=768&&Q<=879)return!0;if(Q===8203||Q===8204||Q===8205||Q===8288||Q===65279)return!0;if(Q>=65024&&Q<=65039)return!0;if(Q>=917760&&Q<=917999)return!0;return!1}function R$(Q){let Z=null,$=0;for(let J of Q){let z=J.codePointAt(0);if(JQ(z)<0&&!zZ(z)){if(Z===null)Z=Q.slice(0,$);Z+=`[U+${z.toString(16).toUpperCase()}]`}else if(Z!==null)Z+=J;$+=J.length}return Z??Q}function GZ(Q,Z,$,J){let z=Z?$?N$(W$(Q)):F$(JZ(Q)):Q;return I$(j$(R$(z),J.cols),J.maxLines,J.maxChars)}function VZ(Q,Z,$){if(!$)return 2*zQ+Z.cols*Z.cw;let J=0;for(let G of Q){let Y=0;for(let H of G)Y+=yQ(H.codePointAt(0));if(Y>Z.cols)Y=Z.cols;if(Y>J)J=Y}let z=2*zQ+J*Z.cw,V=2*zQ+Z.cw;return z>V?z:V}var tQ=new Map;function eQ(Q,Z,$,J,z,V,G){let Y=JQ(z);if(Y<0)return 0;let H=i(Y),K=H?2*V.cw:V.cw,O;if(G==="normal")O=CQ(Y);else{let B=tQ.get(Y);if(B===void 0)B=iQ(Y,K,V.ch),tQ.set(Y,B);O=B}for(let B=0;B<V.ch;B++){let X=(J+B)*Z+$,D=B*K;for(let U=0;U<K;U++){let M=O[D+U];if(M>0){let I=X+U;if(M>Q[I])Q[I]=M}}}return H?2:1}function P$(Q,Z,$,J){let z=VZ(Q,Z,$),V=2*OQ+Q.length*Z.ch,G=new Uint8Array(z*V),Y=0;for(let H=0;H<Q.length;H++){let K=OQ+H*Z.ch,O=0;for(let B of Q[H]){if(O>=Z.cols)break;let X=zQ+O*Z.cw,D=eQ(G,z,X,K,B.codePointAt(0),Z,J);if(D===0){if(D=1,zZ(B.codePointAt(0)));else if(Y++,B!==" ")eQ(G,z,X,K,O$,Z,J)}O+=D}}for(let H=0;H<G.length;H++)G[H]=255-G[H];return{png:rQ(G,z,V),width:z,height:V,dropped:Y}}function m(Q,Z,$,J){let z=ZZ(J),V=GZ(Q,Z,$,z).map((K)=>P$(K,z,$,J)),G=0,Y=0,H=0;for(let K of V)G+=K.width*K.height,Y+=YZ(K.width,K.height),H+=K.dropped;return{pages:V,pixels:G,tokens:Y,dropped:H}}function UQ(Q,Z,$,J){let z=ZZ(J),V=GZ(Q,Z,$,z),G=0,Y=0,H=[];for(let K of V){let O=VZ(K,z,$),B=2*OQ+K.length*z.ch;H.push([O,B]),G+=O*B,Y+=YZ(O,B)}return{pages:V.length,pixels:G,tokens:Y,dims:H}}var QZ=28;function YZ(Q,Z){return Math.ceil(Q/QZ)*Math.ceil(Z/QZ)}import{readFileSync as A$}from"node:fs";import{join as _$}from"node:path";//! pxpipe measurement-log summary (~/.pxpipe/events.jsonl), same math as the
|
|
41
73
|
//! node MCP: actual = every way input bytes get billed (input + cache reads +
|
|
42
74
|
//! cache creates) — ignoring cache_read would fake the savings.
|
|
43
|
-
class
|
|
44
|
-
`)){if(
|
|
75
|
+
class r{value;constructor(Q){this.value=Q}}function bQ(){let Q=process.env.TANUKI_EVENTS;if(Q!==void 0)return Q;let Z=process.env.HOME??"";return _$(Z,".pxpipe","events.jsonl")}function l(Q){return typeof Q==="number"&&Number.isSafeInteger(Q)&&Q>=0?Q:0}function HZ(Q){return Q<0?-Math.round(-Q):Math.round(Q)}function KZ(){let Q=bQ(),Z;try{Z=A$(Q,"utf8")}catch{return{available:!1,note:`no ${Q} yet`}}let $=0,J=0,z=0,V=0,G=0,Y=0,H=0;for(let O of Z.split(`
|
|
76
|
+
`)){if(O.trim().length===0)continue;let B;try{B=JSON.parse(O)}catch{continue}$+=1;let X=B!==null&&typeof B==="object"&&!Array.isArray(B)?B:{};if(X.compressed===!0)J+=1,z+=l(X.orig_chars),V+=l(X.image_count);G+=l(X.baseline_tokens),Y+=l(X.input_tokens)+l(X.cache_read_tokens)+l(X.cache_create_tokens),H+=l(X.output_tokens)}let K=G>0&&Y>0?new r(HZ((1-Y/G)*1000)/10):null;return{available:!0,requests:$,compressedRequests:J,imagedChars:z,imagesEmitted:V,baselineTokens:G,actualInputTokens:Y,outputTokens:H,outputSharePct:H>0?new r(HZ(H/(Y+H)*1000)/10):null,estInputSavedPct:K}}//! Implicit mode: a local Anthropic middlebox, the pxpipe deployment shape
|
|
45
77
|
//! without pxpipe's structural flaw. Rules that keep it injection-shaped-free:
|
|
46
78
|
//!
|
|
47
79
|
//! 1. The system prompt and tool definitions are NEVER touched.
|
|
@@ -57,12 +89,12 @@ class ZJ{value;constructor(J){this.value=J}}function TJ(){let J=process.env.TANU
|
|
|
57
89
|
//!
|
|
58
90
|
//! Responses stream through untouched; usage is scraped from the stream for
|
|
59
91
|
//! the ~/.pxpipe/events.jsonl savings log (same format tanuki_stats reads).
|
|
60
|
-
var
|
|
61
|
-
`)}catch{}}function
|
|
92
|
+
var a={level:0,distill:!1,table:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,minSave:300,maxPages:20};function fQ(Q){let Z=0;for(let $=0;$<Q.length;$++){let J=Q.charCodeAt($);if(J<55296||J>56319)Z++}return Z}function y$(Q,Z){let $=fQ(Q);if($<Z.minChars)return null;let J=Q;if(Z.table){let K=d(J);if(K!==null)J=K.text}if(Z.distill)J=w(J,null,2).distilled;let z=0;if(Z.codebook){let K=t(J);J=K.text,z=K.entries}if(Z.level>0)J=ZQ(J,Z.level).compressed;let V=Math.round($/4),G=m(J,!0,!0,Z.font);if(G.pages.length>Z.maxPages)return null;if(G.tokens>V*Z.ratio||V-G.tokens<Z.minSave)return null;let H=[{type:"text",text:`[tanuki-context: ${$} chars imaged in place as ${G.pages.length} PNG page(s), ~${G.tokens} vs ~${V} text tokens. ↵=newline →=tab ⇥N=indent`+(z>0?`; ·legend· line maps ${z} sigils`:"")+"]"}];for(let K of G.pages)H.push({type:"image",source:{type:"base64",media_type:"image/png",data:Buffer.from(K.png.buffer,K.png.byteOffset,K.png.byteLength).toString("base64")}});return{blocks:H,origChars:$,pages:G.pages.length,savedTokens:V-G.tokens}}function qQ(Q){return Q!==null&&typeof Q==="object"&&!Array.isArray(Q)}function w$(Q,Z){let $;try{$=JSON.parse(Q)}catch{return null}if(!qQ($)||!Array.isArray($.messages))return null;let J=0,z=0,V=0,G=0,Y=new Map,H=(K)=>{let O=Y.get(K),B;if(O!==void 0){let X=fQ(K),D=`[tanuki-context: ${X} chars, byte-identical to a block imaged above (${O} PNG page(s)); not repeated]`;B={blocks:[{type:"text",text:D}],origChars:X,pages:0,savedTokens:Math.round(X/4)-Math.round(fQ(D)/4)}}else if(B=y$(K,Z),B)Y.set(K,B.pages);if(B)J++,z+=B.origChars,V+=B.pages,G+=B.savedTokens;return B};for(let K=0;K<$.messages.length-1;K++){let O=$.messages[K];if(!qQ(O)||O.role!=="user")continue;if(typeof O.content==="string"){let X=H(O.content);if(X)O.content=X.blocks;continue}if(!Array.isArray(O.content))continue;let B=[];for(let X of O.content){if(!qQ(X)||X.cache_control!==void 0){B.push(X);continue}if(X.type==="text"&&typeof X.text==="string"){let D=H(X.text);if(D)B.push(...D.blocks);else B.push(X);continue}if(X.type==="tool_result"){if(typeof X.content==="string"){let D=H(X.content);if(D)X.content=D.blocks}else if(Array.isArray(X.content)){let D=[];for(let U of X.content){if(qQ(U)&&U.type==="text"&&typeof U.text==="string"&&U.cache_control===void 0){let M=H(U.text);if(M){D.push(...M.blocks);continue}}D.push(U)}X.content=D}}B.push(X)}O.content=B}if(J===0)return null;return{body:JSON.stringify($),imagedBlocks:J,origChars:z,imageCount:V,savedTokens:G}}function b$(Q){let Z=(J)=>{let z=J.exec(Q);return z?Number(z[1]):0},$=0;for(let J of Q.matchAll(/"output_tokens"\s*:\s*(\d+)/g))$=Math.max($,Number(J[1]));return{input:Z(/"input_tokens"\s*:\s*(\d+)/),cacheRead:Z(/"cache_read_input_tokens"\s*:\s*(\d+)/),cacheCreate:Z(/"cache_creation_input_tokens"\s*:\s*(\d+)/),output:$}}function f$(Q){try{let Z=bQ();E$(C$(Z),{recursive:!0}),S$(Z,JSON.stringify(Q)+`
|
|
93
|
+
`)}catch{}}function OZ(Q){let Z=new T$(Q.upstream),$=Z.protocol==="https:"?L$:XZ,J=XZ.createServer((z,V)=>{let G=[];z.on("data",(Y)=>G.push(Y)),z.on("end",()=>{let Y=Buffer.concat(G),H=z.method==="POST"&&(z.url??"").startsWith("/v1/messages")&&!(z.url??"").includes("count_tokens")&&z.headers["content-encoding"]===void 0,K=null;if(H){if(K=w$(Y.toString("utf8"),Q),K)Y=Buffer.from(K.body,"utf8")}let O={...z.headers};if(delete O.host,delete O.connection,O["content-length"]=String(Y.length),H)delete O["accept-encoding"];let B=$.request({protocol:Z.protocol,hostname:Z.hostname,port:Z.port||(Z.protocol==="https:"?443:80),path:z.url,method:z.method,headers:O},(X)=>{if(V.writeHead(X.statusCode??502,X.headers),H){let D=[];X.on("data",(U)=>{D.push(U),V.write(U)}),X.on("end",()=>{V.end();let U=b$(Buffer.concat(D).toString("utf8")),M=U.input+U.cacheRead+U.cacheCreate;f$({ts:Date.now(),tool:"proxy",compressed:K!==null,orig_chars:K?.origChars??0,image_count:K?.imageCount??0,baseline_tokens:M+(K?.savedTokens??0),input_tokens:U.input,cache_read_tokens:U.cacheRead,cache_create_tokens:U.cacheCreate,output_tokens:U.output})})}else X.pipe(V)});B.on("error",(X)=>{V.writeHead(502,{"content-type":"application/json"}),V.end(JSON.stringify({type:"error",error:{type:"api_error",message:`tanuki proxy: upstream unreachable (${X.message})`}}))}),B.end(Y)})});return J.listen(Q.port,"127.0.0.1",()=>{let z=J.address(),V=z!==null&&typeof z==="object"?z.port:Q.port,G=`level=${Q.level} distill=${Q.distill} codebook=${Q.codebook} font=${Q.font} minChars=${Q.minChars} ratio=${Q.ratio} minSave=${Q.minSave}`;process.stderr.write(`tanuki-context proxy on http://127.0.0.1:${V} -> ${Q.upstream}
|
|
62
94
|
${G}
|
|
63
95
|
`+` rules: system prompt & tools untouched · in-place blocks only · latest message never imaged · cache_control skipped · identical blocks imaged once
|
|
64
|
-
`+` point your client at it: export ANTHROPIC_BASE_URL=http://127.0.0.1:${
|
|
65
|
-
`)})
|
|
96
|
+
`+` point your client at it: export ANTHROPIC_BASE_URL=http://127.0.0.1:${V}
|
|
97
|
+
`)}),J}import{createHash as v$}from"node:crypto";import{mkdirSync as x$,readFileSync as k$,writeFileSync as h$}from"node:fs";import BZ from"node:process";//! Stash mode: context-mode's shape (content parked outside the context
|
|
66
98
|
//! window, queried on demand) fused with tanuki's pricing (big answers come
|
|
67
99
|
//! back as dense pages).
|
|
68
100
|
//!
|
|
@@ -71,11 +103,11 @@ var o={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,min
|
|
|
71
103
|
//! repeats, first/last lines, the id). fetch = pull a slice by regex query
|
|
72
104
|
//! (distill-powered) or line range; the caller images it only when pages
|
|
73
105
|
//! clearly win. Contract is byte-identical with the Rust engine.
|
|
74
|
-
function
|
|
75
|
-
`),
|
|
76
|
-
`)}}function
|
|
77
|
-
`),G=Math.max(1,Number(
|
|
78
|
-
`)}return w(
|
|
106
|
+
function DZ(){let Q=BZ.env.TANUKI_STASH;if(Q!==void 0&&Q!=="")return Q;return`${BZ.env.HOME??""}/.tanuki/stash`}function GQ(Q){let Z=v$("sha256").update(Q,"utf8").digest("hex").slice(0,12),$=DZ();x$($,{recursive:!0}),h$(`${$}/${Z}`,Q);let J=Buffer.byteLength(Q,"utf8"),z=Q.split(`
|
|
107
|
+
`),V=w(Q,null,2).stats,G=[`stashed ${Z} · ${J} bytes · ${z.length} lines`,`distill map: ${V.origLines} -> ${V.outLines} lines · ${V.savedPct}% of chars removable · ${V.importantKept} error/warn lines`];if(V.topRepeats.length>0){G.push("top repeats:");for(let H of V.topRepeats.slice(0,5)){let K=H.kind==="template"?" (template)":"";G.push(` ×${H.count}${K} ${H.exemplar}`)}}let Y="";for(let H=z.length-1;H>=0;H--)if(z[H]!==""){Y=z[H];break}return G.push(`first: ${QQ(e(z[0]),160)}`),G.push(`last: ${QQ(e(Y),160)}`),G.push(`fetch: tanuki_fetch {"id":"${Z}","query":"<regex>"} or {"id":"${Z}","lines":"a-b"}`),{id:Z,overview:G.join(`
|
|
108
|
+
`)}}function vQ(Q,Z,$){if(Z===null===($===null))throw Error("give exactly one of query or lines");let J;try{J=k$(`${DZ()}/${Q}`,"utf8")}catch{throw Error(`unknown stash id: ${Q}`)}if($!==null){let z=/^(\d+)-(\d+)$/.exec($);if(z===null)throw Error("bad lines range");let V=J.split(`
|
|
109
|
+
`),G=Math.max(1,Number(z[1])),Y=Math.min(V.length,Number(z[2]));if(Number(z[1])>Number(z[2]))throw Error("bad lines range");return V.slice(G-1,Y).join(`
|
|
110
|
+
`)}return w(J,Z,2).distilled}//! tanuki-context — token-cutting context pipeline.
|
|
79
111
|
//! pipeline: text -> distill (stage 0, logs) -> ladder level 0-4 (stage 1)
|
|
80
112
|
//! -> pxpipe imaging (stage 2, name kept from the original mechanic)
|
|
81
113
|
//!
|
|
@@ -84,30 +116,31 @@ function zQ(){let J=VQ.env.TANUKI_STASH;if(J!==void 0&&J!=="")return J;return`${
|
|
|
84
116
|
//! tanuki-context estimate <file> [level] [--distill]
|
|
85
117
|
//! tanuki-context render <file> [level] [outdir]
|
|
86
118
|
//! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
|
|
87
|
-
var
|
|
88
|
-
`;for(let
|
|
89
|
-
`;K+=H+
|
|
90
|
-
|
|
91
|
-
`;for(let H=0;H<
|
|
92
|
-
`;Y+=G+JSON.stringify(
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
`)}function
|
|
97
|
-
`)
|
|
98
|
-
`);
|
|
99
|
-
`);break}case"
|
|
100
|
-
`);
|
|
101
|
-
`);
|
|
102
|
-
`);break}case"
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
`);
|
|
119
|
+
var g$="0.6.0",xQ=6,kQ=8000;function d$(Q,Z){let $=Math.min(Q.length,Z.length),J=0;while(J<$&&Q.charCodeAt(J)===Z.charCodeAt(J))J++;if(J>=$)return Q.length-Z.length;return Q.codePointAt(J)-Z.codePointAt(J)}function _(Q,Z,$=""){if(Q===null||Q===void 0)return"null";if(Q instanceof r){let H=Q.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let J=typeof Q;if(J==="string")return JSON.stringify(Q);if(J==="number"||J==="boolean")return String(Q);if(Array.isArray(Q)){if(Q.length===0)return"[]";if(!Z){let O="[";for(let B=0;B<Q.length;B++){if(B>0)O+=",";O+=_(Q[B],!1)}return O+"]"}let H=$+" ",K=`[
|
|
120
|
+
`;for(let O=0;O<Q.length;O++){if(O>0)K+=`,
|
|
121
|
+
`;K+=H+_(Q[O],!0,H)}return K+`
|
|
122
|
+
`+$+"]"}let z=Q,V=Object.keys(z).sort(d$);if(V.length===0)return"{}";if(!Z){let H="{";for(let K=0;K<V.length;K++){if(K>0)H+=",";H+=JSON.stringify(V[K])+":"+_(z[V[K]],!1)}return H+"}"}let G=$+" ",Y=`{
|
|
123
|
+
`;for(let H=0;H<V.length;H++){if(H>0)Y+=`,
|
|
124
|
+
`;Y+=G+JSON.stringify(V[H])+": "+_(z[V[H]],!0,G)}return Y+`
|
|
125
|
+
`+$+"}"}function N(Q,Z){return Q!==null&&typeof Q==="object"&&!Array.isArray(Q)?Q[Z]:void 0}function E(Q){return typeof Q==="string"?Q:null}function n(Q){return typeof Q==="boolean"?Q:null}function NZ(Q){return typeof Q==="number"&&Number.isSafeInteger(Q)&&Q>=0?Q:null}function S(Q){let Z=0;for(let $=0;$<Q.length;$++){Z++;let J=Q.charCodeAt($);if(J>=55296&&J<=56319&&$+1<Q.length){let z=Q.charCodeAt($+1);if(z>=56320&&z<=57343)$++}}return Z}function WQ(Q,Z,$,J,z,V){let G=Q,Y=null;if(V){let B=d(G);if(B!==null)G=B.text,Y={rows:B.rows,cols:B.cols}}let H=null;if($||J!==null){let B=w(G,J,2);G=B.distilled,H=B.stats}let K=0;if(z){let B=t(G);G=B.text,K=B.entries}let O=ZQ(G,Z);return{stage0:H,compressed:O.compressed,protectedLines:O.protectedLines,level:O.level,cbEntries:K,table:Y}}function c(Q){return Math.round(Q/4)}function YQ(Q,Z){if(Q===0)return 0;let $=(1-Z/Q)*100;return $<0?-Math.round(-$):Math.round($)}function jZ(Q){return{text:E(N(Q,"text"))??"",level:(NZ(N(Q,"level"))??0)%256,distill:n(N(Q,"distill"))??!1,query:E(N(Q,"query")),reflow:n(N(Q,"reflow"))??!0,pack:n(N(Q,"pack"))??!0,font:E(N(Q,"font"))??"normal",codebook:n(N(Q,"codebook"))??!1,table:n(N(Q,"table"))??!1}}function p$(Q){let Z=(H)=>{let K={codebook:!1,tokens:1/0,pages:0,text:H};for(let O of[!1,!0]){let B=O?t(H).text:H,X=UQ(B,!0,!0,f("normal"));if(X.tokens<K.tokens)K={codebook:O,tokens:X.tokens,pages:X.pages,text:B}}return K},$=Z(Q),J=!1,z=d(Q);if(z!==null){let H=Z(z.text);if(H.tokens<$.tokens)$=H,J=!0}let V=J&&z!==null?z.text:Q,G=Z(w(V,null,2).distilled),Y=UQ($.text,!0,!0,f("tiny"));return{codebook:$.codebook,imageTokens:$.tokens,pages:$.pages,table:J,tinyImageTokens:Y.tokens,withDistill:{codebook:G.codebook,imageTokens:G.tokens}}}function IZ(Q){let Z=jZ(Q),$=WQ(Z.text,Z.level,Z.distill,Z.query,Z.codebook,Z.table),J=f(Z.font),z=UQ($.compressed,Z.reflow,Z.pack,J),V=z.tokens,G=S(Z.text),Y=S($.compressed),H=c(G),[K,O]=KQ[$.level],B=E(N(Q,"model")),X=n(N(Q,"cached"))??!1,D={engine:"pxpipe",level:`${$.level} ${K}`,loss:O,distill:$.stage0,origChars:G,stage1Chars:Y,stage1SavedPct:YQ(G,Y),pages:z.pages,imageTokens:V,rawTextTokens:H,totalSavedPct:YQ(H,V),protectedLines:$.protectedLines,pack:Z.pack,font:J==="tiny"?"tiny":"normal",codebook:Z.codebook?$.cbEntries:!1,table:$.table!==null?$.table:!1,verdict:V<H?"PIPELINE cheaper":"TEXT cheaper",recommend:p$(Z.text)};if(B!==null||X)D.cost=dQ(H,V,{model:B,cached:X},{dims:z.dims});return D}function l$(Q){let Z=jZ(Q),$=WQ(Z.text,Z.level,Z.distill,Z.query,Z.codebook,Z.table),J=f(Z.font),z=m($.compressed,Z.reflow,Z.pack,J),V=z.tokens,G=S(Z.text),Y=S($.compressed),H=c(G),[K,O]=KQ[$.level],B="";if($.table!==null)B+=`table: ${$.table.rows} rows x ${$.table.cols} cols, keys stated once
|
|
126
|
+
`;if($.stage0!==null){let U=$.stage0;B+=`distill: ${U.origLines} -> ${U.outLines} lines (-${U.savedPct}% chars, ${U.collapsedRuns} runs, ${U.suppressedLines} exact + ${U.templateSuppressed} template suppressed, ${U.importantKept} error/warn kept)
|
|
127
|
+
`}if(B+=`L${$.level} ${K} (${O}): ${G} chars -> ${Y} chars (stage1 -${YQ(G,Y)}%) -> ${z.pages.length} page(s), ~${V} image-tokens
|
|
128
|
+
vs ~${H} text-tokens raw = TOTAL -${YQ(H,V)}%`,$.protectedLines>0)B+=` · ${$.protectedLines} lines kept verbatim`;if(z.dropped>0)B+=` · ${z.dropped} unmapped glyphs -> ▯`;if($.cbEntries>0)B+=` · codebook: ${$.cbEntries} sigils (see ·legend·)`;if(J==="tiny")B+=" · font: tiny 4x6";if(Z.pack)B+=" · packed (⇥N indent, → tab)";if(Z.reflow)B+=" · ↵ = newline · engine: pxpipe";let X=[{type:"text",text:B}],D=Math.min(z.pages.length,xQ);for(let U=0;U<D;U++){let M=z.pages[U].png;X.push({type:"image",data:Buffer.from(M.buffer,M.byteOffset,M.byteLength).toString("base64"),mimeType:"image/png"})}if(z.pages.length>xQ)X.push({type:"text",text:`(+${z.pages.length-xQ} more page(s))`});return X}function n$(Q){let $=E(N(Q,"text"))??"",J=null;if(n(N(Q,"table"))??!1){let G=d($);if(G!==null)$=G.text,J={rows:G.rows,cols:G.cols}}let z=w($,E(N(Q,"query")),2),V=J!==null?{...z.stats,table:J}:z.stats;return[{type:"text",text:_(V,!0)},{type:"text",text:z.distilled}]}function c$(Q){let Z=E(N(Q,"text"))??"",$=(NZ(N(Q,"level"))??1)%256,J=ZQ(Z,$),[z,V,G]=KQ[J.level],Y=S(Z),H=S(J.compressed),K=c(Y),O=c(H),B={level:`${J.level} ${z}`,loss:V,note:G,origChars:Y,outChars:H,approxOrigTokens:K,approxOutTokens:O,savedPct:YQ(K,O),protectedLines:J.protectedLines};return[{type:"text",text:_(B,!0)},{type:"text",text:J.compressed}]}function o$(Q){let Z=E(N(Q,"text"))??"";return[{type:"text",text:GQ(Z).overview}]}function i$(Q){let Z=E(N(Q,"id"))??"",$=vQ(Z,E(N(Q,"query"))??null,E(N(Q,"lines"))??null),J=c(S($)),z=m($,!0,!0,f("normal"));if(!(z.tokens<=J*0.75&&J-z.tokens>=300&&z.pages.length<=6))return[{type:"text",text:$}];let Y=[{type:"text",text:`[tanuki-context stash ${Z}: slice of ${S($)} chars imaged as ${z.pages.length} PNG page(s), ~${z.tokens} vs ~${J} text tokens. ↵=newline →=tab ⇥N=indent]`}];for(let H of z.pages){let K=H.png;Y.push({type:"image",data:Buffer.from(K.buffer,K.byteOffset,K.byteLength).toString("base64"),mimeType:"image/png"})}return Y}function r$(){let Q={type:"string"},Z={type:"integer",minimum:0,maximum:4};return{tools:[{name:"tanuki_render",description:"Token-cut pipeline: optional columnar table (whole-JSON input: keys stated once in a ·cols· header, rows as tab-separated JSON cells — value-lossless), optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow (single-cell tabs, ⇥N indent runs, width-trimmed pages). font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",inputSchema:{type:"object",properties:{text:Q,level:Z,distill:{type:"boolean"},query:{type:"string"},reflow:{type:"boolean"},pack:{type:"boolean"},font:{type:"string",enum:["normal","tiny"]},codebook:{type:"boolean"},table:{type:"boolean"}},required:["text"]}},{name:"tanuki_estimate",description:"Estimate tokens for the pipeline (table -> distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field prices the reversible knobs (pack/codebook, and table for whole-JSON input — keys stated once, value-lossless) and, separately under 'withDistill', the lossy-but-counted log route. Pass 'model' (e.g. claude-opus-4, gpt-5, gemini-2.5) and/or cached:true to add a 'cost' field that prices the decision in real dollars with provider-correct image counting (Anthropic 28px patches, OpenAI 512px tiles, Gemini 768px tiles) and cache-read rates (a cached text token costs ~0.1x a fresh one on Anthropic), so imaging already-cached content usually loses even when it has fewer tokens. One call replaces manual knob probing.",inputSchema:{type:"object",properties:{text:Q,level:Z,distill:{type:"boolean"},query:{type:"string"},reflow:{type:"boolean"},pack:{type:"boolean"},font:{type:"string",enum:["normal","tiny"]},codebook:{type:"boolean"},table:{type:"boolean"},model:{type:"string"},cached:{type:"boolean"}},required:["text"]}},{name:"tanuki_distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes (exact + same-template) with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. table:true first columnar-encodes whole-JSON input (keys stated once) so identical rows collapse harder. Deterministic, order-preserving.",inputSchema:{type:"object",properties:{text:Q,query:{type:"string"},table:{type:"boolean"}},required:["text"]}},{name:"tanuki_compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",inputSchema:{type:"object",properties:{text:Q,level:Z},required:["text"]}},{name:"tanuki_stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates), and the output-token share of the bill — the part no input-side tool can cut.",inputSchema:{type:"object",properties:{}}},{name:"tanuki_stash",description:"Park bulky text outside the context window (content-addressed file under TANUKI_STASH or ~/.tanuki/stash) and get back a compact map: distill stats, top repeats, first/last lines, and the stash id. Pay a few hundred tokens now, fetch slices later - the retrieval pattern, with tanuki pricing on the way back.",inputSchema:{type:"object",properties:{text:Q},required:["text"]}},{name:"tanuki_fetch",description:"Pull a slice of stashed text by id: query (regex, distill-powered: matches + error/warn lines + context) or lines 'a-b'. Big slices come back as dense PNG pages automatically when they clearly win (>=25% and >=300 tokens cheaper, <=6 pages); small ones stay text.",inputSchema:{type:"object",properties:{id:{type:"string"},query:{type:"string"},lines:{type:"string"}},required:["id"]}}]}}function a$(Q){let Z=E(N(Q,"name"))??"",$=N(Q,"arguments"),J;switch(Z){case"tanuki_render":J=l$($);break;case"tanuki_estimate":J=[{type:"text",text:_(IZ($),!0)}];break;case"tanuki_distill":J=n$($);break;case"tanuki_compress":J=c$($);break;case"tanuki_stats":J=[{type:"text",text:_(KZ(),!0)}];break;case"tanuki_stash":J=o$($);break;case"tanuki_fetch":try{J=i$($)}catch(z){return{ok:!1,error:z instanceof Error?z.message:String(z)}}break;default:return{ok:!1,error:`unknown tool: ${Z}`}}return{ok:!0,value:{content:J}}}function MZ(Q){let Z=Q.endsWith("\r")?Q.slice(0,-1):Q;if(Z.trim().length===0)return;let $;try{$=JSON.parse(Z)}catch{return}let J=$!==null&&typeof $==="object"&&!Array.isArray($)&&Object.prototype.hasOwnProperty.call($,"id"),z=J?$.id??null:null,V=E(N($,"method")),G;switch(V){case"initialize":{let Y=E(N(N($,"params"),"protocolVersion"))??"2025-06-18";G={jsonrpc:"2.0",id:z,result:{protocolVersion:Y,capabilities:{tools:{}},serverInfo:{name:"tanuki-context",version:g$}}};break}case"notifications/initialized":case"notifications/cancelled":G=void 0;break;case"ping":G={jsonrpc:"2.0",id:z,result:{}};break;case"tools/list":G={jsonrpc:"2.0",id:z,result:r$()};break;case"tools/call":{let Y=a$(N($,"params"));G=Y.ok?{jsonrpc:"2.0",id:z,result:Y.value}:{jsonrpc:"2.0",id:z,error:{code:-32602,message:Y.error}};break}default:G=J?{jsonrpc:"2.0",id:z,error:{code:-32601,message:"method not found"}}:void 0}if(G!==void 0)P.stdout.write(_(G,!1)+`
|
|
129
|
+
`)}function s$(){let Q=Buffer.alloc(0);P.stdin.on("data",(Z)=>{let $=Q.length>0?Buffer.concat([Q,Z]):Z,J=0,z;while((z=$.indexOf(10,J))!==-1)MZ($.toString("utf8",J,z)),J=z+1;Q=$.subarray(J)}),P.stdin.on("end",()=>{if(Q.length>0)MZ(Q.toString("utf8")),Q=Buffer.alloc(0)})}function A(Q){P.stderr.write(Q+`
|
|
130
|
+
`),P.exit(101)}function VQ(Q){try{return u$(Q,"utf8")}catch{A("read file")}}function MQ(Q,Z){if(!/^\+?[0-9]+$/.test(Q))return null;let $=BigInt(Q.startsWith("+")?Q.slice(1):Q);return $>Z?null:Number($)}var WZ=255n,FZ=0xffffffffffffffffn;function RZ(){let Q=P.argv.slice(1);switch(Q[1]){case"distill":{let Z=Q[2]??A("usage: tanuki-context distill <file> [query] [--table]"),J=VQ(Z);if(Q.includes("--table")){let G=d(J);if(G!==null)J=G.text}let z=Q.slice(3).filter((G)=>!G.startsWith("--")),V=w(J,z[0]??null,2);P.stdout.write(_(V.stats,!1)+`
|
|
131
|
+
`);break}case"estimate":{let Z=Q[2]??A("usage: tanuki-context estimate <file> [level] [--distill] [--table] [--no-pack] [--font tiny] [--codebook] [--model <id>] [--cached]"),$=VQ(Z),J=Q.slice(3).filter((O)=>!O.startsWith("--")),z=J.length>0?MQ(J[0],FZ)??0:0,V=Q.indexOf("--font"),G=V!==-1&&Q[V+1]!==void 0?Q[V+1]:"normal",Y=Q.indexOf("--model"),H=Y!==-1&&Q[Y+1]!==void 0?Q[Y+1]:null,K=IZ({text:$,level:z,distill:Q.includes("--distill"),pack:!Q.includes("--no-pack"),font:G,codebook:Q.includes("--codebook"),table:Q.includes("--table"),model:H,cached:Q.includes("--cached")});P.stdout.write(_(K,!1)+`
|
|
132
|
+
`);break}case"render":{let Z=Q[2]??A("usage: tanuki-context render <file> [level] [outdir] [--distill] [--table] [--no-pack] [--font tiny] [--codebook]"),$=VQ(Z),J=Q.slice(3).filter((D)=>!D.startsWith("--")),z=J.length>0?MQ(J[0],WZ)??0:0,V=!Q.includes("--no-pack"),G=Q.includes("--codebook"),Y=Q.indexOf("--font"),H=f(Y!==-1&&Q[Y+1]!==void 0?Q[Y+1]:"normal"),K=WQ($,z,Q.includes("--distill"),null,G,Q.includes("--table")),O=m(K.compressed,!0,V,H),B=O.tokens;P.stdout.write(_({pages:O.pages.length,imageTokens:B,dropped:O.dropped,rawTextTokens:c(S($))},!1)+`
|
|
133
|
+
`);let X=J[1];if(X!==void 0){try{UZ(X,{recursive:!0})}catch{A("mkdir")}for(let D=0;D<O.pages.length;D++)try{qZ(`${X}/page${D}.png`,O.pages[D].png)}catch{A("write png")}}break}case"bench":{let Z=Q[2]??A("usage: tanuki-context bench <file> <op> [level] [runs] [--distill]"),$=Q[3]??"pipeline",J=Q[4]!==void 0?MQ(Q[4],WZ)??0:0,z=Q[5]!==void 0?MQ(Q[5],FZ)??3:3,V=Q.includes("--distill"),G=VQ(Z),Y=[],H=null;for(let K=0;K<=z;K++){let O=performance.now();if($==="distill")H=w(G,null,2).stats;else{let B=WQ(G,J,V,null,!1,!1),X=m(B.compressed,!0,!1,"normal");H={pages:X.pages.length,imageTokens:X.tokens,stage1Chars:S(B.compressed),dropped:X.dropped}}if(K>0)Y.push(performance.now()-O)}if(Y.sort((K,O)=>K-O),Y.length===0)A("index out of bounds: the len is 0 but the index is 0");P.stdout.write(_({medianMs:new r(Y[Math.floor(Y.length/2)]),runs:z,result:H},!1)+`
|
|
134
|
+
`);break}case"proxy":{let Z=(z,V)=>{let G=Q.indexOf(z);if(G===-1||Q[G+1]===void 0)return V;let Y=Number(Q[G+1]);return Number.isFinite(Y)?Y:V},$=Q.indexOf("--upstream"),J=Q.indexOf("--font");OZ({port:Z("--port",8484),upstream:$!==-1&&Q[$+1]!==void 0?Q[$+1]:P.env.TANUKI_UPSTREAM??"https://api.anthropic.com",level:Z("--level",a.level),distill:Q.includes("--distill"),table:Q.includes("--table"),codebook:Q.includes("--codebook"),font:f(J!==-1&&Q[J+1]!==void 0?Q[J+1]:"normal"),minChars:Z("--min-chars",a.minChars),ratio:Z("--ratio",a.ratio),minSave:Z("--min-save",a.minSave),maxPages:Z("--max-pages",a.maxPages)});break}case"stash":{let Z=Q[2]??A("usage: tanuki-context stash <file>"),$=GQ(VQ(Z));P.stdout.write($.overview+`
|
|
135
|
+
`);break}case"fetch":{let Z=Q[2]??A("usage: tanuki-context fetch <id> [outdir] [--query re] [--lines a-b]"),$=Q.indexOf("--query"),J=Q.indexOf("--lines"),z="";try{z=vQ(Z,$!==-1?Q[$+1]??null:null,J!==-1?Q[J+1]??null:null)}catch(O){A(O instanceof Error?O.message:String(O))}let V=c(S(z)),G=m(z,!0,!0,f("normal"));if(!(G.tokens<=V*0.75&&V-G.tokens>=300&&G.pages.length<=6)){P.stdout.write(_({mode:"text"},!1)+`
|
|
136
|
+
`+z+`
|
|
137
|
+
`);break}P.stdout.write(_({imageTokens:G.tokens,mode:"pages",pages:G.pages.length,rawTextTokens:V},!1)+`
|
|
138
|
+
`);let H=new Set([$+1,J+1]),K;for(let O=3;O<Q.length;O++)if(!Q[O].startsWith("--")&&!H.has(O)){K=Q[O];break}if(K!==void 0){try{UZ(K,{recursive:!0})}catch{A("mkdir")}for(let O=0;O<G.pages.length;O++)try{qZ(`${K}/page${O}.png`,G.pages[O].png)}catch{A("write png")}}break}case"run":{let Z=Q.indexOf("--"),$=Z!==-1?Q.slice(Z+1):[];if($.length===0)A("usage: tanuki-context run [--query re] -- <command> [args...]");let J=Q.indexOf("--query"),z=J!==-1&&J<Z?Q[J+1]??null:null,V=m$($[0],$.slice(1),{encoding:"utf8",maxBuffer:268435456});if(V.error!==void 0)A(`spawn failed: ${V.error.message}`);let G=(V.stdout??"")+((V.stderr??"")!==""?`
|
|
106
139
|
--- stderr ---
|
|
107
|
-
${
|
|
140
|
+
${V.stderr}`:""),Y=V.status??0,H=w(G,z,2),K=H.stats,O=[`[tanuki run] exit ${Y} · ${K.origLines} -> ${K.outLines} lines · ${K.savedPct}% of chars removed`];if(S(H.distilled)<=kQ||S(G)<=kQ){if(O.push(H.distilled),S(G)>kQ){let B=GQ(G);O.push(`full output stashed: tanuki-context fetch ${B.id} [--query re] [--lines a-b]`)}}else{let B=GQ(G);O.push(B.overview)}P.stdout.write(O.join(`
|
|
108
141
|
`)+`
|
|
109
|
-
`),
|
|
142
|
+
`),P.exit(Y)}case"serve":case void 0:s$();break;default:P.stderr.write(`unknown command: ${Q[1]}
|
|
110
143
|
usage: tanuki-context [serve|proxy|distill|estimate|render|bench|stash|fetch|run] ...
|
|
111
|
-
`),
|
|
144
|
+
`),P.exit(1)}}//! Bin entry: keeps src/main.ts importable as a library (src/agent.ts) without
|
|
112
145
|
//! starting the MCP server as an import side effect.
|
|
113
|
-
|
|
146
|
+
RZ();
|
package/dist/pi.js
CHANGED
|
@@ -11,5 +11,5 @@ import{spawn as W}from"node:child_process";import{fileURLToPath as X}from"node:u
|
|
|
11
11
|
//! ToolResult content shape, so results pass through untouched.
|
|
12
12
|
function Y(){let F=process.env.TANUKI_BIN;if(F)return{cmd:F,args:[]};return{cmd:process.execPath,args:[X(new URL("./cli.js",import.meta.url))]}}class Q{proc;buf="";nextId=1;pending=new Map;ready;constructor(){let{cmd:F,args:B}=Y();this.proc=W(F,B,{stdio:["pipe","pipe","inherit"]}),this.proc.stdout.setEncoding("utf8"),this.proc.stdout.on("data",(G)=>{this.buf+=G;let z;while((z=this.buf.indexOf(`
|
|
13
13
|
`))!==-1){let H=this.buf.slice(0,z).trim();if(this.buf=this.buf.slice(z+1),!H)continue;let D;try{D=JSON.parse(H)}catch{continue}let J=D.id!==void 0?this.pending.get(D.id):void 0;if(!J)continue;if(this.pending.delete(D.id),D.error)J.reject(Error(D.error.message??"tanuki-context error"));else J.resolve(D.result)}}),this.proc.on("exit",(G)=>{let z=Error(`tanuki-context server exited (code ${G})`);for(let H of this.pending.values())H.reject(z);this.pending.clear()}),this.ready=this.request("initialize",{})}request(F,B){let G=this.nextId++,{promise:z,resolve:H,reject:D}=Promise.withResolvers();if(!this.proc.stdin?.writable)return D(Error("tanuki-context server is gone")),z;return this.pending.set(G,{resolve:H,reject:D}),this.proc.stdin.write(JSON.stringify({jsonrpc:"2.0",id:G,method:F,params:B})+`
|
|
14
|
-
`),z}async call(F,B){return await this.ready,this.request("tools/call",{name:F,arguments:B})}kill(){this.proc.kill()}get alive(){return this.proc.exitCode===null&&!this.proc.killed}}var K=q.String({description:"The text to process"}),V=q.Optional(q.Integer({minimum:0,maximum:4,description:"0 raw · 1 whitespace · 2 prose · 3 dense · 4 caveman"})),N=q.Object({text:K,level:V,distill:q.Optional(q.Boolean()),query:q.Optional(q.String()),reflow:q.Optional(q.Boolean()),pack:q.Optional(q.Boolean()),font:q.Optional(q.String({enum:["normal","tiny"]})),codebook:q.Optional(q.Boolean())}),Z=[{name:"tanuki_render",label:"Tanuki Render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow. font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",parameters:N,snippet:"Render bulky text/logs as dense PNG pages that cost a fraction of the tokens"},{name:"tanuki_estimate",label:"Tanuki Estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field prices the reversible knobs (pack/codebook,
|
|
14
|
+
`),z}async call(F,B){return await this.ready,this.request("tools/call",{name:F,arguments:B})}kill(){this.proc.kill()}get alive(){return this.proc.exitCode===null&&!this.proc.killed}}var K=q.String({description:"The text to process"}),V=q.Optional(q.Integer({minimum:0,maximum:4,description:"0 raw · 1 whitespace · 2 prose · 3 dense · 4 caveman"})),N=q.Object({text:K,level:V,distill:q.Optional(q.Boolean()),query:q.Optional(q.String()),reflow:q.Optional(q.Boolean()),pack:q.Optional(q.Boolean()),font:q.Optional(q.String({enum:["normal","tiny"]})),codebook:q.Optional(q.Boolean()),table:q.Optional(q.Boolean({description:"columnar-encode whole-JSON input (keys stated once, value-lossless)"})),model:q.Optional(q.String({description:"price the decision for this model (opus/sonnet/haiku/gpt/gemini)"})),cached:q.Optional(q.Boolean({description:"text already prompt-cached this turn? imaging it usually loses"}))}),Z=[{name:"tanuki_render",label:"Tanuki Render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow. font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",parameters:N,snippet:"Render bulky text/logs as dense PNG pages that cost a fraction of the tokens"},{name:"tanuki_estimate",label:"Tanuki Estimate",description:"Estimate tokens for the pipeline (table -> distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field prices the reversible knobs (pack/codebook, and table for whole-JSON input) and, separately under 'withDistill', the lossy-but-counted log route. Pass 'model' and/or cached:true for a 'cost' field in real dollars with provider-correct image counting (Anthropic 28px patches, OpenAI 512px tiles, Gemini 768px tiles) - a cached text token costs ~0.1x a fresh one on Anthropic, so imaging already-cached content usually loses. One call replaces manual knob probing.",parameters:N,snippet:"Instant token verdict: would imaging this text beat sending it as text?"},{name:"tanuki_distill",label:"Tanuki Distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. Deterministic, order-preserving.",parameters:q.Object({text:K,query:q.Optional(q.String())}),snippet:"Deterministically dedupe noisy logs, keeping every error line verbatim"},{name:"tanuki_compress",label:"Tanuki Compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",parameters:q.Object({text:K,level:V}),snippet:"Graded text compression (lossless whitespace up to gist-only)"},{name:"tanuki_stats",label:"Tanuki Stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates).",parameters:q.Object({}),snippet:"Session savings summary from the tanuki/pxpipe event log"},{name:"tanuki_stash",label:"Tanuki Stash",description:"Park bulky text outside the context window (content-addressed file under TANUKI_STASH or ~/.tanuki/stash) and get back a compact map: distill stats, top repeats, first/last lines, and the stash id. Pay a few hundred tokens now, fetch slices later - the retrieval pattern, with tanuki pricing on the way back.",parameters:q.Object({text:K}),snippet:"Park huge text on disk for a few hundred tokens of map; fetch slices later"},{name:"tanuki_fetch",label:"Tanuki Fetch",description:"Pull a slice of stashed text by id: query (regex, distill-powered: matches + error/warn lines + context) or lines 'a-b'. Big slices come back as dense PNG pages automatically when they clearly win (>=25% and >=300 tokens cheaper, <=6 pages); small ones stay text.",parameters:q.Object({id:q.String(),query:q.Optional(q.String()),lines:q.Optional(q.String())}),snippet:"Pull a stashed slice; big answers arrive as cheap dense pages"}];function $(F){let B=null,G=()=>{if(!B||!B.alive)B=new Q;return B};F.on("session_shutdown",async()=>{B?.kill(),B=null});for(let z of Z)F.registerTool({name:z.name,label:z.label,description:z.description,promptSnippet:z.snippet,parameters:z.parameters,async execute(H,D){let J=await G().call(z.name,D??{}),M=(J.content??[]).map((I)=>I.type==="image"?{type:"image",data:I.data??"",mimeType:I.mimeType??"image/png"}:{type:"text",text:I.text??""});if(J.isError)throw Error(M.map((I)=>("text"in I)?I.text:"").join(`
|
|
15
15
|
`)||z.name+" failed");return{content:M,details:{}}}})}export{$ as default};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tanuki-context",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.6.0",
|
|
4
4
|
"description": "Token-cutting context pipeline as an MCP server: distill logs, compress text, render dense PNG pages (pxpipe imaging engine). Zero dependencies.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: tanuki-context
|
|
3
|
-
version: 0.
|
|
3
|
+
version: 0.6.0
|
|
4
4
|
description: |
|
|
5
5
|
Cut input-token cost by rendering bulky text (logs, command output, long
|
|
6
6
|
docs) as dense PNG pages the model reads at a fraction of the price, or by
|
|
@@ -22,7 +22,8 @@ verbatim, and every drop is counted.
|
|
|
22
22
|
|
|
23
23
|
1. **Estimate first, always.** `tanuki_estimate { text }` is instant and
|
|
24
24
|
never renders pixels. Read two things from the answer:
|
|
25
|
-
- `recommend` - the cheapest reversible knob set (pack/codebook
|
|
25
|
+
- `recommend` - the cheapest reversible knob set (pack/codebook, and
|
|
26
|
+
`table` for whole-JSON input), priced.
|
|
26
27
|
- `recommend.withDistill` - the log route (repeats collapsed with exact
|
|
27
28
|
counts). Cheaper, and honest for logs; do not use it on source code.
|
|
28
29
|
2. **Act on the verdict.**
|
|
@@ -34,12 +35,29 @@ verbatim, and every drop is counted.
|
|
|
34
35
|
`{ id, lines: "a-b" }`. Big slices arrive as pages automatically.
|
|
35
36
|
- `"TEXT cheaper"` -> just use the text. Small inputs are not worth an
|
|
36
37
|
image even when the math technically favors one.
|
|
38
|
+
- **Already cached?** If the text is already in the prompt cache (a file
|
|
39
|
+
the harness re-sends every turn), pass `{ model: "<your model>",
|
|
40
|
+
cached: true }`. `tanuki_estimate` adds a `cost` block in real dollars:
|
|
41
|
+
a cache-read token costs ~0.1x a fresh one, so imaging cached content
|
|
42
|
+
usually *loses* even with fewer tokens - `cost.cheaper` says "TEXT" and
|
|
43
|
+
you leave it alone. Pass `model` alone to price a one-shot decision;
|
|
44
|
+
image tokens are counted with that provider's own rule (OpenAI 512px
|
|
45
|
+
tiles, Gemini 768px tiles - Gemini flagged approximate), so the verdict
|
|
46
|
+
is not an Anthropic-only guess.
|
|
37
47
|
3. **For noisy logs**, add `distill: true` (and `query: "regex"` for a
|
|
38
48
|
slice). Error/warn/fail lines survive verbatim; repeats become one
|
|
39
49
|
exemplar plus an exact xN count.
|
|
40
|
-
4. **
|
|
50
|
+
4. **For whole-JSON input** (arrays of objects, NDJSON like
|
|
51
|
+
`journalctl -o json`), pass `table: true` - keys are stated once in a
|
|
52
|
+
`·cols·` header and rows become tab-separated JSON cells. Value-lossless,
|
|
53
|
+
and uniform rows then collapse harder under distill. `recommend` probes
|
|
54
|
+
this on its own, so trusting it is enough.
|
|
55
|
+
5. **Shell commands with chatty output**: run them as
|
|
41
56
|
`tanuki-context run -- <cmd>` instead of reading the firehose. Exit code
|
|
42
57
|
passes through; the full capture is stashed and fetchable.
|
|
58
|
+
6. **End of session**: `tanuki_stats` totals the savings log and reports
|
|
59
|
+
`outputSharePct` - the share of the bill that is the model's own output,
|
|
60
|
+
which no input-side tool (including this one) can cut.
|
|
43
61
|
|
|
44
62
|
## Reading the pages
|
|
45
63
|
|