tanuki-context 0.2.3 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +61 -14
- package/dist/agent.d.ts +2 -0
- package/dist/agent.js +42 -28
- package/dist/cli.js +56 -39
- package/dist/pi.js +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -115,7 +115,7 @@ Both modes run the same engine and log their savings to the same file, which
|
|
|
115
115
|
Agent SDK glue (below), with a canned instruction block that teaches every
|
|
116
116
|
agent the estimate-first habit.
|
|
117
117
|
|
|
118
|
-
## The
|
|
118
|
+
## The seven tools
|
|
119
119
|
|
|
120
120
|
| tool | what it does | when to reach for it |
|
|
121
121
|
| ----------------- | ----------------------------------------------------------- | ---------------------------------------- |
|
|
@@ -123,8 +123,49 @@ Both modes run the same engine and log their savings to the same file, which
|
|
|
123
123
|
| `tanuki_render` | the full pipeline, returns PNG pages + a breakdown | when estimate says the pipeline wins |
|
|
124
124
|
| `tanuki_distill` | de-noises a log, output stays text | when you want greppable text, not images |
|
|
125
125
|
| `tanuki_compress` | text-only compression, five levels | prose you'll paraphrase anyway |
|
|
126
|
+
| `tanuki_stash` | parks text on disk, returns a ~300-token map + an id | huge references you'll consult, not read |
|
|
127
|
+
| `tanuki_fetch` | pulls a slice by regex or line range; auto-imaged when big | after a stash, when you actually need it |
|
|
126
128
|
| `tanuki_stats` | totals from the session's savings log | end-of-session accounting |
|
|
127
129
|
|
|
130
|
+
### Stash: the retrieval pattern, absorbed
|
|
131
|
+
|
|
132
|
+
Sometimes even 5,000 tokens of pages is too much — the model may only need
|
|
133
|
+
two slices of that log, ever. That is retrieval territory (context-mode's
|
|
134
|
+
home game), and since 0.3.0 tanuki plays it too, with two twists.
|
|
135
|
+
|
|
136
|
+
`tanuki_stash` writes the text to a content-addressed file (nothing enters
|
|
137
|
+
context) and returns a map. The real one, for the 200 KB journal — 305
|
|
138
|
+
tokens:
|
|
139
|
+
|
|
140
|
+
```
|
|
141
|
+
stashed a57d29474429 · 204800 bytes · 1434 lines
|
|
142
|
+
distill map: 1434 -> 641 lines · 51% of chars removable · 310 error/warn lines
|
|
143
|
+
top repeats:
|
|
144
|
+
×156 2026-07-26T11:10:53+02:00 demo-box tailscaled[740]: magicsock: derp-18 does not know about peer [xfiv4], removing route
|
|
145
|
+
×99 2026-07-26T11:27:22+02:00 demo-box tailscaled[740]: open-conn-track: flow TCP ... got RST by peer
|
|
146
|
+
...
|
|
147
|
+
first: 2026-07-26T11:10:32+02:00 demo-box tailscaled[740]: [RATELIMIT] ...
|
|
148
|
+
last: 2026-07-26T16:19:40+02:00 demo-box wpa_supplicant[844]: wlan0: CTRL-EVENT-REGDOM ...
|
|
149
|
+
fetch: tanuki_fetch {"id":"a57d29474429","query":"<regex>"} or {"id":"a57d29474429","lines":"a-b"}
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Twist one: that map is *awareness*. A pure retrieval store tells the model
|
|
153
|
+
nothing — it has to fish with blind queries. This tells it the time range,
|
|
154
|
+
the noisiest units, the error count, and what the repetition looks like,
|
|
155
|
+
for the price of one paragraph.
|
|
156
|
+
|
|
157
|
+
Twist two: fetches come back at tanuki pricing. Measured on that stash:
|
|
158
|
+
|
|
159
|
+
| fetch | as text | via tanuki_fetch |
|
|
160
|
+
| ------------------------------ | ------: | ----------------------- |
|
|
161
|
+
| every error/reset line (query) | 22,111 | **4,704** (4 PNG pages) |
|
|
162
|
+
| lines 1-12 | 446 | **112** (1 page) |
|
|
163
|
+
| lines 1-2 | ~20 | ~20 (stays text) |
|
|
164
|
+
|
|
165
|
+
The gate is the proxy's: pages only when they win by >=25% and >=300
|
|
166
|
+
tokens, six pages max, text otherwise. Same engine, same honesty — the
|
|
167
|
+
slice arrives verbatim or drawn, never paraphrased.
|
|
168
|
+
|
|
128
169
|
---
|
|
129
170
|
|
|
130
171
|
Everything below is for readers who want the machinery.
|
|
@@ -196,6 +237,7 @@ escapes. Nothing disappears silently.
|
|
|
196
237
|
| `recommend` | the estimate call walks all safe knob combos server-side | saves ~590 tokens of tool-call probing |
|
|
197
238
|
| proxy dedupe | byte-identical repeats become a one-line pointer | ~1,400 tokens per repeated 30 KB block |
|
|
198
239
|
| append-stable pages | appending text never changes earlier pages | prompt caching keeps pricing them at cache rates |
|
|
240
|
+
| stash + fetch | park text on disk, return a distill map + content-address id | retrieval economics with awareness and imaged slices |
|
|
199
241
|
|
|
200
242
|
## Benchmarks
|
|
201
243
|
|
|
@@ -224,17 +266,17 @@ the reference server is the original node MCP wrapping pxpipe's library):
|
|
|
224
266
|
| distill a 12 MB log | 0.42 s | 0.31 s | 0.28 s | - |
|
|
225
267
|
| install | 0.98 MB tarball, zero deps | same | 5.7 MB static binary | node_modules tree |
|
|
226
268
|
|
|
227
|
-
Correctness:
|
|
269
|
+
Correctness: 46 TypeScript tests, 32 Rust tests, and a 103-check parity
|
|
228
270
|
harness that holds the two engines to byte-identical JSON and
|
|
229
271
|
pixel-identical PNGs on every knob combination, including distilled
|
|
230
272
|
renders and a full MCP session with error paths.
|
|
231
273
|
|
|
232
274
|
## How it compares
|
|
233
275
|
|
|
234
|
-
|
|
235
|
-
(51,200 tokens raw) on the same machine. They are not
|
|
236
|
-
|
|
237
|
-
|
|
276
|
+
Six approaches to the same problem, measured on the same 200 KB journal
|
|
277
|
+
(51,200 tokens raw) on the same machine. They are not competitors — they
|
|
278
|
+
are different answers to "does the model need to see this at all?", and
|
|
279
|
+
several of them compose.
|
|
238
280
|
|
|
239
281
|
| approach | what enters context | tokens | what the model can still do |
|
|
240
282
|
| --- | --- | ---: | --- |
|
|
@@ -244,16 +286,19 @@ and several of them compose.
|
|
|
244
286
|
| [pxpipe](https://github.com/teamchong/pxpipe) (its own `export` CLI on this corpus) | 8 PNG pages + prompt + factsheet | ~11,965 (-77%) | read everything back; 96 precision-critical strings ride verbatim in the factsheet |
|
|
245
287
|
| tanuki, distill + render | 4 PNG pages | 5,264 (-90%) | read everything that survives distill: repeats collapsed with exact counts, all 310 error/warn lines verbatim |
|
|
246
288
|
| tanuki, + codebook + tiny font | 2 PNG pages | 2,576 (-95%) | same content, lossy glyphs (99.7% read-back) — opt-in |
|
|
289
|
+
| tanuki, stash + fetch (0.3.0) | a 305-token map now, slices on demand | 305 + 112..4,704 per slice | knows the log's shape immediately; opens only the slices it needs, imaged when big, text when small |
|
|
247
290
|
| [context-mode](https://www.npmjs.com/package/context-mode) (sandbox pass over the file) | one analysis result | ~270 per question (-99.5%) | only the answer it asked for; the file never enters context at all |
|
|
248
291
|
|
|
249
292
|
Reading that honestly:
|
|
250
293
|
|
|
251
|
-
- **context-mode wins whenever a question suffices.** If the model
|
|
252
|
-
needs to *see* the log, ~270 tokens per query is unbeatable.
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
294
|
+
- **context-mode wins whenever a narrow question suffices.** If the model
|
|
295
|
+
never needs to *see* the log, ~270 tokens per query is unbeatable. Its
|
|
296
|
+
blind spots are awareness (the store tells the model nothing until it
|
|
297
|
+
guesses the right question) and big answers (a large slice comes back at
|
|
298
|
+
full text price). `tanuki_stash`/`tanuki_fetch` is that pattern with both
|
|
299
|
+
spots covered: a 305-token map up front, and slices that arrive as pages
|
|
300
|
+
whenever pages win. For the "show me every failure line" job measured
|
|
301
|
+
above: 305 + 4,704 = 5,009 tokens vs 22,111 for the same lines as text.
|
|
257
302
|
- **pxpipe and tanuki are the same engine**, so the gap between -77% and
|
|
258
303
|
-90% here is not the imaging — it is distill (pxpipe does not de-noise
|
|
259
304
|
logs) plus the sidecar text its export flow pastes alongside. On prose,
|
|
@@ -411,6 +456,8 @@ npx tanuki-context distill <file> [query] # stats JSON to stdout
|
|
|
411
456
|
npx tanuki-context estimate <file> [level] [--distill] [--no-pack] [--font tiny] [--codebook]
|
|
412
457
|
npx tanuki-context render <file> [level] [outdir] [--distill] [--no-pack] [--font tiny] [--codebook]
|
|
413
458
|
npx tanuki-context bench <file> <distill|pipeline> [level] [runs] # in-process timing
|
|
459
|
+
npx tanuki-context stash <file> # park text, print the map + id
|
|
460
|
+
npx tanuki-context fetch <id> [outdir] [--query re] [--lines a-b]
|
|
414
461
|
```
|
|
415
462
|
|
|
416
463
|
The example page above is one command:
|
|
@@ -465,8 +512,8 @@ Node-compatible files:
|
|
|
465
512
|
|
|
466
513
|
```
|
|
467
514
|
bun run build # dist/cli.js + dist/agent.js + dist/pi.js
|
|
468
|
-
bun test #
|
|
469
|
-
bun run parity # TS vs rust binary,
|
|
515
|
+
bun test # 46 tests
|
|
516
|
+
bun run parity # TS vs rust binary, 103 checks (needs TANUKI_BIN)
|
|
470
517
|
```
|
|
471
518
|
|
|
472
519
|
Regenerating glyphs after a pxpipe atlas rebuild (needs a pxpipe checkout
|
package/dist/agent.d.ts
CHANGED
package/dist/agent.js
CHANGED
|
@@ -1,19 +1,19 @@
|
|
|
1
|
-
import{createRequire as
|
|
2
|
-
·legend· `;for(let B=0;B<H.length;B++)X+=H[B].sig+"="+H[B].val+" ";return
|
|
3
|
-
`),Y=G.length,H=Y,z=Array(Y),K=Array(Y),
|
|
4
|
-
`),
|
|
5
|
-
`),G=Array(V.length);for(let H=0;H<V.length;H++){let z=V[H],K=z.length;while(K>0){let X=z.charCodeAt(K-1);if(X!==32&&X!==9)break;K--}let
|
|
6
|
-
`).replace(
|
|
1
|
+
import{createRequire as KQ}from"node:module";var EJ=KQ(import.meta.url);import{existsSync as UZ}from"node:fs";import{fileURLToPath as OZ}from"node:url";function XQ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function KJ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}Q++}return Q}function BQ(J,Q){let Z=0,$=0;while(Z<J.length&&$<Q.length){let V=J.codePointAt(Z),G=Q.codePointAt($);if(V!==G)return V-G;Z+=V>65535?2:1,$+=G>65535?2:1}return J.length-Z-(Q.length-$)}function t(J){let Q=new Map,Z=(B)=>{Q.set(B,(Q.get(B)??0)+1)},$=(B)=>{if(KJ(B)>=12)Z(B);if(B.includes("/")){let D=B.split("/"),q="";for(let M=0;M<D.length;M++){if(M>0)q+="/";if(q+=D[M],M>=2){let R=q+"/";if(KJ(R)>=12)Z(R)}}}},V=-1;for(let B=0;B<J.length;){let D=J.codePointAt(B),q=D>65535?2:1;if(XQ(D)){if(V>=0)$(J.slice(V,B)),V=-1}else if(V<0)V=B;B+=q}if(V>=0)$(J.slice(V));let G=[];for(let[B,D]of Q)if(D>=3){let q=KJ(B);G.push({k:B,c:D,len:q,saved:(q-1)*D})}G.sort((B,D)=>D.saved-B.saved||BQ(B.k,D.k));let Y=[];for(let B of"§¤¢£¥µ¶ª°±¬×÷ØÞßæðøþ¡¿")if(!J.includes(B))Y.push(B);let H=[],z=[];for(let{k:B,c:D,len:q}of G){if(H.length>=Y.length)break;if((q-1)*D<=q+3)continue;let M=!1;for(let P=0;P<z.length;P++){let T=z[P];if(T.startsWith(B)||B.startsWith(T)){M=!0;break}}if(M)continue;let R=Y[H.length];z.push(B),H.push({sig:R,val:B,len:q})}if(H.length===0)return{text:J,entries:0};let K=H.map((B,D)=>D);K.sort((B,D)=>H[D].len-H[B].len);let U=J;for(let B=0;B<K.length;B++){let{sig:D,val:q}=H[K[B]];U=U.replaceAll(q,D)}let X=`
|
|
2
|
+
·legend· `;for(let B=0;B<H.length;B++)X+=H[B].sig+"="+H[B].val+" ";return U+=X.slice(0,-1),{text:U,entries:H.length}}import{Buffer as CJ}from"node:buffer";var UQ=/\x1b\[[0-9;]*[A-Za-z]/g,XJ=/\b([0-9A-Za-z_]*(error|exception)s?|err|warn(ing)?s?|fail(s|ed|ure|ures)?|panic(s|ked)?|fatal|critical|traceback|denied|refused|timeouts?|timed.?out|assert(s|ed|ion|ions)?|segfault(s|ed)?)\b/i,OQ=/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/g,qQ=/\b[0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?\b/g,DQ=/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,MQ=/\b[0-9a-f]{7,64}\b/gi,FQ=/\b[0-9]+(\.[0-9]+)?[ \t\u00a0]?(ms|us|µs|ns|s|m|h|%|[KMGT]i?B)?\b/gi,WQ=/[0-9a-f]/i,NQ=3,jQ=8,BJ=2,SJ=40;function IQ(J){if(!WQ.test(J))return J;let Q=J.replace(OQ,"<ts>");return Q=Q.replace(qQ,"<time>"),Q=Q.replace(DQ,"<uuid>"),Q=Q.replace(MQ,"<hex>"),Q.replace(FQ,"<n>")}function m(J){if(J===32)return!0;if(J<9)return!1;if(J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function p(J){let Q=0,Z=J.length;while(Q<Z&&m(J.charCodeAt(Q)))Q++;while(Z>Q&&m(J.charCodeAt(Z-1)))Z--;return Q===0&&Z===J.length?J:J.slice(Q,Z)}function RQ(J,Q){let Z=0,$=J.length;while(Z<$&&m(J.charCodeAt(Z)))Z++;while($>Z&&m(J.charCodeAt($-1)))$--;let V=0;for(let G=Z;G<$&&V<Q;V++){let Y=J.charCodeAt(G);G+=Y>=55296&&Y<=56319&&G+1<$?2:1}return V<Q}function PQ(J){let Q=0,Z=J.length;while(Q<Z&&m(J.charCodeAt(Q)))Q++;return J.charCodeAt(Q)===91&&J.charCodeAt(Q+1)===215}function AQ(J){let Q=[],Z=-1,$=!0,V=!1,G=!1,Y=!0;for(let H=0;H<J.length;){let z=J.codePointAt(H),K=z>65535?2:1,U=m(z);if(Y){if(Y=!1,U)Q.push("<v>")}if(U){if(Z>=0)Q.push($?J.slice(Z,H):"<v>"),Z=-1;G=!0}else{if(V=!0,G=!1,Z<0)Z=H,$=!0;if($){let X=z|32;if(X<97||X>122||z>127)$=!1}}H+=K}if(Z>=0)Q.push($?J.slice(Z):"<v>");if(V&&G)Q.push("<v>");return Q.join(" ")}function n(J,Q){let Z=0,$=0,V=J.length;while(Z<V&&$<Q){let G=J.charCodeAt(Z);Z+=G>=55296&&G<=56319&&Z+1<V?2:1,$++}return Z>=V?J:J.slice(0,Z)}var _Q=/[.*+?^${}()|[\]\\]/g;function b(J,Q=null,Z=2){let $=Z,G=J.replace(UQ,"").split(`
|
|
3
|
+
`),Y=G.length,H=Y,z=Array(Y),K=Array(Y),U=0;for(let O=0;O<Y;O++){z[O]=IQ(G[O]);let F=XJ.test(G[O]);if(K[O]=F,F)U++}let X=[],B=[],D=[],q=0,M=0;while(M<Y){let O=0,F=0;for(let W=1;W<=jQ&&M+2*W<=Y;W++){if(K[M+W-1])break;let j=1;J:for(;;){let N=M+j*W;if(N+W>Y)break;for(let _=0;_<W;_++)if(K[N+_]||z[N+_]!==z[M+_])break J;j++}if(j>=NQ&&W*(j-1)>O*(F>0?F-1:0))O=W,F=j}if(O>0){for(let W=M;W<M+O;W++)X.push(G[W]),B.push(K[W]),D.push(z[W]);X.push(O===1?` [×${F} similar]`:` [×${F} similar ${O}-line blocks]`),B.push(!1),D.push(null),q++,M+=O*F}else X.push(G[M]),B.push(K[M]),D.push(z[M]),M++}let R=new Map,P=new Map,T=BJ+1,E=[],A=[],y=0,w=0;for(let O=0;O<X.length;O++){let F=X[O],W=B[O];if(W||PQ(F)||RQ(F,4)){E.push(F),A.push(W);continue}let j=D[O],N=R.get(j);if(N!==void 0){if(N.count++,N.count<=BJ)E.push(F),A.push(!1);else y++;continue}let _=AQ(j);R.set(j,{count:1,exemplar:F});let l=P.get(_);if(l!==void 0)if(l.count++,l.count<=T)E.push(F),A.push(!1);else w++;else P.set(_,{count:1,exemplar:F}),E.push(F),A.push(!1)}let k=[];for(let O of R.values())if(O.count>BJ)k.push([O.count,"exact",O.exemplar]);for(let O of P.values())if(O.count>T)k.push([O.count,"template",O.exemplar]);if(k.sort((O,F)=>F[0]-O[0]),k.length>SJ)k.length=SJ;if(y+w>0){let O=`── ${y+w} repeated lines suppressed (${y} exact ×N, ${w} same-template; first occurrences kept above) ──`;E.push(O),A.push(XJ.test(O));for(let[F,W,j]of k){let _=` ×${F}${W==="template"?" (template)":""} ${n(p(j),160)}`;E.push(_),A.push(XJ.test(_))}}let f;if(Q!=null){let O;try{O=new RegExp(Q,"i")}catch{O=new RegExp(Q.replace(_Q,"\\$&"),"i")}let F=E.length,W=new Uint8Array(F);for(let N=0;N<F;N++)if(A[N]||O.test(E[N])){let _=N>$?N-$:0,l=Math.min(N+$,F-1);W.fill(1,_,l+1)}f=[];let j=0;for(let N=0;N<F;N++)if(W[N]){if(j>0)f.push(`… ${j} lines omitted`),j=0;f.push(E[N])}else j++;if(j>0)f.push(`… ${j} lines omitted`)}else f=E;let PJ=f.join(`
|
|
4
|
+
`),AJ=CJ.byteLength(J),_J=CJ.byteLength(PJ),zJ=(1-_J/AJ)*100,YQ=J.length===0?0:zJ<0?-Math.round(-zJ):Math.round(zJ),HQ=k.map(([O,F,W])=>({count:O,exemplar:n(p(W),160),kind:F})),zQ={collapsedRuns:q,importantKept:U,origChars:AJ,origLines:H,outChars:_J,outLines:f.length,query:Q??null,savedPct:YQ,suppressedLines:y,templateSuppressed:w,topRepeats:HQ};return{distilled:PJ,stats:zQ}}var s=[["none","none","passthrough (baseline)"],["whitespace","lossless","trailing whitespace + blank-line runs collapsed; safe for code"],["prose","light","L1 + prose lines: collapse spaces, cut redundant filler phrases (code/IDs protected)"],["dense","medium","L2 + prose: drop articles & intensifiers"],["caveman","heavy","L3 + prose: telegraphic — drop function words; gist only, NOT verbatim"]];var C=(J)=>new RegExp(`(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:${J})(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])`,"giu"),UJ=[[C("in order to"),"to"],[C("due to the fact that"),"because"],[C("at this point in time"),"now"],[C("in the event that"),"if"],[C("for the purpose of"),"for"],[C("with regard to"),"about"],[C("a large number of"),"many"],[C("it is important to note that"),""],[C("please note that"),""],[C("as a matter of fact"),""],[C("in terms of"),"for"],[C("the fact that"),"that"]],EQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:the|an|a)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),CQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:very|really|just|actually|basically|simply|quite|rather|essentially|literally)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),SQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:is|are|was|were|am|be|been|being|do|does|did|have|has|had|will|would|shall|should|can|could|may|might|of|to|in|on|at|for|with|that|this|these|those|it|its|there|here)(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]*","giu"),LJ=/ {2,}/g,LQ=new RegExp("[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+([.,;:!?])","gu"),TQ=/\n{3,}/g;function OJ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function yQ(J){let Q=0,Z=J.length;while(Q<Z){let $=J.codePointAt(Q);if(!OJ($))break;Q+=$>65535?2:1}while(Z>Q){let $=J.charCodeAt(Z-1);if(!OJ($))break;Z--}return Q===0&&Z===J.length?J:J.slice(Q,Z)}function wQ(J){if(J.length===0)return!1;let Q=J.charCodeAt(0);if(Q===32||Q===9)return!0;let Z=0,$=0,V=0,G=!1;for(let Y=0;Y<J.length;){let H=J.codePointAt(Y);if(Y+=H>65535?2:1,Z++,OJ(H)){V=0;continue}if(V++,V>=24)G=!0;if(H>=48&&H<=57||H>=65&&H<=90||H>=97&&H<=122)continue;switch(H){case 46:case 44:case 59:case 58:case 39:case 34:case 33:case 63:case 40:case 41:case 45:continue}$++}if($/Z>0.3)return!0;return G}function bQ(J,Q){let Z=J.replace(LJ," ");for(let V=0;V<UJ.length;V++)Z=Z.replace(UJ[V][0],UJ[V][1]);if(Q>=3)Z=Z.replace(EQ,""),Z=Z.replace(CQ,"");if(Q>=4)Z=Z.replace(SQ,"");Z=Z.replace(LJ," "),Z=yQ(Z.replace(LQ,"$1"));let $=Z.charCodeAt(0);if($>=97&&$<=122)Z=String.fromCharCode($-32)+Z.slice(1);return Z}function e(J,Q){let Z=Math.min(Q,4);if(Z===0)return{compressed:J,protectedLines:0,level:Z};let $=0,V=J.split(`
|
|
5
|
+
`),G=Array(V.length);for(let H=0;H<V.length;H++){let z=V[H],K=z.length;while(K>0){let X=z.charCodeAt(K-1);if(X!==32&&X!==9)break;K--}let U=K===z.length?z:z.slice(0,K);if(Z===1){G[H]=U;continue}if(wQ(U)){$++,G[H]=U;continue}G[H]=bQ(U,Z)}return{compressed:G.join(`
|
|
6
|
+
`).replace(TQ,`
|
|
7
7
|
|
|
8
|
-
`),protectedLines:$,level:Z}}import{readFileSync as
|
|
8
|
+
`),protectedLines:$,level:Z}}import{readFileSync as DJ}from"node:fs";import{inflateSync as vQ}from"node:zlib";//! Full-BMP glyph atlas (Spleen 5x8 for ASCII/code + Unifont fallback),
|
|
9
9
|
//! extracted from pxpipe's generated gray atlas by `tools/gen-glyphs.mjs`.
|
|
10
10
|
//!
|
|
11
11
|
//! Codepoints + wide flags load eagerly (~175 KB, needed for wrap math);
|
|
12
12
|
//! coverage pixels stay zlib-packed on disk and decompress lazily on the
|
|
13
13
|
//! first blit, so `estimate` never pays for them.
|
|
14
|
-
var
|
|
14
|
+
var v=5,c=8,h=DJ(new URL("../assets/glyphs.cps",import.meta.url)),JJ=DJ(new URL("../assets/glyphs.wide",import.meta.url)),TJ=(()=>{let J=h.byteLength>>>2;if((h.byteOffset&3)===0)return new Uint32Array(h.buffer,h.byteOffset,J);return new Uint32Array(h.buffer.slice(h.byteOffset,h.byteOffset+(J<<2)))})(),xQ=(()=>{let J=new Uint32Array(JJ.length),Q=0;for(let Z=0;Z<JJ.length;Z++)J[Z]=Q,Q+=(JJ[Z]===1?2*v:v)*c;return J})(),qJ=null;function o(J){let Q=0,Z=TJ.length-1;while(Q<=Z){let $=Q+Z>>>1,V=TJ[$];if(V<J)Q=$+1;else if(V>J)Z=$-1;else return $}return-1}function u(J){return JJ[J]===1}function MJ(J){if(qJ===null)qJ=vQ(DJ(new URL("../assets/glyphs.pix.z",import.meta.url)));let Q=xQ[J],Z=u(J)?2*v:v;return qJ.subarray(Q,Q+Z*c)}function yJ(J,Q,Z){let $=MJ(J),V=u(J)?2*v:v;return kQ($,V,c,Q,Z)}function kQ(J,Q,Z,$,V){let G=new Uint8Array($*V),Y=Q/$,H=Z/V;for(let z=0;z<V;z++){let K=z*H,U=(z+1)*H;for(let X=0;X<$;X++){let B=X*Y,D=(X+1)*Y,q=0,M=0,R=Math.min(Math.ceil(U),Z);for(let P=Math.floor(K);P<R;P++){let T=Math.min(P+1,U)-Math.max(P,K);if(T<=0)continue;let E=Math.min(Math.ceil(D),Q);for(let A=Math.floor(B);A<E;A++){let y=Math.min(A+1,D)-Math.max(A,B);if(y<=0)continue;let w=y*T;q+=w*J[P*Q+A],M+=w}}G[z*$+X]=M>0?Math.min(255,Math.max(0,Math.round(q/M))):0}}return G}import{deflateSync as fQ}from"node:zlib";//! Minimal grayscale PNG encoder (bit depth 8, color type 0), mirroring
|
|
15
15
|
//! pxpipe's png.ts: IHDR + one IDAT (zlib) + IEND, filter byte 0 per row.
|
|
16
|
-
var
|
|
16
|
+
var hQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $=0;$<8;$++)Z=(Z&1)!==0?3988292384^Z>>>1:Z>>>1;J[Q]=Z>>>0}return J})();function FJ(J,Q,Z,$){let V=$.length;J[Q]=V>>>24&255,J[Q+1]=V>>>16&255,J[Q+2]=V>>>8&255,J[Q+3]=V&255;let G=Q+4;for(let z=0;z<4;z++)J[G+z]=Z.charCodeAt(z);J.set($,G+4);let Y=4294967295;for(let z=G;z<G+4+V;z++)Y=hQ[(Y^J[z])&255]^Y>>>8;Y=~Y>>>0;let H=G+4+V;return J[H]=Y>>>24&255,J[H+1]=Y>>>16&255,J[H+2]=Y>>>8&255,J[H+3]=Y&255,H+4}function wJ(J,Q,Z){let $=new Uint8Array(Z*(Q+1));for(let z=0,K=0,U=0;z<Z;z++)$[K++]=0,$.set(J.subarray(U,U+Q),K),K+=Q,U+=Q;let V=fQ($,{level:6}),G=new Uint8Array(33+(12+V.length)+12);G[0]=137,G[1]=80,G[2]=78,G[3]=71,G[4]=13,G[5]=10,G[6]=26,G[7]=10;let Y=new Uint8Array(13);Y[0]=Q>>>24&255,Y[1]=Q>>>16&255,Y[2]=Q>>>8&255,Y[3]=Q&255,Y[4]=Z>>>24&255,Y[5]=Z>>>16&255,Y[6]=Z>>>8&255,Y[7]=Z&255,Y[8]=8,Y[9]=0;let H=FJ(G,8,"IHDR",Y);return H=FJ(G,H,"IDAT",V),FJ(G,H,"IEND",new Uint8Array(0)),G}//! Stage 2: the `pxpipe` imaging engine, ported from pxpipe's render.ts
|
|
17
17
|
//! production dense path (bare 5x8 AA cell, 312 cols, 1568x728 pages).
|
|
18
18
|
//! Glyphs cover BMP (pxpipe's atlas: Spleen for ASCII/code, Unifont fallback)
|
|
19
19
|
//! PLUS the astral planes (unifont_upper, incl. emoji) — beyond pxpipe, which
|
|
@@ -28,19 +28,19 @@ var wQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $
|
|
|
28
28
|
//! * `font` — `Tiny` renders the same atlas box-filtered into a 4x6 cell
|
|
29
29
|
//! (390 cols x 120 rows/page), ~40% fewer image-tokens; opt-in,
|
|
30
30
|
//! transcription-accuracy gated.
|
|
31
|
-
var
|
|
31
|
+
var i=4,QJ=4,mQ=1568,uQ=728,ZJ="↵",gQ="⏎",$J="→",dQ="⇢",WJ="⇥",lQ="⇨",pQ=9647,bJ=4,nQ=3,vJ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",cQ=/\n{4,}/g;function g(J){return J.length===4&&J.replace(/[A-Z]/g,(Q)=>Q.toLowerCase())==="tiny"?"tiny":"normal"}function hJ(J){let Q=J==="tiny"?4:v,Z=J==="tiny"?6:c,$=Math.floor((mQ-2*i)/Q),V=Math.floor((uQ-2*QJ)/Z);return{cw:Q,ch:Z,cols:$,maxLines:V,maxChars:$*V}}var oQ=(()=>{let J=new Uint8Array(128);for(let Q=0;Q<128;Q++){let Z=o(Q);J[Q]=Z>=0&&u(Z)?2:1}return J})();function NJ(J){if(J<128)return oQ[J];let Q=o(J);return Q>=0&&u(Q)?2:1}function iQ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<56320&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<57344)Z++}Q++}return Q}function jJ(J){let Q=J.split(`
|
|
32
32
|
`);for(let Z=0;Z<Q.length;Z++){let $=Q[Z],V=$.length;while(V>0){let G=$.charCodeAt(V-1);if(G!==32&&G!==9)break;V--}if(V!==$.length)Q[Z]=$.slice(0,V)}return Q.join(`
|
|
33
|
-
`).replace(
|
|
33
|
+
`).replace(cQ,`
|
|
34
34
|
|
|
35
35
|
|
|
36
|
-
`)}function mJ(J){if(!J.includes("\t"))return J;let Q="",Z=0;for(let $ of J)if($==="\t"){let V=
|
|
37
|
-
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=mJ(Q[Z]);return Q.join(
|
|
38
|
-
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=
|
|
39
|
-
`)){let G=mJ(V);if(G.length===0){Z.push("");continue}let Y="",H=0;for(let z of G){let K=
|
|
36
|
+
`)}function mJ(J){if(!J.includes("\t"))return J;let Q="",Z=0;for(let $ of J)if($==="\t"){let V=bJ-Z%bJ;Q+=$J;for(let G=1;G<V;G++)Q+=" ";Z+=V}else Q+=$,Z+=NJ($.codePointAt(0));return Q}function rQ(J){let Q=J.includes("\t")?J.replaceAll("\t",$J):J,Z=0;while(Z<Q.length&&Q.charCodeAt(Z)===32)Z++;if(Z>=nQ&&Z<vJ.length)return WJ+vJ[Z]+Q.slice(Z);return Q}function uJ(J){return J.includes(ZJ)?J.replaceAll(ZJ,gQ):J}function aQ(J){let Q=uJ(J);if(Q.includes($J))Q=Q.replaceAll($J,dQ);if(Q.includes(WJ))Q=Q.replaceAll(WJ,lQ);return Q}function tQ(J){let Q=jJ(J).split(`
|
|
37
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=mJ(Q[Z]);return Q.join(ZJ)}function sQ(J){let Q=jJ(J).split(`
|
|
38
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=rQ(Q[Z]);return Q.join(ZJ)}function eQ(J,Q){let Z=[],$=jJ(J);for(let V of $.split(`
|
|
39
|
+
`)){let G=mJ(V);if(G.length===0){Z.push("");continue}let Y="",H=0;for(let z of G){let K=NJ(z.codePointAt(0));if(H+K>Q)Z.push(Y),Y=z,H=K;else Y+=z,H+=K}if(Y.length!==0)Z.push(Y)}return Z}function JZ(J,Q,Z){let $=[],V=[],G=0;for(let Y of J){let H=iQ(Y),z=H+(V.length!==0?1:0);if(V.length!==0&&(V.length>=Q||G+z>Z))$.push(V),V=[],G=0;G+=H+(V.length!==0?1:0),V.push(Y)}if(V.length!==0)$.push(V);return $}function gJ(J){if(J<32)return!0;if(J>=127&&J<=159)return!0;if(J>=768&&J<=879)return!0;if(J===8203||J===8204||J===8205||J===8288||J===65279)return!0;if(J>=65024&&J<=65039)return!0;if(J>=917760&&J<=917999)return!0;return!1}function QZ(J){let Q=null,Z=0;for(let $ of J){let V=$.codePointAt(0);if(o(V)<0&&!gJ(V)){if(Q===null)Q=J.slice(0,Z);Q+=`[U+${V.toString(16).toUpperCase()}]`}else if(Q!==null)Q+=$;Z+=$.length}return Q??J}function dJ(J,Q,Z,$){let V=Q?Z?sQ(aQ(J)):tQ(uJ(J)):J;return JZ(eQ(QZ(V),$.cols),$.maxLines,$.maxChars)}function lJ(J,Q,Z){if(!Z)return 2*i+Q.cols*Q.cw;let $=0;for(let Y of J){let H=0;for(let z of Y)H+=NJ(z.codePointAt(0));if(H>Q.cols)H=Q.cols;if(H>$)$=H}let V=2*i+$*Q.cw,G=2*i+Q.cw;return V>G?V:G}var xJ=new Map;function kJ(J,Q,Z,$,V,G,Y){let H=o(V);if(H<0)return 0;let z=u(H),K=z?2*G.cw:G.cw,U;if(Y==="normal")U=MJ(H);else{let X=xJ.get(H);if(X===void 0)X=yJ(H,K,G.ch),xJ.set(H,X);U=X}for(let X=0;X<G.ch;X++){let B=($+X)*Q+Z,D=X*K;for(let q=0;q<K;q++){let M=U[D+q];if(M>0){let R=B+q;if(M>J[R])J[R]=M}}}return z?2:1}function ZZ(J,Q,Z,$){let V=lJ(J,Q,Z),G=2*QJ+J.length*Q.ch,Y=new Uint8Array(V*G),H=0;for(let z=0;z<J.length;z++){let K=QJ+z*Q.ch,U=0;for(let X of J[z]){if(U>=Q.cols)break;let B=i+U*Q.cw,D=kJ(Y,V,B,K,X.codePointAt(0),Q,$);if(D===0){if(D=1,gJ(X.codePointAt(0)));else if(H++,X!==" ")kJ(Y,V,B,K,pQ,Q,$)}U+=D}}for(let z=0;z<Y.length;z++)Y[z]=255-Y[z];return{png:wJ(Y,V,G),width:V,height:G,dropped:H}}function VJ(J,Q,Z,$){let V=hJ($),G=dJ(J,Q,Z,V).map((K)=>ZZ(K,V,Z,$)),Y=0,H=0,z=0;for(let K of G)Y+=K.width*K.height,H+=pJ(K.width,K.height),z+=K.dropped;return{pages:G,pixels:Y,tokens:H,dropped:z}}function GJ(J,Q,Z,$){let V=hJ($),G=dJ(J,Q,Z,V),Y=0,H=0;for(let z of G){let K=lJ(z,V,Z),U=2*QJ+z.length*V.ch;Y+=K*U,H+=pJ(K,U)}return{pages:G.length,pixels:Y,tokens:H}}var fJ=28;function pJ(J,Q){return Math.ceil(J/fJ)*Math.ceil(Q/fJ)}import{readFileSync as $Z}from"node:fs";import{join as VZ}from"node:path";//! pxpipe measurement-log summary (~/.pxpipe/events.jsonl), same math as the
|
|
40
40
|
//! node MCP: actual = every way input bytes get billed (input + cache reads +
|
|
41
41
|
//! cache creates) — ignoring cache_read would fake the savings.
|
|
42
|
-
class
|
|
43
|
-
`)){if(K.trim().length===0)continue;let
|
|
42
|
+
class YJ{value;constructor(J){this.value=J}}function nJ(){let J=process.env.TANUKI_EVENTS;if(J!==void 0)return J;let Q=process.env.HOME??"";return VZ(Q,".pxpipe","events.jsonl")}function d(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:0}function GZ(J){return J<0?-Math.round(-J):Math.round(J)}function IJ(){let J=nJ(),Q;try{Q=$Z(J,"utf8")}catch{return{available:!1,note:`no ${J} yet`}}let Z=0,$=0,V=0,G=0,Y=0,H=0;for(let K of Q.split(`
|
|
43
|
+
`)){if(K.trim().length===0)continue;let U;try{U=JSON.parse(K)}catch{continue}Z+=1;let X=U!==null&&typeof U==="object"&&!Array.isArray(U)?U:{};if(X.compressed===!0)$+=1,V+=d(X.orig_chars),G+=d(X.image_count);Y+=d(X.baseline_tokens),H+=d(X.input_tokens)+d(X.cache_read_tokens)+d(X.cache_create_tokens)}let z=Y>0&&H>0?new YJ(GZ((1-H/Y)*1000)/10):null;return{available:!0,requests:Z,compressedRequests:$,imagedChars:V,imagesEmitted:G,baselineTokens:Y,actualInputTokens:H,estInputSavedPct:z}}//! Implicit mode: a local Anthropic middlebox, the pxpipe deployment shape
|
|
44
44
|
//! without pxpipe's structural flaw. Rules that keep it injection-shaped-free:
|
|
45
45
|
//!
|
|
46
46
|
//! 1. The system prompt and tool definitions are NEVER touched.
|
|
@@ -56,7 +56,20 @@ class ZJ{value;constructor(J){this.value=J}}function nJ(){let J=process.env.TANU
|
|
|
56
56
|
//!
|
|
57
57
|
//! Responses stream through untouched; usage is scraped from the stream for
|
|
58
58
|
//! the ~/.pxpipe/events.jsonl savings log (same format tanuki_stats reads).
|
|
59
|
-
|
|
59
|
+
import{createHash as YZ}from"node:crypto";import{mkdirSync as HZ,readFileSync as zZ,writeFileSync as KZ}from"node:fs";import cJ from"node:process";//! Stash mode: context-mode's shape (content parked outside the context
|
|
60
|
+
//! window, queried on demand) fused with tanuki's pricing (big answers come
|
|
61
|
+
//! back as dense pages).
|
|
62
|
+
//!
|
|
63
|
+
//! stash = write the text to a content-addressed file and pay a few hundred
|
|
64
|
+
//! tokens for a deterministic map of what's there (distill stats, top
|
|
65
|
+
//! repeats, first/last lines, the id). fetch = pull a slice by regex query
|
|
66
|
+
//! (distill-powered) or line range; the caller images it only when pages
|
|
67
|
+
//! clearly win. Contract is byte-identical with the Rust engine.
|
|
68
|
+
function oJ(){let J=cJ.env.TANUKI_STASH;if(J!==void 0&&J!=="")return J;return`${cJ.env.HOME??""}/.tanuki/stash`}function iJ(J){let Q=YZ("sha256").update(J,"utf8").digest("hex").slice(0,12),Z=oJ();HZ(Z,{recursive:!0}),KZ(`${Z}/${Q}`,J);let $=Buffer.byteLength(J,"utf8"),V=J.split(`
|
|
69
|
+
`),G=b(J,null,2).stats,Y=[`stashed ${Q} · ${$} bytes · ${V.length} lines`,`distill map: ${G.origLines} -> ${G.outLines} lines · ${G.savedPct}% of chars removable · ${G.importantKept} error/warn lines`];if(G.topRepeats.length>0){Y.push("top repeats:");for(let z of G.topRepeats.slice(0,5)){let K=z.kind==="template"?" (template)":"";Y.push(` ×${z.count}${K} ${z.exemplar}`)}}let H="";for(let z=V.length-1;z>=0;z--)if(V[z]!==""){H=V[z];break}return Y.push(`first: ${n(p(V[0]),160)}`),Y.push(`last: ${n(p(H),160)}`),Y.push(`fetch: tanuki_fetch {"id":"${Q}","query":"<regex>"} or {"id":"${Q}","lines":"a-b"}`),{id:Q,overview:Y.join(`
|
|
70
|
+
`)}}function rJ(J,Q,Z){if(Q===null===(Z===null))throw Error("give exactly one of query or lines");let $;try{$=zZ(`${oJ()}/${J}`,"utf8")}catch{throw Error(`unknown stash id: ${J}`)}if(Z!==null){let V=/^(\d+)-(\d+)$/.exec(Z);if(V===null)throw Error("bad lines range");let G=$.split(`
|
|
71
|
+
`),Y=Math.max(1,Number(V[1])),H=Math.min(G.length,Number(V[2]));if(Number(V[1])>Number(V[2]))throw Error("bad lines range");return G.slice(Y-1,H).join(`
|
|
72
|
+
`)}return b($,Q,2).distilled}//! tanuki-context — token-cutting context pipeline.
|
|
60
73
|
//! pipeline: text -> distill (stage 0, logs) -> ladder level 0-4 (stage 1)
|
|
61
74
|
//! -> pxpipe imaging (stage 2, name kept from the original mechanic)
|
|
62
75
|
//!
|
|
@@ -65,15 +78,15 @@ class ZJ{value;constructor(J){this.value=J}}function nJ(){let J=process.env.TANU
|
|
|
65
78
|
//! tanuki-context estimate <file> [level] [--distill]
|
|
66
79
|
//! tanuki-context render <file> [level] [outdir]
|
|
67
80
|
//! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
|
|
68
|
-
var
|
|
69
|
-
`;for(let
|
|
70
|
-
`;K+=z+
|
|
71
|
-
`+Z+"]"}let V=J,G=Object.keys(V).sort(
|
|
81
|
+
var aJ="0.3.0",RJ=6;function XZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function L(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof YJ){let z=J.value;return Number.isFinite(z)&&Number.isInteger(z)?z.toFixed(1):String(z)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let U="[";for(let X=0;X<J.length;X++){if(X>0)U+=",";U+=L(J[X],!1)}return U+"]"}let z=Z+" ",K=`[
|
|
82
|
+
`;for(let U=0;U<J.length;U++){if(U>0)K+=`,
|
|
83
|
+
`;K+=z+L(J[U],!0,z)}return K+`
|
|
84
|
+
`+Z+"]"}let V=J,G=Object.keys(V).sort(XZ);if(G.length===0)return"{}";if(!Q){let z="{";for(let K=0;K<G.length;K++){if(K>0)z+=",";z+=JSON.stringify(G[K])+":"+L(V[G[K]],!1)}return z+"}"}let Y=Z+" ",H=`{
|
|
72
85
|
`;for(let z=0;z<G.length;z++){if(z>0)H+=`,
|
|
73
|
-
`;H+=Y+JSON.stringify(G[z])+": "+
|
|
74
|
-
`+Z+"}"}function
|
|
75
|
-
`}if(X+=`L${Z.level} ${K} (${
|
|
76
|
-
vs ~${z} text-tokens raw = TOTAL -${
|
|
86
|
+
`;H+=Y+JSON.stringify(G[z])+": "+L(V[G[z]],!0,Y)}return H+`
|
|
87
|
+
`+Z+"}"}function I(J,Q){return J!==null&&typeof J==="object"&&!Array.isArray(J)?J[Q]:void 0}function S(J){return typeof J==="string"?J:null}function HJ(J){return typeof J==="boolean"?J:null}function tJ(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:null}function x(J){let Q=0;for(let Z=0;Z<J.length;Z++){Q++;let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}}return Q}function sJ(J,Q,Z,$,V){let G,Y=null;if(Z||$!==null){let K=b(J,$,2);G=K.distilled,Y=K.stats}else G=J;let H=0;if(V){let K=t(G);G=K.text,H=K.entries}let z=e(G,Q);return{stage0:Y,compressed:z.compressed,protectedLines:z.protectedLines,level:z.level,cbEntries:H}}function r(J){return Math.round(J/4)}function a(J,Q){if(J===0)return 0;let Z=(1-Q/J)*100;return Z<0?-Math.round(-Z):Math.round(Z)}function eJ(J){return{text:S(I(J,"text"))??"",level:(tJ(I(J,"level"))??0)%256,distill:HJ(I(J,"distill"))??!1,query:S(I(J,"query")),reflow:HJ(I(J,"reflow"))??!0,pack:HJ(I(J,"pack"))??!0,font:S(I(J,"font"))??"normal",codebook:HJ(I(J,"codebook"))??!1}}function BZ(J){let Q=b(J,null,2).distilled,Z=[[!1,J],[!0,Q]],$={distill:!1,codebook:!1,text:J,tokens:1/0,pages:0};for(let[G,Y]of Z)for(let H of[!1,!0]){let z=H?t(Y).text:Y,K=GJ(z,!0,!0,g("normal"));if(K.tokens<$.tokens)$={distill:G,codebook:H,text:z,tokens:K.tokens,pages:K.pages}}let V=GJ($.text,!0,!0,g("tiny"));return{codebook:$.codebook,distill:$.distill,imageTokens:$.tokens,pages:$.pages,tinyImageTokens:V.tokens}}function JQ(J){let Q=eJ(J),Z=sJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=g(Q.font),V=GJ(Z.compressed,Q.reflow,Q.pack,$),G=V.tokens,Y=x(Q.text),H=x(Z.compressed),z=r(Y),[K,U]=s[Z.level];return{engine:"pxpipe",level:`${Z.level} ${K}`,loss:U,distill:Z.stage0,origChars:Y,stage1Chars:H,stage1SavedPct:a(Y,H),pages:V.pages,imageTokens:G,rawTextTokens:z,totalSavedPct:a(z,G),protectedLines:Z.protectedLines,pack:Q.pack,font:$==="tiny"?"tiny":"normal",codebook:Q.codebook?Z.cbEntries:!1,verdict:G<z?"PIPELINE cheaper":"TEXT cheaper",recommend:BZ(Q.text)}}function QQ(J){let Q=eJ(J),Z=sJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=g(Q.font),V=VJ(Z.compressed,Q.reflow,Q.pack,$),G=V.tokens,Y=x(Q.text),H=x(Z.compressed),z=r(Y),[K,U]=s[Z.level],X="";if(Z.stage0!==null){let q=Z.stage0;X+=`distill: ${q.origLines} -> ${q.outLines} lines (-${q.savedPct}% chars, ${q.collapsedRuns} runs, ${q.suppressedLines} exact + ${q.templateSuppressed} template suppressed, ${q.importantKept} error/warn kept)
|
|
88
|
+
`}if(X+=`L${Z.level} ${K} (${U}): ${Y} chars -> ${H} chars (stage1 -${a(Y,H)}%) -> ${V.pages.length} page(s), ~${G} image-tokens
|
|
89
|
+
vs ~${z} text-tokens raw = TOTAL -${a(z,G)}%`,Z.protectedLines>0)X+=` · ${Z.protectedLines} lines kept verbatim`;if(V.dropped>0)X+=` · ${V.dropped} unmapped glyphs -> ▯`;if(Z.cbEntries>0)X+=` · codebook: ${Z.cbEntries} sigils (see ·legend·)`;if($==="tiny")X+=" · font: tiny 4x6";if(Q.pack)X+=" · packed (⇥N indent, → tab)";if(Q.reflow)X+=" · ↵ = newline · engine: pxpipe";let B=[{type:"text",text:X}],D=Math.min(V.pages.length,RJ);for(let q=0;q<D;q++){let M=V.pages[q].png;B.push({type:"image",data:Buffer.from(M.buffer,M.byteOffset,M.byteLength).toString("base64"),mimeType:"image/png"})}if(V.pages.length>RJ)B.push({type:"text",text:`(+${V.pages.length-RJ} more page(s))`});return B}function ZQ(J){let Q=S(I(J,"text"))??"",Z=b(Q,S(I(J,"query")),2);return[{type:"text",text:L(Z.stats,!0)},{type:"text",text:Z.distilled}]}function $Q(J){let Q=S(I(J,"text"))??"",Z=(tJ(I(J,"level"))??1)%256,$=e(Q,Z),[V,G,Y]=s[$.level],H=x(Q),z=x($.compressed),K=r(H),U=r(z),X={level:`${$.level} ${V}`,loss:G,note:Y,origChars:H,outChars:z,approxOrigTokens:K,approxOutTokens:U,savedPct:a(K,U),protectedLines:$.protectedLines};return[{type:"text",text:L(X,!0)},{type:"text",text:$.compressed}]}function VQ(J){let Q=S(I(J,"text"))??"";return[{type:"text",text:iJ(Q).overview}]}function GQ(J){let Q=S(I(J,"id"))??"",Z=rJ(Q,S(I(J,"query"))??null,S(I(J,"lines"))??null),$=r(x(Z)),V=VJ(Z,!0,!0,g("normal"));if(!(V.tokens<=$*0.75&&$-V.tokens>=300&&V.pages.length<=6))return[{type:"text",text:Z}];let H=[{type:"text",text:`[tanuki-context stash ${Q}: slice of ${x(Z)} chars imaged as ${V.pages.length} PNG page(s), ~${V.tokens} vs ~${$} text tokens. ↵=newline →=tab ⇥N=indent]`}];for(let z of V.pages){let K=z.png;H.push({type:"image",data:Buffer.from(K.buffer,K.byteOffset,K.byteLength).toString("base64"),mimeType:"image/png"})}return H}//! Claude Agent SDK integration (`tanuki-context/agent`).
|
|
77
90
|
//!
|
|
78
91
|
//! Two flavours, both one-liners from an agent's point of view:
|
|
79
92
|
//!
|
|
@@ -90,7 +103,8 @@ vs ~${z} text-tokens raw = TOTAL -${o(z,G)}%`,Z.protectedLines>0)X+=` · ${Z.pro
|
|
|
90
103
|
//! The core package stays zero-dependency: the SDK and zod are touched only
|
|
91
104
|
//! inside `tanukiSdkServer()` via dynamic import, and both are already present
|
|
92
105
|
//! in any Agent SDK project (zod is the SDK's own peer dependency).
|
|
93
|
-
var
|
|
106
|
+
var qZ=["tanuki_render","tanuki_estimate","tanuki_distill","tanuki_compress","tanuki_stats","tanuki_stash","tanuki_fetch"],DZ=`tanuki-context turns bulky text (logs, command output, docs) into dense PNG pages that cost a fraction of the text tokens.
|
|
94
107
|
Workflow: call tanuki_estimate first (instant, exact, never renders pixels). Its "recommend" field already names the cheapest safe knob set, priced - do not probe combos by hand. If the verdict says "PIPELINE cheaper", call tanuki_render with the recommended knobs and use the returned pages instead of pasting the text.
|
|
95
108
|
For logs, pass distill:true (repeats collapse, error/warn lines stay verbatim; add query:"regex" to slice). For prose you will not quote verbatim, level 2-3 shrinks it further. codebook:true helps path-heavy logs. Never image content you must quote byte-exact at level 4 or font tiny.
|
|
96
|
-
|
|
109
|
+
For huge references you will only consult occasionally: tanuki_stash parks the text outside context for a few hundred tokens of map; tanuki_fetch pulls slices later, auto-imaged when pages win.
|
|
110
|
+
Pages decode as: ↵ = newline, → = tab, ⇥N = N leading spaces, a trailing ·legend· line maps sigils back to full tokens.`;function MZ(){let J=new URL("./cli.js",import.meta.url);if(J.protocol==="file:"){let Q=OZ(J);if(UZ(Q))return{type:"stdio",command:process.execPath,args:[Q]}}return{type:"stdio",command:"npx",args:["-y","tanuki-context"]}}function FZ(J="tanuki"){return qZ.map((Q)=>`mcp__${J}__${Q}`)}function H$(J,Q={}){let Z=J??{},$=Q.key??"tanuki";return{...Z,mcpServers:{...Z.mcpServers??{},[$]:Q.server??MZ()},allowedTools:[...Z.allowedTools??[],...FZ($)]}}function WZ(J){let Q=J.string().describe("the bulky text to process"),Z=J.number().int().min(0).max(4).optional().describe("ladder level 0-4 (default 0)"),$={text:Q,level:Z,distill:J.boolean().optional().describe("stage 0 log distiller"),query:J.string().optional().describe("distill: keep matching lines +context"),reflow:J.boolean().optional().describe("pack short lines into full rows (default true)"),pack:J.boolean().optional().describe("indent RLE + width trim, lossless (default true)"),font:J.enum(["normal","tiny"]).optional().describe("tiny = 4x6 cells, ~40% fewer tokens, gated"),codebook:J.boolean().optional().describe("repeated tokens/paths -> sigils + legend")},V=(Y)=>({content:Y}),G=async(Y)=>{try{return V(Y())}catch(H){return{content:[{type:"text",text:`tanuki-context error: ${H instanceof Error?H.message:String(H)}`}],isError:!0}}};return[{name:"tanuki_render",description:"Render text through the pipeline (optional distill/level/codebook) into dense PNG pages. Call after tanuki_estimate says PIPELINE cheaper.",inputSchema:$,handler:(Y)=>G(()=>QQ(Y))},{name:"tanuki_estimate",description:"Exact page/token math for the same arguments as tanuki_render, without touching pixels. Instant; call this first.",inputSchema:$,handler:(Y)=>G(()=>[{type:"text",text:L(JQ(Y),!0)}])},{name:"tanuki_distill",description:"Stage 0 alone: collapse repeated log lines/blocks and template near-dupes; error/warn lines kept verbatim. Output stays greppable text.",inputSchema:{text:Q,query:$.query},handler:(Y)=>G(()=>ZQ(Y))},{name:"tanuki_compress",description:"Stage 1 alone: graded text compression, levels 0-4, code/paths/hashes protected from level 2 up.",inputSchema:{text:Q,level:Z},handler:(Y)=>G(()=>$Q(Y))},{name:"tanuki_stats",description:"Session savings summary from the events log (honest denominator: input + cache reads + cache creates).",inputSchema:{},handler:()=>G(()=>[{type:"text",text:L(IJ(),!0)}])},{name:"tanuki_stash",description:"Park bulky text outside the context window; returns a compact map (distill stats, top repeats, id). Retrieval pattern, tanuki pricing on the way back.",inputSchema:{text:Q},handler:(Y)=>G(()=>VQ(Y))},{name:"tanuki_fetch",description:"Pull a slice of stashed text by id + query regex or lines 'a-b'. Big slices return as dense PNG pages automatically.",inputSchema:{id:$.query,query:$.query,lines:$.query},handler:(Y)=>G(()=>GQ(Y))}]}async function z$(){let J,Q;try{J=await import("@anthropic-ai/claude-agent-sdk");let $=await import("zod");Q=$.z??$}catch($){throw Error(`tanukiSdkServer() needs the host project's @anthropic-ai/claude-agent-sdk and zod (npm i @anthropic-ai/claude-agent-sdk zod). Import failed: ${$ instanceof Error?$.message:String($)}`)}let Z=WZ(Q).map(($)=>J.tool($.name,$.description,$.inputSchema,(V)=>$.handler(V)));return J.createSdkMcpServer({name:"tanuki-context",version:aJ,instructions:DZ,tools:Z})}export{H$ as withTanuki,WZ as tanukiSdkToolSpecs,z$ as tanukiSdkServer,MZ as tanukiMcpServer,FZ as tanukiAllowedTools,qZ as TANUKI_TOOL_NAMES,DZ as TANUKI_INSTRUCTIONS};
|
package/dist/cli.js
CHANGED
|
@@ -1,20 +1,20 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import{readFileSync as
|
|
3
|
-
·legend· `;for(let
|
|
4
|
-
`),
|
|
5
|
-
`),
|
|
6
|
-
`),z=Array(V.length);for(let
|
|
7
|
-
`).replace(
|
|
2
|
+
import{readFileSync as EZ,mkdirSync as zQ,writeFileSync as GQ}from"node:fs";import R from"node:process";function FQ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function MJ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}Q++}return Q}function WQ(J,Q){let Z=0,$=0;while(Z<J.length&&$<Q.length){let V=J.codePointAt(Z),z=Q.codePointAt($);if(V!==z)return V-z;Z+=V>65535?2:1,$+=z>65535?2:1}return J.length-Z-(Q.length-$)}function a(J){let Q=new Map,Z=(X)=>{Q.set(X,(Q.get(X)??0)+1)},$=(X)=>{if(MJ(X)>=12)Z(X);if(X.includes("/")){let O=X.split("/"),q="";for(let M=0;M<O.length;M++){if(M>0)q+="/";if(q+=O[M],M>=2){let P=q+"/";if(MJ(P)>=12)Z(P)}}}},V=-1;for(let X=0;X<J.length;){let O=J.codePointAt(X),q=O>65535?2:1;if(FQ(O)){if(V>=0)$(J.slice(V,X)),V=-1}else if(V<0)V=X;X+=q}if(V>=0)$(J.slice(V));let z=[];for(let[X,O]of Q)if(O>=3){let q=MJ(X);z.push({k:X,c:O,len:q,saved:(q-1)*O})}z.sort((X,O)=>O.saved-X.saved||WQ(X.k,O.k));let G=[];for(let X of"§¤¢£¥µ¶ª°±¬×÷ØÞßæðøþ¡¿")if(!J.includes(X))G.push(X);let Y=[],H=[];for(let{k:X,c:O,len:q}of z){if(Y.length>=G.length)break;if((q-1)*O<=q+3)continue;let M=!1;for(let _=0;_<H.length;_++){let k=H[_];if(k.startsWith(X)||X.startsWith(k)){M=!0;break}}if(M)continue;let P=G[Y.length];H.push(X),Y.push({sig:P,val:X,len:q})}if(Y.length===0)return{text:J,entries:0};let K=Y.map((X,O)=>O);K.sort((X,O)=>Y[O].len-Y[X].len);let U=J;for(let X=0;X<K.length;X++){let{sig:O,val:q}=Y[K[X]];U=U.replaceAll(q,O)}let B=`
|
|
3
|
+
·legend· `;for(let X=0;X<Y.length;X++)B+=Y[X].sig+"="+Y[X].val+" ";return U+=B.slice(0,-1),{text:U,entries:Y.length}}import{Buffer as xJ}from"node:buffer";var NQ=/\x1b\[[0-9;]*[A-Za-z]/g,FJ=/\b([0-9A-Za-z_]*(error|exception)s?|err|warn(ing)?s?|fail(s|ed|ure|ures)?|panic(s|ked)?|fatal|critical|traceback|denied|refused|timeouts?|timed.?out|assert(s|ed|ion|ions)?|segfault(s|ed)?)\b/i,jQ=/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/g,IQ=/\b[0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?\b/g,RQ=/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,AQ=/\b[0-9a-f]{7,64}\b/gi,PQ=/\b[0-9]+(\.[0-9]+)?[ \t\u00a0]?(ms|us|µs|ns|s|m|h|%|[KMGT]i?B)?\b/gi,_Q=/[0-9a-f]/i,EQ=3,SQ=8,WJ=2,fJ=40;function CQ(J){if(!_Q.test(J))return J;let Q=J.replace(jQ,"<ts>");return Q=Q.replace(IQ,"<time>"),Q=Q.replace(RQ,"<uuid>"),Q=Q.replace(AQ,"<hex>"),Q.replace(PQ,"<n>")}function p(J){if(J===32)return!0;if(J<9)return!1;if(J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function r(J){let Q=0,Z=J.length;while(Q<Z&&p(J.charCodeAt(Q)))Q++;while(Z>Q&&p(J.charCodeAt(Z-1)))Z--;return Q===0&&Z===J.length?J:J.slice(Q,Z)}function TQ(J,Q){let Z=0,$=J.length;while(Z<$&&p(J.charCodeAt(Z)))Z++;while($>Z&&p(J.charCodeAt($-1)))$--;let V=0;for(let z=Z;z<$&&V<Q;V++){let G=J.charCodeAt(z);z+=G>=55296&&G<=56319&&z+1<$?2:1}return V<Q}function LQ(J){let Q=0,Z=J.length;while(Q<Z&&p(J.charCodeAt(Q)))Q++;return J.charCodeAt(Q)===91&&J.charCodeAt(Q+1)===215}function yQ(J){let Q=[],Z=-1,$=!0,V=!1,z=!1,G=!0;for(let Y=0;Y<J.length;){let H=J.codePointAt(Y),K=H>65535?2:1,U=p(H);if(G){if(G=!1,U)Q.push("<v>")}if(U){if(Z>=0)Q.push($?J.slice(Z,Y):"<v>"),Z=-1;z=!0}else{if(V=!0,z=!1,Z<0)Z=Y,$=!0;if($){let B=H|32;if(B<97||B>122||H>127)$=!1}}Y+=K}if(Z>=0)Q.push($?J.slice(Z):"<v>");if(V&&z)Q.push("<v>");return Q.join(" ")}function t(J,Q){let Z=0,$=0,V=J.length;while(Z<V&&$<Q){let z=J.charCodeAt(Z);Z+=z>=55296&&z<=56319&&Z+1<V?2:1,$++}return Z>=V?J:J.slice(0,Z)}var wQ=/[.*+?^${}()|[\]\\]/g;function w(J,Q=null,Z=2){let $=Z,z=J.replace(NQ,"").split(`
|
|
4
|
+
`),G=z.length,Y=G,H=Array(G),K=Array(G),U=0;for(let D=0;D<G;D++){H[D]=CQ(z[D]);let F=FJ.test(z[D]);if(K[D]=F,F)U++}let B=[],X=[],O=[],q=0,M=0;while(M<G){let D=0,F=0;for(let W=1;W<=SQ&&M+2*W<=G;W++){if(K[M+W-1])break;let I=1;J:for(;;){let j=M+I*W;if(j+W>G)break;for(let S=0;S<W;S++)if(K[j+S]||H[j+S]!==H[M+S])break J;I++}if(I>=EQ&&W*(I-1)>D*(F>0?F-1:0))D=W,F=I}if(D>0){for(let W=M;W<M+D;W++)B.push(z[W]),X.push(K[W]),O.push(H[W]);B.push(D===1?` [×${F} similar]`:` [×${F} similar ${D}-line blocks]`),X.push(!1),O.push(null),q++,M+=D*F}else B.push(z[M]),X.push(K[M]),O.push(H[M]),M++}let P=new Map,_=new Map,k=WJ+1,L=[],E=[],x=0,f=0;for(let D=0;D<B.length;D++){let F=B[D],W=X[D];if(W||LQ(F)||TQ(F,4)){L.push(F),E.push(W);continue}let I=O[D],j=P.get(I);if(j!==void 0){if(j.count++,j.count<=WJ)L.push(F),E.push(!1);else x++;continue}let S=yQ(I);P.set(I,{count:1,exemplar:F});let i=_.get(S);if(i!==void 0)if(i.count++,i.count<=k)L.push(F),E.push(!1);else f++;else _.set(S,{count:1,exemplar:F}),L.push(F),E.push(!1)}let u=[];for(let D of P.values())if(D.count>WJ)u.push([D.count,"exact",D.exemplar]);for(let D of _.values())if(D.count>k)u.push([D.count,"template",D.exemplar]);if(u.sort((D,F)=>F[0]-D[0]),u.length>fJ)u.length=fJ;if(x+f>0){let D=`── ${x+f} repeated lines suppressed (${x} exact ×N, ${f} same-template; first occurrences kept above) ──`;L.push(D),E.push(FJ.test(D));for(let[F,W,I]of u){let S=` ×${F}${W==="template"?" (template)":""} ${t(r(I),160)}`;L.push(S),E.push(FJ.test(S))}}let g;if(Q!=null){let D;try{D=new RegExp(Q,"i")}catch{D=new RegExp(Q.replace(wQ,"\\$&"),"i")}let F=L.length,W=new Uint8Array(F);for(let j=0;j<F;j++)if(E[j]||D.test(L[j])){let S=j>$?j-$:0,i=Math.min(j+$,F-1);W.fill(1,S,i+1)}g=[];let I=0;for(let j=0;j<F;j++)if(W[j]){if(I>0)g.push(`… ${I} lines omitted`),I=0;g.push(L[j])}else I++;if(I>0)g.push(`… ${I} lines omitted`)}else g=L;let bJ=g.join(`
|
|
5
|
+
`),vJ=xJ.byteLength(J),kJ=xJ.byteLength(bJ),DJ=(1-kJ/vJ)*100,qQ=J.length===0?0:DJ<0?-Math.round(-DJ):Math.round(DJ),DQ=u.map(([D,F,W])=>({count:D,exemplar:t(r(W),160),kind:F})),MQ={collapsedRuns:q,importantKept:U,origChars:vJ,origLines:Y,outChars:kJ,outLines:g.length,query:Q??null,savedPct:qQ,suppressedLines:x,templateSuppressed:f,topRepeats:DQ};return{distilled:bJ,stats:MQ}}var zJ=[["none","none","passthrough (baseline)"],["whitespace","lossless","trailing whitespace + blank-line runs collapsed; safe for code"],["prose","light","L1 + prose lines: collapse spaces, cut redundant filler phrases (code/IDs protected)"],["dense","medium","L2 + prose: drop articles & intensifiers"],["caveman","heavy","L3 + prose: telegraphic — drop function words; gist only, NOT verbatim"]];var y=(J)=>new RegExp(`(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:${J})(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])`,"giu"),NJ=[[y("in order to"),"to"],[y("due to the fact that"),"because"],[y("at this point in time"),"now"],[y("in the event that"),"if"],[y("for the purpose of"),"for"],[y("with regard to"),"about"],[y("a large number of"),"many"],[y("it is important to note that"),""],[y("please note that"),""],[y("as a matter of fact"),""],[y("in terms of"),"for"],[y("the fact that"),"that"]],bQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:the|an|a)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),vQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:very|really|just|actually|basically|simply|quite|rather|essentially|literally)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),kQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:is|are|was|were|am|be|been|being|do|does|did|have|has|had|will|would|shall|should|can|could|may|might|of|to|in|on|at|for|with|that|this|these|those|it|its|there|here)(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]*","giu"),hJ=/ {2,}/g,xQ=new RegExp("[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+([.,;:!?])","gu"),fQ=/\n{3,}/g;function jJ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function hQ(J){let Q=0,Z=J.length;while(Q<Z){let $=J.codePointAt(Q);if(!jJ($))break;Q+=$>65535?2:1}while(Z>Q){let $=J.charCodeAt(Z-1);if(!jJ($))break;Z--}return Q===0&&Z===J.length?J:J.slice(Q,Z)}function mQ(J){if(J.length===0)return!1;let Q=J.charCodeAt(0);if(Q===32||Q===9)return!0;let Z=0,$=0,V=0,z=!1;for(let G=0;G<J.length;){let Y=J.codePointAt(G);if(G+=Y>65535?2:1,Z++,jJ(Y)){V=0;continue}if(V++,V>=24)z=!0;if(Y>=48&&Y<=57||Y>=65&&Y<=90||Y>=97&&Y<=122)continue;switch(Y){case 46:case 44:case 59:case 58:case 39:case 34:case 33:case 63:case 40:case 41:case 45:continue}$++}if($/Z>0.3)return!0;return z}function uQ(J,Q){let Z=J.replace(hJ," ");for(let V=0;V<NJ.length;V++)Z=Z.replace(NJ[V][0],NJ[V][1]);if(Q>=3)Z=Z.replace(bQ,""),Z=Z.replace(vQ,"");if(Q>=4)Z=Z.replace(kQ,"");Z=Z.replace(hJ," "),Z=hQ(Z.replace(xQ,"$1"));let $=Z.charCodeAt(0);if($>=97&&$<=122)Z=String.fromCharCode($-32)+Z.slice(1);return Z}function s(J,Q){let Z=Math.min(Q,4);if(Z===0)return{compressed:J,protectedLines:0,level:Z};let $=0,V=J.split(`
|
|
6
|
+
`),z=Array(V.length);for(let Y=0;Y<V.length;Y++){let H=V[Y],K=H.length;while(K>0){let B=H.charCodeAt(K-1);if(B!==32&&B!==9)break;K--}let U=K===H.length?H:H.slice(0,K);if(Z===1){z[Y]=U;continue}if(mQ(U)){$++,z[Y]=U;continue}z[Y]=uQ(U,Z)}return{compressed:z.join(`
|
|
7
|
+
`).replace(fQ,`
|
|
8
8
|
|
|
9
|
-
`),protectedLines:$,level:Z}}import{appendFileSync as
|
|
9
|
+
`),protectedLines:$,level:Z}}import{appendFileSync as OZ,mkdirSync as qZ}from"node:fs";import{dirname as DZ}from"node:path";import QQ from"node:http";import MZ from"node:https";import{URL as FZ}from"node:url";import{readFileSync as RJ}from"node:fs";import{inflateSync as gQ}from"node:zlib";//! Full-BMP glyph atlas (Spleen 5x8 for ASCII/code + Unifont fallback),
|
|
10
10
|
//! extracted from pxpipe's generated gray atlas by `tools/gen-glyphs.mjs`.
|
|
11
11
|
//!
|
|
12
12
|
//! Codepoints + wide flags load eagerly (~175 KB, needed for wrap math);
|
|
13
13
|
//! coverage pixels stay zlib-packed on disk and decompress lazily on the
|
|
14
14
|
//! first blit, so `estimate` never pays for them.
|
|
15
|
-
var
|
|
15
|
+
var h=5,e=8,d=RJ(new URL("../assets/glyphs.cps",import.meta.url)),GJ=RJ(new URL("../assets/glyphs.wide",import.meta.url)),mJ=(()=>{let J=d.byteLength>>>2;if((d.byteOffset&3)===0)return new Uint32Array(d.buffer,d.byteOffset,J);return new Uint32Array(d.buffer.slice(d.byteOffset,d.byteOffset+(J<<2)))})(),dQ=(()=>{let J=new Uint32Array(GJ.length),Q=0;for(let Z=0;Z<GJ.length;Z++)J[Z]=Q,Q+=(GJ[Z]===1?2*h:h)*e;return J})(),IJ=null;function JJ(J){let Q=0,Z=mJ.length-1;while(Q<=Z){let $=Q+Z>>>1,V=mJ[$];if(V<J)Q=$+1;else if(V>J)Z=$-1;else return $}return-1}function n(J){return GJ[J]===1}function AJ(J){if(IJ===null)IJ=gQ(RJ(new URL("../assets/glyphs.pix.z",import.meta.url)));let Q=dQ[J],Z=n(J)?2*h:h;return IJ.subarray(Q,Q+Z*e)}function uJ(J,Q,Z){let $=AJ(J),V=n(J)?2*h:h;return lQ($,V,e,Q,Z)}function lQ(J,Q,Z,$,V){let z=new Uint8Array($*V),G=Q/$,Y=Z/V;for(let H=0;H<V;H++){let K=H*Y,U=(H+1)*Y;for(let B=0;B<$;B++){let X=B*G,O=(B+1)*G,q=0,M=0,P=Math.min(Math.ceil(U),Z);for(let _=Math.floor(K);_<P;_++){let k=Math.min(_+1,U)-Math.max(_,K);if(k<=0)continue;let L=Math.min(Math.ceil(O),Q);for(let E=Math.floor(X);E<L;E++){let x=Math.min(E+1,O)-Math.max(E,X);if(x<=0)continue;let f=x*k;q+=f*J[_*Q+E],M+=f}}z[H*$+B]=M>0?Math.min(255,Math.max(0,Math.round(q/M))):0}}return z}import{deflateSync as pQ}from"node:zlib";//! Minimal grayscale PNG encoder (bit depth 8, color type 0), mirroring
|
|
16
16
|
//! pxpipe's png.ts: IHDR + one IDAT (zlib) + IEND, filter byte 0 per row.
|
|
17
|
-
var
|
|
17
|
+
var nQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $=0;$<8;$++)Z=(Z&1)!==0?3988292384^Z>>>1:Z>>>1;J[Q]=Z>>>0}return J})();function PJ(J,Q,Z,$){let V=$.length;J[Q]=V>>>24&255,J[Q+1]=V>>>16&255,J[Q+2]=V>>>8&255,J[Q+3]=V&255;let z=Q+4;for(let H=0;H<4;H++)J[z+H]=Z.charCodeAt(H);J.set($,z+4);let G=4294967295;for(let H=z;H<z+4+V;H++)G=nQ[(G^J[H])&255]^G>>>8;G=~G>>>0;let Y=z+4+V;return J[Y]=G>>>24&255,J[Y+1]=G>>>16&255,J[Y+2]=G>>>8&255,J[Y+3]=G&255,Y+4}function gJ(J,Q,Z){let $=new Uint8Array(Z*(Q+1));for(let H=0,K=0,U=0;H<Z;H++)$[K++]=0,$.set(J.subarray(U,U+Q),K),K+=Q,U+=Q;let V=pQ($,{level:6}),z=new Uint8Array(33+(12+V.length)+12);z[0]=137,z[1]=80,z[2]=78,z[3]=71,z[4]=13,z[5]=10,z[6]=26,z[7]=10;let G=new Uint8Array(13);G[0]=Q>>>24&255,G[1]=Q>>>16&255,G[2]=Q>>>8&255,G[3]=Q&255,G[4]=Z>>>24&255,G[5]=Z>>>16&255,G[6]=Z>>>8&255,G[7]=Z&255,G[8]=8,G[9]=0;let Y=PJ(z,8,"IHDR",G);return Y=PJ(z,Y,"IDAT",V),PJ(z,Y,"IEND",new Uint8Array(0)),z}//! Stage 2: the `pxpipe` imaging engine, ported from pxpipe's render.ts
|
|
18
18
|
//! production dense path (bare 5x8 AA cell, 312 cols, 1568x728 pages).
|
|
19
19
|
//! Glyphs cover BMP (pxpipe's atlas: Spleen for ASCII/code, Unifont fallback)
|
|
20
20
|
//! PLUS the astral planes (unifont_upper, incl. emoji) — beyond pxpipe, which
|
|
@@ -29,19 +29,19 @@ var mQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $
|
|
|
29
29
|
//! * `font` — `Tiny` renders the same atlas box-filtered into a 4x6 cell
|
|
30
30
|
//! (390 cols x 120 rows/page), ~40% fewer image-tokens; opt-in,
|
|
31
31
|
//! transcription-accuracy gated.
|
|
32
|
-
var
|
|
32
|
+
var QJ=4,YJ=4,cQ=1568,oQ=728,HJ="↵",iQ="⏎",KJ="→",aQ="⇢",_J="⇥",rQ="⇨",tQ=9647,dJ=4,sQ=3,lJ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",eQ=/\n{4,}/g;function v(J){return J.length===4&&J.replace(/[A-Z]/g,(Q)=>Q.toLowerCase())==="tiny"?"tiny":"normal"}function oJ(J){let Q=J==="tiny"?4:h,Z=J==="tiny"?6:e,$=Math.floor((cQ-2*QJ)/Q),V=Math.floor((oQ-2*YJ)/Z);return{cw:Q,ch:Z,cols:$,maxLines:V,maxChars:$*V}}var JZ=(()=>{let J=new Uint8Array(128);for(let Q=0;Q<128;Q++){let Z=JJ(Q);J[Q]=Z>=0&&n(Z)?2:1}return J})();function EJ(J){if(J<128)return JZ[J];let Q=JJ(J);return Q>=0&&n(Q)?2:1}function QZ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<56320&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<57344)Z++}Q++}return Q}function SJ(J){let Q=J.split(`
|
|
33
33
|
`);for(let Z=0;Z<Q.length;Z++){let $=Q[Z],V=$.length;while(V>0){let z=$.charCodeAt(V-1);if(z!==32&&z!==9)break;V--}if(V!==$.length)Q[Z]=$.slice(0,V)}return Q.join(`
|
|
34
|
-
`).replace(
|
|
34
|
+
`).replace(eQ,`
|
|
35
35
|
|
|
36
36
|
|
|
37
|
-
`)}function
|
|
38
|
-
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=
|
|
39
|
-
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=
|
|
40
|
-
`)){let z=
|
|
37
|
+
`)}function iJ(J){if(!J.includes("\t"))return J;let Q="",Z=0;for(let $ of J)if($==="\t"){let V=dJ-Z%dJ;Q+=KJ;for(let z=1;z<V;z++)Q+=" ";Z+=V}else Q+=$,Z+=EJ($.codePointAt(0));return Q}function ZZ(J){let Q=J.includes("\t")?J.replaceAll("\t",KJ):J,Z=0;while(Z<Q.length&&Q.charCodeAt(Z)===32)Z++;if(Z>=sQ&&Z<lJ.length)return _J+lJ[Z]+Q.slice(Z);return Q}function aJ(J){return J.includes(HJ)?J.replaceAll(HJ,iQ):J}function $Z(J){let Q=aJ(J);if(Q.includes(KJ))Q=Q.replaceAll(KJ,aQ);if(Q.includes(_J))Q=Q.replaceAll(_J,rQ);return Q}function VZ(J){let Q=SJ(J).split(`
|
|
38
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=iJ(Q[Z]);return Q.join(HJ)}function zZ(J){let Q=SJ(J).split(`
|
|
39
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=ZZ(Q[Z]);return Q.join(HJ)}function GZ(J,Q){let Z=[],$=SJ(J);for(let V of $.split(`
|
|
40
|
+
`)){let z=iJ(V);if(z.length===0){Z.push("");continue}let G="",Y=0;for(let H of z){let K=EJ(H.codePointAt(0));if(Y+K>Q)Z.push(G),G=H,Y=K;else G+=H,Y+=K}if(G.length!==0)Z.push(G)}return Z}function YZ(J,Q,Z){let $=[],V=[],z=0;for(let G of J){let Y=QZ(G),H=Y+(V.length!==0?1:0);if(V.length!==0&&(V.length>=Q||z+H>Z))$.push(V),V=[],z=0;z+=Y+(V.length!==0?1:0),V.push(G)}if(V.length!==0)$.push(V);return $}function rJ(J){if(J<32)return!0;if(J>=127&&J<=159)return!0;if(J>=768&&J<=879)return!0;if(J===8203||J===8204||J===8205||J===8288||J===65279)return!0;if(J>=65024&&J<=65039)return!0;if(J>=917760&&J<=917999)return!0;return!1}function HZ(J){let Q=null,Z=0;for(let $ of J){let V=$.codePointAt(0);if(JJ(V)<0&&!rJ(V)){if(Q===null)Q=J.slice(0,Z);Q+=`[U+${V.toString(16).toUpperCase()}]`}else if(Q!==null)Q+=$;Z+=$.length}return Q??J}function tJ(J,Q,Z,$){let V=Q?Z?zZ($Z(J)):VZ(aJ(J)):J;return YZ(GZ(HZ(V),$.cols),$.maxLines,$.maxChars)}function sJ(J,Q,Z){if(!Z)return 2*QJ+Q.cols*Q.cw;let $=0;for(let G of J){let Y=0;for(let H of G)Y+=EJ(H.codePointAt(0));if(Y>Q.cols)Y=Q.cols;if(Y>$)$=Y}let V=2*QJ+$*Q.cw,z=2*QJ+Q.cw;return V>z?V:z}var pJ=new Map;function nJ(J,Q,Z,$,V,z,G){let Y=JJ(V);if(Y<0)return 0;let H=n(Y),K=H?2*z.cw:z.cw,U;if(G==="normal")U=AJ(Y);else{let B=pJ.get(Y);if(B===void 0)B=uJ(Y,K,z.ch),pJ.set(Y,B);U=B}for(let B=0;B<z.ch;B++){let X=($+B)*Q+Z,O=B*K;for(let q=0;q<K;q++){let M=U[O+q];if(M>0){let P=X+q;if(M>J[P])J[P]=M}}}return H?2:1}function KZ(J,Q,Z,$){let V=sJ(J,Q,Z),z=2*YJ+J.length*Q.ch,G=new Uint8Array(V*z),Y=0;for(let H=0;H<J.length;H++){let K=YJ+H*Q.ch,U=0;for(let B of J[H]){if(U>=Q.cols)break;let X=QJ+U*Q.cw,O=nJ(G,V,X,K,B.codePointAt(0),Q,$);if(O===0){if(O=1,rJ(B.codePointAt(0)));else if(Y++,B!==" ")nJ(G,V,X,K,tQ,Q,$)}U+=O}}for(let H=0;H<G.length;H++)G[H]=255-G[H];return{png:gJ(G,V,z),width:V,height:z,dropped:Y}}function m(J,Q,Z,$){let V=oJ($),z=tJ(J,Q,Z,V).map((K)=>KZ(K,V,Z,$)),G=0,Y=0,H=0;for(let K of z)G+=K.width*K.height,Y+=eJ(K.width,K.height),H+=K.dropped;return{pages:z,pixels:G,tokens:Y,dropped:H}}function UJ(J,Q,Z,$){let V=oJ($),z=tJ(J,Q,Z,V),G=0,Y=0;for(let H of z){let K=sJ(H,V,Z),U=2*YJ+H.length*V.ch;G+=K*U,Y+=eJ(K,U)}return{pages:z.length,pixels:G,tokens:Y}}var cJ=28;function eJ(J,Q){return Math.ceil(J/cJ)*Math.ceil(Q/cJ)}import{readFileSync as UZ}from"node:fs";import{join as XZ}from"node:path";//! pxpipe measurement-log summary (~/.pxpipe/events.jsonl), same math as the
|
|
41
41
|
//! node MCP: actual = every way input bytes get billed (input + cache reads +
|
|
42
42
|
//! cache creates) — ignoring cache_read would fake the savings.
|
|
43
|
-
class
|
|
44
|
-
`)){if(K.trim().length===0)continue;let
|
|
43
|
+
class ZJ{value;constructor(J){this.value=J}}function CJ(){let J=process.env.TANUKI_EVENTS;if(J!==void 0)return J;let Q=process.env.HOME??"";return XZ(Q,".pxpipe","events.jsonl")}function c(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:0}function BZ(J){return J<0?-Math.round(-J):Math.round(J)}function JQ(){let J=CJ(),Q;try{Q=UZ(J,"utf8")}catch{return{available:!1,note:`no ${J} yet`}}let Z=0,$=0,V=0,z=0,G=0,Y=0;for(let K of Q.split(`
|
|
44
|
+
`)){if(K.trim().length===0)continue;let U;try{U=JSON.parse(K)}catch{continue}Z+=1;let B=U!==null&&typeof U==="object"&&!Array.isArray(U)?U:{};if(B.compressed===!0)$+=1,V+=c(B.orig_chars),z+=c(B.image_count);G+=c(B.baseline_tokens),Y+=c(B.input_tokens)+c(B.cache_read_tokens)+c(B.cache_create_tokens)}let H=G>0&&Y>0?new ZJ(BZ((1-Y/G)*1000)/10):null;return{available:!0,requests:Z,compressedRequests:$,imagedChars:V,imagesEmitted:z,baselineTokens:G,actualInputTokens:Y,estInputSavedPct:H}}//! Implicit mode: a local Anthropic middlebox, the pxpipe deployment shape
|
|
45
45
|
//! without pxpipe's structural flaw. Rules that keep it injection-shaped-free:
|
|
46
46
|
//!
|
|
47
47
|
//! 1. The system prompt and tool definitions are NEVER touched.
|
|
@@ -57,12 +57,25 @@ class e{value;constructor(J){this.value=J}}function SJ(){let J=process.env.TANUK
|
|
|
57
57
|
//!
|
|
58
58
|
//! Responses stream through untouched; usage is scraped from the stream for
|
|
59
59
|
//! the ~/.pxpipe/events.jsonl savings log (same format tanuki_stats reads).
|
|
60
|
-
var
|
|
61
|
-
`)}catch{}}function
|
|
62
|
-
${
|
|
60
|
+
var o={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,minSave:300,maxPages:20};function TJ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($<55296||$>56319)Q++}return Q}function WZ(J,Q){let Z=TJ(J);if(Z<Q.minChars)return null;let $=J;if(Q.distill)$=w($,null,2).distilled;let V=0;if(Q.codebook){let K=a($);$=K.text,V=K.entries}if(Q.level>0)$=s($,Q.level).compressed;let z=Math.round(Z/4),G=m($,!0,!0,Q.font);if(G.pages.length>Q.maxPages)return null;if(G.tokens>z*Q.ratio||z-G.tokens<Q.minSave)return null;let H=[{type:"text",text:`[tanuki-context: ${Z} chars imaged in place as ${G.pages.length} PNG page(s), ~${G.tokens} vs ~${z} text tokens. ↵=newline →=tab ⇥N=indent`+(V>0?`; ·legend· line maps ${V} sigils`:"")+"]"}];for(let K of G.pages)H.push({type:"image",source:{type:"base64",media_type:"image/png",data:Buffer.from(K.png.buffer,K.png.byteOffset,K.png.byteLength).toString("base64")}});return{blocks:H,origChars:Z,pages:G.pages.length,savedTokens:z-G.tokens}}function XJ(J){return J!==null&&typeof J==="object"&&!Array.isArray(J)}function NZ(J,Q){let Z;try{Z=JSON.parse(J)}catch{return null}if(!XJ(Z)||!Array.isArray(Z.messages))return null;let $=0,V=0,z=0,G=0,Y=new Map,H=(K)=>{let U=Y.get(K),B;if(U!==void 0){let X=TJ(K),O=`[tanuki-context: ${X} chars, byte-identical to a block imaged above (${U} PNG page(s)); not repeated]`;B={blocks:[{type:"text",text:O}],origChars:X,pages:0,savedTokens:Math.round(X/4)-Math.round(TJ(O)/4)}}else if(B=WZ(K,Q),B)Y.set(K,B.pages);if(B)$++,V+=B.origChars,z+=B.pages,G+=B.savedTokens;return B};for(let K=0;K<Z.messages.length-1;K++){let U=Z.messages[K];if(!XJ(U)||U.role!=="user")continue;if(typeof U.content==="string"){let X=H(U.content);if(X)U.content=X.blocks;continue}if(!Array.isArray(U.content))continue;let B=[];for(let X of U.content){if(!XJ(X)||X.cache_control!==void 0){B.push(X);continue}if(X.type==="text"&&typeof X.text==="string"){let O=H(X.text);if(O)B.push(...O.blocks);else B.push(X);continue}if(X.type==="tool_result"){if(typeof X.content==="string"){let O=H(X.content);if(O)X.content=O.blocks}else if(Array.isArray(X.content)){let O=[];for(let q of X.content){if(XJ(q)&&q.type==="text"&&typeof q.text==="string"&&q.cache_control===void 0){let M=H(q.text);if(M){O.push(...M.blocks);continue}}O.push(q)}X.content=O}}B.push(X)}U.content=B}if($===0)return null;return{body:JSON.stringify(Z),imagedBlocks:$,origChars:V,imageCount:z,savedTokens:G}}function jZ(J){let Q=(Z)=>{let $=Z.exec(J);return $?Number($[1]):0};return{input:Q(/"input_tokens"\s*:\s*(\d+)/),cacheRead:Q(/"cache_read_input_tokens"\s*:\s*(\d+)/),cacheCreate:Q(/"cache_creation_input_tokens"\s*:\s*(\d+)/)}}function IZ(J){try{let Q=CJ();qZ(DZ(Q),{recursive:!0}),OZ(Q,JSON.stringify(J)+`
|
|
61
|
+
`)}catch{}}function ZQ(J){let Q=new FZ(J.upstream),Z=Q.protocol==="https:"?MZ:QQ,$=QQ.createServer((V,z)=>{let G=[];V.on("data",(Y)=>G.push(Y)),V.on("end",()=>{let Y=Buffer.concat(G),H=V.method==="POST"&&(V.url??"").startsWith("/v1/messages")&&!(V.url??"").includes("count_tokens")&&V.headers["content-encoding"]===void 0,K=null;if(H){if(K=NZ(Y.toString("utf8"),J),K)Y=Buffer.from(K.body,"utf8")}let U={...V.headers};if(delete U.host,delete U.connection,U["content-length"]=String(Y.length),H)delete U["accept-encoding"];let B=Z.request({protocol:Q.protocol,hostname:Q.hostname,port:Q.port||(Q.protocol==="https:"?443:80),path:V.url,method:V.method,headers:U},(X)=>{if(z.writeHead(X.statusCode??502,X.headers),H){let O=[];X.on("data",(q)=>{O.push(q),z.write(q)}),X.on("end",()=>{z.end();let q=jZ(Buffer.concat(O).toString("utf8")),M=q.input+q.cacheRead+q.cacheCreate;IZ({ts:Date.now(),tool:"proxy",compressed:K!==null,orig_chars:K?.origChars??0,image_count:K?.imageCount??0,baseline_tokens:M+(K?.savedTokens??0),input_tokens:q.input,cache_read_tokens:q.cacheRead,cache_create_tokens:q.cacheCreate})})}else X.pipe(z)});B.on("error",(X)=>{z.writeHead(502,{"content-type":"application/json"}),z.end(JSON.stringify({type:"error",error:{type:"api_error",message:`tanuki proxy: upstream unreachable (${X.message})`}}))}),B.end(Y)})});return $.listen(J.port,"127.0.0.1",()=>{let V=$.address(),z=V!==null&&typeof V==="object"?V.port:J.port,G=`level=${J.level} distill=${J.distill} codebook=${J.codebook} font=${J.font} minChars=${J.minChars} ratio=${J.ratio} minSave=${J.minSave}`;process.stderr.write(`tanuki-context proxy on http://127.0.0.1:${z} -> ${J.upstream}
|
|
62
|
+
${G}
|
|
63
63
|
`+` rules: system prompt & tools untouched · in-place blocks only · latest message never imaged · cache_control skipped · identical blocks imaged once
|
|
64
64
|
`+` point your client at it: export ANTHROPIC_BASE_URL=http://127.0.0.1:${z}
|
|
65
|
-
`)}),$}
|
|
65
|
+
`)}),$}import{createHash as RZ}from"node:crypto";import{mkdirSync as AZ,readFileSync as PZ,writeFileSync as _Z}from"node:fs";import $Q from"node:process";//! Stash mode: context-mode's shape (content parked outside the context
|
|
66
|
+
//! window, queried on demand) fused with tanuki's pricing (big answers come
|
|
67
|
+
//! back as dense pages).
|
|
68
|
+
//!
|
|
69
|
+
//! stash = write the text to a content-addressed file and pay a few hundred
|
|
70
|
+
//! tokens for a deterministic map of what's there (distill stats, top
|
|
71
|
+
//! repeats, first/last lines, the id). fetch = pull a slice by regex query
|
|
72
|
+
//! (distill-powered) or line range; the caller images it only when pages
|
|
73
|
+
//! clearly win. Contract is byte-identical with the Rust engine.
|
|
74
|
+
function VQ(){let J=$Q.env.TANUKI_STASH;if(J!==void 0&&J!=="")return J;return`${$Q.env.HOME??""}/.tanuki/stash`}function LJ(J){let Q=RZ("sha256").update(J,"utf8").digest("hex").slice(0,12),Z=VQ();AZ(Z,{recursive:!0}),_Z(`${Z}/${Q}`,J);let $=Buffer.byteLength(J,"utf8"),V=J.split(`
|
|
75
|
+
`),z=w(J,null,2).stats,G=[`stashed ${Q} · ${$} bytes · ${V.length} lines`,`distill map: ${z.origLines} -> ${z.outLines} lines · ${z.savedPct}% of chars removable · ${z.importantKept} error/warn lines`];if(z.topRepeats.length>0){G.push("top repeats:");for(let H of z.topRepeats.slice(0,5)){let K=H.kind==="template"?" (template)":"";G.push(` ×${H.count}${K} ${H.exemplar}`)}}let Y="";for(let H=V.length-1;H>=0;H--)if(V[H]!==""){Y=V[H];break}return G.push(`first: ${t(r(V[0]),160)}`),G.push(`last: ${t(r(Y),160)}`),G.push(`fetch: tanuki_fetch {"id":"${Q}","query":"<regex>"} or {"id":"${Q}","lines":"a-b"}`),{id:Q,overview:G.join(`
|
|
76
|
+
`)}}function yJ(J,Q,Z){if(Q===null===(Z===null))throw Error("give exactly one of query or lines");let $;try{$=PZ(`${VQ()}/${J}`,"utf8")}catch{throw Error(`unknown stash id: ${J}`)}if(Z!==null){let V=/^(\d+)-(\d+)$/.exec(Z);if(V===null)throw Error("bad lines range");let z=$.split(`
|
|
77
|
+
`),G=Math.max(1,Number(V[1])),Y=Math.min(z.length,Number(V[2]));if(Number(V[1])>Number(V[2]))throw Error("bad lines range");return z.slice(G-1,Y).join(`
|
|
78
|
+
`)}return w($,Q,2).distilled}//! tanuki-context — token-cutting context pipeline.
|
|
66
79
|
//! pipeline: text -> distill (stage 0, logs) -> ladder level 0-4 (stage 1)
|
|
67
80
|
//! -> pxpipe imaging (stage 2, name kept from the original mechanic)
|
|
68
81
|
//!
|
|
@@ -71,22 +84,26 @@ var c={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,min
|
|
|
71
84
|
//! tanuki-context estimate <file> [level] [--distill]
|
|
72
85
|
//! tanuki-context render <file> [level] [outdir]
|
|
73
86
|
//! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
|
|
74
|
-
var
|
|
75
|
-
`;for(let
|
|
76
|
-
`;K+=H+
|
|
77
|
-
`+Z+"]"}let V=J,z=Object.keys(V).sort(
|
|
78
|
-
`;for(let H=0;H<z.length;H++){if(H>0)
|
|
79
|
-
`;G
|
|
80
|
-
`+Z+"}"}function
|
|
81
|
-
`}if(
|
|
82
|
-
vs ~${H} text-tokens raw = TOTAL -${
|
|
83
|
-
`)}function
|
|
84
|
-
`),
|
|
85
|
-
`);break}case"estimate":{let Q=J[2]??
|
|
86
|
-
`);break}case"render":{let Q=J[2]??
|
|
87
|
-
`);let
|
|
88
|
-
`);break}case"proxy":{let Q=(V,z)=>{let
|
|
89
|
-
usage: tanuki-context [
|
|
90
|
-
|
|
87
|
+
var SZ="0.3.0",wJ=6;function CZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function A(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof ZJ){let H=J.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let U="[";for(let B=0;B<J.length;B++){if(B>0)U+=",";U+=A(J[B],!1)}return U+"]"}let H=Z+" ",K=`[
|
|
88
|
+
`;for(let U=0;U<J.length;U++){if(U>0)K+=`,
|
|
89
|
+
`;K+=H+A(J[U],!0,H)}return K+`
|
|
90
|
+
`+Z+"]"}let V=J,z=Object.keys(V).sort(CZ);if(z.length===0)return"{}";if(!Q){let H="{";for(let K=0;K<z.length;K++){if(K>0)H+=",";H+=JSON.stringify(z[K])+":"+A(V[z[K]],!1)}return H+"}"}let G=Z+" ",Y=`{
|
|
91
|
+
`;for(let H=0;H<z.length;H++){if(H>0)Y+=`,
|
|
92
|
+
`;Y+=G+JSON.stringify(z[H])+": "+A(V[z[H]],!0,G)}return Y+`
|
|
93
|
+
`+Z+"}"}function N(J,Q){return J!==null&&typeof J==="object"&&!Array.isArray(J)?J[Q]:void 0}function T(J){return typeof J==="string"?J:null}function BJ(J){return typeof J==="boolean"?J:null}function UQ(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:null}function b(J){let Q=0;for(let Z=0;Z<J.length;Z++){Q++;let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}}return Q}function qJ(J,Q,Z,$,V){let z,G=null;if(Z||$!==null){let K=w(J,$,2);z=K.distilled,G=K.stats}else z=J;let Y=0;if(V){let K=a(z);z=K.text,Y=K.entries}let H=s(z,Q);return{stage0:G,compressed:H.compressed,protectedLines:H.protectedLines,level:H.level,cbEntries:Y}}function l(J){return Math.round(J/4)}function VJ(J,Q){if(J===0)return 0;let Z=(1-Q/J)*100;return Z<0?-Math.round(-Z):Math.round(Z)}function XQ(J){return{text:T(N(J,"text"))??"",level:(UQ(N(J,"level"))??0)%256,distill:BJ(N(J,"distill"))??!1,query:T(N(J,"query")),reflow:BJ(N(J,"reflow"))??!0,pack:BJ(N(J,"pack"))??!0,font:T(N(J,"font"))??"normal",codebook:BJ(N(J,"codebook"))??!1}}function TZ(J){let Q=w(J,null,2).distilled,Z=[[!1,J],[!0,Q]],$={distill:!1,codebook:!1,text:J,tokens:1/0,pages:0};for(let[z,G]of Z)for(let Y of[!1,!0]){let H=Y?a(G).text:G,K=UJ(H,!0,!0,v("normal"));if(K.tokens<$.tokens)$={distill:z,codebook:Y,text:H,tokens:K.tokens,pages:K.pages}}let V=UJ($.text,!0,!0,v("tiny"));return{codebook:$.codebook,distill:$.distill,imageTokens:$.tokens,pages:$.pages,tinyImageTokens:V.tokens}}function BQ(J){let Q=XQ(J),Z=qJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=v(Q.font),V=UJ(Z.compressed,Q.reflow,Q.pack,$),z=V.tokens,G=b(Q.text),Y=b(Z.compressed),H=l(G),[K,U]=zJ[Z.level];return{engine:"pxpipe",level:`${Z.level} ${K}`,loss:U,distill:Z.stage0,origChars:G,stage1Chars:Y,stage1SavedPct:VJ(G,Y),pages:V.pages,imageTokens:z,rawTextTokens:H,totalSavedPct:VJ(H,z),protectedLines:Z.protectedLines,pack:Q.pack,font:$==="tiny"?"tiny":"normal",codebook:Q.codebook?Z.cbEntries:!1,verdict:z<H?"PIPELINE cheaper":"TEXT cheaper",recommend:TZ(Q.text)}}function LZ(J){let Q=XQ(J),Z=qJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=v(Q.font),V=m(Z.compressed,Q.reflow,Q.pack,$),z=V.tokens,G=b(Q.text),Y=b(Z.compressed),H=l(G),[K,U]=zJ[Z.level],B="";if(Z.stage0!==null){let q=Z.stage0;B+=`distill: ${q.origLines} -> ${q.outLines} lines (-${q.savedPct}% chars, ${q.collapsedRuns} runs, ${q.suppressedLines} exact + ${q.templateSuppressed} template suppressed, ${q.importantKept} error/warn kept)
|
|
94
|
+
`}if(B+=`L${Z.level} ${K} (${U}): ${G} chars -> ${Y} chars (stage1 -${VJ(G,Y)}%) -> ${V.pages.length} page(s), ~${z} image-tokens
|
|
95
|
+
vs ~${H} text-tokens raw = TOTAL -${VJ(H,z)}%`,Z.protectedLines>0)B+=` · ${Z.protectedLines} lines kept verbatim`;if(V.dropped>0)B+=` · ${V.dropped} unmapped glyphs -> ▯`;if(Z.cbEntries>0)B+=` · codebook: ${Z.cbEntries} sigils (see ·legend·)`;if($==="tiny")B+=" · font: tiny 4x6";if(Q.pack)B+=" · packed (⇥N indent, → tab)";if(Q.reflow)B+=" · ↵ = newline · engine: pxpipe";let X=[{type:"text",text:B}],O=Math.min(V.pages.length,wJ);for(let q=0;q<O;q++){let M=V.pages[q].png;X.push({type:"image",data:Buffer.from(M.buffer,M.byteOffset,M.byteLength).toString("base64"),mimeType:"image/png"})}if(V.pages.length>wJ)X.push({type:"text",text:`(+${V.pages.length-wJ} more page(s))`});return X}function yZ(J){let Q=T(N(J,"text"))??"",Z=w(Q,T(N(J,"query")),2);return[{type:"text",text:A(Z.stats,!0)},{type:"text",text:Z.distilled}]}function wZ(J){let Q=T(N(J,"text"))??"",Z=(UQ(N(J,"level"))??1)%256,$=s(Q,Z),[V,z,G]=zJ[$.level],Y=b(Q),H=b($.compressed),K=l(Y),U=l(H),B={level:`${$.level} ${V}`,loss:z,note:G,origChars:Y,outChars:H,approxOrigTokens:K,approxOutTokens:U,savedPct:VJ(K,U),protectedLines:$.protectedLines};return[{type:"text",text:A(B,!0)},{type:"text",text:$.compressed}]}function bZ(J){let Q=T(N(J,"text"))??"";return[{type:"text",text:LJ(Q).overview}]}function vZ(J){let Q=T(N(J,"id"))??"",Z=yJ(Q,T(N(J,"query"))??null,T(N(J,"lines"))??null),$=l(b(Z)),V=m(Z,!0,!0,v("normal"));if(!(V.tokens<=$*0.75&&$-V.tokens>=300&&V.pages.length<=6))return[{type:"text",text:Z}];let Y=[{type:"text",text:`[tanuki-context stash ${Q}: slice of ${b(Z)} chars imaged as ${V.pages.length} PNG page(s), ~${V.tokens} vs ~${$} text tokens. ↵=newline →=tab ⇥N=indent]`}];for(let H of V.pages){let K=H.png;Y.push({type:"image",data:Buffer.from(K.buffer,K.byteOffset,K.byteLength).toString("base64"),mimeType:"image/png"})}return Y}function kZ(){let J={type:"string"},Q={type:"integer",minimum:0,maximum:4};return{tools:[{name:"tanuki_render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow (single-cell tabs, ⇥N indent runs, width-trimmed pages). font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",inputSchema:{type:"object",properties:{text:J,level:Q,distill:{type:"boolean"},query:{type:"string"},reflow:{type:"boolean"},pack:{type:"boolean"},font:{type:"string",enum:["normal","tiny"]},codebook:{type:"boolean"}},required:["text"]}},{name:"tanuki_estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field names the cheapest safe knob set (level 0), so one call replaces manual knob probing.",inputSchema:{type:"object",properties:{text:J,level:Q,distill:{type:"boolean"},query:{type:"string"},reflow:{type:"boolean"},pack:{type:"boolean"},font:{type:"string",enum:["normal","tiny"]},codebook:{type:"boolean"}},required:["text"]}},{name:"tanuki_distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes (exact + same-template) with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. Deterministic, order-preserving.",inputSchema:{type:"object",properties:{text:J,query:{type:"string"}},required:["text"]}},{name:"tanuki_compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",inputSchema:{type:"object",properties:{text:J,level:Q},required:["text"]}},{name:"tanuki_stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates).",inputSchema:{type:"object",properties:{}}},{name:"tanuki_stash",description:"Park bulky text outside the context window (content-addressed file under TANUKI_STASH or ~/.tanuki/stash) and get back a compact map: distill stats, top repeats, first/last lines, and the stash id. Pay a few hundred tokens now, fetch slices later - the retrieval pattern, with tanuki pricing on the way back.",inputSchema:{type:"object",properties:{text:J},required:["text"]}},{name:"tanuki_fetch",description:"Pull a slice of stashed text by id: query (regex, distill-powered: matches + error/warn lines + context) or lines 'a-b'. Big slices come back as dense PNG pages automatically when they clearly win (>=25% and >=300 tokens cheaper, <=6 pages); small ones stay text.",inputSchema:{type:"object",properties:{id:{type:"string"},query:{type:"string"},lines:{type:"string"}},required:["id"]}}]}}function xZ(J){let Q=T(N(J,"name"))??"",Z=N(J,"arguments"),$;switch(Q){case"tanuki_render":$=LZ(Z);break;case"tanuki_estimate":$=[{type:"text",text:A(BQ(Z),!0)}];break;case"tanuki_distill":$=yZ(Z);break;case"tanuki_compress":$=wZ(Z);break;case"tanuki_stats":$=[{type:"text",text:A(JQ(),!0)}];break;case"tanuki_stash":$=bZ(Z);break;case"tanuki_fetch":try{$=vZ(Z)}catch(V){return{ok:!1,error:V instanceof Error?V.message:String(V)}}break;default:return{ok:!1,error:`unknown tool: ${Q}`}}return{ok:!0,value:{content:$}}}function YQ(J){let Q=J.endsWith("\r")?J.slice(0,-1):J;if(Q.trim().length===0)return;let Z;try{Z=JSON.parse(Q)}catch{return}let $=Z!==null&&typeof Z==="object"&&!Array.isArray(Z)&&Object.prototype.hasOwnProperty.call(Z,"id"),V=$?Z.id??null:null,z=T(N(Z,"method")),G;switch(z){case"initialize":{let Y=T(N(N(Z,"params"),"protocolVersion"))??"2025-06-18";G={jsonrpc:"2.0",id:V,result:{protocolVersion:Y,capabilities:{tools:{}},serverInfo:{name:"tanuki-context",version:SZ}}};break}case"notifications/initialized":case"notifications/cancelled":G=void 0;break;case"ping":G={jsonrpc:"2.0",id:V,result:{}};break;case"tools/list":G={jsonrpc:"2.0",id:V,result:kZ()};break;case"tools/call":{let Y=xZ(N(Z,"params"));G=Y.ok?{jsonrpc:"2.0",id:V,result:Y.value}:{jsonrpc:"2.0",id:V,error:{code:-32602,message:Y.error}};break}default:G=$?{jsonrpc:"2.0",id:V,error:{code:-32601,message:"method not found"}}:void 0}if(G!==void 0)R.stdout.write(A(G,!1)+`
|
|
96
|
+
`)}function fZ(){let J=Buffer.alloc(0);R.stdin.on("data",(Q)=>{let Z=J.length>0?Buffer.concat([J,Q]):Q,$=0,V;while((V=Z.indexOf(10,$))!==-1)YQ(Z.toString("utf8",$,V)),$=V+1;J=Z.subarray($)}),R.stdin.on("end",()=>{if(J.length>0)YQ(J.toString("utf8")),J=Buffer.alloc(0)})}function C(J){R.stderr.write(J+`
|
|
97
|
+
`),R.exit(101)}function $J(J){try{return EZ(J,"utf8")}catch{C("read file")}}function OJ(J,Q){if(!/^\+?[0-9]+$/.test(J))return null;let Z=BigInt(J.startsWith("+")?J.slice(1):J);return Z>Q?null:Number(Z)}var HQ=255n,KQ=0xffffffffffffffffn;function OQ(){let J=R.argv.slice(1);switch(J[1]){case"distill":{let Q=J[2]??C("usage: tanuki-context distill <file> [query]"),Z=$J(Q),$=w(Z,J[3]??null,2);R.stdout.write(A($.stats,!1)+`
|
|
98
|
+
`);break}case"estimate":{let Q=J[2]??C("usage: tanuki-context estimate <file> [level] [--distill] [--no-pack] [--font tiny] [--codebook]"),Z=$J(Q),$=J.slice(3).filter((H)=>!H.startsWith("--")),V=$.length>0?OJ($[0],KQ)??0:0,z=J.indexOf("--font"),G=z!==-1&&J[z+1]!==void 0?J[z+1]:"normal",Y=BQ({text:Z,level:V,distill:J.includes("--distill"),pack:!J.includes("--no-pack"),font:G,codebook:J.includes("--codebook")});R.stdout.write(A(Y,!1)+`
|
|
99
|
+
`);break}case"render":{let Q=J[2]??C("usage: tanuki-context render <file> [level] [outdir] [--distill] [--no-pack] [--font tiny] [--codebook]"),Z=$J(Q),$=J.slice(3).filter((O)=>!O.startsWith("--")),V=$.length>0?OJ($[0],HQ)??0:0,z=!J.includes("--no-pack"),G=J.includes("--codebook"),Y=J.indexOf("--font"),H=v(Y!==-1&&J[Y+1]!==void 0?J[Y+1]:"normal"),K=qJ(Z,V,J.includes("--distill"),null,G),U=m(K.compressed,!0,z,H),B=U.tokens;R.stdout.write(A({pages:U.pages.length,imageTokens:B,dropped:U.dropped,rawTextTokens:l(b(Z))},!1)+`
|
|
100
|
+
`);let X=$[1];if(X!==void 0){try{zQ(X,{recursive:!0})}catch{C("mkdir")}for(let O=0;O<U.pages.length;O++)try{GQ(`${X}/page${O}.png`,U.pages[O].png)}catch{C("write png")}}break}case"bench":{let Q=J[2]??C("usage: tanuki-context bench <file> <op> [level] [runs] [--distill]"),Z=J[3]??"pipeline",$=J[4]!==void 0?OJ(J[4],HQ)??0:0,V=J[5]!==void 0?OJ(J[5],KQ)??3:3,z=J.includes("--distill"),G=$J(Q),Y=[],H=null;for(let K=0;K<=V;K++){let U=performance.now();if(Z==="distill")H=w(G,null,2).stats;else{let B=qJ(G,$,z,null,!1),X=m(B.compressed,!0,!1,"normal");H={pages:X.pages.length,imageTokens:X.tokens,stage1Chars:b(B.compressed),dropped:X.dropped}}if(K>0)Y.push(performance.now()-U)}if(Y.sort((K,U)=>K-U),Y.length===0)C("index out of bounds: the len is 0 but the index is 0");R.stdout.write(A({medianMs:new ZJ(Y[Math.floor(Y.length/2)]),runs:V,result:H},!1)+`
|
|
101
|
+
`);break}case"proxy":{let Q=(V,z)=>{let G=J.indexOf(V);if(G===-1||J[G+1]===void 0)return z;let Y=Number(J[G+1]);return Number.isFinite(Y)?Y:z},Z=J.indexOf("--upstream"),$=J.indexOf("--font");ZQ({port:Q("--port",8484),upstream:Z!==-1&&J[Z+1]!==void 0?J[Z+1]:R.env.TANUKI_UPSTREAM??"https://api.anthropic.com",level:Q("--level",o.level),distill:J.includes("--distill"),codebook:J.includes("--codebook"),font:v($!==-1&&J[$+1]!==void 0?J[$+1]:"normal"),minChars:Q("--min-chars",o.minChars),ratio:Q("--ratio",o.ratio),minSave:Q("--min-save",o.minSave),maxPages:Q("--max-pages",o.maxPages)});break}case"stash":{let Q=J[2]??C("usage: tanuki-context stash <file>"),Z=LJ($J(Q));R.stdout.write(Z.overview+`
|
|
102
|
+
`);break}case"fetch":{let Q=J[2]??C("usage: tanuki-context fetch <id> [outdir] [--query re] [--lines a-b]"),Z=J.indexOf("--query"),$=J.indexOf("--lines"),V="";try{V=yJ(Q,Z!==-1?J[Z+1]??null:null,$!==-1?J[$+1]??null:null)}catch(U){C(U instanceof Error?U.message:String(U))}let z=l(b(V)),G=m(V,!0,!0,v("normal"));if(!(G.tokens<=z*0.75&&z-G.tokens>=300&&G.pages.length<=6)){R.stdout.write(A({mode:"text"},!1)+`
|
|
103
|
+
`+V+`
|
|
104
|
+
`);break}R.stdout.write(A({imageTokens:G.tokens,mode:"pages",pages:G.pages.length,rawTextTokens:z},!1)+`
|
|
105
|
+
`);let H=new Set([Z+1,$+1]),K;for(let U=3;U<J.length;U++)if(!J[U].startsWith("--")&&!H.has(U)){K=J[U];break}if(K!==void 0){try{zQ(K,{recursive:!0})}catch{C("mkdir")}for(let U=0;U<G.pages.length;U++)try{GQ(`${K}/page${U}.png`,G.pages[U].png)}catch{C("write png")}}break}case"serve":case void 0:fZ();break;default:R.stderr.write(`unknown command: ${J[1]}
|
|
106
|
+
usage: tanuki-context [serve|proxy|distill|estimate|render|bench|stash|fetch] ...
|
|
107
|
+
`),R.exit(1)}}//! Bin entry: keeps src/main.ts importable as a library (src/agent.ts) without
|
|
91
108
|
//! starting the MCP server as an import side effect.
|
|
92
|
-
|
|
109
|
+
OQ();
|
package/dist/pi.js
CHANGED
|
@@ -11,5 +11,5 @@ import{spawn as W}from"node:child_process";import{fileURLToPath as X}from"node:u
|
|
|
11
11
|
//! ToolResult content shape, so results pass through untouched.
|
|
12
12
|
function Y(){let F=process.env.TANUKI_BIN;if(F)return{cmd:F,args:[]};return{cmd:process.execPath,args:[X(new URL("./cli.js",import.meta.url))]}}class Q{proc;buf="";nextId=1;pending=new Map;ready;constructor(){let{cmd:F,args:B}=Y();this.proc=W(F,B,{stdio:["pipe","pipe","inherit"]}),this.proc.stdout.setEncoding("utf8"),this.proc.stdout.on("data",(G)=>{this.buf+=G;let z;while((z=this.buf.indexOf(`
|
|
13
13
|
`))!==-1){let H=this.buf.slice(0,z).trim();if(this.buf=this.buf.slice(z+1),!H)continue;let D;try{D=JSON.parse(H)}catch{continue}let J=D.id!==void 0?this.pending.get(D.id):void 0;if(!J)continue;if(this.pending.delete(D.id),D.error)J.reject(Error(D.error.message??"tanuki-context error"));else J.resolve(D.result)}}),this.proc.on("exit",(G)=>{let z=Error(`tanuki-context server exited (code ${G})`);for(let H of this.pending.values())H.reject(z);this.pending.clear()}),this.ready=this.request("initialize",{})}request(F,B){let G=this.nextId++,{promise:z,resolve:H,reject:D}=Promise.withResolvers();if(!this.proc.stdin?.writable)return D(Error("tanuki-context server is gone")),z;return this.pending.set(G,{resolve:H,reject:D}),this.proc.stdin.write(JSON.stringify({jsonrpc:"2.0",id:G,method:F,params:B})+`
|
|
14
|
-
`),z}async call(F,B){return await this.ready,this.request("tools/call",{name:F,arguments:B})}kill(){this.proc.kill()}get alive(){return this.proc.exitCode===null&&!this.proc.killed}}var K=q.String({description:"The text to process"}),V=q.Optional(q.Integer({minimum:0,maximum:4,description:"0 raw · 1 whitespace · 2 prose · 3 dense · 4 caveman"})),N=q.Object({text:K,level:V,distill:q.Optional(q.Boolean()),query:q.Optional(q.String()),reflow:q.Optional(q.Boolean()),pack:q.Optional(q.Boolean()),font:q.Optional(q.String({enum:["normal","tiny"]})),codebook:q.Optional(q.Boolean())}),Z=[{name:"tanuki_render",label:"Tanuki Render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow. font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",parameters:N,snippet:"Render bulky text/logs as dense PNG pages that cost a fraction of the tokens"},{name:"tanuki_estimate",label:"Tanuki Estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field names the cheapest safe knob set (level 0), so one call replaces manual knob probing.",parameters:N,snippet:"Instant token verdict: would imaging this text beat sending it as text?"},{name:"tanuki_distill",label:"Tanuki Distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. Deterministic, order-preserving.",parameters:q.Object({text:K,query:q.Optional(q.String())}),snippet:"Deterministically dedupe noisy logs, keeping every error line verbatim"},{name:"tanuki_compress",label:"Tanuki Compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",parameters:q.Object({text:K,level:V}),snippet:"Graded text compression (lossless whitespace up to gist-only)"},{name:"tanuki_stats",label:"Tanuki Stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates).",parameters:q.Object({}),snippet:"Session savings summary from the tanuki/pxpipe event log"}];function $(F){let B=null,G=()=>{if(!B||!B.alive)B=new Q;return B};F.on("session_shutdown",async()=>{B?.kill(),B=null});for(let z of Z)F.registerTool({name:z.name,label:z.label,description:z.description,promptSnippet:z.snippet,parameters:z.parameters,async execute(H,D){let J=await G().call(z.name,D??{}),M=(J.content??[]).map((I)=>I.type==="image"?{type:"image",data:I.data??"",mimeType:I.mimeType??"image/png"}:{type:"text",text:I.text??""});if(J.isError)throw Error(M.map((I)=>("text"in I)?I.text:"").join(`
|
|
14
|
+
`),z}async call(F,B){return await this.ready,this.request("tools/call",{name:F,arguments:B})}kill(){this.proc.kill()}get alive(){return this.proc.exitCode===null&&!this.proc.killed}}var K=q.String({description:"The text to process"}),V=q.Optional(q.Integer({minimum:0,maximum:4,description:"0 raw · 1 whitespace · 2 prose · 3 dense · 4 caveman"})),N=q.Object({text:K,level:V,distill:q.Optional(q.Boolean()),query:q.Optional(q.String()),reflow:q.Optional(q.Boolean()),pack:q.Optional(q.Boolean()),font:q.Optional(q.String({enum:["normal","tiny"]})),codebook:q.Optional(q.Boolean())}),Z=[{name:"tanuki_render",label:"Tanuki Render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow. font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",parameters:N,snippet:"Render bulky text/logs as dense PNG pages that cost a fraction of the tokens"},{name:"tanuki_estimate",label:"Tanuki Estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff. The result's 'recommend' field names the cheapest safe knob set (level 0), so one call replaces manual knob probing.",parameters:N,snippet:"Instant token verdict: would imaging this text beat sending it as text?"},{name:"tanuki_distill",label:"Tanuki Distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. Deterministic, order-preserving.",parameters:q.Object({text:K,query:q.Optional(q.String())}),snippet:"Deterministically dedupe noisy logs, keeping every error line verbatim"},{name:"tanuki_compress",label:"Tanuki Compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",parameters:q.Object({text:K,level:V}),snippet:"Graded text compression (lossless whitespace up to gist-only)"},{name:"tanuki_stats",label:"Tanuki Stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates).",parameters:q.Object({}),snippet:"Session savings summary from the tanuki/pxpipe event log"},{name:"tanuki_stash",label:"Tanuki Stash",description:"Park bulky text outside the context window (content-addressed file under TANUKI_STASH or ~/.tanuki/stash) and get back a compact map: distill stats, top repeats, first/last lines, and the stash id. Pay a few hundred tokens now, fetch slices later - the retrieval pattern, with tanuki pricing on the way back.",parameters:q.Object({text:K}),snippet:"Park huge text on disk for a few hundred tokens of map; fetch slices later"},{name:"tanuki_fetch",label:"Tanuki Fetch",description:"Pull a slice of stashed text by id: query (regex, distill-powered: matches + error/warn lines + context) or lines 'a-b'. Big slices come back as dense PNG pages automatically when they clearly win (>=25% and >=300 tokens cheaper, <=6 pages); small ones stay text.",parameters:q.Object({id:q.String(),query:q.Optional(q.String()),lines:q.Optional(q.String())}),snippet:"Pull a stashed slice; big answers arrive as cheap dense pages"}];function $(F){let B=null,G=()=>{if(!B||!B.alive)B=new Q;return B};F.on("session_shutdown",async()=>{B?.kill(),B=null});for(let z of Z)F.registerTool({name:z.name,label:z.label,description:z.description,promptSnippet:z.snippet,parameters:z.parameters,async execute(H,D){let J=await G().call(z.name,D??{}),M=(J.content??[]).map((I)=>I.type==="image"?{type:"image",data:I.data??"",mimeType:I.mimeType??"image/png"}:{type:"text",text:I.text??""});if(J.isError)throw Error(M.map((I)=>("text"in I)?I.text:"").join(`
|
|
15
15
|
`)||z.name+" failed");return{content:M,details:{}}}})}export{$ as default};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tanuki-context",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "Token-cutting context pipeline as an MCP server: distill logs, compress text, render dense PNG pages (pxpipe imaging engine). Zero dependencies.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|