tanuki-context 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +317 -73
- package/assets/glyphs.pix.z +0 -0
- package/dist/agent.d.ts +67 -0
- package/dist/agent.js +96 -0
- package/dist/cli.js +59 -35
- package/dist/pi.js +15 -0
- package/package.json +33 -3
package/README.md
CHANGED
|
@@ -1,116 +1,360 @@
|
|
|
1
1
|
# tanuki-context
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/tanuki-context)
|
|
4
|
+
Zero dependencies. 0.98 MB tarball. Runs anywhere Node >= 18 runs. MIT.
|
|
5
|
+
|
|
6
|
+
A model pays about 1 token per 4 characters to read text. A PNG page is
|
|
7
|
+
billed by pixel dimensions (Anthropic's 28x28-pixel patch grid), and a dense
|
|
8
|
+
page fits 3+ characters per token. [pxpipe](https://github.com/teamchong/pxpipe)
|
|
9
|
+
measured ~3.1 chars per image-token on real Claude Code traffic and built a
|
|
10
|
+
transparent proxy on that gap. tanuki-context is the same imaging engine with
|
|
11
|
+
a different contract: the model calls a tool when it wants the cut, sees
|
|
12
|
+
exactly what got imaged, and nothing is rewritten behind its back.
|
|
13
|
+
|
|
14
|
+
It takes bulky context (logs, docs, command output), optionally distills and
|
|
15
|
+
compresses it, then renders it as dense PNG pages. You get the numbers before
|
|
16
|
+
committing: `estimate` prices both routes and says which one wins.
|
|
17
|
+
|
|
18
|
+
## Measured
|
|
19
|
+
|
|
20
|
+
A 200 KB slice of this machine's system journal: 1,370 lines from tailscaled,
|
|
21
|
+
NetworkManager, the kernel, wpa_supplicant, and friends. Not a synthetic
|
|
22
|
+
fixture, and not cherry-picked to repeat itself (a plain `journalctl` tail is
|
|
23
|
+
so repetitive that distill flattens it to 5 lines, which looks fake even
|
|
24
|
+
though it isn't).
|
|
25
|
+
|
|
26
|
+
| how it enters context | tokens | saved |
|
|
27
|
+
| -------------------------------- | -----: | --------: |
|
|
28
|
+
| pasted as raw text | 51,200 | 0 |
|
|
29
|
+
| rendered as pages (level 0) | 10,752 | **-79%** |
|
|
30
|
+
| distilled first, then rendered | 5,320 | **-90%** |
|
|
31
|
+
| distilled + codebook + tiny font | 2,352 | **-95%** |
|
|
32
|
+
|
|
33
|
+
Reproduce on your own log; every row is one command:
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
npx tanuki-context estimate your.log 0
|
|
37
|
+
npx tanuki-context estimate your.log 0 --distill
|
|
38
|
+
npx tanuki-context estimate your.log 0 --distill --codebook --font tiny
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Savings depend on the content. Repetitive logs crush well. Dense unique text
|
|
42
|
+
still gets the flat ~3-4x from imaging alone. Small inputs technically win
|
|
43
|
+
too, but by so little it is rarely worth the modality switch; see
|
|
44
|
+
[Limits](#limits-plainly).
|
|
45
|
+
|
|
46
|
+
## Quick start
|
|
5
47
|
|
|
6
48
|
```
|
|
7
49
|
npx tanuki-context
|
|
8
50
|
```
|
|
9
51
|
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
[
|
|
52
|
+
That is the MCP stdio server. Register it in any client:
|
|
53
|
+
|
|
54
|
+
```json
|
|
55
|
+
{ "mcpServers": { "tanuki-context": { "command": "npx", "args": ["-y", "tanuki-context"] } } }
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Or poke at the CLI first. Real output, same 200 KB journal slice:
|
|
59
|
+
|
|
60
|
+
```
|
|
61
|
+
npx tanuki-context estimate journal.log 2 --distill --codebook
|
|
62
|
+
# { "imageTokens": 3920, "rawTextTokens": 51200, "pages": 3, "verdict": "PIPELINE cheaper", ... }
|
|
63
|
+
npx tanuki-context render journal.log 2 ./pages --codebook
|
|
64
|
+
# ./pages/page0.png ...
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Clients
|
|
14
68
|
|
|
69
|
+
**OMP (oh-my-pi)** - `~/.omp/agent/mcp.json`, or `.omp/mcp.json` in the project:
|
|
70
|
+
|
|
71
|
+
```json
|
|
72
|
+
{ "mcpServers": { "tanuki-context": { "command": "npx", "args": ["-y", "tanuki-context"] } } }
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
**jcode** - `~/.jcode/mcp.json`, or `.jcode/mcp.json` in the project. jcode
|
|
76
|
+
speaks stdio only; `"shared": true` reuses one server across sessions:
|
|
77
|
+
|
|
78
|
+
```json
|
|
79
|
+
{ "mcpServers": { "tanuki-context": { "command": "npx", "args": ["-y", "tanuki-context"], "shared": true } } }
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
**pi** - pi has no MCP layer, so this package doubles as a pi extension (the
|
|
83
|
+
`"pi"` manifest field in package.json):
|
|
84
|
+
|
|
85
|
+
```
|
|
86
|
+
pi install npm:tanuki-context
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
The five `tanuki_*` tools register natively; the extension spawns this
|
|
90
|
+
package's own stdio server per session.
|
|
91
|
+
|
|
92
|
+
**Claude Code**:
|
|
93
|
+
|
|
94
|
+
```
|
|
95
|
+
claude mcp add tanuki-context -- npx -y tanuki-context
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
**Rust instead of Node?** Same pipeline, same numbers, one 5.7 MB static
|
|
99
|
+
binary:
|
|
100
|
+
|
|
101
|
+
```
|
|
102
|
+
cargo install --git https://github.com/Osyna/tanuki-context --branch rust
|
|
15
103
|
```
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
104
|
+
|
|
105
|
+
Then `"command": "tanuki-context", "args": []` in any snippet above, or
|
|
106
|
+
`TANUKI_BIN=~/.cargo/bin/tanuki-context` for the pi extension. The two
|
|
107
|
+
engines are parity-tested against each other on every change; see
|
|
108
|
+
[Two engines](#two-engines) below.
|
|
109
|
+
|
|
110
|
+
## How it works
|
|
111
|
+
|
|
112
|
+
```mermaid
|
|
113
|
+
flowchart LR
|
|
114
|
+
A[text / logs] --> B{log-shaped?}
|
|
115
|
+
B -- yes --> C["stage 0: distill<br/>collapse repeats xN<br/>errors stay verbatim<br/>optional query slice"]
|
|
116
|
+
B -- no --> D
|
|
117
|
+
C --> D["stage 1: ladder<br/>compression level 0-4"]
|
|
118
|
+
D --> E["stage 2: imaging<br/>312-col pages, 5x8 cells<br/>width-trimmed"]
|
|
119
|
+
E --> F["PNG pages<br/>tokens = 28-px patches"]
|
|
20
120
|
```
|
|
21
121
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
122
|
+
Three stages. The first two are optional.
|
|
123
|
+
|
|
124
|
+
**Stage 0, distill.** Built for logs. Repeated lines and repeated multi-line
|
|
125
|
+
blocks collapse to one exemplar plus an exact `xN` count. Near-duplicates
|
|
126
|
+
that differ only in timestamps, ids, or numbers fold into a template, also
|
|
127
|
+
counted. Error, warning, and fail lines are never touched. A `query` regex
|
|
128
|
+
returns only the slice you care about, with context. On the journal slice
|
|
129
|
+
above: 1,371 lines to 621, half the characters gone, all 318 error/warn
|
|
130
|
+
lines kept verbatim.
|
|
131
|
+
|
|
132
|
+
**Stage 1, the ladder.** Five text-compression levels, 0 (off) to 4 (gist
|
|
133
|
+
only). From level 2 up an exact-recall guard keeps code, identifiers, hashes,
|
|
134
|
+
paths, and indented lines verbatim, so loss is confined to prose. Honest
|
|
135
|
+
consequence: on technical files there is little prose to cut. This repo's
|
|
136
|
+
README loses 10% of its characters at level 2 and 14% at level 4; DESIGN.md
|
|
137
|
+
loses 6% at level 4. The ladder earns its keep on wordy prose (meeting notes,
|
|
138
|
+
chat transcripts, tickets); on code-heavy content it mostly just guards.
|
|
139
|
+
Level 1 (whitespace cleanup) is lossless and always safe.
|
|
140
|
+
|
|
141
|
+
**Stage 2, imaging.** The renderer packs text into 312-column pages of 5x8
|
|
142
|
+
antialiased cells (1568x728 px max; short pages get width-trimmed) and
|
|
143
|
+
encodes grayscale PNGs. This is where the big cut comes from, and it is flat:
|
|
144
|
+
raw text pays ~1 token per 4 chars, a full page carries ~28,000 chars for
|
|
145
|
+
1,456 image tokens. Full Unicode, 92,812 codepoints: CJK, Cyrillic, emoji,
|
|
146
|
+
astral planes. Unassigned codepoints render as readable `[U+HEX]` escapes.
|
|
147
|
+
Nothing disappears silently.
|
|
29
148
|
|
|
30
|
-
##
|
|
149
|
+
## Density knobs
|
|
31
150
|
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
|
37
|
-
|
|
|
38
|
-
|
|
|
151
|
+
Measured by `reference/methods-report.mjs` on three corpora; regenerate the
|
|
152
|
+
table with `bun reference/methods-report.mjs`. Percentages are against the
|
|
153
|
+
baseline renderer on the same content.
|
|
154
|
+
|
|
155
|
+
| knob | what it does | code | prose | log |
|
|
156
|
+
| ------------------- | ----------------------------------------------------------------------------- | ---: | ----: | ---: |
|
|
157
|
+
| `pack` (default on) | single-cell tabs, `⇥N` indent runs, width-trimmed pages; byte-exact | -5% | 0% | 0% |
|
|
158
|
+
| `codebook` | repeated tokens and path prefixes become 1-cell sigils plus a `·legend·` line | -9% | 0% | -38% |
|
|
159
|
+
| `font: "tiny"` | glyphs box-filtered into 4x6 cells; opt-in, see limits below | -36% | -38% | -40% |
|
|
160
|
+
| all three stacked | | **-45%** | **-38%** | **-62%** |
|
|
161
|
+
|
|
162
|
+
The three corpora: `src/main.ts` (code), `DESIGN.md` (prose), a path-heavy
|
|
163
|
+
synthetic log. Knobs compose but do not add up linearly; `codebook` needs
|
|
164
|
+
repetition to bite, `pack` needs indentation.
|
|
39
165
|
|
|
40
166
|
## Tools (MCP, stdio)
|
|
41
167
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
168
|
+
| tool | arguments | returns |
|
|
169
|
+
| ----------------- | --------------------------------------------------------- | ------------------------------------------------ |
|
|
170
|
+
| `tanuki_render` | `text, level?, distill?, query?, pack?, font?, codebook?` | PNG page blocks + savings breakdown |
|
|
171
|
+
| `tanuki_estimate` | same as render | exact page geometry and token math, numbers only |
|
|
172
|
+
| `tanuki_distill` | `text, query?` | stage 0 alone; output stays greppable text |
|
|
173
|
+
| `tanuki_compress` | `text, level` | stage 1 alone |
|
|
174
|
+
| `tanuki_stats` | none | savings summary from the session event log |
|
|
175
|
+
|
|
176
|
+
`estimate` never touches pixel data. Call it on everything; render only when
|
|
177
|
+
the verdict says the pipeline wins.
|
|
178
|
+
|
|
179
|
+
## Limits, plainly
|
|
180
|
+
|
|
181
|
+
- **Rendering is exact; reading is a model skill.** Pages are pixel-faithful
|
|
182
|
+
to pxpipe's production renderer, and pxpipe has receipts on how well models
|
|
183
|
+
read them: near-perfect on arithmetic, gist, and state, but 13/15 on exact
|
|
184
|
+
12-char hex strings for claude-fable-5, and misses are silent. Their
|
|
185
|
+
numbers: [pxpipe benchmarks](https://github.com/teamchong/pxpipe#benchmark-results-and-receipts).
|
|
186
|
+
Keep byte-exact-critical values (secrets, hashes you must retype) in text.
|
|
187
|
+
- **`font: "tiny"` trades legibility for tokens.** We measured 99.7%
|
|
188
|
+
character read-back, and `M`/`H` is the confusable pair. Fine for logs and
|
|
189
|
+
prose; skip it when the model must transcribe identifiers exactly.
|
|
190
|
+
- **Levels 2-4 reword prose.** The guard keeps code and identifiers verbatim,
|
|
191
|
+
but prose comes back paraphrase-shaped. Level 4 is gist only. Do not use it
|
|
192
|
+
for anything you may need to quote.
|
|
193
|
+
- **Tiny wins are not worth taking.** Width-trimming makes even small pages
|
|
194
|
+
cheap on paper (12 short lines: 56 image tokens vs 115 as text), but a
|
|
195
|
+
50-token win buys you an image the model has to read back instead of text
|
|
196
|
+
it can quote. The proxy gate defaults to at least 25% and 300 tokens saved
|
|
197
|
+
for exactly this reason; in tool mode, just leave small things alone.
|
|
198
|
+
|
|
199
|
+
## Implicit mode (proxy)
|
|
200
|
+
|
|
201
|
+
If you cannot touch the client, tanuki also runs as a local middlebox, the
|
|
202
|
+
way pxpipe deploys. We left the proxy model early on because rewriting the
|
|
203
|
+
system prompt into a user turn reads exactly like a prompt injection (an
|
|
204
|
+
agent once flagged it as an attack, and it was right to). The middlebox
|
|
205
|
+
came back with rules that remove that exact failure:
|
|
206
|
+
|
|
207
|
+
```
|
|
208
|
+
npx tanuki-context proxy # listens on 127.0.0.1:8484
|
|
209
|
+
export ANTHROPIC_BASE_URL=http://127.0.0.1:8484
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
Oversized text blocks in user messages and tool results are replaced in
|
|
213
|
+
place by a short visible marker plus PNG blocks, only when `estimate` says
|
|
214
|
+
imaging wins by a clear margin (default: at least 25% and 300 tokens). What
|
|
215
|
+
it never does:
|
|
216
|
+
|
|
217
|
+
- touch the system prompt or tool definitions
|
|
218
|
+
- move content between roles or positions
|
|
219
|
+
- image the latest message (you may need to quote it)
|
|
220
|
+
- rewrite blocks carrying `cache_control` (that would break their cache)
|
|
221
|
+
- rewrite anything when text is cheaper; those requests forward byte-for-byte
|
|
222
|
+
|
|
223
|
+
Responses stream through untouched. Savings land in `~/.pxpipe/events.jsonl`
|
|
224
|
+
(the same file `tanuki_stats` reads), with the baseline named: what Anthropic
|
|
225
|
+
billed, plus what the imaged blocks would have cost as text.
|
|
226
|
+
|
|
227
|
+
Knobs: `--port N` `--upstream URL` `--level 0-4` `--distill` `--codebook`
|
|
228
|
+
`--font tiny` `--min-chars N` `--ratio X` `--min-save N` `--max-pages N`.
|
|
229
|
+
Defaults are conservative: level 0, nothing lossy on.
|
|
230
|
+
|
|
231
|
+
## Claude Agent SDK
|
|
47
232
|
|
|
48
|
-
|
|
233
|
+
`tanuki-context/agent` wires the pipeline into agents built on the
|
|
234
|
+
[Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-typescript).
|
|
49
235
|
|
|
50
|
-
|
|
51
|
-
font=normal, codebook=false` = byte-identical to pxpipe).
|
|
236
|
+
External (subprocess per session, zero extra dependencies):
|
|
52
237
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
238
|
+
```ts
|
|
239
|
+
import { query } from "@anthropic-ai/claude-agent-sdk";
|
|
240
|
+
import { withTanuki } from "tanuki-context/agent";
|
|
241
|
+
|
|
242
|
+
for await (const msg of query({ prompt: task, options: withTanuki({ model: "claude-..." }) })) {
|
|
243
|
+
// agent now has tanuki_estimate / tanuki_render / tanuki_distill / ...
|
|
244
|
+
}
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
In-process (one server instance shared by every agent in the process):
|
|
248
|
+
|
|
249
|
+
```ts
|
|
250
|
+
import { tanukiSdkServer, tanukiAllowedTools, TANUKI_INSTRUCTIONS } from "tanuki-context/agent";
|
|
251
|
+
|
|
252
|
+
const tanuki = await tanukiSdkServer(); // needs the SDK + zod, which agent projects already have
|
|
253
|
+
const options = {
|
|
254
|
+
mcpServers: { tanuki },
|
|
255
|
+
allowedTools: tanukiAllowedTools(),
|
|
256
|
+
systemPrompt: { type: "preset", preset: "claude_code", append: TANUKI_INSTRUCTIONS },
|
|
257
|
+
};
|
|
258
|
+
// hand the same `options` to every agent in the team
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
`withTanuki(options?)` merges into existing options without clobbering other
|
|
262
|
+
servers or tools. `TANUKI_INSTRUCTIONS` is a canned prompt block that teaches
|
|
263
|
+
agents the estimate-first workflow and the page decode grammar. The core
|
|
264
|
+
package stays zero-dependency: the SDK and zod are optional peers, touched
|
|
265
|
+
only inside `tanukiSdkServer()`.
|
|
266
|
+
|
|
267
|
+
Python SDK, plain dict config:
|
|
268
|
+
|
|
269
|
+
```python
|
|
270
|
+
from claude_agent_sdk import ClaudeAgentOptions
|
|
271
|
+
|
|
272
|
+
options = ClaudeAgentOptions(
|
|
273
|
+
mcp_servers={"tanuki": {"command": "npx", "args": ["-y", "tanuki-context"]}},
|
|
274
|
+
allowed_tools=[f"mcp__tanuki__tanuki_{t}" for t in
|
|
275
|
+
["render", "estimate", "distill", "compress", "stats"]],
|
|
276
|
+
)
|
|
277
|
+
```
|
|
59
278
|
|
|
60
279
|
## CLI
|
|
61
280
|
|
|
62
281
|
```
|
|
63
282
|
npx tanuki-context # MCP stdio server (default)
|
|
283
|
+
npx tanuki-context proxy [--port 8484] [--upstream URL] [knobs] # implicit middlebox
|
|
64
284
|
npx tanuki-context distill <file> [query] # stats JSON to stdout
|
|
65
285
|
npx tanuki-context estimate <file> [level] [--distill] [--no-pack] [--font tiny] [--codebook]
|
|
66
286
|
npx tanuki-context render <file> [level] [outdir] [--no-pack] [--font tiny] [--codebook]
|
|
287
|
+
npx tanuki-context bench <file> <distill|pipeline> [level] [runs] # in-process timing
|
|
67
288
|
```
|
|
68
289
|
|
|
69
|
-
##
|
|
290
|
+
## Footprint
|
|
70
291
|
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
292
|
+
Measured 2026-07-26 on a Ryzen 7 9700X, Linux. Spawn time is the median of 5
|
|
293
|
+
cold starts to the first MCP response; the reference server is the original
|
|
294
|
+
node MCP that wraps pxpipe's library (kept in `reference/node-mcp/`).
|
|
74
295
|
|
|
75
|
-
|
|
296
|
+
| metric | node | bun | rust | node ref (pxpipe lib) |
|
|
297
|
+
| ----------------------- | ------- | ------- | ------- | --------------------- |
|
|
298
|
+
| spawn to first response | 35 ms | 27 ms | 0.4 ms | 158 ms |
|
|
299
|
+
| idle server RSS | 87 MB | 50 MB | 3.8 MB | 177 MB |
|
|
300
|
+
| distill a 12 MB log | 0.42 s | 0.31 s | 0.28 s | - |
|
|
301
|
+
| install | 0.98 MB tarball, zero deps | same | 5.7 MB static binary | node_modules tree |
|
|
76
302
|
|
|
77
|
-
|
|
78
|
-
|---|---|
|
|
79
|
-
| `src/main.ts` | MCP stdio server (hand-rolled JSON-RPC, serde_json-compatible serializer) + CLI |
|
|
80
|
-
| `src/distill.ts` | stage 0: 3-pass log distiller (runs/blocks ×N, exact+template near-dupes, query) |
|
|
81
|
-
| `src/ladder.ts` | stage 1: levels 0–4 with the exact-recall guard |
|
|
82
|
-
| `src/codebook.ts` | stage 0.5: repeated tokens/path prefixes → sigils + `·legend·` (opt-in, reversible) |
|
|
83
|
-
| `src/render.ts` | stage 2: reflow (`↵`), pack (`⇥N`/single-cell tab/width-trim), wrap, page split, AA blit, tiny 4×6 font |
|
|
84
|
-
| `src/atlas.ts` | full-Unicode glyph atlas (92,812 cps): codepoints/wide eager, pixels lazily inflated |
|
|
85
|
-
| `src/png.ts` | minimal grayscale PNG encoder (zlib via `node:zlib`, filter-0 rows) |
|
|
86
|
-
| `src/stats.ts` | events.jsonl summary |
|
|
87
|
-
| `assets/glyphs.*` | generated glyph data (35,501 codepoints, 0.4 MB packed) |
|
|
88
|
-
| `tools/gen-glyphs.mjs` | regenerates `assets/` from pxpipe's atlas |
|
|
89
|
-
| `reference/parity-ts.mjs` | parity harness: this port vs the rust binary — CLI JSON deep-equal, MCP replies, PNG pixels |
|
|
90
|
-
| `reference/parity.mjs` | original harness: rust vs pxpipe |
|
|
91
|
-
| `reference/node-mcp/` | the pre-rewrite node implementation, kept for comparison |
|
|
303
|
+
## Two engines
|
|
92
304
|
|
|
93
|
-
|
|
305
|
+
`main` is this TypeScript package. The [`rust` branch](../../tree/rust) is
|
|
306
|
+
the same pipeline in Rust: same patch-grid token model, same escapes, same
|
|
307
|
+
glyph atlas, same proxy rules. `reference/parity-ts.mjs` holds them to
|
|
308
|
+
byte-identical JSON and pixel-identical PNGs on every knob combination, plus
|
|
309
|
+
a full MCP session including error paths. When one engine changes, the
|
|
310
|
+
other follows, or the parity harness fails loudly. The npm packaging and
|
|
311
|
+
the Agent SDK / pi glue are TypeScript-only; everything the model sees is
|
|
312
|
+
identical.
|
|
94
313
|
|
|
95
|
-
|
|
96
|
-
|
|
314
|
+
The imaging engine is a remake of [pxpipe](https://github.com/teamchong/pxpipe):
|
|
315
|
+
page geometry and glyphs come from pxpipe's own generated atlas (Spleen 5x8
|
|
316
|
+
for ASCII and code, Unifont for the rest), so pages are pixel-faithful to
|
|
317
|
+
its production renderer, and the default path (`pack` off, normal font) is
|
|
318
|
+
byte-identical to it. Astral-plane coverage comes from GNU `unifont_upper`,
|
|
319
|
+
box-filtered to the same cells. pxpipe escapes astral codepoints as
|
|
320
|
+
`[U+HEX]`; tanuki renders them and reserves the escape for genuinely
|
|
321
|
+
unassigned codepoints.
|
|
97
322
|
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
323
|
+
## Repository layout
|
|
324
|
+
|
|
325
|
+
| path | role |
|
|
326
|
+
| ---------------------- | -------------------------------------------------------------------------------------------- |
|
|
327
|
+
| `src/main.ts` | MCP stdio server (hand-rolled JSON-RPC) + CLI (entry: `src/cli.ts`) |
|
|
328
|
+
| `src/agent.ts` | Claude Agent SDK glue: `withTanuki`, in-process `tanukiSdkServer` |
|
|
329
|
+
| `src/pi.ts` | pi extension: five native tools over a spawned stdio server (`TANUKI_BIN` picks the engine) |
|
|
330
|
+
| `src/distill.ts` | stage 0: 3-pass log distiller (runs, blocks, template near-dupes, query) |
|
|
331
|
+
| `src/ladder.ts` | stage 1: levels 0-4 with the exact-recall guard |
|
|
332
|
+
| `src/codebook.ts` | repeated tokens and path prefixes to sigils plus `·legend·` (opt-in) |
|
|
333
|
+
| `src/render.ts` | stage 2: reflow, pack, wrap, page split, AA blit, tiny 4x6 font |
|
|
334
|
+
| `src/proxy.ts` | implicit mode: local Anthropic middlebox, in-place block imaging |
|
|
335
|
+
| `src/atlas.ts` | glyph atlas (92,812 codepoints): metadata eager, pixels inflated lazily |
|
|
336
|
+
| `src/png.ts` | minimal grayscale PNG encoder (`node:zlib`, filter-0 rows) |
|
|
337
|
+
| `src/stats.ts` | event log summary |
|
|
338
|
+
| `assets/glyphs.*` | generated glyph data (0.4 MB packed) |
|
|
339
|
+
| `tools/gen-glyphs.mjs` | regenerates `assets/` from pxpipe's atlas |
|
|
340
|
+
| `reference/` | parity and benchmark harnesses (the retired node MCP lives here as an oracle) |
|
|
341
|
+
|
|
342
|
+
Architecture notes and the reasoning behind each stage live in
|
|
343
|
+
[DESIGN.md](DESIGN.md).
|
|
344
|
+
|
|
345
|
+
## Build
|
|
102
346
|
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
--manifest-path /tmp/tanuki-rust/Cargo.toml`, or set `TANUKI_BIN`):
|
|
347
|
+
Runs from source with Bun (`bun src/cli.ts`) or as the bundled,
|
|
348
|
+
Node-compatible files:
|
|
106
349
|
|
|
107
350
|
```
|
|
108
|
-
|
|
109
|
-
|
|
351
|
+
bun run build # dist/cli.js + dist/agent.js + dist/pi.js
|
|
352
|
+
bun test # 35 tests
|
|
353
|
+
bun run parity # TS vs rust binary, byte/pixel-exact (needs TANUKI_BIN)
|
|
110
354
|
```
|
|
111
355
|
|
|
112
|
-
Regenerating glyphs after a pxpipe atlas rebuild (
|
|
113
|
-
with `dist/` built):
|
|
356
|
+
Regenerating glyphs after a pxpipe atlas rebuild (needs a pxpipe checkout
|
|
357
|
+
with `dist/` built; the generator fetches `unifont_upper` on first run):
|
|
114
358
|
|
|
115
359
|
```
|
|
116
360
|
PXPIPE_DIST=~/Projects/pxpipe/dist node tools/gen-glyphs.mjs
|
package/assets/glyphs.pix.z
CHANGED
|
Binary file
|
package/dist/agent.d.ts
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
// Hand-maintained types for the `tanuki-context/agent` subpath (the built
|
|
2
|
+
// dist/agent.js is a minified bundle; keep this in sync with src/agent.ts).
|
|
3
|
+
|
|
4
|
+
export declare const TANUKI_TOOL_NAMES: readonly [
|
|
5
|
+
"tanuki_render",
|
|
6
|
+
"tanuki_estimate",
|
|
7
|
+
"tanuki_distill",
|
|
8
|
+
"tanuki_compress",
|
|
9
|
+
"tanuki_stats",
|
|
10
|
+
];
|
|
11
|
+
|
|
12
|
+
/** Canned agent guidance: estimate first, render on a winning verdict,
|
|
13
|
+
* distill logs, and the page decode grammar. */
|
|
14
|
+
export declare const TANUKI_INSTRUCTIONS: string;
|
|
15
|
+
|
|
16
|
+
export interface StdioServerConfig {
|
|
17
|
+
type: "stdio";
|
|
18
|
+
command: string;
|
|
19
|
+
args: string[];
|
|
20
|
+
env?: Record<string, string>;
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
/** External stdio MCP server config for `options.mcpServers`. */
|
|
24
|
+
export declare function tanukiMcpServer(): StdioServerConfig;
|
|
25
|
+
|
|
26
|
+
/** `mcp__<key>__<tool>` names for `options.allowedTools`. */
|
|
27
|
+
export declare function tanukiAllowedTools(key?: string): string[];
|
|
28
|
+
|
|
29
|
+
export interface TanukiOptions {
|
|
30
|
+
mcpServers?: Record<string, unknown>;
|
|
31
|
+
allowedTools?: string[];
|
|
32
|
+
[k: string]: unknown;
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/** Merge tanuki (server + allowed tools) into an Agent SDK options object. */
|
|
36
|
+
export declare function withTanuki<T extends TanukiOptions>(
|
|
37
|
+
options?: T,
|
|
38
|
+
opts?: { key?: string; server?: unknown },
|
|
39
|
+
): T & TanukiOptions;
|
|
40
|
+
|
|
41
|
+
export interface ZodChain {
|
|
42
|
+
optional(): ZodChain;
|
|
43
|
+
int(): ZodChain;
|
|
44
|
+
min(n: number): ZodChain;
|
|
45
|
+
max(n: number): ZodChain;
|
|
46
|
+
describe(text: string): ZodChain;
|
|
47
|
+
}
|
|
48
|
+
export interface ZodNamespace {
|
|
49
|
+
string(): ZodChain;
|
|
50
|
+
boolean(): ZodChain;
|
|
51
|
+
number(): ZodChain;
|
|
52
|
+
enum(values: readonly [string, ...string[]]): ZodChain;
|
|
53
|
+
}
|
|
54
|
+
export interface SdkToolSpec {
|
|
55
|
+
name: string;
|
|
56
|
+
description: string;
|
|
57
|
+
inputSchema: Record<string, ZodChain>;
|
|
58
|
+
handler: (args: Record<string, unknown>) => Promise<{ content: unknown[]; isError?: boolean }>;
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
/** The five tools as SDK tool specs (schema factory + handlers). */
|
|
62
|
+
export declare function tanukiSdkToolSpecs(z: ZodNamespace): SdkToolSpec[];
|
|
63
|
+
|
|
64
|
+
/** In-process MCP server (no subprocess; shareable across a team of agents).
|
|
65
|
+
* Requires @anthropic-ai/claude-agent-sdk and zod in the host project.
|
|
66
|
+
* Returned value goes straight into `options.mcpServers`. */
|
|
67
|
+
export declare function tanukiSdkServer(): Promise<unknown>;
|
package/dist/agent.js
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
import{createRequire as $Q}from"node:module";var PJ=$Q(import.meta.url);import{existsSync as QZ}from"node:fs";import{fileURLToPath as ZZ}from"node:url";function VQ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function $J(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}Q++}return Q}function GQ(J,Q){let Z=0,$=0;while(Z<J.length&&$<Q.length){let V=J.codePointAt(Z),G=Q.codePointAt($);if(V!==G)return V-G;Z+=V>65535?2:1,$+=G>65535?2:1}return J.length-Z-(Q.length-$)}function VJ(J){let Q=new Map,Z=(B)=>{Q.set(B,(Q.get(B)??0)+1)},$=(B)=>{if($J(B)>=12)Z(B);if(B.includes("/")){let F=B.split("/"),O="";for(let D=0;D<F.length;D++){if(D>0)O+="/";if(O+=F[D],D>=2){let I=O+"/";if($J(I)>=12)Z(I)}}}},V=-1;for(let B=0;B<J.length;){let F=J.codePointAt(B),O=F>65535?2:1;if(VQ(F)){if(V>=0)$(J.slice(V,B)),V=-1}else if(V<0)V=B;B+=O}if(V>=0)$(J.slice(V));let G=[];for(let[B,F]of Q)if(F>=3){let O=$J(B);G.push({k:B,c:F,len:O,saved:(O-1)*F})}G.sort((B,F)=>F.saved-B.saved||GQ(B.k,F.k));let Y=[];for(let B of"§¤¢£¥µ¶ª°±¬×÷ØÞßæðøþ¡¿")if(!J.includes(B))Y.push(B);let H=[],z=[];for(let{k:B,c:F,len:O}of G){if(H.length>=Y.length)break;if((O-1)*F<=O+3)continue;let D=!1;for(let R=0;R<z.length;R++){let T=z[R];if(T.startsWith(B)||B.startsWith(T)){D=!0;break}}if(D)continue;let I=Y[H.length];z.push(B),H.push({sig:I,val:B,len:O})}if(H.length===0)return{text:J,entries:0};let K=H.map((B,F)=>F);K.sort((B,F)=>H[F].len-H[B].len);let q=J;for(let B=0;B<K.length;B++){let{sig:F,val:O}=H[K[B]];q=q.replaceAll(O,F)}let X=`
|
|
2
|
+
·legend· `;for(let B=0;B<H.length;B++)X+=H[B].sig+"="+H[B].val+" ";return q+=X.slice(0,-1),{text:q,entries:H.length}}import{Buffer as AJ}from"node:buffer";var YQ=/\x1b\[[0-9;]*[A-Za-z]/g,GJ=/\b([0-9A-Za-z_]*(error|exception)s?|err|warn(ing)?s?|fail(s|ed|ure|ures)?|panic(s|ked)?|fatal|critical|traceback|denied|refused|timeouts?|timed.?out|assert(s|ed|ion|ions)?|segfault(s|ed)?)\b/i,HQ=/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/g,zQ=/\b[0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?\b/g,KQ=/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,XQ=/\b[0-9a-f]{7,64}\b/gi,BQ=/\b[0-9]+(\.[0-9]+)?[ \t\u00a0]?(ms|us|µs|ns|s|m|h|%|[KMGT]i?B)?\b/gi,qQ=/[0-9a-f]/i,UQ=3,OQ=8,YJ=2,_J=40;function FQ(J){if(!qQ.test(J))return J;let Q=J.replace(HQ,"<ts>");return Q=Q.replace(zQ,"<time>"),Q=Q.replace(KQ,"<uuid>"),Q=Q.replace(XQ,"<hex>"),Q.replace(BQ,"<n>")}function h(J){if(J===32)return!0;if(J<9)return!1;if(J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function EJ(J){let Q=0,Z=J.length;while(Q<Z&&h(J.charCodeAt(Q)))Q++;while(Z>Q&&h(J.charCodeAt(Z-1)))Z--;return Q===0&&Z===J.length?J:J.slice(Q,Z)}function DQ(J,Q){let Z=0,$=J.length;while(Z<$&&h(J.charCodeAt(Z)))Z++;while($>Z&&h(J.charCodeAt($-1)))$--;let V=0;for(let G=Z;G<$&&V<Q;V++){let Y=J.charCodeAt(G);G+=Y>=55296&&Y<=56319&&G+1<$?2:1}return V<Q}function MQ(J){let Q=0,Z=J.length;while(Q<Z&&h(J.charCodeAt(Q)))Q++;return J.charCodeAt(Q)===91&&J.charCodeAt(Q+1)===215}function WQ(J){let Q=[],Z=-1,$=!0,V=!1,G=!1,Y=!0;for(let H=0;H<J.length;){let z=J.codePointAt(H),K=z>65535?2:1,q=h(z);if(Y){if(Y=!1,q)Q.push("<v>")}if(q){if(Z>=0)Q.push($?J.slice(Z,H):"<v>"),Z=-1;G=!0}else{if(V=!0,G=!1,Z<0)Z=H,$=!0;if($){let X=z|32;if(X<97||X>122||z>127)$=!1}}H+=K}if(Z>=0)Q.push($?J.slice(Z):"<v>");if(V&&G)Q.push("<v>");return Q.join(" ")}function SJ(J,Q){let Z=0,$=0,V=J.length;while(Z<V&&$<Q){let G=J.charCodeAt(Z);Z+=G>=55296&&G<=56319&&Z+1<V?2:1,$++}return Z>=V?J:J.slice(0,Z)}var jQ=/[.*+?^${}()|[\]\\]/g;function c(J,Q=null,Z=2){let $=Z,G=J.replace(YQ,"").split(`
|
|
3
|
+
`),Y=G.length,H=Y,z=Array(Y),K=Array(Y),q=0;for(let U=0;U<Y;U++){z[U]=FQ(G[U]);let M=GJ.test(G[U]);if(K[U]=M,M)q++}let X=[],B=[],F=[],O=0,D=0;while(D<Y){let U=0,M=0;for(let W=1;W<=OQ&&D+2*W<=Y;W++){if(K[D+W-1])break;let N=1;J:for(;;){let j=D+N*W;if(j+W>Y)break;for(let A=0;A<W;A++)if(K[j+A]||z[j+A]!==z[D+A])break J;N++}if(N>=UQ&&W*(N-1)>U*(M>0?M-1:0))U=W,M=N}if(U>0){for(let W=D;W<D+U;W++)X.push(G[W]),B.push(K[W]),F.push(z[W]);X.push(U===1?` [×${M} similar]`:` [×${M} similar ${U}-line blocks]`),B.push(!1),F.push(null),O++,D+=U*M}else X.push(G[D]),B.push(K[D]),F.push(z[D]),D++}let I=new Map,R=new Map,T=YJ+1,_=[],P=[],L=0,y=0;for(let U=0;U<X.length;U++){let M=X[U],W=B[U];if(W||MQ(M)||DQ(M,4)){_.push(M),P.push(W);continue}let N=F[U],j=I.get(N);if(j!==void 0){if(j.count++,j.count<=YJ)_.push(M),P.push(!1);else L++;continue}let A=WQ(N);I.set(N,{count:1,exemplar:M});let g=R.get(A);if(g!==void 0)if(g.count++,g.count<=T)_.push(M),P.push(!1);else y++;else R.set(A,{count:1,exemplar:M}),_.push(M),P.push(!1)}let b=[];for(let U of I.values())if(U.count>YJ)b.push([U.count,"exact",U.exemplar]);for(let U of R.values())if(U.count>T)b.push([U.count,"template",U.exemplar]);if(b.sort((U,M)=>M[0]-U[0]),b.length>_J)b.length=_J;if(L+y>0){let U=`── ${L+y} repeated lines suppressed (${L} exact ×N, ${y} same-template; first occurrences kept above) ──`;_.push(U),P.push(GJ.test(U));for(let[M,W,N]of b){let A=` ×${M}${W==="template"?" (template)":""} ${SJ(EJ(N),160)}`;_.push(A),P.push(GJ.test(A))}}let v;if(Q!=null){let U;try{U=new RegExp(Q,"i")}catch{U=new RegExp(Q.replace(jQ,"\\$&"),"i")}let M=_.length,W=new Uint8Array(M);for(let j=0;j<M;j++)if(P[j]||U.test(_[j])){let A=j>$?j-$:0,g=Math.min(j+$,M-1);W.fill(1,A,g+1)}v=[];let N=0;for(let j=0;j<M;j++)if(W[j]){if(N>0)v.push(`… ${N} lines omitted`),N=0;v.push(_[j])}else N++;if(N>0)v.push(`… ${N} lines omitted`)}else v=_;let NJ=v.join(`
|
|
4
|
+
`),IJ=AJ.byteLength(J),RJ=AJ.byteLength(NJ),ZJ=(1-RJ/IJ)*100,JQ=J.length===0?0:ZJ<0?-Math.round(-ZJ):Math.round(ZJ),QQ=b.map(([U,M,W])=>({count:U,exemplar:SJ(EJ(W),160),kind:M})),ZQ={collapsedRuns:O,importantKept:q,origChars:IJ,origLines:H,outChars:RJ,outLines:v.length,query:Q??null,savedPct:JQ,suppressedLines:L,templateSuppressed:y,topRepeats:QQ};return{distilled:NJ,stats:ZQ}}var o=[["none","none","passthrough (baseline)"],["whitespace","lossless","trailing whitespace + blank-line runs collapsed; safe for code"],["prose","light","L1 + prose lines: collapse spaces, cut redundant filler phrases (code/IDs protected)"],["dense","medium","L2 + prose: drop articles & intensifiers"],["caveman","heavy","L3 + prose: telegraphic — drop function words; gist only, NOT verbatim"]];var E=(J)=>new RegExp(`(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:${J})(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])`,"giu"),HJ=[[E("in order to"),"to"],[E("due to the fact that"),"because"],[E("at this point in time"),"now"],[E("in the event that"),"if"],[E("for the purpose of"),"for"],[E("with regard to"),"about"],[E("a large number of"),"many"],[E("it is important to note that"),""],[E("please note that"),""],[E("as a matter of fact"),""],[E("in terms of"),"for"],[E("the fact that"),"that"]],NQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:the|an|a)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),IQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:very|really|just|actually|basically|simply|quite|rather|essentially|literally)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),RQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:is|are|was|were|am|be|been|being|do|does|did|have|has|had|will|would|shall|should|can|could|may|might|of|to|in|on|at|for|with|that|this|these|those|it|its|there|here)(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]*","giu"),CJ=/ {2,}/g,PQ=new RegExp("[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+([.,;:!?])","gu"),AQ=/\n{3,}/g;function zJ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function _Q(J){let Q=0,Z=J.length;while(Q<Z){let $=J.codePointAt(Q);if(!zJ($))break;Q+=$>65535?2:1}while(Z>Q){let $=J.charCodeAt(Z-1);if(!zJ($))break;Z--}return Q===0&&Z===J.length?J:J.slice(Q,Z)}function EQ(J){if(J.length===0)return!1;let Q=J.charCodeAt(0);if(Q===32||Q===9)return!0;let Z=0,$=0,V=0,G=!1;for(let Y=0;Y<J.length;){let H=J.codePointAt(Y);if(Y+=H>65535?2:1,Z++,zJ(H)){V=0;continue}if(V++,V>=24)G=!0;if(H>=48&&H<=57||H>=65&&H<=90||H>=97&&H<=122)continue;switch(H){case 46:case 44:case 59:case 58:case 39:case 34:case 33:case 63:case 40:case 41:case 45:continue}$++}if($/Z>0.3)return!0;return G}function SQ(J,Q){let Z=J.replace(CJ," ");for(let V=0;V<HJ.length;V++)Z=Z.replace(HJ[V][0],HJ[V][1]);if(Q>=3)Z=Z.replace(NQ,""),Z=Z.replace(IQ,"");if(Q>=4)Z=Z.replace(RQ,"");Z=Z.replace(CJ," "),Z=_Q(Z.replace(PQ,"$1"));let $=Z.charCodeAt(0);if($>=97&&$<=122)Z=String.fromCharCode($-32)+Z.slice(1);return Z}function i(J,Q){let Z=Math.min(Q,4);if(Z===0)return{compressed:J,protectedLines:0,level:Z};let $=0,V=J.split(`
|
|
5
|
+
`),G=Array(V.length);for(let H=0;H<V.length;H++){let z=V[H],K=z.length;while(K>0){let X=z.charCodeAt(K-1);if(X!==32&&X!==9)break;K--}let q=K===z.length?z:z.slice(0,K);if(Z===1){G[H]=q;continue}if(EQ(q)){$++,G[H]=q;continue}G[H]=SQ(q,Z)}return{compressed:G.join(`
|
|
6
|
+
`).replace(AQ,`
|
|
7
|
+
|
|
8
|
+
`),protectedLines:$,level:Z}}import{readFileSync as XJ}from"node:fs";import{inflateSync as CQ}from"node:zlib";//! Full-BMP glyph atlas (Spleen 5x8 for ASCII/code + Unifont fallback),
|
|
9
|
+
//! extracted from pxpipe's generated gray atlas by `tools/gen-glyphs.mjs`.
|
|
10
|
+
//!
|
|
11
|
+
//! Codepoints + wide flags load eagerly (~175 KB, needed for wrap math);
|
|
12
|
+
//! coverage pixels stay zlib-packed on disk and decompress lazily on the
|
|
13
|
+
//! first blit, so `estimate` never pays for them.
|
|
14
|
+
var w=5,d=8,x=XJ(new URL("../assets/glyphs.cps",import.meta.url)),r=XJ(new URL("../assets/glyphs.wide",import.meta.url)),TJ=(()=>{let J=x.byteLength>>>2;if((x.byteOffset&3)===0)return new Uint32Array(x.buffer,x.byteOffset,J);return new Uint32Array(x.buffer.slice(x.byteOffset,x.byteOffset+(J<<2)))})(),TQ=(()=>{let J=new Uint32Array(r.length),Q=0;for(let Z=0;Z<r.length;Z++)J[Z]=Q,Q+=(r[Z]===1?2*w:w)*d;return J})(),KJ=null;function l(J){let Q=0,Z=TJ.length-1;while(Q<=Z){let $=Q+Z>>>1,V=TJ[$];if(V<J)Q=$+1;else if(V>J)Z=$-1;else return $}return-1}function k(J){return r[J]===1}function BJ(J){if(KJ===null)KJ=CQ(XJ(new URL("../assets/glyphs.pix.z",import.meta.url)));let Q=TQ[J],Z=k(J)?2*w:w;return KJ.subarray(Q,Q+Z*d)}function LJ(J,Q,Z){let $=BJ(J),V=k(J)?2*w:w;return LQ($,V,d,Q,Z)}function LQ(J,Q,Z,$,V){let G=new Uint8Array($*V),Y=Q/$,H=Z/V;for(let z=0;z<V;z++){let K=z*H,q=(z+1)*H;for(let X=0;X<$;X++){let B=X*Y,F=(X+1)*Y,O=0,D=0,I=Math.min(Math.ceil(q),Z);for(let R=Math.floor(K);R<I;R++){let T=Math.min(R+1,q)-Math.max(R,K);if(T<=0)continue;let _=Math.min(Math.ceil(F),Q);for(let P=Math.floor(B);P<_;P++){let L=Math.min(P+1,F)-Math.max(P,B);if(L<=0)continue;let y=L*T;O+=y*J[R*Q+P],D+=y}}G[z*$+X]=D>0?Math.min(255,Math.max(0,Math.round(O/D))):0}}return G}import{deflateSync as yQ}from"node:zlib";//! Minimal grayscale PNG encoder (bit depth 8, color type 0), mirroring
|
|
15
|
+
//! pxpipe's png.ts: IHDR + one IDAT (zlib) + IEND, filter byte 0 per row.
|
|
16
|
+
var wQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $=0;$<8;$++)Z=(Z&1)!==0?3988292384^Z>>>1:Z>>>1;J[Q]=Z>>>0}return J})();function qJ(J,Q,Z,$){let V=$.length;J[Q]=V>>>24&255,J[Q+1]=V>>>16&255,J[Q+2]=V>>>8&255,J[Q+3]=V&255;let G=Q+4;for(let z=0;z<4;z++)J[G+z]=Z.charCodeAt(z);J.set($,G+4);let Y=4294967295;for(let z=G;z<G+4+V;z++)Y=wQ[(Y^J[z])&255]^Y>>>8;Y=~Y>>>0;let H=G+4+V;return J[H]=Y>>>24&255,J[H+1]=Y>>>16&255,J[H+2]=Y>>>8&255,J[H+3]=Y&255,H+4}function yJ(J,Q,Z){let $=new Uint8Array(Z*(Q+1));for(let z=0,K=0,q=0;z<Z;z++)$[K++]=0,$.set(J.subarray(q,q+Q),K),K+=Q,q+=Q;let V=yQ($,{level:6}),G=new Uint8Array(33+(12+V.length)+12);G[0]=137,G[1]=80,G[2]=78,G[3]=71,G[4]=13,G[5]=10,G[6]=26,G[7]=10;let Y=new Uint8Array(13);Y[0]=Q>>>24&255,Y[1]=Q>>>16&255,Y[2]=Q>>>8&255,Y[3]=Q&255,Y[4]=Z>>>24&255,Y[5]=Z>>>16&255,Y[6]=Z>>>8&255,Y[7]=Z&255,Y[8]=8,Y[9]=0;let H=qJ(G,8,"IHDR",Y);return H=qJ(G,H,"IDAT",V),qJ(G,H,"IEND",new Uint8Array(0)),G}//! Stage 2: the `pxpipe` imaging engine, ported from pxpipe's render.ts
|
|
17
|
+
//! production dense path (bare 5x8 AA cell, 312 cols, 1568x728 pages).
|
|
18
|
+
//! Glyphs cover BMP (pxpipe's atlas: Spleen for ASCII/code, Unifont fallback)
|
|
19
|
+
//! PLUS the astral planes (unifont_upper, incl. emoji) — beyond pxpipe, which
|
|
20
|
+
//! drops astral. Only unassigned codepoints fall back to `▯` and are counted
|
|
21
|
+
//! as dropped.
|
|
22
|
+
//!
|
|
23
|
+
//! Two tanuki-only extensions over the faithful port, both behind knobs so the
|
|
24
|
+
//! `pack=false, font=Normal` path stays byte-identical to pxpipe (parity):
|
|
25
|
+
//! * `pack` — lossless reflow tighter than pxpipe: single-cell tabs (no
|
|
26
|
+
//! 4-col padding) + indent run-length (`⇥N`), plus per-page
|
|
27
|
+
//! width-trim so short payloads stop paying for 1568px rows.
|
|
28
|
+
//! * `font` — `Tiny` renders the same atlas box-filtered into a 4x6 cell
|
|
29
|
+
//! (390 cols x 120 rows/page), ~40% fewer image-tokens; opt-in,
|
|
30
|
+
//! transcription-accuracy gated.
|
|
31
|
+
var p=4,a=4,bQ=1568,vQ=728,t="↵",xQ="⏎",s="→",hQ="⇢",UJ="⇥",kQ="⇨",fQ=9647,wJ=4,mQ=3,bJ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",uQ=/\n{4,}/g;function OJ(J){return J.length===4&&J.replace(/[A-Z]/g,(Q)=>Q.toLowerCase())==="tiny"?"tiny":"normal"}function kJ(J){let Q=J==="tiny"?4:w,Z=J==="tiny"?6:d,$=Math.floor((bQ-2*p)/Q),V=Math.floor((vQ-2*a)/Z);return{cw:Q,ch:Z,cols:$,maxLines:V,maxChars:$*V}}var gQ=(()=>{let J=new Uint8Array(128);for(let Q=0;Q<128;Q++){let Z=l(Q);J[Q]=Z>=0&&k(Z)?2:1}return J})();function FJ(J){if(J<128)return gQ[J];let Q=l(J);return Q>=0&&k(Q)?2:1}function dQ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<56320&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<57344)Z++}Q++}return Q}function DJ(J){let Q=J.split(`
|
|
32
|
+
`);for(let Z=0;Z<Q.length;Z++){let $=Q[Z],V=$.length;while(V>0){let G=$.charCodeAt(V-1);if(G!==32&&G!==9)break;V--}if(V!==$.length)Q[Z]=$.slice(0,V)}return Q.join(`
|
|
33
|
+
`).replace(uQ,`
|
|
34
|
+
|
|
35
|
+
|
|
36
|
+
`)}function fJ(J){if(!J.includes("\t"))return J;let Q="",Z=0;for(let $ of J)if($==="\t"){let V=wJ-Z%wJ;Q+=s;for(let G=1;G<V;G++)Q+=" ";Z+=V}else Q+=$,Z+=FJ($.codePointAt(0));return Q}function lQ(J){let Q=J.includes("\t")?J.replaceAll("\t",s):J,Z=0;while(Z<Q.length&&Q.charCodeAt(Z)===32)Z++;if(Z>=mQ&&Z<bJ.length)return UJ+bJ[Z]+Q.slice(Z);return Q}function mJ(J){return J.includes(t)?J.replaceAll(t,xQ):J}function pQ(J){let Q=mJ(J);if(Q.includes(s))Q=Q.replaceAll(s,hQ);if(Q.includes(UJ))Q=Q.replaceAll(UJ,kQ);return Q}function nQ(J){let Q=DJ(J).split(`
|
|
37
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=fJ(Q[Z]);return Q.join(t)}function cQ(J){let Q=DJ(J).split(`
|
|
38
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=lQ(Q[Z]);return Q.join(t)}function oQ(J,Q){let Z=[],$=DJ(J);for(let V of $.split(`
|
|
39
|
+
`)){let G=fJ(V);if(G.length===0){Z.push("");continue}let Y="",H=0;for(let z of G){let K=FJ(z.codePointAt(0));if(H+K>Q)Z.push(Y),Y=z,H=K;else Y+=z,H+=K}if(Y.length!==0)Z.push(Y)}return Z}function iQ(J,Q,Z){let $=[],V=[],G=0;for(let Y of J){let H=dQ(Y),z=H+(V.length!==0?1:0);if(V.length!==0&&(V.length>=Q||G+z>Z))$.push(V),V=[],G=0;G+=H+(V.length!==0?1:0),V.push(Y)}if(V.length!==0)$.push(V);return $}function uJ(J){if(J<32)return!0;if(J>=127&&J<=159)return!0;if(J>=768&&J<=879)return!0;if(J===8203||J===8204||J===8205||J===8288||J===65279)return!0;if(J>=65024&&J<=65039)return!0;if(J>=917760&&J<=917999)return!0;return!1}function rQ(J){let Q=null,Z=0;for(let $ of J){let V=$.codePointAt(0);if(l(V)<0&&!uJ(V)){if(Q===null)Q=J.slice(0,Z);Q+=`[U+${V.toString(16).toUpperCase()}]`}else if(Q!==null)Q+=$;Z+=$.length}return Q??J}function gJ(J,Q,Z,$){let V=Q?Z?cQ(pQ(J)):nQ(mJ(J)):J;return iQ(oQ(rQ(V),$.cols),$.maxLines,$.maxChars)}function dJ(J,Q,Z){if(!Z)return 2*p+Q.cols*Q.cw;let $=0;for(let Y of J){let H=0;for(let z of Y)H+=FJ(z.codePointAt(0));if(H>Q.cols)H=Q.cols;if(H>$)$=H}let V=2*p+$*Q.cw,G=2*p+Q.cw;return V>G?V:G}var vJ=new Map;function xJ(J,Q,Z,$,V,G,Y){let H=l(V);if(H<0)return 0;let z=k(H),K=z?2*G.cw:G.cw,q;if(Y==="normal")q=BJ(H);else{let X=vJ.get(H);if(X===void 0)X=LJ(H,K,G.ch),vJ.set(H,X);q=X}for(let X=0;X<G.ch;X++){let B=($+X)*Q+Z,F=X*K;for(let O=0;O<K;O++){let D=q[F+O];if(D>0){let I=B+O;if(D>J[I])J[I]=D}}}return z?2:1}function aQ(J,Q,Z,$){let V=dJ(J,Q,Z),G=2*a+J.length*Q.ch,Y=new Uint8Array(V*G),H=0;for(let z=0;z<J.length;z++){let K=a+z*Q.ch,q=0;for(let X of J[z]){if(q>=Q.cols)break;let B=p+q*Q.cw,F=xJ(Y,V,B,K,X.codePointAt(0),Q,$);if(F===0){if(F=1,uJ(X.codePointAt(0)));else if(H++,X!==" ")xJ(Y,V,B,K,fQ,Q,$)}q+=F}}for(let z=0;z<Y.length;z++)Y[z]=255-Y[z];return{png:yJ(Y,V,G),width:V,height:G,dropped:H}}function MJ(J,Q,Z,$){let V=kJ($),G=gJ(J,Q,Z,V).map((K)=>aQ(K,V,Z,$)),Y=0,H=0,z=0;for(let K of G)Y+=K.width*K.height,H+=pJ(K.width,K.height),z+=K.dropped;return{pages:G,pixels:Y,tokens:H,dropped:z}}function lJ(J,Q,Z,$){let V=kJ($),G=gJ(J,Q,Z,V),Y=0,H=0;for(let z of G){let K=dJ(z,V,Z),q=2*a+z.length*V.ch;Y+=K*q,H+=pJ(K,q)}return{pages:G.length,pixels:Y,tokens:H}}var hJ=28;function pJ(J,Q){return Math.ceil(J/hJ)*Math.ceil(Q/hJ)}import{readFileSync as tQ}from"node:fs";import{join as sQ}from"node:path";//! pxpipe measurement-log summary (~/.pxpipe/events.jsonl), same math as the
|
|
40
|
+
//! node MCP: actual = every way input bytes get billed (input + cache reads +
|
|
41
|
+
//! cache creates) — ignoring cache_read would fake the savings.
|
|
42
|
+
class e{value;constructor(J){this.value=J}}function nJ(){let J=process.env.TANUKI_EVENTS;if(J!==void 0)return J;let Q=process.env.HOME??"";return sQ(Q,".pxpipe","events.jsonl")}function f(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:0}function eQ(J){return J<0?-Math.round(-J):Math.round(J)}function WJ(){let J=nJ(),Q;try{Q=tQ(J,"utf8")}catch{return{available:!1,note:`no ${J} yet`}}let Z=0,$=0,V=0,G=0,Y=0,H=0;for(let K of Q.split(`
|
|
43
|
+
`)){if(K.trim().length===0)continue;let q;try{q=JSON.parse(K)}catch{continue}Z+=1;let X=q!==null&&typeof q==="object"&&!Array.isArray(q)?q:{};if(X.compressed===!0)$+=1,V+=f(X.orig_chars),G+=f(X.image_count);Y+=f(X.baseline_tokens),H+=f(X.input_tokens)+f(X.cache_read_tokens)+f(X.cache_create_tokens)}let z=Y>0&&H>0?new e(eQ((1-H/Y)*1000)/10):null;return{available:!0,requests:Z,compressedRequests:$,imagedChars:V,imagesEmitted:G,baselineTokens:Y,actualInputTokens:H,estInputSavedPct:z}}//! Implicit mode: a local Anthropic middlebox, the pxpipe deployment shape
|
|
44
|
+
//! without pxpipe's structural flaw. Rules that keep it injection-shaped-free:
|
|
45
|
+
//!
|
|
46
|
+
//! 1. The system prompt and tool definitions are NEVER touched.
|
|
47
|
+
//! 2. Nothing moves between roles or positions: an oversized text block is
|
|
48
|
+
//! replaced IN PLACE by a short overt marker + PNG page blocks, in the
|
|
49
|
+
//! same user-role message (Anthropic allows image blocks in user content
|
|
50
|
+
//! and inside tool_result content).
|
|
51
|
+
//! 3. The latest message is never imaged (the model may need to quote it).
|
|
52
|
+
//! 4. Blocks carrying cache_control are never touched (rewriting would
|
|
53
|
+
//! defeat the cache they exist for).
|
|
54
|
+
//! 5. Imaging only happens when `estimate` says it wins by a clear margin;
|
|
55
|
+
//! everything else passes through byte-for-byte.
|
|
56
|
+
//!
|
|
57
|
+
//! Responses stream through untouched; usage is scraped from the stream for
|
|
58
|
+
//! the ~/.pxpipe/events.jsonl savings log (same format tanuki_stats reads).
|
|
59
|
+
//! tanuki-context — token-cutting context pipeline.
|
|
60
|
+
//! pipeline: text -> distill (stage 0, logs) -> ladder level 0-4 (stage 1)
|
|
61
|
+
//! -> pxpipe imaging (stage 2, name kept from the original mechanic)
|
|
62
|
+
//!
|
|
63
|
+
//! Default: MCP stdio server (newline-delimited JSON-RPC 2.0).
|
|
64
|
+
//! CLI: tanuki-context distill <file> [query]
|
|
65
|
+
//! tanuki-context estimate <file> [level] [--distill]
|
|
66
|
+
//! tanuki-context render <file> [level] [outdir]
|
|
67
|
+
//! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
|
|
68
|
+
var cJ="0.1.0",jJ=6;function JZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function C(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof e){let z=J.value;return Number.isFinite(z)&&Number.isInteger(z)?z.toFixed(1):String(z)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let q="[";for(let X=0;X<J.length;X++){if(X>0)q+=",";q+=C(J[X],!1)}return q+"]"}let z=Z+" ",K=`[
|
|
69
|
+
`;for(let q=0;q<J.length;q++){if(q>0)K+=`,
|
|
70
|
+
`;K+=z+C(J[q],!0,z)}return K+`
|
|
71
|
+
`+Z+"]"}let V=J,G=Object.keys(V).sort(JZ);if(G.length===0)return"{}";if(!Q){let z="{";for(let K=0;K<G.length;K++){if(K>0)z+=",";z+=JSON.stringify(G[K])+":"+C(V[G[K]],!1)}return z+"}"}let Y=Z+" ",H=`{
|
|
72
|
+
`;for(let z=0;z<G.length;z++){if(z>0)H+=`,
|
|
73
|
+
`;H+=Y+JSON.stringify(G[z])+": "+C(V[G[z]],!0,Y)}return H+`
|
|
74
|
+
`+Z+"}"}function S(J,Q){return J!==null&&typeof J==="object"&&!Array.isArray(J)?J[Q]:void 0}function m(J){return typeof J==="string"?J:null}function JJ(J){return typeof J==="boolean"?J:null}function oJ(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:null}function u(J){let Q=0;for(let Z=0;Z<J.length;Z++){Q++;let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}}return Q}function iJ(J,Q,Z,$,V){let G,Y=null;if(Z||$!==null){let K=c(J,$,2);G=K.distilled,Y=K.stats}else G=J;let H=0;if(V){let K=VJ(G);G=K.text,H=K.entries}let z=i(G,Q);return{stage0:Y,compressed:z.compressed,protectedLines:z.protectedLines,level:z.level,cbEntries:H}}function QJ(J){return Math.round(J/4)}function n(J,Q){if(J===0)return 0;let Z=(1-Q/J)*100;return Z<0?-Math.round(-Z):Math.round(Z)}function rJ(J){return{text:m(S(J,"text"))??"",level:(oJ(S(J,"level"))??0)%256,distill:JJ(S(J,"distill"))??!1,query:m(S(J,"query")),reflow:JJ(S(J,"reflow"))??!0,pack:JJ(S(J,"pack"))??!0,font:m(S(J,"font"))??"normal",codebook:JJ(S(J,"codebook"))??!1}}function aJ(J){let Q=rJ(J),Z=iJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=OJ(Q.font),V=lJ(Z.compressed,Q.reflow,Q.pack,$),G=V.tokens,Y=u(Q.text),H=u(Z.compressed),z=QJ(Y),[K,q]=o[Z.level];return{engine:"pxpipe",level:`${Z.level} ${K}`,loss:q,distill:Z.stage0,origChars:Y,stage1Chars:H,stage1SavedPct:n(Y,H),pages:V.pages,imageTokens:G,rawTextTokens:z,totalSavedPct:n(z,G),protectedLines:Z.protectedLines,pack:Q.pack,font:$==="tiny"?"tiny":"normal",codebook:Q.codebook?Z.cbEntries:!1,verdict:G<z?"PIPELINE cheaper":"TEXT cheaper"}}function tJ(J){let Q=rJ(J),Z=iJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=OJ(Q.font),V=MJ(Z.compressed,Q.reflow,Q.pack,$),G=V.tokens,Y=u(Q.text),H=u(Z.compressed),z=QJ(Y),[K,q]=o[Z.level],X="";if(Z.stage0!==null){let O=Z.stage0;X+=`distill: ${O.origLines} -> ${O.outLines} lines (-${O.savedPct}% chars, ${O.collapsedRuns} runs, ${O.suppressedLines} exact + ${O.templateSuppressed} template suppressed, ${O.importantKept} error/warn kept)
|
|
75
|
+
`}if(X+=`L${Z.level} ${K} (${q}): ${Y} chars -> ${H} chars (stage1 -${n(Y,H)}%) -> ${V.pages.length} page(s), ~${G} image-tokens
|
|
76
|
+
vs ~${z} text-tokens raw = TOTAL -${n(z,G)}%`,Z.protectedLines>0)X+=` · ${Z.protectedLines} lines kept verbatim`;if(V.dropped>0)X+=` · ${V.dropped} unmapped glyphs -> ▯`;if(Z.cbEntries>0)X+=` · codebook: ${Z.cbEntries} sigils (see ·legend·)`;if($==="tiny")X+=" · font: tiny 4x6";if(Q.pack)X+=" · packed (⇥N indent, → tab)";if(Q.reflow)X+=" · ↵ = newline · engine: pxpipe";let B=[{type:"text",text:X}],F=Math.min(V.pages.length,jJ);for(let O=0;O<F;O++){let D=V.pages[O].png;B.push({type:"image",data:Buffer.from(D.buffer,D.byteOffset,D.byteLength).toString("base64"),mimeType:"image/png"})}if(V.pages.length>jJ)B.push({type:"text",text:`(+${V.pages.length-jJ} more page(s))`});return B}function sJ(J){let Q=m(S(J,"text"))??"",Z=c(Q,m(S(J,"query")),2);return[{type:"text",text:C(Z.stats,!0)},{type:"text",text:Z.distilled}]}function eJ(J){let Q=m(S(J,"text"))??"",Z=(oJ(S(J,"level"))??1)%256,$=i(Q,Z),[V,G,Y]=o[$.level],H=u(Q),z=u($.compressed),K=QJ(H),q=QJ(z),X={level:`${$.level} ${V}`,loss:G,note:Y,origChars:H,outChars:z,approxOrigTokens:K,approxOutTokens:q,savedPct:n(K,q),protectedLines:$.protectedLines};return[{type:"text",text:C(X,!0)},{type:"text",text:$.compressed}]}//! Claude Agent SDK integration (`tanuki-context/agent`).
|
|
77
|
+
//!
|
|
78
|
+
//! Two flavours, both one-liners from an agent's point of view:
|
|
79
|
+
//!
|
|
80
|
+
//! external (subprocess per session, works everywhere):
|
|
81
|
+
//! import { query } from "@anthropic-ai/claude-agent-sdk";
|
|
82
|
+
//! import { withTanuki } from "tanuki-context/agent";
|
|
83
|
+
//! for await (const m of query({ prompt, options: withTanuki() })) { ... }
|
|
84
|
+
//!
|
|
85
|
+
//! in-process (one process shared by a whole team of agents):
|
|
86
|
+
//! import { tanukiSdkServer, tanukiAllowedTools, TANUKI_INSTRUCTIONS } from "tanuki-context/agent";
|
|
87
|
+
//! const tanuki = await tanukiSdkServer();
|
|
88
|
+
//! query({ prompt, options: { mcpServers: { tanuki }, allowedTools: tanukiAllowedTools() } })
|
|
89
|
+
//!
|
|
90
|
+
//! The core package stays zero-dependency: the SDK and zod are touched only
|
|
91
|
+
//! inside `tanukiSdkServer()` via dynamic import, and both are already present
|
|
92
|
+
//! in any Agent SDK project (zod is the SDK's own peer dependency).
|
|
93
|
+
var $Z=["tanuki_render","tanuki_estimate","tanuki_distill","tanuki_compress","tanuki_stats"],VZ=`tanuki-context turns bulky text (logs, command output, docs) into dense PNG pages that cost a fraction of the text tokens.
|
|
94
|
+
Workflow: call tanuki_estimate first (instant, exact, never renders pixels). If the verdict says "PIPELINE cheaper", call tanuki_render with the same arguments and use the returned pages instead of pasting the text.
|
|
95
|
+
For logs, pass distill:true (repeats collapse, error/warn lines stay verbatim; add query:"regex" to slice). For prose you will not quote verbatim, level 2-3 shrinks it further. codebook:true helps path-heavy logs. Never image content you must quote byte-exact at level 4 or font tiny.
|
|
96
|
+
Pages decode as: ↵ = newline, → = tab, ⇥N = N leading spaces, a trailing ·legend· line maps sigils back to full tokens.`;function GZ(){let J=new URL("./cli.js",import.meta.url);if(J.protocol==="file:"){let Q=ZZ(J);if(QZ(Q))return{type:"stdio",command:process.execPath,args:[Q]}}return{type:"stdio",command:"npx",args:["-y","tanuki-context"]}}function YZ(J="tanuki"){return $Z.map((Q)=>`mcp__${J}__${Q}`)}function pZ(J,Q={}){let Z=J??{},$=Q.key??"tanuki";return{...Z,mcpServers:{...Z.mcpServers??{},[$]:Q.server??GZ()},allowedTools:[...Z.allowedTools??[],...YZ($)]}}function HZ(J){let Q=J.string().describe("the bulky text to process"),Z=J.number().int().min(0).max(4).optional().describe("ladder level 0-4 (default 0)"),$={text:Q,level:Z,distill:J.boolean().optional().describe("stage 0 log distiller"),query:J.string().optional().describe("distill: keep matching lines +context"),reflow:J.boolean().optional().describe("pack short lines into full rows (default true)"),pack:J.boolean().optional().describe("indent RLE + width trim, lossless (default true)"),font:J.enum(["normal","tiny"]).optional().describe("tiny = 4x6 cells, ~40% fewer tokens, gated"),codebook:J.boolean().optional().describe("repeated tokens/paths -> sigils + legend")},V=(Y)=>({content:Y}),G=async(Y)=>{try{return V(Y())}catch(H){return{content:[{type:"text",text:`tanuki-context error: ${H instanceof Error?H.message:String(H)}`}],isError:!0}}};return[{name:"tanuki_render",description:"Render text through the pipeline (optional distill/level/codebook) into dense PNG pages. Call after tanuki_estimate says PIPELINE cheaper.",inputSchema:$,handler:(Y)=>G(()=>tJ(Y))},{name:"tanuki_estimate",description:"Exact page/token math for the same arguments as tanuki_render, without touching pixels. Instant; call this first.",inputSchema:$,handler:(Y)=>G(()=>[{type:"text",text:C(aJ(Y),!0)}])},{name:"tanuki_distill",description:"Stage 0 alone: collapse repeated log lines/blocks and template near-dupes; error/warn lines kept verbatim. Output stays greppable text.",inputSchema:{text:Q,query:$.query},handler:(Y)=>G(()=>sJ(Y))},{name:"tanuki_compress",description:"Stage 1 alone: graded text compression, levels 0-4, code/paths/hashes protected from level 2 up.",inputSchema:{text:Q,level:Z},handler:(Y)=>G(()=>eJ(Y))},{name:"tanuki_stats",description:"Session savings summary from the events log (honest denominator: input + cache reads + cache creates).",inputSchema:{},handler:()=>G(()=>[{type:"text",text:C(WJ(),!0)}])}]}async function nZ(){let J,Q;try{J=await import("@anthropic-ai/claude-agent-sdk");let $=await import("zod");Q=$.z??$}catch($){throw Error(`tanukiSdkServer() needs the host project's @anthropic-ai/claude-agent-sdk and zod (npm i @anthropic-ai/claude-agent-sdk zod). Import failed: ${$ instanceof Error?$.message:String($)}`)}let Z=HZ(Q).map(($)=>J.tool($.name,$.description,$.inputSchema,(V)=>$.handler(V)));return J.createSdkMcpServer({name:"tanuki-context",version:cJ,instructions:VZ,tools:Z})}export{pZ as withTanuki,HZ as tanukiSdkToolSpecs,nZ as tanukiSdkServer,GZ as tanukiMcpServer,YZ as tanukiAllowedTools,$Z as TANUKI_TOOL_NAMES,VZ as TANUKI_INSTRUCTIONS};
|
package/dist/cli.js
CHANGED
|
@@ -1,20 +1,20 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import{readFileSync as
|
|
3
|
-
·legend· `;for(let
|
|
4
|
-
`),Y=z.length,G=Y,
|
|
5
|
-
`),
|
|
6
|
-
`),z=Array(V.length);for(let G=0;G<V.length;G++){let
|
|
7
|
-
`).replace(
|
|
2
|
+
import{readFileSync as FZ,mkdirSync as MZ,writeFileSync as WZ}from"node:fs";import S from"node:process";function UQ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function OJ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}Q++}return Q}function XQ(J,Q){let Z=0,$=0;while(Z<J.length&&$<Q.length){let V=J.codePointAt(Z),z=Q.codePointAt($);if(V!==z)return V-z;Z+=V>65535?2:1,$+=z>65535?2:1}return J.length-Z-(Q.length-$)}function QJ(J){let Q=new Map,Z=(X)=>{Q.set(X,(Q.get(X)??0)+1)},$=(X)=>{if(OJ(X)>=12)Z(X);if(X.includes("/")){let O=X.split("/"),q="";for(let F=0;F<O.length;F++){if(F>0)q+="/";if(q+=O[F],F>=2){let R=q+"/";if(OJ(R)>=12)Z(R)}}}},V=-1;for(let X=0;X<J.length;){let O=J.codePointAt(X),q=O>65535?2:1;if(UQ(O)){if(V>=0)$(J.slice(V,X)),V=-1}else if(V<0)V=X;X+=q}if(V>=0)$(J.slice(V));let z=[];for(let[X,O]of Q)if(O>=3){let q=OJ(X);z.push({k:X,c:O,len:q,saved:(q-1)*O})}z.sort((X,O)=>O.saved-X.saved||XQ(X.k,O.k));let Y=[];for(let X of"§¤¢£¥µ¶ª°±¬×÷ØÞßæðøþ¡¿")if(!J.includes(X))Y.push(X);let G=[],H=[];for(let{k:X,c:O,len:q}of z){if(G.length>=Y.length)break;if((q-1)*O<=q+3)continue;let F=!1;for(let A=0;A<H.length;A++){let y=H[A];if(y.startsWith(X)||X.startsWith(y)){F=!0;break}}if(F)continue;let R=Y[G.length];H.push(X),G.push({sig:R,val:X,len:q})}if(G.length===0)return{text:J,entries:0};let K=G.map((X,O)=>O);K.sort((X,O)=>G[O].len-G[X].len);let B=J;for(let X=0;X<K.length;X++){let{sig:O,val:q}=G[K[X]];B=B.replaceAll(q,O)}let U=`
|
|
3
|
+
·legend· `;for(let X=0;X<G.length;X++)U+=G[X].sig+"="+G[X].val+" ";return B+=U.slice(0,-1),{text:B,entries:G.length}}import{Buffer as LJ}from"node:buffer";var BQ=/\x1b\[[0-9;]*[A-Za-z]/g,qJ=/\b([0-9A-Za-z_]*(error|exception)s?|err|warn(ing)?s?|fail(s|ed|ure|ures)?|panic(s|ked)?|fatal|critical|traceback|denied|refused|timeouts?|timed.?out|assert(s|ed|ion|ions)?|segfault(s|ed)?)\b/i,OQ=/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/g,qQ=/\b[0-9]{2}:[0-9]{2}:[0-9]{2}([.,][0-9]+)?\b/g,DQ=/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/gi,FQ=/\b[0-9a-f]{7,64}\b/gi,MQ=/\b[0-9]+(\.[0-9]+)?[ \t\u00a0]?(ms|us|µs|ns|s|m|h|%|[KMGT]i?B)?\b/gi,WQ=/[0-9a-f]/i,NQ=3,jQ=8,DJ=2,yJ=40;function IQ(J){if(!WQ.test(J))return J;let Q=J.replace(OQ,"<ts>");return Q=Q.replace(qQ,"<time>"),Q=Q.replace(DQ,"<uuid>"),Q=Q.replace(FQ,"<hex>"),Q.replace(MQ,"<n>")}function g(J){if(J===32)return!0;if(J<9)return!1;if(J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function wJ(J){let Q=0,Z=J.length;while(Q<Z&&g(J.charCodeAt(Q)))Q++;while(Z>Q&&g(J.charCodeAt(Z-1)))Z--;return Q===0&&Z===J.length?J:J.slice(Q,Z)}function RQ(J,Q){let Z=0,$=J.length;while(Z<$&&g(J.charCodeAt(Z)))Z++;while($>Z&&g(J.charCodeAt($-1)))$--;let V=0;for(let z=Z;z<$&&V<Q;V++){let Y=J.charCodeAt(z);z+=Y>=55296&&Y<=56319&&z+1<$?2:1}return V<Q}function AQ(J){let Q=0,Z=J.length;while(Q<Z&&g(J.charCodeAt(Q)))Q++;return J.charCodeAt(Q)===91&&J.charCodeAt(Q+1)===215}function PQ(J){let Q=[],Z=-1,$=!0,V=!1,z=!1,Y=!0;for(let G=0;G<J.length;){let H=J.codePointAt(G),K=H>65535?2:1,B=g(H);if(Y){if(Y=!1,B)Q.push("<v>")}if(B){if(Z>=0)Q.push($?J.slice(Z,G):"<v>"),Z=-1;z=!0}else{if(V=!0,z=!1,Z<0)Z=G,$=!0;if($){let U=H|32;if(U<97||U>122||H>127)$=!1}}G+=K}if(Z>=0)Q.push($?J.slice(Z):"<v>");if(V&&z)Q.push("<v>");return Q.join(" ")}function bJ(J,Q){let Z=0,$=0,V=J.length;while(Z<V&&$<Q){let z=J.charCodeAt(Z);Z+=z>=55296&&z<=56319&&Z+1<V?2:1,$++}return Z>=V?J:J.slice(0,Z)}var _Q=/[.*+?^${}()|[\]\\]/g;function m(J,Q=null,Z=2){let $=Z,z=J.replace(BQ,"").split(`
|
|
4
|
+
`),Y=z.length,G=Y,H=Array(Y),K=Array(Y),B=0;for(let D=0;D<Y;D++){H[D]=IQ(z[D]);let M=qJ.test(z[D]);if(K[D]=M,M)B++}let U=[],X=[],O=[],q=0,F=0;while(F<Y){let D=0,M=0;for(let W=1;W<=jQ&&F+2*W<=Y;W++){if(K[F+W-1])break;let j=1;J:for(;;){let N=F+j*W;if(N+W>Y)break;for(let _=0;_<W;_++)if(K[N+_]||H[N+_]!==H[F+_])break J;j++}if(j>=NQ&&W*(j-1)>D*(M>0?M-1:0))D=W,M=j}if(D>0){for(let W=F;W<F+D;W++)U.push(z[W]),X.push(K[W]),O.push(H[W]);U.push(D===1?` [×${M} similar]`:` [×${M} similar ${D}-line blocks]`),X.push(!1),O.push(null),q++,F+=D*M}else U.push(z[F]),X.push(K[F]),O.push(H[F]),F++}let R=new Map,A=new Map,y=DJ+1,T=[],P=[],w=0,b=0;for(let D=0;D<U.length;D++){let M=U[D],W=X[D];if(W||AQ(M)||RQ(M,4)){T.push(M),P.push(W);continue}let j=O[D],N=R.get(j);if(N!==void 0){if(N.count++,N.count<=DJ)T.push(M),P.push(!1);else w++;continue}let _=PQ(j);R.set(j,{count:1,exemplar:M});let c=A.get(_);if(c!==void 0)if(c.count++,c.count<=y)T.push(M),P.push(!1);else b++;else A.set(_,{count:1,exemplar:M}),T.push(M),P.push(!1)}let h=[];for(let D of R.values())if(D.count>DJ)h.push([D.count,"exact",D.exemplar]);for(let D of A.values())if(D.count>y)h.push([D.count,"template",D.exemplar]);if(h.sort((D,M)=>M[0]-D[0]),h.length>yJ)h.length=yJ;if(w+b>0){let D=`── ${w+b} repeated lines suppressed (${w} exact ×N, ${b} same-template; first occurrences kept above) ──`;T.push(D),P.push(qJ.test(D));for(let[M,W,j]of h){let _=` ×${M}${W==="template"?" (template)":""} ${bJ(wJ(j),160)}`;T.push(_),P.push(qJ.test(_))}}let f;if(Q!=null){let D;try{D=new RegExp(Q,"i")}catch{D=new RegExp(Q.replace(_Q,"\\$&"),"i")}let M=T.length,W=new Uint8Array(M);for(let N=0;N<M;N++)if(P[N]||D.test(T[N])){let _=N>$?N-$:0,c=Math.min(N+$,M-1);W.fill(1,_,c+1)}f=[];let j=0;for(let N=0;N<M;N++)if(W[N]){if(j>0)f.push(`… ${j} lines omitted`),j=0;f.push(T[N])}else j++;if(j>0)f.push(`… ${j} lines omitted`)}else f=T;let EJ=f.join(`
|
|
5
|
+
`),TJ=LJ.byteLength(J),CJ=LJ.byteLength(EJ),BJ=(1-CJ/TJ)*100,YQ=J.length===0?0:BJ<0?-Math.round(-BJ):Math.round(BJ),HQ=h.map(([D,M,W])=>({count:D,exemplar:bJ(wJ(W),160),kind:M})),KQ={collapsedRuns:q,importantKept:B,origChars:TJ,origLines:G,outChars:CJ,outLines:f.length,query:Q??null,savedPct:YQ,suppressedLines:w,templateSuppressed:b,topRepeats:HQ};return{distilled:EJ,stats:KQ}}var ZJ=[["none","none","passthrough (baseline)"],["whitespace","lossless","trailing whitespace + blank-line runs collapsed; safe for code"],["prose","light","L1 + prose lines: collapse spaces, cut redundant filler phrases (code/IDs protected)"],["dense","medium","L2 + prose: drop articles & intensifiers"],["caveman","heavy","L3 + prose: telegraphic — drop function words; gist only, NOT verbatim"]];var C=(J)=>new RegExp(`(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:${J})(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])`,"giu"),FJ=[[C("in order to"),"to"],[C("due to the fact that"),"because"],[C("at this point in time"),"now"],[C("in the event that"),"if"],[C("for the purpose of"),"for"],[C("with regard to"),"about"],[C("a large number of"),"many"],[C("it is important to note that"),""],[C("please note that"),""],[C("as a matter of fact"),""],[C("in terms of"),"for"],[C("the fact that"),"that"]],SQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:the|an|a)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),EQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:very|really|just|actually|basically|simply|quite|rather|essentially|literally)[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+","giu"),TQ=new RegExp("(?<![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])(?:is|are|was|were|am|be|been|being|do|does|did|have|has|had|will|would|shall|should|can|could|may|might|of|to|in|on|at|for|with|that|this|these|those|it|its|there|here)(?![\\p{Alphabetic}\\p{Mark}\\p{Decimal_Number}\\p{Connector_Punctuation}\\p{Join_Control}])[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]*","giu"),vJ=/ {2,}/g,CQ=new RegExp("[\\t\\n\\x0B\\f\\r \\u0085\\u00A0\\u1680\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]+([.,;:!?])","gu"),LQ=/\n{3,}/g;function MJ(J){if(J===32||J>=9&&J<=13)return!0;if(J<133)return!1;return J===133||J===160||J===5760||J>=8192&&J<=8202||J===8232||J===8233||J===8239||J===8287||J===12288}function yQ(J){let Q=0,Z=J.length;while(Q<Z){let $=J.codePointAt(Q);if(!MJ($))break;Q+=$>65535?2:1}while(Z>Q){let $=J.charCodeAt(Z-1);if(!MJ($))break;Z--}return Q===0&&Z===J.length?J:J.slice(Q,Z)}function wQ(J){if(J.length===0)return!1;let Q=J.charCodeAt(0);if(Q===32||Q===9)return!0;let Z=0,$=0,V=0,z=!1;for(let Y=0;Y<J.length;){let G=J.codePointAt(Y);if(Y+=G>65535?2:1,Z++,MJ(G)){V=0;continue}if(V++,V>=24)z=!0;if(G>=48&&G<=57||G>=65&&G<=90||G>=97&&G<=122)continue;switch(G){case 46:case 44:case 59:case 58:case 39:case 34:case 33:case 63:case 40:case 41:case 45:continue}$++}if($/Z>0.3)return!0;return z}function bQ(J,Q){let Z=J.replace(vJ," ");for(let V=0;V<FJ.length;V++)Z=Z.replace(FJ[V][0],FJ[V][1]);if(Q>=3)Z=Z.replace(SQ,""),Z=Z.replace(EQ,"");if(Q>=4)Z=Z.replace(TQ,"");Z=Z.replace(vJ," "),Z=yQ(Z.replace(CQ,"$1"));let $=Z.charCodeAt(0);if($>=97&&$<=122)Z=String.fromCharCode($-32)+Z.slice(1);return Z}function o(J,Q){let Z=Math.min(Q,4);if(Z===0)return{compressed:J,protectedLines:0,level:Z};let $=0,V=J.split(`
|
|
6
|
+
`),z=Array(V.length);for(let G=0;G<V.length;G++){let H=V[G],K=H.length;while(K>0){let U=H.charCodeAt(K-1);if(U!==32&&U!==9)break;K--}let B=K===H.length?H:H.slice(0,K);if(Z===1){z[G]=B;continue}if(wQ(B)){$++,z[G]=B;continue}z[G]=bQ(B,Z)}return{compressed:z.join(`
|
|
7
|
+
`).replace(LQ,`
|
|
8
8
|
|
|
9
|
-
`),protectedLines:$,level:
|
|
9
|
+
`),protectedLines:$,level:Z}}import{appendFileSync as GZ,mkdirSync as YZ}from"node:fs";import{dirname as HZ}from"node:path";import sJ from"node:http";import KZ from"node:https";import{URL as UZ}from"node:url";import{readFileSync as NJ}from"node:fs";import{inflateSync as vQ}from"node:zlib";//! Full-BMP glyph atlas (Spleen 5x8 for ASCII/code + Unifont fallback),
|
|
10
10
|
//! extracted from pxpipe's generated gray atlas by `tools/gen-glyphs.mjs`.
|
|
11
11
|
//!
|
|
12
12
|
//! Codepoints + wide flags load eagerly (~175 KB, needed for wrap math);
|
|
13
13
|
//! coverage pixels stay zlib-packed on disk and decompress lazily on the
|
|
14
14
|
//! first blit, so `estimate` never pays for them.
|
|
15
|
-
var v=5,
|
|
15
|
+
var v=5,i=8,u=NJ(new URL("../assets/glyphs.cps",import.meta.url)),$J=NJ(new URL("../assets/glyphs.wide",import.meta.url)),xJ=(()=>{let J=u.byteLength>>>2;if((u.byteOffset&3)===0)return new Uint32Array(u.buffer,u.byteOffset,J);return new Uint32Array(u.buffer.slice(u.byteOffset,u.byteOffset+(J<<2)))})(),xQ=(()=>{let J=new Uint32Array($J.length),Q=0;for(let Z=0;Z<$J.length;Z++)J[Z]=Q,Q+=($J[Z]===1?2*v:v)*i;return J})(),WJ=null;function r(J){let Q=0,Z=xJ.length-1;while(Q<=Z){let $=Q+Z>>>1,V=xJ[$];if(V<J)Q=$+1;else if(V>J)Z=$-1;else return $}return-1}function d(J){return $J[J]===1}function jJ(J){if(WJ===null)WJ=vQ(NJ(new URL("../assets/glyphs.pix.z",import.meta.url)));let Q=xQ[J],Z=d(J)?2*v:v;return WJ.subarray(Q,Q+Z*i)}function kJ(J,Q,Z){let $=jJ(J),V=d(J)?2*v:v;return kQ($,V,i,Q,Z)}function kQ(J,Q,Z,$,V){let z=new Uint8Array($*V),Y=Q/$,G=Z/V;for(let H=0;H<V;H++){let K=H*G,B=(H+1)*G;for(let U=0;U<$;U++){let X=U*Y,O=(U+1)*Y,q=0,F=0,R=Math.min(Math.ceil(B),Z);for(let A=Math.floor(K);A<R;A++){let y=Math.min(A+1,B)-Math.max(A,K);if(y<=0)continue;let T=Math.min(Math.ceil(O),Q);for(let P=Math.floor(X);P<T;P++){let w=Math.min(P+1,O)-Math.max(P,X);if(w<=0)continue;let b=w*y;q+=b*J[A*Q+P],F+=b}}z[H*$+U]=F>0?Math.min(255,Math.max(0,Math.round(q/F))):0}}return z}import{deflateSync as hQ}from"node:zlib";//! Minimal grayscale PNG encoder (bit depth 8, color type 0), mirroring
|
|
16
16
|
//! pxpipe's png.ts: IHDR + one IDAT (zlib) + IEND, filter byte 0 per row.
|
|
17
|
-
var
|
|
17
|
+
var fQ=(()=>{let J=new Uint32Array(256);for(let Q=0;Q<256;Q++){let Z=Q;for(let $=0;$<8;$++)Z=(Z&1)!==0?3988292384^Z>>>1:Z>>>1;J[Q]=Z>>>0}return J})();function IJ(J,Q,Z,$){let V=$.length;J[Q]=V>>>24&255,J[Q+1]=V>>>16&255,J[Q+2]=V>>>8&255,J[Q+3]=V&255;let z=Q+4;for(let H=0;H<4;H++)J[z+H]=Z.charCodeAt(H);J.set($,z+4);let Y=4294967295;for(let H=z;H<z+4+V;H++)Y=fQ[(Y^J[H])&255]^Y>>>8;Y=~Y>>>0;let G=z+4+V;return J[G]=Y>>>24&255,J[G+1]=Y>>>16&255,J[G+2]=Y>>>8&255,J[G+3]=Y&255,G+4}function hJ(J,Q,Z){let $=new Uint8Array(Z*(Q+1));for(let H=0,K=0,B=0;H<Z;H++)$[K++]=0,$.set(J.subarray(B,B+Q),K),K+=Q,B+=Q;let V=hQ($,{level:6}),z=new Uint8Array(33+(12+V.length)+12);z[0]=137,z[1]=80,z[2]=78,z[3]=71,z[4]=13,z[5]=10,z[6]=26,z[7]=10;let Y=new Uint8Array(13);Y[0]=Q>>>24&255,Y[1]=Q>>>16&255,Y[2]=Q>>>8&255,Y[3]=Q&255,Y[4]=Z>>>24&255,Y[5]=Z>>>16&255,Y[6]=Z>>>8&255,Y[7]=Z&255,Y[8]=8,Y[9]=0;let G=IJ(z,8,"IHDR",Y);return G=IJ(z,G,"IDAT",V),IJ(z,G,"IEND",new Uint8Array(0)),z}//! Stage 2: the `pxpipe` imaging engine, ported from pxpipe's render.ts
|
|
18
18
|
//! production dense path (bare 5x8 AA cell, 312 cols, 1568x728 pages).
|
|
19
19
|
//! Glyphs cover BMP (pxpipe's atlas: Spleen for ASCII/code, Unifont fallback)
|
|
20
20
|
//! PLUS the astral planes (unifont_upper, incl. emoji) — beyond pxpipe, which
|
|
@@ -29,19 +29,40 @@ var yQ=(()=>{let J=new Uint32Array(256);for(let Z=0;Z<256;Z++){let Q=Z;for(let $
|
|
|
29
29
|
//! * `font` — `Tiny` renders the same atlas box-filtered into a 4x6 cell
|
|
30
30
|
//! (390 cols x 120 rows/page), ~40% fewer image-tokens; opt-in,
|
|
31
31
|
//! transcription-accuracy gated.
|
|
32
|
-
var
|
|
33
|
-
`);for(let
|
|
34
|
-
`).replace(
|
|
32
|
+
var a=4,VJ=4,mQ=1568,uQ=728,zJ="↵",gQ="⏎",GJ="→",dQ="⇢",RJ="⇥",lQ="⇨",pQ=9647,fJ=4,nQ=3,mJ="0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz",cQ=/\n{4,}/g;function t(J){return J.length===4&&J.replace(/[A-Z]/g,(Q)=>Q.toLowerCase())==="tiny"?"tiny":"normal"}function lJ(J){let Q=J==="tiny"?4:v,Z=J==="tiny"?6:i,$=Math.floor((mQ-2*a)/Q),V=Math.floor((uQ-2*VJ)/Z);return{cw:Q,ch:Z,cols:$,maxLines:V,maxChars:$*V}}var oQ=(()=>{let J=new Uint8Array(128);for(let Q=0;Q<128;Q++){let Z=r(Q);J[Q]=Z>=0&&d(Z)?2:1}return J})();function AJ(J){if(J<128)return oQ[J];let Q=r(J);return Q>=0&&d(Q)?2:1}function iQ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($>=55296&&$<56320&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<57344)Z++}Q++}return Q}function PJ(J){let Q=J.split(`
|
|
33
|
+
`);for(let Z=0;Z<Q.length;Z++){let $=Q[Z],V=$.length;while(V>0){let z=$.charCodeAt(V-1);if(z!==32&&z!==9)break;V--}if(V!==$.length)Q[Z]=$.slice(0,V)}return Q.join(`
|
|
34
|
+
`).replace(cQ,`
|
|
35
35
|
|
|
36
36
|
|
|
37
|
-
`)}function
|
|
38
|
-
`);for(let
|
|
39
|
-
`);for(let
|
|
40
|
-
`)){let z=
|
|
37
|
+
`)}function pJ(J){if(!J.includes("\t"))return J;let Q="",Z=0;for(let $ of J)if($==="\t"){let V=fJ-Z%fJ;Q+=GJ;for(let z=1;z<V;z++)Q+=" ";Z+=V}else Q+=$,Z+=AJ($.codePointAt(0));return Q}function rQ(J){let Q=J.includes("\t")?J.replaceAll("\t",GJ):J,Z=0;while(Z<Q.length&&Q.charCodeAt(Z)===32)Z++;if(Z>=nQ&&Z<mJ.length)return RJ+mJ[Z]+Q.slice(Z);return Q}function nJ(J){return J.includes(zJ)?J.replaceAll(zJ,gQ):J}function aQ(J){let Q=nJ(J);if(Q.includes(GJ))Q=Q.replaceAll(GJ,dQ);if(Q.includes(RJ))Q=Q.replaceAll(RJ,lQ);return Q}function tQ(J){let Q=PJ(J).split(`
|
|
38
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=pJ(Q[Z]);return Q.join(zJ)}function sQ(J){let Q=PJ(J).split(`
|
|
39
|
+
`);for(let Z=0;Z<Q.length;Z++)Q[Z]=rQ(Q[Z]);return Q.join(zJ)}function eQ(J,Q){let Z=[],$=PJ(J);for(let V of $.split(`
|
|
40
|
+
`)){let z=pJ(V);if(z.length===0){Z.push("");continue}let Y="",G=0;for(let H of z){let K=AJ(H.codePointAt(0));if(G+K>Q)Z.push(Y),Y=H,G=K;else Y+=H,G+=K}if(Y.length!==0)Z.push(Y)}return Z}function JZ(J,Q,Z){let $=[],V=[],z=0;for(let Y of J){let G=iQ(Y),H=G+(V.length!==0?1:0);if(V.length!==0&&(V.length>=Q||z+H>Z))$.push(V),V=[],z=0;z+=G+(V.length!==0?1:0),V.push(Y)}if(V.length!==0)$.push(V);return $}function cJ(J){if(J<32)return!0;if(J>=127&&J<=159)return!0;if(J>=768&&J<=879)return!0;if(J===8203||J===8204||J===8205||J===8288||J===65279)return!0;if(J>=65024&&J<=65039)return!0;if(J>=917760&&J<=917999)return!0;return!1}function QZ(J){let Q=null,Z=0;for(let $ of J){let V=$.codePointAt(0);if(r(V)<0&&!cJ(V)){if(Q===null)Q=J.slice(0,Z);Q+=`[U+${V.toString(16).toUpperCase()}]`}else if(Q!==null)Q+=$;Z+=$.length}return Q??J}function oJ(J,Q,Z,$){let V=Q?Z?sQ(aQ(J)):tQ(nJ(J)):J;return JZ(eQ(QZ(V),$.cols),$.maxLines,$.maxChars)}function iJ(J,Q,Z){if(!Z)return 2*a+Q.cols*Q.cw;let $=0;for(let Y of J){let G=0;for(let H of Y)G+=AJ(H.codePointAt(0));if(G>Q.cols)G=Q.cols;if(G>$)$=G}let V=2*a+$*Q.cw,z=2*a+Q.cw;return V>z?V:z}var uJ=new Map;function gJ(J,Q,Z,$,V,z,Y){let G=r(V);if(G<0)return 0;let H=d(G),K=H?2*z.cw:z.cw,B;if(Y==="normal")B=jJ(G);else{let U=uJ.get(G);if(U===void 0)U=kJ(G,K,z.ch),uJ.set(G,U);B=U}for(let U=0;U<z.ch;U++){let X=($+U)*Q+Z,O=U*K;for(let q=0;q<K;q++){let F=B[O+q];if(F>0){let R=X+q;if(F>J[R])J[R]=F}}}return H?2:1}function ZZ(J,Q,Z,$){let V=iJ(J,Q,Z),z=2*VJ+J.length*Q.ch,Y=new Uint8Array(V*z),G=0;for(let H=0;H<J.length;H++){let K=VJ+H*Q.ch,B=0;for(let U of J[H]){if(B>=Q.cols)break;let X=a+B*Q.cw,O=gJ(Y,V,X,K,U.codePointAt(0),Q,$);if(O===0){if(O=1,cJ(U.codePointAt(0)));else if(G++,U!==" ")gJ(Y,V,X,K,pQ,Q,$)}B+=O}}for(let H=0;H<Y.length;H++)Y[H]=255-Y[H];return{png:hJ(Y,V,z),width:V,height:z,dropped:G}}function l(J,Q,Z,$){let V=lJ($),z=oJ(J,Q,Z,V).map((K)=>ZZ(K,V,Z,$)),Y=0,G=0,H=0;for(let K of z)Y+=K.width*K.height,G+=aJ(K.width,K.height),H+=K.dropped;return{pages:z,pixels:Y,tokens:G,dropped:H}}function rJ(J,Q,Z,$){let V=lJ($),z=oJ(J,Q,Z,V),Y=0,G=0;for(let H of z){let K=iJ(H,V,Z),B=2*VJ+H.length*V.ch;Y+=K*B,G+=aJ(K,B)}return{pages:z.length,pixels:Y,tokens:G}}var dJ=28;function aJ(J,Q){return Math.ceil(J/dJ)*Math.ceil(Q/dJ)}import{readFileSync as $Z}from"node:fs";import{join as VZ}from"node:path";//! pxpipe measurement-log summary (~/.pxpipe/events.jsonl), same math as the
|
|
41
41
|
//! node MCP: actual = every way input bytes get billed (input + cache reads +
|
|
42
42
|
//! cache creates) — ignoring cache_read would fake the savings.
|
|
43
|
-
class
|
|
44
|
-
`)){if(
|
|
43
|
+
class s{value;constructor(J){this.value=J}}function _J(){let J=process.env.TANUKI_EVENTS;if(J!==void 0)return J;let Q=process.env.HOME??"";return VZ(Q,".pxpipe","events.jsonl")}function p(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:0}function zZ(J){return J<0?-Math.round(-J):Math.round(J)}function tJ(){let J=_J(),Q;try{Q=$Z(J,"utf8")}catch{return{available:!1,note:`no ${J} yet`}}let Z=0,$=0,V=0,z=0,Y=0,G=0;for(let K of Q.split(`
|
|
44
|
+
`)){if(K.trim().length===0)continue;let B;try{B=JSON.parse(K)}catch{continue}Z+=1;let U=B!==null&&typeof B==="object"&&!Array.isArray(B)?B:{};if(U.compressed===!0)$+=1,V+=p(U.orig_chars),z+=p(U.image_count);Y+=p(U.baseline_tokens),G+=p(U.input_tokens)+p(U.cache_read_tokens)+p(U.cache_create_tokens)}let H=Y>0&&G>0?new s(zZ((1-G/Y)*1000)/10):null;return{available:!0,requests:Z,compressedRequests:$,imagedChars:V,imagesEmitted:z,baselineTokens:Y,actualInputTokens:G,estInputSavedPct:H}}//! Implicit mode: a local Anthropic middlebox, the pxpipe deployment shape
|
|
45
|
+
//! without pxpipe's structural flaw. Rules that keep it injection-shaped-free:
|
|
46
|
+
//!
|
|
47
|
+
//! 1. The system prompt and tool definitions are NEVER touched.
|
|
48
|
+
//! 2. Nothing moves between roles or positions: an oversized text block is
|
|
49
|
+
//! replaced IN PLACE by a short overt marker + PNG page blocks, in the
|
|
50
|
+
//! same user-role message (Anthropic allows image blocks in user content
|
|
51
|
+
//! and inside tool_result content).
|
|
52
|
+
//! 3. The latest message is never imaged (the model may need to quote it).
|
|
53
|
+
//! 4. Blocks carrying cache_control are never touched (rewriting would
|
|
54
|
+
//! defeat the cache they exist for).
|
|
55
|
+
//! 5. Imaging only happens when `estimate` says it wins by a clear margin;
|
|
56
|
+
//! everything else passes through byte-for-byte.
|
|
57
|
+
//!
|
|
58
|
+
//! Responses stream through untouched; usage is scraped from the stream for
|
|
59
|
+
//! the ~/.pxpipe/events.jsonl savings log (same format tanuki_stats reads).
|
|
60
|
+
var n={level:0,distill:!1,codebook:!1,font:"normal",minChars:4000,ratio:0.75,minSave:300,maxPages:20};function XZ(J){let Q=0;for(let Z=0;Z<J.length;Z++){let $=J.charCodeAt(Z);if($<55296||$>56319)Q++}return Q}function BZ(J,Q){let Z=XZ(J);if(Z<Q.minChars)return null;let $=J;if(Q.distill)$=m($,null,2).distilled;let V=0;if(Q.codebook){let K=QJ($);$=K.text,V=K.entries}if(Q.level>0)$=o($,Q.level).compressed;let z=Math.round(Z/4),Y=l($,!0,!0,Q.font);if(Y.pages.length>Q.maxPages)return null;if(Y.tokens>z*Q.ratio||z-Y.tokens<Q.minSave)return null;let H=[{type:"text",text:`[tanuki-context: ${Z} chars imaged in place as ${Y.pages.length} PNG page(s), ~${Y.tokens} vs ~${z} text tokens. ↵=newline →=tab ⇥N=indent`+(V>0?`; ·legend· line maps ${V} sigils`:"")+"]"}];for(let K of Y.pages)H.push({type:"image",source:{type:"base64",media_type:"image/png",data:Buffer.from(K.png.buffer,K.png.byteOffset,K.png.byteLength).toString("base64")}});return{blocks:H,origChars:Z,pages:Y.pages.length,savedTokens:z-Y.tokens}}function YJ(J){return J!==null&&typeof J==="object"&&!Array.isArray(J)}function OZ(J,Q){let Z;try{Z=JSON.parse(J)}catch{return null}if(!YJ(Z)||!Array.isArray(Z.messages))return null;let $=0,V=0,z=0,Y=0,G=(H)=>{let K=BZ(H,Q);if(K)$++,V+=K.origChars,z+=K.pages,Y+=K.savedTokens;return K};for(let H=0;H<Z.messages.length-1;H++){let K=Z.messages[H];if(!YJ(K)||K.role!=="user")continue;if(typeof K.content==="string"){let U=G(K.content);if(U)K.content=U.blocks;continue}if(!Array.isArray(K.content))continue;let B=[];for(let U of K.content){if(!YJ(U)||U.cache_control!==void 0){B.push(U);continue}if(U.type==="text"&&typeof U.text==="string"){let X=G(U.text);if(X)B.push(...X.blocks);else B.push(U);continue}if(U.type==="tool_result"){if(typeof U.content==="string"){let X=G(U.content);if(X)U.content=X.blocks}else if(Array.isArray(U.content)){let X=[];for(let O of U.content){if(YJ(O)&&O.type==="text"&&typeof O.text==="string"&&O.cache_control===void 0){let q=G(O.text);if(q){X.push(...q.blocks);continue}}X.push(O)}U.content=X}}B.push(U)}K.content=B}if($===0)return null;return{body:JSON.stringify(Z),imagedBlocks:$,origChars:V,imageCount:z,savedTokens:Y}}function qZ(J){let Q=(Z)=>{let $=Z.exec(J);return $?Number($[1]):0};return{input:Q(/"input_tokens"\s*:\s*(\d+)/),cacheRead:Q(/"cache_read_input_tokens"\s*:\s*(\d+)/),cacheCreate:Q(/"cache_creation_input_tokens"\s*:\s*(\d+)/)}}function DZ(J){try{let Q=_J();YZ(HZ(Q),{recursive:!0}),GZ(Q,JSON.stringify(J)+`
|
|
61
|
+
`)}catch{}}function eJ(J){let Q=new UZ(J.upstream),Z=Q.protocol==="https:"?KZ:sJ,$=sJ.createServer((V,z)=>{let Y=[];V.on("data",(G)=>Y.push(G)),V.on("end",()=>{let G=Buffer.concat(Y),H=V.method==="POST"&&(V.url??"").startsWith("/v1/messages")&&!(V.url??"").includes("count_tokens")&&V.headers["content-encoding"]===void 0,K=null;if(H){if(K=OZ(G.toString("utf8"),J),K)G=Buffer.from(K.body,"utf8")}let B={...V.headers};if(delete B.host,delete B.connection,B["content-length"]=String(G.length),H)delete B["accept-encoding"];let U=Z.request({protocol:Q.protocol,hostname:Q.hostname,port:Q.port||(Q.protocol==="https:"?443:80),path:V.url,method:V.method,headers:B},(X)=>{if(z.writeHead(X.statusCode??502,X.headers),H){let O=[];X.on("data",(q)=>{O.push(q),z.write(q)}),X.on("end",()=>{z.end();let q=qZ(Buffer.concat(O).toString("utf8")),F=q.input+q.cacheRead+q.cacheCreate;DZ({ts:Date.now(),tool:"proxy",compressed:K!==null,orig_chars:K?.origChars??0,image_count:K?.imageCount??0,baseline_tokens:F+(K?.savedTokens??0),input_tokens:q.input,cache_read_tokens:q.cacheRead,cache_create_tokens:q.cacheCreate})})}else X.pipe(z)});U.on("error",(X)=>{z.writeHead(502,{"content-type":"application/json"}),z.end(JSON.stringify({type:"error",error:{type:"api_error",message:`tanuki proxy: upstream unreachable (${X.message})`}}))}),U.end(G)})});return $.listen(J.port,"127.0.0.1",()=>{let V=$.address(),z=V!==null&&typeof V==="object"?V.port:J.port,Y=`level=${J.level} distill=${J.distill} codebook=${J.codebook} font=${J.font} minChars=${J.minChars} ratio=${J.ratio} minSave=${J.minSave}`;process.stderr.write(`tanuki-context proxy on http://127.0.0.1:${z} -> ${J.upstream}
|
|
62
|
+
${Y}
|
|
63
|
+
`+` rules: system prompt & tools untouched · in-place blocks only · latest message never imaged · cache_control skipped
|
|
64
|
+
`+` point your client at it: export ANTHROPIC_BASE_URL=http://127.0.0.1:${z}
|
|
65
|
+
`)}),$}//! tanuki-context — token-cutting context pipeline.
|
|
45
66
|
//! pipeline: text -> distill (stage 0, logs) -> ladder level 0-4 (stage 1)
|
|
46
67
|
//! -> pxpipe imaging (stage 2, name kept from the original mechanic)
|
|
47
68
|
//!
|
|
@@ -49,20 +70,23 @@ class r{value;constructor(J){this.value=J}}function aQ(){let J=process.env.TANUK
|
|
|
49
70
|
//! CLI: tanuki-context distill <file> [query]
|
|
50
71
|
//! tanuki-context estimate <file> [level] [--distill]
|
|
51
72
|
//! tanuki-context render <file> [level] [outdir]
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
`;
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
`;
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
`)
|
|
63
|
-
`)
|
|
64
|
-
`);break}case"
|
|
65
|
-
`);
|
|
66
|
-
`);break}case"
|
|
67
|
-
|
|
68
|
-
|
|
73
|
+
//! tanuki-context proxy [--port N] [--upstream URL] [knobs] (implicit mode)
|
|
74
|
+
var NZ="0.1.0",SJ=6;function jZ(J,Q){let Z=Math.min(J.length,Q.length),$=0;while($<Z&&J.charCodeAt($)===Q.charCodeAt($))$++;if($>=Z)return J.length-Q.length;return J.codePointAt($)-Q.codePointAt($)}function E(J,Q,Z=""){if(J===null||J===void 0)return"null";if(J instanceof s){let H=J.value;return Number.isFinite(H)&&Number.isInteger(H)?H.toFixed(1):String(H)}let $=typeof J;if($==="string")return JSON.stringify(J);if($==="number"||$==="boolean")return String(J);if(Array.isArray(J)){if(J.length===0)return"[]";if(!Q){let B="[";for(let U=0;U<J.length;U++){if(U>0)B+=",";B+=E(J[U],!1)}return B+"]"}let H=Z+" ",K=`[
|
|
75
|
+
`;for(let B=0;B<J.length;B++){if(B>0)K+=`,
|
|
76
|
+
`;K+=H+E(J[B],!0,H)}return K+`
|
|
77
|
+
`+Z+"]"}let V=J,z=Object.keys(V).sort(jZ);if(z.length===0)return"{}";if(!Q){let H="{";for(let K=0;K<z.length;K++){if(K>0)H+=",";H+=JSON.stringify(z[K])+":"+E(V[z[K]],!1)}return H+"}"}let Y=Z+" ",G=`{
|
|
78
|
+
`;for(let H=0;H<z.length;H++){if(H>0)G+=`,
|
|
79
|
+
`;G+=Y+JSON.stringify(z[H])+": "+E(V[z[H]],!0,Y)}return G+`
|
|
80
|
+
`+Z+"}"}function I(J,Q){return J!==null&&typeof J==="object"&&!Array.isArray(J)?J[Q]:void 0}function L(J){return typeof J==="string"?J:null}function HJ(J){return typeof J==="boolean"?J:null}function $Q(J){return typeof J==="number"&&Number.isSafeInteger(J)&&J>=0?J:null}function k(J){let Q=0;for(let Z=0;Z<J.length;Z++){Q++;let $=J.charCodeAt(Z);if($>=55296&&$<=56319&&Z+1<J.length){let V=J.charCodeAt(Z+1);if(V>=56320&&V<=57343)Z++}}return Q}function XJ(J,Q,Z,$,V){let z,Y=null;if(Z||$!==null){let K=m(J,$,2);z=K.distilled,Y=K.stats}else z=J;let G=0;if(V){let K=QJ(z);z=K.text,G=K.entries}let H=o(z,Q);return{stage0:Y,compressed:H.compressed,protectedLines:H.protectedLines,level:H.level,cbEntries:G}}function e(J){return Math.round(J/4)}function JJ(J,Q){if(J===0)return 0;let Z=(1-Q/J)*100;return Z<0?-Math.round(-Z):Math.round(Z)}function VQ(J){return{text:L(I(J,"text"))??"",level:($Q(I(J,"level"))??0)%256,distill:HJ(I(J,"distill"))??!1,query:L(I(J,"query")),reflow:HJ(I(J,"reflow"))??!0,pack:HJ(I(J,"pack"))??!0,font:L(I(J,"font"))??"normal",codebook:HJ(I(J,"codebook"))??!1}}function zQ(J){let Q=VQ(J),Z=XJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=t(Q.font),V=rJ(Z.compressed,Q.reflow,Q.pack,$),z=V.tokens,Y=k(Q.text),G=k(Z.compressed),H=e(Y),[K,B]=ZJ[Z.level];return{engine:"pxpipe",level:`${Z.level} ${K}`,loss:B,distill:Z.stage0,origChars:Y,stage1Chars:G,stage1SavedPct:JJ(Y,G),pages:V.pages,imageTokens:z,rawTextTokens:H,totalSavedPct:JJ(H,z),protectedLines:Z.protectedLines,pack:Q.pack,font:$==="tiny"?"tiny":"normal",codebook:Q.codebook?Z.cbEntries:!1,verdict:z<H?"PIPELINE cheaper":"TEXT cheaper"}}function IZ(J){let Q=VQ(J),Z=XJ(Q.text,Q.level,Q.distill,Q.query,Q.codebook),$=t(Q.font),V=l(Z.compressed,Q.reflow,Q.pack,$),z=V.tokens,Y=k(Q.text),G=k(Z.compressed),H=e(Y),[K,B]=ZJ[Z.level],U="";if(Z.stage0!==null){let q=Z.stage0;U+=`distill: ${q.origLines} -> ${q.outLines} lines (-${q.savedPct}% chars, ${q.collapsedRuns} runs, ${q.suppressedLines} exact + ${q.templateSuppressed} template suppressed, ${q.importantKept} error/warn kept)
|
|
81
|
+
`}if(U+=`L${Z.level} ${K} (${B}): ${Y} chars -> ${G} chars (stage1 -${JJ(Y,G)}%) -> ${V.pages.length} page(s), ~${z} image-tokens
|
|
82
|
+
vs ~${H} text-tokens raw = TOTAL -${JJ(H,z)}%`,Z.protectedLines>0)U+=` · ${Z.protectedLines} lines kept verbatim`;if(V.dropped>0)U+=` · ${V.dropped} unmapped glyphs -> ▯`;if(Z.cbEntries>0)U+=` · codebook: ${Z.cbEntries} sigils (see ·legend·)`;if($==="tiny")U+=" · font: tiny 4x6";if(Q.pack)U+=" · packed (⇥N indent, → tab)";if(Q.reflow)U+=" · ↵ = newline · engine: pxpipe";let X=[{type:"text",text:U}],O=Math.min(V.pages.length,SJ);for(let q=0;q<O;q++){let F=V.pages[q].png;X.push({type:"image",data:Buffer.from(F.buffer,F.byteOffset,F.byteLength).toString("base64"),mimeType:"image/png"})}if(V.pages.length>SJ)X.push({type:"text",text:`(+${V.pages.length-SJ} more page(s))`});return X}function RZ(J){let Q=L(I(J,"text"))??"",Z=m(Q,L(I(J,"query")),2);return[{type:"text",text:E(Z.stats,!0)},{type:"text",text:Z.distilled}]}function AZ(J){let Q=L(I(J,"text"))??"",Z=($Q(I(J,"level"))??1)%256,$=o(Q,Z),[V,z,Y]=ZJ[$.level],G=k(Q),H=k($.compressed),K=e(G),B=e(H),U={level:`${$.level} ${V}`,loss:z,note:Y,origChars:G,outChars:H,approxOrigTokens:K,approxOutTokens:B,savedPct:JJ(K,B),protectedLines:$.protectedLines};return[{type:"text",text:E(U,!0)},{type:"text",text:$.compressed}]}function PZ(){let J={type:"string"},Q={type:"integer",minimum:0,maximum:4};return{tools:[{name:"tanuki_render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow (single-cell tabs, ⇥N indent runs, width-trimmed pages). font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",inputSchema:{type:"object",properties:{text:J,level:Q,distill:{type:"boolean"},query:{type:"string"},reflow:{type:"boolean"},pack:{type:"boolean"},font:{type:"string",enum:["normal","tiny"]},codebook:{type:"boolean"}},required:["text"]}},{name:"tanuki_estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff.",inputSchema:{type:"object",properties:{text:J,level:Q,distill:{type:"boolean"},query:{type:"string"},reflow:{type:"boolean"},pack:{type:"boolean"},font:{type:"string",enum:["normal","tiny"]},codebook:{type:"boolean"}},required:["text"]}},{name:"tanuki_distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes (exact + same-template) with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. Deterministic, order-preserving.",inputSchema:{type:"object",properties:{text:J,query:{type:"string"}},required:["text"]}},{name:"tanuki_compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",inputSchema:{type:"object",properties:{text:J,level:Q},required:["text"]}},{name:"tanuki_stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates).",inputSchema:{type:"object",properties:{}}}]}}function _Z(J){let Q=L(I(J,"name"))??"",Z=I(J,"arguments"),$;switch(Q){case"tanuki_render":$=IZ(Z);break;case"tanuki_estimate":$=[{type:"text",text:E(zQ(Z),!0)}];break;case"tanuki_distill":$=RZ(Z);break;case"tanuki_compress":$=AZ(Z);break;case"tanuki_stats":$=[{type:"text",text:E(tJ(),!0)}];break;default:return{ok:!1,error:`unknown tool: ${Q}`}}return{ok:!0,value:{content:$}}}function JQ(J){let Q=J.endsWith("\r")?J.slice(0,-1):J;if(Q.trim().length===0)return;let Z;try{Z=JSON.parse(Q)}catch{return}let $=Z!==null&&typeof Z==="object"&&!Array.isArray(Z)&&Object.prototype.hasOwnProperty.call(Z,"id"),V=$?Z.id??null:null,z=L(I(Z,"method")),Y;switch(z){case"initialize":{let G=L(I(I(Z,"params"),"protocolVersion"))??"2025-06-18";Y={jsonrpc:"2.0",id:V,result:{protocolVersion:G,capabilities:{tools:{}},serverInfo:{name:"tanuki-context",version:NZ}}};break}case"notifications/initialized":case"notifications/cancelled":Y=void 0;break;case"ping":Y={jsonrpc:"2.0",id:V,result:{}};break;case"tools/list":Y={jsonrpc:"2.0",id:V,result:PZ()};break;case"tools/call":{let G=_Z(I(Z,"params"));Y=G.ok?{jsonrpc:"2.0",id:V,result:G.value}:{jsonrpc:"2.0",id:V,error:{code:-32602,message:G.error}};break}default:Y=$?{jsonrpc:"2.0",id:V,error:{code:-32601,message:"method not found"}}:void 0}if(Y!==void 0)S.stdout.write(E(Y,!1)+`
|
|
83
|
+
`)}function SZ(){let J=Buffer.alloc(0);S.stdin.on("data",(Q)=>{let Z=J.length>0?Buffer.concat([J,Q]):Q,$=0,V;while((V=Z.indexOf(10,$))!==-1)JQ(Z.toString("utf8",$,V)),$=V+1;J=Z.subarray($)}),S.stdin.on("end",()=>{if(J.length>0)JQ(J.toString("utf8")),J=Buffer.alloc(0)})}function x(J){S.stderr.write(J+`
|
|
84
|
+
`),S.exit(101)}function KJ(J){try{return FZ(J,"utf8")}catch{x("read file")}}function UJ(J,Q){if(!/^\+?[0-9]+$/.test(J))return null;let Z=BigInt(J.startsWith("+")?J.slice(1):J);return Z>Q?null:Number(Z)}var QQ=255n,ZQ=0xffffffffffffffffn;function GQ(){let J=S.argv.slice(1);switch(J[1]){case"distill":{let Q=J[2]??x("usage: tanuki-context distill <file> [query]"),Z=KJ(Q),$=m(Z,J[3]??null,2);S.stdout.write(E($.stats,!1)+`
|
|
85
|
+
`);break}case"estimate":{let Q=J[2]??x("usage: tanuki-context estimate <file> [level] [--distill] [--no-pack] [--font tiny] [--codebook]"),Z=KJ(Q),$=J.slice(3).filter((H)=>!H.startsWith("--")),V=$.length>0?UJ($[0],ZQ)??0:0,z=J.indexOf("--font"),Y=z!==-1&&J[z+1]!==void 0?J[z+1]:"normal",G=zQ({text:Z,level:V,distill:J.includes("--distill"),pack:!J.includes("--no-pack"),font:Y,codebook:J.includes("--codebook")});S.stdout.write(E(G,!1)+`
|
|
86
|
+
`);break}case"render":{let Q=J[2]??x("usage: tanuki-context render <file> [level] [outdir] [--no-pack] [--font tiny] [--codebook]"),Z=KJ(Q),$=J.slice(3).filter((O)=>!O.startsWith("--")),V=$.length>0?UJ($[0],QQ)??0:0,z=!J.includes("--no-pack"),Y=J.includes("--codebook"),G=J.indexOf("--font"),H=t(G!==-1&&J[G+1]!==void 0?J[G+1]:"normal"),K=XJ(Z,V,!1,null,Y),B=l(K.compressed,!0,z,H),U=B.tokens;S.stdout.write(E({pages:B.pages.length,imageTokens:U,dropped:B.dropped,rawTextTokens:e(k(Z))},!1)+`
|
|
87
|
+
`);let X=$[1];if(X!==void 0){try{MZ(X,{recursive:!0})}catch{x("mkdir")}for(let O=0;O<B.pages.length;O++)try{WZ(`${X}/page${O}.png`,B.pages[O].png)}catch{x("write png")}}break}case"bench":{let Q=J[2]??x("usage: tanuki-context bench <file> <op> [level] [runs] [--distill]"),Z=J[3]??"pipeline",$=J[4]!==void 0?UJ(J[4],QQ)??0:0,V=J[5]!==void 0?UJ(J[5],ZQ)??3:3,z=J.includes("--distill"),Y=KJ(Q),G=[],H=null;for(let K=0;K<=V;K++){let B=performance.now();if(Z==="distill")H=m(Y,null,2).stats;else{let U=XJ(Y,$,z,null,!1),X=l(U.compressed,!0,!1,"normal");H={pages:X.pages.length,imageTokens:X.tokens,stage1Chars:k(U.compressed),dropped:X.dropped}}if(K>0)G.push(performance.now()-B)}if(G.sort((K,B)=>K-B),G.length===0)x("index out of bounds: the len is 0 but the index is 0");S.stdout.write(E({medianMs:new s(G[Math.floor(G.length/2)]),runs:V,result:H},!1)+`
|
|
88
|
+
`);break}case"proxy":{let Q=(V,z)=>{let Y=J.indexOf(V);if(Y===-1||J[Y+1]===void 0)return z;let G=Number(J[Y+1]);return Number.isFinite(G)?G:z},Z=J.indexOf("--upstream"),$=J.indexOf("--font");eJ({port:Q("--port",8484),upstream:Z!==-1&&J[Z+1]!==void 0?J[Z+1]:S.env.TANUKI_UPSTREAM??"https://api.anthropic.com",level:Q("--level",n.level),distill:J.includes("--distill"),codebook:J.includes("--codebook"),font:t($!==-1&&J[$+1]!==void 0?J[$+1]:"normal"),minChars:Q("--min-chars",n.minChars),ratio:Q("--ratio",n.ratio),minSave:Q("--min-save",n.minSave),maxPages:Q("--max-pages",n.maxPages)});break}case"serve":case void 0:SZ();break;default:S.stderr.write(`unknown command: ${J[1]}
|
|
89
|
+
usage: tanuki-context [serve|proxy|distill|estimate|render] ...
|
|
90
|
+
`),S.exit(1)}}//! Bin entry: keeps src/main.ts importable as a library (src/agent.ts) without
|
|
91
|
+
//! starting the MCP server as an import side effect.
|
|
92
|
+
GQ();
|
package/dist/pi.js
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
import{spawn as W}from"node:child_process";import{fileURLToPath as X}from"node:url";import{Type as q}from"typebox";//! Pi (pi-mono / oh-my-pi lineage) extension: registers the five tanuki tools.
|
|
2
|
+
//!
|
|
3
|
+
//! Engine-agnostic by design: the extension is a thin stdio JSON-RPC client to
|
|
4
|
+
//! a `tanuki-context` MCP server it spawns itself, so the same file serves
|
|
5
|
+
//! - the npm/TS engine (default: `node <this package>/dist/cli.js`), and
|
|
6
|
+
//! - the Rust engine (`TANUKI_BIN=/path/to/tanuki-context`).
|
|
7
|
+
//! One child per pi session, spawned lazily on first tool call (pi docs forbid
|
|
8
|
+
//! starting processes in the factory), killed on session_shutdown.
|
|
9
|
+
//!
|
|
10
|
+
//! MCP tool-result content blocks ({type:"text"|"image"}) are pi's own
|
|
11
|
+
//! ToolResult content shape, so results pass through untouched.
|
|
12
|
+
function Y(){let F=process.env.TANUKI_BIN;if(F)return{cmd:F,args:[]};return{cmd:process.execPath,args:[X(new URL("./cli.js",import.meta.url))]}}class Q{proc;buf="";nextId=1;pending=new Map;ready;constructor(){let{cmd:F,args:B}=Y();this.proc=W(F,B,{stdio:["pipe","pipe","inherit"]}),this.proc.stdout.setEncoding("utf8"),this.proc.stdout.on("data",(G)=>{this.buf+=G;let z;while((z=this.buf.indexOf(`
|
|
13
|
+
`))!==-1){let H=this.buf.slice(0,z).trim();if(this.buf=this.buf.slice(z+1),!H)continue;let D;try{D=JSON.parse(H)}catch{continue}let J=D.id!==void 0?this.pending.get(D.id):void 0;if(!J)continue;if(this.pending.delete(D.id),D.error)J.reject(Error(D.error.message??"tanuki-context error"));else J.resolve(D.result)}}),this.proc.on("exit",(G)=>{let z=Error(`tanuki-context server exited (code ${G})`);for(let H of this.pending.values())H.reject(z);this.pending.clear()}),this.ready=this.request("initialize",{})}request(F,B){let G=this.nextId++,{promise:z,resolve:H,reject:D}=Promise.withResolvers();if(!this.proc.stdin?.writable)return D(Error("tanuki-context server is gone")),z;return this.pending.set(G,{resolve:H,reject:D}),this.proc.stdin.write(JSON.stringify({jsonrpc:"2.0",id:G,method:F,params:B})+`
|
|
14
|
+
`),z}async call(F,B){return await this.ready,this.request("tools/call",{name:F,arguments:B})}kill(){this.proc.kill()}get alive(){return this.proc.exitCode===null&&!this.proc.killed}}var K=q.String({description:"The text to process"}),V=q.Optional(q.Integer({minimum:0,maximum:4,description:"0 raw · 1 whitespace · 2 prose · 3 dense · 4 caveman"})),N=q.Object({text:K,level:V,distill:q.Optional(q.Boolean()),query:q.Optional(q.String()),reflow:q.Optional(q.Boolean()),pack:q.Optional(q.Boolean()),font:q.Optional(q.String({enum:["normal","tiny"]})),codebook:q.Optional(q.Boolean())}),Z=[{name:"tanuki_render",label:"Tanuki Render",description:"Token-cut pipeline: optional log distillation (dedupe noise, keep errors verbatim, optional query filter), optional codebook (repeated long tokens/path prefixes -> 1-cell sigils + a ·legend· line), then a ladder level, then dense PNG page(s) via the pxpipe imaging engine. level 0 raw · 1 whitespace (lossless) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths stay verbatim. pack (default true) = lossless tight reflow. font 'tiny' = 4x6 cell, ~40% fewer image-tokens (opt-in). Image tokens are pixel-priced, so every earlier cut compounds. Returns image blocks + a breakdown.",parameters:N,snippet:"Render bulky text/logs as dense PNG pages that cost a fraction of the tokens"},{name:"tanuki_estimate",label:"Tanuki Estimate",description:"Estimate tokens for the pipeline (distill -> codebook -> level -> pxpipe imaging) vs sending the raw text as text. Exact page geometry, no image data returned. Compare levels/pack/font/codebook to pick a loss/size tradeoff.",parameters:N,snippet:"Instant token verdict: would imaging this text beat sending it as text?"},{name:"tanuki_distill",label:"Tanuki Distill",description:"Stage 0 alone: make noisy logs/output small and readable WITHOUT imaging. Strips ANSI, collapses runs of near-identical lines/blocks into '[×N similar]', suppresses global near-dupes with exact counts, always keeps error/warn/fail lines verbatim, optional query (regex) returns only the relevant slice. Deterministic, order-preserving.",parameters:q.Object({text:K,query:q.Optional(q.String())}),snippet:"Deterministically dedupe noisy logs, keeping every error line verbatim"},{name:"tanuki_compress",label:"Tanuki Compress",description:"Stage 1 alone: graded text compression for content that stays TEXT. level 0 none · 1 whitespace (lossless, safe for code) · 2 prose · 3 dense · 4 caveman (gist only). From level 2 up code/IDs/hashes/paths are preserved verbatim.",parameters:q.Object({text:K,level:V}),snippet:"Graded text compression (lossless whitespace up to gist-only)"},{name:"tanuki_stats",label:"Tanuki Stats",description:"Summarize the pxpipe measurement log (~/.pxpipe/events.jsonl): requests, compression counts, honest input-token savings (input + cache reads + cache creates).",parameters:q.Object({}),snippet:"Session savings summary from the tanuki/pxpipe event log"}];function $(F){let B=null,G=()=>{if(!B||!B.alive)B=new Q;return B};F.on("session_shutdown",async()=>{B?.kill(),B=null});for(let z of Z)F.registerTool({name:z.name,label:z.label,description:z.description,promptSnippet:z.snippet,parameters:z.parameters,async execute(H,D){let J=await G().call(z.name,D??{}),M=(J.content??[]).map((I)=>I.type==="image"?{type:"image",data:I.data??"",mimeType:I.mimeType??"image/png"}:{type:"text",text:I.text??""});if(J.isError)throw Error(M.map((I)=>("text"in I)?I.text:"").join(`
|
|
15
|
+
`)||z.name+" failed");return{content:M,details:{}}}})}export{$ as default};
|
package/package.json
CHANGED
|
@@ -1,12 +1,19 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tanuki-context",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.1",
|
|
4
4
|
"description": "Token-cutting context pipeline as an MCP server: distill logs, compress text, render dense PNG pages (pxpipe imaging engine). Zero dependencies.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
7
7
|
"bin": {
|
|
8
8
|
"tanuki-context": "dist/cli.js"
|
|
9
9
|
},
|
|
10
|
+
"exports": {
|
|
11
|
+
"./agent": {
|
|
12
|
+
"types": "./dist/agent.d.ts",
|
|
13
|
+
"default": "./dist/agent.js"
|
|
14
|
+
},
|
|
15
|
+
"./package.json": "./package.json"
|
|
16
|
+
},
|
|
10
17
|
"files": [
|
|
11
18
|
"dist",
|
|
12
19
|
"assets",
|
|
@@ -14,9 +21,32 @@
|
|
|
14
21
|
"LICENSE"
|
|
15
22
|
],
|
|
16
23
|
"scripts": {
|
|
17
|
-
"build": "bun build src/
|
|
24
|
+
"build": "bun build src/cli.ts --target=node --minify --outfile=dist/cli.js && bun build src/agent.ts --target=node --minify --external @anthropic-ai/claude-agent-sdk --external zod --outfile=dist/agent.js && bun build src/pi.ts --target=node --minify --external typebox --outfile=dist/pi.js && cp types/agent.d.ts dist/agent.d.ts",
|
|
18
25
|
"prepublishOnly": "bun run build",
|
|
19
|
-
"parity": "bun reference/parity-ts.mjs"
|
|
26
|
+
"parity": "bun reference/parity-ts.mjs",
|
|
27
|
+
"test": "bun test"
|
|
28
|
+
},
|
|
29
|
+
"peerDependencies": {
|
|
30
|
+
"@anthropic-ai/claude-agent-sdk": ">=0.3.0",
|
|
31
|
+
"zod": "^3.25.0 || ^4.0.0"
|
|
32
|
+
},
|
|
33
|
+
"peerDependenciesMeta": {
|
|
34
|
+
"@anthropic-ai/claude-agent-sdk": {
|
|
35
|
+
"optional": true
|
|
36
|
+
},
|
|
37
|
+
"zod": {
|
|
38
|
+
"optional": true
|
|
39
|
+
}
|
|
40
|
+
},
|
|
41
|
+
"devDependencies": {
|
|
42
|
+
"@anthropic-ai/claude-agent-sdk": "^0.3.220",
|
|
43
|
+
"typebox": "^1.0.0",
|
|
44
|
+
"zod": "^4.0.0"
|
|
45
|
+
},
|
|
46
|
+
"pi": {
|
|
47
|
+
"extensions": [
|
|
48
|
+
"./dist/pi.js"
|
|
49
|
+
]
|
|
20
50
|
},
|
|
21
51
|
"engines": {
|
|
22
52
|
"node": ">=18"
|