@tokenade/cli 0.7.0 → 0.7.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +534 -18
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,32 +1,548 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
output filtering and compaction, applied automatically as a local MCP server +
|
|
5
|
-
inspectable agent hooks. Paid product (freemium) — see <https://tokenade.net>.
|
|
3
|
+
# ⚡ Tokenade
|
|
6
4
|
|
|
7
|
-
|
|
5
|
+
### The #1 tool to cut your AI agent's token bill.
|
|
6
|
+
|
|
7
|
+
**Ranked #1 on the [Token-Harness Optimizer Leaderboard](https://pi-infected.github.io/token-harness-optimizer-leaderboard/) — zero config.**
|
|
8
|
+
|
|
9
|
+
**Tokenade is the simplest way to slash what your coding agent sends to the model.**
|
|
10
|
+
Set it up once, save on every prompt — same results, a fraction of the tokens.
|
|
11
|
+
|
|
12
|
+
**Freemium** — start free (10M tokens/month, no credit card), upgrade only when you scale.
|
|
13
|
+
|
|
14
|
+
[](https://www.npmjs.com/package/@tokenade/cli)
|
|
15
|
+
[](https://tokenade.net)
|
|
16
|
+
[](https://nodejs.org)
|
|
17
|
+
[](https://tokenade.net/signup)
|
|
18
|
+
|
|
19
|
+
[**🚀 Start free**](https://tokenade.net/signup) · [**💸 Pricing**](https://tokenade.net/pricing) · [**📊 Dashboard**](https://tokenade.net/dashboard) · [**🐛 Report an issue**](https://github.com/pi-infected/tokenade-npm/issues)
|
|
20
|
+
|
|
21
|
+
</div>
|
|
22
|
+
|
|
23
|
+
---
|
|
8
24
|
|
|
9
25
|
```sh
|
|
10
26
|
npm install -g @tokenade/cli
|
|
27
|
+
tokenade login # opens your browser to authorize this machine — no key to type
|
|
28
|
+
tokenade install # wires Tokenade into your coding agent automatically
|
|
11
29
|
```
|
|
12
30
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
31
|
+
That's it. Tokenade now trims tokens on **every prompt**, in the background. No config, nothing to remember.
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## Why Tokenade
|
|
16
36
|
|
|
17
|
-
|
|
37
|
+
Your coding agent burns tokens on things the model never needed to see — whole files when one function mattered, 2,000-line build logs, every MCP tool definition on every turn. Tokenade strips that waste **before it reaches the model**, locally, automatically.
|
|
38
|
+
|
|
39
|
+
> **Less tokens, same results — automatically, on every prompt.**
|
|
40
|
+
|
|
41
|
+
## ✨ What it does
|
|
42
|
+
|
|
43
|
+
| | Feature | What you get |
|
|
44
|
+
|---|---|---|
|
|
45
|
+
| 🔎 | **Semantic code search** | Finds the right files by *meaning*, so your agent reads only what matters — not your whole project. |
|
|
46
|
+
| 🧹 | **Command optimization** | Trims long command outputs (logs, builds, tests) down to what counts before your agent ever sees them. |
|
|
47
|
+
| 🧩 | **MCP optimization** | Loads only the MCP tools your agent actually needs, on demand — instead of every tool, every time. |
|
|
48
|
+
| 📊 | **Token dashboard** | See exactly how many tokens Tokenade saved you — clear numbers, no guesswork. |
|
|
49
|
+
| 🛟 | **Free to start** | Save up to **10M tokens/month** on the free plan — unlimited machines, no credit card. |
|
|
50
|
+
| ⚙️ | **One-command install** | Ready in seconds. Connects to your agent automatically, with zero configuration. |
|
|
51
|
+
|
|
52
|
+
## 🤝 Works with the agents you already use
|
|
53
|
+
|
|
54
|
+
**Claude Code** · **Cursor** · **Codex** · **GitHub Copilot** · **Gemini CLI** · **Qwen Code** · **Windsurf** · **Cline** · **Kilo Code** · **OpenCode** · **Aider** · **Hermes** · **OpenClaw** · **Antigravity** · **Grok**
|
|
55
|
+
|
|
56
|
+
## 📦 Install
|
|
18
57
|
|
|
19
58
|
```sh
|
|
20
|
-
|
|
21
|
-
tokenade install # registers the MCP server + hooks with your coding agent
|
|
59
|
+
npm install -g @tokenade/cli
|
|
22
60
|
```
|
|
23
61
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
62
|
+
The `postinstall` step downloads the prebuilt binary for your platform from the **signed release manifest** (`downloads.tokenade.net/manifest.json`) and verifies its **SHA‑256**. Nothing is run blindly — and the launcher in [`bin/tokenade.js`](bin/tokenade.js) and the installer in [`install.js`](install.js) are right here in this repo for you to inspect.
|
|
63
|
+
|
|
64
|
+
**Supported:** Linux · macOS · Windows · x64 & arm64 · Node ≥ 18.
|
|
65
|
+
|
|
66
|
+
## 🔓 Activate (browser — no key to type)
|
|
67
|
+
|
|
68
|
+
```sh
|
|
69
|
+
tokenade login # opens tokenade.net so you (logged into your dashboard) approve this device
|
|
70
|
+
tokenade install # registers the hooks with your coding agent
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
**Free** and **Pro** include **unlimited machines**; **Enterprise** includes 3 (plus paid extra seats). Manage them anytime at [tokenade.net/dashboard](https://tokenade.net/dashboard).
|
|
74
|
+
|
|
75
|
+
## 🔒 What leaves your machine
|
|
76
|
+
|
|
77
|
+
**Nothing**, except anonymous token-savings counts and an update check. Tokenade runs **locally**; its agent hooks live in your editor/agent config — **inspectable and reversible**.
|
|
78
|
+
|
|
79
|
+
## 💸 Pricing
|
|
80
|
+
|
|
81
|
+
| | **Free** | **Pro** | **Enterprise** |
|
|
82
|
+
|---|---|---|---|
|
|
83
|
+
| Token savings | Up to **10M / mo** | Up to **100M / mo** | **Unlimited** |
|
|
84
|
+
| Machines | **Unlimited** | **Unlimited** | 3 (+ paid seats) |
|
|
85
|
+
| All reduction features | ✅ | ✅ | ✅ |
|
|
86
|
+
| Token-savings dashboard | ✅ | ✅ | ✅ |
|
|
87
|
+
| Priority support | — | ✅ | ✅ |
|
|
88
|
+
| Price | **$0** — no card | **$19.90/mo** — flat, no per-token billing | **$2,990/yr** — 3 seats incl. |
|
|
89
|
+
| | [Start free →](https://tokenade.net/signup) | [Go Pro →](https://tokenade.net/pricing) | [Go Enterprise →](https://tokenade.net/pricing) |
|
|
90
|
+
|
|
91
|
+
## 🐛 Found a bug? Tell us!
|
|
92
|
+
|
|
93
|
+
Tokenade is in **active beta** and your reports make it better — fast.
|
|
94
|
+
|
|
95
|
+
- **Open an issue:** https://github.com/pi-infected/tokenade-npm/issues
|
|
96
|
+
- **Join the beta testers on WhatsApp:** https://chat.whatsapp.com/JmBZxT41XB6AtQWgW9oNif
|
|
97
|
+
|
|
98
|
+
When reporting, please include your OS, `tokenade --version`, your coding agent, and what you expected vs. what happened. The more detail, the faster we fix it. 🙏
|
|
99
|
+
|
|
100
|
+
---
|
|
101
|
+
|
|
102
|
+
## 🧰 CLI reference — every command
|
|
103
|
+
|
|
104
|
+
> **You rarely need any of these by hand.** After `tokenade install`, Tokenade works automatically through your agent's hooks. The CLI below is for power users, scripting, and inspection. Run `tokenade help` for the live list, and `tokenade <command> --help` where available.
|
|
105
|
+
|
|
106
|
+
### Setup & account
|
|
107
|
+
|
|
108
|
+
| Command | What it does |
|
|
109
|
+
|---|---|
|
|
110
|
+
| `install` | Register Tokenade with your detected agent (hooks). Flags: `--only <agent>`, `--with-hook`, `--with-claude-md`, `--with-lsp-enforcement-hook`, `--with-brevity-skill`, `--dry-run`. |
|
|
111
|
+
| `uninstall [aliases]` | Remove the integration. `uninstall aliases` removes only the shell proxy. `--dry-run` to preview. |
|
|
112
|
+
| `login` | Activate via browser device-flow — no key to type. |
|
|
113
|
+
| `activate <key>` | Link this machine to a tokenade.net license (free or paid) with a key. |
|
|
114
|
+
| `upgrade` | Self-update in place (aliases: `self-update`). `--on`/`--off` toggles the 24h background updater. |
|
|
115
|
+
| `healthcheck` | Diagnose the install (aliases: `health`, `doctor`). |
|
|
116
|
+
| `detect` | Print the detected AI agent. |
|
|
117
|
+
| `discover` | Scan `~/.claude` logs, surface top tools + install suggestions. |
|
|
118
|
+
| `hooks <status\|tail [N]>` | Show installed hooks + recent activity / live tail. |
|
|
119
|
+
| `statusline` | One-line summary for `~/.claude/settings.json` `statusLine`. |
|
|
120
|
+
| `incompatibilities` | Check for other token tools that conflict with Tokenade and propose fixes (aliases: `competitors`, `scan`). |
|
|
121
|
+
|
|
122
|
+
### Code intelligence — read less, find faster
|
|
123
|
+
|
|
124
|
+
| Command | What it does |
|
|
125
|
+
|---|---|
|
|
126
|
+
| `map` | Print a compact codebase structure map. |
|
|
127
|
+
| `skeleton <file>` | Signatures-only skeleton of a file. |
|
|
128
|
+
| `query <symbol>` | Locate a symbol without reading files. |
|
|
129
|
+
| `impact <file>` | Refactor impact radius (dependents + symbols). |
|
|
130
|
+
| `semantic <query>` | Hybrid BM25 + dense, framework-aware search (aliases: `ask`, `sem`). |
|
|
131
|
+
| `file-search <path> <query>` | Per-file embedder lookup (`--k=N`). |
|
|
132
|
+
| `index` | Build & cache the codebase symbol index. |
|
|
133
|
+
| `watch` | Keep the index fresh: incremental reindex on every save. |
|
|
134
|
+
| `measure` | Report agent-loaded-file startup-token cost. |
|
|
135
|
+
| `review-context` | Token-budgeted code-review context (`--budget N`, `--base ref`). |
|
|
136
|
+
| `pack` | Budget-bounded repo export, one fenced bundle (`--budget N`, `--diff`, `--out FILE`). |
|
|
137
|
+
| `semantic-status` | Per-project state of `~/.tokenade/semantic.db`. |
|
|
138
|
+
|
|
139
|
+
### Run commands through compaction
|
|
140
|
+
|
|
141
|
+
| Command | What it does |
|
|
142
|
+
|---|---|
|
|
143
|
+
| `wrap <cmd>` | Run a command through the auto-detected compactor (default for noisy commands). |
|
|
144
|
+
| `proxy <cmd...>` | Run a command and emit compressed output (logs savings). |
|
|
145
|
+
| `raw <cmd...>` | Run a command **without** compaction (aliases: `bypass`, `noproxy`). |
|
|
146
|
+
| `filter <cmd...>` | Filter stdin as if it were produced by `<cmd>`. |
|
|
147
|
+
| `execute --lang L --script CODE` | Sandboxed "think-in-code" runner (bash/python/node/jq/awk/ruby/perl) — returns only stdout, redacted + compacted, keeps raw data out of context. |
|
|
148
|
+
| `shell-init` | Print the shell proxy fragment for your shell (`--shell …`, `--all`). |
|
|
149
|
+
| `shellwrap` | TTY-aware alias target for csh/tcsh + cmd.exe. |
|
|
150
|
+
|
|
151
|
+
### Format compactors
|
|
152
|
+
|
|
153
|
+
`auto <file|->` auto-detects the format and dispatches to the right compactor — **the one to call** if unsure. Or target a format directly (all read `<file>` or `-` for stdin):
|
|
154
|
+
|
|
155
|
+
| Domain | Commands |
|
|
156
|
+
|---|---|
|
|
157
|
+
| **Data & config** | `json` · `csv` · `tsv` · `yaml` · `ndjson` · `lock` · `sql` · `env` · `md` · `openapi` · `graphql` · `sourcemap` |
|
|
158
|
+
| **Logs & traces** | `logs` · `access` · `journal` · `container-logs` · `ghactions` · `stack` · `strace` · `tcpdump` · `metrics` |
|
|
159
|
+
| **Build / VCS / infra** | `diff` · `gitlog` · `archive` · `cargo-bench` · `jvmbuild` (aliases `mvn`/`gradle`/`sbt`) · `pkginstall` · `terraform` · `ansible` · `k8s-manifest` · `describe` · `systemd` |
|
|
160
|
+
| **System & network** | `table` · `disk` · `netstat` · `dns` · `vmstat` · `filelist` · `grep` |
|
|
161
|
+
| **Web & docs** | `web` · `serp` (aliases `google`/`bing`/`ddg`) · `snapshot` · `stealth` · `curl` · `docs` · `pdf` · `notebook` |
|
|
162
|
+
|
|
163
|
+
### Savings, receipts & reporting
|
|
164
|
+
|
|
165
|
+
| Command | What it does |
|
|
166
|
+
|---|---|
|
|
167
|
+
| `gain` | Cumulative token savings (`--json`, `--history`, `--by-source`). |
|
|
168
|
+
| `dashboard` | Screenshot-ready savings overview (aliases: `dash`, `stats`; `--html`/`--json`). |
|
|
169
|
+
| `receipt` | Signed, tamper-evident savings receipt over the local ledger (`--verify <file>`). |
|
|
170
|
+
| `cache-note` | Prompt-cache-aware framing of savings (net vs gross). |
|
|
171
|
+
| `mcp-stats` | MCP manifest rollup (servers/wrapped) + token-aware skills index. |
|
|
172
|
+
| `report` | Share redacted Tokenade + agent logs (consent-gated, typed phrase) to improve compression (`--dry-run`, `--message`, `--accept`). |
|
|
173
|
+
|
|
174
|
+
### Recovery & disclosure — get folded bytes back
|
|
175
|
+
|
|
176
|
+
| Command | What it does |
|
|
177
|
+
|---|---|
|
|
178
|
+
| `expand-ref <hash>` | Re-emit the verbatim bytes a compactor folded, by hash (alias: `xref`). |
|
|
179
|
+
| `expand-fold <id>` | Expand a folded log run back to its raw lines (alias: `xfold`). |
|
|
180
|
+
| `disclose <stash\|read\|search>` | Progressive disclosure: stash a big blob to a file, read/search by hash. |
|
|
181
|
+
| `search-stash <hash> <query>` | Fuzzy / natural-language recall within a stashed blob (BM25). |
|
|
182
|
+
| `turn-summary` | Extractive summary of a conversation turn (`--max N`). |
|
|
183
|
+
|
|
184
|
+
### Security & maintenance
|
|
185
|
+
|
|
186
|
+
| Command | What it does |
|
|
187
|
+
|---|---|
|
|
188
|
+
| `scrub-secret <substring>` | **Emergency:** purge a leaked credential from all Tokenade-owned files + caches (`--dry-run`). |
|
|
189
|
+
| `scrub-cache` | Re-apply built-in redaction patterns to every persisted byte (use after upgrade). |
|
|
190
|
+
| `redact <file…>` | Redact secret-shaped values to `<redacted>` in any file (`--in-place`, `--dry-run`). Streams line-by-line, idempotent. |
|
|
191
|
+
| `evict-stale` | Prune 7-day-stale cache rows + zero-saved noise, then VACUUM to reclaim disk. |
|
|
192
|
+
| `mistake <list\|add\|from-revert>` | Persistent "bad direction" memory (records git reverts). |
|
|
193
|
+
| `audit-claude-md [DIR]` | Audit the `CLAUDE.md` cascade for bloat + cross-file dupes (alias: `claudemd`). |
|
|
194
|
+
|
|
195
|
+
### Agent integration & config
|
|
196
|
+
|
|
197
|
+
| Command | What it does |
|
|
198
|
+
|---|---|
|
|
199
|
+
| `style <chat\|coding\|off>` | Response-style preamble injected before each prompt. |
|
|
200
|
+
| `read-mode` | Show the active fold read-mode ladder (`TOKENADE_READ_MODE`). |
|
|
201
|
+
| `unwrap-mcps` | Restore wrapped MCP servers to their original form (idempotent). |
|
|
202
|
+
| `mcp-proxy <bin>` | Transparent JSON-RPC proxy in front of an MCP server (written by `install`; internal). |
|
|
203
|
+
| `--version` | Print the installed version. |
|
|
204
|
+
|
|
205
|
+
> **User presets:** drop TOML files in `~/.config/tokenade/presets/` to add command-rewrite rules for your own CLIs (helm, ansible, internal scripts).
|
|
206
|
+
|
|
207
|
+
---
|
|
208
|
+
|
|
209
|
+
## 🧪 Examples & recipes
|
|
210
|
+
|
|
211
|
+
Real commands, real (trimmed) output — so both humans **and coding agents** know exactly what each command does and when to reach for it. Every compactor reads a file **or `-` for stdin**, so you can pipe: `some-noisy-cmd | tokenade auto -`.
|
|
212
|
+
|
|
213
|
+
### Setup & account
|
|
214
|
+
|
|
215
|
+
**`install`** — wire Tokenade into your agent (run once).
|
|
216
|
+
```text
|
|
217
|
+
$ tokenade install
|
|
218
|
+
✓ detected agent: claude-code
|
|
219
|
+
✓ hooks registered in ~/.claude/settings.json (Bash|Read + Edit/Write)
|
|
220
|
+
✓ shell proxy added to ~/.zshrc
|
|
221
|
+
── Incompatible tools — suggested fixes ── # only if any are found
|
|
222
|
+
✓ install complete — token savings start on your next prompt.
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
**`login`** — authorize this machine in your browser (no key to paste).
|
|
226
|
+
```text
|
|
227
|
+
$ tokenade login
|
|
228
|
+
→ opening https://tokenade.net/device?code=ABCD-1234 …
|
|
229
|
+
✓ this machine is now linked to your account (plan: free).
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
**`activate <key>`** — link a machine with a license key instead of the browser.
|
|
233
|
+
```text
|
|
234
|
+
$ tokenade activate tk_live_xxx
|
|
235
|
+
✓ activated — plan: pro · unlimited machines.
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
**`healthcheck`** (aliases `health`, `doctor`) — confirm everything is wired.
|
|
239
|
+
```text
|
|
240
|
+
$ tokenade healthcheck
|
|
241
|
+
─── tokenade healthcheck ───
|
|
242
|
+
OK binary at ~/.local/bin/tokenade (0.6.2)
|
|
243
|
+
OK gain ledger writable at ~/.tokenade/gain.jsonl
|
|
244
|
+
OK license active — plan: pro
|
|
245
|
+
OK agent detected: claude-code
|
|
246
|
+
OK hookwrap found in ~/.claude/settings.json
|
|
247
|
+
all checks passed
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
**`detect`** — print the detected agent. **`discover`** — scan your logs and suggest installs. **`hooks status`** — show installed hooks. **`statusline`** — one line for your editor status bar. **`upgrade`** — self-update; **`uninstall`** — remove cleanly.
|
|
251
|
+
```text
|
|
252
|
+
$ tokenade detect
|
|
253
|
+
claude-code
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
**`incompatibilities`** (aliases `competitors`, `scan`) — find other token tools that conflict with Tokenade and propose fixes.
|
|
257
|
+
```text
|
|
258
|
+
$ tokenade incompatibilities
|
|
259
|
+
── Found 1 potentially incompatible tool(s) ──
|
|
260
|
+
|
|
261
|
+
[redundant] rtk
|
|
262
|
+
Output-filtering CLI proxy (filters/compresses bash output)
|
|
263
|
+
→ binary at ~/.cargo/bin/rtk
|
|
264
|
+
fix: Tokenade subsumes rtk: `tokenade proxy <cmd>` does the same AND
|
|
265
|
+
auto-detects 48 formats — usually with better savings.
|
|
266
|
+
|
|
267
|
+
Run `tokenade install` to interactively migrate & fix the [redundant] /
|
|
268
|
+
[overlapping] tools (the fix-up pass is on by default; opt out with `--no-cleanup`).
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
### Code intelligence — read less, find faster
|
|
272
|
+
|
|
273
|
+
**`map`** — get the lay of the land without reading anything.
|
|
274
|
+
```text
|
|
275
|
+
$ tokenade map
|
|
276
|
+
4722 files, 49898 symbols indexed
|
|
277
|
+
by directory:
|
|
278
|
+
crates/tokenade-core/src/ 140 files, 4802 symbols
|
|
279
|
+
crates/tokenade-core/src/cmd/ 2637 files, 25741 symbols
|
|
280
|
+
crates/tokenade-cli/src/ 12 files, 1188 symbols
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
**`skeleton <file>`** — see a file's shape (signatures only), not its body.
|
|
284
|
+
```text
|
|
285
|
+
$ tokenade skeleton crates/tokenade-core/src/redact.rs
|
|
286
|
+
use regex::Regex;
|
|
287
|
+
// … 3 lines …
|
|
288
|
+
pub fn redact(s: &str) -> String {
|
|
289
|
+
// … 8 lines …
|
|
290
|
+
pub fn redact_argv(argv: &[String]) -> Vec<String> {
|
|
291
|
+
struct PatternSet {
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
**`query <symbol>`** — jump straight to where a symbol is defined.
|
|
295
|
+
```text
|
|
296
|
+
$ tokenade query redact
|
|
297
|
+
module redact — crates/tokenade-core/src/lib.rs:137
|
|
298
|
+
fn redact — crates/tokenade-core/src/redact.rs:28
|
|
299
|
+
fn redact_argv — crates/tokenade-core/src/redact.rs:38
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
**`impact <file>`** — before a refactor, see what depends on it.
|
|
303
|
+
```text
|
|
304
|
+
$ tokenade impact crates/tokenade-core/src/redact.rs
|
|
305
|
+
crates/tokenade-core/src/redact.rs declares (47 symbols): fn redact, redact_argv, …
|
|
306
|
+
dependents (3):
|
|
307
|
+
crates/tokenade-core/src/debug_log.rs
|
|
308
|
+
crates/tokenade-core/src/sandbox.rs
|
|
309
|
+
crates/tokenade-core/tests/proptest_compactors.rs
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
**`semantic "<query>"`** (aliases `ask`, `sem`) — find code by meaning, not by string match.
|
|
313
|
+
```text
|
|
314
|
+
$ tokenade semantic "where do we redact secrets before logging"
|
|
315
|
+
1. crates/tokenade-core/src/redact.rs:28 fn redact — applies all secret patterns
|
|
316
|
+
2. crates/tokenade-core/src/gain.rs:231 record_inner — redacts label before write
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
**`measure`** — see the per-session token cost of your agent-loaded files (CLAUDE.md, etc.).
|
|
320
|
+
```text
|
|
321
|
+
$ tokenade measure
|
|
322
|
+
file tokens
|
|
323
|
+
CLAUDE.md 642
|
|
324
|
+
AGENTS.md 385
|
|
325
|
+
PER SESSION (one agent loads one) 642
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
Also: **`index`** (build the cache up front) · **`watch`** (keep it fresh on save) · **`file-search <path> <query>`** (search within one file) · **`review-context --base main`** (budget-bounded diff context for reviews) · **`pack --budget 8000`** (export a repo slice as one bundle) · **`semantic-status`** (index state).
|
|
329
|
+
|
|
330
|
+
### Run commands through compaction
|
|
331
|
+
|
|
332
|
+
**`wrap <cmd>`** — the default: run any noisy command, get the compacted output.
|
|
333
|
+
```text
|
|
334
|
+
$ tokenade wrap 'kubectl get pods -A'
|
|
335
|
+
NAMESPACE NAME READY STATUS RESTARTS AGE
|
|
336
|
+
default web-xx 1/1 Running 0 2h
|
|
337
|
+
… 47 rows · grouped by status: Running=46, Pending=1
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
**`execute --lang <L> --script <code>`** — "think in code": run a snippet in a sandbox, get **only stdout** back (raw data never enters your context).
|
|
341
|
+
```text
|
|
342
|
+
$ tokenade execute --lang python --script 'print(sum(range(101)))'
|
|
343
|
+
5050
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
**`proxy <cmd…>`** — like `wrap`, logs savings to your ledger. **`raw <cmd…>`** — run *without* compaction (escape hatch). **`filter <cmd…>`** — treat stdin as if produced by `<cmd>`. **`shell-init`** — print the shell fragment that auto-wraps noisy commands:
|
|
347
|
+
```text
|
|
348
|
+
$ eval "$(tokenade shell-init)" # add to ~/.zshrc — git/cargo/kubectl/… auto-compact
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
### Format compactors
|
|
352
|
+
|
|
353
|
+
`tokenade auto <file|->` auto-detects and dispatches — **the one to reach for**. Or call a format directly. All read a file or `-` (stdin).
|
|
354
|
+
|
|
355
|
+
**`json`** — folds arrays of objects into a compact TOON table.
|
|
356
|
+
```text
|
|
357
|
+
$ tokenade json api.json
|
|
358
|
+
{ "page": 1, "total": 20, "users": "<see [fold @ users] block below>" }
|
|
359
|
+
|
|
360
|
+
[fold @ users] [×20 rows · toon] active,id,name,role
|
|
361
|
+
true,1,"u1","user"
|
|
362
|
+
…
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
**`logs`** — collapses runs of templated-equal lines (timestamps/IDs normalised).
|
|
366
|
+
```text
|
|
367
|
+
$ tokenade logs app.log # 200 lines →
|
|
368
|
+
<TS> INFO worker job=<N> status=done ms=<N> [×200 lines · e.g. job=1, ms=11]
|
|
369
|
+
```
|
|
370
|
+
|
|
371
|
+
**`diff`** — caps context, folds multi-file patches.
|
|
372
|
+
```text
|
|
373
|
+
$ git diff | tokenade diff -
|
|
374
|
+
diff --git a/redact.rs b/redact.rs
|
|
375
|
+
@@ -175,17 +175,24 @@ fn patterns() …
|
|
376
|
+
…2 context line(s) elided…
|
|
377
|
+
+ r#"…(?:'[^']*'|"[^"]*"|[^\s]+)"#,
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
**`csv` / `tsv`** — drop empty columns, cap rows, truncate cells.
|
|
381
|
+
```text
|
|
382
|
+
$ tokenade csv users.csv
|
|
383
|
+
id,name,email,signup,plan
|
|
384
|
+
1,Ada,ada@x.io,2024-01-02,pro
|
|
385
|
+
# 1 empty column(s) dropped
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
**`yaml`** — strips `managedFields`, `lastTransitionTime`, and other k8s noise.
|
|
389
|
+
```text
|
|
390
|
+
$ kubectl get deploy web -o yaml | tokenade yaml -
|
|
391
|
+
metadata:
|
|
392
|
+
name: web
|
|
393
|
+
managedFields: <dropped>
|
|
394
|
+
status:
|
|
395
|
+
conditions:
|
|
396
|
+
- type: Available
|
|
397
|
+
lastTransitionTime: <dropped>
|
|
398
|
+
```
|
|
399
|
+
|
|
400
|
+
**`env`** — compact an env dump **and redact secrets**.
|
|
401
|
+
```text
|
|
402
|
+
$ env | tokenade env -
|
|
403
|
+
AWS_SECRET_ACCESS_KEY=<redacted 17 chars>
|
|
404
|
+
DATABASE_URL=<redacted 21 chars>
|
|
405
|
+
# [tokenade:env] 2 secret(s) redacted
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
**`table`** — any columnar CLI output (`docker ps`, `kubectl get`, `ps aux`).
|
|
409
|
+
```text
|
|
410
|
+
$ docker ps | tokenade table -
|
|
411
|
+
IMAGE STATUS PORTS NAMES
|
|
412
|
+
nginx:latest Up 2 hours 0.0.0.0:80->80/tcp web
|
|
413
|
+
```
|
|
414
|
+
|
|
415
|
+
**`stack`** — fold framework frames in a stack trace, keep your code + the error.
|
|
416
|
+
```text
|
|
417
|
+
$ tokenade stack trace.txt
|
|
418
|
+
/app/main.py:42 in <module> run()
|
|
419
|
+
/app/db.py:17 in query raise ValueError("bad row")
|
|
420
|
+
ValueError: bad row
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
**`dns`** — keep the ANSWER section of `dig`/`host`, drop the pseudo-sections.
|
|
424
|
+
```text
|
|
425
|
+
$ dig example.com | tokenade dns -
|
|
426
|
+
example.com A NOERROR: 2 answers
|
|
427
|
+
example.com. IN A 93.184.216.34
|
|
428
|
+
```
|
|
429
|
+
|
|
430
|
+
**`access`** — group an nginx/Apache log by method + path + status.
|
|
431
|
+
```text
|
|
432
|
+
$ tokenade access access.log
|
|
433
|
+
3 request(s) · status counts: 200=2, 500=1
|
|
434
|
+
2× GET /api/users [200]
|
|
435
|
+
1× POST /api/pay [500]
|
|
436
|
+
```
|
|
437
|
+
|
|
438
|
+
The full family — same shape (`tokenade <name> file|-`), each tuned to one format:
|
|
439
|
+
|
|
440
|
+
| Command | Common use → output |
|
|
441
|
+
|---|---|
|
|
442
|
+
| `md` | Markdown → drop front-matter, fold huge code blocks, strip ref-links |
|
|
443
|
+
| `ndjson` | JSONL cloud logs → severity histogram + key-union schema |
|
|
444
|
+
| `lock` | `Cargo.lock`/`yarn.lock` → flat `name@version` list |
|
|
445
|
+
| `sql` | psql/mysql result table → capped, borderless rows |
|
|
446
|
+
| `openapi` / `graphql` | API spec → endpoint / type + field list, no bodies |
|
|
447
|
+
| `sourcemap` | `.map` → sources list, drop mappings blob |
|
|
448
|
+
| `gitlog` | `git log` → `hash author subject` per commit |
|
|
449
|
+
| `archive` | `tar -tvf`/`unzip -l` → names only, perms/size dropped |
|
|
450
|
+
| `terraform` | `terraform plan` → fold "Refreshing state", drop unchanged attrs |
|
|
451
|
+
| `k8s-manifest` | `helm template` → per-kind summary, drop spec bodies |
|
|
452
|
+
| `describe` | `kubectl describe` → identity + status + warnings |
|
|
453
|
+
| `systemd` | `systemctl status` → Active + Mem/CPU + ERR/WARN tail |
|
|
454
|
+
| `journal` | `journalctl`/`dmesg` → group by service, surface ERR/WARN |
|
|
455
|
+
| `container-logs` | `kubectl logs --prefix` → per-stream fold |
|
|
456
|
+
| `grep` | `grep -r`/`rg` → group by file, top hits |
|
|
457
|
+
| `filelist` | `find`/`tree`/`ls -laR` → per-dir counts + ext histogram |
|
|
458
|
+
| `disk` | `du -h` → top entries by size |
|
|
459
|
+
| `netstat` | `ss`/`netstat -an` → per-state counts, top LISTEN |
|
|
460
|
+
| `vmstat` | `vmstat 1 N` → per-column min/mean/max + peaks |
|
|
461
|
+
| `strace` | `strace`/`ltrace` → syscall counts + error summary |
|
|
462
|
+
| `tcpdump` | `tcpdump` text → per-flow aggregation |
|
|
463
|
+
| `metrics` | Prometheus `/metrics` → grouped, capped label combos |
|
|
464
|
+
| `curl` | `curl -v` → fold TLS handshake + cert chain |
|
|
465
|
+
| `cargo-bench` | criterion output → Δ-vs-baseline + regressions |
|
|
466
|
+
| `jvmbuild` (`mvn`/`gradle`) | Maven/Gradle build → per-phase outcome |
|
|
467
|
+
| `pkginstall` | pip/npm/cargo install → drop progress, keep installed list |
|
|
468
|
+
| `ghactions` | GH Actions log → strip timestamps, keep `##[error]` |
|
|
469
|
+
| `ansible` | `ansible-playbook` → per-task outcome, failures verbatim |
|
|
470
|
+
| `web` | HTML page → clean markdown |
|
|
471
|
+
| `serp` (`google`/`bing`/`ddg`) | raw search-results HTML → title/url/snippet |
|
|
472
|
+
| `snapshot` | Playwright a11y-tree → compact tree |
|
|
473
|
+
| `stealth` | page → detect Cloudflare/hCaptcha + remediation |
|
|
474
|
+
| `docs` / `pdf` | `.docx`/`.xlsx`/`.pptx`/`.pdf` → plain text |
|
|
475
|
+
| `notebook` | Jupyter `.ipynb` → code + outputs, strip base64 images |
|
|
476
|
+
|
|
477
|
+
### Savings, receipts & reporting
|
|
478
|
+
|
|
479
|
+
**`gain`** — how many tokens you've saved.
|
|
480
|
+
```text
|
|
481
|
+
$ tokenade gain
|
|
482
|
+
operations: 136
|
|
483
|
+
tokens: 1,333,261 → 99,204
|
|
484
|
+
saved: 1,234,057 (92.6%)
|
|
485
|
+
by op: auto:json 9× −36% · auto-compact 4× −96% · …
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
**`dashboard`** (aliases `dash`, `stats`) — the big-picture banner (`--html` / `--json`).
|
|
489
|
+
```text
|
|
490
|
+
$ tokenade dashboard
|
|
491
|
+
25.5M measured · 97.9% · 1571 ops
|
|
492
|
+
```
|
|
493
|
+
|
|
494
|
+
**`receipt`** — a signed, tamper-evident savings receipt (`--verify <file>` to check one). **`cache-note --saved N`** — frames savings honestly (net vs gross of prompt-cache). **`mcp-stats`** — rollup of your MCP servers + skills index.
|
|
495
|
+
|
|
496
|
+
**`report`** — share redacted logs (consent-gated) so we can improve compression. A typed-phrase contract is required before anything leaves your machine; `--dry-run` packs locally and uploads nothing.
|
|
497
|
+
```text
|
|
498
|
+
$ tokenade report --dry-run
|
|
499
|
+
collected 42 transcripts since install · redacted 17 secrets
|
|
500
|
+
wrote ~/.tokenade/last-report.zip (4.2 MB) — nothing uploaded
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
### Recovery & disclosure — get folded bytes back
|
|
504
|
+
|
|
505
|
+
Compaction is never lossy-by-surprise: every fold is recoverable.
|
|
506
|
+
```text
|
|
507
|
+
$ tokenade expand-ref 927a921273b2 # re-emit the exact bytes a banner folded
|
|
508
|
+
$ tokenade expand-fold app-3f2a # expand one folded log run to raw lines
|
|
509
|
+
$ tokenade disclose read <hash> # read a stashed big blob
|
|
510
|
+
$ tokenade search-stash <hash> "timeout" # fuzzy recall inside a stash
|
|
511
|
+
$ tokenade turn-summary --max 5 chat.txt # top-5 salient sentences of a turn
|
|
512
|
+
```
|
|
513
|
+
|
|
514
|
+
### Security & maintenance
|
|
515
|
+
|
|
516
|
+
```text
|
|
517
|
+
$ tokenade scrub-secret 'ak_98_leakedkey' --dry-run # purge a leaked credential everywhere
|
|
518
|
+
$ tokenade scrub-cache # re-apply redaction to all persisted bytes (after upgrade)
|
|
519
|
+
$ tokenade redact secrets.log --dry-run # count secret-shaped values in a file; write nothing
|
|
520
|
+
$ tokenade evict-stale # prune 7-day-stale cache rows + VACUUM
|
|
521
|
+
$ tokenade audit-claude-md # find bloat + dupes across your CLAUDE.md cascade
|
|
522
|
+
$ tokenade mistake from-revert # record a 'bad direction' from a git revert
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
### Agent integration & config
|
|
526
|
+
|
|
527
|
+
**`read-mode`** — show (and tune via `TOKENADE_READ_MODE`) how aggressively file reads are folded.
|
|
528
|
+
```text
|
|
529
|
+
$ tokenade read-mode
|
|
530
|
+
active: task (default)
|
|
531
|
+
ladder: aggressive 0.4× · → task 1.0× · reference 3.0× · entropy
|
|
532
|
+
```
|
|
533
|
+
|
|
534
|
+
**`style <chat|coding|off>`** — inject a response-style preamble before each prompt. **`unwrap-mcps`** — restore wrapped MCP servers. **`--version`** — print the version.
|
|
535
|
+
```text
|
|
536
|
+
$ tokenade style coding # terse, code-first replies
|
|
537
|
+
current style: coding
|
|
538
|
+
```
|
|
539
|
+
|
|
540
|
+
> **User-defined presets:** drop TOML files in `~/.config/tokenade/presets/` to teach Tokenade your own CLIs (helm, ansible, internal scripts) — same compaction, your commands.
|
|
541
|
+
|
|
542
|
+
---
|
|
543
|
+
|
|
544
|
+
<div align="center">
|
|
27
545
|
|
|
28
|
-
|
|
546
|
+
**[tokenade.net](https://tokenade.net)** · Made for people who'd rather spend tokens on shipping than on overhead.
|
|
29
547
|
|
|
30
|
-
|
|
31
|
-
runs locally; its agent hooks live in your editor/agent config (inspectable,
|
|
32
|
-
reversible).
|
|
548
|
+
</div>
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tokenade/cli",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.2",
|
|
4
4
|
"description": "Tokenade — cut your AI coding agent's token bill. Installs the Tokenade CLI (a local, paid token-reduction tool; activate via your browser).",
|
|
5
5
|
"homepage": "https://tokenade.net",
|
|
6
6
|
"license": "UNLICENSED",
|