@tokmizer/plugin 1.0.1 → 1.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/.codex-plugin/plugin.json +1 -1
- package/README.md +259 -272
- package/dist/bin/tkr-codex-hook.js +2 -2
- package/dist/bin/tkr-daemon.js +2 -2
- package/dist/bin/tkr-exec +2 -2
- package/dist/bin/tkr-exec.js +2 -2
- package/dist/bin/tkr-processor.js +2 -2
- package/dist/bin/tkr-shim.js +2 -2
- package/dist/bin/tkr.js +2 -2
- package/dist/hooks-host/post-tool-use.js +2 -2
- package/dist/hooks-host/reducer-stop.js +1 -1
- package/dist/hooks-host/session-recap.js +1 -1
- package/dist/index.js +2 -2
- package/dist/native/manifest.json +12 -12
- package/dist/native/single.json +12 -12
- package/dist/native/tkr-exec-darwin-arm64 +0 -0
- package/dist/native/tkr-exec-darwin-x64 +0 -0
- package/dist/native/tkr-exec-linux-arm64 +0 -0
- package/dist/native/tkr-exec-linux-x64 +0 -0
- package/dist/native/tkr-exec-win32-arm64.exe +0 -0
- package/dist/native/tkr-exec-win32-x64.exe +0 -0
- package/package.json +7 -7
package/README.md
CHANGED
|
@@ -1,172 +1,265 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
|
|
1
3
|
# Tokmizer
|
|
2
4
|
|
|
3
|
-
|
|
5
|
+
**Your AI coding agent reads up to 93% fewer tokens of command output, and keeps every error.**
|
|
6
|
+
|
|
7
|
+
Free · no account · Claude Code, Codex, Cursor, Gemini CLI, Copilot and 16 more agents · macOS, Linux, Windows
|
|
8
|
+
|
|
9
|
+
[](https://www.npmjs.com/package/@tokmizer/plugin)
|
|
10
|
+
[](https://github.com/norwelltw/Tokmizer/blob/main/packages/plugin/LICENSES.md)
|
|
11
|
+
[](https://nodejs.org)
|
|
12
|
+
|
|
13
|
+
[Website](https://tokmizer.com) · [Docs](https://tokmizer.com/en/docs) · [Benchmark vs rtk](https://tokmizer.com/en/compare/rtk) · [CI](https://tokmizer.com/en/ci)
|
|
14
|
+
|
|
15
|
+
</div>
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
Every `npm test`, `cargo build`, `git log` or `grep -rn` your agent runs pours its full output into the context window: passing tests, progress bars, compile lines, the same warning 400 times. Tokmizer sits between the command and the model and hands over what matters: the failures, the diagnostics, the result. The full output stays one command away.
|
|
20
|
+
|
|
21
|
+
## Before and after
|
|
22
|
+
|
|
23
|
+
`cargo test` with three failing tests ([fixture](https://github.com/norwelltw/Tokmizer/blob/main/packages/plugin/src/__tests__/fixtures/test-failures/cargo.txt)).
|
|
24
|
+
|
|
25
|
+
**Before:** 42 lines, 1,102 bytes. Compile lines, every passing test, panic headers, a backtrace hint, and the failure list printed twice.
|
|
26
|
+
|
|
27
|
+
**After:** 11 lines, 382 bytes.
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
× tests::truthy src/lib.rs:12:27
|
|
31
|
+
assertion failed: add(0, 0) > 0
|
|
32
|
+
× tests::big src/lib.rs:10:24
|
|
33
|
+
assertion `left == right` failed: adding 6 and 1
|
|
34
|
+
left: 8
|
|
35
|
+
right: 7
|
|
36
|
+
× tests::empty src/lib.rs:2:34
|
|
37
|
+
index out of bounds: the len is 0 but the index is 0
|
|
38
|
+
test result: FAILED. 4 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out
|
|
39
|
+
error: test failed, to rerun pass `--lib`
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Every assertion, location and value is still there.
|
|
4
43
|
|
|
5
|
-
|
|
44
|
+
## Numbers
|
|
45
|
+
|
|
46
|
+
68 real command outputs replayed offline through each tool's Claude Code hooks (Tokmizer 0.9.71 and rtk 1.0.1 with default settings, o200k_base tokenizer, median of 3 runs, 2026-10-19):
|
|
47
|
+
|
|
48
|
+
| | Tokmizer, Claude Code default mode | Tokmizer, commands rewritten | rtk 1.0.1 |
|
|
49
|
+
| --- | ---: | ---: | ---: |
|
|
50
|
+
| Tokens removed | **93.0%** | **94.3%** | 55.5% |
|
|
51
|
+
| Failure details kept, out of 307 | **307** | **307** | 133 |
|
|
52
|
+
| Failing runs that lost a detail, out of 26 | **0** | **0** | 10 |
|
|
53
|
+
| Cuts without a marker | none found | none found | found |
|
|
54
|
+
|
|
55
|
+
- Claude Code's own `Read`, `Grep` and `Glob` tools are shortened too: large file reads −62% to −91%, `grep` −27% to −94%, `glob` −70%. Smaller code files stay whole.
|
|
56
|
+
- The hook that runs before each command takes about 3 ms with the native launcher, against 11 to 17 ms for rtk (measured 2026-10-16).
|
|
57
|
+
|
|
58
|
+
These figures count command output, not your bill, task success or speed. Small outputs gain less: 51% on the outputs under 30,000 characters. Method, every case and the raw JSON: [tokmizer.com/en/compare/rtk](https://tokmizer.com/en/compare/rtk). A public benchmark you can replay yourself: [BENCHMARK.md](https://github.com/norwelltw/Tokmizer/blob/main/BENCHMARK.md).
|
|
6
59
|
|
|
7
60
|
## Install
|
|
8
61
|
|
|
9
|
-
|
|
62
|
+
### Claude Code
|
|
10
63
|
|
|
11
64
|
```
|
|
12
65
|
/plugin marketplace add norwelltw/Tokmizer
|
|
13
66
|
/plugin install tokmizer@tokmizer
|
|
14
67
|
```
|
|
15
68
|
|
|
16
|
-
|
|
69
|
+
Restart Claude Code once (hooks load at startup), then run `/tkr-status`. You should see `Status active`.
|
|
17
70
|
|
|
18
|
-
|
|
71
|
+
### Every other agent
|
|
72
|
+
|
|
73
|
+
Install the CLI, then let `tkr setup --all` hook every agent it finds on this machine:
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
curl -fsSL https://tokmizer.com/install.sh | bash
|
|
77
|
+
tkr setup --all
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
The script uses npm when Node.js 20.11+ is installed, and otherwise downloads a standalone build for macOS or Linux after checking its signature and SHA-256. Your shell profile is not touched.
|
|
81
|
+
|
|
82
|
+
<table>
|
|
83
|
+
<tr><td><b>npm</b></td><td>
|
|
19
84
|
|
|
20
85
|
```bash
|
|
21
86
|
npm install -g @tokmizer/plugin
|
|
22
87
|
tkr setup --all
|
|
23
88
|
```
|
|
24
89
|
|
|
25
|
-
|
|
90
|
+
</td></tr>
|
|
91
|
+
<tr><td><b>Homebrew</b></td><td>
|
|
26
92
|
|
|
27
93
|
```bash
|
|
28
|
-
|
|
94
|
+
brew install norwelltw/tokmizer/tokmizer
|
|
95
|
+
tkr setup --all
|
|
29
96
|
```
|
|
30
97
|
|
|
31
|
-
|
|
98
|
+
</td></tr>
|
|
99
|
+
<tr><td><b>Windows</b><br>PowerShell</td><td>
|
|
32
100
|
|
|
33
101
|
```powershell
|
|
34
102
|
npm.cmd install -g @tokmizer/plugin
|
|
35
103
|
tkr.cmd setup --all
|
|
36
104
|
```
|
|
37
105
|
|
|
38
|
-
|
|
106
|
+
</td></tr>
|
|
107
|
+
</table>
|
|
39
108
|
|
|
40
|
-
|
|
109
|
+
`tkr setup` without `--all` lists what it found and asks. Restart each agent after setup, then run `tkr status`. Installed with npm or Homebrew, `tkr` updates itself 24 hours after a version ships; `tkr auto-update off` stops that.
|
|
41
110
|
|
|
42
|
-
|
|
43
|
-
tkr setup-codex
|
|
44
|
-
```
|
|
111
|
+
### Supported agents
|
|
45
112
|
|
|
46
|
-
|
|
113
|
+
| Agent | Setup | Through |
|
|
114
|
+
| --- | --- | --- |
|
|
115
|
+
| Claude Code | plugin above, or `tkr setup-claude` | native hooks |
|
|
116
|
+
| Codex | `tkr setup-codex`, then approve the hook in `/hooks` | native hooks |
|
|
117
|
+
| Cursor | `tkr setup-cursor` | `preToolUse` hook |
|
|
118
|
+
| Gemini CLI | `tkr setup-gemini` | `BeforeTool` hook |
|
|
119
|
+
| GitHub Copilot, VS Code and CLI | `tkr setup-copilot` | `PreToolUse` hook |
|
|
120
|
+
| Factory Droid | `tkr setup-droid` | `PreToolUse` hook |
|
|
121
|
+
| Trae, Trae CN | `tkr setup-trae` | `PreToolUse` hook |
|
|
122
|
+
| Qwen Code | `tkr setup-qwen` | `PreToolUse` hook |
|
|
123
|
+
| Crush | `tkr setup-crush` | `PreToolUse` hook |
|
|
124
|
+
| Junie CLI | `tkr setup-junie` | `PreToolUse` hook |
|
|
125
|
+
| ZCode | `tkr setup-zcode` | `PreToolUse` hook |
|
|
126
|
+
| Grok Build 1.0.4+ | `tkr setup-grok` | `PreToolUse` hook |
|
|
127
|
+
| Antigravity | `tkr setup-antigravity` | `PreToolUse` hook |
|
|
128
|
+
| Mistral Vibe | `tkr setup-vibe` | `pre_tool` hook |
|
|
129
|
+
| OpenCode | `tkr setup-opencode` | plugin |
|
|
130
|
+
| Kilo Code, CLI and VS Code | `tkr setup-kilo` | plugin |
|
|
131
|
+
| Amp | `tkr setup-amp` | plugin |
|
|
132
|
+
| OpenClaw | `tkr setup-openclaw` | plugin |
|
|
133
|
+
| Hermes v2026.8.16+ | `tkr setup-hermes` | plugin |
|
|
134
|
+
| Pi | `tkr setup-pi` | extension |
|
|
135
|
+
| Oh My Pi | `tkr setup-omp` | extension |
|
|
136
|
+
| Aider, Cline, Windsurf | `tkr wrap aider` | command shims, for that session only |
|
|
137
|
+
|
|
138
|
+
Every setup command keeps your other hooks and settings, saves the file as it was in `~/.tokmizer/backup/config/`, and takes itself out with `--remove`. If `tkr` is missing, slow or fails, the command runs exactly as typed.
|
|
139
|
+
|
|
140
|
+
<details>
|
|
141
|
+
<summary>Files each setup command edits</summary>
|
|
142
|
+
|
|
143
|
+
| Command | File |
|
|
144
|
+
| --- | --- |
|
|
145
|
+
| `tkr setup-cursor` | `~/.cursor/hooks.json` |
|
|
146
|
+
| `tkr setup-gemini` | `~/.gemini/settings.json` |
|
|
147
|
+
| `tkr setup-copilot` | `~/.copilot/hooks/tokmizer.json`, or `$COPILOT_HOME/hooks/` |
|
|
148
|
+
| `tkr setup-droid` | `~/.factory/hooks.json`, or `~/.factory/settings.json` when your own `PreToolUse` hooks live there |
|
|
149
|
+
| `tkr setup-trae` | `~/.trae/hooks.json`, plus `~/.trae-cn/hooks.json` when Trae CN is installed |
|
|
150
|
+
| `tkr setup-qwen` | `~/.qwen/settings.json`, or `$QWEN_HOME` |
|
|
151
|
+
| `tkr setup-crush` | `~/.config/crush/crush.json`; a `crush.json` with comments is left untouched and reported |
|
|
152
|
+
| `tkr setup-junie` | `~/.junie/config.json` |
|
|
153
|
+
| `tkr setup-zcode` | `~/.zcode/cli/config.json` |
|
|
154
|
+
| `tkr setup-grok` | `~/.grok/hooks/tokmizer.json` |
|
|
155
|
+
| `tkr setup-antigravity` | `~/.gemini/config/hooks.json` |
|
|
156
|
+
| `tkr setup-vibe` | `~/.vibe/hooks.toml` |
|
|
157
|
+
| `tkr setup-opencode` | `~/.config/opencode/plugins/tokmizer.js` |
|
|
158
|
+
| `tkr setup-kilo` | `~/.config/kilo/plugin/tokmizer.js` |
|
|
159
|
+
| `tkr setup-amp` | `~/.config/amp/plugins/tokmizer.js` |
|
|
160
|
+
| `tkr setup-openclaw` | `~/.openclaw/extensions/tokmizer/`, then `openclaw plugins enable tokmizer` |
|
|
161
|
+
| `tkr setup-hermes` | `~/.hermes/plugins/tokmizer/`, then `hermes plugins enable tokmizer` |
|
|
162
|
+
| `tkr setup-pi` | `~/.pi/agent/extensions/tokmizer.js` |
|
|
163
|
+
| `tkr setup-omp` | `~/.omp/agent/extensions/tokmizer.js` |
|
|
47
164
|
|
|
48
|
-
|
|
165
|
+
`XDG_CONFIG_HOME`, `VIBE_HOME`, `PI_CODING_AGENT_DIR` and `HERMES_HOME` are respected.
|
|
49
166
|
|
|
50
|
-
|
|
167
|
+
</details>
|
|
51
168
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
tkr setup-gemini
|
|
55
|
-
```
|
|
169
|
+
<details>
|
|
170
|
+
<summary>Hooks copied between agents, Codex <code>/import</code></summary>
|
|
56
171
|
|
|
57
|
-
|
|
172
|
+
Codex `/import` copies your Claude Code hooks, and config files get copied by hand. Tokmizer hooks are safe to bring along: each one reads which agent is calling it and answers in that agent's format; if the input mixes two agents, it changes nothing. Afterwards run `tkr setup --all` or `tkr doctor --fix`: each agent gets its own hook, the copied ones are removed, your own hooks stay, and every changed file is backed up.
|
|
58
173
|
|
|
59
|
-
|
|
174
|
+
If Codex reports `PreToolUse hook returned updatedInput without permissionDecision:allow`, it is running Claude Code hooks. Run `tkr doctor --fix`, approve the Tokmizer hook in `/hooks` and restart Codex.
|
|
60
175
|
|
|
61
|
-
|
|
62
|
-
| --- | --- | --- |
|
|
63
|
-
| OpenCode | `tkr setup-opencode` | plugin `~/.config/opencode/plugins/tokmizer.js` |
|
|
64
|
-
| Kilo Code (CLI and VS Code extension) | `tkr setup-kilo` | plugin `~/.config/kilo/plugin/tokmizer.js` |
|
|
65
|
-
| Amp | `tkr setup-amp` | plugin `~/.config/amp/plugins/tokmizer.js` |
|
|
66
|
-
| Mistral Vibe | `tkr setup-vibe` | `pre_tool` hook in `~/.vibe/hooks.toml` |
|
|
67
|
-
| Antigravity | `tkr setup-antigravity` | `PreToolUse` hook in `~/.gemini/config/hooks.json` |
|
|
68
|
-
| OpenClaw | `tkr setup-openclaw` | plugin `~/.openclaw/extensions/tokmizer/`, then `openclaw plugins enable tokmizer` |
|
|
69
|
-
| Pi | `tkr setup-pi` | extension `~/.pi/agent/extensions/tokmizer.js` |
|
|
70
|
-
| Oh My Pi | `tkr setup-omp` | extension `~/.omp/agent/extensions/tokmizer.js` |
|
|
71
|
-
| Hermes | `tkr setup-hermes` | plugin `~/.hermes/plugins/tokmizer/`, then `hermes plugins enable tokmizer` (Hermes v2026.8.16 or newer) |
|
|
176
|
+
</details>
|
|
72
177
|
|
|
73
|
-
|
|
178
|
+
## How it works
|
|
74
179
|
|
|
75
|
-
|
|
180
|
+
1. **Before a command runs**, the agent's hook asks Tokmizer whether to route it. If so, it runs through `tkr-exec`, which picks a built-in rule for it: `cargo`, `pytest`, `vitest`, `jest`, `go test`, `gradle`, `git`, `docker`, `kubectl`, `terraform` and many more.
|
|
181
|
+
2. **The rule shortens the output.** Passing tests, progress, repeated lines and boilerplate go; failures, diagnostics, locations and the exit code stay. A shortened output is never longer than the original.
|
|
182
|
+
3. **A copy of the full output** stays on your machine. When something was cut, the agent sees a `[tkr restore <hash>]` hint and can fetch it back.
|
|
76
183
|
|
|
77
|
-
|
|
78
|
-
tkr setup-copilot
|
|
79
|
-
tkr setup-droid
|
|
80
|
-
tkr setup-trae
|
|
81
|
-
tkr setup-qwen
|
|
82
|
-
tkr setup-crush
|
|
83
|
-
tkr setup-junie
|
|
84
|
-
tkr setup-zcode
|
|
85
|
-
tkr setup-grok
|
|
86
|
-
```
|
|
184
|
+
### Your permission rules stay in force
|
|
87
185
|
|
|
88
|
-
|
|
89
|
-
|---|---|---|
|
|
90
|
-
| `tkr setup-copilot` | Copilot in VS Code and Copilot CLI | `~/.copilot/hooks/tokmizer.json` (or `$COPILOT_HOME/hooks/`) |
|
|
91
|
-
| `tkr setup-droid` | Factory Droid | `~/.factory/hooks.json`, or `~/.factory/settings.json` when your own `PreToolUse` hooks live there |
|
|
92
|
-
| `tkr setup-trae` | Trae | `~/.trae/hooks.json`, plus `~/.trae-cn/hooks.json` when Trae CN is installed |
|
|
93
|
-
| `tkr setup-qwen` | Qwen Code | `~/.qwen/settings.json` (or `$QWEN_HOME`) |
|
|
94
|
-
| `tkr setup-crush` | Crush | `~/.config/crush/crush.json` (or `$XDG_CONFIG_HOME/crush/`) |
|
|
95
|
-
| `tkr setup-junie` | Junie CLI | `~/.junie/config.json` |
|
|
96
|
-
| `tkr setup-zcode` | ZCode | `~/.zcode/cli/config.json` (`hooks.events.PreToolUse`) |
|
|
97
|
-
| `tkr setup-grok` | Grok Build 1.0.4 or newer | `~/.grok/hooks/tokmizer.json` |
|
|
186
|
+
Agents check their allow, ask and deny rules against the command text. Tokmizer reads those rules and leaves a command untouched whenever one of your deny or ask rules matches it, so `git push` or `rm` run as typed and still ask you. For agents whose rules it cannot read, only inspection, build and test commands are routed. Shell functions, aliases and builtins always run as typed, and if a settings file cannot be read, nothing is routed.
|
|
98
187
|
|
|
99
|
-
|
|
188
|
+
<details>
|
|
189
|
+
<summary>Claude Code permission modes</summary>
|
|
100
190
|
|
|
101
|
-
|
|
191
|
+
| Mode | What Tokmizer does |
|
|
192
|
+
| --- | --- |
|
|
193
|
+
| Default | Build and test commands (`npm test`, `cargo test`, `pytest`, `./gradlew build`…) are shortened before they run; Claude Code asks for them once, as it would anyway. Read-only commands (`ls`, `cat`, `git status`) run as typed, so Claude Code still approves them on its own, and their output is shortened once they succeed. Claude Code runs that after-command hook only for successful commands, so a failing read-only command reaches Claude in full. |
|
|
194
|
+
| Default, with a narrow allow rule of yours | A single command typed on its own (no `&&`, `;`, pipe, redirection, `$( )` or wildcard) that one of your allow rules covers, such as `Bash(pytest:*)` in your user or managed settings, is rewritten too when no deny or ask rule matches. Tokmizer then gives the approval Claude Code would have given to the command as typed, so no new prompt appears. |
|
|
195
|
+
| `bypassPermissions`, or a plain `Bash` allow rule | In `bypassPermissions` mode, or when a plain `Bash` allow rule in your user or managed settings applies and the session is in default, `acceptEdits` or `dontAsk` mode, every routed command is rewritten, and failing output is shortened too. |
|
|
102
196
|
|
|
103
|
-
|
|
197
|
+
Your deny and ask rules, `--disallowedTools`, `--settings`, managed policies and the claude.ai console rules always win. In a linked git worktree, commands that mention git run as typed so Claude Code's worktree check never refuses them.
|
|
104
198
|
|
|
105
|
-
|
|
199
|
+
</details>
|
|
106
200
|
|
|
107
|
-
|
|
201
|
+
## Commands
|
|
108
202
|
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
203
|
+
| Claude Code | Terminal | What it does |
|
|
204
|
+
| --- | --- | --- |
|
|
205
|
+
| `/tkr-status` | `tkr status` | account, engine and rule status |
|
|
206
|
+
| `/tkr-gain` | `tkr gain` | tokens saved, by day, project or rule |
|
|
207
|
+
| `/tkr-compression` | `tkr compression light\|balanced\|max` | how hard output is shortened, `balanced` by default |
|
|
208
|
+
| `/tkr-reducer` | `tkr reducer off\|light\|balanced\|max` | shorter replies from the model itself, `off` by default |
|
|
209
|
+
| `/tkr-yolo` | `tkr yolo on\|off\|status` | very aggressive mode, see below |
|
|
210
|
+
| `/power` | `$power` in Codex, `/power` in most agents | plan, build, test and ship a change in one go |
|
|
211
|
+
| `/tkr` | `tkr settings show\|set\|configure` | every setting |
|
|
212
|
+
| `/tkr-telemetry` | `tkr telemetry on\|off\|status` | usage stats |
|
|
213
|
+
| `/tkr-link` | `tkr link <key>` | optional account: synced settings and a dashboard |
|
|
214
|
+
| `/tkr-unlink` | `tkr unlink` | detach this machine from the account |
|
|
215
|
+
| `/tkr-update` | npm, brew or auto-update | latest version |
|
|
216
|
+
|
|
217
|
+
Also in the terminal: `tkr restore`, `tkr doctor [--fix]`, `tkr discover`, `tkr session`, `tkr network`, `tkr filters`, `tkr migrate rtk`, `tkr setup`, `tkr uninstall`. `tkr --help` lists everything.
|
|
218
|
+
|
|
219
|
+
## Compression modes
|
|
220
|
+
|
|
221
|
+
| Mode | Use it for |
|
|
222
|
+
| --- | --- |
|
|
223
|
+
| `light` | gentle shortening; file reads up to 16 KB stay whole. Pick it when you need implementation details. |
|
|
224
|
+
| `balanced` | the default: compact output, code bodies can be folded in large reads |
|
|
225
|
+
| `max` | the strongest standard level; repeated log lines become one line with a count |
|
|
112
226
|
|
|
113
|
-
|
|
227
|
+
All three keep failures, distinct diagnostics and exit codes. Short or exact output is often identical across levels. `normal`, `compact` and `minimal` work as aliases.
|
|
114
228
|
|
|
115
|
-
|
|
229
|
+
> **Warning: YOLO** (`tkr yolo on`) goes further and **can remove errors, list entries and useful details**. It keeps no restore copy, so what it drops cannot be recovered. It is off by default, applies to this machine only and cannot be turned on from the dashboard. Keep it off for debugging, reviews, audits or anything that needs the full output.
|
|
116
230
|
|
|
117
|
-
|
|
231
|
+
The **reducer** is separate: from the next session it asks the model to answer in fewer words, while keeping code, errors and security warnings. It works in Claude Code and in every agent set up with `tkr setup-<agent>`. How much it saves depends on the model.
|
|
118
232
|
|
|
119
|
-
|
|
233
|
+
## Getting the full output back
|
|
120
234
|
|
|
121
235
|
```bash
|
|
122
|
-
|
|
123
|
-
tkr
|
|
236
|
+
tkr restore <hash> # the whole original output
|
|
237
|
+
tkr restore <hash> --lines 120:180 # only some lines
|
|
238
|
+
tkr restore <hash> --grep <pattern> # matching lines with their numbers
|
|
239
|
+
tkr restore --grep <pattern> # search every cached output of this project
|
|
240
|
+
tkr restore --list # what is cached here; --all-projects for every project
|
|
124
241
|
```
|
|
125
242
|
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
## Free access
|
|
243
|
+
Copies stay on your machine. Entries are kept 30 days, up to 8 MB each and 200 MB in total; the oldest go first. Private keys, GitHub, npm, AWS, Slack and Stripe tokens, `Bearer` headers and passwords in URLs are masked before a copy is saved; what the agent sees is unchanged. `TOKMIZER_CCR_DISABLED=1` turns copies off.
|
|
129
244
|
|
|
130
|
-
|
|
245
|
+
## Measuring your own savings
|
|
131
246
|
|
|
132
247
|
```bash
|
|
133
|
-
tkr
|
|
248
|
+
tkr gain # all time
|
|
249
|
+
tkr gain --week --graph # last 7 days, one bar per day
|
|
250
|
+
tkr gain --by-rule # which rules saved the most
|
|
251
|
+
tkr gain --failures # commands left unchanged, and why
|
|
252
|
+
tkr gain --quota 5x # rough share of a Claude plan budget (pro, 5x or 20x)
|
|
134
253
|
```
|
|
135
254
|
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
In Claude Code:
|
|
139
|
-
|
|
140
|
-
- `/tkr` shows or changes settings
|
|
141
|
-
- `/tkr-link tokmizer_yourkey` connects an optional account
|
|
142
|
-
- `/tkr-unlink` unlinks this machine from your account
|
|
143
|
-
- `/tkr-gain` shows the estimated reduction in command-output tokens
|
|
144
|
-
- `/tkr-status` shows your account and engine status
|
|
145
|
-
- `/tkr-compression` picks the compression level (light, balanced, max)
|
|
146
|
-
- `/tkr-yolo on` explicitly enables YOLO on this machine; `/tkr-yolo off` disables it
|
|
147
|
-
- `/tkr-reducer` makes the assistant's replies shorter (off, light, balanced, max), in Claude Code and in every agent set up with `tkr setup-<agent>`
|
|
148
|
-
- `/tkr-telemetry` views or changes usage stats (command names and token counts; on by default, off in one command)
|
|
149
|
-
- `/tkr-update` updates the plugin when a new version is out
|
|
150
|
-
|
|
151
|
-
From any terminal (after installing with npm, the install script or Homebrew):
|
|
152
|
-
|
|
153
|
-
- `tkr status` shows your account and engine status
|
|
154
|
-
- `tkr gain` shows the estimated reduction in command-output tokens
|
|
155
|
-
- `tkr link <key>` connects an optional account
|
|
156
|
-
- `tkr unlink` unlinks this machine
|
|
157
|
-
- `tkr compression light|balanced|max` picks how hard output gets compressed (default balanced)
|
|
158
|
-
- `tkr yolo on|off|status` enables, disables, or checks the local YOLO mode
|
|
159
|
-
- `tkr reducer off|light|balanced|max` makes the assistant's replies shorter (default off)
|
|
160
|
-
- `tkr settings show|set|configure` manages settings and syncs them across your machines
|
|
161
|
-
- `tkr network off|on|status` blocks or allows every connection to tokmizer.com
|
|
162
|
-
- `tkr migrate rtk` moves your rtk filters, settings and history over and turns off rtk hooks where Tokmizer runs (see [Switching from rtk to Tokmizer](#switching-from-rtk-to-tokmizer))
|
|
163
|
-
- `tkr setup [--all]` sets up the agents found on this machine; `tkr setup-<agent> --remove` takes one out
|
|
164
|
-
- `tkr doctor [--fix]` shows what is detected and repairs outdated shims and hooks
|
|
165
|
-
- `tkr uninstall` removes every hook, shim and PATH line Tokmizer added (see [Uninstall](#uninstall))
|
|
255
|
+
Each view also takes `--project` and `--json`. `tkr gain` estimates command-output tokens as bytes divided by four; it is not your actual usage. Restores count against the saving.
|
|
166
256
|
|
|
167
257
|
## /power and the reducer in other agents
|
|
168
258
|
|
|
169
|
-
`tkr setup-<agent>` also installs `/power` and the reducer where the agent offers a documented way to do it
|
|
259
|
+
`tkr setup-<agent>` also installs `/power` and the reducer where the agent offers a documented way to do it. `/power` runs only the steps the agent's tools allow and says what is left when it cannot run tests or open a pull request. A command or skill you wrote yourself is never overwritten. In rules files, Tokmizer only edits the block between `<!-- tokmizer:begin managed-by: tokmizer -->` and `<!-- tokmizer:end -->`.
|
|
260
|
+
|
|
261
|
+
<details>
|
|
262
|
+
<summary>Where each agent gets them</summary>
|
|
170
263
|
|
|
171
264
|
| Agent | `/power` | Reducer |
|
|
172
265
|
| --- | --- | --- |
|
|
@@ -193,90 +286,11 @@ From any terminal (after installing with npm, the install script or Homebrew):
|
|
|
193
286
|
| OpenClaw | skill `/power` (`~/.openclaw/skills`) | via plugin: `before_prompt_build`, once you set `plugins.entries.tokmizer.hooks.allowConversationAccess` to `true` |
|
|
194
287
|
| Cline, Windsurf, Aider | not installed: these agents only get command shims and have no `tkr setup-<agent>` command | not installed |
|
|
195
288
|
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
## Choosing a compression mode
|
|
199
|
-
|
|
200
|
-
Choose `light` for gentle reduction, `balanced` (the default) for more compact output, or `max` for the strongest standard reduction. The aliases `normal`, `compact`, and `minimal` map to these levels respectively. Short or exact output can be identical across levels.
|
|
201
|
-
|
|
202
|
-
Since 0.9.54, Light keeps recognized file reads of up to 16 KB complete; larger reads are shortened again. Balanced and Max can fold code bodies. Choose Light when you need implementation details. Explicit outlining remains available to callers that request it. Host limits and the CLI's output buffer still apply.
|
|
203
|
-
|
|
204
|
-
Since 0.9.52, short code reads and small explicit excerpts stay complete by default in all three standard modes. Larger reads can still be shortened. Recognized failures retain their diagnostics; Max can summarize repeated log lines with an occurrence count. Distinct diagnostic messages and command exit codes are preserved.
|
|
205
|
-
|
|
206
|
-
Without a linked account, standard settings stay on this machine and work offline after initial activation. With a linked account, changes require a connection and sync across machines. Linking an account restores its server settings.
|
|
207
|
-
|
|
208
|
-
The separate `reducer` setting provides response-style instructions, starting with the next session, for Claude Code and for every agent set up with its `tkr setup-<agent>` command (see the table below). The instructions ask the model to preserve code, errors, and security information; compliance and actual savings depend on the model. If the ruleset is unavailable, the CLI reports it instead of claiming activation. Reconnect and run `tkr heartbeat` before starting a new session.
|
|
209
|
-
|
|
210
|
-
YOLO is off by default and requires an explicit `on`. It applies only to this machine, is not synchronized, and cannot be enabled from the dashboard.
|
|
211
|
-
|
|
212
|
-
| Command | Effect |
|
|
213
|
-
| --- | --- |
|
|
214
|
-
| `tkr yolo on` | Enable more aggressive compression on this machine |
|
|
215
|
-
| `tkr yolo off` | Return to the usual compression mode |
|
|
216
|
-
| `tkr yolo status` | Show whether YOLO is enabled |
|
|
217
|
-
| `tkr compression light`, `tkr compression balanced`, or `tkr compression max` | Disable YOLO locally. Linked accounts need a connection to sync the standard level |
|
|
218
|
-
|
|
219
|
-
**YOLO can remove errors, list entries, and useful details.** It creates no new restoration copies of compressed output. Existing cached output is not deleted, but turning YOLO off cannot recover details discarded while it was on. Command exit codes remain unchanged.
|
|
220
|
-
|
|
221
|
-
Use YOLO for repeated commands only when you accept incomplete output and can check the result independently. Turn it off for debugging, diagnosis, exhaustive code review, audits, or any task that needs the complete output. A preserved exit code alone does not verify the result. YOLO is not a promise of faster coding, rendering, or builds.
|
|
222
|
-
|
|
223
|
-
Terminal commands managed by Tokmizer use the mode selected when execution starts. Native agent tools use the mode active when their output is received. Output processing that has already started in YOLO remains without a restoration copy after YOLO is turned off. Turning YOLO on blocks new restoration copies. Change modes between commands. This concerns Tokmizer's copies, not any history kept by your agent or terminal.
|
|
224
|
-
|
|
225
|
-
`tkr gain` measures output size before and after compression and estimates tokens as UTF-8 bytes divided by four. Token and cost figures are estimates about command output, not a measure of your actual API usage, task quality or completion speed.
|
|
226
|
-
|
|
227
|
-
Since 0.9.53, successful `tkr restore` calls are counted as additional output in the local gain history, even with telemetry disabled. They reduce the net saving; repeated restorations count again and can make it negative. The display shows restored tokens separately. `gain --json` exposes the net estimate in `observed.savedTokens` and restoration totals in `restoration`; the existing top-level `savedTokens` remains the total for reduced outputs only. Earlier restorations cannot be reconstructed from local history. Manually repeated commands, prompts, replies and replayed conversation context are not attributed to restoration costs.
|
|
228
|
-
|
|
229
|
-
Other local views: `tkr gain --by-rule` groups savings by rule (your custom filters by name, built-in rules by command), `tkr gain --failures` lists commands whose output went through unchanged and why (no rule, nothing to cut, failed run kept in full), and `tkr gain --quota [pro|5x|20x]` estimates the share of a Claude plan budget the savings represent. The quota view is an estimate, not a plan limit: tokens saved per day × 30, divided by an assumed 44,000 tokens per 5-hour window × 144 windows × the plan multiplier (1, 5 or 20). Anthropic does not publish plan limits in tokens. All three combine with a period, `--project` and `--json`.
|
|
230
|
-
|
|
231
|
-
The percentages in `tkr gain` measure byte reduction. A byte percentage can differ substantially from the reduction counted by a model's tokenizer. Restoring a full file after reading its outline can cost more tokens than reading it once. Compression is not a promise of faster task completion or a correct answer.
|
|
232
|
-
|
|
233
|
-
## Restoring shortened output
|
|
234
|
-
|
|
235
|
-
When Tokmizer shortens an output it can add a `[tkr restore <hash>]` hint. `tkr restore <hash>` prints the complete original output when it is at most 8 MB, except detected secrets, which are masked; for a larger output, the entry holds only the part that was left out and `tkr restore` says so. A unique prefix of the hash is enough.
|
|
236
|
-
|
|
237
|
-
| Command | Effect |
|
|
238
|
-
| --- | --- |
|
|
239
|
-
| `tkr restore <hash>` | Print the cached output |
|
|
240
|
-
| `tkr restore <hash> --lines 120:180` | Print only lines 120 to 180 (`120:`, `:40` and `57` also work) |
|
|
241
|
-
| `tkr restore <hash> --grep <pattern>` | Print matching lines with their line numbers |
|
|
242
|
-
| `tkr restore --grep <pattern>` | Search the cached outputs of the current project |
|
|
243
|
-
| `tkr restore --grep <pattern> --all-projects` | Search the cached outputs of every project |
|
|
244
|
-
| `tkr restore --list` | List this project's cached outputs with age, size, line count and command (`--all-projects` for every project) |
|
|
245
|
-
|
|
246
|
-
Before a copy is saved, private keys, GitHub, npm, AWS, Slack and Stripe live tokens, `Authorization: Bearer` values and passwords in URLs are replaced by `[tkr: secret masked]`; the output your agent sees is unchanged. Entries are kept 30 days, up to 8 MB each and 200 MB in total; the oldest go first. `TOKMIZER_CCR_MAX_AGE_MS`, `TOKMIZER_CCR_ENTRY_MAX_BYTES` and `TOKMIZER_CCR_DIR_CAP_BYTES` change these limits, `TOKMIZER_CCR_DISABLED=1` turns copies off. `tkr gain` shows the restored tokens and the cache size.
|
|
247
|
-
|
|
248
|
-
## Measured comparison
|
|
249
|
-
|
|
250
|
-
On October 9, 2026, a fixed set of 30 real and synthetic command outputs contained 774,863 tokens before compression. The table counts tokens with the cl100k tokenizer, excluding restoration notices. It is a test corpus, not a sample of customer usage.
|
|
289
|
+
</details>
|
|
251
290
|
|
|
252
|
-
|
|
291
|
+
## Your own rules
|
|
253
292
|
|
|
254
|
-
|
|
255
|
-
| --- | ---: | ---: |
|
|
256
|
-
| `light` | 112,816 | 85 / 85 |
|
|
257
|
-
| `balanced` | 102,052 | 85 / 85 |
|
|
258
|
-
| `max` | 93,131 | 85 / 85 |
|
|
259
|
-
| YOLO | 1,925 | 44 / 85 |
|
|
260
|
-
|
|
261
|
-
The 85 checks cover 13 of the 30 cases. The other 17 have no complete preservation checks, so these results do not prove universal fidelity. YOLO's smaller output comes with observed information loss.
|
|
262
|
-
|
|
263
|
-
The result also depends on the output. In a separate set of five JSON examples, `light` produced 285 tokens, `balanced` 253, and both `max` and YOLO 197. YOLO does not always save more than `max`.
|
|
264
|
-
|
|
265
|
-
In one real failed test, `max` kept 302 tokens and YOLO kept 77. Both retained the actual and expected values. YOLO removed the explanation about invoice 42 and the location `payment.test.mjs:4:10`, which were retained by `max`. That missing context can matter when fixing the failure.
|
|
266
|
-
|
|
267
|
-
In a separate controlled test for 0.9.52, a repetitive failure log went from 1,094 to 84 tokens in Max (92.3% fewer, counted with cl100k). Every distinct diagnostic and the repetition counts were retained. This result applies to that example, not to typical usage.
|
|
268
|
-
|
|
269
|
-
These tokenizer counts are separate from the byte-based estimates in `tkr gain`. They measure output volume, not API usage, an agent's task success, or execution speed. See the [mode guide](https://tokmizer.com/en/docs#configuration) for usage and limits.
|
|
270
|
-
|
|
271
|
-
### Benchmark of October 17, 2026
|
|
272
|
-
|
|
273
|
-
Each tool's Claude Code hook chain replayed 68 real command outputs offline (network cut, empty home folder, no account, median of 3 runs) with Tokmizer 0.9.60 and its default settings, against rtk 1.0.1. Tokmizer removed 84% of tokens in Claude Code's default mode and up to 93% when commands are rewritten (`bypassPermissions` or a blanket `Bash` allow); rtk removed 55%. All 307 failure details were kept in both Tokmizer modes. A later offline run of the same 68 cases with Tokmizer 0.9.63, counted with Anthropic's legacy tokenizer rather than o200k_base, gave 84.2% in the default mode without allow rules, 92.8% in the default mode with narrow allow rules for the test and build commands and 93.5% in `bypassPermissions`, against 57.2% for rtk; 307/307 failure details were kept in every Tokmizer mode. The full results are on [tokmizer.com/en/compare/rtk](https://tokmizer.com/en/compare/rtk).
|
|
274
|
-
|
|
275
|
-
## Custom filters
|
|
276
|
-
|
|
277
|
-
Add your own output rules as `*.toml` files in `~/.tokmizer/filters/` (or `$TOKMIZER_HOME/filters/`). They load next to the built-in rules and win over them for the commands they match. They stay on your machine and are never uploaded. A file or rule that fails validation is skipped; `tkr doctor` shows what was loaded and why something was skipped.
|
|
278
|
-
|
|
279
|
-
A file is used only after you trust it: review it, then run `tkr filters trust` (all files) or `tkr filters trust <file>`. Tokmizer records the file's SHA-256; editing the file stops it from being used until you trust it again. `tkr filters untrust [file]` removes the trust.
|
|
293
|
+
Drop a `*.toml` file in `~/.tokmizer/filters/`. Your rules load next to the built-in ones, win for the commands they match, and never leave your machine.
|
|
280
294
|
|
|
281
295
|
```toml
|
|
282
296
|
schema_version = 1
|
|
@@ -293,122 +307,95 @@ input = "make[1]: Entering directory '/src'\nok\n"
|
|
|
293
307
|
expected = "ok"
|
|
294
308
|
```
|
|
295
309
|
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
## Switching from rtk to Tokmizer
|
|
301
|
-
|
|
302
|
-
If you already use rtk, one command brings over what can be brought over and stops commands from going through two hooks:
|
|
303
|
-
|
|
304
|
-
```sh
|
|
305
|
-
tkr migrate rtk # show the plan, then ask before changing anything
|
|
306
|
-
tkr migrate rtk --dry-run # only show the plan
|
|
307
|
-
tkr migrate rtk --yes # apply without asking
|
|
308
|
-
tkr migrate rtk --undo # put back every file the migrations changed, newest first
|
|
310
|
+
```bash
|
|
311
|
+
tkr filters test # run the inline tests, exit 1 on failure
|
|
312
|
+
tkr filters trust # after review: records the file's SHA-256 and starts using it
|
|
309
313
|
```
|
|
310
314
|
|
|
311
|
-
|
|
312
|
-
- **Settings.** From `config.toml`: `exclude_commands` (the output of those commands is left unchanged), the `rtk recall` retention or its `disabled` mode (`~/.tokmizer/restore.json`), and telemetry turned off (`tkr telemetry off`). Every other setting is listed as not imported.
|
|
313
|
-
- **History.** The totals in rtk's `history.db` are read without changing it. `tkr gain` shows them on their own "Imported from rtk" line, never added to the savings Tokmizer measured. An unreadable file is skipped.
|
|
314
|
-
- **Agent hooks.** For each agent where both rtk and Tokmizer are installed (Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Factory Droid, Trae, Mistral Vibe, Antigravity, OpenCode, Pi, Oh My Pi), the rtk hook is removed so commands are not wrapped twice. Where Tokmizer is not installed yet, the rtk hook stays and the plan says which `tkr setup-…` to run first. rtk's own proxy setting (`ANTHROPIC_BASE_URL`) is left alone. `tkr doctor` reports any agent where both still hook the same commands.
|
|
315
|
+
An edited file stops being used until you trust it again. Also available: `strip_ansi`, `keep_failures`, `truncate_lines_at` and `priority`. `tkr doctor` shows what loaded and why something was skipped.
|
|
315
316
|
|
|
316
|
-
|
|
317
|
+
## Coming from rtk
|
|
317
318
|
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
319
|
+
```bash
|
|
320
|
+
tkr migrate rtk --dry-run # show the plan
|
|
321
|
+
tkr migrate rtk # show it, ask, then apply
|
|
322
|
+
tkr migrate rtk --undo # put every changed file back
|
|
323
|
+
```
|
|
323
324
|
|
|
324
|
-
|
|
325
|
-
- The public repository contains the full source of the package: CLI, hooks, agent adapters, native launcher, setup commands, the TOML rule catalogue and the compression engine.
|
|
325
|
+
It brings over your rtk filters and runs their tests on Tokmizer, plus `exclude_commands`, the `rtk recall` retention and a telemetry opt-out. rtk's history shows on its own line in `tkr gain`. In each agent where Tokmizer is installed, rtk's hook is removed so commands are not wrapped twice. Anything that cannot be imported is listed with the reason, and every changed file is backed up in `~/.tokmizer/migrate-rtk/`. `--shim` adds an `rtk` command that forwards `rtk gain` and `rtk recall` to Tokmizer.
|
|
326
326
|
|
|
327
|
-
##
|
|
327
|
+
## In CI
|
|
328
328
|
|
|
329
|
-
|
|
329
|
+
Add Tokmizer to Claude Code jobs that run with `anthropics/claude-code-action`:
|
|
330
330
|
|
|
331
|
-
```
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
331
|
+
```yaml
|
|
332
|
+
- id: tokmizer
|
|
333
|
+
uses: norwelltw/Tokmizer@main
|
|
334
|
+
- uses: anthropics/claude-code-action@v1
|
|
335
|
+
with:
|
|
336
|
+
plugin_marketplaces: ${{ steps.tokmizer.outputs.plugin_marketplaces }}
|
|
337
|
+
plugins: ${{ steps.tokmizer.outputs.plugins }}
|
|
337
338
|
```
|
|
338
339
|
|
|
339
|
-
|
|
340
|
+
Full workflow and options: [action/README.md](https://github.com/norwelltw/Tokmizer/blob/main/action/README.md).
|
|
340
341
|
|
|
341
|
-
|
|
342
|
+
## Network and privacy
|
|
342
343
|
|
|
343
|
-
|
|
344
|
+
Tokmizer works without an account or email. It contacts `tokmizer.com` to keep its rules current and to learn which commands to optimize next. **Never sent: command arguments, command output, file contents, file paths.**
|
|
344
345
|
|
|
345
|
-
|
|
346
|
+
```bash
|
|
347
|
+
tkr telemetry off # stop usage stats
|
|
348
|
+
tkr network off # stop every connection; compression keeps working with the built-in rules
|
|
349
|
+
```
|
|
346
350
|
|
|
347
|
-
|
|
351
|
+
`TOKMIZER_NETWORK=0` does the same for one shell or a CI job. Rules and commands downloaded from `tokmizer.com` are signed and checked against a key pinned in the CLI.
|
|
348
352
|
|
|
349
|
-
|
|
353
|
+
<details>
|
|
354
|
+
<summary>Exactly what leaves your machine</summary>
|
|
350
355
|
|
|
351
356
|
| When | What is sent | Why |
|
|
352
|
-
|
|
353
|
-
| First online run
|
|
354
|
-
| `tkr link <key>` | The same, plus your key, and your email only
|
|
355
|
-
|
|
|
356
|
-
| Rule downloads | Install ID, fingerprint, plugin version |
|
|
357
|
-
| Settings sync, linked accounts only | Your Tokmizer settings |
|
|
358
|
-
| Usage stats, only while
|
|
359
|
-
| `tkr unlink` | Install ID, fingerprint |
|
|
360
|
-
|
|
|
361
|
-
|
|
362
|
-
The hashes are not anonymous: the same machine always gives the same values, and a user name hash can be guessed. Like any web server, `tokmizer.com` sees your IP address and keeps it in its security log. Tokmizer never sends command arguments, command output, file contents or file paths.
|
|
363
|
-
|
|
364
|
-
Usage stats are on by default and off in one command: `tkr telemetry off` (or `tkr network off` to stop every connection). The choice is saved on this machine and survives updates; an install that turned them off stays off. Before anything is sent, Tokmizer says so once, in the terminal or in the Claude Code session notice. Check with `tkr telemetry status`.
|
|
365
|
-
|
|
366
|
-
### Usage stats without an account
|
|
357
|
+
| --- | --- | --- |
|
|
358
|
+
| First online run | A random install ID, computer name, OS and version, CPU model and core count, SHA-256 hashes of your network card address and user name, a fingerprint of these, your time zone and the plugin version | create the install token used to download rules |
|
|
359
|
+
| `tkr link <key>` | The same, plus your key, and your email only with `--email` | attach this machine to your account |
|
|
360
|
+
| Check-in (heartbeat) | Install ID, fingerprint, plugin version, local date and time zone; while usage stats are on, install health counts such as outdated shims, broken hooks and repairs | refresh access, announce new versions, spot broken installs |
|
|
361
|
+
| Rule downloads | Install ID, fingerprint, plugin version | latest rules, `/power` and reducer instructions |
|
|
362
|
+
| Settings sync, linked accounts only | Your Tokmizer settings | keep them in sync with the dashboard |
|
|
363
|
+
| Usage stats, only while on | Command names such as `npm install` (never arguments), token counts, rule IDs with output sizes, session ID, agent name | see which commands to optimize next |
|
|
364
|
+
| `tkr unlink` | Install ID, fingerprint | remove this device from your account |
|
|
365
|
+
| Self-update | `npm i -g @tokmizer/plugin@<version>` or `brew upgrade tokmizer` | keep the CLI current; `tkr auto-update off` stops it |
|
|
367
366
|
|
|
368
|
-
|
|
367
|
+
The hashes are not anonymous: the same machine always gives the same values, and a user name hash can be guessed. Like any web server, `tokmizer.com` sees your IP address and keeps it in its security log. Usage stats are on by default; Tokmizer says so once before anything is sent, and turning them off survives updates. Detailed rows are kept 90 days, then only a running total. With the network off, Tokmizer uses the last downloaded rules for up to 7 days, then the rules built into the CLI. `tkr network off` does not cover Claude Code's own marketplace auto-update. Full policy: [tokmizer.com/en/privacy](https://tokmizer.com/en/privacy).
|
|
369
368
|
|
|
370
|
-
|
|
371
|
-
- **Only with your consent:** the plugin uploads them only while `tkr telemetry status` says `on`, and tells the server so with each upload. Nothing is accepted from an install that is off or whose network is off.
|
|
372
|
-
- **Once per install, before the first upload:** Tokmizer says so once (first run, setup, session start or upgrade) and sends nothing until that notice has been shown, then stays quiet.
|
|
373
|
-
- **Kept:** detailed rows for 90 days, then only a running total of tokens saved per key.
|
|
374
|
-
- **Linking later:** `tkr link` moves the install's past usage stats to your account, so your dashboard includes them.
|
|
375
|
-
- **Turn off:** `tkr telemetry off`. To stop every connection: `tkr network off`.
|
|
369
|
+
</details>
|
|
376
370
|
|
|
377
|
-
|
|
371
|
+
## Uninstall
|
|
378
372
|
|
|
379
373
|
```bash
|
|
380
|
-
tkr
|
|
381
|
-
|
|
382
|
-
tkr network on
|
|
374
|
+
tkr uninstall # every hook, shim, PATH line, /power and reducer block Tokmizer added
|
|
375
|
+
npm uninstall -g @tokmizer/plugin # or: brew uninstall tokmizer
|
|
383
376
|
```
|
|
384
377
|
|
|
385
|
-
In
|
|
386
|
-
|
|
387
|
-
While it is off there is no sign-up, heartbeat, rule download, settings sync, usage stats or update check. Tokmizer keeps compressing locally: with the last downloaded rules for up to 7 days, then with the rules built into the CLI. Settings are saved on this machine only. `tkr link` and `tkr heartbeat` explain how to turn the network back on.
|
|
378
|
+
In Claude Code: `/plugin uninstall tokmizer@tokmizer`. Your savings history stays in `~/.tokmizer` until you delete it. To pause without uninstalling: `tkr settings set enabled false`.
|
|
388
379
|
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
### Local data
|
|
392
|
-
|
|
393
|
-
- Restoration copies are local and kept up to 30 days. A restore hint identifies available cached content; there is no guarantee that every removed detail has a saved copy. YOLO creates no new restoration copies.
|
|
394
|
-
- The [redaction helpers](https://github.com/norwelltw/Tokmizer/blob/main/packages/plugin/src/learn/redact.ts) are available for inspection, like the rest of the source.
|
|
395
|
-
- Local state lives in `~/.tokmizer/`.
|
|
380
|
+
## Build from source
|
|
396
381
|
|
|
397
|
-
|
|
382
|
+
Node.js 20.11+ and npm. The optional native launcher needs Rust.
|
|
398
383
|
|
|
399
384
|
```bash
|
|
400
|
-
|
|
385
|
+
git clone https://github.com/norwelltw/Tokmizer
|
|
386
|
+
cd Tokmizer
|
|
387
|
+
npm install
|
|
388
|
+
npm run build
|
|
389
|
+
npm test
|
|
390
|
+
node packages/plugin/dist/bin/tkr.js help
|
|
401
391
|
```
|
|
402
392
|
|
|
403
|
-
|
|
393
|
+
Native launcher: `npm --workspace packages/plugin run build:native`, then `npm run build` again. To try your build in Claude Code, install the folder as a local plugin or run `npm --workspace packages/plugin link`. The `tokmizer.com` service (accounts, rule downloads, telemetry) is not part of this repository.
|
|
404
394
|
|
|
405
|
-
|
|
406
|
-
npm uninstall -g @tokmizer/plugin
|
|
407
|
-
brew uninstall tokmizer
|
|
408
|
-
```
|
|
395
|
+
## Licence
|
|
409
396
|
|
|
410
|
-
|
|
397
|
+
Apache-2.0, except the compression engine (`packages/plugin/src/engine/`) and the rule catalogue (`packages/plugin/filters/`), which are under the Business Source License 1.1. Production use is allowed, including inside your own tools; offering them as part of a competing token-optimization product or hosted service is not. Each release becomes Apache-2.0 four years after it ships. Rules you write yourself stay yours. Details: [LICENSES.md](https://github.com/norwelltw/Tokmizer/blob/main/packages/plugin/LICENSES.md).
|
|
411
398
|
|
|
412
|
-
##
|
|
399
|
+
## Bugs and security
|
|
413
400
|
|
|
414
|
-
|
|
401
|
+
Found a bug? [Open an issue](https://github.com/norwelltw/Tokmizer/issues/new/choose). You can suggest a fix in it; the maintainer reviews and applies changes, see [CONTRIBUTING.md](https://github.com/norwelltw/Tokmizer/blob/main/CONTRIBUTING.md). Security reports: [SECURITY.md](https://github.com/norwelltw/Tokmizer/blob/main/SECURITY.md).
|