@tokmizer/plugin 0.9.63 → 0.9.64
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/.codex-plugin/plugin.json +1 -1
- package/README.md +4 -4
- package/dist/bin/tkr-codex-hook.js +27 -27
- package/dist/bin/tkr-daemon.js +118 -116
- package/dist/bin/tkr-exec +73 -71
- package/dist/bin/tkr-exec.js +73 -71
- package/dist/bin/tkr-processor.js +92 -90
- package/dist/bin/tkr-shim.js +67 -67
- package/dist/bin/tkr.js +213 -213
- package/dist/hooks-host/post-tool-use.js +85 -83
- package/dist/hooks-host/reducer-stop.js +24 -24
- package/dist/hooks-host/session-recap.js +1 -1
- package/dist/index.js +82 -82
- package/dist/native/manifest.json +12 -12
- package/dist/native/single.json +12 -12
- package/dist/native/tkr-exec-darwin-arm64 +0 -0
- package/dist/native/tkr-exec-darwin-x64 +0 -0
- package/dist/native/tkr-exec-linux-arm64 +0 -0
- package/dist/native/tkr-exec-linux-x64 +0 -0
- package/dist/native/tkr-exec-win32-arm64.exe +0 -0
- package/dist/native/tkr-exec-win32-x64.exe +0 -0
- package/hooks/on-session-start.mjs +16 -5
- package/package.json +8 -8
- package/scripts/install-mac-linux.sh +1 -1
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Tokmizer reduces token consumption for 21 AI coding agents, including Claude Code, OpenAI Codex, Cursor, Gemini CLI and GitHub Copilot. Aider, Cline and Windsurf run through `tkr wrap`. Works on macOS, Linux, and Windows.
|
|
4
4
|
|
|
5
|
-
It shortens command output before it reaches the model, reducing the space it takes in the context window. The reduction depends on the output and the chosen mode. This is not a
|
|
5
|
+
It shortens command output before it reaches the model, reducing the space it takes in the context window. The reduction depends on the output and the chosen mode. This is not a promise of lower usage costs or faster work.
|
|
6
6
|
|
|
7
7
|
## Install
|
|
8
8
|
|
|
@@ -222,7 +222,7 @@ Use YOLO for repeated commands only when you accept incomplete output and can ch
|
|
|
222
222
|
|
|
223
223
|
Terminal commands managed by Tokmizer use the mode selected when execution starts. Native agent tools use the mode active when their output is received. Output processing that has already started in YOLO remains without a restoration copy after YOLO is turned off. Turning YOLO on blocks new restoration copies. Change modes between commands. This concerns Tokmizer's copies, not any history kept by your agent or terminal.
|
|
224
224
|
|
|
225
|
-
`tkr gain` measures output size before and after compression and estimates tokens as UTF-8 bytes divided by four. Token and cost figures are estimates about command output, not
|
|
225
|
+
`tkr gain` measures output size before and after compression and estimates tokens as UTF-8 bytes divided by four. Token and cost figures are estimates about command output, not a measure of your actual API usage, task quality or completion speed.
|
|
226
226
|
|
|
227
227
|
Since 0.9.53, successful `tkr restore` calls are counted as additional output in the local gain history, even with telemetry disabled. They reduce the net saving; repeated restorations count again and can make it negative. The display shows restored tokens separately. `gain --json` exposes the net estimate in `observed.savedTokens` and restoration totals in `restoration`; the existing top-level `savedTokens` remains the total for reduced outputs only. Earlier restorations cannot be reconstructed from local history. Manually repeated commands, prompts, replies and replayed conversation context are not attributed to restoration costs.
|
|
228
228
|
|
|
@@ -265,11 +265,11 @@ In one real failed test, `max` kept 302 tokens and YOLO kept 77. Both retained t
|
|
|
265
265
|
|
|
266
266
|
In a separate controlled test for 0.9.52, a repetitive failure log went from 1,094 to 84 tokens in Max (92.3% fewer, counted with cl100k). Every distinct diagnostic and the repetition counts were retained. This result applies to that example, not to typical usage.
|
|
267
267
|
|
|
268
|
-
These tokenizer counts are separate from the byte-based estimates in `tkr gain`. They measure output volume, not
|
|
268
|
+
These tokenizer counts are separate from the byte-based estimates in `tkr gain`. They measure output volume, not API usage, an agent's task success, or execution speed. See the [mode guide](https://tokmizer.com/en/docs#configuration) for usage and limits.
|
|
269
269
|
|
|
270
270
|
### Benchmark of October 17, 2026
|
|
271
271
|
|
|
272
|
-
Each tool's Claude Code hook chain replayed 68 real command outputs offline (network cut, empty home folder, no account, median of 3 runs) with Tokmizer 0.9.60 and its default settings, against rtk 1.0.1. Tokmizer removed 84% of tokens in Claude Code's default mode and up to 93% when commands are rewritten (`bypassPermissions` or a
|
|
272
|
+
Each tool's Claude Code hook chain replayed 68 real command outputs offline (network cut, empty home folder, no account, median of 3 runs) with Tokmizer 0.9.60 and its default settings, against rtk 1.0.1. Tokmizer removed 84% of tokens in Claude Code's default mode and up to 93% when commands are rewritten (`bypassPermissions` or a blanket `Bash` allow); rtk removed 55%. All 307 failure details were kept in both Tokmizer modes. A later offline run of the same 68 cases with Tokmizer 0.9.63, counted with Anthropic's legacy tokenizer rather than o200k_base, gave 84.2% in the default mode without allow rules, 92.8% in the default mode with narrow allow rules for the test and build commands and 93.5% in `bypassPermissions`, against 57.2% for rtk; 307/307 failure details were kept in every Tokmizer mode. The full results are on [tokmizer.com/en/compare/rtk](https://tokmizer.com/en/compare/rtk).
|
|
273
273
|
|
|
274
274
|
## Custom filters
|
|
275
275
|
|