anolisa-tokenless 0.7.7 → 0.7.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -7,7 +7,7 @@
7
7
  Token-Less combines complementary strategies to minimize LLM token consumption:
8
8
 
9
9
  - **Schema & Response Compression** — Compresses OpenAI Function Calling tool definitions and API responses via the `tokenless-schema` library, cutting structural overhead before tokens ever reach the context window.
10
- - **TOON Context Compression** — Encodes JSON responses to TOON (Token-Oriented Object Notation) format via the `toon` binary, reducing token usage by 15-40% for structured data.
10
+ - **TOON Context Compression** — Encodes JSON responses to TOON (Token-Oriented Object Notation) format via the `toon-format` library linked into `tokenless`, reducing token usage by 15-40% for structured data.
11
11
  - **Command Rewriting** — Integrates [RTK](https://github.com/rtk-ai/rtk) to filter and rewrite CLI command output, eliminating noise that would otherwise waste 60–90% of tokens.
12
12
  - **Tool Ready (legacy, hard-disabled)** — Its pre-call dependency checks are retained in source but unconditionally bypassed while the readiness model is redesigned.
13
13
 
@@ -22,8 +22,8 @@ Agent adapters are available for:
22
22
  - **OpenCode plugin** — schema/response/TOON compression, registered but hard-disabled Tool Ready, and command rewriting via OpenCode's local plugin API.
23
23
  - **DeepSeek Harness plugin** — native response compression and environment-error attribution through DSH's `tools/post-execute` seam.
24
24
 
25
- For framework developers, the separate **AgentScope Python integration** replaces successful
26
- final tool responses and provides a marker-scoped native retrieval Tool.
25
+ For framework developers, the self-contained Python SDK and separate **AgentScope integration**
26
+ cover schema compression, RTK rewriting, response compression, TOON, retrieval, and attribution.
27
27
 
28
28
  ## Features
29
29
 
@@ -43,7 +43,7 @@ final tool responses and provides a marker-scoped native retrieval Tool.
43
43
  | Codex plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Response compression ✅, TOON ✅ |
44
44
  | OpenCode plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Schema compression ✅, Response compression ✅, TOON ✅ |
45
45
  | DeepSeek Harness plugin | — | Response compression ✅, Environment-error attribution ✅ |
46
- | AgentScope framework integration | — | Response compression ✅, Native retrieval Tool ✅ |
46
+ | AgentScope framework integration | — | Schema ✅, RTK ✅, Response ✅, TOON ✅, Retrieval ✅ |
47
47
  | Zero runtime deps | — | Pure Rust, single static binary |
48
48
 
49
49
  ## Applicable Scenarios & Expected Effects
@@ -119,7 +119,7 @@ Token-Less/
119
119
  Install the published component with the ANOLISA CLI:
120
120
 
121
121
  The install script places `anolisa` in `~/.local/bin`, and a user-mode
122
- Tokenless installation places `tokenless`, `rtk`, and `toon` in that same
122
+ Tokenless installation places `tokenless` and `rtk` in that same
123
123
  directory. Export it once if the current shell has not picked it up yet.
124
124
 
125
125
  ```bash
@@ -180,8 +180,8 @@ cd Token-Less
180
180
  make setup
181
181
  ```
182
182
 
183
- The source setup installs `tokenless` to `~/.local/bin`, places the `rtk` and
184
- `toon` helpers alongside it, and deploys all adapters for development.
183
+ The source setup installs `tokenless` to `~/.local/bin`, places the `rtk`
184
+ helper alongside it, and deploys all adapters for development.
185
185
 
186
186
  ### Build the Python runtime
187
187
 
@@ -201,12 +201,35 @@ uses `uvx` to provision Maturin by default. Install
201
201
  plain workspace-default Cargo commands exclude the Python extension.
202
202
 
203
203
  The `anolisa_tokenless` module supports CPython 3.11 and later on the platform
204
- where its native wheel was built. It currently exposes JSON response
205
- compression and Stash retrieval; it does not bundle the CLI, RTK, TOON, or a
206
- framework integration. The package is built and tested in this repository but is
207
- not yet published to PyPI. See the [runtime design](docs/design/runtime-library.md)
204
+ where its native wheel was built. It exposes the four Tokenless lifecycle
205
+ methods and bundles the matching RTK executable; TOON is linked into the native
206
+ runtime. It does not require the Tokenless CLI or system helper binaries. The
207
+ package is built and tested in this repository but is not yet published to
208
+ PyPI. See the [runtime design](docs/design/runtime-library.md)
208
209
  and the [user manual](../../docs/user-guide/en/token-saving/tokenless/user-manual.md#build-the-python-runtime-from-source).
209
210
 
211
+ The same wheel provides typed, read-only statistics queries without requiring
212
+ the CLI. Point `TokenlessStats` at the state directory used by the runtime, or
213
+ use the lazy `sdk.stats` client:
214
+
215
+ ```python
216
+ from anolisa_tokenless import TokenlessStats
217
+
218
+ stats = TokenlessStats("/absolute/path/to/tokenless-data")
219
+ summary = stats.summary()
220
+ print(summary.total.tokens_saved, summary.total.tokens_saved_percent)
221
+ ```
222
+
223
+ Token counts are estimates and only operations with positive savings are
224
+ recorded. `show()` and detailed `diff()` results may contain sensitive tool
225
+ input and output stored in `stats.db`. Read-only describes the API surface:
226
+ opening the client follows CLI initialization and may create or migrate
227
+ `stats.db`, so the data directory must be writable. `summary(limit=None)` and
228
+ `compare(..., limit=None)` inspect at most the newest 10,000 records. For a
229
+ session or tool-use diff, at most the newest 10,000 matching records are read.
230
+ For a meaningful comparison, pass a dry-run session first and an active
231
+ Tokenless session second.
232
+
210
233
  ## CLI Usage
211
234
 
212
235
  ### compress-schema
@@ -581,9 +604,9 @@ The public entry point and configuration are the same across both major
581
604
  versions. AgentScope 1.x and 2.x expose different lifecycle hooks, so only the
582
605
  final attachment step differs.
583
606
 
584
- AgentScope 1.x must install the integration after the Agent and all of its tool
585
- functions have been created. Installation binds retrieval to that Agent's
586
- memory so a stash hash cannot be retrieved unless its marker is visible there.
607
+ AgentScope 1.x uses a Tokenless Toolkit so tools registered before or after
608
+ Agent construction, including MCP tools, receive the same lifecycle handling.
609
+ Installation requires an explicit session identifier.
587
610
 
588
611
  ```python
589
612
  from agentscope.agent import ReActAgent
@@ -595,8 +618,10 @@ integration = TokenlessAgentScope(
595
618
  data_dir="/absolute/path/to/tenant-tokenless-data",
596
619
  ),
597
620
  )
621
+ toolkit = integration.create_toolkit()
622
+ toolkit.register_tool_function(application_tool)
598
623
  agent = ReActAgent(..., toolkit=toolkit)
599
- integration.install(agent)
624
+ integration.install(agent, session_id="conversation-id")
600
625
  ```
601
626
 
602
627
  AgentScope 2.x receives the retrieval Tool and middleware during construction;
@@ -650,25 +675,28 @@ Toolkit mutation or automatic Tool collection.
650
675
  | `balanced` | Skip Read/Glob/Grep; use 65,536 / 128 / depth 8 for Shell and conservative limits elsewhere |
651
676
  | `aggressive` | Skip Read/Glob/Grep; use CLI defaults of 4,096 / 32 / depth 8 elsewhere |
652
677
 
653
- `balanced` is the default. The read-only retrieval Tool is auto-allowed only
654
- for a 24-character hash whose marker is present in AgentScope 1.x memory or the
655
- AgentScope 2.x context/summary. In 1.x, call `install()` only after registering
656
- the tools that should be compressed; tools registered later are not wrapped.
657
- Pass a different absolute `data_dir` to each user or tenant for direct Agents;
678
+ `balanced` is the default. The read-only retrieval Tool is published to the
679
+ model only when a marker is visible and accepts only a hash from the exact
680
+ marker set retained for that model call. Pass a different absolute `data_dir`
681
+ to each user or tenant for direct Agents;
658
682
  `TOKENLESS_DATA_DIR` is only a process-wide fallback when `data_dir` is omitted.
659
683
  Retain the default one-hour stash TTL unless the application has a deliberate
660
- lifecycle policy, and do not expect retrieval across nodes. This integration
661
- does not enable Shell, MCP, TOON, RTK, or schema compression. Its source lives
662
- under `python/agentscope/` for independent wheel distribution.
684
+ lifecycle policy, and do not expect retrieval across nodes.
685
+
686
+ Both AgentScope adapters enable schema compression, RTK command rewriting,
687
+ response compression, TOON, retrieval, environment-error guidance, and
688
+ per-call attribution. The native wheel contains RTK and links TOON directly;
689
+ it does not search for system executables. Host objects and streaming chunks
690
+ remain unchanged; only copied call arguments and final model-visible text are
691
+ transformed. Tool Ready remains hard-disabled.
663
692
 
664
693
 
665
694
  ## Build
666
695
 
667
696
  | Target | Description |
668
697
  |---|---|
669
- | `make build` | Build `tokenless` + `rtk` + `toon` (release mode) |
698
+ | `make build` | Build `tokenless` + `rtk` (release mode) |
670
699
  | `make build-tokenless` | Build `tokenless` + `rtk` (via justfile) |
671
- | `make build-toon` | Install TOON binary via `cargo install toon-format` |
672
700
  | `make python-wheel` | Build the native `anolisa-tokenless` wheel |
673
701
  | `make agentscope-wheel` | Build the pure-Python AgentScope integration wheel |
674
702
  | `make test-python-runtime` | Install and test the wheel in an isolated environment |
@@ -706,7 +734,7 @@ make install BIN_DIR=/usr/local/bin
706
734
 
707
735
  ## Raw Packaging
708
736
 
709
- Raw packaging accepts already-built `tokenless`, `rtk`, and `toon`
737
+ Raw packaging accepts already-built `tokenless` and `rtk`
710
738
  executables in one directory and applies the stable component payload layout:
711
739
 
712
740
  ```bash
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenless",
3
- "version": "0.7.7",
3
+ "version": "0.7.10",
4
4
  "description": "Token-Less context compression for Claude Code — RTK command rewriting, response/TOON compression, and Tool Ready environment pre-check",
5
5
  "author": { "name": "ANOLISA" },
6
6
  "license": "Apache-2.0",
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenless",
3
- "version": "0.7.7",
3
+ "version": "0.7.10",
4
4
  "description": "Intelligent tool response compression, environment error detection, and token optimization. Strips noise (debug fields, nulls, empty values), truncates long strings/arrays, applies TOON encoding for JSON responses, and classifies environment errors with actionable fix hints.",
5
5
  "keywords": [
6
6
  "compression",
@@ -166,6 +166,10 @@ else:
166
166
  AGENT_ID = os.environ.get("TOKENLESS_AGENT_ID", "codex")
167
167
  MIN_RESPONSE_CHARS: int = 500
168
168
  LARGE_RESPONSE_CHARS: int = 4000
169
+ # TOON on small JSON saves only a few characters (observed ~0.3% below
170
+ # ~500 chars) while the per-event encode cost stays the same, so payloads
171
+ # under this threshold keep the compressed form and skip the TOON pass.
172
+ MIN_TOON_CHARS: int = 500
169
173
 
170
174
  # 3-layer compression strategy:
171
175
  # Layer 1: Content retrieval + task management → skip all compression
@@ -372,7 +376,7 @@ def main() -> None:
372
376
  toon_input = compression_input
373
377
 
374
378
  toon_parsed = _try_parse_json(toon_input)
375
- if toon_parsed is not None:
379
+ if toon_parsed is not None and len(toon_input) >= MIN_TOON_CHARS:
376
380
  toon_text = _run_tokenless(
377
381
  tokenless_bin, "compress-toon",
378
382
  toon_input, session_id, tool_use_id, timeout=10,
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenless",
3
- "version": "0.7.0",
3
+ "version": "0.7.10",
4
4
  "contextFileName": "COPILOT.md",
5
5
  "hooks": {
6
6
  "PreToolUse": [
@@ -20,7 +20,7 @@
20
20
  ]
21
21
  },
22
22
  {
23
- "matcher": "^(Bash|run_shell_command|terminal|Shell|exec|process)$",
23
+ "matcher": "^(Bash|run_shell_command|terminal|Shell|shell|exec|process)$",
24
24
  "hooks": [
25
25
  {
26
26
  "type": "command",
@@ -81,6 +81,12 @@ from hook_utils import (
81
81
 
82
82
  _MIN_RESPONSE_CHARS = 200
83
83
 
84
+ # Minimum payload size for the TOON encoding step. TOON on small JSON
85
+ # saves only a few characters (observed ~0.3% below ~500 chars) while the
86
+ # per-event encode cost stays the same, so payloads under this threshold
87
+ # keep the response-compressed form and skip the TOON pass entirely.
88
+ _MIN_TOON_CHARS = 500
89
+
84
90
  # Claude Code added hookSpecificOutput.updatedToolOutput (normal-path tool
85
91
  # output replacement for all tools) in v2.1.121. Older versions only support
86
92
  # the additive additionalContext, which would duplicate the payload.
@@ -231,7 +237,11 @@ def _build_replacement_output(
231
237
  return False, None
232
238
 
233
239
  # Restoring empty schema fields can cancel out a marginal win.
234
- serialized = json.dumps(updated_output, separators=(",", ":"))
240
+ # ensure_ascii=False keeps the size comparison in Unicode characters,
241
+ # consistent with the non-escaped normalization below.
242
+ serialized = json.dumps(
243
+ updated_output, separators=(",", ":"), ensure_ascii=False
244
+ )
235
245
  if len(serialized) >= len(tool_response):
236
246
  return False, None
237
247
  return True, updated_output
@@ -313,7 +323,13 @@ def main() -> None:
313
323
  skip() # Plain text, not JSON
314
324
  tool_response = unwrapped
315
325
  elif isinstance(model_visible_before, (dict, list)):
316
- tool_response = json.dumps(model_visible_before, separators=(",", ":"))
326
+ # ensure_ascii=False: size gates below must count Unicode
327
+ # characters (code points), not \uXXXX escape sequences, so
328
+ # structured payloads are measured the same way as JSON string
329
+ # inputs and the OpenClaw adapter.
330
+ tool_response = json.dumps(
331
+ model_visible_before, separators=(",", ":"), ensure_ascii=False
332
+ )
317
333
  else:
318
334
  skip()
319
335
 
@@ -380,10 +396,12 @@ def main() -> None:
380
396
  except Exception as e:
381
397
  warn(f"Response compression error: {e}")
382
398
 
383
- # 16. Step 2: TOON encoding
399
+ # 16. Step 2: TOON encoding — only for payloads at or above the
400
+ # minimum threshold; small JSON gains near-zero chars from TOON but
401
+ # would still pay the full encode cost on every PostToolUse event.
384
402
  toon_output = ""
385
403
 
386
- if tokenless_bin:
404
+ if tokenless_bin and len(compressed) >= _MIN_TOON_CHARS:
387
405
  toon_parsed = try_parse_json(compressed)
388
406
  if toon_parsed is not None:
389
407
  toon_cmd = [tokenless_bin, "compress-toon", "--agent-id", agent_id]
@@ -39,7 +39,11 @@ from hook_utils import (
39
39
  # -- constants ---------------------------------------------------------------
40
40
 
41
41
  _AGENT_ID = resolve_agent_id()
42
- _MIN_RESPONSE_CHARS = 200
42
+
43
+ # Minimum payload size for TOON encoding. TOON on small JSON saves only a
44
+ # few characters (observed ~0.3% below ~500 chars) while the per-event
45
+ # encode cost stays the same, so smaller responses pass through untouched.
46
+ _MIN_TOON_CHARS = 500
43
47
 
44
48
 
45
49
  # -- main --------------------------------------------------------------------
@@ -84,15 +88,22 @@ def main() -> None:
84
88
  if tool_response is None:
85
89
  skip() # Plain text, not JSON
86
90
  elif isinstance(tool_response_raw, (dict, list)):
87
- tool_response = json.dumps(tool_response_raw, separators=(",", ":"))
91
+ # ensure_ascii=False: the threshold below counts Unicode
92
+ # characters (code points), not \uXXXX escape sequences, so
93
+ # structured payloads are measured the same way as JSON string
94
+ # inputs and the OpenClaw adapter.
95
+ tool_response = json.dumps(
96
+ tool_response_raw, separators=(",", ":"), ensure_ascii=False
97
+ )
88
98
  else:
89
99
  skip()
90
100
 
91
101
  if not tool_response:
92
102
  skip()
93
103
 
94
- # 7. Skip small responses (character count, not byte length)
95
- if len(tool_response) < _MIN_RESPONSE_CHARS:
104
+ # 7. Skip payloads below the TOON minimum threshold (character count,
105
+ # not byte length): TOON savings on small JSON are near-zero
106
+ if len(tool_response) < _MIN_TOON_CHARS:
96
107
  skip()
97
108
 
98
109
  # 8. Validate it's JSON
@@ -47,7 +47,7 @@ _RTK_LOCAL_SHARE = _user_path(
47
47
  )
48
48
  _RTK_LOCAL_LIB = _user_path(".local", "lib", "anolisa", "tokenless", "rtk")
49
49
 
50
- _TOKENLESS_HELPER_BINARIES = frozenset({"rtk", "toon"})
50
+ _TOKENLESS_HELPER_BINARIES = frozenset({"rtk"})
51
51
 
52
52
 
53
53
  def _known_binary_paths(name: str, home: str | None = None) -> tuple[str, ...]:
@@ -408,7 +408,13 @@ def unwrap_string_json(raw: str) -> str | None:
408
408
  if isinstance(inner, str):
409
409
  inner_obj = try_parse_json(inner)
410
410
  if inner_obj is not None and isinstance(inner_obj, (dict, list)):
411
- return json.dumps(inner_obj, separators=(",", ":"))
411
+ # ensure_ascii=False: downstream size gates count Unicode
412
+ # characters (code points), not \uXXXX escape sequences, so
413
+ # string-wrapped payloads are measured the same way as the
414
+ # dict/list branch and the OpenClaw adapter.
415
+ return json.dumps(
416
+ inner_obj, separators=(",", ":"), ensure_ascii=False
417
+ )
412
418
  return None
413
419
  return raw
414
420
 
@@ -5,7 +5,10 @@ Reads a PreToolUse JSON from stdin, extracts the shell command,
5
5
  invokes ``rtk rewrite`` via subprocess, and writes a HookOutput
6
6
  JSON to stdout.
7
7
 
8
- Hook point: **PreToolUse** — matcher: ``Shell``
8
+ Hook point: **PreToolUse** — matcher: shell-family tool names
9
+ (``Bash``, ``run_shell_command``, ``terminal``, ``Shell``, ``shell``,
10
+ ``exec``, ``process``). The lowercase ``shell`` alternative covers
11
+ cosh-ng, whose built-in shell tool is named ``shell`` on the wire.
9
12
 
10
13
  The agent ID is read from the TOKENLESS_AGENT_ID environment variable
11
14
  (set by the install action script). Fallback paths follow the ANOLISA
@@ -295,7 +295,7 @@ resolve_binary() {
295
295
  candidates+=("$USER_HOME/.local/bin/$name")
296
296
  fi
297
297
  case "$name" in
298
- rtk|toon)
298
+ rtk)
299
299
  if [ -n "$USER_HOME" ]; then
300
300
  candidates+=(
301
301
  "$USER_HOME/.local/lib/anolisa/libexec/tokenless/$name"
@@ -306,13 +306,13 @@ resolve_binary() {
306
306
  esac
307
307
  candidates+=("/usr/local/bin/$name")
308
308
  case "$name" in
309
- rtk|toon)
309
+ rtk)
310
310
  candidates+=("/usr/local/libexec/anolisa/tokenless/$name")
311
311
  ;;
312
312
  esac
313
313
  candidates+=("/usr/bin/$name")
314
314
  case "$name" in
315
- rtk|toon)
315
+ rtk)
316
316
  candidates+=(
317
317
  "/usr/libexec/anolisa/tokenless/$name"
318
318
  "/usr/lib/anolisa/tokenless/$name"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@anolisa/dsh-tokenless",
3
- "version": "0.7.7",
3
+ "version": "0.7.10",
4
4
  "description": "Native DeepSeek Harness response compression for Tokenless",
5
5
  "type": "module",
6
6
  "main": "./dist/index.js",
@@ -180,6 +180,12 @@ logger = logging.getLogger(__name__)
180
180
  AGENT_ID = "hermes-agent"
181
181
  _MIN_RESPONSE_LEN = 200
182
182
 
183
+ # Minimum payload size for the TOON encoding step. TOON on small JSON
184
+ # saves only a few characters (observed ~0.3% below ~500 chars) while
185
+ # the per-event encode cost stays the same, so payloads under this
186
+ # threshold keep the response-compressed form and skip TOON entirely.
187
+ _MIN_TOON_CHARS = 500
188
+
183
189
  _SKIP_TOOLS: set[str] = _SKIP_TOOLS_SHARED | {
184
190
  "session_search", "list_sessions",
185
191
  }
@@ -513,8 +519,12 @@ def on_transform_tool_result(
513
519
  str(session_id), str(tool_call_id))
514
520
  current = compressed if compressed else result
515
521
 
516
- # Step 2: TOON encoding
517
- toon_result = _encode_toon(current, str(session_id), str(tool_call_id))
522
+ # Step 2: TOON encoding — only for payloads at or above the minimum
523
+ # threshold; small JSON gains near-zero chars from TOON but would still
524
+ # pay the full encode cost on every tool result.
525
+ toon_result = None
526
+ if len(current) >= _MIN_TOON_CHARS:
527
+ toon_result = _encode_toon(current, str(session_id), str(tool_call_id))
518
528
  used_compression = compressed is not None
519
529
  used_toon = toon_result is not None
520
530
 
@@ -1,5 +1,5 @@
1
1
  name: tokenless
2
- version: "0.7.7"
2
+ version: "0.7.10"
3
3
  description: "Token-Less context compression for Hermes Agent — response compression, TOON encoding, command rewriting, and registered but hard-disabled Tool Ready"
4
4
  author: ANOLISA
5
5
  requires_env: []
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "component": "tokenless",
3
- "version": "0.7.7",
3
+ "version": "0.7.10",
4
4
  "targets": {
5
5
  "cosh": {
6
6
  "compatibleVersions": "*",
@@ -55,6 +55,25 @@ function mergeExecContextEnv(params, context) {
55
55
  }
56
56
  // ---- Binary availability cache (with TTL for negative results) -----------------
57
57
  const CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes — retry after auto-fix installs
58
+ // Minimum payload size for the TOON encoding step. TOON on small JSON saves
59
+ // only a few characters (observed ~0.3% below ~500 chars) while the
60
+ // per-event encode cost stays the same, so payloads under this threshold
61
+ // keep the response-compressed form and skip TOON entirely.
62
+ const MIN_TOON_CHARS = 500;
63
+ // True when `text` contains at least `threshold` Unicode code points.
64
+ // Iterating a string counts code points (a surrogate pair counts once), so
65
+ // the threshold uses the same unit as the Python adapters' len(). The loop
66
+ // returns as soon as the threshold is reached, so large payloads only pay
67
+ // for scanning the first `threshold` characters.
68
+ function hasAtLeastChars(text, threshold) {
69
+ let count = 0;
70
+ for (const _ch of text) {
71
+ count += 1;
72
+ if (count >= threshold)
73
+ return true;
74
+ }
75
+ return false;
76
+ }
58
77
  let rtkAvailable = null;
59
78
  let rtkCheckedAt = null;
60
79
  let tokenlessAvailable = null;
@@ -274,6 +293,13 @@ function tryCompressResponse(response, sessionId, toolCallId, thresholds) {
274
293
  function tryCompressToon(response, sessionId, toolCallId) {
275
294
  try {
276
295
  const input = JSON.stringify(response);
296
+ // Skip payloads below the minimum threshold: TOON savings on small
297
+ // JSON are near-zero but the encode cost is paid on every tool result.
298
+ // Count Unicode code points, not UTF-16 code units (String.length), so
299
+ // non-BMP text (e.g. emoji) is measured the same way as the Python
300
+ // adapters' character counts.
301
+ if (!hasAtLeastChars(input, MIN_TOON_CHARS))
302
+ return null;
277
303
  const beforeChars = input.length;
278
304
  const args = ["compress-toon", "--agent-id", "openclaw"];
279
305
  if (sessionId)
@@ -88,6 +88,26 @@ function mergeExecContextEnv(
88
88
 
89
89
  const CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes — retry after auto-fix installs
90
90
 
91
+ // Minimum payload size for the TOON encoding step. TOON on small JSON saves
92
+ // only a few characters (observed ~0.3% below ~500 chars) while the
93
+ // per-event encode cost stays the same, so payloads under this threshold
94
+ // keep the response-compressed form and skip TOON entirely.
95
+ const MIN_TOON_CHARS = 500;
96
+
97
+ // True when `text` contains at least `threshold` Unicode code points.
98
+ // Iterating a string counts code points (a surrogate pair counts once), so
99
+ // the threshold uses the same unit as the Python adapters' len(). The loop
100
+ // returns as soon as the threshold is reached, so large payloads only pay
101
+ // for scanning the first `threshold` characters.
102
+ function hasAtLeastChars(text: string, threshold: number): boolean {
103
+ let count = 0;
104
+ for (const _ch of text) {
105
+ count += 1;
106
+ if (count >= threshold) return true;
107
+ }
108
+ return false;
109
+ }
110
+
91
111
  let rtkAvailable: boolean | null = null;
92
112
  let rtkCheckedAt: number | null = null;
93
113
  let tokenlessAvailable: boolean | null = null;
@@ -326,6 +346,12 @@ function tryCompressResponse(response: any, sessionId?: string, toolCallId?: str
326
346
  function tryCompressToon(response: any, sessionId?: string, toolCallId?: string): { toonText: string; savingsPct: number } | null {
327
347
  try {
328
348
  const input = JSON.stringify(response);
349
+ // Skip payloads below the minimum threshold: TOON savings on small
350
+ // JSON are near-zero but the encode cost is paid on every tool result.
351
+ // Count Unicode code points, not UTF-16 code units (String.length), so
352
+ // non-BMP text (e.g. emoji) is measured the same way as the Python
353
+ // adapters' character counts.
354
+ if (!hasAtLeastChars(input, MIN_TOON_CHARS)) return null;
329
355
  const beforeChars = input.length;
330
356
  const args = ["compress-toon", "--agent-id", "openclaw"];
331
357
  if (sessionId) args.push("--session-id", sessionId);
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "id": "tokenless",
3
3
  "name": "Tokenless",
4
- "version": "0.7.7",
4
+ "version": "0.7.10",
5
5
  "description": "Unified RTK command rewriting + response/TOON compression + registered but hard-disabled Tool Ready. Wraps tokenless and rtk system binaries via child_process — this is expected and not malicious.",
6
6
  "activation": {
7
7
  "onCapabilities": ["hook"]
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@tokenless/openclaw-plugin",
3
- "version": "0.7.7",
3
+ "version": "0.7.10",
4
4
  "description": "Unified OpenClaw plugin — RTK command rewriting + tokenless schema/response compression for 60-90% LLM token savings",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenless",
3
- "version": "0.7.7",
3
+ "version": "0.7.10",
4
4
  "description": "Token-Less LLM Token Optimization Toolkit for Qoder CLI",
5
5
  "author": {
6
6
  "name": "Shile Zhang",
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenless",
3
- "version": "0.7.7",
3
+ "version": "0.7.10",
4
4
  "description": "Token-Less context compression for Qwen Code — RTK command rewriting, response/TOON/schema compression, and registered but hard-disabled Tool Ready",
5
5
  "author": { "name": "ANOLISA" },
6
6
  "license": "Apache-2.0",
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "anolisa-tokenless",
3
3
  "type": "module",
4
- "version": "0.7.7",
4
+ "version": "0.7.10",
5
5
  "description": "Token-Less — LLM token optimization toolkit (schema/response compression, command rewriting, tool readiness)",
6
6
  "license": "Apache-2.0",
7
7
  "repository": {
@@ -20,8 +20,7 @@
20
20
  ],
21
21
  "bin": {
22
22
  "tokenless": "bin/tokenless",
23
- "rtk": "bin/rtk",
24
- "toon": "bin/toon"
23
+ "rtk": "bin/rtk"
25
24
  },
26
25
  "files": [
27
26
  "bin/",
@@ -45,10 +44,10 @@
45
44
  "arm64"
46
45
  ],
47
46
  "optionalDependencies": {
48
- "@anolisa/tokenless-linux-x64": "0.7.7",
49
- "@anolisa/tokenless-linux-arm64": "0.7.7",
50
- "@anolisa/tokenless-darwin-x64": "0.7.7",
51
- "@anolisa/tokenless-darwin-arm64": "0.7.7"
47
+ "@anolisa/tokenless-linux-x64": "0.7.10",
48
+ "@anolisa/tokenless-linux-arm64": "0.7.10",
49
+ "@anolisa/tokenless-darwin-x64": "0.7.10",
50
+ "@anolisa/tokenless-darwin-arm64": "0.7.10"
52
51
  },
53
52
  "publishConfig": {
54
53
  "registry": "https://registry.npmjs.org/",
@@ -17,8 +17,8 @@
17
17
  * Platform packages follow the naming convention:
18
18
  * @anolisa/tokenless-{os}-{arch}
19
19
  *
20
- * Each platform package ships three native binaries:
21
- * bin/tokenless, bin/rtk, bin/toon
20
+ * Each platform package ships two native binaries:
21
+ * bin/tokenless, bin/rtk
22
22
  *
23
23
  * Exit codes: on a supported platform, a missing platform package or a
24
24
  * missing binary is a hard failure (non-zero exit) so `npm install` fails
@@ -47,7 +47,7 @@ const require = createRequire(import.meta.url);
47
47
  const packageRoot = join(__dirname, '..');
48
48
  const binDir = join(packageRoot, 'bin');
49
49
 
50
- const BINARIES = ['tokenless', 'rtk', 'toon'];
50
+ const BINARIES = ['tokenless', 'rtk'];
51
51
 
52
52
  // Map Node.js platform/arch to package names
53
53
  const PLATFORM_MAP = {
package/bin/toon DELETED
@@ -1,6 +0,0 @@
1
- #!/usr/bin/env node
2
- console.error(
3
- 'anolisa-tokenless: postinstall has not run yet. ' +
4
- 'Run "npm rebuild anolisa-tokenless" to fix.',
5
- );
6
- process.exit(1);