anolisa-tokenless 0.7.14 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. package/README.md +206 -79
  2. package/adapters/tokenless/claude-code/.claude-plugin/plugin.json +1 -1
  3. package/adapters/tokenless/claude-code/hooks/run-hook.sh +62 -0
  4. package/adapters/tokenless/codex/.codex-plugin/plugin.json +1 -1
  5. package/adapters/tokenless/common/cosh-extension.json +4 -4
  6. package/adapters/tokenless/common/hooks/compress_response_hook.py +180 -96
  7. package/adapters/tokenless/common/hooks/compress_schema_hook.py +19 -31
  8. package/adapters/tokenless/common/hooks/hook_utils.py +240 -70
  9. package/adapters/tokenless/common/hooks/rewrite_hook.py +53 -169
  10. package/adapters/tokenless/dsh/dist/index.js +353 -264
  11. package/adapters/tokenless/dsh/package.json +2 -2
  12. package/adapters/tokenless/hermes/__init__.py +191 -357
  13. package/adapters/tokenless/hermes/plugin.yaml +2 -2
  14. package/adapters/tokenless/manifest.json +17 -1
  15. package/adapters/tokenless/openclaw/dist/index.d.ts +4 -16
  16. package/adapters/tokenless/openclaw/dist/index.js +291 -507
  17. package/adapters/tokenless/openclaw/index.ts +408 -628
  18. package/adapters/tokenless/openclaw/openclaw.plugin.json +4 -20
  19. package/adapters/tokenless/openclaw/package.json +6 -4
  20. package/adapters/tokenless/qoder/.qoder-plugin/plugin.json +1 -1
  21. package/adapters/tokenless/qwencode/hooks/run-hook.sh +62 -0
  22. package/adapters/tokenless/qwencode/qwen-extension.json +4 -4
  23. package/adapters/tokenless/qwenpaw/plugin.json +17 -0
  24. package/adapters/tokenless/qwenpaw/plugin.py +390 -0
  25. package/adapters/tokenless/qwenpaw/requirements.txt +6 -0
  26. package/adapters/tokenless/qwenpaw/scripts/detect.sh +131 -0
  27. package/adapters/tokenless/qwenpaw/scripts/install.sh +98 -0
  28. package/adapters/tokenless/qwenpaw/scripts/uninstall.sh +61 -0
  29. package/package.json +5 -5
  30. package/adapters/tokenless/common/hooks/compress_toon_hook.py +0 -174
package/README.md CHANGED
@@ -2,52 +2,58 @@
2
2
 
3
3
  [中文版](README_zh.md)
4
4
 
5
- **LLM token optimization toolkit** — schema/response compression + command rewriting + tool environment readiness.
5
+ **LLM token optimization toolkit** — content-aware compression + command rewriting + diagnostics.
6
6
 
7
7
  Token-Less combines complementary strategies to minimize LLM token consumption:
8
8
 
9
- - **Schema & Response Compression** — Compresses OpenAI Function Calling tool definitions and API responses via the `tokenless-schema` library, cutting structural overhead before tokens ever reach the context window.
9
+ - **Lifecycle-aware Compression** — Protocol v2 owns BeforeModel schema handling, PreTool RTK rewriting, PostTool routing, and authorized Retrieve; the PostTool Pipeline compresses JSON and recognized build/test command logs.
10
10
  - **TOON Context Compression** — Encodes JSON responses to TOON (Token-Oriented Object Notation) format via the `toon-format` library linked into `tokenless`, reducing syntax overhead for suitable structured data.
11
11
  - **Command Rewriting** — Integrates [RTK](https://github.com/rtk-ai/rtk) to filter and rewrite CLI command output, eliminating noise that would otherwise waste 60–90% of tokens.
12
12
  - **Tool Ready (legacy, hard-disabled)** — Its pre-call dependency checks are retained in source but unconditionally bypassed while the readiness model is redesigned.
13
13
 
14
14
  Agent adapters are available for:
15
15
 
16
- - **OpenClaw plugin** — covers command rewriting, response compression, and schema compression in one plugin.
16
+ - **OpenClaw plugin** — delegates PreTool RTK rewriting and PostTool optimization to Protocol v2 Core.
17
17
  - **copilot-shell hook** — intercepts Shell commands via a PreToolUse hook and delegates to RTK for command rewriting + output filtering.
18
- - **Hermes Agent plugin** — response compression, TOON encoding, command rewriting (block + suggest), and registered but hard-disabled Tool Ready via Hermes's native plugin system.
19
- - **Qoder CLI plugin** — registered but hard-disabled Tool Ready, command rewriting, and response compression via Qoder's native hook system.
20
- - **Claude Code plugin** — RTK command rewriting, response/TOON compression, and registered but hard-disabled Tool Ready via Claude Code's official plugin marketplace.
18
+ - **Hermes Agent plugin** — delegates block-and-suggest command rewriting and model-bound result optimization to Core, with Marker-directed recovery through its shell tool.
19
+ - **Qoder CLI plugin** — registered but hard-disabled Tool Ready, command rewriting, response compression, and Marker-directed recovery via Qoder's native hook system.
20
+ - **Claude Code plugin** — RTK command rewriting, response/TOON compression, Marker-directed recovery, and registered but hard-disabled Tool Ready via Claude Code's official plugin marketplace.
21
21
  - **Codex plugin** — RTK command rewriting, environment-failure diagnostics, and registered but hard-disabled Tool Ready via Codex's native hook system.
22
- - **OpenCode plugin** — schema/response/TOON compression, registered but hard-disabled Tool Ready, and command rewriting via OpenCode's local plugin API.
23
- - **DeepSeek Harness plugin** — native response compression and environment-error attribution through DSH's `tools/post-execute` seam.
22
+ - **OpenCode plugin** — schema/response/TOON compression, Marker-directed recovery, registered but hard-disabled Tool Ready, and command rewriting via OpenCode's local plugin API.
23
+ - **Qwen Code extension** — command rewriting and registered but hard-disabled Tool Ready; current host releases cannot replace post-tool output and skip the declared schema event.
24
+ - **QwenPaw plugin** — schema compression, RTK command rewriting, response/TOON compression, and static-tool recovery through an AgentScope middleware registered by QwenPaw's plugin system; the plugin embeds the `anolisa_tokenless` wheel in-process.
25
+ - **DeepSeek Harness plugin** — native response compression, Marker-directed recovery, and environment-error attribution through DSH's `tools/post-execute` seam.
24
26
 
25
- For framework developers, the self-contained Python SDK and separate **AgentScope integration**
26
- cover schema compression, RTK rewriting, response compression, TOON, retrieval, and attribution.
27
+ For framework developers, the Python SDK has a framework-neutral layer and an **AgentScope-specific
28
+ layer**. Together they cover schema compression, RTK rewriting, response compression, TOON,
29
+ retrieval, and attribution.
27
30
 
28
31
  ## Features
29
32
 
30
33
  | Capability | Savings indicator | Details |
31
34
  |---|---|---|
32
35
  | Schema compression | 47.3% on reference fixture | Compresses OpenAI Function Calling tool schemas |
33
- | Response compression | 65.8% on reference fixture | Compresses API / tool responses |
34
- | Reversible compression (stash) | | Dropped array items are stashed and retrievable via `<<tokenless:KEY>>` markers |
36
+ | Content-aware response compression | 36.3% lossless savings on the JSON reference fixture | Routes successful JSON through `JsonCompressor`; lossless candidates saving at least 15% take priority, while recoverable record arrays can be reduced to a 32-record base budget |
37
+ | Build-log compression | workload-dependent | Cleans terminal control output and reduces repeated routine progress in recognized Cargo, pytest, npm/Jest, Go, Make/C, and generic command logs while preserving diagnostics, summaries, phases, and stack traces |
38
+ | Reversible compression (stash) | — | Omitted record collections and bounded values are stashed; supported agents run `tokenless retrieve HASH` or call their static Retrieve Tool when full data is needed |
35
39
  | TOON context compression | 17.0% on reference response | Encodes JSON to TOON format for LLMs |
36
40
  | Command rewriting | 60–90% | Filters CLI output via RTK (70+ commands supported) |
37
41
  | Tool Ready | reduces retry waste | Legacy pre-call check, auto-fix, and blocking; hard-disabled |
38
- | OpenClaw plugin | — | Command rewriting ✅, Response compression ✅, Schema compression |
39
- | copilot-shell hooks | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Response compression ✅, TOON ✅, Schema compression |
40
- | Hermes Agent plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Response compression ✅, TOON ✅, Schema compression |
41
- | Qoder CLI plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Response compression ✅ |
42
- | Claude Code plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Response compression ✅, TOON ✅ |
42
+ | OpenClaw plugin | — | RTK ✅, lossless transcript PostTool ✅, Schema/Retrieve unavailable in the host |
43
+ | copilot-shell hooks | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅; Cosh-NG supports response compression and Marker-command recovery, while legacy copilot-shell remains lossless-only; Common BeforeModel passes schemas through without authorized Retrieve |
44
+ | Hermes Agent plugin | — | Tool Ready ⛔ hard-disabled, Core-owned command rewriting/response/TOON ✅, Marker-command recovery ✅, Schema unavailable |
45
+ | Qoder CLI plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Response compression ✅, Marker-command recovery ✅ |
46
+ | Claude Code plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Response compression ✅, TOON ✅, Marker-command recovery on Claude Code 2.1.121 or newer |
43
47
  | Codex plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Environment diagnostics ✅, Response compression — protocol-blocked |
44
- | OpenCode plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Schema compression ✅, Response compression ✅, TOON ✅ |
45
- | DeepSeek Harness plugin | — | Response compression ✅, Environment-error attribution |
48
+ | OpenCode plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Schema compression ✅, Response compression ✅, TOON ✅, Marker-command recovery ✅ |
49
+ | Qwen Code extension | — | Tool Ready hard-disabled, Command rewriting ✅, Response/Schema replacement unavailable in current host |
50
+ | QwenPaw plugin | — | Schema compression ✅, Command rewriting ✅, Response compression ✅, TOON ✅, Retrieve Tool recovery ✅ |
51
+ | DeepSeek Harness plugin | — | Response compression ✅, Marker-command recovery ✅, Environment-error attribution ✅ |
46
52
  | AgentScope framework integration | — | Schema ✅, RTK ✅, Response ✅, TOON ✅, Retrieval ✅ |
47
53
  | Zero runtime deps | — | Pure Rust, single static binary |
48
54
 
49
- The schema, response, and TOON figures above are isolated Tokenless 0.7.11
50
- results on the repository's committed reference fixtures; they are neither a
55
+ The schema, response, and TOON figures above are isolated results on the
56
+ repository's committed reference fixtures; they are neither a
51
57
  production range nor additive. Compression depends on payload size and shape,
52
58
  removable fields, configured thresholds, and the share of tool data in the
53
59
  session. Short or already compact payloads may save only a few percent or pass
@@ -65,7 +71,7 @@ on the share and shape of that content in the session.
65
71
 
66
72
  | Workload | Primary strategy | Why |
67
73
  |----------|-----------------|-----|
68
- | Shell-heavy (build/test/triage) | Command rewriting (RTK) | `cargo`/`npm`/`go`/`pytest` output carries lots of progress/warning noise; RTK cuts 60–90% |
74
+ | Shell-heavy (build/test/triage) | Build-log compression + RTK | Recognized build/test commands keep their native output for PostTool compression; other supported shell commands use RTK |
69
75
  | API/fetch-heavy (REST, web_fetch) | Response compression + TOON | JSON may carry removable debug/null/empty fields; sufficiently large, regular structures also have reducible syntax overhead |
70
76
  | Agents with many tools | Schema compression | Many Function Calling definitions carry verbose descriptions and removable metadata |
71
77
  | Long responses that must stay faithful | Reversible compression (Stash) | Truncated content is `retrieve`-able end-to-end lossless; thresholds can be tightened safely |
@@ -97,16 +103,18 @@ on the share and shape of that content in the session.
97
103
 
98
104
  Example: dashboard shows 60% compression rate, but if tool responses account for 20% of total consumption, the actual savings rate is 60% × 20% = **12%**. This is why savings feel "lighter than a feather" in experiments consuming 15 million tokens — tokenless only optimizes the ~3 million tokens of tool responses.
99
105
 
100
- > Stash makes compression **end-to-end lossless**: you can tighten truncation thresholds for higher inline savings and recover the original via the `<<tokenless:KEY>>` marker when needed, with no correctness impact. Use `TOKENLESS_COMPRESSION_ENABLED=0/1` dual runs to compare real savings.
106
+ > Recoverable omissions include an optional action: `If needed, run in shell: tokenless retrieve HASH`. AgentScope instead sees `If needed, call tool tokenless_retrieve with hash_or_marker=HASH` (using its configured static Tool name). Recovery returns the saved payload while it remains in Stash; it costs additional tokens and is not required for every omission. Historical `<<tokenless:HASH>>` markers remain readable but are no longer generated. Use `TOKENLESS_COMPRESSION_ENABLED=0/1` dual runs to compare real savings.
101
107
  > See [user manual](../../docs/user-guide/en/token-saving/tokenless/user-manual.md) for per-strategy trigger conditions.
102
108
 
103
109
  ## Architecture
104
110
 
105
111
  ```
106
112
  Token-Less/
107
- ├── crates/tokenless-schema/ # Core library: SchemaCompressor + ResponseCompressor
113
+ ├── crates/tokenless-schema/ # BeforeModel tool-schema compressor
108
114
  ├── crates/tokenless-ccr/ # Reversible compression stash (Compress-Cache-Retrieve)
109
- ├── crates/tokenless-runtime/ # Stateful in-process compression and retrieval API
115
+ ├── crates/tokenless-runtime/ # Lifecycle API and Runtime-owned PostTool pipeline
116
+ ├── crates/tokenless-protocol/ # Versioned adapter contract and token estimator
117
+ ├── crates/tokenless-compressors/ # JSON and build-log domain compressors
110
118
  ├── crates/tokenless-cli/ # CLI binary: `tokenless` command (env-check, compress, retrieve, stats)
111
119
  ├── python/tokenless/ # PyO3 package: `anolisa_tokenless`
112
120
  ├── python/agentscope/ # Pure-Python AgentScope integration package
@@ -124,6 +132,7 @@ Token-Less/
124
132
  │ ├── claude-code/ # Claude Code plugin + marketplace + hooks
125
133
  │ ├── codex/ # Codex plugin + scripts
126
134
  │ ├── opencode/ # OpenCode local plugin + scripts
135
+ │ ├── qwenpaw/ # QwenPaw plugin (AgentScope middleware) + scripts
127
136
  │ └── dsh/ # Native DeepSeek Harness bundle
128
137
  ├── third_party/rtk/ # RTK vendored source (justfile clone+patch from GitHub)
129
138
  ├── third_party/patches/ # Patches for vendored third_party sources
@@ -200,7 +209,7 @@ make setup
200
209
  The source setup installs `tokenless` to `~/.local/bin`, places the `rtk`
201
210
  helper alongside it, and deploys all adapters for development.
202
211
 
203
- ### Build the Python runtime
212
+ ### Build the Python SDK
204
213
 
205
214
  Framework authors can build the in-process Python API from source:
206
215
 
@@ -222,8 +231,12 @@ where its native wheel was built. It exposes the four Tokenless lifecycle
222
231
  methods and bundles the matching RTK executable; TOON is linked into the native
223
232
  runtime. It does not require the Tokenless CLI or system helper binaries. The
224
233
  package is built and tested in this repository but is not yet published to
225
- PyPI. See the [runtime design](docs/design/runtime-library.md)
226
- and the [user manual](../../docs/user-guide/en/token-saving/tokenless/user-manual.md#build-the-python-runtime-from-source).
234
+ PyPI. See the [Python SDK guide](../../docs/user-guide/en/token-saving/tokenless/sdk.md) for
235
+ runnable lifecycle and Stats examples, the
236
+ [AgentScope SDK integration](../../docs/user-guide/en/token-saving/tokenless/sdk/agentscope.md) for
237
+ AgentScope attachment, the
238
+ [Agent integration guide](../../docs/user-guide/en/token-saving/tokenless/framework-integration.md)
239
+ for product adapters, and the [runtime design](docs/design/runtime-library.md) for internal contracts.
227
240
 
228
241
  The same wheel provides typed, read-only statistics queries without requiring
229
242
  the CLI. Point `TokenlessStats` at the state directory used by the runtime, or
@@ -255,7 +268,31 @@ The description, string, array, and depth limits in the
255
268
  [CLI reference](../../docs/user-guide/en/token-saving/tokenless/cli-reference.md)
256
269
  trigger individual transformations; they are not minimum total payload sizes.
257
270
  Agent adapters may apply separate pre-check thresholds; see the
258
- [framework integration guide](../../docs/user-guide/en/token-saving/tokenless/framework-integration.md#adapter-processing-rules).
271
+ [Agent integration guide](../../docs/user-guide/en/token-saving/tokenless/framework-integration.md#adapter-processing-rules).
272
+
273
+ ### compress
274
+
275
+ Shared Agent hooks send lifecycle requests to `tokenless compress`; only
276
+ successful, non-bypassed PostTool JSON and eligible command-output build logs
277
+ enter the Runtime-owned Pipeline. Tool errors bypass compression and keep their
278
+ original output while Core attaches environment-diagnostic context.
279
+
280
+ PreTool leaves recognized Cargo, pytest, npm/Jest, Go, and Make build/test
281
+ commands unchanged so their native output has a single PostTool owner. Other
282
+ supported commands may be rewritten by RTK, and their results continue to
283
+ bypass PostTool compression.
284
+
285
+ Claude Code 2.1.121 or newer, Qoder CLI, OpenCode, and Cosh-NG can replace the
286
+ live result. Their PostTool requests enable recovery when bare `tokenless`
287
+ also resolves on the shell `PATH`. A compression Marker then tells the model to
288
+ run the exact `tokenless retrieve` command through the existing shell tool. The
289
+ hook recognizes only a successful, standalone command
290
+ with a valid Hash or Marker and sends its output through the Retrieve bypass,
291
+ preventing a second compression pass. Legacy copilot-shell and other hosts that
292
+ cannot replace the result remain lossless-only. BeforeModel Schema compression
293
+ still requires its separate authorized recovery capability. See the
294
+ [CLI reference](../../docs/user-guide/en/token-saving/tokenless/cli-reference.md#compress)
295
+ for the request/response contract and an executable example.
259
296
 
260
297
  ### compress-schema
261
298
 
@@ -314,9 +351,8 @@ hash or any text containing a `<<tokenless:HASH>>` marker:
314
351
  # Bare hash
315
352
  tokenless retrieve c30ccf5ed1125e0ed871ba8e
316
353
 
317
- # Or paste the whole truncation line — the hash is extracted automatically.
318
- # (Use the FULL 24-hex hash from your output; the value below is shorthand.)
319
- tokenless retrieve "<... 160 items truncated, run: tokenless retrieve '<<tokenless:c30ccf5ed1125e0ed871ba8e>>'>"
354
+ # Historical markers remain accepted
355
+ tokenless retrieve '<<tokenless:c30ccf5ed1125e0ed871ba8e>>'
320
356
  ```
321
357
 
322
358
  ### compress-toon / decompress-toon
@@ -386,7 +422,7 @@ The adapter provides hooks that are auto-discovered by copilot-shell via the cos
386
422
  | Tool Ready (hard-disabled) | PreToolUse (all tools) | `tool_ready_hook.sh` | Silent pass-through; no check, repair, context, or block |
387
423
  | Command rewriting | PreToolUse (Shell) | `rewrite_hook.py` | Rewrite commands via RTK |
388
424
  | Response compression + attribution + TOON | PostToolUse | `compress_response_hook.py` | Compress + env error attribution + TOON |
389
- | Schema compression | BeforeModel | `compress_schema_hook.py` | Compress tool schemas |
425
+ | Schema compression | BeforeModel | `compress_schema_hook.py` | Passes through lossy transformations until the host exposes marker-authorized recovery |
390
426
 
391
427
  ### Install
392
428
 
@@ -463,49 +499,69 @@ String format `"jq"` is also supported (auto-converts to object).
463
499
 
464
500
  ## OpenClaw Plugin
465
501
 
466
- The plugin hooks into the OpenClaw agent loop at two stages:
502
+ The plugin translates two OpenClaw events into Protocol v2 lifecycle operations:
467
503
 
468
504
  | Hook | Event | Action | Status |
469
505
  |---|---|---|---|
470
506
  | Tool Ready | `before_tool_call` | Registered silent pass-through; no check, repair, context, or block | ⛔ Hard-disabled |
471
- | Command rewriting | `before_tool_call` | Rewrites `exec` commands to RTK equivalents for filtered output | ✅ Active |
472
- | Response compression | `tool_result_persist` | Compresses tool results before they enter the context window | ✅ Active |
473
- | Schema compression | — | Not supported by OpenClaw's hook system | |
507
+ | PreTool | `before_tool_call` | Sends `exec` arguments to Core and applies the returned RTK rewrite | ✅ Active |
508
+ | PostTool | `tool_result_persist` | Rewrites supported OpenClaw-owned transcript tool results | ✅ Active |
509
+ | BeforeModel / Retrieve | — | OpenClaw exposes neither a reliable schema-transform seam nor marker-authorized recovery | — |
474
510
 
475
- **Response compression details:**
476
- - Automatically compresses results from all tool types (`web_search`, `web_fetch`, `read_file`, etc.)
477
- - Skips `exec` tool results when RTK is enabled RTK already produces optimized output, avoiding double-compression
478
- - Observed savings: **~78%** on `web_fetch` results, varies by content type
511
+ Core owns RTK execution, JSON detection, cleanup, TOON selection, thresholds, diagnostics, and final
512
+ arbitration. The plugin carries Core's per-call `output_optimization` from PreTool into the matching
513
+ PostTool request, so RTK output is not compressed twice. The local CLI recovery command is a trusted
514
+ operator entry rather than Agent authorization, so the plugin applies only lossless candidates.
479
515
 
480
- Each hook degrades gracefully if the corresponding binary (`rtk` or `tokenless`) is not installed, that hook is silently skipped.
516
+ `tool_result_persist` is a synchronous OpenClaw transcript seam. It can replace a persisted string,
517
+ a structured value, or a single text block while preserving the surrounding Tool Result envelope;
518
+ media and multi-block results pass through. It does not replace a tool result already consumed by
519
+ the model in the same turn, and it does not cover non-OpenClaw transcript implementations.
520
+
521
+ Both operations use the single `tokenless compress` entry point and fail open if Tokenless is
522
+ missing or returns an invalid response.
481
523
 
482
524
  ### Configuration
483
525
 
526
+ The adapter requires OpenClaw Plugin API `2026.4.22` or newer; package metadata enforces this
527
+ minimum during installation on hosts that support compatibility checks.
528
+
484
529
  Options in `openclaw.plugin.json`:
485
530
 
486
531
  | Option | Default | Description |
487
532
  |---|---|---|
488
533
  | `rtk_enabled` | `true` | Enable RTK command rewriting |
489
- | `schema_compression_enabled` | `true` | Enable tool schema compression (pending OpenClaw support) |
490
- | `response_compression_enabled` | `true` | Enable tool response compression via `tool_result_persist` |
491
- | `verbose` | `true` | Log detailed rewrite/compression info |
534
+ | `post_tool_enabled` | `true` | Enable Protocol v2 PostTool handling of persisted tool results |
535
+ | `tool_ready_enabled` | `true` | Register the currently hard-disabled Tool Ready hook |
536
+ | `verbose` | `false` | Log lifecycle rewrites and applied PostTool results |
537
+
538
+ The previous response, TOON, skip-tool, and shell-tool configuration keys are removed; Core now
539
+ owns those decisions.
492
540
 
493
541
  ## Hermes Agent Plugin
494
542
 
495
- The plugin registers hooks at three Hermes events, covering five strategies:
543
+ The plugin registers hooks at three Hermes events while Core owns the lifecycle policy:
496
544
 
497
545
  | Strategy | Event | Action | Status |
498
546
  |---|---|---|---|
499
547
  | Tool Ready | `pre_tool_call` | Registered silent pass-through; no check, repair, context, or block | ⛔ Hard-disabled |
500
- | Command rewriting | `pre_tool_call` | Blocks original command, suggests `rtk`-rewritten version (one extra round-trip) | ✅ Active |
501
- | Response compression | `transform_tool_result` | Compresses tool results via `tokenless compress-response` | ✅ Active |
502
- | TOON encoding | `transform_tool_result` | Pipeline step after response compression — encodes JSON to TOON format | ✅ Active |
548
+ | Command rewriting | `pre_tool_call` | Sends the command to Core, then blocks and suggests the returned RTK form | ✅ Active |
549
+ | PostTool optimization | `transform_tool_result` | Sends the final model-bound result to Core and applies accepted output | ✅ Active |
503
550
  | Session tracking | `on_session_start` | Propagates agent/session IDs for stats recording | ✅ Active |
504
- | Schema compression | — | Not supported by Hermes hook system (no hook exposes tool schemas) | Blocked |
551
+ | Schema compression | — | Hermes exposes no schema-transform seam | |
552
+ | Marker recovery | `transform_tool_result` | Labels a successful standalone `tokenless retrieve` shell result for the Core bypass | ✅ Active |
505
553
 
506
- **How command rewriting works in Hermes**: Hermes's `pre_tool_call` hook can only block tool execution (not modify arguments), so the plugin blocks the original shell command and returns a message suggesting the RTK-rewritten version. The agent then re-executes with the optimized command, adding one extra tool-call round-trip. This is safe — `rtk rewrite` only does text substitution and never executes the command.
554
+ **How command rewriting works in Hermes**: to remain compatible with Hermes releases that only
555
+ support blocking, the plugin asks Core for a rewrite, blocks the original shell command, and tells
556
+ the agent to retry with the returned command. The retry adds one tool-call round-trip. The final
557
+ hook recognizes Core's attributed RTK wrapper from the command Hermes actually executed, so RTK
558
+ output bypasses a second compression pass without correlating two different tool-call IDs.
507
559
 
508
- Each hook degrades gracefully if the corresponding binary is not installed, that hook is silently skipped.
560
+ When compression omits recoverable data, the Marker tells Hermes to run
561
+ `tokenless retrieve` through its existing shell tool. The adapter checks the
562
+ actual executed command; a successful standalone retrieve is returned unchanged
563
+ and never compressed again. If the Tokenless operation is unavailable or fails,
564
+ the hook leaves the host value unchanged.
509
565
 
510
566
  ### Install
511
567
 
@@ -539,6 +595,10 @@ The plugin registers hooks at three Qoder events, covering three strategies:
539
595
 
540
596
  Each hook degrades gracefully — if the corresponding binary is not installed, that hook is silently skipped.
541
597
 
598
+ When a compressed result contains a Retrieve Marker, Qoder can execute its
599
+ command through the existing shell tool. The successful result bypasses response
600
+ compression so the recovered payload reaches the model unchanged.
601
+
542
602
  ### Install
543
603
 
544
604
  ```bash
@@ -558,6 +618,10 @@ The plugin registers hooks at two Claude Code events, covering four strategies:
558
618
 
559
619
  Claude Code v2 requires plugins to be sourced from a registered marketplace. We expose the adapter's `claude-code/` directory as a single-plugin marketplace (`anolisa-tokenless`), then install `tokenless@anolisa-tokenless` from it. The marketplace name is component-scoped so multiple ANOLISA components can each register their own without colliding.
560
620
 
621
+ On Claude Code 2.1.121 or newer, a compressed result can direct the model to
622
+ run `tokenless retrieve` through Bash. The successful command result bypasses
623
+ compression and restores the complete payload.
624
+
561
625
  ### Install
562
626
 
563
627
  ```bash
@@ -609,13 +673,55 @@ The installer creates a `tokenless.js` symbolic link in OpenCode's global
609
673
  `OPENCODE_CONFIG_DIR`, `XDG_CONFIG_HOME`, and the explicit
610
674
  `TOKENLESS_OPENCODE_CONFIG_DIR` override.
611
675
 
676
+ If a response contains a Retrieve Marker, OpenCode can run the embedded
677
+ `tokenless retrieve` command through its existing shell tool. The adapter sends
678
+ the successful recovery result through the Core bypass without recompressing it.
679
+
680
+ ## QwenPaw Plugin
681
+
682
+ The QwenPaw adapter is a native QwenPaw plugin. Its `plugin.py` registers an
683
+ AgentScope middleware through `api.register_middleware` and a
684
+ `tokenless_retrieve` tool through `api.register_tool`, and calls the in-process
685
+ `anolisa_tokenless.TokenlessSdk` directly:
686
+
687
+ | Feature | Middleware hook | Behavior | Status |
688
+ |---|---|---|---|
689
+ | Schema compression | `on_model_call` | Compresses tool schemas and appends the retrieve tool | ✅ Active |
690
+ | Command rewriting | `on_acting` | Rewrites `execute_shell_command` input via RTK after QwenPaw's approval step | ✅ Active |
691
+ | Response + TOON compression | `on_acting` | Replaces text blocks of the tool result for QwenPaw's built-in tools; file readers and tools outside the built-in table pass through untouched | ✅ Active |
692
+ | Recovery | `tokenless_retrieve` tool | Restores omitted content from the hash in a visible recovery instruction | ✅ Active |
693
+
694
+ ```bash
695
+ make qwenpaw-install
696
+ ```
697
+
698
+ The installer runs `qwenpaw plugin install <bundle> --force`; QwenPaw copies the
699
+ bundle into `<working dir>/plugins/tokenless/` (`QWENPAW_WORKING_DIR`, else
700
+ `COPAW_WORKING_DIR`, else an existing `~/.copaw`, else `~/.qwenpaw`) and installs
701
+ the `anolisa_tokenless` wheel listed in `requirements.txt` from the matching
702
+ GitHub Release. Records are written under `<workspace>/.tokenless`.
703
+
612
704
  ## DeepSeek Harness Plugin
613
705
 
614
- The native DSH bundle compresses successful single-block JSON tool results
615
- through `tools/post-execute` and keeps the original result unless the Tokenless
616
- CLI returns strictly smaller valid JSON. Content-retrieval tools remain
617
- lossless by default. Environment-error attribution stays active when response
618
- compression is disabled, skipped, or unable to reduce the result.
706
+ The native DSH bundle sends replaceable single-text tool results through
707
+ Tokenless PostTool Core on `tools/post-execute`. Core owns content detection,
708
+ JSON cleanup, TOON selection, acceptance, and environment-error diagnostics.
709
+ When a reduced response contains a Retrieve Marker, DSH can run the embedded
710
+ `tokenless retrieve` command through its existing shell tool. The adapter
711
+ classifies the successful standalone command from `exec.arguments.command` and
712
+ sends its output through the Core Retrieve bypass, so it remains unchanged.
713
+ Recoverable compression is enabled only when bare `tokenless` resolves on the
714
+ shell `PATH` to the same executable selected by `tokenlessBin` or
715
+ `TOKENLESS_BIN` for Core. An absolute plugin-only or different binary is not
716
+ sufficient because the Marker emits the bare command.
717
+ DSH removes inherited `TOKENLESS_*` variables from model shell commands, so the
718
+ adapter publishes the selected state directory and optional statistics/Stash
719
+ database overrides as managed DSH shell facts. Core uses the same paths. The
720
+ data directory defaults to `.tokenless` in the session workspace and contains
721
+ a self-ignoring `.gitignore`; set `TOKENLESS_DATA_DIR`, `TOKENLESS_STATS_DB`, or
722
+ `TOKENLESS_STASH_DB` before starting DSH to select other absolute paths that
723
+ DSH's shell sandbox can access.
724
+ Error guidance stays active when response compression is disabled.
619
725
 
620
726
  Enable the bundle for every desired DSH profile in one command by repeating
621
727
  `--profile`:
@@ -659,13 +765,16 @@ Installation requires an explicit session identifier.
659
765
 
660
766
  ```python
661
767
  from agentscope.agent import ReActAgent
662
- from tokenless_agentscope import TokenlessAgentScope, TokenlessConfig
768
+ from anolisa_tokenless import ContentOrigin
769
+ from tokenless_agentscope import TokenlessAgentScope, TokenlessConfig, ToolContract
663
770
 
664
771
  integration = TokenlessAgentScope(
665
772
  TokenlessConfig(
666
- mode="balanced",
667
773
  data_dir="/absolute/path/to/tenant-tokenless-data",
668
774
  ),
775
+ tool_contracts={
776
+ "application_tool": ToolContract(ContentOrigin.API_RESPONSE),
777
+ },
669
778
  )
670
779
  toolkit = integration.create_toolkit()
671
780
  toolkit.register_tool_function(application_tool)
@@ -680,14 +789,17 @@ patch versions.
680
789
  ```python
681
790
  from agentscope.agent import Agent
682
791
  from agentscope.tool import Toolkit
683
- from tokenless_agentscope import TokenlessAgentScope, TokenlessConfig
792
+ from anolisa_tokenless import ContentOrigin
793
+ from tokenless_agentscope import TokenlessAgentScope, TokenlessConfig, ToolContract
684
794
 
685
795
  integration = TokenlessAgentScope(
686
796
  TokenlessConfig(
687
- mode="balanced",
688
797
  data_dir="/absolute/path/to/tenant-tokenless-data",
689
798
  # retrieve_tool_name="tenant_tokenless_retrieve",
690
799
  ),
800
+ tool_contracts={
801
+ "application_tool": ToolContract(ContentOrigin.API_RESPONSE),
802
+ },
691
803
  )
692
804
  toolkit = Toolkit(tools=[*application_tools, *integration.tools])
693
805
 
@@ -698,7 +810,7 @@ agent = Agent(
698
810
  )
699
811
  ```
700
812
 
701
- AgentScope App is supported from 2.0.1. It derives an isolated Tokenless data
813
+ AgentScope App is supported from 2.0.3. It derives an isolated Tokenless data
702
814
  directory for every user/agent/session below the configured absolute base
703
815
  directory:
704
816
 
@@ -708,25 +820,31 @@ from agentscope.app import create_app
708
820
  app = create_app(..., **integration.app_options())
709
821
  ```
710
822
 
823
+ `app_options()` supplies one Middleware factory. AgentScope publishes that
824
+ Middleware instance's static Retrieve Tool through `list_tools()` and persists
825
+ Marker authorization in `AgentState.middle_context`.
826
+
711
827
  Set a unique `retrieve_tool_name` in `TokenlessConfig` if the application
712
828
  already defines `tokenless_retrieve`; App assembly does not expose the other
713
829
  tools to this factory for a preflight collision check.
714
830
 
715
- AgentScope 2.0.0 does not expose App-level Agent middleware or Tool injection,
716
- so that patch release supports direct Agent construction only. The existing
717
- `TokenlessMiddleware` 2.x API remains available for compatibility; new code
718
- should use `TokenlessAgentScope` so it does not depend on patch-specific
719
- Toolkit mutation or automatic Tool collection.
720
-
721
- | Mode | Policy |
722
- |---|---|
723
- | `conservative` | Compress every non-excluded tool with 1 MiB / 65,536 / depth 32 limits |
724
- | `balanced` | Skip Read/Glob/Grep; use 65,536 / 128 / depth 8 for Shell and conservative limits elsewhere |
725
- | `aggressive` | Skip Read/Glob/Grep; use CLI defaults of 4,096 / 32 / depth 8 elsewhere |
726
-
727
- `balanced` is the default. The read-only retrieval Tool is published to the
728
- model only when a marker is visible and accepts only a hash from the exact
729
- marker set retained for that model call. Pass a different absolute `data_dir`
831
+ AgentScope 2.0.0 through 2.0.2 support direct Agent construction only; their App
832
+ APIs do not provide both Middleware-owned Tool publication and persisted
833
+ Middleware state. The existing `TokenlessMiddleware` 2.x API remains available
834
+ for compatibility; new code should use `TokenlessAgentScope`.
835
+
836
+ AgentScope supplies explicit contracts for its known shell, file, and API tools.
837
+ Register every custom tool with `ToolContract`: select `COMMAND_OUTPUT`,
838
+ `FILE_CONTENT`, or `API_RESPONSE`, and set `command_field` only for commands
839
+ that may be rewritten by RTK. Unknown custom tools fail during registration or
840
+ at the model boundary rather than guessing from output text. Compression
841
+ thresholds, TOON selection, diagnostics, and retrieval authorization remain in
842
+ Rust Core.
843
+
844
+ The read-only retrieval Tool has a static declaration and remains in the model
845
+ tool list across calls, avoiding tool-list churn when Marker visibility changes.
846
+ It accepts only a hash from the exact Marker set retained for the current model
847
+ call. Pass a different absolute `data_dir`
730
848
  to each user or tenant for direct Agents;
731
849
  `TOKENLESS_DATA_DIR` is only a process-wide fallback when `data_dir` is omitted.
732
850
  Retain the default one-hour stash TTL unless the application has a deliberate
@@ -742,6 +860,10 @@ transformed. Tool Ready remains hard-disabled.
742
860
 
743
861
  ## Build
744
862
 
863
+ For installed-package checks and optional real Agent tasks, see the
864
+ [release regression suite](tests/release_regression/README.md). It reports
865
+ tool-output savings and retrieval overhead separately.
866
+
745
867
  | Target | Description |
746
868
  |---|---|
747
869
  | `make build` | Build `tokenless` + `rtk` (release mode) |
@@ -773,6 +895,8 @@ transformed. Tool Ready remains hard-disabled.
773
895
  | `make codex-uninstall` | Remove Codex plugin |
774
896
  | `make opencode-install` | Install OpenCode local plugin |
775
897
  | `make opencode-uninstall` | Remove OpenCode local plugin |
898
+ | `make qwenpaw-install` | Install QwenPaw plugin via the qwenpaw CLI |
899
+ | `make qwenpaw-uninstall` | Remove QwenPaw plugin |
776
900
  | `make setup` | Full setup: build + install + all adapters |
777
901
 
778
902
  Override install paths:
@@ -819,8 +943,10 @@ layout and single-target interface.
819
943
  | Path | Description |
820
944
  |---|---|
821
945
  | `crates/tokenless-cli/` | CLI binary — `tokenless` command (compress, stats, env-check) |
822
- | `crates/tokenless-schema/` | Core Rust library — `SchemaCompressor` and `ResponseCompressor` |
823
- | `crates/tokenless-runtime/` | Stateful Rust API shared by the CLI and language bindings |
946
+ | `crates/tokenless-schema/` | BeforeModel tool-schema compression — `SchemaCompressor` |
947
+ | `crates/tokenless-compressors/` | Content-domain engines `JsonCompressor` and `BuildLogCompressor` are connected to PostTool |
948
+ | `crates/tokenless-runtime/` | Lifecycle API and Runtime-owned `PostToolPipeline`, shared by CLI and language bindings |
949
+ | `crates/tokenless-protocol/` | Versioned adapter contract and shared `heuristic-v1` token estimator |
824
950
  | `python/tokenless/` | PyO3 package exposing `anolisa_tokenless` for CPython 3.11+ |
825
951
  | `python/agentscope/` | Independent AgentScope framework integration and wheel metadata |
826
952
  | `adapters/tokenless/` | FHS adapter bundle — manifest, env-check spec/fix, hooks, OpenClaw plugin |
@@ -829,6 +955,7 @@ layout and single-target interface.
829
955
  | `adapters/tokenless/claude-code/` | Claude Code adapter — marketplace + plugin + hooks dispatcher |
830
956
  | `adapters/tokenless/codex/` | Codex adapter — plugin + Python hook scripts |
831
957
  | `adapters/tokenless/opencode/` | OpenCode adapter — local JavaScript plugin + lifecycle scripts |
958
+ | `adapters/tokenless/qwenpaw/` | QwenPaw adapter — plugin manifest, AgentScope middleware, wheel requirements + lifecycle scripts |
832
959
  | `third_party/rtk/` | RTK vendored source — command rewriting engine (justfile clone+patch) |
833
960
  | `third_party/patches/` | Patches for vendored third_party sources |
834
961
  | `packaging/raw/` | Component-owned ANOLISA raw packer and target validation |
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenless",
3
- "version": "0.7.14",
3
+ "version": "0.8.0",
4
4
  "description": "Token-Less context compression for Claude Code — RTK command rewriting, response/TOON compression, and Tool Ready environment pre-check",
5
5
  "author": { "name": "ANOLISA" },
6
6
  "license": "Apache-2.0",
@@ -0,0 +1,62 @@
1
+ #!/usr/bin/env bash
2
+ # run-hook.sh — Locate and exec a shared tokenless hook script.
3
+ #
4
+ # Prefer hooks from the wrapper's own adapter tree so concurrent RPM, Makefile,
5
+ # and raw installations cannot cross-load another version. Hosts that copy the
6
+ # wrapper into a detached cache still use the historical FHS fallbacks.
7
+ #
8
+ # Usage: run-hook.sh <hook-script-basename> [args...]
9
+ # Examples: run-hook.sh rewrite_hook.py
10
+ # run-hook.sh compress_response_hook.py
11
+ # run-hook.sh tool_ready_hook.sh
12
+ #
13
+ # Fail-open contract: any not-found / missing-interpreter condition emits
14
+ # an empty JSON object on stdout and exits 0, so the host never blocks
15
+ # on us.
16
+ #
17
+ # PreToolUse matcher overlap is by design: hooks.json registers both a
18
+ # Bash-specific entry (rewrite) and a catch-all entry (tool-ready). Bash
19
+ # tool calls therefore fire both — the host evaluates each matching
20
+ # matcher independently, so this is the documented way to attach a
21
+ # tool-specific hook alongside a global one. The timeout values are
22
+ # upper bounds; observed runtimes are well under them.
23
+ set -euo pipefail
24
+
25
+ SCRIPT_DIR="$(CDPATH='' cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
26
+ SCRIPT="${1:?usage: run-hook.sh <hook-script-basename> [args...]}"
27
+ shift
28
+
29
+ fail_open() { echo "{}"; exit 0; }
30
+
31
+ # Reject anything but a bare basename. Practical risk is low — hooks.json is
32
+ # RPM-installed root:root 0644 and unwritable at runtime — but a defense-in-
33
+ # depth check costs one line and stops any future caller from smuggling a
34
+ # traversal segment through this argument.
35
+ case "$SCRIPT" in
36
+ */*|"") fail_open ;;
37
+ esac
38
+
39
+ CANDIDATES=(
40
+ "${SCRIPT_DIR}/../../common/hooks/${SCRIPT}"
41
+ "/usr/local/share/anolisa/adapters/tokenless/common/hooks/${SCRIPT}"
42
+ "/usr/share/anolisa/adapters/tokenless/common/hooks/${SCRIPT}"
43
+ "${HOME}/.local/share/anolisa/adapters/tokenless/common/hooks/${SCRIPT}"
44
+ )
45
+
46
+ for candidate in "${CANDIDATES[@]}"; do
47
+ [ -f "$candidate" ] || continue
48
+ case "$candidate" in
49
+ *.py)
50
+ command -v python3 >/dev/null 2>&1 || fail_open
51
+ exec python3 "$candidate" "$@"
52
+ ;;
53
+ *.sh)
54
+ exec bash "$candidate" "$@"
55
+ ;;
56
+ *)
57
+ fail_open
58
+ ;;
59
+ esac
60
+ done
61
+
62
+ fail_open
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenless",
3
- "version": "0.7.14",
3
+ "version": "0.8.0",
4
4
  "description": "Shell command output reduction and environment error diagnostics for Codex. Rewrites supported commands through RTK before execution and classifies environment failures with actionable fix hints.",
5
5
  "keywords": [
6
6
  "rtk",
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tokenless",
3
- "version": "0.7.14",
3
+ "version": "0.8.0",
4
4
  "contextFileName": "COPILOT.md",
5
5
  "hooks": {
6
6
  "PreToolUse": [
@@ -26,8 +26,8 @@
26
26
  "type": "command",
27
27
  "name": "tokenless-rewrite",
28
28
  "description": "Rewrites shell commands via rtk for token savings",
29
- "command": "python3 ${extensionPath}/hooks/rewrite_hook.py",
30
- "timeout": 5000,
29
+ "command": "python3 ${extensionPath}/hooks/rewrite_hook.py --agent-id copilot-shell",
30
+ "timeout": 10000,
31
31
  "env": { "TOKENLESS_AGENT_ID": "copilot-shell" }
32
32
  }
33
33
  ]
@@ -40,7 +40,7 @@
40
40
  "type": "command",
41
41
  "name": "tokenless-compress-response",
42
42
  "description": "Compresses tool responses and encodes to TOON format",
43
- "command": "python3 ${extensionPath}/hooks/compress_response_hook.py",
43
+ "command": "python3 ${extensionPath}/hooks/compress_response_hook.py --agent-id copilot-shell",
44
44
  "timeout": 10000,
45
45
  "env": { "TOKENLESS_AGENT_ID": "copilot-shell" }
46
46
  }