anolisa-tokenless 0.7.7 → 0.7.9

package/README.md

@@ -22,8 +22,8 @@ Agent adapters are available for:
 - **OpenCode plugin** — schema/response/TOON compression, registered but hard-disabled Tool Ready, and command rewriting via OpenCode's local plugin API.
 - **DeepSeek Harness plugin** — native response compression and environment-error attribution through DSH's `tools/post-execute` seam.
 
-For framework developers, the separate **AgentScope Python integration** replaces successful
-final tool responses and provides a marker-scoped native retrieval Tool.
+For framework developers, the self-contained Python SDK and separate **AgentScope integration**
+cover schema compression, RTK rewriting, response compression, TOON, retrieval, and attribution.
 
 ## Features
 
@@ -43,7 +43,7 @@ final tool responses and provides a marker-scoped native retrieval Tool.
 | Codex plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Response compression ✅, TOON ✅ |
 | OpenCode plugin | — | Tool Ready ⛔ hard-disabled, Command rewriting ✅, Schema compression ✅, Response compression ✅, TOON ✅ |
 | DeepSeek Harness plugin | — | Response compression ✅, Environment-error attribution ✅ |
-| AgentScope framework integration | — | Response compression ✅, Native retrieval Tool ✅ |
+| AgentScope framework integration | — | Schema ✅, RTK ✅, Response ✅, TOON ✅, Retrieval ✅ |
 | Zero runtime deps | — | Pure Rust, single static binary |
 
 ## Applicable Scenarios & Expected Effects
@@ -201,10 +201,11 @@ uses `uvx` to provision Maturin by default. Install
 plain workspace-default Cargo commands exclude the Python extension.
 
 The `anolisa_tokenless` module supports CPython 3.11 and later on the platform
-where its native wheel was built. It currently exposes JSON response
-compression and Stash retrieval; it does not bundle the CLI, RTK, TOON, or a
-framework integration. The package is built and tested in this repository but is
-not yet published to PyPI. See the [runtime design](docs/design/runtime-library.md)
+where its native wheel was built. It exposes the four Tokenless lifecycle
+methods and bundles the matching RTK executable; TOON is linked into the native
+runtime. It does not require the Tokenless CLI or system helper binaries. The
+package is built and tested in this repository but is not yet published to
+PyPI. See the [runtime design](docs/design/runtime-library.md)
 and the [user manual](../../docs/user-guide/en/token-saving/tokenless/user-manual.md#build-the-python-runtime-from-source).
 
 ## CLI Usage
@@ -581,9 +582,9 @@ The public entry point and configuration are the same across both major
 versions. AgentScope 1.x and 2.x expose different lifecycle hooks, so only the
 final attachment step differs.
 
-AgentScope 1.x must install the integration after the Agent and all of its tool
-functions have been created. Installation binds retrieval to that Agent's
-memory so a stash hash cannot be retrieved unless its marker is visible there.
+AgentScope 1.x uses a Tokenless Toolkit so tools registered before or after
+Agent construction, including MCP tools, receive the same lifecycle handling.
+Installation requires an explicit session identifier.
 
 ```python
 from agentscope.agent import ReActAgent
@@ -595,8 +596,10 @@ integration = TokenlessAgentScope(
         data_dir="/absolute/path/to/tenant-tokenless-data",
     ),
 )
+toolkit = integration.create_toolkit()
+toolkit.register_tool_function(application_tool)
 agent = ReActAgent(..., toolkit=toolkit)
-integration.install(agent)
+integration.install(agent, session_id="conversation-id")
 ```
 
 AgentScope 2.x receives the retrieval Tool and middleware during construction;
@@ -650,16 +653,20 @@ Toolkit mutation or automatic Tool collection.
 | `balanced` | Skip Read/Glob/Grep; use 65,536 / 128 / depth 8 for Shell and conservative limits elsewhere |
 | `aggressive` | Skip Read/Glob/Grep; use CLI defaults of 4,096 / 32 / depth 8 elsewhere |
 
-`balanced` is the default. The read-only retrieval Tool is auto-allowed only
-for a 24-character hash whose marker is present in AgentScope 1.x memory or the
-AgentScope 2.x context/summary. In 1.x, call `install()` only after registering
-the tools that should be compressed; tools registered later are not wrapped.
-Pass a different absolute `data_dir` to each user or tenant for direct Agents;
+`balanced` is the default. The read-only retrieval Tool is published to the
+model only when a marker is visible and accepts only a hash from the exact
+marker set retained for that model call. Pass a different absolute `data_dir`
+to each user or tenant for direct Agents;
 `TOKENLESS_DATA_DIR` is only a process-wide fallback when `data_dir` is omitted.
 Retain the default one-hour stash TTL unless the application has a deliberate
-lifecycle policy, and do not expect retrieval across nodes. This integration
-does not enable Shell, MCP, TOON, RTK, or schema compression. Its source lives
-under `python/agentscope/` for independent wheel distribution.
+lifecycle policy, and do not expect retrieval across nodes.
+
+Both AgentScope adapters enable schema compression, RTK command rewriting,
+response compression, TOON, retrieval, environment-error guidance, and
+per-call attribution. The native wheel contains RTK and links TOON directly;
+it does not search for system executables. Host objects and streaming chunks
+remain unchanged; only copied call arguments and final model-visible text are
+transformed. Tool Ready remains hard-disabled.
 
 
 ## Build

package/adapters/tokenless/claude-code/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "description": "Token-Less context compression for Claude Code — RTK command rewriting, response/TOON compression, and Tool Ready environment pre-check",
   "author": { "name": "ANOLISA" },
   "license": "Apache-2.0",

package/adapters/tokenless/codex/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "description": "Intelligent tool response compression, environment error detection, and token optimization. Strips noise (debug fields, nulls, empty values), truncates long strings/arrays, applies TOON encoding for JSON responses, and classifies environment errors with actionable fix hints.",
   "keywords": [
     "compression",

package/adapters/tokenless/codex/scripts/compress-response

@@ -166,6 +166,10 @@ else:
 AGENT_ID = os.environ.get("TOKENLESS_AGENT_ID", "codex")
 MIN_RESPONSE_CHARS: int = 500
 LARGE_RESPONSE_CHARS: int = 4000
+# TOON on small JSON saves only a few characters (observed ~0.3% below
+# ~500 chars) while the per-event encode cost stays the same, so payloads
+# under this threshold keep the compressed form and skip the TOON pass.
+MIN_TOON_CHARS: int = 500
 
 # 3-layer compression strategy:
 #   Layer 1: Content retrieval + task management → skip all compression
@@ -372,7 +376,7 @@ def main() -> None:
         toon_input = compression_input
 
     toon_parsed = _try_parse_json(toon_input)
-    if toon_parsed is not None:
+    if toon_parsed is not None and len(toon_input) >= MIN_TOON_CHARS:
         toon_text = _run_tokenless(
             tokenless_bin, "compress-toon",
             toon_input, session_id, tool_use_id, timeout=10,

package/adapters/tokenless/common/cosh-extension.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.0",
+  "version": "0.7.9",
   "contextFileName": "COPILOT.md",
   "hooks": {
     "PreToolUse": [
@@ -20,7 +20,7 @@
         ]
       },
       {
-        "matcher": "^(Bash|run_shell_command|terminal|Shell|exec|process)$",
+        "matcher": "^(Bash|run_shell_command|terminal|Shell|shell|exec|process)$",
         "hooks": [
           {
             "type": "command",

package/adapters/tokenless/common/hooks/compress_response_hook.py

@@ -81,6 +81,12 @@ from hook_utils import (
 
 _MIN_RESPONSE_CHARS = 200
 
+# Minimum payload size for the TOON encoding step. TOON on small JSON
+# saves only a few characters (observed ~0.3% below ~500 chars) while the
+# per-event encode cost stays the same, so payloads under this threshold
+# keep the response-compressed form and skip the TOON pass entirely.
+_MIN_TOON_CHARS = 500
+
 # Claude Code added hookSpecificOutput.updatedToolOutput (normal-path tool
 # output replacement for all tools) in v2.1.121. Older versions only support
 # the additive additionalContext, which would duplicate the payload.
@@ -231,7 +237,11 @@ def _build_replacement_output(
         return False, None
 
     # Restoring empty schema fields can cancel out a marginal win.
-    serialized = json.dumps(updated_output, separators=(",", ":"))
+    # ensure_ascii=False keeps the size comparison in Unicode characters,
+    # consistent with the non-escaped normalization below.
+    serialized = json.dumps(
+        updated_output, separators=(",", ":"), ensure_ascii=False
+    )
     if len(serialized) >= len(tool_response):
         return False, None
     return True, updated_output
@@ -313,7 +323,13 @@ def main() -> None:
             skip()  # Plain text, not JSON
         tool_response = unwrapped
     elif isinstance(model_visible_before, (dict, list)):
-        tool_response = json.dumps(model_visible_before, separators=(",", ":"))
+        # ensure_ascii=False: size gates below must count Unicode
+        # characters (code points), not \uXXXX escape sequences, so
+        # structured payloads are measured the same way as JSON string
+        # inputs and the OpenClaw adapter.
+        tool_response = json.dumps(
+            model_visible_before, separators=(",", ":"), ensure_ascii=False
+        )
     else:
         skip()
 
@@ -380,10 +396,12 @@ def main() -> None:
         except Exception as e:
             warn(f"Response compression error: {e}")
 
-    # 16. Step 2: TOON encoding
+    # 16. Step 2: TOON encoding — only for payloads at or above the
+    # minimum threshold; small JSON gains near-zero chars from TOON but
+    # would still pay the full encode cost on every PostToolUse event.
     toon_output = ""
 
-    if tokenless_bin:
+    if tokenless_bin and len(compressed) >= _MIN_TOON_CHARS:
         toon_parsed = try_parse_json(compressed)
         if toon_parsed is not None:
             toon_cmd = [tokenless_bin, "compress-toon", "--agent-id", agent_id]

package/adapters/tokenless/common/hooks/compress_toon_hook.py

@@ -39,7 +39,11 @@ from hook_utils import (
 # -- constants ---------------------------------------------------------------
 
 _AGENT_ID = resolve_agent_id()
-_MIN_RESPONSE_CHARS = 200
+
+# Minimum payload size for TOON encoding. TOON on small JSON saves only a
+# few characters (observed ~0.3% below ~500 chars) while the per-event
+# encode cost stays the same, so smaller responses pass through untouched.
+_MIN_TOON_CHARS = 500
 
 
 # -- main --------------------------------------------------------------------
@@ -84,15 +88,22 @@ def main() -> None:
         if tool_response is None:
             skip()  # Plain text, not JSON
     elif isinstance(tool_response_raw, (dict, list)):
-        tool_response = json.dumps(tool_response_raw, separators=(",", ":"))
+        # ensure_ascii=False: the threshold below counts Unicode
+        # characters (code points), not \uXXXX escape sequences, so
+        # structured payloads are measured the same way as JSON string
+        # inputs and the OpenClaw adapter.
+        tool_response = json.dumps(
+            tool_response_raw, separators=(",", ":"), ensure_ascii=False
+        )
     else:
         skip()
 
     if not tool_response:
         skip()
 
-    # 7. Skip small responses (character count, not byte length)
-    if len(tool_response) < _MIN_RESPONSE_CHARS:
+    # 7. Skip payloads below the TOON minimum threshold (character count,
+    # not byte length): TOON savings on small JSON are near-zero
+    if len(tool_response) < _MIN_TOON_CHARS:
         skip()
 
     # 8. Validate it's JSON

package/adapters/tokenless/common/hooks/hook_utils.py

@@ -408,7 +408,13 @@ def unwrap_string_json(raw: str) -> str | None:
     if isinstance(inner, str):
         inner_obj = try_parse_json(inner)
         if inner_obj is not None and isinstance(inner_obj, (dict, list)):
-            return json.dumps(inner_obj, separators=(",", ":"))
+            # ensure_ascii=False: downstream size gates count Unicode
+            # characters (code points), not \uXXXX escape sequences, so
+            # string-wrapped payloads are measured the same way as the
+            # dict/list branch and the OpenClaw adapter.
+            return json.dumps(
+                inner_obj, separators=(",", ":"), ensure_ascii=False
+            )
         return None
     return raw
 

package/adapters/tokenless/common/hooks/rewrite_hook.py

@@ -5,7 +5,10 @@ Reads a PreToolUse JSON from stdin, extracts the shell command,
 invokes ``rtk rewrite`` via subprocess, and writes a HookOutput
 JSON to stdout.
 
-Hook point: **PreToolUse** — matcher: ``Shell``
+Hook point: **PreToolUse** — matcher: shell-family tool names
+(``Bash``, ``run_shell_command``, ``terminal``, ``Shell``, ``shell``,
+``exec``, ``process``).  The lowercase ``shell`` alternative covers
+cosh-ng, whose built-in shell tool is named ``shell`` on the wire.
 
 The agent ID is read from the TOKENLESS_AGENT_ID environment variable
 (set by the install action script).  Fallback paths follow the ANOLISA

package/adapters/tokenless/dsh/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@anolisa/dsh-tokenless",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "description": "Native DeepSeek Harness response compression for Tokenless",
   "type": "module",
   "main": "./dist/index.js",

package/adapters/tokenless/hermes/__init__.py

@@ -180,6 +180,12 @@ logger = logging.getLogger(__name__)
 AGENT_ID = "hermes-agent"
 _MIN_RESPONSE_LEN = 200
 
+# Minimum payload size for the TOON encoding step. TOON on small JSON
+# saves only a few characters (observed ~0.3% below ~500 chars) while
+# the per-event encode cost stays the same, so payloads under this
+# threshold keep the response-compressed form and skip TOON entirely.
+_MIN_TOON_CHARS = 500
+
 _SKIP_TOOLS: set[str] = _SKIP_TOOLS_SHARED | {
     "session_search", "list_sessions",
 }
@@ -513,8 +519,12 @@ def on_transform_tool_result(
                                      str(session_id), str(tool_call_id))
     current = compressed if compressed else result
 
-    # Step 2: TOON encoding
-    toon_result = _encode_toon(current, str(session_id), str(tool_call_id))
+    # Step 2: TOON encoding — only for payloads at or above the minimum
+    # threshold; small JSON gains near-zero chars from TOON but would still
+    # pay the full encode cost on every tool result.
+    toon_result = None
+    if len(current) >= _MIN_TOON_CHARS:
+        toon_result = _encode_toon(current, str(session_id), str(tool_call_id))
     used_compression = compressed is not None
     used_toon = toon_result is not None
 

package/adapters/tokenless/hermes/plugin.yaml

@@ -1,5 +1,5 @@
 name: tokenless
-version: "0.7.7"
+version: "0.7.9"
 description: "Token-Less context compression for Hermes Agent — response compression, TOON encoding, command rewriting, and registered but hard-disabled Tool Ready"
 author: ANOLISA
 requires_env: []

package/adapters/tokenless/manifest.json

@@ -1,6 +1,6 @@
 {
   "component": "tokenless",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "targets": {
     "cosh": {
       "compatibleVersions": "*",

package/adapters/tokenless/openclaw/dist/index.js

@@ -55,6 +55,25 @@ function mergeExecContextEnv(params, context) {
 }
 // ---- Binary availability cache (with TTL for negative results) -----------------
 const CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes — retry after auto-fix installs
+// Minimum payload size for the TOON encoding step. TOON on small JSON saves
+// only a few characters (observed ~0.3% below ~500 chars) while the
+// per-event encode cost stays the same, so payloads under this threshold
+// keep the response-compressed form and skip TOON entirely.
+const MIN_TOON_CHARS = 500;
+// True when `text` contains at least `threshold` Unicode code points.
+// Iterating a string counts code points (a surrogate pair counts once), so
+// the threshold uses the same unit as the Python adapters' len(). The loop
+// returns as soon as the threshold is reached, so large payloads only pay
+// for scanning the first `threshold` characters.
+function hasAtLeastChars(text, threshold) {
+    let count = 0;
+    for (const _ch of text) {
+        count += 1;
+        if (count >= threshold)
+            return true;
+    }
+    return false;
+}
 let rtkAvailable = null;
 let rtkCheckedAt = null;
 let tokenlessAvailable = null;
@@ -274,6 +293,13 @@ function tryCompressResponse(response, sessionId, toolCallId, thresholds) {
 function tryCompressToon(response, sessionId, toolCallId) {
     try {
         const input = JSON.stringify(response);
+        // Skip payloads below the minimum threshold: TOON savings on small
+        // JSON are near-zero but the encode cost is paid on every tool result.
+        // Count Unicode code points, not UTF-16 code units (String.length), so
+        // non-BMP text (e.g. emoji) is measured the same way as the Python
+        // adapters' character counts.
+        if (!hasAtLeastChars(input, MIN_TOON_CHARS))
+            return null;
         const beforeChars = input.length;
         const args = ["compress-toon", "--agent-id", "openclaw"];
         if (sessionId)

package/adapters/tokenless/openclaw/index.ts

@@ -88,6 +88,26 @@ function mergeExecContextEnv(
 
 const CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes — retry after auto-fix installs
 
+// Minimum payload size for the TOON encoding step. TOON on small JSON saves
+// only a few characters (observed ~0.3% below ~500 chars) while the
+// per-event encode cost stays the same, so payloads under this threshold
+// keep the response-compressed form and skip TOON entirely.
+const MIN_TOON_CHARS = 500;
+
+// True when `text` contains at least `threshold` Unicode code points.
+// Iterating a string counts code points (a surrogate pair counts once), so
+// the threshold uses the same unit as the Python adapters' len(). The loop
+// returns as soon as the threshold is reached, so large payloads only pay
+// for scanning the first `threshold` characters.
+function hasAtLeastChars(text: string, threshold: number): boolean {
+  let count = 0;
+  for (const _ch of text) {
+    count += 1;
+    if (count >= threshold) return true;
+  }
+  return false;
+}
+
 let rtkAvailable: boolean | null = null;
 let rtkCheckedAt: number | null = null;
 let tokenlessAvailable: boolean | null = null;
@@ -326,6 +346,12 @@ function tryCompressResponse(response: any, sessionId?: string, toolCallId?: str
 function tryCompressToon(response: any, sessionId?: string, toolCallId?: string): { toonText: string; savingsPct: number } | null {
   try {
     const input = JSON.stringify(response);
+    // Skip payloads below the minimum threshold: TOON savings on small
+    // JSON are near-zero but the encode cost is paid on every tool result.
+    // Count Unicode code points, not UTF-16 code units (String.length), so
+    // non-BMP text (e.g. emoji) is measured the same way as the Python
+    // adapters' character counts.
+    if (!hasAtLeastChars(input, MIN_TOON_CHARS)) return null;
     const beforeChars = input.length;
     const args = ["compress-toon", "--agent-id", "openclaw"];
     if (sessionId) args.push("--session-id", sessionId);

package/adapters/tokenless/openclaw/openclaw.plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "tokenless",
   "name": "Tokenless",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "description": "Unified RTK command rewriting + response/TOON compression + registered but hard-disabled Tool Ready. Wraps tokenless and rtk system binaries via child_process — this is expected and not malicious.",
   "activation": {
     "onCapabilities": ["hook"]

package/adapters/tokenless/openclaw/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tokenless/openclaw-plugin",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "description": "Unified OpenClaw plugin — RTK command rewriting + tokenless schema/response compression for 60-90% LLM token savings",
   "type": "module",
   "main": "dist/index.js",

package/adapters/tokenless/qoder/.qoder-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "description": "Token-Less LLM Token Optimization Toolkit for Qoder CLI",
   "author": {
     "name": "Shile Zhang",

package/adapters/tokenless/qwencode/qwen-extension.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "description": "Token-Less context compression for Qwen Code — RTK command rewriting, response/TOON/schema compression, and registered but hard-disabled Tool Ready",
   "author": { "name": "ANOLISA" },
   "license": "Apache-2.0",

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "anolisa-tokenless",
   "type": "module",
-  "version": "0.7.7",
+  "version": "0.7.9",
   "description": "Token-Less — LLM token optimization toolkit (schema/response compression, command rewriting, tool readiness)",
   "license": "Apache-2.0",
   "repository": {
@@ -45,10 +45,10 @@
     "arm64"
   ],
   "optionalDependencies": {
-    "@anolisa/tokenless-linux-x64": "0.7.7",
-    "@anolisa/tokenless-linux-arm64": "0.7.7",
-    "@anolisa/tokenless-darwin-x64": "0.7.7",
-    "@anolisa/tokenless-darwin-arm64": "0.7.7"
+    "@anolisa/tokenless-linux-x64": "0.7.9",
+    "@anolisa/tokenless-linux-arm64": "0.7.9",
+    "@anolisa/tokenless-darwin-x64": "0.7.9",
+    "@anolisa/tokenless-darwin-arm64": "0.7.9"
   },
   "publishConfig": {
     "registry": "https://registry.npmjs.org/",