anolisa-tokenless 0.7.11 → 0.7.12

package/README.md

@@ -7,7 +7,7 @@
 Token-Less combines complementary strategies to minimize LLM token consumption:
 
 - **Schema & Response Compression** — Compresses OpenAI Function Calling tool definitions and API responses via the `tokenless-schema` library, cutting structural overhead before tokens ever reach the context window.
-- **TOON Context Compression** — Encodes JSON responses to TOON (Token-Oriented Object Notation) format via the `toon-format` library linked into `tokenless`, reducing token usage by 15-40% for structured data.
+- **TOON Context Compression** — Encodes JSON responses to TOON (Token-Oriented Object Notation) format via the `toon-format` library linked into `tokenless`, reducing syntax overhead for suitable structured data.
 - **Command Rewriting** — Integrates [RTK](https://github.com/rtk-ai/rtk) to filter and rewrite CLI command output, eliminating noise that would otherwise waste 60–90% of tokens.
 - **Tool Ready (legacy, hard-disabled)** — Its pre-call dependency checks are retained in source but unconditionally bypassed while the readiness model is redesigned.
 
@@ -27,12 +27,12 @@ cover schema compression, RTK rewriting, response compression, TOON, retrieval,
 
 ## Features
 
-| Capability | Token Savings | Details |
+| Capability | Savings indicator | Details |
 |---|---|---|
-| Schema compression | ~57% | Compresses OpenAI Function Calling tool schemas |
-| Response compression | ~26–78% | Compresses API / tool responses (varies by content type) |
+| Schema compression | 47.3% on reference fixture | Compresses OpenAI Function Calling tool schemas |
+| Response compression | 65.8% on reference fixture | Compresses API / tool responses |
 | Reversible compression (stash) | — | Dropped array items are stashed and retrievable via `<<tokenless:KEY>>` markers |
-| TOON context compression | 15–40% | Encodes JSON to TOON format for LLMs |
+| TOON context compression | 17.0% on reference response | Encodes JSON to TOON format for LLMs |
 | Command rewriting | 60–90% | Filters CLI output via RTK (70+ commands supported) |
 | Tool Ready | reduces retry waste | Legacy pre-call check, auto-fix, and blocking; hard-disabled |
 | OpenClaw plugin | — | Command rewriting ✅, Response compression ✅, Schema compression ✅ |
@@ -46,17 +46,28 @@ cover schema compression, RTK rewriting, response compression, TOON, retrieval,
 | AgentScope framework integration | — | Schema ✅, RTK ✅, Response ✅, TOON ✅, Retrieval ✅ |
 | Zero runtime deps | — | Pure Rust, single static binary |
 
+The schema, response, and TOON figures above are isolated Tokenless 0.7.11
+results on the repository's committed reference fixtures; they are neither a
+production range nor additive. Compression depends on payload size and shape,
+removable fields, configured thresholds, and the share of tool data in the
+session. Short or already compact payloads may save only a few percent or pass
+through unchanged. See [Measuring Tokenless Savings](../../docs/user-guide/en/token-saving/tokenless/measuring-savings.md#run-the-repository-reference-workload)
+for the exact inputs, command, full output, and limitations.
+
 ## Applicable Scenarios & Expected Effects
 
-tokenless only removes redundancy from **tool call responses** before they enter the LLM context; it does not touch model reasoning or conversation history. The payoff depends heavily on the share and shape of tool responses in the session.
+tokenless optimizes the tool-related content it handles—tool schemas, tool/API
+responses, and supported shell output—before it enters the LLM context. It does
+not touch model reasoning or conversation history. The payoff depends heavily
+on the share and shape of that content in the session.
 
 ### Where it pays off
 
 | Workload | Primary strategy | Why |
 |----------|-----------------|-----|
 | Shell-heavy (build/test/triage) | Command rewriting (RTK) | `cargo`/`npm`/`go`/`pytest` output carries lots of progress/warning noise; RTK cuts 60–90% |
-| API/fetch-heavy (REST, web_fetch) | Response compression + TOON | JSON carries debug/null/empty and syntax overhead; 26–78% compression, TOON adds 15–40% |
-| Agents with many tools | Schema compression | Many Function Calling definitions carry verbose descriptions; ~57% |
+| API/fetch-heavy (REST, web_fetch) | Response compression + TOON | JSON may carry removable debug/null/empty fields; sufficiently large, regular structures also have reducible syntax overhead |
+| Agents with many tools | Schema compression | Many Function Calling definitions carry verbose descriptions and removable metadata |
 | Long responses that must stay faithful | Reversible compression (Stash) | Truncated content is `retrieve`-able end-to-end lossless; thresholds can be tightened safely |
 
 ### Where it pays little or doesn't apply
@@ -264,6 +275,15 @@ Compress a batch of tools (JSON array):
 tokenless compress-schema -f tools.json --batch
 ```
 
+A top-level request object with a `tools` array is also accepted without
+`--batch`; OpenAI wrappers, Gemini `functionDeclarations` tool objects, and
+bare Function Calling declarations are compressed while non-function tools and
+fields outside `tools` are preserved:
+
+```bash
+tokenless compress-schema -f request.json
+```
+
 ### compress-response
 
 Compress an API response:
@@ -276,8 +296,12 @@ tokenless compress-response -f response.json
 curl -s https://api.example.com/data | tokenless compress-response
 ```
 
-By default `compress-response` stashes dropped array items so they can be
-retrieved later (see [Reversible compression](docs/stash-reversible-compression.md)).
+Long arrays are truncated to a head+tail window: the first
+`--truncate-arrays-at` items (default 32) plus the last
+`--array-tail-preserve` items (default 8), with a truncation marker in
+between; pass `--array-tail-preserve 0` for head-only truncation. By default
+`compress-response` stashes the dropped middle segment so it can be retrieved
+later (see [Reversible compression](docs/stash-reversible-compression.md)).
 Pass `--no-stash` for lossy truncation, or `--stash-db <path>` to override the
 stash database (default `~/.tokenless/stash.db`).
 
@@ -292,7 +316,7 @@ tokenless retrieve c30ccf5ed1125e0ed871ba8e
 
 # Or paste the whole truncation line — the hash is extracted automatically.
 # (Use the FULL 24-hex hash from your output; the value below is shorthand.)
-tokenless retrieve "<... 195 items truncated, retrieve with <<tokenless:c30ccf5ed1125e0ed871ba8e>>"
+tokenless retrieve "<... 160 items truncated, retrieve with <<tokenless:c30ccf5ed1125e0ed871ba8e>>"
 ```
 
 ### compress-toon / decompress-toon
@@ -312,10 +336,13 @@ echo 'name: Alice\nage: 30' | tokenless decompress-toon
 
 ### Inspect token savings
 
-Use `show` to print the complete stored before/after payload, or `diff` to
-explain the estimated token saving and highlight only changed lines:
+Use `stats summary` for totals, `show` for the stored before/after payload, or
+`diff` to explain the estimated token saving and highlight only changed lines:
 
 ```bash
+tokenless stats summary
+tokenless stats summary --limit 1000
+tokenless stats summary --compare <baseline-session> <active-session>
 tokenless stats show 42
 tokenless stats diff 42
 tokenless stats diff --session <session-id>
@@ -323,10 +350,13 @@ tokenless stats diff --session <session-id> --tool-use-id <tool-use-id>
 tokenless stats diff 42 --json
 ```
 
-Session overviews contain metrics only. Record and tool-use reports include a
-unified content diff; consecutive active stages are linked only when their
-stored output/input content matches exactly, avoiding duplicate intermediate
-token counts. See [Measuring Tokenless Savings](../../docs/user-guide/en/token-saving/tokenless/measuring-savings.md)
+`stats summary --limit` must be a positive integer; `--limit 0` is rejected at
+parse time. `--compare` fails if either session has no records instead of
+reporting 0% savings. Session overviews contain metrics only. Record and
+tool-use reports include a unified content diff; consecutive active stages are
+linked only when their stored output/input content matches exactly, avoiding
+duplicate intermediate token counts. See
+[Measuring Tokenless Savings](../../docs/user-guide/en/token-saving/tokenless/measuring-savings.md)
 for options and measurement limits.
 
 ### Database location

package/adapters/tokenless/claude-code/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "description": "Token-Less context compression for Claude Code — RTK command rewriting, response/TOON compression, and Tool Ready environment pre-check",
   "author": { "name": "ANOLISA" },
   "license": "Apache-2.0",

package/adapters/tokenless/codex/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "description": "Intelligent tool response compression, environment error detection, and token optimization. Strips noise (debug fields, nulls, empty values), truncates long strings/arrays, applies TOON encoding for JSON responses, and classifies environment errors with actionable fix hints.",
   "keywords": [
     "compression",

package/adapters/tokenless/common/cosh-extension.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "contextFileName": "COPILOT.md",
   "hooks": {
     "PreToolUse": [

package/adapters/tokenless/common/hooks/compress_schema_hook.py

@@ -11,11 +11,19 @@ The agent ID is read from the TOKENLESS_AGENT_ID environment variable
 (set by the install action script).
 """
 
+from __future__ import annotations
+
+import contextlib
 import json
 import os
 import subprocess
 import sys
 
+try:  # POSIX hosts (cosh / Cosh-NG) — the platforms these hooks target.
+    import fcntl
+except ImportError:  # pragma: no cover - non-POSIX fallback keeps best-effort
+    fcntl = None
+
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 
 from hook_utils import (
@@ -25,6 +33,7 @@ from hook_utils import (
     resolve_agent_id,
     resolve_binary,
     resolve_tool_call_id,
+    secure_write_text,
     skip,
     warn,
 )
@@ -33,6 +42,63 @@ from hook_utils import (
 
 _AGENT_ID = resolve_agent_id()
 
+# One marker file holds the session keys that already emitted the "no tool
+# declarations" warning — one key per line, most recent last — so the warning
+# repeats at most once per session even though BeforeModel fires on every
+# model turn. A single-value marker would not survive concurrent sessions
+# under one HOME: session B's first warning would overwrite session A's key
+# and make A warn again on its next turn. The list is bounded (oldest entries
+# trimmed) so it cannot grow without limit; losing an evicted key only
+# re-arms the warning for a long-idle session. All reads and writes of the
+# marker run under the exclusive sidecar lock below: an unlocked
+# read-modify-write lets two hook processes read the same old key set and
+# write back over each other, silently losing one session's key.
+_NO_TOOLS_WARN_MARKER = os.path.join(
+    os.path.expanduser("~"), ".tokenless", ".schema-hook-nowarn-session"
+)
+
+# Best-effort dedup bound: far above any plausible number of concurrently
+# active sessions sharing one HOME.
+_MAX_WARNED_SESSIONS = 64
+
+
+@contextlib.contextmanager
+def _marker_lock():
+    """Exclusive advisory lock guarding the marker read-modify-write.
+
+    Two hook processes warning for different sessions at the same time must
+    not read the same old key set and write back over each other. The lock
+    lives on a dedicated sidecar file that the marker write never replaces,
+    so the locked inode cannot be swapped out mid-section, and the kernel
+    drops an ``flock`` when its holder exits or crashes, so a dead hook
+    process cannot wedge later sessions. On platforms without ``fcntl``, or
+    when the lock file cannot be created, this degrades to the historical
+    unlocked best-effort behaviour instead of failing the warning.
+    """
+    if fcntl is None:
+        yield
+        return
+    lock_path = _NO_TOOLS_WARN_MARKER + ".lock"
+    fd = -1
+    try:
+        os.makedirs(os.path.dirname(lock_path), mode=0o700, exist_ok=True)
+        if os.path.islink(lock_path):
+            os.unlink(lock_path)
+        flags = os.O_RDWR | os.O_CREAT
+        if hasattr(os, "O_NOFOLLOW"):
+            flags |= os.O_NOFOLLOW
+        fd = os.open(lock_path, flags, 0o600)
+        fcntl.flock(fd, fcntl.LOCK_EX)
+    except OSError:
+        if fd >= 0:
+            os.close(fd)
+        yield  # lock unavailable — keep the unlocked best-effort behaviour
+        return
+    try:
+        yield
+    finally:
+        os.close(fd)
+
 
 # -- helpers -----------------------------------------------------------------
 
@@ -45,6 +111,100 @@ def _is_json_array(data: str) -> bool:
         return False
 
 
+def _session_warn_key(session_id: str) -> str:
+    """Normalize the session ID into the dedup key for the no-tools warning.
+
+    Hosts that omit a session ID share one key, so the warning still appears
+    at most once for them instead of on every turn.
+    """
+    key = (session_id or "").strip()
+    return key or "<no-session>"
+
+
+def _read_warned_sessions() -> list:
+    """Read the warned session keys (one per line) from the marker file.
+
+    Tolerates a missing or corrupt marker — dedup is best-effort and a miss
+    simply means the session warns once (again).
+    """
+    try:
+        with open(_NO_TOOLS_WARN_MARKER, "r", encoding="utf-8") as handle:
+            return [line.strip() for line in handle if line.strip()]
+    except OSError:
+        return []  # first invocation, or unreadable marker
+    except UnicodeDecodeError:
+        return []  # corrupt marker — rebuild it on the next warning
+
+
+def _should_warn_no_tools(session_id: str) -> bool:
+    """Return True if the no-tools warning has not been emitted for this
+    session yet, and record that it is being emitted now.
+
+    Every warned session key stays in the marker (bounded to the newest
+    ``_MAX_WARNED_SESSIONS`` entries), so concurrent sessions under one HOME
+    do not invalidate each other's dedup state. The check-and-record runs
+    inside ``_marker_lock`` so two first warnings racing for different
+    sessions serialize instead of overwriting each other's keys. Best-effort
+    by design: the marker lives under ``~/.tokenless`` and is read/written
+    with the same hardened helpers as other hook state, but any failure
+    falls back to warning — observability beats silence, and the warning
+    itself never affects the pass-through behaviour.
+    """
+    key = _session_warn_key(session_id)
+    with _marker_lock():
+        warned = _read_warned_sessions()
+        if key in warned:
+            return False
+        warned.append(key)
+        warned = warned[-_MAX_WARNED_SESSIONS:]
+        try:
+            secure_write_text(
+                _NO_TOOLS_WARN_MARKER, "".join(entry + "\n" for entry in warned)
+            )
+        except OSError:
+            pass  # state dir unwritable — still warn this once
+        return True
+
+
+def _skip_with_message(msg: str) -> None:
+    """Pass through unchanged while surfacing ``msg`` to the user.
+
+    ``systemMessage`` is the protocol field both hosts surface for humans:
+    Cosh-NG renders it as a hook notification and copilot-shell records it in
+    its hook log, while neither injects it into the model context. The
+    warning is also printed to stderr so hosts that forward hook stderr get
+    the same text.
+    """
+    warn(msg)
+    print(json.dumps({"systemMessage": msg}))
+    sys.exit(0)
+
+
+def _extract_tools(input_data: dict) -> tuple[object, bool]:
+    """Extract the tool declarations from a BeforeModel payload.
+
+    Returns ``(tools, declared)`` where ``declared`` is True when the host
+    carried a tools field at any known position — even an empty one.
+
+    ``config.tools`` is the canonical position (both copilot-shell's Hook
+    Translator and Cosh-NG put it there); the top-level ``tools`` is the
+    older position, kept for hosts that still emit it. Presence of the
+    canonical key decides, not its truthiness: a host that declares no tools
+    sends an empty canonical array, and falling through to a stale legacy
+    field there would compress declarations this request never carried. This
+    mirrors the host-side precedence.
+    """
+    llm_request = input_data.get("llm_request")
+    if not isinstance(llm_request, dict):
+        return None, False
+    config = llm_request.get("config")
+    if isinstance(config, dict) and "tools" in config:
+        return config["tools"], True
+    if "tools" in llm_request:
+        return llm_request.get("tools"), True
+    return None, False
+
+
 # -- main --------------------------------------------------------------------
 
 
@@ -68,21 +228,42 @@ def main() -> None:
     except (json.JSONDecodeError, EOFError, ValueError):
         warn("failed to read BeforeModel payload. Passing through unchanged.")
         skip()
+    if not isinstance(input_data, dict):
+        # No session_id available outside a dict payload — the shared
+        # "<no-session>" key still bounds this to one warning.
+        if _should_warn_no_tools(""):
+            _skip_with_message(
+                "BeforeModel payload is not a JSON object. Passing through "
+                "unchanged (warned once per session)."
+            )
+        skip()
 
-    # 3. Extract tools array. `config.tools` is the canonical position (both
-    # copilot-shell's Hook Translator and Cosh-NG put it there); the top-level
-    # `tools` is the older position, kept for hosts that still emit it.
-    # Presence of the canonical key decides, not its truthiness: a host that
-    # declares no tools sends an empty canonical array, and falling through to
-    # a stale legacy field there would compress declarations this request never
-    # carried. This mirrors the host-side precedence.
-    llm_request = input_data.get("llm_request", {})
-    config = llm_request.get("config")
-    if isinstance(config, dict) and "tools" in config:
-        tools = config["tools"]
-    else:
-        tools = llm_request.get("tools")
+    # 3. Extract tools array.
+    tools, tools_declared = _extract_tools(input_data)
     if not tools:
+        # A host that declares no tools for this turn (empty canonical array)
+        # skips silently — that is a normal request. But a BeforeModel event
+        # with no tools field at any known position means there is nothing
+        # for schema compression to work on. That skip used to be silent, so
+        # "0 schema records" could not be told apart from "hook never ran";
+        # warn once per session instead.
+        if not tools_declared and _should_warn_no_tools(
+            str(input_data.get("session_id", ""))
+        ):
+            if not isinstance(input_data.get("llm_request"), dict):
+                _skip_with_message(
+                    "BeforeModel payload carries no llm_request object, so schema "
+                    "compression cannot find tool declarations. Passing through "
+                    "unchanged (warned once per session)."
+                )
+            else:
+                _skip_with_message(
+                    "BeforeModel event carries no tool declarations at "
+                    "llm_request.config.tools or llm_request.tools, so schema "
+                    "compression was skipped. If the host declares tools, its "
+                    "event format may not match the hook. Passing through "
+                    "unchanged (warned once per session)."
+                )
         skip()
 
     tools_json = json.dumps(tools, separators=(",", ":"))

package/adapters/tokenless/dsh/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@anolisa/dsh-tokenless",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "description": "Native DeepSeek Harness response compression for Tokenless",
   "type": "module",
   "main": "./dist/index.js",

package/adapters/tokenless/hermes/plugin.yaml

@@ -1,5 +1,5 @@
 name: tokenless
-version: "0.7.11"
+version: "0.7.12"
 description: "Token-Less context compression for Hermes Agent — response compression, TOON encoding, command rewriting, and registered but hard-disabled Tool Ready"
 author: ANOLISA
 requires_env: []

package/adapters/tokenless/manifest.json

@@ -1,6 +1,6 @@
 {
   "component": "tokenless",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "targets": {
     "cosh": {
       "compatibleVersions": "*",

package/adapters/tokenless/openclaw/openclaw.plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "tokenless",
   "name": "Tokenless",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "description": "Unified RTK command rewriting + response/TOON compression + registered but hard-disabled Tool Ready. Wraps tokenless and rtk system binaries via child_process — this is expected and not malicious.",
   "activation": {
     "onCapabilities": ["hook"]

package/adapters/tokenless/openclaw/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tokenless/openclaw-plugin",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "description": "Unified OpenClaw plugin — RTK command rewriting + tokenless schema/response compression for 60-90% LLM token savings",
   "type": "module",
   "main": "dist/index.js",

package/adapters/tokenless/qoder/.qoder-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "description": "Token-Less LLM Token Optimization Toolkit for Qoder CLI",
   "author": {
     "name": "Shile Zhang",

package/adapters/tokenless/qwencode/qwen-extension.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenless",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "description": "Token-Less context compression for Qwen Code — RTK command rewriting, response/TOON/schema compression, and registered but hard-disabled Tool Ready",
   "author": { "name": "ANOLISA" },
   "license": "Apache-2.0",

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "anolisa-tokenless",
   "type": "module",
-  "version": "0.7.11",
+  "version": "0.7.12",
   "description": "Token-Less — LLM token optimization toolkit (schema/response compression, command rewriting, tool readiness)",
   "license": "Apache-2.0",
   "repository": {
@@ -44,10 +44,10 @@
     "arm64"
   ],
   "optionalDependencies": {
-    "@anolisa/tokenless-linux-x64": "0.7.11",
-    "@anolisa/tokenless-linux-arm64": "0.7.11",
-    "@anolisa/tokenless-darwin-x64": "0.7.11",
-    "@anolisa/tokenless-darwin-arm64": "0.7.11"
+    "@anolisa/tokenless-linux-x64": "0.7.12",
+    "@anolisa/tokenless-linux-arm64": "0.7.12",
+    "@anolisa/tokenless-darwin-x64": "0.7.12",
+    "@anolisa/tokenless-darwin-arm64": "0.7.12"
   },
   "publishConfig": {
     "registry": "https://registry.npmjs.org/",