anolisa-tokenless 0.8.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +24 -4
- package/adapters/tokenless/claude-code/.claude-plugin/plugin.json +1 -1
- package/adapters/tokenless/claude-code/scripts/detect.sh +50 -7
- package/adapters/tokenless/codex/.codex-plugin/plugin.json +1 -1
- package/adapters/tokenless/common/cosh-extension.json +1 -1
- package/adapters/tokenless/dsh/package.json +1 -1
- package/adapters/tokenless/hermes/__init__.py +147 -6
- package/adapters/tokenless/hermes/plugin.yaml +1 -1
- package/adapters/tokenless/manifest.json +1 -1
- package/adapters/tokenless/openclaw/openclaw.plugin.json +1 -1
- package/adapters/tokenless/openclaw/package.json +1 -1
- package/adapters/tokenless/openclaw/scripts/install.sh +51 -12
- package/adapters/tokenless/qoder/.qoder-plugin/plugin.json +1 -1
- package/adapters/tokenless/qwencode/qwen-extension.json +1 -1
- package/adapters/tokenless/qwenpaw/plugin.json +1 -1
- package/adapters/tokenless/qwenpaw/requirements.txt +3 -3
- package/package.json +5 -5
package/README.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
Token-Less combines complementary strategies to minimize LLM token consumption:
|
|
8
8
|
|
|
9
|
-
- **Lifecycle-aware Compression** — Protocol v2 owns BeforeModel schema handling, PreTool RTK rewriting, PostTool routing, and authorized Retrieve; the PostTool Pipeline compresses JSON and recognized build/test command logs.
|
|
9
|
+
- **Lifecycle-aware Compression** — Protocol v2 owns BeforeModel schema handling, PreTool RTK rewriting, PostTool routing, and authorized Retrieve; the PostTool Pipeline compresses JSON, CSV/TSV tables, and recognized build/test command logs.
|
|
10
10
|
- **TOON Context Compression** — Encodes JSON responses to TOON (Token-Oriented Object Notation) format via the `toon-format` library linked into `tokenless`, reducing syntax overhead for suitable structured data.
|
|
11
11
|
- **Command Rewriting** — Integrates [RTK](https://github.com/rtk-ai/rtk) to filter and rewrite CLI command output, eliminating noise that would otherwise waste 60–90% of tokens.
|
|
12
12
|
- **Tool Ready (legacy, hard-disabled)** — Its pre-call dependency checks are retained in source but unconditionally bypassed while the readiness model is redesigned.
|
|
@@ -35,6 +35,7 @@ retrieval, and attribution.
|
|
|
35
35
|
| Schema compression | 47.3% on reference fixture | Compresses OpenAI Function Calling tool schemas |
|
|
36
36
|
| Content-aware response compression | 36.3% lossless savings on the JSON reference fixture | Routes successful JSON through `JsonCompressor`; lossless candidates saving at least 15% take priority, while recoverable record arrays can be reduced to a 32-record base budget |
|
|
37
37
|
| Build-log compression | workload-dependent | Cleans terminal control output and reduces repeated routine progress in recognized Cargo, pytest, npm/Jest, Go, Make/C, and generic command logs while preserving diagnostics, summaries, phases, and stack traces |
|
|
38
|
+
| CSV/TSV table compression | workload-dependent | Preserves every cell when compacting quoting and record separators; larger tables can retain selected rows with an explicit incomplete-table notice and byte-exact original retrieval. Requires a text replacement slot; file reads pass through |
|
|
38
39
|
| Reversible compression (stash) | — | Omitted record collections and bounded values are stashed; supported agents run `tokenless retrieve HASH` or call their static Retrieve Tool when full data is needed |
|
|
39
40
|
| TOON context compression | 17.0% on reference response | Encodes JSON to TOON format for LLMs |
|
|
40
41
|
| Command rewriting | 60–90% | Filters CLI output via RTK (70+ commands supported) |
|
|
@@ -114,7 +115,7 @@ Token-Less/
|
|
|
114
115
|
├── crates/tokenless-ccr/ # Reversible compression stash (Compress-Cache-Retrieve)
|
|
115
116
|
├── crates/tokenless-runtime/ # Lifecycle API and Runtime-owned PostTool pipeline
|
|
116
117
|
├── crates/tokenless-protocol/ # Versioned adapter contract and token estimator
|
|
117
|
-
├── crates/tokenless-compressors/ # JSON and build-log
|
|
118
|
+
├── crates/tokenless-compressors/ # JSON, tabular, and build-log compressors
|
|
118
119
|
├── crates/tokenless-cli/ # CLI binary: `tokenless` command (env-check, compress, retrieve, stats)
|
|
119
120
|
├── python/tokenless/ # PyO3 package: `anolisa_tokenless`
|
|
120
121
|
├── python/agentscope/ # Pure-Python AgentScope integration package
|
|
@@ -273,7 +274,7 @@ Agent adapters may apply separate pre-check thresholds; see the
|
|
|
273
274
|
### compress
|
|
274
275
|
|
|
275
276
|
Shared Agent hooks send lifecycle requests to `tokenless compress`; only
|
|
276
|
-
successful, non-bypassed PostTool JSON and eligible command-output build logs
|
|
277
|
+
successful, non-bypassed PostTool JSON, CSV/TSV tables, and eligible command-output build logs
|
|
277
278
|
enter the Runtime-owned Pipeline. Tool errors bypass compression and keep their
|
|
278
279
|
original output while Core attaches environment-diagnostic context.
|
|
279
280
|
|
|
@@ -397,6 +398,25 @@ duplicate intermediate token counts. See
|
|
|
397
398
|
[Measuring Tokenless Savings](../../docs/user-guide/en/token-saving/tokenless/measuring-savings.md)
|
|
398
399
|
for options and measurement limits.
|
|
399
400
|
|
|
401
|
+
### Trace correlation
|
|
402
|
+
|
|
403
|
+
Exported SLS records carry the trace identity of the host span they were
|
|
404
|
+
produced under, so an observability backend such as AgentLoop can attribute
|
|
405
|
+
token savings to a trace. Two optional environment variables supply it:
|
|
406
|
+
|
|
407
|
+
- `TOKENLESS_TRACEPARENT` — adapter-facing override, read first.
|
|
408
|
+
- `TRACEPARENT` — standard W3C variable, used when the override is absent,
|
|
409
|
+
empty, or unparsable.
|
|
410
|
+
|
|
411
|
+
Injecting one is the launcher's responsibility: OpenTelemetry keeps the active
|
|
412
|
+
span in an in-process carrier and does not export it to child processes, so a
|
|
413
|
+
host or adapter that wants correlation must set one of them before spawning
|
|
414
|
+
Tokenless. Without a usable context the record shape is unchanged, and the
|
|
415
|
+
identity is written only to the SLS JSONL — never to the local `stats.db`. See
|
|
416
|
+
[Measuring Tokenless Savings](../../docs/user-guide/en/token-saving/tokenless/measuring-savings.md)
|
|
417
|
+
and
|
|
418
|
+
[Configuration and Data Privacy](../../docs/user-guide/en/token-saving/tokenless/configuration-and-privacy.md).
|
|
419
|
+
|
|
400
420
|
### Database location
|
|
401
421
|
|
|
402
422
|
Tokenless stores statistics and reversible-compression data in
|
|
@@ -944,7 +964,7 @@ layout and single-target interface.
|
|
|
944
964
|
|---|---|
|
|
945
965
|
| `crates/tokenless-cli/` | CLI binary — `tokenless` command (compress, stats, env-check) |
|
|
946
966
|
| `crates/tokenless-schema/` | BeforeModel tool-schema compression — `SchemaCompressor` |
|
|
947
|
-
| `crates/tokenless-compressors/` | Content-domain engines — `JsonCompressor` and `BuildLogCompressor` are connected to PostTool |
|
|
967
|
+
| `crates/tokenless-compressors/` | Content-domain engines — `JsonCompressor`, `TabularCompressor` and `BuildLogCompressor` are connected to PostTool |
|
|
948
968
|
| `crates/tokenless-runtime/` | Lifecycle API and Runtime-owned `PostToolPipeline`, shared by CLI and language bindings |
|
|
949
969
|
| `crates/tokenless-protocol/` | Versioned adapter contract and shared `heuristic-v1` token estimator |
|
|
950
970
|
| `python/tokenless/` | PyO3 package exposing `anolisa_tokenless` for CPython 3.11+ |
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tokenless",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"description": "Token-Less context compression for Claude Code — RTK command rewriting, response/TOON compression, and Tool Ready environment pre-check",
|
|
5
5
|
"author": { "name": "ANOLISA" },
|
|
6
6
|
"license": "Apache-2.0",
|
|
@@ -24,6 +24,14 @@ export PATH="$HOME/.local/bin:/usr/local/bin:$PATH"
|
|
|
24
24
|
# fast.
|
|
25
25
|
DETECT_RETRIES="${TOKENLESS_DETECT_RETRIES:-3}"
|
|
26
26
|
DETECT_RETRY_DELAY="${TOKENLESS_DETECT_RETRY_DELAY:-1}"
|
|
27
|
+
# Separate, smaller budget for one result that only looks definitive: a
|
|
28
|
+
# `plugin list` that succeeds but omits the plugin. `claude plugin install`
|
|
29
|
+
# writes marketplace.json/plugin.json, yet the CLI's plugin registry index
|
|
30
|
+
# only picks them up on a later scan, so the very first list right after
|
|
31
|
+
# provisioning can succeed and still omit the just-installed plugin (GH
|
|
32
|
+
# #3082). Kept apart from DETECT_RETRIES so a genuinely absent plugin pays a
|
|
33
|
+
# couple of cheap re-lists rather than the whole settling budget.
|
|
34
|
+
DETECT_PLUGIN_RELISTS="${TOKENLESS_DETECT_PLUGIN_RELISTS:-2}"
|
|
27
35
|
|
|
28
36
|
# settle <cmd...> — run cmd once; if it reports a retryable failure (exit
|
|
29
37
|
# status 1), sleep DETECT_RETRY_DELAY and retry, up to DETECT_RETRIES retries
|
|
@@ -97,13 +105,21 @@ else
|
|
|
97
105
|
field "plugin.json" "missing (run: make stamp-adapter-templates)"
|
|
98
106
|
fi
|
|
99
107
|
|
|
108
|
+
# plugin_manifests_staged — true when the local manifests that
|
|
109
|
+
# `claude plugin install` consumes (the single-plugin marketplace plus the
|
|
110
|
+
# stamped plugin manifest) are both on disk. Their presence is what makes an
|
|
111
|
+
# omitted plugin ambiguous: the installer had something to register, so the
|
|
112
|
+
# CLI's registry index may simply not have caught up with it yet.
|
|
113
|
+
plugin_manifests_staged() {
|
|
114
|
+
[ -f "$PLUGIN_SRC/.claude-plugin/marketplace.json" ] \
|
|
115
|
+
&& [ -f "$PLUGIN_SRC/.claude-plugin/plugin.json" ]
|
|
116
|
+
}
|
|
117
|
+
|
|
100
118
|
# claude_plugin_listed — probe the plugin registry, using the settle()
|
|
101
119
|
# exit-status contract: 0 = plugin listed; 1 = `claude plugin list` itself
|
|
102
120
|
# failed (the CLI may still be initializing ~/.claude on first run, so a
|
|
103
121
|
# retry may still succeed); 2 = `plugin list` ran successfully but did not
|
|
104
|
-
# list the plugin.
|
|
105
|
-
# it — so settle() must return immediately instead of sleeping out the
|
|
106
|
-
# retry budget and invoking the CLI DETECT_RETRIES + 1 times.
|
|
122
|
+
# list the plugin.
|
|
107
123
|
claude_plugin_listed() {
|
|
108
124
|
local listing
|
|
109
125
|
if ! listing="$("$CLAUDE_BIN" plugin list 2>&1)"; then
|
|
@@ -115,12 +131,39 @@ claude_plugin_listed() {
|
|
|
115
131
|
return 2
|
|
116
132
|
}
|
|
117
133
|
|
|
134
|
+
# settle_plugin_listed — settle() for the plugin probe, extended with a
|
|
135
|
+
# bounded re-list of the "list succeeded but omitted the plugin" case.
|
|
136
|
+
#
|
|
137
|
+
# With nothing staged on disk that omission is definitive — there was no
|
|
138
|
+
# plugin for the registry to index — so it is returned immediately (status
|
|
139
|
+
# 2) without spending any budget. With the manifests staged it is also the
|
|
140
|
+
# shape of the GH #3082 first-run race, where the registry index lags one
|
|
141
|
+
# scan behind the installer and the just-installed plugin is missing from an
|
|
142
|
+
# otherwise successful list. Re-list up to DETECT_PLUGIN_RELISTS times to
|
|
143
|
+
# ride out that refresh window; a genuinely absent plugin still ends up
|
|
144
|
+
# reported as "not installed", only a few cheap calls later.
|
|
145
|
+
#
|
|
146
|
+
# Like settle(), this must be called in a condition context so that `set -e`
|
|
147
|
+
# stays suppressed while the probe reports a non-zero status.
|
|
148
|
+
settle_plugin_listed() {
|
|
149
|
+
local relist=0 rc=0
|
|
150
|
+
settle claude_plugin_listed; rc=$?
|
|
151
|
+
while [ "$rc" -eq 2 ] && [ "$relist" -lt "$DETECT_PLUGIN_RELISTS" ] \
|
|
152
|
+
&& plugin_manifests_staged; do
|
|
153
|
+
relist=$((relist + 1))
|
|
154
|
+
sleep "$DETECT_RETRY_DELAY"
|
|
155
|
+
settle claude_plugin_listed; rc=$?
|
|
156
|
+
done
|
|
157
|
+
return "$rc"
|
|
158
|
+
}
|
|
159
|
+
|
|
118
160
|
if [ -n "$CLAUDE_BIN" ] && [ -x "$CLAUDE_BIN" ]; then
|
|
119
161
|
# First-run race: `claude plugin list` may transiently fail while the CLI
|
|
120
|
-
# initializes ~/.claude
|
|
121
|
-
#
|
|
122
|
-
#
|
|
123
|
-
|
|
162
|
+
# initializes ~/.claude (status 1), and may transiently omit a plugin
|
|
163
|
+
# whose manifests were staged only moments ago (status 2). Both are
|
|
164
|
+
# retried within their own bounded budgets, and "not installed" is
|
|
165
|
+
# reported only once those budgets are exhausted.
|
|
166
|
+
if settle_plugin_listed; then
|
|
124
167
|
field "plugin install" "installed ($PLUGIN_ID)"
|
|
125
168
|
else
|
|
126
169
|
field "plugin install" "not installed"
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tokenless",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"description": "Shell command output reduction and environment error diagnostics for Codex. Rewrites supported commands through RTK before execution and classifies environment failures with actionable fix hints.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"rtk",
|
|
@@ -14,6 +14,7 @@ Activation is controlled by the Hermes plugin system — list ``tokenless`` in
|
|
|
14
14
|
|
|
15
15
|
from __future__ import annotations
|
|
16
16
|
|
|
17
|
+
import inspect
|
|
17
18
|
import json
|
|
18
19
|
import logging
|
|
19
20
|
import os
|
|
@@ -76,13 +77,147 @@ def _validate_hooks_dir(path: str) -> str | None:
|
|
|
76
77
|
return None
|
|
77
78
|
|
|
78
79
|
|
|
80
|
+
# Symbols that must exist in the shared hook_utils module, paired with the
|
|
81
|
+
# exact call shape this adapter uses at its hook entry points. When a
|
|
82
|
+
# candidate passes the trust check but ships an older hook_utils.py (e.g. a
|
|
83
|
+
# stale install from a previous adapter version), the candidate is rejected and
|
|
84
|
+
# the search continues to later paths. The lifecycle migration made the Hermes
|
|
85
|
+
# adapter a thin Core client, so the required symbols are the Protocol v2
|
|
86
|
+
# request builders, the compress runner, and the Retrieve helpers this module
|
|
87
|
+
# from-imports at load time. The remaining load-time imports (resolve_binary,
|
|
88
|
+
# SHELL_TOOLS, SKIP_TOOLS, and the local-path constants) predate Protocol v2,
|
|
89
|
+
# so every module that ships the v2 API also ships them. A module missing any
|
|
90
|
+
# listed symbol would pass a narrower check and then raise ImportError at the
|
|
91
|
+
# top-level from-import, after the candidate search has already ended — with no
|
|
92
|
+
# fallback to later complete candidates.
|
|
93
|
+
#
|
|
94
|
+
# Symbol names alone are not enough: hook_utils has also changed *call
|
|
95
|
+
# signatures* while keeping names stable. ``build_post_tool_request`` took a
|
|
96
|
+
# keyword-only ``retrieval_available`` flag until c2c7e580e replaced it with
|
|
97
|
+
# the ``recovery`` mapping, so the module shipped by 9f109d559 exports all five
|
|
98
|
+
# required symbols yet rejects the ``recovery=`` this adapter passes — the
|
|
99
|
+
# plugin would import cleanly and every PostTool request would then die with
|
|
100
|
+
# "unexpected keyword argument 'recovery'", again after the candidate search
|
|
101
|
+
# has ended and with no fallback. Each shape is therefore bound with
|
|
102
|
+
# :func:`inspect.signature`; the values below are placeholders mirroring the
|
|
103
|
+
# real call sites (``bind`` only checks arity and parameter names, so nothing
|
|
104
|
+
# is ever called here).
|
|
105
|
+
_HOOK_UTILS_CALL_SHAPES: tuple[tuple[str, tuple[Any, ...], dict[str, Any]], ...] = (
|
|
106
|
+
# on_compress_pre_tool:
|
|
107
|
+
# build_pre_tool_request(args, AGENT_ID, tool_name, "command",
|
|
108
|
+
# session_id, tool_call_id,
|
|
109
|
+
# replace_arguments=False, block_and_suggest=True)
|
|
110
|
+
(
|
|
111
|
+
"build_pre_tool_request",
|
|
112
|
+
({}, "", "", "", "", ""),
|
|
113
|
+
{"replace_arguments": False, "block_and_suggest": True},
|
|
114
|
+
),
|
|
115
|
+
# on_transform_tool_result:
|
|
116
|
+
# build_post_tool_request(content, AGENT_ID, tool_name, protocol_status,
|
|
117
|
+
# content_origin, output_optimization,
|
|
118
|
+
# result_kind=..., recovery=..., session_id=...,
|
|
119
|
+
# tool_use_id=..., replace_output=True,
|
|
120
|
+
# replace_with_text=True)
|
|
121
|
+
(
|
|
122
|
+
"build_post_tool_request",
|
|
123
|
+
("", "", "", "", "", ""),
|
|
124
|
+
{
|
|
125
|
+
"result_kind": "tool",
|
|
126
|
+
"recovery": {"kind": "none"},
|
|
127
|
+
"session_id": "",
|
|
128
|
+
"tool_use_id": "",
|
|
129
|
+
"replace_output": True,
|
|
130
|
+
"replace_with_text": True,
|
|
131
|
+
},
|
|
132
|
+
),
|
|
133
|
+
# Both hooks: run_compress(tokenless_bin, request, timeout, operation)
|
|
134
|
+
("run_compress", ("", {}, 0, ""), {}),
|
|
135
|
+
# on_transform_tool_result: is_tokenless_retrieve_command(tool_name, args)
|
|
136
|
+
("is_tokenless_retrieve_command", ("", {}), {}),
|
|
137
|
+
# on_transform_tool_result: tokenless_retrieve_command_available()
|
|
138
|
+
("tokenless_retrieve_command_available", (), {}),
|
|
139
|
+
)
|
|
140
|
+
|
|
141
|
+
_HOOK_UTILS_REQUIRED_SYMBOLS = tuple(name for name, _, _ in _HOOK_UTILS_CALL_SHAPES)
|
|
142
|
+
|
|
143
|
+
|
|
144
|
+
def _restore_cached_hook_utils(saved: Any) -> None:
|
|
145
|
+
"""Reinstate the module cached before a trial import, or drop the trial."""
|
|
146
|
+
if saved is not None:
|
|
147
|
+
sys.modules["hook_utils"] = saved
|
|
148
|
+
else:
|
|
149
|
+
sys.modules.pop("hook_utils", None)
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
def _call_shape_rejection(
|
|
153
|
+
symbol: Any, name: str, args: tuple[Any, ...], kwargs: dict[str, Any]
|
|
154
|
+
) -> str | None:
|
|
155
|
+
"""Return why *symbol* cannot take the adapter call shape, else ``None``."""
|
|
156
|
+
try:
|
|
157
|
+
signature = inspect.signature(symbol)
|
|
158
|
+
except (TypeError, ValueError) as exc:
|
|
159
|
+
return f"{name} is not an introspectable callable ({exc})"
|
|
160
|
+
try:
|
|
161
|
+
signature.bind(*args, **kwargs)
|
|
162
|
+
except TypeError as exc:
|
|
163
|
+
# Name the shape the adapter needs, not only what the module offers:
|
|
164
|
+
# a renamed keyword (retrieval_available -> recovery) reports as a
|
|
165
|
+
# missing argument on some Python versions, which alone would not tell
|
|
166
|
+
# the reader which side of the contract moved.
|
|
167
|
+
return (
|
|
168
|
+
f"{name}{signature} rejects the adapter call shape "
|
|
169
|
+
f"({len(args)} positional, kwargs: {', '.join(sorted(kwargs)) or 'none'}): "
|
|
170
|
+
f"{exc}"
|
|
171
|
+
)
|
|
172
|
+
return None
|
|
173
|
+
|
|
174
|
+
|
|
175
|
+
def _check_api_compat(candidate_dir: str) -> str | None:
|
|
176
|
+
"""Trial-import hook_utils from *candidate_dir* and verify its API.
|
|
177
|
+
|
|
178
|
+
Returns ``None`` when the module loads, exposes every symbol in
|
|
179
|
+
:data:`_HOOK_UTILS_REQUIRED_SYMBOLS`, and every one of them accepts the
|
|
180
|
+
matching call shape in :data:`_HOOK_UTILS_CALL_SHAPES`; otherwise a
|
|
181
|
+
human-readable rejection reason. On success the freshly imported module is
|
|
182
|
+
kept in ``sys.modules`` so the subsequent ``from hook_utils import …``
|
|
183
|
+
reuses it rather than a stale cached copy. On rejection the ``sys.path``
|
|
184
|
+
mutation is cleaned up and the previously cached module (if any) is
|
|
185
|
+
restored so later candidates start from a clean state.
|
|
186
|
+
"""
|
|
187
|
+
sys.path.insert(0, candidate_dir)
|
|
188
|
+
saved = sys.modules.pop("hook_utils", None)
|
|
189
|
+
try:
|
|
190
|
+
import hook_utils as _trial # type: ignore[import-not-found]
|
|
191
|
+
missing = [s for s in _HOOK_UTILS_REQUIRED_SYMBOLS
|
|
192
|
+
if not hasattr(_trial, s)]
|
|
193
|
+
if missing:
|
|
194
|
+
_restore_cached_hook_utils(saved)
|
|
195
|
+
return f"API mismatch: missing {', '.join(missing)}"
|
|
196
|
+
mismatched = []
|
|
197
|
+
for name, args, kwargs in _HOOK_UTILS_CALL_SHAPES:
|
|
198
|
+
reason = _call_shape_rejection(getattr(_trial, name), name, args, kwargs)
|
|
199
|
+
if reason is not None:
|
|
200
|
+
mismatched.append(reason)
|
|
201
|
+
if mismatched:
|
|
202
|
+
_restore_cached_hook_utils(saved)
|
|
203
|
+
return "API mismatch: " + "; ".join(mismatched)
|
|
204
|
+
# Success — keep the freshly imported module in sys.modules.
|
|
205
|
+
return None
|
|
206
|
+
except Exception as exc:
|
|
207
|
+
_restore_cached_hook_utils(saved)
|
|
208
|
+
return f"import failed: {exc}"
|
|
209
|
+
finally:
|
|
210
|
+
sys.path.pop(0)
|
|
211
|
+
|
|
212
|
+
|
|
213
|
+
|
|
79
214
|
def _resolve_hook_utils() -> tuple[str, list[str]]:
|
|
80
215
|
"""Locate a trusted shared hooks directory and make it importable.
|
|
81
216
|
|
|
82
217
|
Returns ``(resolved_path, candidate_list)``. The resolved path is
|
|
83
218
|
inserted at the front of ``sys.path`` so the shared ``hook_utils``
|
|
84
219
|
module can be imported. Raises :exc:`ImportError` when no candidate
|
|
85
|
-
passes the trust policy.
|
|
220
|
+
passes both the trust policy and the API compatibility check.
|
|
86
221
|
"""
|
|
87
222
|
# Resolve real home from passwd DB for user-install fallback path
|
|
88
223
|
# (NOT $HOME — env-controllable).
|
|
@@ -119,11 +254,17 @@ def _resolve_hook_utils() -> tuple[str, list[str]]:
|
|
|
119
254
|
rejections: list[str] = []
|
|
120
255
|
for candidate in candidates:
|
|
121
256
|
reason = _validate_hooks_dir(candidate)
|
|
122
|
-
if reason is None:
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
257
|
+
if reason is not None:
|
|
258
|
+
rejections.append(f" - {candidate}: {reason}")
|
|
259
|
+
continue
|
|
260
|
+
# Trust check passed — verify API compat (version mismatch guard).
|
|
261
|
+
resolved = os.path.realpath(candidate)
|
|
262
|
+
api_reason = _check_api_compat(resolved)
|
|
263
|
+
if api_reason is not None:
|
|
264
|
+
rejections.append(f" - {candidate}: {api_reason}")
|
|
265
|
+
continue
|
|
266
|
+
sys.path.insert(0, resolved)
|
|
267
|
+
return resolved, candidates
|
|
127
268
|
|
|
128
269
|
raise ImportError(
|
|
129
270
|
"tokenless: no trusted shared hook_utils module (common/hooks/) found.\n"
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"id": "tokenless",
|
|
3
3
|
"name": "Tokenless",
|
|
4
|
-
"version": "0.8.
|
|
4
|
+
"version": "0.8.1",
|
|
5
5
|
"description": "Protocol v2 PreTool RTK rewriting and PostTool optimization. Wraps the tokenless system binary via child_process — this is expected and not malicious.",
|
|
6
6
|
"activation": {
|
|
7
7
|
"onCapabilities": ["hook"]
|
|
@@ -45,7 +45,9 @@ if [ ! -f "$PLUGIN_SRC/dist/index.js" ]; then
|
|
|
45
45
|
fi
|
|
46
46
|
|
|
47
47
|
if [ "$DRY_RUN" = "1" ]; then
|
|
48
|
-
echo "DRY-RUN: env -u OPENCLAW_HOME OPENCLAW_STATE_DIR=$OPENCLAW_STATE_DIR $OPENCLAW_BIN plugins install $PLUGIN_SRC --force
|
|
48
|
+
echo "DRY-RUN: env -u OPENCLAW_HOME OPENCLAW_STATE_DIR=$OPENCLAW_STATE_DIR $OPENCLAW_BIN plugins install $PLUGIN_SRC --force"
|
|
49
|
+
echo "DRY-RUN: add --dangerously-force-unsafe-install only while the advertised option still has effect (legacy hosts)"
|
|
50
|
+
echo "DRY-RUN: add --accept-capabilities only if advertised by plugins install --help"
|
|
49
51
|
exit 0
|
|
50
52
|
fi
|
|
51
53
|
|
|
@@ -55,19 +57,56 @@ if ! command -v "$OPENCLAW_BIN" &>/dev/null; then
|
|
|
55
57
|
exit 0
|
|
56
58
|
fi
|
|
57
59
|
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
#
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
60
|
+
INSTALL_ARGS=(plugins install "$PLUGIN_SRC" --force)
|
|
61
|
+
if ! INSTALL_HELP="$(env -u OPENCLAW_HOME OPENCLAW_STATE_DIR="$OPENCLAW_STATE_DIR" "$OPENCLAW_BIN" plugins install --help 2>&1)"; then
|
|
62
|
+
printf '[%s] Cannot inspect OpenClaw installer options: %s\n' "$COMPONENT" "$INSTALL_HELP" >&2
|
|
63
|
+
exit 1
|
|
64
|
+
fi
|
|
65
|
+
# Invoking this installer accepts declared capabilities. Older hosts must
|
|
66
|
+
# never receive an unsupported flag or a similarly named option.
|
|
67
|
+
if [[ "$INSTALL_HELP" =~ (^|[^[:alnum:]_.-])--accept-capabilities([^[:alnum:]_.-]|$) ]]; then
|
|
68
|
+
INSTALL_ARGS+=(--accept-capabilities)
|
|
69
|
+
fi
|
|
70
|
+
|
|
71
|
+
# Legacy hosts gate installs on a safety scan of child_process usage whose only
|
|
72
|
+
# non-interactive bypass is --dangerously-force-unsafe-install. OpenClaw 2026.9.2
|
|
73
|
+
# keeps the token as a deprecated no-op: passing it there accomplishes nothing
|
|
74
|
+
# and breaks again once the token is removed outright, so keep it only while the
|
|
75
|
+
# advertised option still has effect (the scanner moved to
|
|
76
|
+
# security.installPolicy on no-op hosts). The npm package ships this script to
|
|
77
|
+
# macOS, where /bin/bash stays at 3.2 — no ${var,,} here.
|
|
78
|
+
UNSAFE_SUPPORT=absent
|
|
79
|
+
while IFS= read -r help_line; do
|
|
80
|
+
if [[ "$help_line" =~ (^|[^[:alnum:]_.-])--dangerously-force-unsafe-install([^[:alnum:]_.-]|$) ]]; then
|
|
81
|
+
help_line_lc="$(printf '%s' "$help_line" | tr '[:upper:]' '[:lower:]')"
|
|
82
|
+
case "$help_line_lc" in
|
|
83
|
+
*"no op"*|*"no-op"*) UNSAFE_SUPPORT=noop ;;
|
|
84
|
+
*) UNSAFE_SUPPORT=effective; INSTALL_ARGS+=(--dangerously-force-unsafe-install) ;;
|
|
85
|
+
esac
|
|
86
|
+
break
|
|
87
|
+
fi
|
|
88
|
+
done <<< "$INSTALL_HELP"
|
|
89
|
+
|
|
90
|
+
# Why the bypass is legitimate on the hosts that still need it: their scanner
|
|
91
|
+
# reads child_process imports as "dangerous code patterns", and the tokenless
|
|
92
|
+
# plugin does delegate to the tokenless and rtk system binaries via
|
|
93
|
+
# execFileSync/spawnSync — with fixed paths and timeouts. No shell injection
|
|
94
|
+
# vector exists: all subprocess arguments are hardcoded or come from
|
|
95
|
+
# resolveBinaryPath(), never from user input. Hosts that dropped install-time
|
|
96
|
+
# scanning get no note, because nothing was bypassed.
|
|
97
|
+
if [ "$UNSAFE_SUPPORT" = "effective" ]; then
|
|
98
|
+
echo "[${COMPONENT}] Note: --dangerously-force-unsafe-install is required because"
|
|
99
|
+
echo "[${COMPONENT}] this plugin wraps tokenless/rtk system binaries via child_process."
|
|
100
|
+
echo "[${COMPONENT}] See https://github.com/alibaba/anolisa for source."
|
|
101
|
+
fi
|
|
67
102
|
|
|
68
|
-
env -u OPENCLAW_HOME OPENCLAW_STATE_DIR="$OPENCLAW_STATE_DIR" "$OPENCLAW_BIN"
|
|
69
|
-
--force --dangerously-force-unsafe-install || {
|
|
103
|
+
env -u OPENCLAW_HOME OPENCLAW_STATE_DIR="$OPENCLAW_STATE_DIR" "$OPENCLAW_BIN" "${INSTALL_ARGS[@]}" || {
|
|
70
104
|
echo "[${COMPONENT}] openclaw CLI install failed — check OpenClaw version >= 5.0.0" >&2
|
|
105
|
+
if [ "$UNSAFE_SUPPORT" = "noop" ]; then
|
|
106
|
+
echo "[${COMPONENT}] this OpenClaw advertises --dangerously-force-unsafe-install as a" >&2
|
|
107
|
+
echo "[${COMPONENT}] deprecated no-op; the safety scan follows the operator-owned" >&2
|
|
108
|
+
echo "[${COMPONENT}] security.installPolicy instead." >&2
|
|
109
|
+
fi
|
|
71
110
|
exit 1
|
|
72
111
|
}
|
|
73
112
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tokenless",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"description": "Token-Less context compression for Qwen Code — RTK command rewriting, response/TOON/schema compression, and registered but hard-disabled Tool Ready",
|
|
5
5
|
"author": { "name": "ANOLISA" },
|
|
6
6
|
"license": "Apache-2.0",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Installed by QwenPaw's plugin loader into QwenPaw's own Python environment.
|
|
2
2
|
# The native wheel is published only as a Tokenless GitHub Release asset; pip
|
|
3
3
|
# picks the single line whose environment marker matches this host.
|
|
4
|
-
anolisa-tokenless @ https://github.com/alibaba/anolisa/releases/download/tokenless/v0.8.
|
|
5
|
-
anolisa-tokenless @ https://github.com/alibaba/anolisa/releases/download/tokenless/v0.8.
|
|
6
|
-
anolisa-tokenless @ https://github.com/alibaba/anolisa/releases/download/tokenless/v0.8.
|
|
4
|
+
anolisa-tokenless @ https://github.com/alibaba/anolisa/releases/download/tokenless/v0.8.1/anolisa_tokenless-0.8.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl ; sys_platform == "linux" and platform_machine == "x86_64"
|
|
5
|
+
anolisa-tokenless @ https://github.com/alibaba/anolisa/releases/download/tokenless/v0.8.1/anolisa_tokenless-0.8.1-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl ; sys_platform == "linux" and platform_machine == "aarch64"
|
|
6
|
+
anolisa-tokenless @ https://github.com/alibaba/anolisa/releases/download/tokenless/v0.8.1/anolisa_tokenless-0.8.1-cp311-abi3-macosx_11_0_arm64.whl ; sys_platform == "darwin" and platform_machine == "arm64"
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "anolisa-tokenless",
|
|
3
3
|
"type": "module",
|
|
4
|
-
"version": "0.8.
|
|
4
|
+
"version": "0.8.1",
|
|
5
5
|
"description": "Token-Less — LLM token optimization toolkit (schema/response compression, command rewriting, tool readiness)",
|
|
6
6
|
"license": "Apache-2.0",
|
|
7
7
|
"repository": {
|
|
@@ -44,10 +44,10 @@
|
|
|
44
44
|
"arm64"
|
|
45
45
|
],
|
|
46
46
|
"optionalDependencies": {
|
|
47
|
-
"@anolisa/tokenless-linux-x64": "0.8.
|
|
48
|
-
"@anolisa/tokenless-linux-arm64": "0.8.
|
|
49
|
-
"@anolisa/tokenless-darwin-x64": "0.8.
|
|
50
|
-
"@anolisa/tokenless-darwin-arm64": "0.8.
|
|
47
|
+
"@anolisa/tokenless-linux-x64": "0.8.1",
|
|
48
|
+
"@anolisa/tokenless-linux-arm64": "0.8.1",
|
|
49
|
+
"@anolisa/tokenless-darwin-x64": "0.8.1",
|
|
50
|
+
"@anolisa/tokenless-darwin-arm64": "0.8.1"
|
|
51
51
|
},
|
|
52
52
|
"publishConfig": {
|
|
53
53
|
"registry": "https://registry.npmjs.org/",
|