@softspark/ai-toolkit 4.16.0 → 4.17.0

package/kb/reference/tool-output-filter.md

@@ -1,288 +0,0 @@
----
-title: "Native Tool Output Filter"
-category: reference
-service: ai-toolkit
-tags: [output-filter, hooks, recovery, telemetry, claude-code]
-version: "1.0.0"
-created: "2026-07-23"
-last_updated: "2026-07-23"
-description: "Contract, configuration, safety boundaries, recovery, CLI, and runtime support for the native ai-toolkit output filter."
----
-
-# Native Tool Output Filter
-
-## Overview
-
-ai-toolkit includes an original, dependency-free filter for selected
-post-execution tool results. It is disabled by default and does not depend on,
-vendor, execute, or copy another output-filter package.
-
-The active adapter targets Claude Code because its `PostToolUse` contract can
-replace a native tool response through
-`hookSpecificOutput.updatedToolOutput`. The replacement object retains the
-native response shape and changes only `stdout`. See the
-[Claude Code hooks reference](https://code.claude.com/docs/en/hooks).
-
-The filter never changes the command, arguments, environment, working
-directory, permission decision, exit status, or signal. [PATH:
-scripts/tool_output_filter/hook_runtime.py] [PATH:
-scripts/tool_output_filter/engine.py]
-
-## Modes
-
-| Mode | Model-visible result | Recovery | Telemetry |
-|------|----------------------|----------|-----------|
-| `off` | Original | None | None |
-| `observe` | Original | No raw response | Content-free decision metadata |
-| `safe` | Replacement only after every gate passes | Exact native response saved first | Content-free decision metadata |
-
-`off` is a shell fast path, so the Python runtime is not started. Any runtime
-error, malformed payload, unsafe command, failed invariant, unavailable secure
-storage, or insufficient saving leaves the original response unchanged.
-
-Three consecutive profile, invariant, or recovery safety failures open a
-persistent session-scoped circuit breaker. One bounded Claude system message
-reports the bypass, then later results stay unchanged for that session.
-
-## Configuration
-
-Configure the project in `.softspark-toolkit.json`:
-
-```json
-{
-  "toolOutputFilter": {
-    "mode": "observe",
-    "profiles": ["repeat-lines", "tap-success"],
-    "maxInputBytes": 8388608,
-    "minSavingsBytes": 1024,
-    "minSavingsRatio": 0.15,
-    "recovery": {
-      "mode": "ephemeral",
-      "ttlMinutes": 60,
-      "maxSessionBytes": 33554432
-    }
-  }
-}
-```
-
-Run `ai-toolkit install --local` or `ai-toolkit update --local` to materialize
-the effective policy as:
-
-```text
-<project>/.claude/ai-toolkit-output-filter.json
-<project>/.claude/.ai-toolkit-output-filter.owner
-```
-
-The managed files use mode `0600`. The hook accepts a project policy only when
-the project root is registered in `~/.softspark/ai-toolkit/projects.json`
-**and** the regular owner marker matches ai-toolkit. Registration is the
-security boundary: the owner marker is a public constant, so requiring the
-registry stops a cloned or untrusted checkout from self-enabling filtering by
-committing its own marker. An unregistered project, a missing or foreign
-marker, or a symlinked project root or `.claude` directory falls back to the
-installed global policy at
-`~/.softspark/ai-toolkit/hooks/output-filter-policy.json`, which defaults to
-`off`. [PATH: app/hooks/filter-tool-output.sh] [PATH:
-scripts/install_steps/ai_tools.py]
-
-`jq` is a required system dependency for the lifecycle hooks and is verified
-by `python3 scripts/check_deps.py` alongside `python3`, `git`, and `node`.
-
-Before executing anything, the hook validates the resolved Python runtime path:
-it must be a readable regular file and must not be a symlink. A missing,
-non-regular, unreadable, or symlinked runtime makes the hook exit silently and
-leave the tool response unchanged, so a tampered or half-installed runtime
-cannot be invoked. The same regular-file rule applies to every policy file the
-hook reads. [PATH: app/hooks/filter-tool-output.sh]
-
-Set `AI_TOOLKIT_OUTPUT_FILTER_DISABLE=1` for an immediate bypass without
-reinstalling. `AI_TOOLKIT_OUTPUT_FILTER_POLICY` may point the hook to an
-explicit regular policy file for controlled operational testing. The hook is
-also skipped by the `minimal` hook profile and may be listed in
-`AI_TOOLKIT_DISABLED_HOOKS`. `AI_TOOLKIT_OUTPUT_FILTER_HOOK_RUNTIME` is reserved
-for controlled runtime testing; the manual and cleanup CLI remains
-`output_filter_cli.py`.
-
-## Eligibility
-
-The Claude adapter considers only a completed `PostToolUse` event with:
-
-- tool name `Bash`;
-- a non-empty native session ID of at most 160 ASCII letters, digits,
-  underscores, or hyphens;
-- string `stdout`;
-- empty `stderr`;
-- `interrupted: false`;
-- `isImage: false`;
-- a command that matches a strict allowlist for test, lint, typecheck, or
-  validation tools;
-- input at or below 8 MiB;
-- valid text without binary or terminal-control content.
-
-The following always pass through unchanged:
-
-- failed, interrupted, image, binary, invalid-text, TTY, or streaming results;
-- pipes, redirects, shell chaining, substitutions, and multiline commands;
-- deployment, release, migration, publish, destroy, audit, and security-scanner
-  commands;
-- arbitrary Python scripts and unknown command shapes;
-- output with non-empty stderr;
-- unknown profiles or native payload shapes;
-- candidates that save less than both the configured byte and ratio threshold.
-
-The command classifier is eligibility logic only. It never parses and
-re-executes a command. [PATH: scripts/tool_output_filter/hook_runtime.py]
-
-## Profiles
-
-### `repeat-lines`
-
-Collapses only adjacent identical, non-diagnostic lines. It retains the first
-line and adds a versioned marker with the exact number of omitted copies.
-Warnings, failures, permissions, security diagnostics, blank lines, comments,
-existing filter markers, and control-bearing output are not collapsed.
-
-### `tap-success`
-
-Accepts only a strict, complete, successful TAP stream with a single plan and
-contiguous `ok` result numbers. It retains the TAP version, plan, directives,
-comments, totals, duration, and other summary lines. Diagnostics, `not ok`,
-non-zero failure summaries, malformed plans, gaps, duplicates, and unknown
-content reject the whole profile.
-
-Both profiles are deterministic and idempotent. A safe replacement must remain
-smaller after the recovery marker is added. [PATH:
-scripts/tool_output_filter/profiles/] [PATH:
-tests/test_tool_output_filter_properties.py]
-
-## Exact Recovery and Privacy
-
-Before `safe` mode emits a replacement, it stores and reloads the complete
-native tool-response object. Equality must succeed before the hook prints
-`updatedToolOutput`.
-
-```text
-~/.softspark/ai-toolkit/sessions/<repo-key>/
-└── output-filter/
-    └── <hashed-session>/
-        ├── <opaque-handle>.json
-        ├── .circuit-state.json
-        └── .telemetry.jsonl
-```
-
-Recovery directories use `0700`; response, state, and telemetry files use
-`0600`. Creation and cleanup use pinned directory descriptors, no-follow
-operations, atomic publication, opaque random handles, a per-session quota,
-and TTL cleanup. If this secure contract is unavailable, `safe` mode returns
-the original response.
-
-The replacement ends with a marker similar to:
-
-```text
-[ai-toolkit-output-filter repeat-lines/v1; original_lines=500; emitted_lines=3; recovery=<opaque-handle>]
-```
-
-The recovery file can contain everything returned by the tool, including
-secrets. Treat the session directory as sensitive. Telemetry never stores raw
-output, commands, paths, environment values, session IDs, or recovery handles.
-It contains only profile/version, input/output byte and line counts, latency,
-outcome, and a bounded fallback reason.
-
-Session end, explicit cleanup, and global uninstall remove only validated
-ai-toolkit-owned filter artifacts. Foreign files and directories are
-preserved. [PATH: scripts/tool_output_filter/recovery.py] [PATH:
-app/hooks/session-end.sh] [PATH: scripts/uninstall.py]
-
-## CLI
-
-Inspect candidate savings without changing output:
-
-```bash
-some-test-command | ai-toolkit output-filter inspect --profile repeat-lines
-some-tap-command | ai-toolkit output-filter inspect --profile tap-success
-```
-
-The JSON report contains counts, eligibility, outcome, and fallback reason. It
-does not echo stdin.
-
-Inspect the effective trusted project or global policy:
-
-```bash
-ai-toolkit output-filter status
-ai-toolkit output-filter status --policy /path/to/materialized-policy.json
-```
-
-Recover the exact native response object using the handle printed in a safe
-replacement:
-
-```bash
-ai-toolkit output-filter recover <opaque-handle>
-```
-
-The default lookup derives the current repository session directory. Advanced
-or test workflows can add `--base-directory PATH` or `--session-id ID`.
-
-Clean the ending session, expired exact responses, or all filter artifacts for
-the current repository:
-
-```bash
-ai-toolkit output-filter clean --session-id <native-session-id>
-ai-toolkit output-filter clean --session-id <native-session-id> --expired
-ai-toolkit output-filter clean
-```
-
-`--expired` requires `--session-id`. Cleanup prints only the removed artifact
-count and scope.
-
-## Runtime Capability Matrix
-
-| Runtime | Active result replacement | Capability |
-|---------|---------------------------|------------|
-| Claude Code | Yes, opt-in | Native `PostToolUse.updatedToolOutput` adapter |
-| Claude Chat / Cowork | No | Plugin export explicitly excludes the Claude Code-only hook |
-| Cursor | No | Manual `output-filter inspect` only |
-| Windsurf / Devin | No | Manual `output-filter inspect` only |
-| GitHub Copilot | No | Manual `output-filter inspect` only |
-| Gemini CLI | No | Manual `output-filter inspect` only |
-| Cline | No | Manual `output-filter inspect` only |
-| Roo Code | No | Manual `output-filter inspect` only |
-| Aider | No | Manual `output-filter inspect` only |
-| Augment | No | Manual `output-filter inspect` only |
-| Google Antigravity | No | Manual `output-filter inspect` only |
-| Codex CLI | No | Manual `output-filter inspect` only |
-| OpenCode | No | Manual `output-filter inspect` only |
-
-An editor hook, extra context message, or command wrapper is not treated as
-result replacement. A new adapter requires a verified native replacement
-contract and dedicated native payload tests.
-
-## Benchmark Semantics
-
-Run the deterministic offline corpus:
-
-```bash
-PYTHONDONTWRITEBYTECODE=1 python3 scripts/benchmark_output_filter.py
-```
-
-The benchmark measures profile p95 latency, production Bash-wrapper latency
-with a fresh Python process per sample in one native session, traced peak
-allocation, and eligible-output byte reduction. It uses 100 samples by default
-to avoid a one-sample p95 swing. The current gates are:
-
-- at least 30% candidate byte reduction;
-- at most 20 ms p95 for profile inputs up to 100 KiB;
-- at most 150 ms p95 for the 8 MiB profile case;
-- at most 75 ms p95 for a cold end-to-end hook process;
-- peak traced allocation no greater than three input sizes plus 16 MiB.
-
-Byte reduction is not billed-token savings and is not a whole-session cost
-claim. Measure actual model token receipts separately before changing the
-default mode. [PATH: scripts/benchmark_output_filter.py] [PATH:
-benchmarks/output-filter/]
-
-## Related
-
-- [Hooks Catalog](hooks-catalog.md)
-- [Supported Tools Registry](supported-tools-registry.md)
-- [Architecture Overview](architecture-overview.md)
-- [Output Token Discipline Plan](../history/completed/output-token-discipline-plan-20260504.md)

package/scripts/benchmark_output_filter.py

@@ -1,343 +0,0 @@
-#!/usr/bin/env python3
-"""Deterministic offline benchmarks for native output profiles."""
-
-from __future__ import annotations
-
-import argparse
-import json
-import math
-import os
-import subprocess
-import sys
-import tempfile
-import time
-import tracemalloc
-from pathlib import Path
-
-from tool_output_filter.profiles import apply_profile
-
-REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
-DEFAULT_CORPUS = (
-    REPOSITORY_ROOT / "benchmarks" / "output-filter" / "scenarios.json"
-)
-MEMORY_ALLOWANCE_BYTES = 16 * 1024 * 1024
-COLD_HOOK_INPUT_BYTES = 4 * 1024
-COLD_HOOK_MAX_P95_MS = 75.0
-DEFAULT_ITERATIONS = 100
-HOOK_RUNTIME = REPOSITORY_ROOT / "scripts" / "output_filter_hook.py"
-HOOK_SCRIPT = REPOSITORY_ROOT / "app" / "hooks" / "filter-tool-output.sh"
-
-
-def _repeat_output(target_bytes: int, line_width: int) -> str:
-    prefix = "synthetic progress "
-    line = prefix + ("x" * max(1, line_width - len(prefix) - 1)) + "\n"
-    count = math.ceil(target_bytes / len(line.encode("utf-8")))
-    return line * count
-
-
-def _tap_output(test_count: int) -> str:
-    result_lines = [
-        f"ok {number} - synthetic benchmark case {number}\n"
-        for number in range(1, test_count + 1)
-    ]
-    return "".join(
-        [
-            "TAP version 13\n",
-            f"1..{test_count}\n",
-            *result_lines,
-            f"# tests {test_count}\n",
-            f"# pass {test_count}\n",
-            "# fail 0\n",
-            "# duration_ms 1\n",
-        ]
-    )
-
-
-def _integer_field(scenario: dict[str, object], key: str) -> int:
-    value = scenario.get(key)
-    if isinstance(value, bool) or not isinstance(value, int):
-        raise ValueError(f"benchmark field {key} must be an integer")
-    return value
-
-
-def _number_field(scenario: dict[str, object], key: str) -> float:
-    value = scenario.get(key)
-    if isinstance(value, bool) or not isinstance(value, (int, float)):
-        raise ValueError(f"benchmark field {key} must be numeric")
-    return float(value)
-
-
-def _string_field(scenario: dict[str, object], key: str) -> str:
-    value = scenario.get(key)
-    if not isinstance(value, str):
-        raise ValueError(f"benchmark field {key} must be text")
-    return value
-
-
-def _scenario_input(scenario: dict[str, object]) -> str:
-    kind = _string_field(scenario, "kind")
-    if kind == "repeat":
-        return _repeat_output(
-            _integer_field(scenario, "targetBytes"),
-            _integer_field(scenario, "lineWidth"),
-        )
-    if kind == "tap":
-        return _tap_output(_integer_field(scenario, "testCount"))
-    raise ValueError(f"unknown benchmark kind: {kind}")
-
-
-def _p95(values: list[float]) -> float:
-    ordered = sorted(values)
-    index = max(0, math.ceil(len(ordered) * 0.95) - 1)
-    return ordered[index]
-
-
-def _measure_scenario(
-    scenario: dict[str, object],
-    iterations: int,
-) -> dict[str, object]:
-    raw_output = _scenario_input(scenario)
-    profile_id = _string_field(scenario, "profile")
-    scenario_name = _string_field(scenario, "name")
-    durations: list[float] = []
-    candidate = apply_profile(profile_id, raw_output)
-    if candidate is None or not candidate.accepted:
-        raise RuntimeError(f"profile rejected benchmark {scenario_name}")
-
-    for _ in range(iterations):
-        started = time.perf_counter_ns()
-        measured = apply_profile(profile_id, raw_output)
-        durations.append((time.perf_counter_ns() - started) / 1_000_000)
-        if measured != candidate:
-            raise RuntimeError(
-                f"non-deterministic benchmark {scenario_name}"
-            )
-
-    tracemalloc.start()
-    apply_profile(profile_id, raw_output)
-    _, peak_bytes = tracemalloc.get_traced_memory()
-    tracemalloc.stop()
-
-    input_bytes = len(raw_output.encode("utf-8"))
-    candidate_bytes = len(candidate.output.encode("utf-8"))
-    savings_ratio = (input_bytes - candidate_bytes) / input_bytes
-    p95_ms = _p95(durations)
-    memory_limit = input_bytes * 3 + MEMORY_ALLOWANCE_BYTES
-    passed = (
-        candidate_bytes < input_bytes
-        and savings_ratio >= 0.3
-        and p95_ms <= _number_field(scenario, "maxP95Ms")
-        and peak_bytes <= memory_limit
-    )
-    return {
-        "name": scenario_name,
-        "profile": profile_id,
-        "inputBytes": input_bytes,
-        "candidateBytes": candidate_bytes,
-        "savingsRatio": round(savings_ratio, 4),
-        "p95Ms": round(p95_ms, 3),
-        "peakBytes": peak_bytes,
-        "memoryLimitBytes": memory_limit,
-        "passed": passed,
-    }
-
-
-def _hook_policy(mode: str) -> dict[str, object]:
-    return {
-        "mode": mode,
-        "profiles": ["repeat-lines"],
-        "maxInputBytes": 8 * 1024 * 1024,
-        "minSavingsBytes": 1024,
-        "minSavingsRatio": 0.15,
-        "recovery": {
-            "mode": "ephemeral",
-            "ttlMinutes": 60,
-            "maxSessionBytes": 32 * 1024 * 1024,
-        },
-    }
-
-
-def _hook_payload(raw_output: str, session_id: str) -> str:
-    payload = {
-        "hook_event_name": "PostToolUse",
-        "tool_name": "Bash",
-        "session_id": session_id,
-        "cwd": str(REPOSITORY_ROOT),
-        "tool_input": {"command": "npm test"},
-        "tool_response": {
-            "stdout": raw_output,
-            "stderr": "",
-            "interrupted": False,
-            "isImage": False,
-        },
-    }
-    return json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
-
-
-def _hook_session_root(temporary_home: Path) -> Path:
-    repo_key = "-" + str(REPOSITORY_ROOT).replace("/", "-").lstrip("-")
-    return (
-        temporary_home
-        / ".softspark"
-        / "ai-toolkit"
-        / "sessions"
-        / repo_key
-    )
-
-
-def _run_cold_hook(
-    payload: str,
-    environment: dict[str, str],
-) -> tuple[float, int]:
-    started = time.perf_counter_ns()
-    result = subprocess.run(
-        ["bash", str(HOOK_SCRIPT)],
-        input=payload,
-        text=True,
-        capture_output=True,
-        cwd=REPOSITORY_ROOT,
-        env=environment,
-        check=False,
-    )
-    duration_ms = (time.perf_counter_ns() - started) / 1_000_000
-    if result.returncode != 0 or result.stderr:
-        raise RuntimeError("cold hook invocation failed")
-    if result.stdout:
-        raise RuntimeError("observe hook unexpectedly changed output")
-    return duration_ms, len(result.stdout.encode("utf-8"))
-
-
-def _measure_cold_hook(
-    iterations: int,
-    mode: str = "observe",
-) -> dict[str, object]:
-    raw_output = _repeat_output(COLD_HOOK_INPUT_BYTES, 96)
-    input_bytes = len(raw_output.encode("utf-8"))
-    durations: list[float] = []
-    emitted_sizes: list[int] = []
-    with tempfile.TemporaryDirectory() as temporary_directory:
-        temporary_path = Path(temporary_directory)
-        policy_path = temporary_path / "policy.json"
-        policy_path.write_text(
-            json.dumps(_hook_policy(mode), separators=(",", ":")),
-            encoding="utf-8",
-        )
-        temporary_home = temporary_path / "home"
-        session_root = _hook_session_root(temporary_home)
-        session_root.mkdir(parents=True, mode=0o700)
-        session_root.chmod(0o700)
-        environment = {
-            **os.environ,
-            "HOME": str(temporary_home),
-            "PYTHONPYCACHEPREFIX": str(temporary_path / "pycache"),
-            "AI_TOOLKIT_DISABLED_HOOKS": "",
-            "AI_TOOLKIT_OUTPUT_FILTER_HOOK_RUNTIME": str(HOOK_RUNTIME),
-            "AI_TOOLKIT_OUTPUT_FILTER_DISABLE": "0",
-            "AI_TOOLKIT_OUTPUT_FILTER_POLICY": str(policy_path),
-            "TOOLKIT_HOOK_PROFILE": "standard",
-        }
-        environment.pop("PYTHONDONTWRITEBYTECODE", None)
-        session_id = "benchmark-session"
-        for iteration in range(iterations):
-            duration_ms, emitted_bytes = _run_cold_hook(
-                _hook_payload(raw_output, session_id),
-                environment,
-            )
-            durations.append(duration_ms)
-            emitted_sizes.append(emitted_bytes)
-        if mode == "observe":
-            telemetry_files = list(session_root.rglob(".telemetry.jsonl"))
-            if len(telemetry_files) != 1:
-                raise RuntimeError(
-                    "cold hook did not complete observe telemetry"
-                )
-            telemetry_lines = telemetry_files[0].read_text(
-                encoding="ascii",
-            ).splitlines()
-            if len(telemetry_lines) != iterations:
-                raise RuntimeError(
-                    "cold hook telemetry event count is incomplete"
-                )
-    p95_ms = _p95(durations)
-    emitted_bytes = max(emitted_sizes)
-    return {
-        "name": "claude-hook-cold-4k",
-        "mode": mode,
-        "adapter": "bash-wrapper",
-        "sessionReused": True,
-        "iterations": iterations,
-        "inputBytes": input_bytes,
-        "emittedBytes": emitted_bytes,
-        "p95Ms": round(p95_ms, 3),
-        "maxP95Ms": COLD_HOOK_MAX_P95_MS,
-        "passed": (
-            emitted_bytes < input_bytes
-            and p95_ms <= COLD_HOOK_MAX_P95_MS
-        ),
-    }
-
-
-def _load_corpus(path: Path) -> list[dict[str, object]]:
-    with path.open(encoding="utf-8") as corpus_file:
-        data = json.load(corpus_file)
-    if not isinstance(data, list):
-        raise ValueError("benchmark corpus must be an array")
-    return data
-
-
-def _build_parser() -> argparse.ArgumentParser:
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--corpus", type=Path, default=DEFAULT_CORPUS)
-    parser.add_argument("--iterations", type=int, default=DEFAULT_ITERATIONS)
-    parser.add_argument("--max-input-bytes", type=int)
-    parser.add_argument("--report-cold-hook-only", action="store_true")
-    parser.add_argument("--json", action="store_true")
-    return parser
-
-
-def main(argv: list[str] | None = None) -> int:
-    args = _build_parser().parse_args(argv)
-    if args.iterations <= 0:
-        return 2
-    scenarios = []
-    for definition in _load_corpus(args.corpus):
-        raw_size = len(_scenario_input(definition).encode("utf-8"))
-        if args.max_input_bytes is not None and raw_size > args.max_input_bytes:
-            continue
-        scenarios.append(_measure_scenario(definition, args.iterations))
-    cold_hook = _measure_cold_hook(args.iterations)
-    profile_gates_passed = bool(scenarios) and all(
-        scenario["passed"] for scenario in scenarios
-    )
-    cold_hook_enforced = not args.report_cold_hook_only
-    report = {
-        "passed": (
-            profile_gates_passed
-            and (
-                cold_hook["passed"]
-                or not cold_hook_enforced
-            )
-        ),
-        "scenarios": scenarios,
-        "coldHook": cold_hook,
-        "coldHookEnforced": cold_hook_enforced,
-    }
-    if args.json:
-        print(json.dumps(report, separators=(",", ":")))
-    else:
-        for scenario in scenarios:
-            print(
-                f"{scenario['name']}: p95={scenario['p95Ms']}ms "
-                f"savings={scenario['savingsRatio']} "
-                f"peak={scenario['peakBytes']} passed={scenario['passed']}"
-            )
-        print(
-            f"{cold_hook['name']}: p95={cold_hook['p95Ms']}ms "
-            f"emitted={cold_hook['emittedBytes']} "
-            f"passed={cold_hook['passed']}"
-        )
-    return 0 if report["passed"] else 1
-
-
-if __name__ == "__main__":
-    sys.exit(main())