@softspark/ai-toolkit 4.16.0 → 4.17.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. package/CHANGELOG.md +41 -0
  2. package/README.md +11 -16
  3. package/app/.claude-plugin/plugin.json +1 -1
  4. package/app/hooks/session-end.sh +1 -13
  5. package/app/hooks.json +0 -10
  6. package/benchmarks/ecosystem-doctor-snapshot.json +14 -15
  7. package/bin/ai-toolkit.js +0 -2
  8. package/kb/history/completed/output-filter-retirement-20260726.md +128 -0
  9. package/kb/reference/architecture-overview.md +2 -3
  10. package/kb/reference/cli-reference.md +3 -13
  11. package/kb/reference/enterprise-config-guide.md +1 -21
  12. package/kb/reference/hooks-catalog.md +3 -60
  13. package/kb/reference/supported-tools-registry.md +0 -4
  14. package/llms-full.txt +143 -395
  15. package/llms.txt +1 -1
  16. package/manifest.json +147 -36
  17. package/package.json +1 -2
  18. package/scripts/claude_app.py +2 -21
  19. package/scripts/config_cli.py +4 -0
  20. package/scripts/config_merger.py +0 -17
  21. package/scripts/config_validator.py +11 -138
  22. package/scripts/doctor.py +3 -20
  23. package/scripts/generate_copilot.py +35 -4
  24. package/scripts/install.py +7 -2
  25. package/scripts/install_steps/ai_tools.py +28 -99
  26. package/scripts/install_steps/hooks.py +26 -24
  27. package/scripts/merge-hooks.py +33 -2
  28. package/scripts/output_filter_retirement.py +395 -0
  29. package/scripts/schemas/ai-toolkit-config.schema.json +0 -60
  30. package/scripts/uninstall.py +13 -27
  31. package/app/hooks/filter-tool-output.sh +0 -76
  32. package/app/output-filter-policy.json +0 -15
  33. package/benchmarks/output-filter/README.md +0 -11
  34. package/benchmarks/output-filter/scenarios.json +0 -25
  35. package/kb/reference/tool-output-filter.md +0 -288
  36. package/scripts/benchmark_output_filter.py +0 -343
  37. package/scripts/output_filter_cli.py +0 -347
  38. package/scripts/output_filter_hook.py +0 -23
  39. package/scripts/tool_output_filter/__init__.py +0 -33
  40. package/scripts/tool_output_filter/contracts.py +0 -173
  41. package/scripts/tool_output_filter/engine.py +0 -260
  42. package/scripts/tool_output_filter/hook_runtime.py +0 -369
  43. package/scripts/tool_output_filter/input.py +0 -56
  44. package/scripts/tool_output_filter/invariants.py +0 -40
  45. package/scripts/tool_output_filter/policy.py +0 -153
  46. package/scripts/tool_output_filter/profiles/__init__.py +0 -68
  47. package/scripts/tool_output_filter/profiles/repeat_lines.py +0 -71
  48. package/scripts/tool_output_filter/profiles/tap_success.py +0 -154
  49. package/scripts/tool_output_filter/recovery.py +0 -846
  50. package/scripts/tool_output_filter/telemetry.py +0 -13
package/CHANGELOG.md CHANGED
@@ -7,6 +7,47 @@ Versioning follows [Semantic Versioning](https://semver.org/).
7
7
 
8
8
  ---
9
9
 
10
+ ## v4.17.0 — Native tool-output filter removed (2026-07-26)
11
+
12
+ ### Removed
13
+
14
+ - **BREAKING for anyone who opted in.** The native tool-output filter shipped
15
+ in v4.16.0 is gone: the `PostToolUse` hook `filter-tool-output.sh`, the
16
+ `scripts/tool_output_filter/` runtime, the `ai-toolkit output-filter` CLI,
17
+ the benchmark corpus, and the `toolOutputFilter` key in
18
+ `.softspark-toolkit.json`. The key is accepted and ignored rather than
19
+ rejected, so an existing config still validates; `ai-toolkit config validate`
20
+ reports it as a retired key you can delete.
21
+ - Why: measured whole-session token saving was **0.0000%** on real traffic.
22
+ Across 134 session transcripts spanning 22 projects, 7600 successful Bash
23
+ results contained 145 that parsed as simple command shapes, 18 that matched a
24
+ registered shape, and 0 that any filter accepted. The filter was correct;
25
+ its premise was not. Agent-issued commands are overwhelmingly compound
26
+ (`&&` and `;` chains, pipelines, heredocs), and the design accepted only
27
+ simple registered shapes. See
28
+ `kb/history/completed/output-filter-retirement-20260726.md`.
29
+
30
+ ### Changed
31
+
32
+ - `ai-toolkit install` and `ai-toolkit update` now remove the artifacts left by
33
+ v4.16.x: the installed hook script, the global
34
+ `hooks/output-filter-policy.json`, the managed project policy and owner
35
+ marker under `.claude/`, and stored recovery trees under
36
+ `sessions/<repo-key>/output-filter/`. Cleanup is idempotent, verifies
37
+ ai-toolkit ownership before deleting, and leaves foreign files untouched.
38
+ - The stale `PostToolUse` entry is removed from `~/.claude/settings.json` by
39
+ the existing hook strip-and-remerge cycle, so no manual edit is needed.
40
+ - Hook counts drop from 29 entries to 28 across the same 14 events.
41
+
42
+ ## v4.16.1 — Copilot skill remnant recovery (2026-07-24)
43
+
44
+ ### Fixed
45
+
46
+ - **Legacy Copilot skill remnants no longer abort local updates** — `.github/skills/ai-toolkit-*` directories left behind by pre-manifest cleanups (assets without `SKILL.md`) are rebuilt in place instead of raising `Refusing user-owned Copilot skill collision`, which aborted the entire `install --local` / `update` run on affected projects. User files inside a remnant that do not collide with generated output are preserved; any directory that still contains a `SKILL.md` stays protected and is never reclaimed.
47
+ - **Readable installer errors** — `scripts/install.py` reports fail-closed `RuntimeError` conditions as a single `ERROR:` line instead of a raw traceback.
48
+
49
+ Test count: 1477 → 1479.
50
+
10
51
  ## v4.16.0 — Native tool-output filtering (2026-07-24)
11
52
 
12
53
  Minor release. Adds a dependency-free, opt-in native tool-output filter for Claude Code and hardens the enterprise configuration, session-isolation, and editor-adapter surfaces around it. The filter ships disabled; `observe` and `safe` are per-project opt-ins. No source catalog count change (44 agents, 108 skills).
package/README.md CHANGED
@@ -6,17 +6,17 @@
6
6
  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
7
7
  [![Skills](https://img.shields.io/badge/skills-108-brightgreen)](app/skills/)
8
8
  [![Agents](https://img.shields.io/badge/agents-44-blue)](app/agents/)
9
- [![Tests](https://img.shields.io/badge/tests-1477%20passing-success)](tests/)
9
+ [![Tests](https://img.shields.io/badge/tests-1445%20passing-success)](tests/)
10
10
 
11
- ## What's New in v4.16.0
11
+ ## What's New in v4.17.0
12
12
 
13
- v4.16.0 adds a dependency-free, opt-in native tool-output filter for Claude Code and hardens the enterprise configuration surface around it.
13
+ v4.17.0 removes the native tool-output filter shipped in v4.16.0.
14
14
 
15
- - **Native tool-output filter**: opt-in `PostToolUse` filter with `off`, `observe`, and `safe` modes; ships disabled, handles only allowlisted successful Bash test/validation output.
16
- - **Exact ephemeral recovery**: every safe replacement keeps a private, bounded, session-scoped copy of the original response, recoverable via `ai-toolkit output-filter recover`.
17
- - **Output-filter CLI**: `ai-toolkit output-filter status|inspect|recover|clean` plus a deterministic benchmark for the built-in profiles.
15
+ - **Native tool-output filter removed**: measured whole-session token saving was 0.0000% on real traffic, because agent-issued commands are overwhelmingly compound and the filter only accepted simple registered shapes. The `PostToolUse` hook, the CLI, and the `toolOutputFilter` config key are gone.
16
+ - **Automatic cleanup on update**: `ai-toolkit update` removes the orphaned hook script, the global and project policy files, and any stored recovery artifacts left by v4.16.x. Foreign files in those directories are preserved.
18
17
  - **Stricter config validation**: unknown top-level config keys are rejected, plugin manifests must declare `requires`, and lock files go stale on toolkit version bumps.
19
18
  - **GitHub Copilot compatibility reference**: new `kb/reference/copilot-compatibility.md` documenting the Copilot integration surface.
19
+ - **Copilot remnant recovery**: asset-only `.github/skills/ai-toolkit-*` leftovers from older cleanups are rebuilt in place instead of aborting `install --local` / `update`.
20
20
 
21
21
  See [CHANGELOG.md](CHANGELOG.md) for full history.
22
22
 
@@ -91,10 +91,7 @@ ai-toolkit claude-app export --verify
91
91
  ```
92
92
 
93
93
  Re-export and re-upload after toolkit or registered-rule updates. Skills work
94
- in Chat and Cowork; hooks and sub-agents are active only in Cowork. The native
95
- tool-output replacement hook remains Claude Code-only and is excluded from the
96
- Claude app archive until that runtime has an independently verified replacement
97
- contract.
94
+ in Chat and Cowork; hooks and sub-agents are active only in Cowork.
98
95
 
99
96
  ### Install Profiles
100
97
 
@@ -126,7 +123,7 @@ See [CLI Reference](kb/reference/cli-reference.md) for all commands and options.
126
123
  | Gemini CLI | `~/.gemini/GEMINI.md` | ✅ | global |
127
124
  | GitHub Copilot | Project: `AGENTS.md` + `.github/copilot-instructions.md` + `.github/{instructions,prompts,agents,skills,hooks}/` + `.github/mcp.json`; user: `$COPILOT_HOME/copilot-instructions.md` + `$COPILOT_HOME/{instructions,agents,skills,hooks}/` + `$COPILOT_HOME/mcp-config.json` | ✅ | project + user |
128
125
  | Cline | `~/Documents/Cline/Rules/*.md` + `~/.cline/skills/*` + `.clinerules/*.md` | — | global + project |
129
- | Roo Code | `~/.roo/rules/*.md` + `.roomodes` + `.roo/rules/*.md` | — | global rules + project |
126
+ | Roo Code / [Zoo Code](https://zoocode.dev/) | `~/.roo/rules/*.md` + `.roomodes` + `.roo/rules/*.md` | — | global rules + project |
130
127
  | Aider | `~/.aider.conf.yml` + `.aider.conf.yml` + `CONVENTIONS.md` | — | global + project |
131
128
  | Augment | `~/.augment/rules/*.md` + `.augment/rules/ai-toolkit-*.md` | ✅ | global + project |
132
129
  | Google Antigravity | `.agents/rules/*.md` + `.agents/workflows/*.md` + skill pointer in `.agent/skills/*` (IDE) and `.agents/skills/*` (CLI) | — | project |
@@ -145,7 +142,7 @@ See [CLI Reference](kb/reference/cli-reference.md) for all commands and options.
145
142
  | `skills/` (hybrid) | 30 | Slash commands with agent knowledge base |
146
143
  | `skills/` (knowledge) | 46 | Domain knowledge auto-loaded by agents (includes 13 `<lang>-rules` skills) |
147
144
  | `agents/` | 44 | Specialized agents across 10 categories |
148
- | `hooks/` | 29 entries / 14 events + statusLine | Quality gates, path safety, prompt governance, loop guard, output filtering, session lifecycle |
145
+ | `hooks/` | 28 entries / 14 events + statusLine | Quality gates, path safety, prompt governance, loop guard, session lifecycle |
149
146
  | `plugins/` | 11 packs | Opt-in domain bundles (security, research, frontend, enterprise, 6 language packs) |
150
147
  | `constitution.md` | 7 articles | Machine-enforced safety rules |
151
148
  | `rules/` | auto-synced | Global/project rule files for Claude and other editors |
@@ -169,7 +166,7 @@ ai-toolkit/
169
166
  │ └── ARCHITECTURE.md # Full system design
170
167
  ├── kb/ # Reference docs, procedures, plans
171
168
  ├── scripts/ # Validation, install, evaluation scripts
172
- ├── tests/ # Bats and Python test suite (1477 tests)
169
+ ├── tests/ # Bats and Python test suite (1445 tests)
173
170
  └── CHANGELOG.md
174
171
  ```
175
172
 
@@ -181,9 +178,7 @@ ai-toolkit/
181
178
 
182
179
  **Machine-enforced constitution** — 7-article safety constitution enforced via `PreToolUse` hooks that actually block `rm -rf`, `DROP TABLE`, and irreversible operations. Not just documentation.
183
180
 
184
- **29 lifecycle hook entries:** Executable handlers across 14 events (SessionStart → SessionEnd, plus InstructionsLoaded + ConfigChange). Guards, governance, quality gates, session persistence, MCP health checks, revert protection, test-cohesion enforcement, loop guard, search-first discipline, and opt-in output filtering. See [Hooks Catalog](kb/reference/hooks-catalog.md).
185
-
186
- **Native tool-output filtering** — dependency-free, post-execution filtering for successful Claude Code Bash output, shipped **disabled by default** (`off`). Neither `off` nor `observe` ever replaces output; opt-in `safe` mode replaces only validated `repeat-lines` or `tap-success` results after exact ephemeral recovery is available. Failures, diagnostics, unsupported payloads, and unavailable recovery always pass through unchanged. See [Tool Output Filter](kb/reference/tool-output-filter.md).
181
+ **28 lifecycle hook entries:** Executable handlers across 14 events (SessionStart → SessionEnd, plus InstructionsLoaded + ConfigChange). Guards, governance, quality gates, session persistence, MCP health checks, revert protection, test-cohesion enforcement, loop guard, and search-first discipline. See [Hooks Catalog](kb/reference/hooks-catalog.md).
187
182
 
188
183
  **Security scanning** — `/skill-audit` for code-level risks, `/cve-scan` for dependency CVEs. Both CI-ready with exit codes.
189
184
 
@@ -3,7 +3,7 @@
3
3
  "name": "ai-toolkit",
4
4
  "displayName": "AI Toolkit",
5
5
  "description": "Professional-grade engineering skills, agents, rules, and lifecycle guardrails for Claude Code, Claude Chat, and Cowork.",
6
- "version": "4.16.0",
6
+ "version": "4.17.0",
7
7
  "author": {
8
8
  "name": "SoftSpark",
9
9
  "url": "https://github.com/softspark"
@@ -3,7 +3,7 @@
3
3
  #
4
4
  # Fires on: SessionEnd
5
5
  # Matcher: all
6
- # The handoff snapshot is skipped for the minimal profile. Owned recovery
6
+ # The handoff snapshot is skipped for the minimal profile. Session-state
7
7
  # cleanup still runs for every profile.
8
8
 
9
9
  # shellcheck source=_session-paths.sh
@@ -18,18 +18,6 @@ if [ ! -t 0 ]; then
18
18
  INPUT=$(cat)
19
19
  fi
20
20
  SESSION_ID=$(hook_session_id)
21
- RECOVERY_ROOT="$SESSION_DIR/output-filter"
22
- OUTPUT_FILTER_CLI="${AI_TOOLKIT_OUTPUT_FILTER_CLI:-$HOME/.softspark/ai-toolkit/scripts/output_filter_cli.py}"
23
- if [ "$SESSION_ID" != "default" ] &&
24
- [ -d "$RECOVERY_ROOT" ] &&
25
- [ ! -L "$RECOVERY_ROOT" ] &&
26
- [ -f "$OUTPUT_FILTER_CLI" ] &&
27
- [ ! -L "$OUTPUT_FILTER_CLI" ] &&
28
- command -v python3 >/dev/null 2>&1; then
29
- python3 -S "$OUTPUT_FILTER_CLI" clean \
30
- --base-directory "$SESSION_DIR" \
31
- --session-id "$SESSION_ID" >/dev/null 2>&1 || true
32
- fi
33
21
 
34
22
  SESSION_STATE_CLI="${AI_TOOLKIT_SESSION_STATE_CLI:-$HOME/.softspark/ai-toolkit/scripts/session_state.py}"
35
23
  if [ "$SESSION_ID" != "default" ] &&
package/app/hooks.json CHANGED
@@ -158,16 +158,6 @@
158
158
  "command": "\"$HOME/.softspark/ai-toolkit/hooks/search-tracker.sh\""
159
159
  }
160
160
  ]
161
- },
162
- {
163
- "_source": "ai-toolkit",
164
- "matcher": "Bash",
165
- "hooks": [
166
- {
167
- "type": "command",
168
- "command": "\"$HOME/.softspark/ai-toolkit/hooks/filter-tool-output.sh\""
169
- }
170
- ]
171
161
  }
172
162
  ],
173
163
  "Stop": [
@@ -1,5 +1,5 @@
1
1
  {
2
- "last_run": "2026-07-24T03:52:58Z",
2
+ "last_run": "2026-07-26T11:36:57Z",
3
3
  "schema_version": 1,
4
4
  "tools": {
5
5
  "aider": {
@@ -24,7 +24,7 @@
24
24
  }
25
25
  },
26
26
  "augment": {
27
- "docs_hash": "388c0b5e25ab4906",
27
+ "docs_hash": "1dd9e9f490e7103a",
28
28
  "headings": [
29
29
  "Admin",
30
30
  "Auggie CLI",
@@ -66,7 +66,7 @@
66
66
  }
67
67
  },
68
68
  "claude-app": {
69
- "docs_hash": "3c7ff437aec0ab98",
69
+ "docs_hash": "9a5cb995316c7c71",
70
70
  "headings": [
71
71
  "Add global and folder instructions",
72
72
  "Availability",
@@ -107,7 +107,7 @@
107
107
  }
108
108
  },
109
109
  "claude-code": {
110
- "docs_hash": "a6ae93657498688c",
110
+ "docs_hash": "d373a3e4db1f4c79",
111
111
  "headings": [
112
112
  "Core concepts",
113
113
  "Documentation Index",
@@ -163,10 +163,10 @@
163
163
  "slash command": false,
164
164
  "sub-agent": true
165
165
  },
166
- "version": "2.1.218 (Claude Code)"
166
+ "version": "2.1.220 (Claude Code)"
167
167
  },
168
168
  "cline": {
169
- "docs_hash": "3eaa20e45d385a1f",
169
+ "docs_hash": "d76db24ff58cb7ea",
170
170
  "headings": [
171
171
  "API Reference",
172
172
  "Best Practices",
@@ -213,7 +213,7 @@
213
213
  }
214
214
  },
215
215
  "codex-cli": {
216
- "docs_hash": "87a5a879a39dc62c",
216
+ "docs_hash": "379d4d3760235c7f",
217
217
  "headings": [
218
218
  "API",
219
219
  "API Reference",
@@ -243,12 +243,10 @@
243
243
  "Connect tools and data",
244
244
  "Connection methods",
245
245
  "Contribute",
246
- "Conversion apps",
247
- "Core Concepts",
246
+ "Conversion specs",
248
247
  "Core concepts",
249
248
  "Cost and throughput",
250
249
  "Customization",
251
- "Deploy",
252
250
  "Deployment and model providers",
253
251
  "Desktop app",
254
252
  "Development workflows",
@@ -309,6 +307,7 @@
309
307
  "Start your first task",
310
308
  "Stay in control",
311
309
  "Suggested",
310
+ "Test and publish",
312
311
  "Text and code",
313
312
  "Third-party integrations",
314
313
  "Topics",
@@ -349,7 +348,7 @@
349
348
  "version": "codex-cli 0.145.0"
350
349
  },
351
350
  "cursor": {
352
- "docs_hash": "5387d04443d26da1",
351
+ "docs_hash": "74d1366066760874",
353
352
  "headings": [],
354
353
  "markers": {
355
354
  ".cursor/rules": false,
@@ -365,7 +364,7 @@
365
364
  }
366
365
  },
367
366
  "gemini-cli": {
368
- "docs_hash": "8db25b552ffbed80",
367
+ "docs_hash": "c6d93617f9359329",
369
368
  "headings": [
370
369
  "Breadcrumbs",
371
370
  "Directory actions",
@@ -408,7 +407,7 @@
408
407
  }
409
408
  },
410
409
  "github-copilot": {
411
- "docs_hash": "5d021df2f3b4d39b",
410
+ "docs_hash": "1c6e0fd930f573fa",
412
411
  "headings": [
413
412
  "About Copilot auto model selection",
414
413
  "About Copilot automations",
@@ -461,7 +460,7 @@
461
460
  }
462
461
  },
463
462
  "opencode": {
464
- "docs_hash": "f1dd69ffcfb280fa",
463
+ "docs_hash": "297813a7e2881afe",
465
464
  "headings": [
466
465
  "Add features",
467
466
  "Ask questions",
@@ -521,7 +520,7 @@
521
520
  }
522
521
  },
523
522
  "windsurf": {
524
- "docs_hash": "bee6ba0ee1a517dd",
523
+ "docs_hash": "d57d379406804a4b",
525
524
  "headings": [
526
525
  "Accounts",
527
526
  "Advanced",
package/bin/ai-toolkit.js CHANGED
@@ -50,7 +50,6 @@ const SCRIPT_COMMANDS = {
50
50
  'benchmark-ecosystem': { script: 'benchmark_ecosystem.py', toolkitCwd: true },
51
51
  'evaluate': { script: 'evaluate_skills.py', toolkitCwd: true },
52
52
  'stats': { script: 'stats.py' },
53
- 'output-filter': { script: 'output_filter_cli.py' },
54
53
  'compile-slm': { script: 'compile_slm.py' },
55
54
  'pack-codebase': { script: 'pack_codebase.py' },
56
55
  'claude-app': { script: 'claude_app.py', toolkitCwd: true },
@@ -80,7 +79,6 @@ const COMMANDS = {
80
79
  'benchmark-ecosystem': 'Generate ecosystem benchmark snapshot (GitHub metadata + offline fallback)',
81
80
  evaluate: 'Run skill evaluation suite',
82
81
  stats: 'Show skill usage statistics (--summary for product telemetry, --reset to clear)',
83
- 'output-filter': 'Manage native tool-output filter (status, inspect, recover, clean)',
84
82
  create: 'Scaffold new skill from template (e.g. create skill my-lint --template=linter)',
85
83
  mcp: 'Manage MCP templates and install native editor MCP configs',
86
84
  config: 'Manage config inheritance (validate, diff, init, create-base, check)',
@@ -0,0 +1,128 @@
1
+ ---
2
+ title: "Retirement: Native Tool-Output Filter — Measured 0% and Removed"
3
+ category: planning
4
+ service: ai-toolkit
5
+ tags:
6
+ - output-filter
7
+ - token-reduction
8
+ - postmortem
9
+ - measurement
10
+ - claude-code
11
+ doc_type: postmortem
12
+ status: completed
13
+ created: "2026-07-26"
14
+ last_updated: "2026-07-26"
15
+ shipped_in: "v4.17.0 (removal)"
16
+ description: "Why the native tool-output filter shipped in v4.16.0 was removed in v4.17.0: measured 0.0000% whole-session token saving on real traffic, because agent-issued commands are compound and the design accepted only simple registered shapes."
17
+ ---
18
+
19
+ # Retirement: Native Tool-Output Filter
20
+
21
+ **Shipped:** v4.16.0 (2026-07-23). **Removed:** v4.17.0 (2026-07-26).
22
+
23
+ ## The number
24
+
25
+ Measured whole-session input-token saving: **0.0000%**.
26
+
27
+ Real Bash results from local Claude Code transcripts were replayed through the
28
+ shipped classifier and the full filter registry. The filters ran on the actual
29
+ captured output; this is a measurement, not an estimate.
30
+
31
+ | Scope | Value |
32
+ |---|---:|
33
+ | Transcripts replayed | 134, across 22 distinct projects |
34
+ | Successful Bash results | 7600 |
35
+ | Of those, parsed as a simple command shape | 145 (1.9%) |
36
+ | Of those, matched a registered shape | 18 (0.24%) — 16 `git diff`, 2 `git show` |
37
+ | Accepted by any filter | **0** |
38
+ | Bytes saved | **0** |
39
+
40
+ The classifier was verified working before the result was accepted: `git
41
+ status`, `git log -n 20`, `pytest -v`, `bats --tap`, and `npm test` each
42
+ produced exactly one candidate. The zero is real.
43
+
44
+ ## Why: the premise, not the implementation
45
+
46
+ Seventeen filters were correct. They cleared their byte floors on owned
47
+ fixtures (44–95% reduction), stayed inside every latency budget at the 8 MiB
48
+ engine cap, passed adversarial safety review, and never once compressed a
49
+ failure. None of that mattered, because the commands they were built for are
50
+ not the commands that get issued.
51
+
52
+ 95% of successful Bash invocations are compound. The byte pool breaks down as:
53
+
54
+ | Class | Share of compound bytes | Why the filter refused it |
55
+ |---|---:|---|
56
+ | `;` chain | 44.3% | multiple output producers, attribution ambiguous |
57
+ | multiline script | 29.1% | rejected at the raw-string boundary |
58
+ | pipeline | 12.7% | the pipe transformed the output |
59
+ | `&&` chain with producing segments | 5.6% | multiple output producers |
60
+ | redirect, substitution | 4.4% | rejected at the raw-string boundary |
61
+ | heredoc | 3.5% | rejected at the raw-string boundary |
62
+
63
+ Every one of those refusals was the correct safety decision in isolation.
64
+ Together they excluded the entire population.
65
+
66
+ The most-frequent single shape was `cd <path> && …`, at 375 results and 480 KB.
67
+ A bounded `cd`-prefix subset had already been designed, threat-modelled, and
68
+ measured during Phase 3, and it was dropped because it covered 0.00% of the
69
+ compound pool. The retirement measurement confirms why: of those 375 results,
70
+ only 5 had a single simple second segment, and those 5 produced 0 bytes of
71
+ output.
72
+
73
+ ## What was already rejected on the way, and still stands
74
+
75
+ - **Read-result coverage: rejected on the `Edit` exact-match hazard.** On
76
+ 670.7 KB of real Read content, adjacent-duplicate collapse saves 0.00%,
77
+ blank-run collapse 0.04%, trailing-whitespace 0.00%. Anything above noise
78
+ requires elision, and 80.7% of `Edit` old-strings target a file read earlier
79
+ in the same session, 63.4% of them multi-line byte-exact quotes. A Read
80
+ result asserts what is on disk, so omission is a false claim rather than a
81
+ summary.
82
+ - **Compound-command subset: designed, measured at 0.00% coverage, dropped.**
83
+ The pipeline-truncator shape failed on an inversion: a truncated document
84
+ parses cleanly exactly where the shape would pay, and rejects exactly where
85
+ truncation is detectable.
86
+
87
+ ## The process lesson
88
+
89
+ The plan validated its **design** exhaustively across five phases and its
90
+ **premise** not at all until the fifth. Fixtures measured the filter; only real
91
+ traffic measured the value, and the two disagreed by two orders of magnitude.
92
+
93
+ The end-to-end replay that produced the 0% took under an hour and could have
94
+ run on day one, before any filter existed. Any future plan of this shape must
95
+ put premise validation in Phase 0, with a kill number published before the
96
+ measurement rather than argued after it.
97
+
98
+ ## Evaluated as a replacement: rtk
99
+
100
+ `rtk` (https://github.com/rtk-ai/rtk, Apache-2.0) rewrites commands at
101
+ `PreToolUse` rather than filtering output afterwards, which is the mechanism
102
+ this project's own safety contract had excluded. Its rewrite pipeline was
103
+ ported and validated against 197 of its own test assertions (197/197 exact
104
+ agreement), then applied to the same traffic:
105
+
106
+ - addresses **31.5%** of successful Bash bytes, **9.7%** of all tool-result
107
+ bytes — genuinely non-zero, so the in-house 0% was a coverage failure rather
108
+ than a law of nature;
109
+ - projected saving is **0.32–0.48%** of session input tokens on rtk's own
110
+ 60–90% claim, and **0.15–0.21%** once its filters' actual behaviour is
111
+ modelled;
112
+ - its two largest families here under-deliver: `rtk read` returns files
113
+ verbatim at the default `--level none`, and `rtk grep` models at 12.3%
114
+ against a claimed 75%;
115
+ - custom TOML filters, the documented extension point, would reach 1.91% of
116
+ Bash bytes. The large misses are structurally unreachable from config:
117
+ `| head` and `| tail` (34.6%) are blocked by the pipeline-final rule, and
118
+ `sed` (19.4%) sits in the hard-ignored prefix list.
119
+
120
+ Not adopted.
121
+
122
+ ## Where the tokens actually are
123
+
124
+ The measurement points somewhere other than command output. In this traffic,
125
+ `Read` is 53.8% of tool-result bytes, and within Bash the two largest buckets
126
+ are `sed` used as a file reader (16.2%) and `| head` / `| tail` pipeline tails
127
+ (34.6%). Those are file-reading patterns, not tool reports. Any future attempt
128
+ at token reduction should start there, and should start by measuring.
@@ -3,9 +3,9 @@ title: "AI Toolkit - Architecture Overview"
3
3
  category: reference
4
4
  service: ai-toolkit
5
5
  tags: [architecture, overview, design, structure]
6
- version: "1.5.0"
6
+ version: "1.6.0"
7
7
  created: "2026-03-23"
8
- last_updated: "2026-07-23"
8
+ last_updated: "2026-07-26"
9
9
  description: "Architecture of ai-toolkit: directory layout, Claude app export, global install model, editor-aware MCP install, Codex translation layer, skill tiers, and integration with projects."
10
10
  ---
11
11
 
@@ -300,7 +300,6 @@ Agents (code-reviewer, debugger, devops-implementer, ...)
300
300
  | UserPromptSubmit | Before user prompt execution | `track-usage.sh` | Record skill invocations to stats.json |
301
301
  | PostToolUse | After edit/write tools | `post-tool-use.sh` | Lightweight validation reminders |
302
302
  | PostToolUse | After any tool | `governance-capture.sh` | Log security-sensitive operations |
303
- | PostToolUse | After successful Bash | `filter-tool-output.sh` | Observe eligible text or replace it only after exact recovery succeeds; Claude Code only |
304
303
  | Stop | After response | `quality-check.sh` | Multi-language lint |
305
304
  | Stop | After response | `save-session.sh` | Persist session context |
306
305
  | Stop | Before final stop | `quality-gate.sh` | Block final response on lint/type errors |
@@ -2,9 +2,9 @@
2
2
  title: "CLI Reference"
3
3
  category: reference
4
4
  service: ai-toolkit
5
- tags: [cli, commands, reference, install, update, plugin, mcp, telemetry, output-filter]
5
+ tags: [cli, commands, reference, install, update, plugin, mcp, telemetry]
6
6
  created: "2026-04-13"
7
- last_updated: "2026-07-23"
7
+ last_updated: "2026-07-26"
8
8
  description: "Complete CLI reference for all ai-toolkit commands, options, and flags."
9
9
  ---
10
10
 
@@ -79,18 +79,8 @@ Usage: ai-toolkit <command> [options]
79
79
 
80
80
  | Command | Description |
81
81
  |---------|-------------|
82
- | `output-filter status [--policy PATH]` | Print the effective trusted project/global policy, or inspect an explicit policy |
83
- | `output-filter inspect --profile repeat-lines\|tap-success` | Read stdin and print content-free candidate metadata without changing or echoing the input |
84
- | `output-filter recover <handle> [--base-directory PATH] [--session-id ID]` | Print the exact stored native response object for an opaque recovery handle |
85
- | `output-filter clean [--base-directory PATH] [--session-id ID]` | Remove owned filter artifacts for one session or all sessions in the current repository |
86
- | `output-filter clean --session-id ID --expired` | Remove expired exact-response artifacts for one native session |
87
- | `output-filter hook --policy PATH` | **Internal, not for manual use.** `PostToolUse` entry point that reads the native hook payload on stdin and emits any replacement. The installed hook `filter-tool-output.sh` calls the same runtime through the lean `output_filter_hook.py` process instead |
88
-
89
- `off` is the default. Configure `toolOutputFilter.mode` in
90
- `.softspark-toolkit.json`, then run `install --local` or `update --local`.
82
+
91
83
  `AI_TOOLKIT_OUTPUT_FILTER_DISABLE=1` bypasses active filtering immediately.
92
- See [Native Tool Output Filter](tool-output-filter.md) for eligibility,
93
- privacy, recovery, and runtime support.
94
84
 
95
85
  ## Project Registry
96
86
 
@@ -10,7 +10,7 @@ tags:
10
10
  - multi-repo
11
11
  doc_type: reference
12
12
  created: "2026-04-11"
13
- last_updated: "2026-07-23"
13
+ last_updated: "2026-07-26"
14
14
  description: "Comprehensive guide for setting up and using ai-toolkit configuration inheritance. Covers base config creation, project setup, enforcement rules, CI integration, and troubleshooting."
15
15
  ---
16
16
 
@@ -77,7 +77,6 @@ ai-toolkit config check # CI enforcement check
77
77
  |-------|------|-------------|
78
78
  | `extends` | string | Base config source (npm, git URL, local path) |
79
79
  | `profile` | enum | `minimal`, `standard`, `strict`, `full`, `offline-slm` |
80
- | `toolOutputFilter` | object | Native Bash result filtering policy: mode, profiles, savings thresholds, and recovery limits |
81
80
  | `agents` | object | `enabled`, `disabled`, `custom` arrays |
82
81
  | `plugins` | object | Resolved `enabled` and `disabled` plugin intent |
83
82
  | `rules` | object | `inject`, `remove` arrays |
@@ -157,25 +156,6 @@ effective intent explicit and causes validation to reject a conflicting
157
156
  `plugins.disabled` entry. Install the named plugin pack separately with
158
157
  `ai-toolkit plugin install <name>` or your organization deployment workflow.
159
158
 
160
- ### Native tool-output filtering
161
-
162
- Projects can opt into the dependency-free output filter without changing the
163
- organization-wide default:
164
-
165
- ```json
166
- {
167
- "toolOutputFilter": {
168
- "mode": "observe",
169
- "profiles": ["repeat-lines", "tap-success"]
170
- }
171
- }
172
- ```
173
-
174
- `ai-toolkit install --local` materializes the effective policy as the managed
175
- `.claude/ai-toolkit-output-filter.json` file. `off` is the default, `observe`
176
- collects metadata without changing model-visible output, and `safe` permits
177
- replacement only when every invariant and exact-recovery check succeeds.
178
-
179
159
  ### Overrides
180
160
 
181
161
  Projects can override base settings, but must declare intent:
@@ -3,9 +3,9 @@ title: "Hooks Catalog"
3
3
  category: reference
4
4
  service: ai-toolkit
5
5
  tags: [hooks, quality, safety, enforcement, settings.json]
6
- version: "1.8.0"
6
+ version: "1.9.0"
7
7
  created: "2026-03-27"
8
- last_updated: "2026-07-23"
8
+ last_updated: "2026-07-26"
9
9
  description: "Complete reference of all ai-toolkit hooks: events, scripts, installation, and runtime behavior."
10
10
  ---
11
11
 
@@ -31,9 +31,6 @@ ai-toolkit update # re-copies scripts, re-merges (idempotent)
31
31
  **File locations:**
32
32
 
33
33
  - Scripts: `~/.softspark/ai-toolkit/hooks/*.sh`
34
- - Output-filter runtime: `~/.softspark/ai-toolkit/scripts/output_filter_hook.py`, `output_filter_cli.py`, and `tool_output_filter/`
35
- - Global output-filter policy: `~/.softspark/ai-toolkit/hooks/output-filter-policy.json`
36
- - Managed project policy: `<project>/.claude/ai-toolkit-output-filter.json`
37
34
  - Config: `~/.claude/settings.json` → `hooks` key
38
35
  - Source: `ai-toolkit/app/hooks/*.sh` + `app/hooks.json`
39
36
 
@@ -178,59 +175,6 @@ arms the corrective Stop hook.
178
175
 
179
176
  Skipped when `TOOLKIT_HOOK_PROFILE=minimal`.
180
177
 
181
- ### PostToolUse (native Bash output filter): `filter-tool-output.sh`
182
-
183
- | Field | Value |
184
- |-------|-------|
185
- | Event | `PostToolUse` |
186
- | Matcher | `Bash` |
187
- | Script | `~/.softspark/ai-toolkit/hooks/filter-tool-output.sh` |
188
- | Fires | After a successful Claude Code Bash tool call |
189
-
190
- The hook is disabled by default and runs last among the installed
191
- `PostToolUse` handlers. Its modes are:
192
-
193
- | Mode | Behavior |
194
- |------|----------|
195
- | `off` | Shell fast path exits before Python starts |
196
- | `observe` | Evaluates eligible output and writes content-free metadata, but emits no replacement |
197
- | `safe` | Replaces eligible output only after invariants pass and the exact native response is stored for recovery |
198
-
199
- Only explicitly allowlisted test, lint, typecheck, and validation command
200
- shapes are eligible. Failed or interrupted tools, non-empty stderr, image or
201
- binary results, pipes, redirects, deployment, migrations, audits, security
202
- scanners, malformed payloads, unavailable recovery, and any uncertain case
203
- remain unchanged.
204
-
205
- The per-project policy `<project>/.claude/ai-toolkit-output-filter.json` is
206
- honored only when **both** checks pass: the project root is registered in
207
- `~/.softspark/ai-toolkit/projects.json`, and the sibling regular file
208
- `<project>/.claude/.ai-toolkit-output-filter.owner` holds the ai-toolkit owner
209
- marker. `ai-toolkit install --local` writes both. Registration is required
210
- because the owner marker is a public constant, so a cloned or untrusted
211
- checkout must never be able to self-enable filtering by shipping its own
212
- marker. An unregistered project, a missing or foreign marker, or a symlinked
213
- project root or `.claude` directory falls back to the installed global policy
214
- at `~/.softspark/ai-toolkit/hooks/output-filter-policy.json`, which ships as
215
- `off`.
216
-
217
- `jq` is a required system dependency (`python3 scripts/check_deps.py`
218
- verifies it alongside `python3`, `git`, and `node`); without it `guard-path.sh`
219
- blocks file tools rather than skipping path validation.
220
-
221
- Recovery data is private and session-scoped under
222
- `~/.softspark/ai-toolkit/sessions/<repo-key>/output-filter/`. Telemetry records
223
- only profile/version, byte and line counts, latency, outcome, and fallback
224
- reason. Recovery-backed modes require a bounded native session ID containing
225
- only ASCII letters, digits, underscores, or hyphens. Three consecutive
226
- profile, invariant, or recovery safety failures open a session circuit
227
- breaker. Set `AI_TOOLKIT_OUTPUT_FILTER_DISABLE=1` for an immediate bypass.
228
-
229
- The replacement adapter is Claude Code-specific. Claude Chat/Cowork exports
230
- exclude it, and the generated hooks for other editors do not activate it.
231
- The `minimal` hook profile and `AI_TOOLKIT_DISABLED_HOOKS=filter-tool-output`
232
- bypass it before the Python runtime starts.
233
-
234
178
  ### Stop (quality check) — `quality-check.sh`
235
179
 
236
180
  | Field | Value |
@@ -344,8 +288,7 @@ Skipped when `TOOLKIT_HOOK_PROFILE=minimal`.
344
288
  | Script | `~/.softspark/ai-toolkit/hooks/session-end.sh` |
345
289
  | Fires | When a Claude session ends |
346
290
 
347
- **Action:** Removes private output-filter artifacts and isolated edit state for
348
- the ending native session, then writes `session-end.md` to the per-repo session store
291
+ **Action:** Removes isolated edit state for the ending native session, then writes `session-end.md` to the per-repo session store
349
292
  (`~/.softspark/ai-toolkit/sessions/<repo-key>/`) with a lightweight handoff note
350
293
  for the next session.
351
294