@softspark/ai-toolkit 4.16.0 → 4.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +41 -0
- package/README.md +11 -16
- package/app/.claude-plugin/plugin.json +1 -1
- package/app/hooks/session-end.sh +1 -13
- package/app/hooks.json +0 -10
- package/benchmarks/ecosystem-doctor-snapshot.json +14 -15
- package/bin/ai-toolkit.js +0 -2
- package/kb/history/completed/output-filter-retirement-20260726.md +128 -0
- package/kb/reference/architecture-overview.md +2 -3
- package/kb/reference/cli-reference.md +3 -13
- package/kb/reference/enterprise-config-guide.md +1 -21
- package/kb/reference/hooks-catalog.md +3 -60
- package/kb/reference/supported-tools-registry.md +0 -4
- package/llms-full.txt +143 -395
- package/llms.txt +1 -1
- package/manifest.json +147 -36
- package/package.json +1 -2
- package/scripts/claude_app.py +2 -21
- package/scripts/config_cli.py +4 -0
- package/scripts/config_merger.py +0 -17
- package/scripts/config_validator.py +11 -138
- package/scripts/doctor.py +3 -20
- package/scripts/generate_copilot.py +35 -4
- package/scripts/install.py +7 -2
- package/scripts/install_steps/ai_tools.py +28 -99
- package/scripts/install_steps/hooks.py +26 -24
- package/scripts/merge-hooks.py +33 -2
- package/scripts/output_filter_retirement.py +395 -0
- package/scripts/schemas/ai-toolkit-config.schema.json +0 -60
- package/scripts/uninstall.py +13 -27
- package/app/hooks/filter-tool-output.sh +0 -76
- package/app/output-filter-policy.json +0 -15
- package/benchmarks/output-filter/README.md +0 -11
- package/benchmarks/output-filter/scenarios.json +0 -25
- package/kb/reference/tool-output-filter.md +0 -288
- package/scripts/benchmark_output_filter.py +0 -343
- package/scripts/output_filter_cli.py +0 -347
- package/scripts/output_filter_hook.py +0 -23
- package/scripts/tool_output_filter/__init__.py +0 -33
- package/scripts/tool_output_filter/contracts.py +0 -173
- package/scripts/tool_output_filter/engine.py +0 -260
- package/scripts/tool_output_filter/hook_runtime.py +0 -369
- package/scripts/tool_output_filter/input.py +0 -56
- package/scripts/tool_output_filter/invariants.py +0 -40
- package/scripts/tool_output_filter/policy.py +0 -153
- package/scripts/tool_output_filter/profiles/__init__.py +0 -68
- package/scripts/tool_output_filter/profiles/repeat_lines.py +0 -71
- package/scripts/tool_output_filter/profiles/tap_success.py +0 -154
- package/scripts/tool_output_filter/recovery.py +0 -846
- package/scripts/tool_output_filter/telemetry.py +0 -13
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,47 @@ Versioning follows [Semantic Versioning](https://semver.org/).
|
|
|
7
7
|
|
|
8
8
|
---
|
|
9
9
|
|
|
10
|
+
## v4.17.0 — Native tool-output filter removed (2026-07-26)
|
|
11
|
+
|
|
12
|
+
### Removed
|
|
13
|
+
|
|
14
|
+
- **BREAKING for anyone who opted in.** The native tool-output filter shipped
|
|
15
|
+
in v4.16.0 is gone: the `PostToolUse` hook `filter-tool-output.sh`, the
|
|
16
|
+
`scripts/tool_output_filter/` runtime, the `ai-toolkit output-filter` CLI,
|
|
17
|
+
the benchmark corpus, and the `toolOutputFilter` key in
|
|
18
|
+
`.softspark-toolkit.json`. The key is accepted and ignored rather than
|
|
19
|
+
rejected, so an existing config still validates; `ai-toolkit config validate`
|
|
20
|
+
reports it as a retired key you can delete.
|
|
21
|
+
- Why: measured whole-session token saving was **0.0000%** on real traffic.
|
|
22
|
+
Across 134 session transcripts spanning 22 projects, 7600 successful Bash
|
|
23
|
+
results contained 145 that parsed as simple command shapes, 18 that matched a
|
|
24
|
+
registered shape, and 0 that any filter accepted. The filter was correct;
|
|
25
|
+
its premise was not. Agent-issued commands are overwhelmingly compound
|
|
26
|
+
(`&&` and `;` chains, pipelines, heredocs), and the design accepted only
|
|
27
|
+
simple registered shapes. See
|
|
28
|
+
`kb/history/completed/output-filter-retirement-20260726.md`.
|
|
29
|
+
|
|
30
|
+
### Changed
|
|
31
|
+
|
|
32
|
+
- `ai-toolkit install` and `ai-toolkit update` now remove the artifacts left by
|
|
33
|
+
v4.16.x: the installed hook script, the global
|
|
34
|
+
`hooks/output-filter-policy.json`, the managed project policy and owner
|
|
35
|
+
marker under `.claude/`, and stored recovery trees under
|
|
36
|
+
`sessions/<repo-key>/output-filter/`. Cleanup is idempotent, verifies
|
|
37
|
+
ai-toolkit ownership before deleting, and leaves foreign files untouched.
|
|
38
|
+
- The stale `PostToolUse` entry is removed from `~/.claude/settings.json` by
|
|
39
|
+
the existing hook strip-and-remerge cycle, so no manual edit is needed.
|
|
40
|
+
- Hook counts drop from 29 entries to 28 across the same 14 events.
|
|
41
|
+
|
|
42
|
+
## v4.16.1 — Copilot skill remnant recovery (2026-07-24)
|
|
43
|
+
|
|
44
|
+
### Fixed
|
|
45
|
+
|
|
46
|
+
- **Legacy Copilot skill remnants no longer abort local updates** — `.github/skills/ai-toolkit-*` directories left behind by pre-manifest cleanups (assets without `SKILL.md`) are rebuilt in place instead of raising `Refusing user-owned Copilot skill collision`, which aborted the entire `install --local` / `update` run on affected projects. User files inside a remnant that do not collide with generated output are preserved; any directory that still contains a `SKILL.md` stays protected and is never reclaimed.
|
|
47
|
+
- **Readable installer errors** — `scripts/install.py` reports fail-closed `RuntimeError` conditions as a single `ERROR:` line instead of a raw traceback.
|
|
48
|
+
|
|
49
|
+
Test count: 1477 → 1479.
|
|
50
|
+
|
|
10
51
|
## v4.16.0 — Native tool-output filtering (2026-07-24)
|
|
11
52
|
|
|
12
53
|
Minor release. Adds a dependency-free, opt-in native tool-output filter for Claude Code and hardens the enterprise configuration, session-isolation, and editor-adapter surfaces around it. The filter ships disabled; `observe` and `safe` are per-project opt-ins. No source catalog count change (44 agents, 108 skills).
|
package/README.md
CHANGED
|
@@ -6,17 +6,17 @@
|
|
|
6
6
|
[](LICENSE)
|
|
7
7
|
[](app/skills/)
|
|
8
8
|
[](app/agents/)
|
|
9
|
-
[](tests/)
|
|
10
10
|
|
|
11
|
-
## What's New in v4.
|
|
11
|
+
## What's New in v4.17.0
|
|
12
12
|
|
|
13
|
-
v4.
|
|
13
|
+
v4.17.0 removes the native tool-output filter shipped in v4.16.0.
|
|
14
14
|
|
|
15
|
-
- **Native tool-output filter**:
|
|
16
|
-
- **
|
|
17
|
-
- **Output-filter CLI**: `ai-toolkit output-filter status|inspect|recover|clean` plus a deterministic benchmark for the built-in profiles.
|
|
15
|
+
- **Native tool-output filter removed**: measured whole-session token saving was 0.0000% on real traffic, because agent-issued commands are overwhelmingly compound and the filter only accepted simple registered shapes. The `PostToolUse` hook, the CLI, and the `toolOutputFilter` config key are gone.
|
|
16
|
+
- **Automatic cleanup on update**: `ai-toolkit update` removes the orphaned hook script, the global and project policy files, and any stored recovery artifacts left by v4.16.x. Foreign files in those directories are preserved.
|
|
18
17
|
- **Stricter config validation**: unknown top-level config keys are rejected, plugin manifests must declare `requires`, and lock files go stale on toolkit version bumps.
|
|
19
18
|
- **GitHub Copilot compatibility reference**: new `kb/reference/copilot-compatibility.md` documenting the Copilot integration surface.
|
|
19
|
+
- **Copilot remnant recovery**: asset-only `.github/skills/ai-toolkit-*` leftovers from older cleanups are rebuilt in place instead of aborting `install --local` / `update`.
|
|
20
20
|
|
|
21
21
|
See [CHANGELOG.md](CHANGELOG.md) for full history.
|
|
22
22
|
|
|
@@ -91,10 +91,7 @@ ai-toolkit claude-app export --verify
|
|
|
91
91
|
```
|
|
92
92
|
|
|
93
93
|
Re-export and re-upload after toolkit or registered-rule updates. Skills work
|
|
94
|
-
in Chat and Cowork; hooks and sub-agents are active only in Cowork.
|
|
95
|
-
tool-output replacement hook remains Claude Code-only and is excluded from the
|
|
96
|
-
Claude app archive until that runtime has an independently verified replacement
|
|
97
|
-
contract.
|
|
94
|
+
in Chat and Cowork; hooks and sub-agents are active only in Cowork.
|
|
98
95
|
|
|
99
96
|
### Install Profiles
|
|
100
97
|
|
|
@@ -126,7 +123,7 @@ See [CLI Reference](kb/reference/cli-reference.md) for all commands and options.
|
|
|
126
123
|
| Gemini CLI | `~/.gemini/GEMINI.md` | ✅ | global |
|
|
127
124
|
| GitHub Copilot | Project: `AGENTS.md` + `.github/copilot-instructions.md` + `.github/{instructions,prompts,agents,skills,hooks}/` + `.github/mcp.json`; user: `$COPILOT_HOME/copilot-instructions.md` + `$COPILOT_HOME/{instructions,agents,skills,hooks}/` + `$COPILOT_HOME/mcp-config.json` | ✅ | project + user |
|
|
128
125
|
| Cline | `~/Documents/Cline/Rules/*.md` + `~/.cline/skills/*` + `.clinerules/*.md` | — | global + project |
|
|
129
|
-
| Roo Code | `~/.roo/rules/*.md` + `.roomodes` + `.roo/rules/*.md` | — | global rules + project |
|
|
126
|
+
| Roo Code / [Zoo Code](https://zoocode.dev/) | `~/.roo/rules/*.md` + `.roomodes` + `.roo/rules/*.md` | — | global rules + project |
|
|
130
127
|
| Aider | `~/.aider.conf.yml` + `.aider.conf.yml` + `CONVENTIONS.md` | — | global + project |
|
|
131
128
|
| Augment | `~/.augment/rules/*.md` + `.augment/rules/ai-toolkit-*.md` | ✅ | global + project |
|
|
132
129
|
| Google Antigravity | `.agents/rules/*.md` + `.agents/workflows/*.md` + skill pointer in `.agent/skills/*` (IDE) and `.agents/skills/*` (CLI) | — | project |
|
|
@@ -145,7 +142,7 @@ See [CLI Reference](kb/reference/cli-reference.md) for all commands and options.
|
|
|
145
142
|
| `skills/` (hybrid) | 30 | Slash commands with agent knowledge base |
|
|
146
143
|
| `skills/` (knowledge) | 46 | Domain knowledge auto-loaded by agents (includes 13 `<lang>-rules` skills) |
|
|
147
144
|
| `agents/` | 44 | Specialized agents across 10 categories |
|
|
148
|
-
| `hooks/` |
|
|
145
|
+
| `hooks/` | 28 entries / 14 events + statusLine | Quality gates, path safety, prompt governance, loop guard, session lifecycle |
|
|
149
146
|
| `plugins/` | 11 packs | Opt-in domain bundles (security, research, frontend, enterprise, 6 language packs) |
|
|
150
147
|
| `constitution.md` | 7 articles | Machine-enforced safety rules |
|
|
151
148
|
| `rules/` | auto-synced | Global/project rule files for Claude and other editors |
|
|
@@ -169,7 +166,7 @@ ai-toolkit/
|
|
|
169
166
|
│ └── ARCHITECTURE.md # Full system design
|
|
170
167
|
├── kb/ # Reference docs, procedures, plans
|
|
171
168
|
├── scripts/ # Validation, install, evaluation scripts
|
|
172
|
-
├── tests/ # Bats and Python test suite (
|
|
169
|
+
├── tests/ # Bats and Python test suite (1445 tests)
|
|
173
170
|
└── CHANGELOG.md
|
|
174
171
|
```
|
|
175
172
|
|
|
@@ -181,9 +178,7 @@ ai-toolkit/
|
|
|
181
178
|
|
|
182
179
|
**Machine-enforced constitution** — 7-article safety constitution enforced via `PreToolUse` hooks that actually block `rm -rf`, `DROP TABLE`, and irreversible operations. Not just documentation.
|
|
183
180
|
|
|
184
|
-
**
|
|
185
|
-
|
|
186
|
-
**Native tool-output filtering** — dependency-free, post-execution filtering for successful Claude Code Bash output, shipped **disabled by default** (`off`). Neither `off` nor `observe` ever replaces output; opt-in `safe` mode replaces only validated `repeat-lines` or `tap-success` results after exact ephemeral recovery is available. Failures, diagnostics, unsupported payloads, and unavailable recovery always pass through unchanged. See [Tool Output Filter](kb/reference/tool-output-filter.md).
|
|
181
|
+
**28 lifecycle hook entries:** Executable handlers across 14 events (SessionStart → SessionEnd, plus InstructionsLoaded + ConfigChange). Guards, governance, quality gates, session persistence, MCP health checks, revert protection, test-cohesion enforcement, loop guard, and search-first discipline. See [Hooks Catalog](kb/reference/hooks-catalog.md).
|
|
187
182
|
|
|
188
183
|
**Security scanning** — `/skill-audit` for code-level risks, `/cve-scan` for dependency CVEs. Both CI-ready with exit codes.
|
|
189
184
|
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
"name": "ai-toolkit",
|
|
4
4
|
"displayName": "AI Toolkit",
|
|
5
5
|
"description": "Professional-grade engineering skills, agents, rules, and lifecycle guardrails for Claude Code, Claude Chat, and Cowork.",
|
|
6
|
-
"version": "4.
|
|
6
|
+
"version": "4.17.0",
|
|
7
7
|
"author": {
|
|
8
8
|
"name": "SoftSpark",
|
|
9
9
|
"url": "https://github.com/softspark"
|
package/app/hooks/session-end.sh
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
#
|
|
4
4
|
# Fires on: SessionEnd
|
|
5
5
|
# Matcher: all
|
|
6
|
-
# The handoff snapshot is skipped for the minimal profile.
|
|
6
|
+
# The handoff snapshot is skipped for the minimal profile. Session-state
|
|
7
7
|
# cleanup still runs for every profile.
|
|
8
8
|
|
|
9
9
|
# shellcheck source=_session-paths.sh
|
|
@@ -18,18 +18,6 @@ if [ ! -t 0 ]; then
|
|
|
18
18
|
INPUT=$(cat)
|
|
19
19
|
fi
|
|
20
20
|
SESSION_ID=$(hook_session_id)
|
|
21
|
-
RECOVERY_ROOT="$SESSION_DIR/output-filter"
|
|
22
|
-
OUTPUT_FILTER_CLI="${AI_TOOLKIT_OUTPUT_FILTER_CLI:-$HOME/.softspark/ai-toolkit/scripts/output_filter_cli.py}"
|
|
23
|
-
if [ "$SESSION_ID" != "default" ] &&
|
|
24
|
-
[ -d "$RECOVERY_ROOT" ] &&
|
|
25
|
-
[ ! -L "$RECOVERY_ROOT" ] &&
|
|
26
|
-
[ -f "$OUTPUT_FILTER_CLI" ] &&
|
|
27
|
-
[ ! -L "$OUTPUT_FILTER_CLI" ] &&
|
|
28
|
-
command -v python3 >/dev/null 2>&1; then
|
|
29
|
-
python3 -S "$OUTPUT_FILTER_CLI" clean \
|
|
30
|
-
--base-directory "$SESSION_DIR" \
|
|
31
|
-
--session-id "$SESSION_ID" >/dev/null 2>&1 || true
|
|
32
|
-
fi
|
|
33
21
|
|
|
34
22
|
SESSION_STATE_CLI="${AI_TOOLKIT_SESSION_STATE_CLI:-$HOME/.softspark/ai-toolkit/scripts/session_state.py}"
|
|
35
23
|
if [ "$SESSION_ID" != "default" ] &&
|
package/app/hooks.json
CHANGED
|
@@ -158,16 +158,6 @@
|
|
|
158
158
|
"command": "\"$HOME/.softspark/ai-toolkit/hooks/search-tracker.sh\""
|
|
159
159
|
}
|
|
160
160
|
]
|
|
161
|
-
},
|
|
162
|
-
{
|
|
163
|
-
"_source": "ai-toolkit",
|
|
164
|
-
"matcher": "Bash",
|
|
165
|
-
"hooks": [
|
|
166
|
-
{
|
|
167
|
-
"type": "command",
|
|
168
|
-
"command": "\"$HOME/.softspark/ai-toolkit/hooks/filter-tool-output.sh\""
|
|
169
|
-
}
|
|
170
|
-
]
|
|
171
161
|
}
|
|
172
162
|
],
|
|
173
163
|
"Stop": [
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
"last_run": "2026-07-
|
|
2
|
+
"last_run": "2026-07-26T11:36:57Z",
|
|
3
3
|
"schema_version": 1,
|
|
4
4
|
"tools": {
|
|
5
5
|
"aider": {
|
|
@@ -24,7 +24,7 @@
|
|
|
24
24
|
}
|
|
25
25
|
},
|
|
26
26
|
"augment": {
|
|
27
|
-
"docs_hash": "
|
|
27
|
+
"docs_hash": "1dd9e9f490e7103a",
|
|
28
28
|
"headings": [
|
|
29
29
|
"Admin",
|
|
30
30
|
"Auggie CLI",
|
|
@@ -66,7 +66,7 @@
|
|
|
66
66
|
}
|
|
67
67
|
},
|
|
68
68
|
"claude-app": {
|
|
69
|
-
"docs_hash": "
|
|
69
|
+
"docs_hash": "9a5cb995316c7c71",
|
|
70
70
|
"headings": [
|
|
71
71
|
"Add global and folder instructions",
|
|
72
72
|
"Availability",
|
|
@@ -107,7 +107,7 @@
|
|
|
107
107
|
}
|
|
108
108
|
},
|
|
109
109
|
"claude-code": {
|
|
110
|
-
"docs_hash": "
|
|
110
|
+
"docs_hash": "d373a3e4db1f4c79",
|
|
111
111
|
"headings": [
|
|
112
112
|
"Core concepts",
|
|
113
113
|
"Documentation Index",
|
|
@@ -163,10 +163,10 @@
|
|
|
163
163
|
"slash command": false,
|
|
164
164
|
"sub-agent": true
|
|
165
165
|
},
|
|
166
|
-
"version": "2.1.
|
|
166
|
+
"version": "2.1.220 (Claude Code)"
|
|
167
167
|
},
|
|
168
168
|
"cline": {
|
|
169
|
-
"docs_hash": "
|
|
169
|
+
"docs_hash": "d76db24ff58cb7ea",
|
|
170
170
|
"headings": [
|
|
171
171
|
"API Reference",
|
|
172
172
|
"Best Practices",
|
|
@@ -213,7 +213,7 @@
|
|
|
213
213
|
}
|
|
214
214
|
},
|
|
215
215
|
"codex-cli": {
|
|
216
|
-
"docs_hash": "
|
|
216
|
+
"docs_hash": "379d4d3760235c7f",
|
|
217
217
|
"headings": [
|
|
218
218
|
"API",
|
|
219
219
|
"API Reference",
|
|
@@ -243,12 +243,10 @@
|
|
|
243
243
|
"Connect tools and data",
|
|
244
244
|
"Connection methods",
|
|
245
245
|
"Contribute",
|
|
246
|
-
"Conversion
|
|
247
|
-
"Core Concepts",
|
|
246
|
+
"Conversion specs",
|
|
248
247
|
"Core concepts",
|
|
249
248
|
"Cost and throughput",
|
|
250
249
|
"Customization",
|
|
251
|
-
"Deploy",
|
|
252
250
|
"Deployment and model providers",
|
|
253
251
|
"Desktop app",
|
|
254
252
|
"Development workflows",
|
|
@@ -309,6 +307,7 @@
|
|
|
309
307
|
"Start your first task",
|
|
310
308
|
"Stay in control",
|
|
311
309
|
"Suggested",
|
|
310
|
+
"Test and publish",
|
|
312
311
|
"Text and code",
|
|
313
312
|
"Third-party integrations",
|
|
314
313
|
"Topics",
|
|
@@ -349,7 +348,7 @@
|
|
|
349
348
|
"version": "codex-cli 0.145.0"
|
|
350
349
|
},
|
|
351
350
|
"cursor": {
|
|
352
|
-
"docs_hash": "
|
|
351
|
+
"docs_hash": "74d1366066760874",
|
|
353
352
|
"headings": [],
|
|
354
353
|
"markers": {
|
|
355
354
|
".cursor/rules": false,
|
|
@@ -365,7 +364,7 @@
|
|
|
365
364
|
}
|
|
366
365
|
},
|
|
367
366
|
"gemini-cli": {
|
|
368
|
-
"docs_hash": "
|
|
367
|
+
"docs_hash": "c6d93617f9359329",
|
|
369
368
|
"headings": [
|
|
370
369
|
"Breadcrumbs",
|
|
371
370
|
"Directory actions",
|
|
@@ -408,7 +407,7 @@
|
|
|
408
407
|
}
|
|
409
408
|
},
|
|
410
409
|
"github-copilot": {
|
|
411
|
-
"docs_hash": "
|
|
410
|
+
"docs_hash": "1c6e0fd930f573fa",
|
|
412
411
|
"headings": [
|
|
413
412
|
"About Copilot auto model selection",
|
|
414
413
|
"About Copilot automations",
|
|
@@ -461,7 +460,7 @@
|
|
|
461
460
|
}
|
|
462
461
|
},
|
|
463
462
|
"opencode": {
|
|
464
|
-
"docs_hash": "
|
|
463
|
+
"docs_hash": "297813a7e2881afe",
|
|
465
464
|
"headings": [
|
|
466
465
|
"Add features",
|
|
467
466
|
"Ask questions",
|
|
@@ -521,7 +520,7 @@
|
|
|
521
520
|
}
|
|
522
521
|
},
|
|
523
522
|
"windsurf": {
|
|
524
|
-
"docs_hash": "
|
|
523
|
+
"docs_hash": "d57d379406804a4b",
|
|
525
524
|
"headings": [
|
|
526
525
|
"Accounts",
|
|
527
526
|
"Advanced",
|
package/bin/ai-toolkit.js
CHANGED
|
@@ -50,7 +50,6 @@ const SCRIPT_COMMANDS = {
|
|
|
50
50
|
'benchmark-ecosystem': { script: 'benchmark_ecosystem.py', toolkitCwd: true },
|
|
51
51
|
'evaluate': { script: 'evaluate_skills.py', toolkitCwd: true },
|
|
52
52
|
'stats': { script: 'stats.py' },
|
|
53
|
-
'output-filter': { script: 'output_filter_cli.py' },
|
|
54
53
|
'compile-slm': { script: 'compile_slm.py' },
|
|
55
54
|
'pack-codebase': { script: 'pack_codebase.py' },
|
|
56
55
|
'claude-app': { script: 'claude_app.py', toolkitCwd: true },
|
|
@@ -80,7 +79,6 @@ const COMMANDS = {
|
|
|
80
79
|
'benchmark-ecosystem': 'Generate ecosystem benchmark snapshot (GitHub metadata + offline fallback)',
|
|
81
80
|
evaluate: 'Run skill evaluation suite',
|
|
82
81
|
stats: 'Show skill usage statistics (--summary for product telemetry, --reset to clear)',
|
|
83
|
-
'output-filter': 'Manage native tool-output filter (status, inspect, recover, clean)',
|
|
84
82
|
create: 'Scaffold new skill from template (e.g. create skill my-lint --template=linter)',
|
|
85
83
|
mcp: 'Manage MCP templates and install native editor MCP configs',
|
|
86
84
|
config: 'Manage config inheritance (validate, diff, init, create-base, check)',
|
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Retirement: Native Tool-Output Filter — Measured 0% and Removed"
|
|
3
|
+
category: planning
|
|
4
|
+
service: ai-toolkit
|
|
5
|
+
tags:
|
|
6
|
+
- output-filter
|
|
7
|
+
- token-reduction
|
|
8
|
+
- postmortem
|
|
9
|
+
- measurement
|
|
10
|
+
- claude-code
|
|
11
|
+
doc_type: postmortem
|
|
12
|
+
status: completed
|
|
13
|
+
created: "2026-07-26"
|
|
14
|
+
last_updated: "2026-07-26"
|
|
15
|
+
shipped_in: "v4.17.0 (removal)"
|
|
16
|
+
description: "Why the native tool-output filter shipped in v4.16.0 was removed in v4.17.0: measured 0.0000% whole-session token saving on real traffic, because agent-issued commands are compound and the design accepted only simple registered shapes."
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
# Retirement: Native Tool-Output Filter
|
|
20
|
+
|
|
21
|
+
**Shipped:** v4.16.0 (2026-07-23). **Removed:** v4.17.0 (2026-07-26).
|
|
22
|
+
|
|
23
|
+
## The number
|
|
24
|
+
|
|
25
|
+
Measured whole-session input-token saving: **0.0000%**.
|
|
26
|
+
|
|
27
|
+
Real Bash results from local Claude Code transcripts were replayed through the
|
|
28
|
+
shipped classifier and the full filter registry. The filters ran on the actual
|
|
29
|
+
captured output; this is a measurement, not an estimate.
|
|
30
|
+
|
|
31
|
+
| Scope | Value |
|
|
32
|
+
|---|---:|
|
|
33
|
+
| Transcripts replayed | 134, across 22 distinct projects |
|
|
34
|
+
| Successful Bash results | 7600 |
|
|
35
|
+
| Of those, parsed as a simple command shape | 145 (1.9%) |
|
|
36
|
+
| Of those, matched a registered shape | 18 (0.24%) — 16 `git diff`, 2 `git show` |
|
|
37
|
+
| Accepted by any filter | **0** |
|
|
38
|
+
| Bytes saved | **0** |
|
|
39
|
+
|
|
40
|
+
The classifier was verified working before the result was accepted: `git
|
|
41
|
+
status`, `git log -n 20`, `pytest -v`, `bats --tap`, and `npm test` each
|
|
42
|
+
produced exactly one candidate. The zero is real.
|
|
43
|
+
|
|
44
|
+
## Why: the premise, not the implementation
|
|
45
|
+
|
|
46
|
+
Seventeen filters were correct. They cleared their byte floors on owned
|
|
47
|
+
fixtures (44–95% reduction), stayed inside every latency budget at the 8 MiB
|
|
48
|
+
engine cap, passed adversarial safety review, and never once compressed a
|
|
49
|
+
failure. None of that mattered, because the commands they were built for are
|
|
50
|
+
not the commands that get issued.
|
|
51
|
+
|
|
52
|
+
95% of successful Bash invocations are compound. The byte pool breaks down as:
|
|
53
|
+
|
|
54
|
+
| Class | Share of compound bytes | Why the filter refused it |
|
|
55
|
+
|---|---:|---|
|
|
56
|
+
| `;` chain | 44.3% | multiple output producers, attribution ambiguous |
|
|
57
|
+
| multiline script | 29.1% | rejected at the raw-string boundary |
|
|
58
|
+
| pipeline | 12.7% | the pipe transformed the output |
|
|
59
|
+
| `&&` chain with producing segments | 5.6% | multiple output producers |
|
|
60
|
+
| redirect, substitution | 4.4% | rejected at the raw-string boundary |
|
|
61
|
+
| heredoc | 3.5% | rejected at the raw-string boundary |
|
|
62
|
+
|
|
63
|
+
Every one of those refusals was the correct safety decision in isolation.
|
|
64
|
+
Together they excluded the entire population.
|
|
65
|
+
|
|
66
|
+
The most-frequent single shape was `cd <path> && …`, at 375 results and 480 KB.
|
|
67
|
+
A bounded `cd`-prefix subset had already been designed, threat-modelled, and
|
|
68
|
+
measured during Phase 3, and it was dropped because it covered 0.00% of the
|
|
69
|
+
compound pool. The retirement measurement confirms why: of those 375 results,
|
|
70
|
+
only 5 had a single simple second segment, and those 5 produced 0 bytes of
|
|
71
|
+
output.
|
|
72
|
+
|
|
73
|
+
## What was already rejected on the way, and still stands
|
|
74
|
+
|
|
75
|
+
- **Read-result coverage: rejected on the `Edit` exact-match hazard.** On
|
|
76
|
+
670.7 KB of real Read content, adjacent-duplicate collapse saves 0.00%,
|
|
77
|
+
blank-run collapse 0.04%, trailing-whitespace 0.00%. Anything above noise
|
|
78
|
+
requires elision, and 80.7% of `Edit` old-strings target a file read earlier
|
|
79
|
+
in the same session, 63.4% of them multi-line byte-exact quotes. A Read
|
|
80
|
+
result asserts what is on disk, so omission is a false claim rather than a
|
|
81
|
+
summary.
|
|
82
|
+
- **Compound-command subset: designed, measured at 0.00% coverage, dropped.**
|
|
83
|
+
The pipeline-truncator shape failed on an inversion: a truncated document
|
|
84
|
+
parses cleanly exactly where the shape would pay, and rejects exactly where
|
|
85
|
+
truncation is detectable.
|
|
86
|
+
|
|
87
|
+
## The process lesson
|
|
88
|
+
|
|
89
|
+
The plan validated its **design** exhaustively across five phases and its
|
|
90
|
+
**premise** not at all until the fifth. Fixtures measured the filter; only real
|
|
91
|
+
traffic measured the value, and the two disagreed by two orders of magnitude.
|
|
92
|
+
|
|
93
|
+
The end-to-end replay that produced the 0% took under an hour and could have
|
|
94
|
+
run on day one, before any filter existed. Any future plan of this shape must
|
|
95
|
+
put premise validation in Phase 0, with a kill number published before the
|
|
96
|
+
measurement rather than argued after it.
|
|
97
|
+
|
|
98
|
+
## Evaluated as a replacement: rtk
|
|
99
|
+
|
|
100
|
+
`rtk` (https://github.com/rtk-ai/rtk, Apache-2.0) rewrites commands at
|
|
101
|
+
`PreToolUse` rather than filtering output afterwards, which is the mechanism
|
|
102
|
+
this project's own safety contract had excluded. Its rewrite pipeline was
|
|
103
|
+
ported and validated against 197 of its own test assertions (197/197 exact
|
|
104
|
+
agreement), then applied to the same traffic:
|
|
105
|
+
|
|
106
|
+
- addresses **31.5%** of successful Bash bytes, **9.7%** of all tool-result
|
|
107
|
+
bytes — genuinely non-zero, so the in-house 0% was a coverage failure rather
|
|
108
|
+
than a law of nature;
|
|
109
|
+
- projected saving is **0.32–0.48%** of session input tokens on rtk's own
|
|
110
|
+
60–90% claim, and **0.15–0.21%** once its filters' actual behaviour is
|
|
111
|
+
modelled;
|
|
112
|
+
- its two largest families here under-deliver: `rtk read` returns files
|
|
113
|
+
verbatim at the default `--level none`, and `rtk grep` models at 12.3%
|
|
114
|
+
against a claimed 75%;
|
|
115
|
+
- custom TOML filters, the documented extension point, would reach 1.91% of
|
|
116
|
+
Bash bytes. The large misses are structurally unreachable from config:
|
|
117
|
+
`| head` and `| tail` (34.6%) are blocked by the pipeline-final rule, and
|
|
118
|
+
`sed` (19.4%) sits in the hard-ignored prefix list.
|
|
119
|
+
|
|
120
|
+
Not adopted.
|
|
121
|
+
|
|
122
|
+
## Where the tokens actually are
|
|
123
|
+
|
|
124
|
+
The measurement points somewhere other than command output. In this traffic,
|
|
125
|
+
`Read` is 53.8% of tool-result bytes, and within Bash the two largest buckets
|
|
126
|
+
are `sed` used as a file reader (16.2%) and `| head` / `| tail` pipeline tails
|
|
127
|
+
(34.6%). Those are file-reading patterns, not tool reports. Any future attempt
|
|
128
|
+
at token reduction should start there, and should start by measuring.
|
|
@@ -3,9 +3,9 @@ title: "AI Toolkit - Architecture Overview"
|
|
|
3
3
|
category: reference
|
|
4
4
|
service: ai-toolkit
|
|
5
5
|
tags: [architecture, overview, design, structure]
|
|
6
|
-
version: "1.
|
|
6
|
+
version: "1.6.0"
|
|
7
7
|
created: "2026-03-23"
|
|
8
|
-
last_updated: "2026-07-
|
|
8
|
+
last_updated: "2026-07-26"
|
|
9
9
|
description: "Architecture of ai-toolkit: directory layout, Claude app export, global install model, editor-aware MCP install, Codex translation layer, skill tiers, and integration with projects."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -300,7 +300,6 @@ Agents (code-reviewer, debugger, devops-implementer, ...)
|
|
|
300
300
|
| UserPromptSubmit | Before user prompt execution | `track-usage.sh` | Record skill invocations to stats.json |
|
|
301
301
|
| PostToolUse | After edit/write tools | `post-tool-use.sh` | Lightweight validation reminders |
|
|
302
302
|
| PostToolUse | After any tool | `governance-capture.sh` | Log security-sensitive operations |
|
|
303
|
-
| PostToolUse | After successful Bash | `filter-tool-output.sh` | Observe eligible text or replace it only after exact recovery succeeds; Claude Code only |
|
|
304
303
|
| Stop | After response | `quality-check.sh` | Multi-language lint |
|
|
305
304
|
| Stop | After response | `save-session.sh` | Persist session context |
|
|
306
305
|
| Stop | Before final stop | `quality-gate.sh` | Block final response on lint/type errors |
|
|
@@ -2,9 +2,9 @@
|
|
|
2
2
|
title: "CLI Reference"
|
|
3
3
|
category: reference
|
|
4
4
|
service: ai-toolkit
|
|
5
|
-
tags: [cli, commands, reference, install, update, plugin, mcp, telemetry
|
|
5
|
+
tags: [cli, commands, reference, install, update, plugin, mcp, telemetry]
|
|
6
6
|
created: "2026-04-13"
|
|
7
|
-
last_updated: "2026-07-
|
|
7
|
+
last_updated: "2026-07-26"
|
|
8
8
|
description: "Complete CLI reference for all ai-toolkit commands, options, and flags."
|
|
9
9
|
---
|
|
10
10
|
|
|
@@ -79,18 +79,8 @@ Usage: ai-toolkit <command> [options]
|
|
|
79
79
|
|
|
80
80
|
| Command | Description |
|
|
81
81
|
|---------|-------------|
|
|
82
|
-
|
|
83
|
-
| `output-filter inspect --profile repeat-lines\|tap-success` | Read stdin and print content-free candidate metadata without changing or echoing the input |
|
|
84
|
-
| `output-filter recover <handle> [--base-directory PATH] [--session-id ID]` | Print the exact stored native response object for an opaque recovery handle |
|
|
85
|
-
| `output-filter clean [--base-directory PATH] [--session-id ID]` | Remove owned filter artifacts for one session or all sessions in the current repository |
|
|
86
|
-
| `output-filter clean --session-id ID --expired` | Remove expired exact-response artifacts for one native session |
|
|
87
|
-
| `output-filter hook --policy PATH` | **Internal, not for manual use.** `PostToolUse` entry point that reads the native hook payload on stdin and emits any replacement. The installed hook `filter-tool-output.sh` calls the same runtime through the lean `output_filter_hook.py` process instead |
|
|
88
|
-
|
|
89
|
-
`off` is the default. Configure `toolOutputFilter.mode` in
|
|
90
|
-
`.softspark-toolkit.json`, then run `install --local` or `update --local`.
|
|
82
|
+
|
|
91
83
|
`AI_TOOLKIT_OUTPUT_FILTER_DISABLE=1` bypasses active filtering immediately.
|
|
92
|
-
See [Native Tool Output Filter](tool-output-filter.md) for eligibility,
|
|
93
|
-
privacy, recovery, and runtime support.
|
|
94
84
|
|
|
95
85
|
## Project Registry
|
|
96
86
|
|
|
@@ -10,7 +10,7 @@ tags:
|
|
|
10
10
|
- multi-repo
|
|
11
11
|
doc_type: reference
|
|
12
12
|
created: "2026-04-11"
|
|
13
|
-
last_updated: "2026-07-
|
|
13
|
+
last_updated: "2026-07-26"
|
|
14
14
|
description: "Comprehensive guide for setting up and using ai-toolkit configuration inheritance. Covers base config creation, project setup, enforcement rules, CI integration, and troubleshooting."
|
|
15
15
|
---
|
|
16
16
|
|
|
@@ -77,7 +77,6 @@ ai-toolkit config check # CI enforcement check
|
|
|
77
77
|
|-------|------|-------------|
|
|
78
78
|
| `extends` | string | Base config source (npm, git URL, local path) |
|
|
79
79
|
| `profile` | enum | `minimal`, `standard`, `strict`, `full`, `offline-slm` |
|
|
80
|
-
| `toolOutputFilter` | object | Native Bash result filtering policy: mode, profiles, savings thresholds, and recovery limits |
|
|
81
80
|
| `agents` | object | `enabled`, `disabled`, `custom` arrays |
|
|
82
81
|
| `plugins` | object | Resolved `enabled` and `disabled` plugin intent |
|
|
83
82
|
| `rules` | object | `inject`, `remove` arrays |
|
|
@@ -157,25 +156,6 @@ effective intent explicit and causes validation to reject a conflicting
|
|
|
157
156
|
`plugins.disabled` entry. Install the named plugin pack separately with
|
|
158
157
|
`ai-toolkit plugin install <name>` or your organization deployment workflow.
|
|
159
158
|
|
|
160
|
-
### Native tool-output filtering
|
|
161
|
-
|
|
162
|
-
Projects can opt into the dependency-free output filter without changing the
|
|
163
|
-
organization-wide default:
|
|
164
|
-
|
|
165
|
-
```json
|
|
166
|
-
{
|
|
167
|
-
"toolOutputFilter": {
|
|
168
|
-
"mode": "observe",
|
|
169
|
-
"profiles": ["repeat-lines", "tap-success"]
|
|
170
|
-
}
|
|
171
|
-
}
|
|
172
|
-
```
|
|
173
|
-
|
|
174
|
-
`ai-toolkit install --local` materializes the effective policy as the managed
|
|
175
|
-
`.claude/ai-toolkit-output-filter.json` file. `off` is the default, `observe`
|
|
176
|
-
collects metadata without changing model-visible output, and `safe` permits
|
|
177
|
-
replacement only when every invariant and exact-recovery check succeeds.
|
|
178
|
-
|
|
179
159
|
### Overrides
|
|
180
160
|
|
|
181
161
|
Projects can override base settings, but must declare intent:
|
|
@@ -3,9 +3,9 @@ title: "Hooks Catalog"
|
|
|
3
3
|
category: reference
|
|
4
4
|
service: ai-toolkit
|
|
5
5
|
tags: [hooks, quality, safety, enforcement, settings.json]
|
|
6
|
-
version: "1.
|
|
6
|
+
version: "1.9.0"
|
|
7
7
|
created: "2026-03-27"
|
|
8
|
-
last_updated: "2026-07-
|
|
8
|
+
last_updated: "2026-07-26"
|
|
9
9
|
description: "Complete reference of all ai-toolkit hooks: events, scripts, installation, and runtime behavior."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -31,9 +31,6 @@ ai-toolkit update # re-copies scripts, re-merges (idempotent)
|
|
|
31
31
|
**File locations:**
|
|
32
32
|
|
|
33
33
|
- Scripts: `~/.softspark/ai-toolkit/hooks/*.sh`
|
|
34
|
-
- Output-filter runtime: `~/.softspark/ai-toolkit/scripts/output_filter_hook.py`, `output_filter_cli.py`, and `tool_output_filter/`
|
|
35
|
-
- Global output-filter policy: `~/.softspark/ai-toolkit/hooks/output-filter-policy.json`
|
|
36
|
-
- Managed project policy: `<project>/.claude/ai-toolkit-output-filter.json`
|
|
37
34
|
- Config: `~/.claude/settings.json` → `hooks` key
|
|
38
35
|
- Source: `ai-toolkit/app/hooks/*.sh` + `app/hooks.json`
|
|
39
36
|
|
|
@@ -178,59 +175,6 @@ arms the corrective Stop hook.
|
|
|
178
175
|
|
|
179
176
|
Skipped when `TOOLKIT_HOOK_PROFILE=minimal`.
|
|
180
177
|
|
|
181
|
-
### PostToolUse (native Bash output filter): `filter-tool-output.sh`
|
|
182
|
-
|
|
183
|
-
| Field | Value |
|
|
184
|
-
|-------|-------|
|
|
185
|
-
| Event | `PostToolUse` |
|
|
186
|
-
| Matcher | `Bash` |
|
|
187
|
-
| Script | `~/.softspark/ai-toolkit/hooks/filter-tool-output.sh` |
|
|
188
|
-
| Fires | After a successful Claude Code Bash tool call |
|
|
189
|
-
|
|
190
|
-
The hook is disabled by default and runs last among the installed
|
|
191
|
-
`PostToolUse` handlers. Its modes are:
|
|
192
|
-
|
|
193
|
-
| Mode | Behavior |
|
|
194
|
-
|------|----------|
|
|
195
|
-
| `off` | Shell fast path exits before Python starts |
|
|
196
|
-
| `observe` | Evaluates eligible output and writes content-free metadata, but emits no replacement |
|
|
197
|
-
| `safe` | Replaces eligible output only after invariants pass and the exact native response is stored for recovery |
|
|
198
|
-
|
|
199
|
-
Only explicitly allowlisted test, lint, typecheck, and validation command
|
|
200
|
-
shapes are eligible. Failed or interrupted tools, non-empty stderr, image or
|
|
201
|
-
binary results, pipes, redirects, deployment, migrations, audits, security
|
|
202
|
-
scanners, malformed payloads, unavailable recovery, and any uncertain case
|
|
203
|
-
remain unchanged.
|
|
204
|
-
|
|
205
|
-
The per-project policy `<project>/.claude/ai-toolkit-output-filter.json` is
|
|
206
|
-
honored only when **both** checks pass: the project root is registered in
|
|
207
|
-
`~/.softspark/ai-toolkit/projects.json`, and the sibling regular file
|
|
208
|
-
`<project>/.claude/.ai-toolkit-output-filter.owner` holds the ai-toolkit owner
|
|
209
|
-
marker. `ai-toolkit install --local` writes both. Registration is required
|
|
210
|
-
because the owner marker is a public constant, so a cloned or untrusted
|
|
211
|
-
checkout must never be able to self-enable filtering by shipping its own
|
|
212
|
-
marker. An unregistered project, a missing or foreign marker, or a symlinked
|
|
213
|
-
project root or `.claude` directory falls back to the installed global policy
|
|
214
|
-
at `~/.softspark/ai-toolkit/hooks/output-filter-policy.json`, which ships as
|
|
215
|
-
`off`.
|
|
216
|
-
|
|
217
|
-
`jq` is a required system dependency (`python3 scripts/check_deps.py`
|
|
218
|
-
verifies it alongside `python3`, `git`, and `node`); without it `guard-path.sh`
|
|
219
|
-
blocks file tools rather than skipping path validation.
|
|
220
|
-
|
|
221
|
-
Recovery data is private and session-scoped under
|
|
222
|
-
`~/.softspark/ai-toolkit/sessions/<repo-key>/output-filter/`. Telemetry records
|
|
223
|
-
only profile/version, byte and line counts, latency, outcome, and fallback
|
|
224
|
-
reason. Recovery-backed modes require a bounded native session ID containing
|
|
225
|
-
only ASCII letters, digits, underscores, or hyphens. Three consecutive
|
|
226
|
-
profile, invariant, or recovery safety failures open a session circuit
|
|
227
|
-
breaker. Set `AI_TOOLKIT_OUTPUT_FILTER_DISABLE=1` for an immediate bypass.
|
|
228
|
-
|
|
229
|
-
The replacement adapter is Claude Code-specific. Claude Chat/Cowork exports
|
|
230
|
-
exclude it, and the generated hooks for other editors do not activate it.
|
|
231
|
-
The `minimal` hook profile and `AI_TOOLKIT_DISABLED_HOOKS=filter-tool-output`
|
|
232
|
-
bypass it before the Python runtime starts.
|
|
233
|
-
|
|
234
178
|
### Stop (quality check) — `quality-check.sh`
|
|
235
179
|
|
|
236
180
|
| Field | Value |
|
|
@@ -344,8 +288,7 @@ Skipped when `TOOLKIT_HOOK_PROFILE=minimal`.
|
|
|
344
288
|
| Script | `~/.softspark/ai-toolkit/hooks/session-end.sh` |
|
|
345
289
|
| Fires | When a Claude session ends |
|
|
346
290
|
|
|
347
|
-
**Action:** Removes
|
|
348
|
-
the ending native session, then writes `session-end.md` to the per-repo session store
|
|
291
|
+
**Action:** Removes isolated edit state for the ending native session, then writes `session-end.md` to the per-repo session store
|
|
349
292
|
(`~/.softspark/ai-toolkit/sessions/<repo-key>/`) with a lightweight handoff note
|
|
350
293
|
for the next session.
|
|
351
294
|
|