@softspark/ai-toolkit 4.15.0 → 4.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +117 -0
- package/CHANGELOG.md +43 -0
- package/README.md +19 -13
- package/app/.claude-plugin/plugin.json +1 -1
- package/app/ARCHITECTURE.md +4 -3
- package/app/hooks/_hook-io.sh +18 -3
- package/app/hooks/ai-toolkit-statusline.sh +30 -5
- package/app/hooks/filter-tool-output.sh +76 -0
- package/app/hooks/governance-capture.sh +1 -1
- package/app/hooks/guard-path.sh +2 -2
- package/app/hooks/post-tool-use.sh +5 -3
- package/app/hooks/pre-compact-save.sh +4 -3
- package/app/hooks/quality-gate.sh +12 -1
- package/app/hooks/revert-guard.sh +5 -2
- package/app/hooks/save-session.sh +4 -2
- package/app/hooks/session-end.sh +36 -4
- package/app/hooks/session-start.sh +11 -5
- package/app/hooks.json +10 -0
- package/app/output-filter-policy.json +15 -0
- package/app/skills/brand-voice/scripts/measure.py +7 -5
- package/benchmarks/ecosystem-doctor-snapshot.json +22 -22
- package/benchmarks/output-filter/README.md +11 -0
- package/benchmarks/output-filter/scenarios.json +25 -0
- package/bin/ai-toolkit.js +2 -0
- package/kb/history/completed/native-tool-output-filter-plan.md +517 -0
- package/kb/procedures/release-preparation-sop.md +6 -5
- package/kb/reference/architecture-overview.md +6 -5
- package/kb/reference/cli-reference.md +19 -2
- package/kb/reference/codex-cli-compatibility.md +1 -0
- package/kb/reference/copilot-compatibility.md +173 -0
- package/kb/reference/enterprise-config-guide.md +28 -2
- package/kb/reference/global-install-model.md +6 -2
- package/kb/reference/hooks-catalog.md +105 -16
- package/kb/reference/opencode-compatibility.md +1 -0
- package/kb/reference/supported-tools-registry.md +10 -5
- package/kb/reference/tool-output-filter.md +288 -0
- package/kb/reference/windows-support.md +4 -3
- package/llms-full.txt +1182 -40
- package/llms.txt +3 -0
- package/manifest.json +9 -6
- package/package.json +3 -2
- package/scripts/benchmark_output_filter.py +343 -0
- package/scripts/check_deps.py +16 -0
- package/scripts/claude_app.py +30 -2
- package/scripts/config_cli.py +4 -4
- package/scripts/config_lock.py +120 -14
- package/scripts/config_merger.py +103 -20
- package/scripts/config_resolver.py +22 -2
- package/scripts/config_validator.py +268 -16
- package/scripts/copilot_legacy_hashes.json +338 -0
- package/scripts/doctor.py +1 -0
- package/scripts/generate_codex_hooks.py +2 -0
- package/scripts/generate_copilot.py +464 -71
- package/scripts/generate_copilot_hooks.py +124 -7
- package/scripts/generate_gemini_hooks.py +33 -10
- package/scripts/generate_opencode_plugin.py +28 -12
- package/scripts/install_steps/ai_tools.py +115 -3
- package/scripts/install_steps/hooks.py +25 -1
- package/scripts/output_filter_cli.py +347 -0
- package/scripts/output_filter_hook.py +23 -0
- package/scripts/plugin_schema.py +27 -1
- package/scripts/schemas/ai-toolkit-config.schema.json +83 -5
- package/scripts/session_state.py +156 -42
- package/scripts/tool_output_filter/__init__.py +33 -0
- package/scripts/tool_output_filter/contracts.py +173 -0
- package/scripts/tool_output_filter/engine.py +260 -0
- package/scripts/tool_output_filter/hook_runtime.py +369 -0
- package/scripts/tool_output_filter/input.py +56 -0
- package/scripts/tool_output_filter/invariants.py +40 -0
- package/scripts/tool_output_filter/policy.py +153 -0
- package/scripts/tool_output_filter/profiles/__init__.py +68 -0
- package/scripts/tool_output_filter/profiles/repeat_lines.py +71 -0
- package/scripts/tool_output_filter/profiles/tap_success.py +154 -0
- package/scripts/tool_output_filter/recovery.py +846 -0
- package/scripts/tool_output_filter/telemetry.py +13 -0
- package/scripts/uninstall.py +96 -3
|
@@ -0,0 +1,517 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Implementation Plan: Native Tool Output Filter"
|
|
3
|
+
category: planning
|
|
4
|
+
service: ai-toolkit
|
|
5
|
+
tags:
|
|
6
|
+
- output-filter
|
|
7
|
+
- hooks
|
|
8
|
+
- recovery
|
|
9
|
+
- performance
|
|
10
|
+
- claude-code
|
|
11
|
+
doc_type: plan
|
|
12
|
+
status: completed
|
|
13
|
+
created: "2026-07-23"
|
|
14
|
+
last_updated: "2026-07-23"
|
|
15
|
+
completed: "2026-07-23"
|
|
16
|
+
completion: "100% of approved Claude Code scope; other runtimes remain manual-only by capability decision"
|
|
17
|
+
shipped_in: "Unreleased"
|
|
18
|
+
description: "Approved implementation plan and completion evidence for the original dependency-free ai-toolkit tool-output filter, including conservative profiles, exact recovery, telemetry, runtime capability gates, and adjacent repairs."
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
# Implementation Plan: Native Tool Output Filter
|
|
22
|
+
|
|
23
|
+
## Status
|
|
24
|
+
|
|
25
|
+
Completed on 2026-07-23 after user approval.
|
|
26
|
+
|
|
27
|
+
Completion evidence:
|
|
28
|
+
|
|
29
|
+
- 69 focused Python tests pass, including the production-wrapper benchmark.
|
|
30
|
+
- The 100-sample cold wrapper measures 63.565 ms p95 against the 75 ms gate.
|
|
31
|
+
- The 8 MiB profile case measures 4.354 ms p95 and 8,792,932 peak traced
|
|
32
|
+
bytes against the 150 ms and 41,943,040 byte gates.
|
|
33
|
+
- The strict repository validator reports 44 agents, 108 skills, and 1477
|
|
34
|
+
tests with zero errors or warnings.
|
|
35
|
+
- Ruff, mypy, ShellCheck, skill audit, generated artifacts, and the final
|
|
36
|
+
repository test gate pass for the changed surface.
|
|
37
|
+
|
|
38
|
+
## Context
|
|
39
|
+
|
|
40
|
+
`ai-toolkit` currently controls assistant response length and reports real
|
|
41
|
+
Claude session tokens, but it does not transform live tool output before that
|
|
42
|
+
output reaches the model. The requested feature is an original, MIT-licensed,
|
|
43
|
+
dependency-free implementation inside `ai-toolkit`. RTK is research input only:
|
|
44
|
+
no runtime dependency, vendoring, translated code, copied filters, fixtures,
|
|
45
|
+
regex tables, CLI names, or configuration keys.
|
|
46
|
+
|
|
47
|
+
The first native integration targets Claude Code because current Claude hooks
|
|
48
|
+
support replacing successful tool output through
|
|
49
|
+
`PostToolUse.hookSpecificOutput.updatedToolOutput`. The transformer remains
|
|
50
|
+
strictly post-execution and cannot alter the command, arguments, environment,
|
|
51
|
+
working directory, permission decision, exit status, or signal.
|
|
52
|
+
|
|
53
|
+
This feature is separate from MCP `tools/list` description trimming. Hooks can
|
|
54
|
+
transform an executed tool result, but they do not intercept MCP catalog
|
|
55
|
+
metadata. [PATH: kb/history/completed/f2-mcp-trim-spike-20260504.md]
|
|
56
|
+
[PATH: kb/planning/mcp-context-trim-v4-prd.md]
|
|
57
|
+
|
|
58
|
+
Relevant local boundaries:
|
|
59
|
+
|
|
60
|
+
- Fixed lifecycle enforcement belongs in hooks. [PATH: CLAUDE.md:10]
|
|
61
|
+
- The current safety guards run in `PreToolUse`. [PATH: app/hooks.json:37]
|
|
62
|
+
- Hook runtime Python helpers are deployed explicitly.
|
|
63
|
+
[PATH: scripts/install_steps/hooks.py:71]
|
|
64
|
+
- Runtime hook schemas differ by editor.
|
|
65
|
+
[PATH: kb/reference/hooks-catalog.md:571]
|
|
66
|
+
- Existing output/token work intentionally uses native mechanisms.
|
|
67
|
+
[PATH: kb/history/completed/output-token-discipline-plan-20260504.md]
|
|
68
|
+
- Claude hook contract:
|
|
69
|
+
<https://code.claude.com/docs/en/hooks>
|
|
70
|
+
|
|
71
|
+
## Scope
|
|
72
|
+
|
|
73
|
+
### Included
|
|
74
|
+
|
|
75
|
+
- Python standard library implementation under `scripts/`.
|
|
76
|
+
- Pure post-execution transformation of successful textual Bash output.
|
|
77
|
+
- `off`, `observe`, and `safe` modes.
|
|
78
|
+
- Claude Code integration through `PostToolUse.updatedToolOutput`.
|
|
79
|
+
- Manual CLI for fixture inspection, status, recovery, and cleanup.
|
|
80
|
+
- Two initial deterministic profiles:
|
|
81
|
+
- `repeat-lines`: aggregate adjacent identical non-diagnostic lines and state
|
|
82
|
+
their multiplicity.
|
|
83
|
+
- `tap-success`: compact valid successful TAP while retaining plan,
|
|
84
|
+
directives, comments, totals, duration, and all diagnostic material.
|
|
85
|
+
- Byte and line savings, latency, outcome, profile ID, and profile version.
|
|
86
|
+
- Exact, bounded, session-scoped raw recovery when `safe` mode is enabled.
|
|
87
|
+
- Capability-gated adapters for additional runtimes after Phase 2.
|
|
88
|
+
|
|
89
|
+
### Excluded
|
|
90
|
+
|
|
91
|
+
- RTK binaries, libraries, source code, filters, fixtures, configuration, or
|
|
92
|
+
branding.
|
|
93
|
+
- Command rewriting, `sh -c`, shell parsing, permission decisions, or command
|
|
94
|
+
execution by the filter.
|
|
95
|
+
- LLM-generated summaries or network calls.
|
|
96
|
+
- Project-defined regex filters in the first release.
|
|
97
|
+
- Failed commands, non-empty `stderr`, signals, TTY/streaming output, binary or
|
|
98
|
+
invalid text, pipes, redirects, security scanners, dependency audits,
|
|
99
|
+
deployment, migrations, permission failures, and destructive-command
|
|
100
|
+
diagnostics.
|
|
101
|
+
- `Read`, web results, arbitrary MCP results, and MCP `tools/list`.
|
|
102
|
+
- Claims that byte reduction equals billed-token or whole-session savings.
|
|
103
|
+
|
|
104
|
+
## Architecture
|
|
105
|
+
|
|
106
|
+
```text
|
|
107
|
+
PostToolUse payload
|
|
108
|
+
|
|
|
109
|
+
v
|
|
110
|
+
runtime adapter validates the native payload and output shape
|
|
111
|
+
|
|
|
112
|
+
+-- unsupported, disabled, unsafe, failed, or malformed --> no hook output
|
|
113
|
+
|
|
|
114
|
+
v
|
|
115
|
+
eligibility policy selects an explicit profile
|
|
116
|
+
|
|
|
117
|
+
v
|
|
118
|
+
pure deterministic transformer
|
|
119
|
+
|
|
|
120
|
+
v
|
|
121
|
+
invariant gate
|
|
122
|
+
- mandatory facts preserved
|
|
123
|
+
- no forbidden invention
|
|
124
|
+
- deterministic and idempotent
|
|
125
|
+
- at least 15% and 1 KiB smaller
|
|
126
|
+
|
|
|
127
|
+
+-- invariant failure --> exact raw passthrough + session circuit breaker
|
|
128
|
+
|
|
|
129
|
+
v
|
|
130
|
+
exact raw response written to bounded ephemeral recovery store
|
|
131
|
+
|
|
|
132
|
+
+-- recovery unavailable --> exact raw passthrough
|
|
133
|
+
|
|
|
134
|
+
v
|
|
135
|
+
hookSpecificOutput.updatedToolOutput
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
### Module layout
|
|
139
|
+
|
|
140
|
+
```text
|
|
141
|
+
scripts/
|
|
142
|
+
tool_output_filter/
|
|
143
|
+
__init__.py
|
|
144
|
+
contracts.py
|
|
145
|
+
engine.py
|
|
146
|
+
hook_runtime.py
|
|
147
|
+
input.py
|
|
148
|
+
policy.py
|
|
149
|
+
invariants.py
|
|
150
|
+
recovery.py
|
|
151
|
+
telemetry.py
|
|
152
|
+
profiles/
|
|
153
|
+
__init__.py
|
|
154
|
+
repeat_lines.py
|
|
155
|
+
tap_success.py
|
|
156
|
+
output_filter_hook.py
|
|
157
|
+
output_filter_cli.py
|
|
158
|
+
benchmark_output_filter.py
|
|
159
|
+
|
|
160
|
+
app/
|
|
161
|
+
hooks/
|
|
162
|
+
filter-tool-output.sh
|
|
163
|
+
session-end.sh
|
|
164
|
+
hooks.json
|
|
165
|
+
output-filter-policy.json
|
|
166
|
+
|
|
167
|
+
tests/
|
|
168
|
+
fixtures/output-filter/
|
|
169
|
+
repeat-basic/
|
|
170
|
+
tap-basic/
|
|
171
|
+
tap-diagnostic/
|
|
172
|
+
ansi-adversarial/
|
|
173
|
+
test_tool_output_filter.py
|
|
174
|
+
test_tool_output_filter_cli.py
|
|
175
|
+
test_tool_output_filter_properties.py
|
|
176
|
+
test_tool_output_filter_recovery.py
|
|
177
|
+
test_tool_output_filter_benchmark.py
|
|
178
|
+
test_output_filter_config.bats
|
|
179
|
+
test_output_filter_hook.bats
|
|
180
|
+
|
|
181
|
+
kb/reference/
|
|
182
|
+
tool-output-filter.md
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
### Configuration contract
|
|
186
|
+
|
|
187
|
+
```json
|
|
188
|
+
{
|
|
189
|
+
"toolOutputFilter": {
|
|
190
|
+
"mode": "off",
|
|
191
|
+
"profiles": ["repeat-lines", "tap-success"],
|
|
192
|
+
"maxInputBytes": 8388608,
|
|
193
|
+
"minSavingsBytes": 1024,
|
|
194
|
+
"minSavingsRatio": 0.15,
|
|
195
|
+
"recovery": {
|
|
196
|
+
"mode": "ephemeral",
|
|
197
|
+
"ttlMinutes": 60,
|
|
198
|
+
"maxSessionBytes": 33554432
|
|
199
|
+
}
|
|
200
|
+
}
|
|
201
|
+
}
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
- `off`: no Python invocation and no metrics.
|
|
205
|
+
- `observe`: calculate an eligible result and metadata, but return no hook
|
|
206
|
+
output, so the model receives the original bytes.
|
|
207
|
+
- `safe`: replace only output that passes every invariant and has exact raw
|
|
208
|
+
recovery available.
|
|
209
|
+
- `AI_TOOLKIT_OUTPUT_FILTER_DISABLE=1`: global emergency bypass.
|
|
210
|
+
- Three consecutive runtime or invariant failures disable filtering for the
|
|
211
|
+
current session and emit one bounded warning.
|
|
212
|
+
- All profiles remain explicit opt-in until GA evidence is reviewed.
|
|
213
|
+
|
|
214
|
+
### Recovery and privacy
|
|
215
|
+
|
|
216
|
+
- Store the exact original tool-response object, not a redacted approximation.
|
|
217
|
+
- Location:
|
|
218
|
+
`~/.softspark/ai-toolkit/sessions/<repo-key>/output-filter/<session>/<opaque-id>`.
|
|
219
|
+
- Directory mode `0700`, file mode `0600`, exclusive no-follow creation,
|
|
220
|
+
opaque random identifiers, atomic publication, quota, and TTL.
|
|
221
|
+
- Never store raw commands, paths, output, environment values, or session IDs
|
|
222
|
+
in telemetry.
|
|
223
|
+
- Telemetry contains only profile ID/version, input/output byte and line
|
|
224
|
+
counts, latency, outcome, and fallback reason.
|
|
225
|
+
- Session end, explicit clean, uninstall, and expired-TTL cleanup remove only
|
|
226
|
+
ai-toolkit-owned recovery artifacts.
|
|
227
|
+
- If the platform cannot provide the secure-store contract, `safe` degrades to
|
|
228
|
+
raw passthrough. It must not silently become lossy.
|
|
229
|
+
|
|
230
|
+
## Success Criteria
|
|
231
|
+
|
|
232
|
+
- [x] No new npm, pip, Cargo, system, or runtime dependency.
|
|
233
|
+
- [x] Shadow/observe mode is byte-identical for 100% of fixtures and native
|
|
234
|
+
hook-contract test payloads.
|
|
235
|
+
- [x] Any exception, timeout, unsupported encoding, unknown profile, malformed
|
|
236
|
+
payload, failed invariant, unavailable recovery store, non-zero exit,
|
|
237
|
+
signal, or non-empty `stderr` returns the exact original output.
|
|
238
|
+
- [x] Every omission is marked with profile/version, original and emitted
|
|
239
|
+
line counts, and an opaque exact-recovery handle.
|
|
240
|
+
- [x] Golden fixtures preserve 100% of declared mandatory facts and introduce
|
|
241
|
+
zero non-marker facts.
|
|
242
|
+
- [x] Filter output is deterministic, idempotent, and never larger than raw.
|
|
243
|
+
- [x] Eligible fixtures achieve median reduction of at least 30%; replacement
|
|
244
|
+
requires at least 15% and 1 KiB saved per call.
|
|
245
|
+
- [x] Hard input cap is 8 MiB; larger payloads pass through without parsing.
|
|
246
|
+
- [x] Algorithm p95 is at most 20 ms for 100 KiB and 150 ms for 8 MiB; cold
|
|
247
|
+
end-to-end hook invocation p95 is at most 75 ms on CI reference runners.
|
|
248
|
+
- [x] Peak traced Python allocation is at most three times input size plus
|
|
249
|
+
16 MiB.
|
|
250
|
+
- [x] No regex has unbounded catastrophic backtracking.
|
|
251
|
+
- [x] Claude Code `safe` mode is opt-in and can be disabled globally without
|
|
252
|
+
reinstalling.
|
|
253
|
+
- [x] Additional runtimes activate only after their native result-replacement
|
|
254
|
+
contract has captured fixtures and passing integration tests.
|
|
255
|
+
- [x] Full repository validation and test suite are green.
|
|
256
|
+
|
|
257
|
+
## Pre-Mortem
|
|
258
|
+
|
|
259
|
+
| Risk | Probability | Impact | Mitigation |
|
|
260
|
+
|---|---|---|---|
|
|
261
|
+
| A profile removes a discriminating fact | High | High | Mandatory-fact fixtures, adversarial sentinels, raw recovery, one regression disables the profile |
|
|
262
|
+
| A runtime changes its tool-output schema | Medium | High | Versioned adapters, strict shape validation, unknown shape returns no hook output |
|
|
263
|
+
| Filtering hides security or failure diagnostics | Medium | High | Explicit exclusion list, successful stdout-only MVP, failures always raw |
|
|
264
|
+
| Recovery leaks secrets or paths | Medium | High | Exact data only in opt-in ephemeral spool, `0700`/`0600`, no-follow, quota, TTL, complete cleanup |
|
|
265
|
+
| Hook latency degrades normal tool use | Medium | Medium | Bash fast-path, size threshold, hard cap, benchmark gates, circuit breaker |
|
|
266
|
+
| Command classification mishandles shell syntax | High | Medium | No command rewriting or execution, conservative allowlist, pipes/redirects bypass |
|
|
267
|
+
| Regex or parser behavior is locale/version-specific | Medium | High | Independently captured multi-version fixtures; malformed/localized output passes raw |
|
|
268
|
+
| Concurrent sessions corrupt metrics or recovery | Medium | Medium | Per-session directories, opaque IDs, atomic writes; do not reuse global `session-edits.json` |
|
|
269
|
+
| Runtime adapters diverge in safety behavior | High | High | Capability matrix and one adapter at a time; no guessed compatibility |
|
|
270
|
+
| Scope expands into MCP catalog proxying | Medium | Medium | Keep MCP trim as a separate PRD and release boundary |
|
|
271
|
+
|
|
272
|
+
## Tasks
|
|
273
|
+
|
|
274
|
+
### Phase 1: Observable vertical slice, no output mutation (M)
|
|
275
|
+
|
|
276
|
+
Success criteria:
|
|
277
|
+
|
|
278
|
+
- Claude PostToolUse payloads are parsed by a native Python stdlib engine.
|
|
279
|
+
- `off` and `observe` modes cannot change model-visible output.
|
|
280
|
+
- Independent fixture format and semantic oracle are operational.
|
|
281
|
+
- No raw tool output or command is persisted.
|
|
282
|
+
|
|
283
|
+
Tasks:
|
|
284
|
+
|
|
285
|
+
- [x] Define contracts, independent fixture provenance ledger, conservative
|
|
286
|
+
eligibility rules, and clean implementation boundary.
|
|
287
|
+
Owner: `infrastructure-architect`.
|
|
288
|
+
Files: `kb/reference/tool-output-filter.md`,
|
|
289
|
+
`tests/fixtures/output-filter/README.md`.
|
|
290
|
+
- [x] Implement pure engine, policy, invariants, `repeat-lines` candidate, and
|
|
291
|
+
`tap-success` candidate in observe-only mode.
|
|
292
|
+
Owner: `backend-specialist`.
|
|
293
|
+
Files: `scripts/tool_output_filter/**`.
|
|
294
|
+
- [x] Add manual `inspect`, `status`, recovery, and cleanup CLI.
|
|
295
|
+
Owner: `backend-specialist`.
|
|
296
|
+
Files: `scripts/output_filter_cli.py`, `bin/ai-toolkit.js`.
|
|
297
|
+
- [x] Add Claude PostToolUse adapter last in hook order and deploy the Python
|
|
298
|
+
runtime package with installed hooks.
|
|
299
|
+
Owner: `command-expert`.
|
|
300
|
+
Files: `app/hooks/filter-tool-output.sh`, `app/hooks.json`,
|
|
301
|
+
`scripts/install_steps/hooks.py`.
|
|
302
|
+
- [x] Add `toolOutputFilter` schema, merge, validation, and effective-policy
|
|
303
|
+
materialization; fix config-lock source/version/integrity staleness and
|
|
304
|
+
the Article VII reserved-number mismatch while those files are open.
|
|
305
|
+
Owner: `backend-specialist`.
|
|
306
|
+
Files: `scripts/schemas/ai-toolkit-config.schema.json`,
|
|
307
|
+
`scripts/config_merger.py`, `scripts/config_validator.py`,
|
|
308
|
+
`scripts/config_lock.py`, `app/output-filter-policy.json`.
|
|
309
|
+
- [x] Build unit, property, hook-contract, malformed, binary, Unicode,
|
|
310
|
+
ANSI/OSC, injection, huge-line, and concurrency tests.
|
|
311
|
+
Owner: `test-engineer`.
|
|
312
|
+
Files: `tests/test_tool_output_filter*.py`,
|
|
313
|
+
`tests/test_tool_output_filter*.bats`,
|
|
314
|
+
`tests/fixtures/output-filter/**`.
|
|
315
|
+
|
|
316
|
+
Rollback/scope cut:
|
|
317
|
+
|
|
318
|
+
- Remove the owned PostToolUse entry and deployed runtime package.
|
|
319
|
+
- If native payload fixtures are unstable, ship only the manual inspect CLI
|
|
320
|
+
and fixture oracle; do not enable the hook.
|
|
321
|
+
|
|
322
|
+
### Phase 2: Claude Code safe mode with exact recovery (L)
|
|
323
|
+
|
|
324
|
+
Dependency: Phase 1 must pass all observe-mode and performance gates.
|
|
325
|
+
|
|
326
|
+
Success criteria:
|
|
327
|
+
|
|
328
|
+
- `safe` is explicit opt-in.
|
|
329
|
+
- Only successful, stdout-only, allowlisted text is replaceable.
|
|
330
|
+
- Exact raw recovery is available before replacement is emitted.
|
|
331
|
+
- Every failure path is byte-identical passthrough.
|
|
332
|
+
|
|
333
|
+
Tasks:
|
|
334
|
+
|
|
335
|
+
- [x] Promote `repeat-lines` and `tap-success` only after mandatory-fact,
|
|
336
|
+
idempotence, determinism, and minimum-savings gates pass.
|
|
337
|
+
Owner: `backend-specialist`.
|
|
338
|
+
Files: `scripts/tool_output_filter/profiles/**`,
|
|
339
|
+
`scripts/tool_output_filter/invariants.py`.
|
|
340
|
+
- [x] Implement bounded secure ephemeral recovery and exact `recover`/`clean`
|
|
341
|
+
CLI operations.
|
|
342
|
+
Owner: `security-architect`.
|
|
343
|
+
Files: `scripts/tool_output_filter/recovery.py`,
|
|
344
|
+
`scripts/output_filter_cli.py`.
|
|
345
|
+
- [x] Add session circuit breaker, metadata-only telemetry, TTL/quota cleanup,
|
|
346
|
+
and owned-artifact removal.
|
|
347
|
+
Owner: `backend-specialist`.
|
|
348
|
+
Files: `scripts/tool_output_filter/telemetry.py`,
|
|
349
|
+
`app/hooks/session-end.sh`, `scripts/uninstall.py`.
|
|
350
|
+
- [x] Verify safety precedence and that filtering cannot emit permission
|
|
351
|
+
decisions, execute output, or change original tool input/status.
|
|
352
|
+
Owner: `security-auditor`.
|
|
353
|
+
Files: `tests/test_tool_output_filter.py`,
|
|
354
|
+
`tests/test_tool_output_filter_recovery.py`,
|
|
355
|
+
`tests/test_output_filter_hook.bats`.
|
|
356
|
+
- [x] Add deterministic benchmark corpus and enforce latency, memory, maximum
|
|
357
|
+
input, and savings thresholds.
|
|
358
|
+
Owner: `performance-optimizer`.
|
|
359
|
+
Files: `scripts/benchmark_output_filter.py`,
|
|
360
|
+
`benchmarks/output-filter/**`.
|
|
361
|
+
|
|
362
|
+
Rollback/scope cut:
|
|
363
|
+
|
|
364
|
+
- Global disable returns the hook to observe/raw behavior immediately.
|
|
365
|
+
- Per-profile disable preserves other validated profiles.
|
|
366
|
+
- If exact secure recovery is unavailable on a platform, that platform stays
|
|
367
|
+
in `observe`; no weaker recovery implementation is accepted.
|
|
368
|
+
- If TAP cannot meet semantic gates, release only `repeat-lines`.
|
|
369
|
+
|
|
370
|
+
### Phase 3: Capability-gated runtime expansion (L)
|
|
371
|
+
|
|
372
|
+
Dependency: Phase 2 must complete Claude safe-mode dogfooding without a
|
|
373
|
+
confirmed semantic loss.
|
|
374
|
+
|
|
375
|
+
Success criteria:
|
|
376
|
+
|
|
377
|
+
- Each runtime has a documented capability: native replacement, observe-only,
|
|
378
|
+
manual CLI, or unsupported.
|
|
379
|
+
- No adapter uses command rewriting or `additionalContext` as fake replacement.
|
|
380
|
+
- Each active adapter has captured native payload/output fixtures.
|
|
381
|
+
|
|
382
|
+
Tasks:
|
|
383
|
+
|
|
384
|
+
- [x] Build and document the capability matrix from current runtime contracts.
|
|
385
|
+
Owner: `technical-researcher`.
|
|
386
|
+
Files: `kb/reference/tool-output-filter.md`,
|
|
387
|
+
`kb/reference/supported-tools-registry.md`.
|
|
388
|
+
- [x] Classify adapters one by one and keep OpenCode, Cursor, and the remaining
|
|
389
|
+
runtimes manual-only because no independently verified native
|
|
390
|
+
result-replacement contract passed the release boundary.
|
|
391
|
+
Owner: `command-expert`.
|
|
392
|
+
Files: `scripts/generate_opencode_plugin.py`,
|
|
393
|
+
`scripts/generate_cursor_hooks.py`, runtime-specific fixture files.
|
|
394
|
+
- [x] Fix ignored OpenCode guard block propagation before adding its filter
|
|
395
|
+
adapter.
|
|
396
|
+
Owner: `backend-specialist`.
|
|
397
|
+
Files: `scripts/generate_opencode_plugin.py`,
|
|
398
|
+
`tests/test_opencode*.bats`.
|
|
399
|
+
- [x] Fix Gemini invalid-settings overwrite and non-atomic write before any
|
|
400
|
+
Gemini filter adapter is attempted.
|
|
401
|
+
Owner: `backend-specialist`.
|
|
402
|
+
Files: `scripts/generate_gemini_hooks.py`,
|
|
403
|
+
`tests/test_gemini.bats`.
|
|
404
|
+
- [x] Add per-runtime uninstall, rollback, user-hook preservation, failure, and
|
|
405
|
+
schema-drift tests.
|
|
406
|
+
Owner: `test-engineer`.
|
|
407
|
+
Files: `tests/test_hooks_per_editor.bats`,
|
|
408
|
+
`tests/test_*hooks*.bats`.
|
|
409
|
+
|
|
410
|
+
Rollback/scope cut:
|
|
411
|
+
|
|
412
|
+
- A runtime without safe native replacement remains manual CLI or
|
|
413
|
+
observe-only.
|
|
414
|
+
- Any runtime-specific regression removes only its owned adapter.
|
|
415
|
+
- Cross-runtime uniformity is not a release requirement.
|
|
416
|
+
|
|
417
|
+
### Phase 4: Documentation and GA decision (M)
|
|
418
|
+
|
|
419
|
+
Dependency: Phases 1 and 2 are mandatory; Phase 3 may remain partial.
|
|
420
|
+
|
|
421
|
+
Success criteria:
|
|
422
|
+
|
|
423
|
+
- Dogfood evidence distinguishes eligible-output byte savings from actual
|
|
424
|
+
session token receipts.
|
|
425
|
+
- Documentation lists exclusions, recovery sensitivity, and bypass procedure.
|
|
426
|
+
- `safe` remains opt-in unless a separate user-approved promotion decision is
|
|
427
|
+
made.
|
|
428
|
+
|
|
429
|
+
Tasks:
|
|
430
|
+
|
|
431
|
+
- [x] Review available shadow/safe evidence, false-positive and
|
|
432
|
+
circuit-breaker paths, and latency. No production Claude JSONL receipt
|
|
433
|
+
evidence was collected in this implementation run, so documentation
|
|
434
|
+
makes no token-savings claim and `safe` remains opt-in.
|
|
435
|
+
Owner: `data-analyst`.
|
|
436
|
+
Files: `kb/reference/tool-output-filter.md`,
|
|
437
|
+
`kb/reference/stats.md`.
|
|
438
|
+
- [x] Update all hook counts, architecture references, CLI help, lifecycle
|
|
439
|
+
catalog, README, changelog, generated instruction surfaces, and release
|
|
440
|
+
notes.
|
|
441
|
+
Owner: `documenter`.
|
|
442
|
+
Files: `README.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
|
|
443
|
+
`app/ARCHITECTURE.md`, `kb/reference/architecture-overview.md`,
|
|
444
|
+
`kb/reference/hooks-catalog.md`, `CHANGELOG.md`, `AGENTS.md`,
|
|
445
|
+
`llms.txt`, `llms-full.txt`, `package.json`, `plugin.json`.
|
|
446
|
+
- [x] Re-read the complete diff for orphaned references, dead code, missing
|
|
447
|
+
behavior coverage, stale docs, and unexpected generated changes.
|
|
448
|
+
Owner: `code-reviewer`.
|
|
449
|
+
Files: all changed files.
|
|
450
|
+
|
|
451
|
+
Rollback/scope cut:
|
|
452
|
+
|
|
453
|
+
- Release `observe` and manual CLI only if active-mode evidence is insufficient.
|
|
454
|
+
- Do not promote any profile to default without separate approval.
|
|
455
|
+
|
|
456
|
+
## Dependencies
|
|
457
|
+
|
|
458
|
+
```text
|
|
459
|
+
Phase 1 → Phase 2 → Phase 3 → Phase 4
|
|
460
|
+
└────────→ Phase 4 if runtime expansion is deferred
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
There are no circular dependencies. Phase 1 is shippable as diagnostics,
|
|
464
|
+
Phase 2 as Claude-only opt-in filtering, and Phase 3 as incremental
|
|
465
|
+
runtime-by-runtime support.
|
|
466
|
+
|
|
467
|
+
## Verification
|
|
468
|
+
|
|
469
|
+
Focused verification:
|
|
470
|
+
|
|
471
|
+
1. `python3 -m unittest discover -s tests -p 'test_tool_output_filter*.py'`
|
|
472
|
+
2. `bats tests/test_output_filter_config.bats tests/test_output_filter_hook.bats`
|
|
473
|
+
3. `python3 scripts/benchmark_output_filter.py`
|
|
474
|
+
4. `shellcheck --severity=warning app/hooks/*.sh`
|
|
475
|
+
|
|
476
|
+
Repository quality gates:
|
|
477
|
+
|
|
478
|
+
1. `npm run generate:all`
|
|
479
|
+
2. `python3 scripts/validate.py --strict`
|
|
480
|
+
3. `python3 scripts/audit_skills.py --ci`
|
|
481
|
+
4. `npm test`
|
|
482
|
+
5. `git diff --check`
|
|
483
|
+
6. Re-read `git diff` and confirm no orphaned references, missing tests, stale
|
|
484
|
+
docs, copied upstream expressions, or new external dependencies.
|
|
485
|
+
|
|
486
|
+
Expected result: every command exits `0`, focused semantic fixtures report zero
|
|
487
|
+
missing mandatory facts and zero forbidden inventions, active replacements meet
|
|
488
|
+
the savings threshold, and all unsafe paths remain exact passthrough.
|
|
489
|
+
|
|
490
|
+
## Agent Assignments
|
|
491
|
+
|
|
492
|
+
| Responsibility | Agent | Model |
|
|
493
|
+
|---|---|---|
|
|
494
|
+
| Architecture and contracts | `infrastructure-architect` | Assigned tier, no override |
|
|
495
|
+
| Python engine and configuration | `backend-specialist` | Assigned tier, no override |
|
|
496
|
+
| Hook and runtime adapters | `command-expert` | Assigned tier, no override |
|
|
497
|
+
| Recovery threat model | `security-architect` | Assigned tier, no override |
|
|
498
|
+
| Security verification | `security-auditor` | Assigned tier, no override |
|
|
499
|
+
| Semantic and integration tests | `test-engineer` | Assigned tier, no override |
|
|
500
|
+
| Performance gates | `performance-optimizer` | Assigned tier, no override |
|
|
501
|
+
| Runtime contract research | `technical-researcher` | Assigned tier, no override |
|
|
502
|
+
| Evidence analysis | `data-analyst` | Assigned tier, no override |
|
|
503
|
+
| Documentation | `documenter` | Assigned tier, no override |
|
|
504
|
+
| Final diff review | `code-reviewer` | Assigned tier, no override |
|
|
505
|
+
|
|
506
|
+
## Independent-Implementation Boundary
|
|
507
|
+
|
|
508
|
+
- Implement from this plan, local ai-toolkit requirements, native runtime
|
|
509
|
+
contracts, and independently captured command outputs.
|
|
510
|
+
- Do not consult or copy RTK source while implementing filters.
|
|
511
|
+
- Do not copy or transliterate identifiers, control flow, comments, regex
|
|
512
|
+
tables, filter order, fixtures, messages, configuration keys, default values,
|
|
513
|
+
benchmarks, or documentation wording.
|
|
514
|
+
- Record the origin and rationale of each filter rule in the fixture provenance
|
|
515
|
+
ledger.
|
|
516
|
+
- If any upstream expression is intentionally adapted, stop and perform an
|
|
517
|
+
Apache-2.0 attribution and NOTICE review before continuing.
|
|
@@ -3,9 +3,9 @@ title: "SOP: Release Preparation"
|
|
|
3
3
|
category: procedures
|
|
4
4
|
service: ai-toolkit
|
|
5
5
|
tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
|
|
6
|
-
version: "1.11.
|
|
6
|
+
version: "1.11.1"
|
|
7
7
|
created: "2026-04-10"
|
|
8
|
-
last_updated: "2026-
|
|
8
|
+
last_updated: "2026-07-24"
|
|
9
9
|
description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, and checksum-pin checks added in v2.8.0, the single-run npm test discipline added in v1.8.0, the ecosystem-sync gate added in v1.9.0, the registry-vs-generators drift gate added in v1.10.0, and the mandatory pre-tag ShellCheck gate added in v1.11.0 (publish.yml does not run ShellCheck, so a hook lint failure can publish while reddening main CI — see the v4.5.1 postmortem in Phase 5)."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -335,9 +335,10 @@ grep -E '\-\-provenance|id-token: write' .github/workflows/publish.yml
|
|
|
335
335
|
**2. URL-sourced rules and hooks are checksum-pinned:**
|
|
336
336
|
|
|
337
337
|
```bash
|
|
338
|
-
# On a machine that has consumed URL rules/hooks at least once
|
|
339
|
-
|
|
340
|
-
jq 'to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/
|
|
338
|
+
# On a machine that has consumed URL rules/hooks at least once
|
|
339
|
+
# (schema_version 1: entries live under the .rules / .hooks key):
|
|
340
|
+
jq '.rules | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/rules/sources.json
|
|
341
|
+
jq '.hooks | to_entries | map(select(.value.url != null and (.value.sha256 // "" | length) == 0))' ~/.softspark/ai-toolkit/hooks/external/sources.json
|
|
341
342
|
```
|
|
342
343
|
|
|
343
344
|
- [ ] Both queries return empty arrays (every URL entry has a `sha256`)
|
|
@@ -3,9 +3,9 @@ title: "AI Toolkit - Architecture Overview"
|
|
|
3
3
|
category: reference
|
|
4
4
|
service: ai-toolkit
|
|
5
5
|
tags: [architecture, overview, design, structure]
|
|
6
|
-
version: "1.
|
|
6
|
+
version: "1.5.0"
|
|
7
7
|
created: "2026-03-23"
|
|
8
|
-
last_updated: "2026-07-
|
|
8
|
+
last_updated: "2026-07-23"
|
|
9
9
|
description: "Architecture of ai-toolkit: directory layout, Claude app export, global install model, editor-aware MCP install, Codex translation layer, skill tiers, and integration with projects."
|
|
10
10
|
---
|
|
11
11
|
|
|
@@ -285,7 +285,7 @@ Agents (code-reviewer, debugger, devops-implementer, ...)
|
|
|
285
285
|
|
|
286
286
|
## Quality Hooks
|
|
287
287
|
|
|
288
|
-
|
|
288
|
+
29 entries across 14 lifecycle events. See [hooks-catalog.md](hooks-catalog.md) for full details.
|
|
289
289
|
|
|
290
290
|
| Hook | Trigger | Script | Action |
|
|
291
291
|
|------|---------|--------|--------|
|
|
@@ -294,12 +294,13 @@ Agents (code-reviewer, debugger, devops-implementer, ...)
|
|
|
294
294
|
| Notification | Claude waiting for input | *(inline)* | macOS desktop notification |
|
|
295
295
|
| PreToolUse | Before Bash | `guard-destructive.sh` | Block destructive commands |
|
|
296
296
|
| PreToolUse | Before file ops (Bash, Read, Edit, Write, MultiEdit, Glob, Grep, NotebookEdit, mcp\_filesystem) | `guard-path.sh` | Block wrong-user path hallucination |
|
|
297
|
-
| PreToolUse | Before Edit/Write/MultiEdit | `guard-config.sh` |
|
|
297
|
+
| PreToolUse | Before Edit/Write/MultiEdit | `guard-config.sh` | Always block protected config edits; remove the hook deliberately when a change is authorized |
|
|
298
298
|
| PreToolUse | Before Bash (git commit) | `commit-quality.sh` | Advisory Conventional Commits format check |
|
|
299
299
|
| UserPromptSubmit | Before user prompt execution | `user-prompt-submit.sh` | Prompt governance reminder |
|
|
300
300
|
| UserPromptSubmit | Before user prompt execution | `track-usage.sh` | Record skill invocations to stats.json |
|
|
301
301
|
| PostToolUse | After edit/write tools | `post-tool-use.sh` | Lightweight validation reminders |
|
|
302
302
|
| PostToolUse | After any tool | `governance-capture.sh` | Log security-sensitive operations |
|
|
303
|
+
| PostToolUse | After successful Bash | `filter-tool-output.sh` | Observe eligible text or replace it only after exact recovery succeeds; Claude Code only |
|
|
303
304
|
| Stop | After response | `quality-check.sh` | Multi-language lint |
|
|
304
305
|
| Stop | After response | `save-session.sh` | Persist session context |
|
|
305
306
|
| Stop | Before final stop | `quality-gate.sh` | Block final response on lint/type errors |
|
|
@@ -309,7 +310,7 @@ Agents (code-reviewer, debugger, devops-implementer, ...)
|
|
|
309
310
|
| SubagentStop | Subagent completion | `subagent-stop.sh` | Handoff checklist for subagents |
|
|
310
311
|
| PreCompact | Before compaction | `pre-compact.sh` | Save prioritized context: instincts > tasks > git state > decisions |
|
|
311
312
|
| PreCompact | Before compaction | `pre-compact-save.sh` | Timestamped context snapshot to audit trail |
|
|
312
|
-
| SessionEnd | Session end | `session-end.sh` |
|
|
313
|
+
| SessionEnd | Session end | `session-end.sh` | Clean owned output recovery and persist the next-session handoff note |
|
|
313
314
|
|
|
314
315
|
Scripts at `~/.softspark/ai-toolkit/hooks/`. See [hooks-catalog.md](hooks-catalog.md) for details.
|
|
315
316
|
|
|
@@ -2,9 +2,9 @@
|
|
|
2
2
|
title: "CLI Reference"
|
|
3
3
|
category: reference
|
|
4
4
|
service: ai-toolkit
|
|
5
|
-
tags: [cli, commands, reference, install, update, plugin, mcp, telemetry]
|
|
5
|
+
tags: [cli, commands, reference, install, update, plugin, mcp, telemetry, output-filter]
|
|
6
6
|
created: "2026-04-13"
|
|
7
|
-
last_updated: "2026-07-
|
|
7
|
+
last_updated: "2026-07-23"
|
|
8
8
|
description: "Complete CLI reference for all ai-toolkit commands, options, and flags."
|
|
9
9
|
---
|
|
10
10
|
|
|
@@ -75,6 +75,23 @@ Usage: ai-toolkit <command> [options]
|
|
|
75
75
|
| `config create-base <name>` | Scaffold base config npm package |
|
|
76
76
|
| `config check [path]` | CI enforcement gate (exit 0=pass, 1=fail, 2=no config; `--json`) |
|
|
77
77
|
|
|
78
|
+
## Native Tool-Output Filter
|
|
79
|
+
|
|
80
|
+
| Command | Description |
|
|
81
|
+
|---------|-------------|
|
|
82
|
+
| `output-filter status [--policy PATH]` | Print the effective trusted project/global policy, or inspect an explicit policy |
|
|
83
|
+
| `output-filter inspect --profile repeat-lines\|tap-success` | Read stdin and print content-free candidate metadata without changing or echoing the input |
|
|
84
|
+
| `output-filter recover <handle> [--base-directory PATH] [--session-id ID]` | Print the exact stored native response object for an opaque recovery handle |
|
|
85
|
+
| `output-filter clean [--base-directory PATH] [--session-id ID]` | Remove owned filter artifacts for one session or all sessions in the current repository |
|
|
86
|
+
| `output-filter clean --session-id ID --expired` | Remove expired exact-response artifacts for one native session |
|
|
87
|
+
| `output-filter hook --policy PATH` | **Internal, not for manual use.** `PostToolUse` entry point that reads the native hook payload on stdin and emits any replacement. The installed hook `filter-tool-output.sh` calls the same runtime through the lean `output_filter_hook.py` process instead |
|
|
88
|
+
|
|
89
|
+
`off` is the default. Configure `toolOutputFilter.mode` in
|
|
90
|
+
`.softspark-toolkit.json`, then run `install --local` or `update --local`.
|
|
91
|
+
`AI_TOOLKIT_OUTPUT_FILTER_DISABLE=1` bypasses active filtering immediately.
|
|
92
|
+
See [Native Tool Output Filter](tool-output-filter.md) for eligibility,
|
|
93
|
+
privacy, recovery, and runtime support.
|
|
94
|
+
|
|
78
95
|
## Project Registry
|
|
79
96
|
|
|
80
97
|
| Command | Description |
|