@softspark/ai-toolkit 4.18.0 → 4.19.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/llms-full.txt CHANGED
@@ -22,15 +22,16 @@
22
22
  - [Retirement: Native Tool-Output Filter](kb/history/completed/output-filter-retirement-20260726.md)
23
23
  - [Plan: Output & Token Discipline](kb/history/completed/output-token-discipline-plan-20260504.md)
24
24
  - [rtk Pack Integration](kb/history/completed/rtk-pack-integration-20260726.md)
25
+ - [Retirement: rtk-pack](kb/history/completed/rtk-pack-retirement-20260727.md)
25
26
  - [How-To Guides](kb/howto/README.md)
26
27
  - [Plan: Cloud Security Pack — Multi-Cloud Audit](kb/planning/cloud-security-pack-plan.md)
27
28
  - [Plan: Drop Cascade hooks after 2026-07-01 sunset](kb/planning/drop-cascade-hooks-after-sunset.md)
28
29
  - [PRD: MCP Context Trim v4.0](kb/planning/mcp-context-trim-v4-prd.md)
29
30
  - [SOP: Ecosystem Sync](kb/procedures/ecosystem-sync-sop.md)
30
31
  - [SOP: AI Toolkit Maintenance](kb/procedures/maintenance-sop.md)
32
+ - [SOP: Post-Release Testing](kb/procedures/post-release-testing-sop.md)
31
33
  - [SOP: Release Preparation](kb/procedures/release-preparation-sop.md)
32
34
  - [SOP: Release Verification](kb/procedures/release-verification-sop.md)
33
- - [SOP: rtk Upstream Sync](kb/procedures/rtk-upstream-sync-sop.md)
34
35
  - [Agents Catalog](kb/reference/agents-catalog.md)
35
36
  - [Anti-Pattern Registry Format](kb/reference/anti-pattern-registry-format.md)
36
37
  - [AI Toolkit Architecture](kb/reference/architecture-overview.md)
@@ -4164,6 +4165,179 @@ archive.
4164
4165
 
4165
4166
  ---
4166
4167
 
4168
+ ## kb/history/completed/rtk-pack-retirement-20260727.md
4169
+
4170
+ ---
4171
+ title: "Retirement: rtk-pack — Broken On Install, Removed One Day After Shipping"
4172
+ category: planning
4173
+ service: ai-toolkit
4174
+ tags:
4175
+ - rtk
4176
+ - plugin-pack
4177
+ - token-reduction
4178
+ - postmortem
4179
+ - measurement
4180
+ - release-process
4181
+ doc_type: postmortem
4182
+ status: completed
4183
+ created: "2026-07-27"
4184
+ last_updated: "2026-07-27"
4185
+ shipped_in: "v4.19.0 (removal)"
4186
+ description: "Why rtk-pack, shipped in v4.18.0, was removed in v4.19.0: the first real install proved every rewritten command failed with exit 127, the wrong architecture was fetched on Apple Silicon, and the pack's own status check reported both as green. Measured value was 0.0615% of input tokens, so neither defect was worth fixing."
4187
+ ---
4188
+
4189
+ # Retirement: rtk-pack
4190
+
4191
+ **Shipped:** v4.18.0 (2026-07-26). **Removed:** v4.19.0 (2026-07-27).
4192
+
4193
+ The pack was installed on a maintainer's machine for the first time one day
4194
+ after release. It did not work, in the strongest sense available: it broke the
4195
+ shell.
4196
+
4197
+ ## Defect 1: every rewritten command failed with exit 127
4198
+
4199
+ rtk emits its rewrite as a bare `rtk git status`. The pack installs its binary
4200
+ at `~/.softspark/ai-toolkit/plugin-scripts/rtk-pack/bin/rtk` and never puts that
4201
+ directory on `PATH` — deliberately, so a checksum-pinned binary cannot shadow
4202
+ anything system-wide. The two decisions are individually defensible and jointly
4203
+ fatal: the shell could not find `rtk`, so every command the hook touched died
4204
+ before running.
4205
+
4206
+ Observed on the first three commands issued after install:
4207
+
4208
+ | Command | Result |
4209
+ |---|---|
4210
+ | `cat ~/.softspark/ai-toolkit/plugins.json` | `command not found: rtk` |
4211
+ | `git --no-pager diff` | `command not found: rtk` |
4212
+ | `find ~/.softspark -path '*rtk*'` | `command not found: rtk` |
4213
+
4214
+ The blast radius is every family in `rtk --help`: `git`, `ls`, `read`, `find`,
4215
+ `grep`, `rg`, `diff`, `docker`, `kubectl`, `npm`, `jest`, `tsc`. On the reference
4216
+ workload that is 35% of Bash bytes, which is the same 35% the pack was built to
4217
+ save. The mechanism that produced the benefit produced the outage.
4218
+
4219
+ ## Defect 2: the Intel build on an Apple Silicon host
4220
+
4221
+ `detect_platform()` trusted `platform.machine()`. The maintainer's `python3` is
4222
+ an Intel Homebrew build at `/usr/local/opt/python@3.14`, so it runs under
4223
+ Rosetta 2, where every architecture API inside the process reports `x86_64` —
4224
+ `platform.machine()`, `os.uname()` and `uname -m` alike. The pack fetched
4225
+ `rtk-x86_64-apple-darwin.tar.gz` onto an `arm64` machine and ran it emulated.
4226
+
4227
+ `sysctl.proc_translated` answers the question that distinguishes the two cases
4228
+ and was not consulted. Note that CI had already met Rosetta on this project:
4229
+ commit `30614ca`, *"build x86_64-darwin on arm64 and verify it under Rosetta"*.
4230
+ The build pipeline knew. The install path did not.
4231
+
4232
+ ## Defect 3: the pack's own health check called both of them green
4233
+
4234
+ `plugin status` reported the binary present, the digest recorded, `runs: rtk
4235
+ 0.44.0`, the hook script present and the hook registered. All true, all useless.
4236
+ The pack's `status.py` was written specifically to *"distinguish installed from
4237
+ working"*, and it checked only the installed half.
4238
+
4239
+ `kb/procedures/post-release-testing-sop.md`, written the day before the release,
4240
+ requires exactly the missing step:
4241
+
4242
+ > Presence is not function. Drive the hook directly.
4243
+
4244
+ The SOP was written and not run. Every defect above would have surfaced in its
4245
+ first five minutes.
4246
+
4247
+ ## The number that made fixing it not worth it
4248
+
4249
+ All three defects were fixed and tested before the removal decision: a `PATH`
4250
+ prefix on the emitted command, `sysctl.proc_translated` in the detector, and a
4251
+ status check that executes what the hook emits and looks for 127. Six tests,
4252
+ all passing, all failing against the previous code. The work was not hard.
4253
+
4254
+ It was measured against this, from `rtk-pack-integration-20260726.md` §10.1:
4255
+
4256
+ | | |
4257
+ |---|---:|
4258
+ | Measured saving | 1.44 MB = 360,529 tokens |
4259
+ | As a share of input tokens | **0.0615%** |
4260
+ | Kill number, published before the measurement | 0.05% |
4261
+ | Margin | ×1.23 |
4262
+
4263
+ The integration plan's own verdict on that margin was *"a pass, not a
4264
+ vindication"*. A pack that survives its kill number by 23%, carries a
4265
+ supply-chain surface, an upstream-sync SOP, a cross-build workflow for five
4266
+ targets, and a hook that rewrites every command before it runs, is not worth
4267
+ three defect classes discovered on first contact. The cost side moved; the
4268
+ benefit side never did.
4269
+
4270
+ ## Why the ceiling was always low, independent of any defect
4271
+
4272
+ `Read` results are 62.8% of tool-result bytes on this traffic, and `rtk read`
4273
+ measures 0.0% — at its default `--level none` it returns files verbatim. Tool
4274
+ results are 4.54% of input token volume, so that is the arithmetic ceiling for
4275
+ any tool-output mechanism here, and rtk addresses 8.8% of it.
4276
+
4277
+ Replayed against upstream's own claims, the families that can be measured
4278
+ deliver 25.5% in aggregate against a claimed 60–90%:
4279
+
4280
+ | Family | Measured | Claimed |
4281
+ |---|---:|---:|
4282
+ | `rtk find` | 35.6% | 70% |
4283
+ | `rtk git` | 33.1% | 70% |
4284
+ | `rtk grep` | 22.3% | 75% |
4285
+ | `rtk rg` | 7.0% | 75% |
4286
+ | `rtk read` | 0.0% | 60% |
4287
+
4288
+ Better engineering does not move any of this. The lever is in the wrong place.
4289
+
4290
+ ## The process lesson
4291
+
4292
+ The previous retirement in this series
4293
+ ([output-filter-retirement-20260726.md](output-filter-retirement-20260726.md))
4294
+ concluded that premise validation must come first, with a kill number published
4295
+ before the measurement. rtk-pack did that, and did it well: Phase 0 ran on 1224
4296
+ transcripts before any build work, the kill number was published in advance, and
4297
+ Phase 3 was cut on a measured 0.008%.
4298
+
4299
+ It then shipped without anyone installing it.
4300
+
4301
+ Measurement discipline and release discipline are different disciplines, and
4302
+ this project now has one clean failure of each. The first shipped a feature that
4303
+ worked and saved nothing. The second shipped a feature that would have saved
4304
+ something and did not work. The next plan of this shape needs both gates, and
4305
+ the second one is the cheap one: install the artifact, run the thing, look at
4306
+ what happens.
4307
+
4308
+ ## What survives
4309
+
4310
+ - **Multi-runtime pack hook wiring.** Packs write user-scope entries to
4311
+ `~/.cursor/hooks.json` and `~/.gemini/settings.json`, tagged per pack, with
4312
+ `plugin remove` taking only its own back out. Generic; no rtk in it.
4313
+ - **`supported_editors` in the manifest.** A pack declares the runtimes it works
4314
+ on instead of installing everywhere and silently doing nothing.
4315
+ - **Generic `plugin status` dispatch.** Any pack can ship `scripts/status.py`.
4316
+ The lesson attached: a status check must prove the working half by exercising
4317
+ it.
4318
+ - **Version-aware `plugin update`.** A pack whose manifest has not moved is a
4319
+ silent no-op.
4320
+ - **`audit_skills.py --ci` and the ShellCheck gate now cover `app/plugins/`.**
4321
+ - **[Post-Release Testing SOP](../../procedures/post-release-testing-sop.md).**
4322
+ Kept, and now carries the note that the one time it existed and was skipped,
4323
+ this happened.
4324
+
4325
+ ## What was removed
4326
+
4327
+ `app/plugins/rtk-pack/`, `.github/workflows/rtk-build.yml`,
4328
+ `scripts/verify_rtk_binary.py`, `tests/test_rtk_pack.bats`,
4329
+ `tests/test_verify_rtk_binary.bats`, `kb/procedures/rtk-upstream-sync-sop.md`,
4330
+ and the GitHub Release `softspark-rtk-v0.44.0-1` holding the five cross-built
4331
+ binaries.
4332
+
4333
+ Anyone who installed the pack under v4.18.0 should run
4334
+ `ai-toolkit plugin remove rtk-pack`. With the release deleted, a fresh
4335
+ `plugin install rtk-pack` on v4.18.0 fails at the fetch and leaves the pack
4336
+ inert rather than half-installed, which is the degraded path the pack was
4337
+ designed for.
4338
+
4339
+ ---
4340
+
4167
4341
  ## kb/howto/README.md
4168
4342
 
4169
4343
  ---
@@ -6367,6 +6541,228 @@ What `uninstall` does:
6367
6541
 
6368
6542
  ---
6369
6543
 
6544
+ ## kb/procedures/post-release-testing-sop.md
6545
+
6546
+ ---
6547
+ title: "SOP: Post-Release Testing"
6548
+ category: procedures
6549
+ service: ai-toolkit
6550
+ tags: [sop, post-release, smoke-test, npm, sandbox, plugin-pack, provenance, isolation]
6551
+ version: "1.0.0"
6552
+ created: "2026-07-26"
6553
+ last_updated: "2026-07-26"
6554
+ description: "Smoke-test a published @softspark/ai-toolkit release from npm in an isolated HOME and npm prefix, without touching the maintainer's real install. Covers provenance, CLI, doctor, and the full plugin-pack lifecycle including the degraded-install path. Written for v4.18.0 and not run; v4.18.0 shipped a pack that broke every command it touched, and every step here would have caught it."
6555
+ ---
6556
+
6557
+ # SOP: Post-Release Testing
6558
+
6559
+ Runs **after** `publish.yml` succeeds on a tag. Verifies the artifact users will
6560
+ actually install, from npm, rather than the working tree.
6561
+
6562
+ Sibling procedures exist for `jira-mcp` and `legal-pl-pack`; this is the
6563
+ ai-toolkit equivalent. It complements
6564
+ [Release Verification](release-verification-sop.md), which checks the toolkit
6565
+ from the maintainer's own installed copy. The difference that matters: this one
6566
+ never writes to the maintainer's `~/.claude` or `~/.softspark`.
6567
+
6568
+ **Time:** 10 minutes.
6569
+
6570
+ ## Why isolation is the first step, not a detail
6571
+
6572
+ The toolkit installs into `$HOME`. Testing a release against your own HOME
6573
+ means the test either pollutes your working setup or, worse, passes because of
6574
+ state your setup already had. Both make the result meaningless.
6575
+
6576
+ Every command below runs against a throwaway HOME and a throwaway npm prefix.
6577
+ Nothing is global.
6578
+
6579
+ ## Phase 1: Sandbox
6580
+
6581
+ ```bash
6582
+ SB=$(mktemp -d)
6583
+ mkdir -p "$SB/home" "$SB/npm"
6584
+ export HOME="$SB/home"
6585
+ AT="$SB/npm/bin/ai-toolkit"
6586
+ echo "sandbox: $SB"
6587
+ ```
6588
+
6589
+ Record the real state now, so Phase 7 can prove it is unchanged:
6590
+
6591
+ ```bash
6592
+ python3 -c "
6593
+ import json, pathlib
6594
+ p = pathlib.Path('$SB/../real-before.json')
6595
+ import os
6596
+ home = pathlib.Path(os.path.expanduser('~'))
6597
+ " 2>/dev/null
6598
+ # Simpler: note what exists today.
6599
+ cat ~/.softspark/ai-toolkit/plugins.json 2>/dev/null
6600
+ ```
6601
+
6602
+ ## Phase 2: Provenance
6603
+
6604
+ Do this before installing anything: an unsigned publish is a release-blocking
6605
+ regression, and there is no point smoke-testing a build you would have to redo.
6606
+
6607
+ ```bash
6608
+ VERSION="X.Y.Z"
6609
+ npm view "@softspark/ai-toolkit@${VERSION}" --json \
6610
+ | python3 -c "
6611
+ import json, sys
6612
+ d = json.load(sys.stdin); att = d['dist'].get('attestations', {})
6613
+ pt = att.get('provenance', {}).get('predicateType')
6614
+ assert pt == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE: {pt}'
6615
+ print('PROVENANCE OK:', att['url'])
6616
+ "
6617
+ ```
6618
+
6619
+ ## Phase 3: Install from npm
6620
+
6621
+ ```bash
6622
+ npm install -g --prefix "$SB/npm" "@softspark/ai-toolkit@${VERSION}"
6623
+ "$AT" --version # must equal VERSION
6624
+ "$AT" --help >/dev/null && echo "help OK"
6625
+ ```
6626
+
6627
+ ## Phase 4: Core surfaces
6628
+
6629
+ ```bash
6630
+ "$AT" install # full global install into the sandbox HOME
6631
+ "$AT" doctor # must end: Errors: 0 | Warnings: 0
6632
+ "$AT" status
6633
+ "$AT" plugin list # pack count must match app/plugins/
6634
+ ```
6635
+
6636
+ **A doctor run before `install` reports `agents directory missing` and
6637
+ `skills directory missing`.** That is the sandbox being empty, not a defect.
6638
+ Install first, then judge doctor.
6639
+
6640
+ ## Phase 5: Plugin-pack lifecycle
6641
+
6642
+ Run this for any pack the release touched. For a pack that downloads a binary,
6643
+ every step below has caught a real defect at least once.
6644
+
6645
+ ```bash
6646
+ "$AT" plugin install <pack>
6647
+ "$AT" plugin status # does it report itself working?
6648
+ find "$HOME/.softspark/ai-toolkit" -path '*<pack>*' # what actually landed
6649
+ ```
6650
+
6651
+ **Verify:**
6652
+ - [ ] `plugin status` distinguishes *installed* from *working*, not just present
6653
+ - [ ] For a binary pack: the binary runs and reports the pinned upstream version
6654
+ - [ ] The hook is registered in `~/.claude/settings.json` with the pack's `_source`
6655
+
6656
+ **Does it do its job?** Presence is not function. Drive the hook directly:
6657
+
6658
+ ```bash
6659
+ printf '%s' '{"tool_name":"Bash","tool_input":{"command":"git status"}}' \
6660
+ | bash "$HOME/.softspark/ai-toolkit/hooks/plugin-<pack>-<hook>.sh"
6661
+ ```
6662
+
6663
+ **Update path:**
6664
+
6665
+ ```bash
6666
+ "$AT" plugin update --editor claude --all # current version: silent
6667
+ "$AT" plugin update --editor claude --all --dry-run # says "up to date"
6668
+ # force a stale marker, then confirm it updates and re-records:
6669
+ python3 -c "
6670
+ import json, pathlib, os
6671
+ p = pathlib.Path(os.environ['HOME'] + '/.softspark/ai-toolkit/plugins.json')
6672
+ d = json.loads(p.read_text()); d['targets']['claude']['versions']['<pack>'] = '0.0.9'
6673
+ p.write_text(json.dumps(d, indent=2))
6674
+ "
6675
+ "$AT" plugin update --editor claude --all # reports 0.0.9 -> <version>
6676
+ "$AT" update # core update leaves a current pack alone
6677
+ ```
6678
+
6679
+ **Removal must be complete:**
6680
+
6681
+ ```bash
6682
+ "$AT" plugin remove <pack>
6683
+ find "$HOME/.softspark" -path '*<pack>*' | wc -l # must be 0
6684
+ python3 -c "
6685
+ import json, pathlib, os
6686
+ d = json.loads(pathlib.Path(os.environ['HOME'] + '/.claude/settings.json').read_text())
6687
+ print('hook still present:', '<pack>' in json.dumps(d.get('hooks', {})))
6688
+ "
6689
+ "$AT" plugin install <pack> # re-install must work
6690
+ ```
6691
+
6692
+ ## Phase 6: The degraded path
6693
+
6694
+ **This is the step most worth keeping.** A pack that fetches anything can fail
6695
+ to fetch, and the failure mode must be inert rather than broken or silent.
6696
+
6697
+ No pack in the toolkit fetches anything today. Run this phase if one ever does
6698
+ again, pointing its source-override variable at a dead URL:
6699
+
6700
+ ```bash
6701
+ "$AT" plugin remove <pack>
6702
+ <PACK>_RELEASE_BASE_URL="file:///nonexistent" "$AT" plugin install <pack>
6703
+ ```
6704
+
6705
+ **Verify:**
6706
+ - [ ] Install reports the failure in words a user can act on, and does not claim success
6707
+ - [ ] No partial artifact is left behind
6708
+ - [ ] The hook is still wired, and passes commands through untouched
6709
+ - [ ] `plugin status` says the pack is inert and names the fix
6710
+ - [ ] Re-installing without the broken source recovers
6711
+
6712
+ ## Phase 7: Prove the real environment is untouched
6713
+
6714
+ ```bash
6715
+ python3 -c "
6716
+ import json, pathlib
6717
+ d = json.loads(pathlib.Path.home().joinpath('.softspark/ai-toolkit/plugins.json').read_text())
6718
+ print('plugins.json:', d['targets']['claude'])
6719
+ p = pathlib.Path.home() / '.claude/settings.json'
6720
+ print('pack hook leaked into real settings:', '<pack>' in json.dumps(json.loads(p.read_text()).get('hooks', {})) if p.exists() else False)
6721
+ print('pack paths in real ~/.softspark:', len(list(pathlib.Path.home().joinpath('.softspark').rglob('*<pack>*'))))
6722
+ "
6723
+ ```
6724
+
6725
+ All three must show the pre-test state.
6726
+
6727
+ ## Phase 8: Clean up
6728
+
6729
+ `guard-destructive.sh` blocks `rm -rf` on a `PreToolUse` hook, so removal goes
6730
+ through an enumerated delete that reports what it removed:
6731
+
6732
+ ```bash
6733
+ python3 -c "
6734
+ import pathlib, shutil
6735
+ sb = pathlib.Path('$SB')
6736
+ assert sb.is_dir() and str(sb).startswith(('/tmp', '/var/folders')), sb
6737
+ n = sum(1 for _ in sb.rglob('*') if _.is_file())
6738
+ shutil.rmtree(sb)
6739
+ print(f'removed {sb} ({n} files)')
6740
+ "
6741
+ ```
6742
+
6743
+ ## Success criteria
6744
+
6745
+ | Area | Criterion |
6746
+ |---|---|
6747
+ | Supply chain | `predicateType == https://slsa.dev/provenance/v1` |
6748
+ | CLI | `--version` equals the tag, `--help` renders |
6749
+ | Health | `doctor` after `install`: 0 errors, 0 warnings |
6750
+ | Catalog | `plugin list` count matches `app/plugins/` |
6751
+ | Pack install | Binary present, runs, digest verified, hook registered |
6752
+ | Pack function | Driving the hook produces the expected effect, not just exit 0 |
6753
+ | Pack update | Current version silent; stale version updates and re-records |
6754
+ | Pack removal | Zero residue in `~/.softspark` and `settings.json`; re-install works |
6755
+ | Degraded path | Fetch failure is inert, loud in status, and recoverable |
6756
+ | Isolation | Real `~/.claude` and `~/.softspark` byte-identical to pre-test |
6757
+
6758
+ ## Related
6759
+
6760
+ - [Release Preparation](release-preparation-sop.md) — run before tagging
6761
+ - [Release Verification](release-verification-sop.md) — the maintainer-install checks
6762
+ - [rtk-pack Retirement](../history/completed/rtk-pack-retirement-20260727.md) — what happened the one time this SOP was written and not run
6763
+
6764
+ ---
6765
+
6370
6766
  ## kb/procedures/release-preparation-sop.md
6371
6767
 
6372
6768
  ---
@@ -6374,9 +6770,9 @@ title: "SOP: Release Preparation"
6374
6770
  category: procedures
6375
6771
  service: ai-toolkit
6376
6772
  tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
6377
- version: "1.11.1"
6773
+ version: "1.11.2"
6378
6774
  created: "2026-04-10"
6379
- last_updated: "2026-07-24"
6775
+ last_updated: "2026-07-27"
6380
6776
  description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, and checksum-pin checks added in v2.8.0, the single-run npm test discipline added in v1.8.0, the ecosystem-sync gate added in v1.9.0, the registry-vs-generators drift gate added in v1.10.0, and the mandatory pre-tag ShellCheck gate added in v1.11.0 (publish.yml does not run ShellCheck, so a hook lint failure can publish while reddening main CI — see the v4.5.1 postmortem in Phase 5)."
6381
6777
  ---
6382
6778
 
@@ -6666,7 +7062,15 @@ diff \
6666
7062
  && echo "OK: registry matches filesystem" \
6667
7063
  || { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
6668
7064
 
6669
- # Run npm test ONCE, cache output, parse from file. The suite is 900+ bats
7065
+ # Stage first IF this release adds or deletes a kb/ file. The test
7066
+ # "npm package KB files match the tracked release set" compares `git ls-files
7067
+ # kb` (the index) against what `npm pack` sees (the working tree), so an
7068
+ # unstaged addition reads as "extra" and an unstaged deletion as "missing".
7069
+ # Phase 6 stages, and it runs after this one, so the ordering fails the test
7070
+ # for any release that touches the KB. Staging early costs nothing.
7071
+ git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
7072
+
7073
+ # Run npm test ONCE, cache output, parse from file. The suite is 1400+ bats
6670
7074
  # cases — rerunning it per check wastes minutes. Do not pipe npm test into
6671
7075
  # tail/grep multiple times in the same session.
6672
7076
  npm test > /tmp/npm-test.log 2>&1
@@ -6823,7 +7227,7 @@ git push origin --delete vX.Y.Z
6823
7227
  | 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
6824
7228
  | 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
6825
7229
  | 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
6826
- | 16 | Tests | `npm test` | All pass |
7230
+ | 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
6827
7231
  | 17 | Commit | `git commit` | Clean working tree |
6828
7232
  | 18 | Tag | `git tag vX.Y.Z` | Tag exists |
6829
7233
  | 19 | Push | `git push origin main --tags` | CI triggered with `id-token: write` |
@@ -7340,290 +7744,6 @@ ai-toolkit eject /tmp/test # retry
7340
7744
 
7341
7745
  ---
7342
7746
 
7343
- ## kb/procedures/rtk-upstream-sync-sop.md
7344
-
7345
- ---
7346
- title: "SOP: rtk Upstream Sync"
7347
- category: procedures
7348
- service: ai-toolkit
7349
- tags: [sop, rtk, rtk-pack, upstream, cross-build, telemetry, checksum, port-validation, advisory]
7350
- version: "1.0.0"
7351
- created: "2026-07-26"
7352
- last_updated: "2026-07-26"
7353
- description: "Procedure for moving rtk-pack to a newer upstream rtk release: detect the tag, review the files we depend on, re-validate the Python port that every coverage number rests on, rebuild five targets with telemetry undefined, verify silence, publish to our own release namespace, and bump the pack. Written after the v0.43.0 to v0.44.0 bump, which changed every file this SOP names."
7354
- ---
7355
-
7356
- # SOP: rtk Upstream Sync
7357
-
7358
- Moves `rtk-pack` from one pinned upstream tag to the next.
7359
-
7360
- Currently pinned: **v0.44.0**, shipped as
7361
- `softspark-rtk-v0.44.0-1`. The pin lives in
7362
- `app/plugins/rtk-pack/plugin.json` under `upstream.version`.
7363
-
7364
- Upstream ships stable tags roughly every two to four weeks behind a long
7365
- release-candidate train (300+ RCs preceded v0.44.0). Do not track RCs.
7366
-
7367
- Background and the measured numbers: `kb/history/completed/rtk-pack-integration-20260726.md`.
7368
-
7369
- ## Why this SOP is not "just rebuild"
7370
-
7371
- The v0.43.0 to v0.44.0 bump was 200 commits and touched **every file listed in
7372
- Phase 2 below**. It also inverted pipeline rewriting: v0.43.0 rewrote the first
7373
- stage of a pipeline, v0.44.0 rewrites the last. A rebuild without the review
7374
- step would have shipped that silently, and the coverage numbers quoted to users
7375
- would have described a version we no longer ship.
7376
-
7377
- ## Phase 1: Detect
7378
-
7379
- ```bash
7380
- gh api repos/rtk-ai/rtk/releases --paginate \
7381
- --jq '.[] | select(.tag_name | test("^v[0-9]")) | "\(.tag_name)\t\(.published_at)"' | head -5
7382
- ```
7383
-
7384
- Compare against `upstream.version` in `app/plugins/rtk-pack/plugin.json`.
7385
-
7386
- Cadence: on demand, plus a check folded into release preparation.
7387
-
7388
- ## Phase 2: Review before building
7389
-
7390
- Fetch the diff for the areas the pack depends on:
7391
-
7392
- ```bash
7393
- gh api repos/rtk-ai/rtk/compare/<pinned>...<new> \
7394
- --jq '{ahead: .ahead_by, files: [.files[] | {f: .filename, add: .additions, del: .deletions}]}'
7395
- ```
7396
-
7397
- Read the changelog, then diff these specifically:
7398
-
7399
- | File | Why it matters |
7400
- |---|---|
7401
- | `src/discover/registry.rs` | rewrite eligibility, pipeline handling, the TOML call sites |
7402
- | `src/discover/rules.rs` | the rule table and `IGNORED_PREFIXES` |
7403
- | `src/discover/lexer.rs` | tokenisation and `contains_unattestable_construct` |
7404
- | `src/core/toml_filter.rs` | the filter DSL, which affects users who write their own filters |
7405
- | `src/hooks/trust.rs` | the trust gate and its paths |
7406
- | `src/hooks/hook_cmd.rs` | the Claude hook contract and permission handling |
7407
- | `src/core/telemetry.rs`, `src/core/telemetry_cmd.rs` | the compile-time endpoint gate |
7408
- | `Cargo.toml`, `Cargo.lock` | native deps, the MSRV, and new advisories |
7409
-
7410
- **A change to the DSL, to trust handling, or to the permission flow is a
7411
- stop-and-think, not a rebuild.** In particular:
7412
-
7413
- - The pack ships no filters of its own, so a DSL change cannot break us. It can
7414
- still break a **user's** `filters.toml`, which upstream then skips silently
7415
- (`toml_filter.rs:220-221`). Worth a release-note line, not a code change.
7416
- - If `hook_cmd.rs` changes when `permissionDecision` is emitted, the trust
7417
- boundary documented in the pack README changes with it.
7418
- - If `IGNORED_PREFIXES` or the `pipeline_final_safe` rule set moves, every
7419
- coverage number is stale.
7420
-
7421
- ## Phase 3: Re-validate the port
7422
-
7423
- Every coverage and saving figure the pack quotes comes from `rtk_port.py`, a
7424
- Python model of rtk's rewrite pipeline. **Anything short of full agreement
7425
- invalidates those numbers until the port is fixed.** Tooling lives in
7426
- `~/rtk-measurement-archive/`.
7427
-
7428
- ```bash
7429
- git clone --depth 1 --branch <new-tag> https://github.com/rtk-ai/rtk.git /tmp/rtk-new
7430
- cd ~/rtk-measurement-archive
7431
-
7432
- # Rewrite assertions from the tag's own test block.
7433
- python3 extract_cases.py /tmp/rtk-new/src/discover/registry.rs cases.json
7434
- python3 validate_port.py cases.json
7435
-
7436
- # The entry gate both real hook paths apply before rewrite_command.
7437
- # Its assertions live in lexer.rs; re-extract if that block moved.
7438
- python3 -c "import entry_gate; print('gate import ok')"
7439
- ```
7440
-
7441
- Baseline at v0.44.0: **203/203** rewrite assertions, **35/35** gate assertions.
7442
-
7443
- If the port diverges, fix the port first, then re-measure:
7444
-
7445
- ```bash
7446
- python3 measure_gated.py gated 0:1224
7447
- ```
7448
-
7449
- Measure over the whole transcript pool, never the default 134-file window: at
7450
- that size the projection swings 8.8x on an unchanged mechanism.
7451
-
7452
- ### 3.1 Re-measure, do not just re-validate
7453
-
7454
- Port agreement proves rtk still rewrites the same commands. It says nothing
7455
- about how much each rewrite saves, and that is where the value actually sits.
7456
-
7457
- **`rtk grep` carries 4.04 MB of the 5.66 MB measurable saving: 71% of the total
7458
- rests on one family.** If upstream changes that one filter, the headline number
7459
- moves even with the port at full agreement. Measured effectiveness has already
7460
- diverged from expectation in both directions once: `rtk grep` measured 22.3%
7461
- against 9.0% modelled, `rtk rg` 7.0% against 30% modelled.
7462
-
7463
- So on every bump, after the port passes, replay against the newly built binary:
7464
-
7465
- ```bash
7466
- python3 replay_rtk.py --pool 1300 --rtk <path-to-the-new-binary>
7467
- ```
7468
-
7469
- Compare per family against the recorded baseline:
7470
-
7471
- | Family | Measured at v0.44.0 | Share of total saving |
7472
- |---|---:|---:|
7473
- | `rtk grep` | 22.3% | 71% |
7474
- | `rtk git` | 33.1% | 21% |
7475
- | `rtk find` | 35.6% | 8% |
7476
-
7477
- **Act on the result, do not just record it.** If the total lands below the
7478
- published kill number of **0.05% of input tokens**, the pack has stopped earning
7479
- its supply-chain surface and retiring it is the correct outcome, exactly as the
7480
- in-house filter was retired. The margin at v0.44.0 is 0.0615%, which is 1.23x
7481
- the kill number, so a single-family regression is enough to cross it.
7482
-
7483
- ## Phase 4: Rebuild
7484
-
7485
- ```bash
7486
- gh workflow run rtk-build.yml --ref main \
7487
- -f upstream_tag=<new-tag> -f build_revision=1 -f rust_version=<pinned> -f publish=false
7488
- ```
7489
-
7490
- Five targets: `x86_64-apple-darwin`, `aarch64-apple-darwin`,
7491
- `x86_64-unknown-linux-musl`, `aarch64-unknown-linux-gnu`,
7492
- `x86_64-pc-windows-msvc`. A target that will not build is dropped, not faked.
7493
-
7494
- Pin `rust_version` explicitly. Upstream uses unpinned `stable` with
7495
- `warnings = "deny"`, so a new rustc lint can turn the build red with no change
7496
- on either side.
7497
-
7498
- **Advisories.** The `audit` job blocks. Re-derive the disposition rather than
7499
- carrying the previous one forward:
7500
-
7501
- ```bash
7502
- # Cross-reference the new lockfile against OSV without waiting for CI.
7503
- python3 - <<'PY'
7504
- import json, re, urllib.request, pathlib
7505
- lock = pathlib.Path("/tmp/rtk-new/Cargo.lock").read_text()
7506
- pkgs = [(re.search(r'^name = "([^"]+)"', b, re.M).group(1),
7507
- re.search(r'^version = "([^"]+)"', b, re.M).group(1))
7508
- for b in lock.split("[[package]]")[1:]
7509
- if re.search(r'^name = ', b, re.M) and re.search(r'^version = ', b, re.M)]
7510
- q = [{"package": {"name": n, "ecosystem": "crates.io"}, "version": v} for n, v in pkgs]
7511
- req = urllib.request.Request("https://api.osv.dev/v1/querybatch",
7512
- data=json.dumps({"queries": q}).encode(), headers={"Content-Type": "application/json"})
7513
- res = json.load(urllib.request.urlopen(req, timeout=60))
7514
- for (n, v), r in zip(pkgs, res["results"]):
7515
- if r.get("vulns"):
7516
- print(n, v, [x["id"] for x in r["vulns"]])
7517
- PY
7518
- ```
7519
-
7520
- For each advisory decide, and record the reason in the workflow:
7521
-
7522
- - **In-range fix** (`cargo update -p <crate>` works): add the crate to
7523
- `RTK_CARGO_UPDATES` in `.github/workflows/rtk-build.yml`. Both the audit job
7524
- and every build job apply it, so we audit what we ship.
7525
- - **Needs a `Cargo.toml` change**: that is a source modification and breaks the
7526
- NOTICE claim. Ignore with a written reason, or escalate.
7527
- - **Carried-forward ignores**: re-check every `--ignore` still applies. The
7528
- quick-xml pair exists only because upstream pins `"0.37"`; **delete both the
7529
- moment upstream moves to 0.41 or later** rather than carrying them.
7530
-
7531
- ## Phase 5: Verify
7532
-
7533
- CI asserts this per target and the run fails on any `fail` verdict:
7534
-
7535
- 1. `RTK_TELEMETRY_URL` and `RTK_TELEMETRY_TOKEN` unset at build time
7536
- 2. the artifact starts and reports the expected version
7537
- 3. no telemetry state written into a sandboxed home
7538
- 4. on Linux, identical behaviour with no network route
7539
- 5. every archive holds exactly one flat entry
7540
-
7541
- Then check by hand:
7542
-
7543
- ```bash
7544
- gh run download <run-id> --dir /tmp/rtk-verify
7545
- cd /tmp/rtk-verify && shasum -a 256 -c checksums.txt
7546
- ```
7547
-
7548
- Three things CI cannot tell you:
7549
-
7550
- - **`strings` markers.** Compare `tls_markers_present` per target against the
7551
- previous build. Their absence is an LLVM outcome under LTO, not a guarantee,
7552
- so a sudden appearance means the telemetry stack survived and is worth
7553
- understanding before shipping.
7554
- - **Reproducibility.** Four of five targets are bit-reproducible; compare
7555
- extracted binaries, never the tarballs, because gzip records a timestamp.
7556
- `x86_64-pc-windows-msvc` differs by 24 bytes per link (MSVC timestamp plus a
7557
- CodeView GUID), so a changed Windows digest proves nothing on its own.
7558
- - **Upstream has no test asserting network silence.** That property can regress
7559
- on any bump without turning their CI red. We own it.
7560
-
7561
- ## Phase 6: Publish
7562
-
7563
- ```bash
7564
- gh workflow run rtk-build.yml --ref main \
7565
- -f upstream_tag=<new-tag> -f build_revision=1 -f rust_version=<pinned> -f publish=true
7566
- ```
7567
-
7568
- Creates `softspark-rtk-<upstream>-<revision>`. Bump the revision, not the
7569
- upstream part, when rebuilding the same upstream tag.
7570
-
7571
- ## Phase 7: Bump the pack
7572
-
7573
- In `app/plugins/rtk-pack/plugin.json`:
7574
-
7575
- - `upstream.version` to the new tag
7576
- - `binary.release_tag` to the new release
7577
- - every `assets.*.sha256` from the published `checksums.txt`
7578
- - `version` (the pack's own) — bump it whether or not the upstream tag moved,
7579
- because `plugin update` skips a pack whose recorded version still matches, so
7580
- an unbumped pack never reaches installed users. The two fields are separate so
7581
- a pack-only fix does not pretend to be an upstream bump
7582
-
7583
- Then:
7584
-
7585
- ```bash
7586
- npm test # tests/test_rtk_pack.bats asserts digest shape and layout
7587
- python3 scripts/validate.py --strict
7588
- python3 scripts/audit_skills.py --ci
7589
- shellcheck --severity=warning app/hooks/*.sh app/plugins/*/hooks/*.sh
7590
- ```
7591
-
7592
- Verify a real install end to end, against the published release rather than a
7593
- mirror:
7594
-
7595
- ```bash
7596
- H=$(mktemp -d)
7597
- AI_TOOLKIT_DATA_DIR="$H" python3 app/plugins/rtk-pack/scripts/init.py
7598
- AI_TOOLKIT_DATA_DIR="$H" python3 app/plugins/rtk-pack/scripts/status.py
7599
- ```
7600
-
7601
- ## Phase 8: Ship
7602
-
7603
- The pack version bump reaches installed users through `ai-toolkit update`,
7604
- which runs `plugin update --editor all --all`. A pack whose recorded version
7605
- matches its manifest is skipped silently, so the bump in Phase 7 is what makes
7606
- the update fire at all. Forgetting it means nobody gets the new binary.
7607
-
7608
- ## Phase 9: Record the licence position
7609
-
7610
- If upstream relicenses, adds a `NOTICE`, or changes `DISCLAIMER.md`, update the
7611
- NOTICE generated in `.github/workflows/rtk-build.yml`. It currently records two
7612
- build-time differences: the undefined telemetry endpoint, and in-range lockfile
7613
- security updates. If Phase 4 changes that set, the NOTICE text changes with it.
7614
-
7615
- `DISCLAIMER.md` is deliberately not redistributed because it states telemetry is
7616
- collected by default, which our builds contradict. Re-check that this is still
7617
- the reason before changing the decision.
7618
-
7619
- ## Related
7620
-
7621
- - `kb/history/completed/rtk-pack-integration-20260726.md` — decisions, measurements, pre-mortem
7622
- - `kb/history/completed/output-filter-retirement-20260726.md` — why premise validation comes first
7623
- - `app/plugins/rtk-pack/README.md` — the user-facing trust boundary
7624
-
7625
- ---
7626
-
7627
7747
  ## kb/reference/agents-catalog.md
7628
7748
 
7629
7749
  ---
@@ -13563,21 +13683,30 @@ ai-toolkit plugin status --editor all # show installed packs with r
13563
13683
  | `kotlin-pack` | kotlin | 0 | 1 | 0 | Kotlin patterns |
13564
13684
  | `swift-pack` | swift | 0 | 1 | 0 | Swift patterns |
13565
13685
  | `ruby-pack` | ruby | 0 | 1 | 0 | Ruby patterns |
13566
- | `rtk-pack` | token-reduction | 0 | 0 | 1 | Command rewriting via a checksum-pinned rtk binary fetched at install |
13567
-
13568
- `rtk-pack` is the first pack to break three assumptions the others share, so it
13569
- is the one to read when extending the contract:
13570
-
13571
- - **It fetches from the network at install time.** `scripts/init.py` downloads a
13572
- platform-specific artifact and verifies its SHA-256 against `plugin.json`
13573
- before installing anything. A mismatch aborts and leaves nothing behind.
13574
- - **It declares platform assets and digests in `plugin.json`.** The manifest
13575
- schema tolerates extra keys, so `upstream` and `binary` are additive; nothing
13576
- validates them, which means a malformed block fails at install rather than in
13577
- `validate.py --strict`.
13578
- - **It reports its own health.** `scripts/status.py` is picked up generically by
13579
- `plugin status`, replacing what used to be a hardcoded `if name == "memory-pack"`
13580
- branch. Any pack can now ship one.
13686
+
13687
+ Every pack here is content that ships in this repository. None fetches anything
13688
+ at install time.
13689
+
13690
+ **A pack that downloads a binary has been tried once and retired.** `rtk-pack`
13691
+ (v4.18.0, removed in v4.19.0) fetched a checksum-pinned artifact in
13692
+ `scripts/init.py`, declared platform assets and digests in `plugin.json`, and
13693
+ rewrote commands at `PreToolUse`. Read
13694
+ `kb/history/completed/rtk-pack-retirement-20260727.md` before proposing another
13695
+ pack of that shape; the two defects that killed it were both invisible to
13696
+ `validate.py --strict` and to the pack's own status check.
13697
+
13698
+ What survives from that work and applies to any pack:
13699
+
13700
+ - **A pack reports its own health.** `scripts/status.py` is picked up
13701
+ generically by `plugin status`, replacing what used to be a hardcoded
13702
+ `if name == "memory-pack"` branch. A status check must distinguish *installed*
13703
+ from *working*, and prove the working part by exercising it rather than by
13704
+ checking that files exist.
13705
+ - **A pack may declare `supported_editors`.** Without it a pack installs on
13706
+ every runtime and silently does nothing on the ones it was never built for.
13707
+ - **The manifest schema tolerates extra keys.** Anything additive is unvalidated,
13708
+ so a malformed block fails at install time rather than in `validate.py
13709
+ --strict`. Do not rely on the schema to catch it.
13581
13710
 
13582
13711
  ## Optional Hook Modules
13583
13712