@softspark/ai-toolkit 4.17.0 → 4.19.0

package/kb/history/completed/rtk-pack-retirement-20260727.md

@@ -0,0 +1,168 @@
+---
+title: "Retirement: rtk-pack — Broken On Install, Removed One Day After Shipping"
+category: planning
+service: ai-toolkit
+tags:
+  - rtk
+  - plugin-pack
+  - token-reduction
+  - postmortem
+  - measurement
+  - release-process
+doc_type: postmortem
+status: completed
+created: "2026-07-27"
+last_updated: "2026-07-27"
+shipped_in: "v4.19.0 (removal)"
+description: "Why rtk-pack, shipped in v4.18.0, was removed in v4.19.0: the first real install proved every rewritten command failed with exit 127, the wrong architecture was fetched on Apple Silicon, and the pack's own status check reported both as green. Measured value was 0.0615% of input tokens, so neither defect was worth fixing."
+---
+
+# Retirement: rtk-pack
+
+**Shipped:** v4.18.0 (2026-07-26). **Removed:** v4.19.0 (2026-07-27).
+
+The pack was installed on a maintainer's machine for the first time one day
+after release. It did not work, in the strongest sense available: it broke the
+shell.
+
+## Defect 1: every rewritten command failed with exit 127
+
+rtk emits its rewrite as a bare `rtk git status`. The pack installs its binary
+at `~/.softspark/ai-toolkit/plugin-scripts/rtk-pack/bin/rtk` and never puts that
+directory on `PATH` — deliberately, so a checksum-pinned binary cannot shadow
+anything system-wide. The two decisions are individually defensible and jointly
+fatal: the shell could not find `rtk`, so every command the hook touched died
+before running.
+
+Observed on the first three commands issued after install:
+
+| Command | Result |
+|---|---|
+| `cat ~/.softspark/ai-toolkit/plugins.json` | `command not found: rtk` |
+| `git --no-pager diff` | `command not found: rtk` |
+| `find ~/.softspark -path '*rtk*'` | `command not found: rtk` |
+
+The blast radius is every family in `rtk --help`: `git`, `ls`, `read`, `find`,
+`grep`, `rg`, `diff`, `docker`, `kubectl`, `npm`, `jest`, `tsc`. On the reference
+workload that is 35% of Bash bytes, which is the same 35% the pack was built to
+save. The mechanism that produced the benefit produced the outage.
+
+## Defect 2: the Intel build on an Apple Silicon host
+
+`detect_platform()` trusted `platform.machine()`. The maintainer's `python3` is
+an Intel Homebrew build at `/usr/local/opt/python@3.14`, so it runs under
+Rosetta 2, where every architecture API inside the process reports `x86_64` —
+`platform.machine()`, `os.uname()` and `uname -m` alike. The pack fetched
+`rtk-x86_64-apple-darwin.tar.gz` onto an `arm64` machine and ran it emulated.
+
+`sysctl.proc_translated` answers the question that distinguishes the two cases
+and was not consulted. Note that CI had already met Rosetta on this project:
+commit `30614ca`, *"build x86_64-darwin on arm64 and verify it under Rosetta"*.
+The build pipeline knew. The install path did not.
+
+## Defect 3: the pack's own health check called both of them green
+
+`plugin status` reported the binary present, the digest recorded, `runs: rtk
+0.44.0`, the hook script present and the hook registered. All true, all useless.
+The pack's `status.py` was written specifically to *"distinguish installed from
+working"*, and it checked only the installed half.
+
+`kb/procedures/post-release-testing-sop.md`, written the day before the release,
+requires exactly the missing step:
+
+> Presence is not function. Drive the hook directly.
+
+The SOP was written and not run. Every defect above would have surfaced in its
+first five minutes.
+
+## The number that made fixing it not worth it
+
+All three defects were fixed and tested before the removal decision: a `PATH`
+prefix on the emitted command, `sysctl.proc_translated` in the detector, and a
+status check that executes what the hook emits and looks for 127. Six tests,
+all passing, all failing against the previous code. The work was not hard.
+
+It was measured against this, from `rtk-pack-integration-20260726.md` §10.1:
+
+| | |
+|---|---:|
+| Measured saving | 1.44 MB = 360,529 tokens |
+| As a share of input tokens | **0.0615%** |
+| Kill number, published before the measurement | 0.05% |
+| Margin | ×1.23 |
+
+The integration plan's own verdict on that margin was *"a pass, not a
+vindication"*. A pack that survives its kill number by 23%, carries a
+supply-chain surface, an upstream-sync SOP, a cross-build workflow for five
+targets, and a hook that rewrites every command before it runs, is not worth
+three defect classes discovered on first contact. The cost side moved; the
+benefit side never did.
+
+## Why the ceiling was always low, independent of any defect
+
+`Read` results are 62.8% of tool-result bytes on this traffic, and `rtk read`
+measures 0.0% — at its default `--level none` it returns files verbatim. Tool
+results are 4.54% of input token volume, so that is the arithmetic ceiling for
+any tool-output mechanism here, and rtk addresses 8.8% of it.
+
+Replayed against upstream's own claims, the families that can be measured
+deliver 25.5% in aggregate against a claimed 60–90%:
+
+| Family | Measured | Claimed |
+|---|---:|---:|
+| `rtk find` | 35.6% | 70% |
+| `rtk git` | 33.1% | 70% |
+| `rtk grep` | 22.3% | 75% |
+| `rtk rg` | 7.0% | 75% |
+| `rtk read` | 0.0% | 60% |
+
+Better engineering does not move any of this. The lever is in the wrong place.
+
+## The process lesson
+
+The previous retirement in this series
+([output-filter-retirement-20260726.md](output-filter-retirement-20260726.md))
+concluded that premise validation must come first, with a kill number published
+before the measurement. rtk-pack did that, and did it well: Phase 0 ran on 1224
+transcripts before any build work, the kill number was published in advance, and
+Phase 3 was cut on a measured 0.008%.
+
+It then shipped without anyone installing it.
+
+Measurement discipline and release discipline are different disciplines, and
+this project now has one clean failure of each. The first shipped a feature that
+worked and saved nothing. The second shipped a feature that would have saved
+something and did not work. The next plan of this shape needs both gates, and
+the second one is the cheap one: install the artifact, run the thing, look at
+what happens.
+
+## What survives
+
+- **Multi-runtime pack hook wiring.** Packs write user-scope entries to
+  `~/.cursor/hooks.json` and `~/.gemini/settings.json`, tagged per pack, with
+  `plugin remove` taking only its own back out. Generic; no rtk in it.
+- **`supported_editors` in the manifest.** A pack declares the runtimes it works
+  on instead of installing everywhere and silently doing nothing.
+- **Generic `plugin status` dispatch.** Any pack can ship `scripts/status.py`.
+  The lesson attached: a status check must prove the working half by exercising
+  it.
+- **Version-aware `plugin update`.** A pack whose manifest has not moved is a
+  silent no-op.
+- **`audit_skills.py --ci` and the ShellCheck gate now cover `app/plugins/`.**
+- **[Post-Release Testing SOP](../../procedures/post-release-testing-sop.md).**
+  Kept, and now carries the note that the one time it existed and was skipped,
+  this happened.
+
+## What was removed
+
+`app/plugins/rtk-pack/`, `.github/workflows/rtk-build.yml`,
+`scripts/verify_rtk_binary.py`, `tests/test_rtk_pack.bats`,
+`tests/test_verify_rtk_binary.bats`, `kb/procedures/rtk-upstream-sync-sop.md`,
+and the GitHub Release `softspark-rtk-v0.44.0-1` holding the five cross-built
+binaries.
+
+Anyone who installed the pack under v4.18.0 should run
+`ai-toolkit plugin remove rtk-pack`. With the release deleted, a fresh
+`plugin install rtk-pack` on v4.18.0 fails at the fetch and leaves the pack
+inert rather than half-installed, which is the degraded path the pack was
+designed for.

package/kb/procedures/maintenance-sop.md

@@ -291,7 +291,7 @@ python3 scripts/validate.py --strict
 ai-toolkit plugin list                             # show available packs
 ai-toolkit plugin install --editor claude <name>  # install for Claude Code global target
 ai-toolkit plugin install --editor codex <name>   # install for Codex global target
-ai-toolkit plugin install --editor all --all      # install all 11 packs for both runtimes
+ai-toolkit plugin install --editor all --all      # install all 12 packs for both runtimes
 ai-toolkit plugin update --editor all --all       # re-apply all installed packs after toolkit updates
 ai-toolkit plugin clean <name>                    # prune data older than 90 days
 ai-toolkit plugin clean <name> --days 30  # custom retention

package/kb/procedures/post-release-testing-sop.md

@@ -0,0 +1,217 @@
+---
+title: "SOP: Post-Release Testing"
+category: procedures
+service: ai-toolkit
+tags: [sop, post-release, smoke-test, npm, sandbox, plugin-pack, provenance, isolation]
+version: "1.0.0"
+created: "2026-07-26"
+last_updated: "2026-07-26"
+description: "Smoke-test a published @softspark/ai-toolkit release from npm in an isolated HOME and npm prefix, without touching the maintainer's real install. Covers provenance, CLI, doctor, and the full plugin-pack lifecycle including the degraded-install path. Written for v4.18.0 and not run; v4.18.0 shipped a pack that broke every command it touched, and every step here would have caught it."
+---
+
+# SOP: Post-Release Testing
+
+Runs **after** `publish.yml` succeeds on a tag. Verifies the artifact users will
+actually install, from npm, rather than the working tree.
+
+Sibling procedures exist for `jira-mcp` and `legal-pl-pack`; this is the
+ai-toolkit equivalent. It complements
+[Release Verification](release-verification-sop.md), which checks the toolkit
+from the maintainer's own installed copy. The difference that matters: this one
+never writes to the maintainer's `~/.claude` or `~/.softspark`.
+
+**Time:** 10 minutes.
+
+## Why isolation is the first step, not a detail
+
+The toolkit installs into `$HOME`. Testing a release against your own HOME
+means the test either pollutes your working setup or, worse, passes because of
+state your setup already had. Both make the result meaningless.
+
+Every command below runs against a throwaway HOME and a throwaway npm prefix.
+Nothing is global.
+
+## Phase 1: Sandbox
+
+```bash
+SB=$(mktemp -d)
+mkdir -p "$SB/home" "$SB/npm"
+export HOME="$SB/home"
+AT="$SB/npm/bin/ai-toolkit"
+echo "sandbox: $SB"
+```
+
+Record the real state now, so Phase 7 can prove it is unchanged:
+
+```bash
+python3 -c "
+import json, pathlib
+p = pathlib.Path('$SB/../real-before.json')
+import os
+home = pathlib.Path(os.path.expanduser('~'))
+" 2>/dev/null
+# Simpler: note what exists today.
+cat ~/.softspark/ai-toolkit/plugins.json 2>/dev/null
+```
+
+## Phase 2: Provenance
+
+Do this before installing anything: an unsigned publish is a release-blocking
+regression, and there is no point smoke-testing a build you would have to redo.
+
+```bash
+VERSION="X.Y.Z"
+npm view "@softspark/ai-toolkit@${VERSION}" --json \
+  | python3 -c "
+import json, sys
+d = json.load(sys.stdin); att = d['dist'].get('attestations', {})
+pt = att.get('provenance', {}).get('predicateType')
+assert pt == 'https://slsa.dev/provenance/v1', f'NO PROVENANCE: {pt}'
+print('PROVENANCE OK:', att['url'])
+"
+```
+
+## Phase 3: Install from npm
+
+```bash
+npm install -g --prefix "$SB/npm" "@softspark/ai-toolkit@${VERSION}"
+"$AT" --version    # must equal VERSION
+"$AT" --help >/dev/null && echo "help OK"
+```
+
+## Phase 4: Core surfaces
+
+```bash
+"$AT" install            # full global install into the sandbox HOME
+"$AT" doctor             # must end: Errors: 0 | Warnings: 0
+"$AT" status
+"$AT" plugin list        # pack count must match app/plugins/
+```
+
+**A doctor run before `install` reports `agents directory missing` and
+`skills directory missing`.** That is the sandbox being empty, not a defect.
+Install first, then judge doctor.
+
+## Phase 5: Plugin-pack lifecycle
+
+Run this for any pack the release touched. For a pack that downloads a binary,
+every step below has caught a real defect at least once.
+
+```bash
+"$AT" plugin install <pack>
+"$AT" plugin status                       # does it report itself working?
+find "$HOME/.softspark/ai-toolkit" -path '*<pack>*'   # what actually landed
+```
+
+**Verify:**
+- [ ] `plugin status` distinguishes *installed* from *working*, not just present
+- [ ] For a binary pack: the binary runs and reports the pinned upstream version
+- [ ] The hook is registered in `~/.claude/settings.json` with the pack's `_source`
+
+**Does it do its job?** Presence is not function. Drive the hook directly:
+
+```bash
+printf '%s' '{"tool_name":"Bash","tool_input":{"command":"git status"}}' \
+  | bash "$HOME/.softspark/ai-toolkit/hooks/plugin-<pack>-<hook>.sh"
+```
+
+**Update path:**
+
+```bash
+"$AT" plugin update --editor claude --all            # current version: silent
+"$AT" plugin update --editor claude --all --dry-run  # says "up to date"
+# force a stale marker, then confirm it updates and re-records:
+python3 -c "
+import json, pathlib, os
+p = pathlib.Path(os.environ['HOME'] + '/.softspark/ai-toolkit/plugins.json')
+d = json.loads(p.read_text()); d['targets']['claude']['versions']['<pack>'] = '0.0.9'
+p.write_text(json.dumps(d, indent=2))
+"
+"$AT" plugin update --editor claude --all            # reports 0.0.9 -> <version>
+"$AT" update                                          # core update leaves a current pack alone
+```
+
+**Removal must be complete:**
+
+```bash
+"$AT" plugin remove <pack>
+find "$HOME/.softspark" -path '*<pack>*' | wc -l      # must be 0
+python3 -c "
+import json, pathlib, os
+d = json.loads(pathlib.Path(os.environ['HOME'] + '/.claude/settings.json').read_text())
+print('hook still present:', '<pack>' in json.dumps(d.get('hooks', {})))
+"
+"$AT" plugin install <pack>                           # re-install must work
+```
+
+## Phase 6: The degraded path
+
+**This is the step most worth keeping.** A pack that fetches anything can fail
+to fetch, and the failure mode must be inert rather than broken or silent.
+
+No pack in the toolkit fetches anything today. Run this phase if one ever does
+again, pointing its source-override variable at a dead URL:
+
+```bash
+"$AT" plugin remove <pack>
+<PACK>_RELEASE_BASE_URL="file:///nonexistent" "$AT" plugin install <pack>
+```
+
+**Verify:**
+- [ ] Install reports the failure in words a user can act on, and does not claim success
+- [ ] No partial artifact is left behind
+- [ ] The hook is still wired, and passes commands through untouched
+- [ ] `plugin status` says the pack is inert and names the fix
+- [ ] Re-installing without the broken source recovers
+
+## Phase 7: Prove the real environment is untouched
+
+```bash
+python3 -c "
+import json, pathlib
+d = json.loads(pathlib.Path.home().joinpath('.softspark/ai-toolkit/plugins.json').read_text())
+print('plugins.json:', d['targets']['claude'])
+p = pathlib.Path.home() / '.claude/settings.json'
+print('pack hook leaked into real settings:', '<pack>' in json.dumps(json.loads(p.read_text()).get('hooks', {})) if p.exists() else False)
+print('pack paths in real ~/.softspark:', len(list(pathlib.Path.home().joinpath('.softspark').rglob('*<pack>*'))))
+"
+```
+
+All three must show the pre-test state.
+
+## Phase 8: Clean up
+
+`guard-destructive.sh` blocks `rm -rf` on a `PreToolUse` hook, so removal goes
+through an enumerated delete that reports what it removed:
+
+```bash
+python3 -c "
+import pathlib, shutil
+sb = pathlib.Path('$SB')
+assert sb.is_dir() and str(sb).startswith(('/tmp', '/var/folders')), sb
+n = sum(1 for _ in sb.rglob('*') if _.is_file())
+shutil.rmtree(sb)
+print(f'removed {sb} ({n} files)')
+"
+```
+
+## Success criteria
+
+| Area | Criterion |
+|---|---|
+| Supply chain | `predicateType == https://slsa.dev/provenance/v1` |
+| CLI | `--version` equals the tag, `--help` renders |
+| Health | `doctor` after `install`: 0 errors, 0 warnings |
+| Catalog | `plugin list` count matches `app/plugins/` |
+| Pack install | Binary present, runs, digest verified, hook registered |
+| Pack function | Driving the hook produces the expected effect, not just exit 0 |
+| Pack update | Current version silent; stale version updates and re-records |
+| Pack removal | Zero residue in `~/.softspark` and `settings.json`; re-install works |
+| Degraded path | Fetch failure is inert, loud in status, and recoverable |
+| Isolation | Real `~/.claude` and `~/.softspark` byte-identical to pre-test |
+
+## Related
+
+- [Release Preparation](release-preparation-sop.md) — run before tagging
+- [Release Verification](release-verification-sop.md) — the maintainer-install checks
+- [rtk-pack Retirement](../history/completed/rtk-pack-retirement-20260727.md) — what happened the one time this SOP was written and not run

package/kb/procedures/release-preparation-sop.md

@@ -3,9 +3,9 @@ title: "SOP: Release Preparation"
 category: procedures
 service: ai-toolkit
 tags: [sop, release, version, publish, changelog, semver, provenance, sarif, ecosystem, shellcheck]
-version: "1.11.1"
+version: "1.11.2"
 created: "2026-04-10"
-last_updated: "2026-07-24"
+last_updated: "2026-07-27"
 description: "Step-by-step checklist for preparing a new ai-toolkit release — ecosystem-sync drift check, version sync, changelog, artifact regeneration, validation, and tagging. Run BEFORE every git tag. Includes mandatory Provenance, SARIF, and checksum-pin checks added in v2.8.0, the single-run npm test discipline added in v1.8.0, the ecosystem-sync gate added in v1.9.0, the registry-vs-generators drift gate added in v1.10.0, and the mandatory pre-tag ShellCheck gate added in v1.11.0 (publish.yml does not run ShellCheck, so a hook lint failure can publish while reddening main CI — see the v4.5.1 postmortem in Phase 5)."
 ---
 
@@ -250,12 +250,17 @@ Add entry at the top of `CHANGELOG.md` (after the header, before previous releas
 
 ## Phase 4: Regenerate Artifacts
 
+Use the npm scripts, not the generators directly:
+
 ```bash
-python3 scripts/generate_agents_md.py > AGENTS.md
-python3 scripts/generate_llms_txt.py > llms.txt
-python3 scripts/generate_llms_txt.py --full > llms-full.txt
+npm run generate:agents   # AI_TOOLKIT_NO_CUSTOM_RULES=1 python3 scripts/generate_agents_md.py > AGENTS.md
+npm run generate:llms     # llms.txt + llms-full.txt
 ```
 
+`generate:agents` sets `AI_TOOLKIT_NO_CUSTOM_RULES=1`. Running
+`generate_agents_md.py` bare picks up whatever is registered in the maintainer's
+own `~/.softspark/ai-toolkit/rules/`, which then ships inside `AGENTS.md`.
+
 Check if anything actually changed:
 
 ```bash
@@ -290,7 +295,15 @@ diff \
   && echo "OK: registry matches filesystem" \
   || { echo "DRIFT: update supported-tools-registry.md before tagging"; exit 1; }
 
-# Run npm test ONCE, cache output, parse from file. The suite is 900+ bats
+# Stage first IF this release adds or deletes a kb/ file. The test
+# "npm package KB files match the tracked release set" compares `git ls-files
+# kb` (the index) against what `npm pack` sees (the working tree), so an
+# unstaged addition reads as "extra" and an unstaged deletion as "missing".
+# Phase 6 stages, and it runs after this one, so the ordering fails the test
+# for any release that touches the KB. Staging early costs nothing.
+git status --porcelain kb/ | grep -qE '^(\?\?| D|\?M)' && git add -A kb/
+
+# Run npm test ONCE, cache output, parse from file. The suite is 1400+ bats
 # cases — rerunning it per check wastes minutes. Do not pipe npm test into
 # tail/grep multiple times in the same session.
 npm test > /tmp/npm-test.log 2>&1
@@ -447,7 +460,7 @@ git push origin --delete vX.Y.Z
 | 13 | ShellCheck hooks | `shellcheck --severity=warning app/hooks/*.sh` | Exit 0, no output (mirrors ci.yml; publish.yml does NOT run it) |
 | 14 | Provenance flag check | `grep -- '--provenance' .github/workflows/publish.yml` | Present |
 | 15 | Checksum-pin backfill | `sources.json` entries all have `sha256` | No unpinned URL sources |
-| 16 | Tests | `npm test` | All pass |
+| 16 | Tests | `git add -A kb/` if the KB changed, then `npm test` | All pass |
 | 17 | Commit | `git commit` | Clean working tree |
 | 18 | Tag | `git tag vX.Y.Z` | Tag exists |
 | 19 | Push | `git push origin main --tags` | CI triggered with `id-token: write` |

package/kb/reference/cli-reference.md

@@ -58,7 +58,7 @@ Usage: ai-toolkit <command> [options]
 |---------|-------------|
 | `plugin list` | Show available plugin packs with install status |
 | `plugin install <name> [--editor claude\|codex\|all]` | Install a plugin pack for Claude Code and/or Codex (`claude` means Claude Code, not the Claude app) |
-| `plugin install --all [--editor claude\|codex\|all]` | Install all 11 plugin packs |
+| `plugin install --all [--editor claude\|codex\|all]` | Install all 12 plugin packs |
 | `plugin update <name> [--editor claude\|codex\|all]` | Update a plugin pack (remove + reinstall, preserves data) |
 | `plugin update --all [--editor claude\|codex\|all]` | Update all installed plugin packs |
 | `plugin clean <name> [--days N]` | Prune old plugin data (default: 90 days) |

package/kb/reference/plugin-pack-conventions.md

@@ -72,10 +72,10 @@ Required keys:
 ## CLI Management
 
 ```bash
-ai-toolkit plugin list               # show all 11 packs with install status
+ai-toolkit plugin list               # show all 12 packs with install status
 ai-toolkit plugin install --editor claude <name>   # Claude Code global target
 ai-toolkit plugin install --editor codex <name>    # Codex global target
-ai-toolkit plugin install --editor all --all       # install all 11 packs for both runtimes
+ai-toolkit plugin install --editor all --all       # install all 12 packs for both runtimes
 ai-toolkit plugin update --editor all --all        # update all installed packs
 ai-toolkit plugin clean <name>       # prune data older than 90 days (default)
 ai-toolkit plugin clean <name> --days 30  # prune data older than 30 days
@@ -139,6 +139,30 @@ ai-toolkit plugin status --editor all              # show installed packs with r
 | `swift-pack` | swift | 0 | 1 | 0 | Swift patterns |
 | `ruby-pack` | ruby | 0 | 1 | 0 | Ruby patterns |
 
+Every pack here is content that ships in this repository. None fetches anything
+at install time.
+
+**A pack that downloads a binary has been tried once and retired.** `rtk-pack`
+(v4.18.0, removed in v4.19.0) fetched a checksum-pinned artifact in
+`scripts/init.py`, declared platform assets and digests in `plugin.json`, and
+rewrote commands at `PreToolUse`. Read
+`kb/history/completed/rtk-pack-retirement-20260727.md` before proposing another
+pack of that shape; the two defects that killed it were both invisible to
+`validate.py --strict` and to the pack's own status check.
+
+What survives from that work and applies to any pack:
+
+- **A pack reports its own health.** `scripts/status.py` is picked up
+  generically by `plugin status`, replacing what used to be a hardcoded
+  `if name == "memory-pack"` branch. A status check must distinguish *installed*
+  from *working*, and prove the working part by exercising it rather than by
+  checking that files exist.
+- **A pack may declare `supported_editors`.** Without it a pack installs on
+  every runtime and silently does nothing on the ones it was never built for.
+- **The manifest schema tolerates extra keys.** Anything additive is unvalidated,
+  so a malformed block fails at install time rather than in `validate.py
+  --strict`. Do not rely on the schema to catch it.
+
 ## Optional Hook Modules
 
 `enterprise-pack` provides two optional hook modules: