@softspark/ai-toolkit 4.17.0 → 4.19.0

package/kb/history/completed/rtk-pack-integration-20260726.md

@@ -0,0 +1,710 @@
+---
+title: "Plan: rtk Pack Integration"
+category: planning
+service: ai-toolkit
+tags:
+  - rtk
+  - plugin-pack
+  - token-reduction
+  - vendored-binaries
+  - cross-build
+  - upstream-sync
+doc_type: postmortem
+status: completed
+created: "2026-07-26"
+last_updated: "2026-07-26"
+completion: "100% — phases 0, 1, 2, 4, 5 delivered; phase 3 cut on the measurement"
+shipped_in: "v4.18.0"
+pinned_upstream: "v0.44.0"
+description: "Integrate rtk as an opt-in ai-toolkit plugin pack: binaries cross-built from source in our CI with telemetry disabled, hosted on our own GitHub Release, fetched and checksum-verified at pack install, auto-update on ai-toolkit update, and an SOP for tracking upstream releases. Custom filter presets were cut on the measurement (0.008% of input tokens). Phase 0 validated the premise against v0.44.0 on the full transcript corpus before any build work."
+---
+
+# rtk Pack Integration
+
+**Completed 2026-07-26, shipped in v4.18.0.** Kept whole rather than summarised,
+because the value here is the measurements and the things that turned out to be
+false, not the plan structure. Live procedure:
+`kb/procedures/rtk-upstream-sync-sop.md`.
+
+Headline: rtk-pack ships opt-in, built from source with telemetry compiled out,
+and saves a **measured 0.0615% of input tokens** against a kill number of 0.05%
+published before the measurement. Phase 3 (our own filter presets) was cut at
+0.008%. The projection this plan published for itself was about twice too
+optimistic, and section 10.1 says why.
+
+## 1. Decision
+
+Ship `rtk` to ai-toolkit users as an **opt-in plugin pack**, with the binary
+supply chain owned end to end by us.
+
+Five choices, made and locked:
+
+| Question | Decision |
+|---|---|
+| Delivery | Plugin pack (`rtk-pack`), never a native feature |
+| Binary hosting | Our own GitHub Release, fetched and SHA-256 verified at pack install |
+| Binary provenance | Cross-built from upstream source in our CI, `RTK_TELEMETRY_URL` unset |
+| Pinned upstream tag | **v0.44.0** (2026-07-26). v0.43.0 was ruled out on evidence, see section 2 |
+| Upstream tracking | An SOP that rebuilds, verifies, and re-publishes when rtk ships a release |
+
+Upstream: https://github.com/rtk-ai/rtk, Apache-2.0, Rust 1.91 plus a C
+toolchain (rusqlite `bundled` compiles SQLite from source).
+
+## 2. Phase 0: premise validated before building anything
+
+The retired output filter validated its design across five phases and its
+premise not at all until the fifth. This plan runs premise validation first.
+Everything below is measured, not estimated.
+
+### 2.1 Why v0.43.0 was ruled out
+
+Two structural facts, read from the source of both tags:
+
+- **Custom TOML filters are not wired into the rewrite path in v0.43.0.**
+  `src/discover/registry.rs` at v0.43.0 contains zero references to
+  `toml_filter`. In v0.44.0 they appear at `registry.rs:1047,1055,1058`.
+  *This reason no longer applies:* section 7 cut our own presets, so nothing we
+  ship depends on that wiring. It is recorded because it was one of the two
+  arguments made at pin time, and the pin now rests on the second alone.
+- **`pipeline_final_safe` does not exist in v0.43.0.** It arrives in v0.44.0
+  (`rules.rs:7,115,124`, `registry.rs:605,659,864,1069`). v0.43.0 rewrote the
+  first pipeline stage; v0.44.0 rewrites the last. On this traffic 56.5% of
+  `rtk grep` hits arrive through the pipeline-final path, so they do not exist
+  at all on v0.43.0.
+
+### 2.2 The port models v0.44.0, not v0.43.0
+
+`rtk_port.py` is re-validated by extracting the `assert_eq!` assertions from
+each tag's `registry.rs` `#[cfg(test)]` block and replaying them:
+
+| Port checked against | Assertions | Exact-string agreement |
+|---|---:|---:|
+| **v0.44.0** | 203 | **203/203** |
+| v0.43.0 | 160 | 147/160 |
+
+The 13 v0.43.0 failures are all the same pipeline inversion. The port also
+passes the 15 assertions v0.44.0 added after the port was written. Pinning
+v0.44.0 makes the existing measurement correct rather than requiring rework.
+
+`registry.rs:590-592` returns `Some(unchanged)` for a simple already-`rtk`
+command; the port's equivalent bail must be mapped to that, not to `None`.
+
+### 2.3 The recorded numbers came from an 11% sample
+
+`measure_rtk.py` sets `N_FILES = 134` and takes the most recently modified
+transcripts. The available pool is **1224 transcripts**. At n=134 the projected
+saving swings **8.8x** across windows (0.071% to 0.628%); at n>=408 it converges
+to within **1.05x**. The spread recorded in earlier revisions of this plan was
+sampling noise.
+
+### 2.4 The port over-counted, and by how much
+
+Both real entry points call `contains_unattestable_construct` **before**
+`registry::rewrite_command`: `hook_cmd.rs:149-151` (Defer) and
+`rewrite_cmd.rs:54-56` (Passthrough). Any command carrying a command
+substitution, a process substitution, or a redirect with a file target is
+rejected whole. Upstream's rewrite tests call `rewrite_command` directly, so a
+port validated only against them passes while still over-counting.
+
+The gate is ported from `lexer.rs:295-347` and validated **35/35** against
+upstream's own assertions at `lexer.rs:1186-1276`. Applying it costs 3.1%
+relative coverage.
+
+### 2.5 Corrected numbers
+
+Full corpus, 1224 transcripts, 552 MB, 106 MB of tool results, 26.6 MB of
+successful Bash, entry gate applied:
+
+| Metric | Earlier revision (n=134, ungated) | Phase 0 (n=1224, gated) |
+|---|---:|---:|
+| Coverage of successful Bash bytes | 31.5% | **35.17%** |
+| Coverage of all tool-result bytes | 9.69% | **8.77%** |
+| Ceiling: tool results as share of input volume | 5.52% | **4.54%** |
+| Projection at rtk's 60-90% claim | 0.32-0.48% | **0.239-0.359%** |
+| Projection, mechanism modelled | 0.15-0.21% | **0.117-0.164%** |
+| Reach of our own TOML presets (the number that cut section 7) | 1.91% of Bash | **1.08% of Bash** |
+| `Read` share of tool-result bytes | 53.8% | **62.8%** |
+
+Modelled effectiveness per family against the claim: `rtk grep` 75% claimed and
+**9.0%** modelled; `rtk read` 60% claimed and **0.0%** modelled; `rtk rg` 75%
+and 30%; `rtk make` 65% and 16%; `rtk jq` 74%.
+
+Largest unaddressed buckets, as a share of successful Bash bytes: pipeline not a
+rewritable final stage **39.78%**, ignored by rtk **10.72%**, entry gate
+**8.29%**, multiline script **4.71%**.
+
+### 2.6 Open question 3 answered: no
+
+Patching `pipeline_final_safe` for `head` and `tail` unlocks 10.56 MB, 39.96% of
+Bash bytes. **88.3% of that (9.33 MB) routes to `rtk read`, measured at 0.00%
+saving** on this traffic because it returns files verbatim at the default
+`--level none`. The largest reachable gap is reachable and worthless. The
+argument for maintaining a fork does not survive its own measurement.
+
+## 3. Why a pack rather than a native feature
+
+- **Opt-in is structural, not a policy.** Plugin packs must not be
+  auto-installed [PATH: kb/reference/plugin-pack-conventions.md]. The retired
+  output filter shipped `off` by default and still had to be wired into every
+  install path, which is why v4.17.0 needed a bespoke migration cleanup.
+- **Removal is already solved.** `plugin remove` strips hook entries from
+  `settings.json`, deletes owned `plugin-<pack>-*` assets, and leaves core and
+  user files alone.
+- **The core package stays lean.** ai-toolkit is 4.4 MB unpacked; the platform
+  artifacts are ~19.5 MB compressed. They never enter the base package.
+- **There is a working precedent.** `memory-pack` ships hooks, a shared script
+  directory, an install-time init script, its own data with retention, and a
+  clean uninstall. `rtk-pack` is the same shape with a binary instead of a
+  database.
+
+## 4. Architecture
+
+```text
+app/plugins/rtk-pack/
+├── plugin.json          # pinned rtk version + per-platform SHA-256
+├── README.md
+├── hooks/
+│   └── rewrite.sh       # PreToolUse; delegates to the fetched binary
+└── scripts/
+    ├── init.py          # platform detect → fetch → verify → install
+    └── status.py        # reports binary, digest and hook wiring
+```
+
+Runtime layout on a user machine:
+
+```text
+~/.softspark/ai-toolkit/
+├── hooks/plugin-rtk-pack-rewrite.sh        # installed hook
+└── plugin-scripts/rtk-pack/
+    ├── bin/rtk                              # verified binary, 0755
+    └── version.json                         # pinned version + recorded digest
+```
+
+Everything the pack writes lives under `plugin-scripts/rtk-pack/`, which
+`plugin remove` deletes wholesale (`plugin.py:500-503`). The pack owns no state
+outside the toolkit data directory: no `filters.toml`, no entry in rtk's trust
+store, nothing in an OS config directory. That is a consequence of cutting
+section 7 and it is what keeps removal complete.
+
+Binary supply chain:
+
+```text
+upstream tag  →  our CI cross-build (RTK_TELEMETRY_URL unset)
+              →  our GitHub Release  softspark-rtk-<upstream>-<build>
+              →  SHA-256 recorded in plugin.json
+              →  plugin install: detect platform, fetch, verify, chmod
+```
+
+## 5. Phase 1: build pipeline and binary release
+
+**Outcome:** we can produce, from an upstream tag, a set of binaries that
+provably make no network calls of their own.
+
+**Targets: five, not six.** `x86_64-apple-darwin`, `aarch64-apple-darwin`,
+`x86_64-unknown-linux-musl`, `aarch64-unknown-linux-gnu`,
+`x86_64-pc-windows-msvc`. Earlier revisions added `x86_64-unknown-linux-gnu` on
+the reasoning that upstream does not ship it. It is dropped: upstream's own
+Homebrew formula routes Linux x86_64 to the **musl** tarball
+(`release.yml:314-316`), a static musl binary runs on glibc, and upstream has
+never validated a gnu x86_64 artifact, so we would be first to ship one and
+would own all its breakage plus an uncontrolled glibc floor inherited from the
+runner image.
+
+Build constraints, all read from the pinned checkout:
+
+- **Pin the toolchain.** Upstream uses `dtolnay/rust-toolchain@stable`
+  (`release.yml:65,116`) with `warnings = "deny"` (`Cargo.toml:70-72`) and ships
+  no `rust-toolchain` file. A future rustc lint turns our builds red with no
+  change on either side. We pin an exact version at or above 1.91.
+- **Windows must build natively.** `build.rs:6-13` emits
+  `cargo:rustc-link-arg=/STACK:8388608` under `#[cfg(windows)]`, which in a
+  build script is a **host** predicate. Cross-building Windows from Linux
+  silently drops the 8 MiB stack reservation that upstream's own comment says is
+  what makes `rtk.exe --version`, `--help`, and hook entry points start
+  reliably. Build on `windows-latest`, or pass the link-arg explicitly.
+- **Both darwin targets build on `macos-latest` (arm64).** `macos-13` is the
+  last x86_64 macOS image and GitHub is winding it down: on runs 30212577757 and
+  30214341444 that job sat queued indefinitely while every other target
+  finished. Upstream builds both on arm64 too, but never runs its x86_64
+  artifact. We do: the verifier detects Rosetta 2 and executes it, and reports
+  `inconclusive` rather than `pass` if it cannot.
+- **Every target needs a target-capable C compiler.** rusqlite `bundled`
+  (`Cargo.toml:26`) is not switchable off; there is no `[features]` table and no
+  `cfg(feature` in `src/`. There is no pure-Rust escape route.
+- **aarch64-linux is the hard case, but upstream already solves it** with
+  `cross: true` on `ubuntu-latest` (`release.yml:51-54`). Copy that. Upstream
+  sets only the linker env var and lets the `cc` crate guess the C compiler;
+  export `CC_aarch64_unknown_linux_gnu` and `AR_aarch64_unknown_linux_gnu`
+  explicitly so a runner image change surfaces as a clear error.
+- **Run our own blocking `cargo audit`.** Upstream's is advisory: `ci.yml:88-96`
+  swallows failures into a warning. Rebuilding from a tag inherits that
+  lockfile, and on v0.44.0 it inherits four advisories across three of its 203
+  crates. The audit and the build jobs apply the same remediation, so we audit
+  the dependency set we ship rather than a different one:
+
+  | Crate | Advisory | Disposition |
+  |---|---|---|
+  | `anyhow` 1.0.102 | RUSTSEC-2026-0190 | `cargo update` to 1.0.103, in range |
+  | `crossbeam-epoch` 0.9.18 | RUSTSEC-2026-0204 | `cargo update` to 0.9.20, in range |
+  | `quick-xml` 0.37.5 | RUSTSEC-2026-0194, RUSTSEC-2026-0195 | ignored with reasons |
+
+  The quick-xml fix lands only in 0.41.0 while rtk pins `"0.37"` as a direct
+  dependency (`Cargo.toml:34`), so taking it means editing `Cargo.toml` and the
+  calling code in `src/cmds/dotnet/dotnet_trx.rs`. That would break the promise
+  that the only difference from an upstream build is the undefined telemetry
+  endpoint. The reachable surface is narrow: quick-xml parses .NET TRX test
+  output only, the input is a report produced locally by the user's own test
+  run, and both advisories are availability-only (CVSS `C:N/I:N/A:H`). The SOP
+  re-checks this on every sync and deletes the ignores once upstream moves to
+  quick-xml 0.41 or later.
+- A target that will not build is dropped, not faked.
+
+### 5.1 Proving the binary is silent
+
+The endpoint is compile-time only: `option_env!("RTK_TELEMETRY_URL")` at
+`telemetry.rs:16` and `telemetry_cmd.rs:176`, with the token at
+`telemetry.rs:17` and `telemetry_cmd.rs:189`. Nothing supplies a default:
+`build.rs` emits no `cargo:rustc-env`, `TelemetryConfig` has no URL field, and
+no runtime `std::env::var` resolves the endpoint. Upstream injects it only in
+`release.yml:85-86,124-125,151-152`. Building without it yields a binary whose
+telemetry destination cannot be re-enabled by any env var or config at runtime.
+
+**The acceptance criterion in earlier revisions was untestable and is replaced.**
+"No telemetry symbols in the binary" cannot be checked: the guard is a runtime
+`if TELEMETRY_URL.is_none()` on a const (`telemetry.rs:23-26`), not a `#[cfg]`,
+so the code compiles in and is only eliminated by LLVM as an optimisation; and
+`Cargo.toml:51` sets `strip = true`, which removes the symbols regardless of
+whether the code is present. A symbol check would pass for the wrong reason.
+
+What we assert instead:
+
+0. **The artifact was actually started.** Every other assertion is worthless on
+   a binary nobody executed, and two of the five targets are not native to their
+   runner. `aarch64-unknown-linux-gnu` runs under `qemu-user` with the cross
+   sysroot passed as `-L`, and `x86_64-apple-darwin` runs under Rosetta 2 on the
+   arm64 runner. A target that cannot be started reports `inconclusive`, never
+   `pass`.
+1. **Build gate.** `RTK_TELEMETRY_URL` and `RTK_TELEMETRY_TOKEN` are unset in
+   the build environment, asserted in CI before `cargo build`, with a clean
+   target directory per build.
+2. **Offline smoke run.** Each artifact runs its real command surface with no
+   network route available and makes zero outbound connections. This is
+   Linux-only: `unshare -rn` has no unprivileged equivalent on macOS or Windows
+   runners, and Ubuntu 24.04's
+   `kernel.apparmor_restrict_unprivileged_userns=1` means even there it needs
+   `sudo`. On the first run this assertion silently degraded to a skip on every
+   target while the verdict still read `pass`, so the manifest now records which
+   isolator was used and a target that could not be started at all reports
+   `inconclusive` rather than `pass`.
+3. **Filesystem assertion.** No telemetry state is created under the resolved
+   data directory.
+4. **Drift detection.** Record artifact size and a string-allowlist hash per
+   target and fail the build on unexplained drift, since whether `ureq`,
+   `rustls`, `ring` and `webpki-roots` are actually eliminated is an LLVM
+   outcome under `lto = true`, not a guarantee.
+
+Three findings that must reach the pack README:
+
+- **`rtk cc` shells out to `npx --yes ccusage`** when `ccusage` is not on PATH
+  (`ccusage.rs:104-119`). That is a runtime npm fetch and third-party code
+  execution. It is not telemetry and not automatic, but it is outbound network
+  from a binary we tell users makes no network calls.
+- **`RTK_TELEMETRY_DISABLED` only accepts the exact string `1`**
+  (`telemetry_cmd.rs:31-33`, locked by the test at `:222-229`). `true` and `yes`
+  are silent no-ops. Never write anything else anywhere in the pack.
+- **`rtk telemetry forget` on an endpoint-free build prints a misleading
+  failure** telling the user to email upstream to complete erasure
+  (`telemetry_cmd.rs:158-169`), when nothing was ever sent.
+
+Apache-2.0 obligations ship alongside: upstream `LICENSE`, a `NOTICE` file
+(upstream has none), and a statement that the only build-time change is leaving
+the telemetry endpoint undefined. **Do not redistribute upstream
+`DISCLAIMER.md` verbatim**: `DISCLAIMER.md:25` states telemetry is collected by
+default, which the code contradicts and which a compliance reviewer would read
+as a reason to block the pack. `docs/TELEMETRY.md:180` calls the mechanism
+"compile-time gating" where "all telemetry code is dead", which is imprecise for
+the same reason our own criterion changed.
+
+**Success criteria:** five artifacts build from the pinned tag; each runs
+`rtk --version` on its target; the silence assertions pass or say plainly that
+they did not run; checksums published.
+
+### 5.2 Result, run 30214882862
+
+All five targets build and verify. Every artifact was actually started, none
+shipped on a skipped assertion.
+
+| Target | `runs` | `offline` |
+|---|---|---|
+| `aarch64-apple-darwin` | native | not applicable |
+| `x86_64-apple-darwin` | translated via Rosetta 2 | not applicable |
+| `x86_64-unknown-linux-musl` | native | passed under `sudo -n unshare -rn` |
+| `aarch64-unknown-linux-gnu` | emulated via `qemu-aarch64-static` | passed under `sudo -n unshare -rn` |
+| `x86_64-pc-windows-msvc` | native | not applicable |
+
+No TLS markers in any artifact, so `ureq`, `rustls`, `ring` and `webpki-roots`
+appear to be eliminated under LTO once the endpoint const is `None`. That is an
+optimisation outcome rather than a guarantee, which is why the fingerprint is
+recorded per target and drift fails the build.
+
+**Four of five targets are bit-reproducible.** Across independent runs with
+identical inputs, `aarch64-apple-darwin`, `x86_64-apple-darwin`,
+`aarch64-unknown-linux-gnu` and `x86_64-unknown-linux-musl` produced
+byte-identical binaries. The concern about unset `trim-paths` embedding registry
+paths does not materialise on these runners.
+
+`x86_64-pc-windows-msvc` did not, and the cause was bounded: **24 bytes out of
+8,018,432 differed**, in five regions. Four were the same two-byte value at
+`0x100` and in the debug directory, which is the MSVC link timestamp; the fifth
+was a 16-byte CodeView GUID. The code was identical; only the link stamp varied.
+
+`-C link-arg=/Brepro` is now set for the Windows target, which makes the linker
+derive that timestamp from content rather than the clock. **The shipped
+`softspark-rtk-v0.44.0-1` artifacts predate that change**, so the Windows digest
+in `plugin.json` is from a non-deterministic link; the flag takes effect on the
+next rebuild, which will produce a different Windows digest for the same source.
+
+This matters for the sync SOP: a changed digest means changed input rather than
+build noise. That inference holds on four targets today and on all five from the
+next rebuild onward.
+
+Note that the **archives** are not reproducible even where the binaries are,
+because gzip records a timestamp. Compare extracted binaries, never the
+tarballs.
+
+Three defects the run surfaced, all now fixed and guarded:
+
+- The gnu target is dynamically linked, so `qemu-user` needs the cross sysroot.
+  It is passed as `-L` rather than `QEMU_LD_PREFIX`, because the offline check
+  runs under `sudo` and `env_reset` drops the variable.
+- The offline check compared a plain run against a namespaced one, so `sudo`'s
+  environment handling was confounded with the network variable and a harness
+  failure was reported as a behaviour difference in the binary. Both sides now
+  run in a namespace and differ only in `-n`.
+- `7z` stored the full relative path, so the Windows zip held
+  `rtk-src/target/<triple>/release/rtk.exe` while every tar.gz held a bare
+  `rtk`. The pack installer extracts all five the same way, so this was a
+  silent install failure on one platform. A `collect` step now asserts every
+  archive holds exactly one flat entry.
+
+## 6. Phase 2: the pack
+
+**Outcome:** `ai-toolkit plugin install rtk-pack` produces a working setup and
+`plugin remove` leaves nothing behind.
+
+- `scripts/init.py` detects platform and libc, fetches the matching artifact,
+  verifies its SHA-256 against `plugin.json`, and installs to
+  `plugin-scripts/rtk-pack/bin/rtk`. A mismatch aborts and removes the partial
+  download. No network at runtime, only at install.
+- Failure to fetch is not an install failure: the pack degrades to inert and
+  says so, matching how the core behaves when `jq` is missing.
+- The hook is wired at `PreToolUse` through the existing pack hook mechanism, so
+  `plugin remove` strips it via the `_source` marker.
+- `plugin status` dispatches to the pack's own `scripts/status.py`, which is
+  generic rather than another hardcoded `if name == ...` branch. It reports the
+  binary, the install record, the live version, and hook wiring, and announces a
+  missing binary as inert rather than letting a green install imply it works.
+
+### 6.1 No trust step, because the pack ships no filters
+
+Section 7 is cut, so the pack writes no `filters.toml` and takes no entry in
+rtk's trust store. That removes the single highest-ranked risk this plan
+carried: an untrusted or content-changed preset produces **zero output on the
+command path with no warning, no stderr line, and no non-zero exit**
+(`toml_filter.rs:220-221`, `:450-458`) — installed, apparently fine, doing
+nothing, which is exactly the shape of failure that retired the in-house filter.
+
+The pack instead reports whether it is actually working. `scripts/status.py`
+distinguishes installed from functioning: a missing binary is announced as
+`MISSING — the hook is inert and every command runs unchanged` rather than
+inferred from a green install.
+
+**If presets are ever revisited, these are the constraints, kept because they
+were expensive to establish:**
+
+- Trust is a byte-exact SHA-256 (`trust.rs:142-163`), so every write, upgrade,
+  repair, trailing newline, or git CRLF normalisation invalidates it.
+- `rtk trust --yes` is indiscriminate: it trusts every gated file that exists,
+  including a `.rtk/filters.toml` an attacker committed into the CWD
+  (`trust.rs:262-305`, project path is CWD-relative at `:207`).
+- It exits 1 both when there is nothing to trust and when the file is invalid
+  TOML (`trust.rs:307-312`), so `set -e` aborts on the benign case.
+- There is no `rtk doctor`, and `rtk trust --list` never re-hashes
+  (`trust.rs:242-256`). Verification means reading the store and recomputing.
+- The global path is not `~/.config/rtk/filters.toml` everywhere: it comes from
+  `dirs::config_dir()` (`trust.rs:206-216`), which on macOS is
+  `~/Library/Application Support`. rtk's own error text hardcodes the Linux path
+  and is wrong there.
+- One global file, no include mechanism, so injection would have to follow the
+  merge-friendly install model [PATH: kb/reference/merge-friendly-install-model.md]
+  with comment markers rather than owning the file.
+
+**Success criteria:** install, status, update, remove, and re-install are
+idempotent; a machine without the pack behaves exactly as today; uninstalling
+ai-toolkit removes every pack artifact.
+
+## 7. Phase 3: cut
+
+**Cut on the measurement, 2026-07-26.** Custom TOML presets reach **1.08% of
+Bash bytes**, which is 0.27% of tool-result bytes. At a generous 60% reduction
+that is **0.008% of input tokens**, roughly one part in twelve thousand.
+
+Against that: presets would have required a trust step that fails silently, a
+doctor check that recomputes digests, marker injection into a file the user may
+also edit, re-trusting after every write, and a per-upstream-release review of a
+DSL whose `deny_unknown_fields` rejects the entire file on one unknown key. The
+maintenance surface is large, the payoff is not measurable, and every item on
+that list is a way for the pack to look installed while doing nothing.
+
+The pack therefore ships upstream rtk's built-in filters and nothing of our own.
+The 35.2% coverage quoted in section 2.5 is already what those built-ins
+deliver; it does not shrink as a result of this cut.
+
+Two consequences recorded so they are not rediscovered:
+
+- The pack owns no state outside `~/.softspark/ai-toolkit/plugin-scripts/rtk-pack/`,
+  which is what makes `plugin remove` complete.
+- One of the two arguments for pinning v0.44.0 was that custom filters are only
+  wired into the rewrite path there. That argument is now moot; the pin rests on
+  `pipeline_final_safe`, which carries 56.5% of `rtk grep` hits on this traffic
+  and does not exist in v0.43.0. See section 2.1.
+
+Reopening this is a plan change, not a task: it needs a workload where rtk's
+built-ins measurably under-perform and a preset that measurably closes the gap,
+demonstrated by replay before any code is written.
+
+## 8. Phase 4: auto-update on `ai-toolkit update`
+
+**Outcome:** a user who installed the pack gets the new binary by running the
+update they already run.
+
+Verified gap: `handleUpdate` in `bin/ai-toolkit.js` reads installed modules from
+`state.json` and does not touch `plugins.json`. Nothing propagates to packs
+today, so this is new wiring, not a configuration change.
+
+**Done.** `ai-toolkit update` now calls `plugin update --editor all --all`
+after the core update. Two things had to change first, both generic rather than
+rtk-specific:
+
+- `update_pack` was an unconditional remove-then-reinstall, so wiring it into
+  the core update would have refetched the binary on every run. `plugins.json`
+  now records the pack version installed per editor, and an update whose
+  manifest version matches is a silent no-op. `--force` overrides.
+- State written before versions were tracked has no `versions` map, so every
+  pack reads as stale exactly once and is updated once. That is the intended
+  migration, not a bug.
+
+`--local` leaves packs alone: they live in `~/.softspark/ai-toolkit` and are
+global, while `--local` is project-local config only.
+
+- After the core update completes, `ai-toolkit update` reads `plugins.json` and
+  runs the equivalent of `plugin update` for every **currently installed** pack.
+  Packs that are not installed stay untouched, which preserves adoption rule 1.
+- The rtk-pack update path is: compare the manifest's pinned version against the
+  recorded pack version; if they differ, fetch and verify the new binary and
+  re-install the hook.
+- `--dry-run` reports what would change per pack.
+- A pack update failure warns and continues; it never fails the core update.
+- Bats coverage in `tests/test_plugin_update.bats`: stale version replaced;
+  current version untouched and silent; pack absent means no work; `--dry-run`
+  reports without acting; legacy state with no version map updates once; a
+  corrupt `plugins.json` does not crash the run.
+
+**Success criteria:** the wiring is generic across packs, not rtk-specific;
+`update` remains idempotent; a failed pack update never leaves a half-installed
+binary.
+
+## 9. Phase 5: upstream sync SOP
+
+**Outcome:** a written procedure so tracking upstream is routine rather than a
+research project each time. **Written**, at
+`kb/procedures/rtk-upstream-sync-sop.md`, modelled on the existing
+ecosystem-sync SOP.
+
+The 0.43.0 to 0.44.0 bump is a worked example of why the review step exists: 200
+commits, and every file the SOP names changed, including a semantic inversion in
+pipeline rewriting and the arrival of the trust gate.
+
+Steps the SOP must cover:
+
+1. **Detect.** Check the upstream releases feed for a tag newer than the pinned
+   one. Their stable cadence is roughly two to four weeks, behind a long
+   release-candidate train. Cadence: on demand, plus a check folded into release
+   preparation.
+2. **Review before building.** Read the changelog and diff `src/discover/rules.rs`,
+   `src/discover/registry.rs`, `src/discover/lexer.rs`, `IGNORED_PREFIXES`, the
+   TOML DSL structs in `src/core/toml_filter.rs`, `src/hooks/trust.rs`, and
+   anything touching telemetry. A change to the DSL or to trust handling is a
+   stop-and-think, not a rebuild.
+3. **Re-validate the port.** Re-extract the `rewrite_command` assertions from the
+   new tag's `registry.rs` test block and the gate assertions from `lexer.rs`,
+   and replay both. Anything short of full agreement invalidates every coverage
+   number until the port is fixed.
+4. **Rebuild** all five targets from the new tag with telemetry unset.
+5. **Verify.** Binary runs on each target; the silence assertions pass; the
+   pack's fixtures still produce the expected decisions.
+6. **Publish** a new release in our namespace and record the new digests.
+7. **Bump** `rtk-pack` version in `plugin.json` and note the upstream version it
+   tracks. The bump is what makes `plugin update` fire at all, since a pack whose
+   recorded version still matches is skipped silently.
+8. **Ship** in the next ai-toolkit release; installed packs pick it up through
+   Phase 4.
+9. **Record** the licence position if upstream relicenses or adds a NOTICE.
+
+There is no upstream test asserting network silence, so that property can
+regress on any bump without turning their CI red. We own that test and re-run it
+every time.
+
+**Success criteria:** a maintainer who has never done it can follow the SOP end
+to end; the review step names specific files rather than saying "check for
+breaking changes".
+
+## 10. Verification
+
+The pack must be able to answer "did this help?" with a number.
+
+**The before-and-after design is replaced.** Phase 0 measured its noise floor:
+on a 134-transcript window the projected saving varies by 8.8x while the
+mechanism is unchanged. A before-and-after comparison cannot detect an effect of
+0.117% to 0.164% against that. Two changes make the measurement possible:
+
+- **Measure over the whole transcript pool, not a recency window.** At n>=408
+  the same metric converges to within 1.05x.
+- **Measure by replay, not by elapsed calendar time.** Run real captured tool
+  output through the built binary and compare byte counts directly. That is the
+  method that produced the honest 0% which retired the in-house filter, and it
+  removes the confound of what work the user happened to do that week.
+
+The harness is `replay_rtk.py` in the measurement archive, not a script inside
+the pack. It pairs each successful Bash result with the command that produced
+it, keeps the ones the validated port says rtk would rewrite, and pipes the
+captured bytes through the shipped binary via `rtk pipe -f <filter>`.
+
+Two limits, stated because the number is meaningless without them:
+
+- `rtk pipe` runs the same filter code as the command path but knows less than
+  it does: no result caps, no exit code, no file set. For search families the
+  replay figure is a **lower bound** on what the command path would save.
+- Families with no pipe filter (`rtk read`, `rtk ls`, `rtk wc`) cannot be
+  measured this way at all. They are reported as **unmeasurable**, never folded
+  in as zero, because a zero that is really an absence is how the previous
+  effort talked itself into shipping.
+
+Session-level token accounting, when it is wanted, comes from the session JSONL
+[PATH: scripts/session_token_stats.py], summing all four usage fields: in a
+cached session most context tokens land in the cache fields rather than
+`input_tokens`.
+
+**Kill number, published before the measurement rather than argued after it:**
+if replay on the full corpus shows the shipped binary saving less than 0.05% of
+input tokens, the pack is not worth its maintenance and supply-chain surface,
+and it is retired the way the output filter was.
+
+### 10.1 Result: measured 0.0615%, kill number survived by 23%
+
+Full pool, 28.56 MB of successful Bash output, replayed through the published
+`softspark-rtk-v0.44.0-1` binary:
+
+| | |
+|---|---:|
+| Addressed by rtk | 9.99 MB, **35.00%** of Bash bytes |
+| Measurable through `rtk pipe` | 5.66 MB |
+| After filtering | 4.22 MB |
+| **Measured saving** | **1.44 MB = 360,529 tokens** |
+| As a share of tool-result bytes | 1.35% |
+| **As a share of input tokens** | **0.0615%** |
+
+Against a kill number of 0.05%, the pack survives by a factor of 1.23. That is a
+pass, not a vindication.
+
+**The projection published in section 2.5 was roughly twice too optimistic.**
+It said 0.117% to 0.164%; the measurement says 0.0615%. The error has a single
+identifiable cause: the model credited families it could not simulate with
+rtk's own claimed 60-90%. Replayed, the families that can be measured deliver
+**25.5%** in aggregate.
+
+Per family, measured against modelled and against upstream's claim:
+
+| Family | Measured | Modelled | Claimed |
+|---|---:|---:|---:|
+| `rtk grep` | **22.3%** | 9.0% | 75% |
+| `rtk git` | **33.1%** | — | 70% |
+| `rtk find` | **35.6%** | — | 70% |
+| `rtk rg` | **7.0%** | 30% | 75% |
+| `rtk ruff` | **0.0%** | — | — |
+
+The model **understated** grep by 2.5x and the total still came out high, so the
+two errors are unrelated: coverage modelling was sound, per-family effectiveness
+was guesswork wherever the source was not read closely.
+
+**What the replay does not settle.** 43.3% of addressed bytes (4.33 MB) have no
+`rtk pipe` filter and are unmeasurable, the largest being `rtk read` at 1.55 MB.
+That family is known from source to return files verbatim at the default
+`--level none`, so the headroom is smaller than 43.3% suggests. No extrapolation
+is applied: a zero that is really an absence of measurement is how the previous
+effort talked itself into shipping. For search families the pipe path also lacks
+the command path's result caps, making these figures a lower bound.
+
+**One independent confirmation.** The replay measured coverage at 35.00% of Bash
+bytes; the gated port measured 35.17% on the same pool by a different method.
+The port is right about *which* commands rtk touches even where it was wrong
+about how much each saves.
+
+Reproduce with `replay_rtk.py --pool 1300 --rtk <binary>` in the measurement
+archive.
+
+## 11. Licence and security obligations
+
+- **Apache-2.0.** Ship upstream `LICENSE` with the binaries, add a `NOTICE`, and
+  state the build-time change. Never relabel any upstream file MIT. Do not ship
+  `DISCLAIMER.md` verbatim, see section 5.1.
+- **Telemetry.** Proven absent per build by the four assertions in section 5.1,
+  re-proven on every upstream sync.
+- **Supply chain.** Digests pinned in `plugin.json`, verified on fetch,
+  re-verified by `doctor`. This mirrors the existing checksum-pin discipline for
+  URL-sourced rules and hooks. Run a blocking `cargo audit` on the pinned
+  lockfile, because upstream's is advisory.
+- **Trust boundary.** rtk rewrites commands before execution, which the retired
+  in-house contract explicitly forbade for itself. Adopting it is a conscious
+  reversal of that constraint and its own threat surface: what runs is not what
+  the model asked for. The pack's README must say this plainly, and
+  `plugin install` must not be silent about it. The README must also disclose
+  the `rtk cc` npx path.
+
+  The sharp edge is more specific than "the command changes". rtk evaluates the
+  host's permission rules against the **original** command and applies the
+  verdict to the **rewritten** one: `decide_hook_action(cmd, ...)` takes the
+  command the model asked for, and `permissionDecision: allow` is then emitted
+  for the substituted `rtk …` form (`hook_cmd.rs:405-436`). An allowlist entry
+  for `git status` therefore authorises `rtk git status`, which the user never
+  wrote a rule for. Upstream documents the absence case in its own test
+  (`hook_cmd.rs:1272-1283`): with no matching allow rule, no decision is emitted
+  and the normal prompt stands. Both halves belong in the README.
+
+## 12. Pre-mortem
+
+| Rank | Failure mode | Probability | Impact | Mitigation |
+|---:|---|:---:|:---:|---|
+| — | ~~Presets install but are untrusted, so they silently do nothing~~ | — | — | **Eliminated**, not mitigated: section 7 is cut, so the pack ships no filters and takes no trust-store entry |
+| 2 | Cross-building with bundled SQLite is fragile in CI | Medium | Medium | Copy upstream's `cross` setup for aarch64-linux; five targets not six; a target that will not build is dropped, not faked |
+| 3 | Pre-execution rewriting changes command semantics | Medium | High | Opt-in pack, documented one-flag disable, upstream's own review process, our integration tests on real commands |
+| 4 | Windows binary cross-built without the 8 MiB stack reservation | Medium | High | Build natively on `windows-latest`, or pass the link-arg explicitly. `build.rs:6-13` |
+| 5 | Upstream velocity breaks the rewrite logic the coverage numbers model | Medium | Medium | Pinned version, SOP review step naming exact files, port re-validation as a gate, fork option preserved by Apache-2.0 |
+| 6 | Unpinned toolchain plus `warnings = "deny"` turns builds red with no change | Medium | Low | Pin an exact rustc version; treat bumps as deliberate |
+| 7 | A fetch failure leaves a half-installed pack | Medium | Medium | Verify-then-install, abort and clean on digest mismatch, `doctor` detects drift |
+| 8 | Live saving lands near the projection, not the claim | High | Medium | Section 10 measures it by replay either way, against a published kill number |
+| 9 | The auto-update wiring makes `update` slower or flakier | Low | Medium | Skip when versions match, warn-and-continue on failure, `--dry-run` coverage |
+| 10 | The pack drifts into being installed by default | Low | High | Adoption rule 1 plus a test asserting `install` never pulls it in |
+
+## 13. Open questions
+
+1. ~~Which upstream tag do we pin first?~~ **Answered: v0.44.0.** See section 2.1.
+2. ~~User scope only, or a project-scoped preset variant too?~~ **Moot:** section 7 is cut, so the pack installs no presets at any scope.
+3. ~~Is patching `pipeline_final_safe` for `head`/`tail` in scope later?~~
+   **Answered: no.** See section 2.6.
+4. ~~Does Phase 3 survive its own measurement at 0.008% of input tokens?~~
+   **Answered: no, cut 2026-07-26.** See section 7.
+5. ~~How does the pack share rtk's single global `filters.toml`?~~ **Moot:** it
+   writes no filters at all. The marker-injection design and the trust
+   constraints that would have applied are kept in section 6.1 in case presets
+   are ever revisited.