mlx-bun 0.0.9 → 0.0.11

package/docs/design/repo-cleanup-plan.md

@@ -0,0 +1,189 @@
+# Repo cleanup plan
+
+Status: PLANNED (2026-07-02, written post-v0.0.10). The audit behind it:
+`.git` was 179 MB, driven by 497 machine-specific golden `.bin` blobs
+(~290 MB working tree, every regen compounding history); root has ~30
+entries of mixed tracked/untracked provenance; `repro/` was undocumented.
+Already done in v0.0.10: `.bin` goldens untracked + gitignored +
+`goldens/README.md` policy; CLAUDE.md root-layout map. This plan is the
+rest, phased so each step is safe, reversible (except C1, which is the
+point), and independently shippable.
+
+Principle: the hygiene discipline IS the product edge — every phase ends
+with a guardrail so the mess can't re-form, not just a one-time sweep.
+
+## What can actually be lost? Nothing. (Read this before re-litigating.)
+
+Every binary this plan untracks or purges survives in MULTIPLE independent
+places, so "we might lose the goldens/fixtures" is not a reason to stall:
+
+1. **Both dev laptops' working trees** — `git rm --cached` and `.gitignore`
+   never touch disk; the files sit exactly where they were.
+2. **Git history** — until Phase C runs, every previously-committed byte is
+   still in `.git` and recoverable with `git show <rev>:<path>`.
+3. **The Phase-C mirror backup** — `scripts/history-rewrite.sh` refuses to
+   rewrite before writing `~/mlx-bun-history-backup-<date>.tar` (a full
+   pre-rewrite mirror, kept forever). Even after the purge, every historical
+   blob is one `tar -x` away.
+4. **Regen scripts** — goldens regenerate from `scripts/regen-*` + the
+   oracle venv; the universal-rope fixtures from their gen script.
+5. **Pinned-hash fetch** — the two LoRA adapter fixtures are sha256-pinned
+   in `scripts/fetch-test-fixtures.sh`, sourced from the `test-fixtures-v1`
+   GitHub release (one command to create — see the script header) or
+   either laptop.
+
+And no test can break: every suite that reads a binary fixture gates on
+FILE PRESENCE and skips cleanly when it's absent (goldens via
+tests/goldens.ts, LoRA via `haveAdapters`, universal-rope via its own
+check). Untracked ≠ deleted ≠ unrunnable.
+
+## Phase A — root consolidation (30 min, zero risk) — DONE 2026-07-02
+
+- `[x]` **A1. `lab/`**: `git mv repro lab/repro && git mv spikes lab/spikes`.
+  One home for "code that proves a point but isn't the product": upstream
+  bug reproductions and feasibility probes. Update the CLAUDE.md layout map
+  + the two docs that reference `repro/` (grep first: `grep -rn "repro/"
+  docs/ *.md`).
+  - Also updated every path reference (CLAUDE.md, PLAN.md, PLAN-archive.md,
+    docs/investigations/lab-build-journal.md, docs/design/docs-reorg-plan.md,
+    and the `repro/`-pointing comments in src/mlx/{ffi,array,autograd,ops}.ts
+    + tests/{ffi-jit,train-autograd}.test.ts + the repro's own ISSUE.md).
+- `[x]` **A2. root strays** (all untracked; no git surgery): delete
+  `train.log`; move `benchmarks-e4b-flag-matrix-2026-06-15.json` into
+  `reports/`; confirm `.DS_Store` is gitignored globally.
+  - `train.log` deleted; benchmark json moved to `reports/`; `.DS_Store`
+    already globally gitignored (`.gitignore`).
+- `[x]` **A3. `archive/` → `docs/archive/`**: the two frozen HTML reports
+  are docs, not a root concern. `git mv`, update CLAUDE.md map.
+- Exit: root = code (`src tests scripts`), product surfaces (`docs website
+  packaging bin extensions`), fixtures (`goldens fixtures lab`), meta
+  (README/PLAN/STATUS/CLAUDE + configs), and gitignored working dirs. Every
+  tracked root entry appears in the CLAUDE.md layout map.
+
+## Phase B — tracked-content policy + guardrail (1 h) — DONE 2026-07-02 (B2 closed same day)
+
+- `[x]` **B1. Binary-in-git gate**: `scripts/check-hygiene.ts` — FAILS on
+  any tracked file that is (a) >1 MB, or (b) matches `*.bin|*.safetensors|
+  *.dylib|*.gguf|*.metallib` — outside an explicit, size-capped, justified
+  allowlist. Wired into `scripts/test.sh` (runs before tests) AND CI
+  (`.github/workflows/ci.yml` `hygiene` job, gates typecheck). Supports
+  `--staged` for pre-commit use. This is the "can't re-form" guarantee.
+- `[x]` **B2. fixtures/ audit**: 13 MB, 2 tracked `.safetensors`.
+  Decision per file:
+  - `fixtures/adapters/{upper,french}/adapters.safetensors` (6.6 MB each):
+    KEEP tracked, on the gate's explicit allowlist (cap 7 MB). They are
+    stable one-time-trained LoRA adapter INPUTS (trained 2026-06-10, never
+    churned) for the opt-in `MLX_BUN_TEST_LORA` hot-swap test, which also
+    requires the e4b base. The tracked `adapter_config.json` + `data-*/`
+    jsonl fully describe how to reproduce them; a bit-exact regen script
+    (trainer-of-record: optiq `lora train` vs mlx-lm — the saved config is
+    ambiguous) is a Josh task to confirm + smoke. NOT untracked now because
+    an untested untrack would break the lora hot-swap parity test, and these
+    do not churn (the goldens-bloat problem was *churning* artifacts). This
+    is a deliberate allowlist exception, not the default `fixtures/ ≤ 2 MB`
+    policy.
+  - **CLOSED later the same day**: untracked + gitignored after pinning the
+    bytes by sha256 in `scripts/fetch-test-fixtures.sh` (fetch from the
+    `test-fixtures-v1` release / either laptop / pre-rewrite history; the
+    gated LoRA test skips cleanly when absent — verified). The regen-
+    ambiguity concern was real but orthogonal: preservation is by pinned
+    original bytes, not by retraining. Allowlist entry removed; gate green
+    with ZERO multi-MB binaries tracked.
+- `[x]` **B3. tests/ binary audit**: 27 tracked `.bin` under
+  `tests/fixtures/universal-rope/` (4–8 KB each, ~108 KB total). All are
+  model-free CI-load-bearing bit-exact oracle fixtures (used by
+  `tests/universal-rope.test.ts`, which skips cleanly if absent),
+  regenerated by `scripts/gen-universal-rope-fixtures.py`. STAY, on the
+  allowlist (cap 16 KB each). Also allowlisted: `tests/fixtures/qwen-delta-golden.json`
+  (1.08 MB text JSON golden, model-free, regen script exists).
+- Exit: `git ls-files | grep -E '\.(bin|safetensors|dylib)$'` returns only
+  allowlisted entries; the gate enforces it forever.
+
+## Phase C — history rewrite (the 179 MB) — DONE 2026-07-02 (Josh go)
+
+Executed 2026-07-02 on Josh's explicit go. Result: `.git` **182 MB →
+20 MB**. Notes on what actually ran (vs the sketch below):
+
+- C1: dry-run `--analyze` confirmed goldens `.bin` = the entire bloat.
+  Actual invocation, narrower than sketched (no size threshold needed):
+  `git filter-repo --invert-paths --path-glob 'goldens/*.bin'
+  --path-glob 'goldens/*/*.bin'` — both globs needed because git
+  pathspec `**` doesn't span the flat + one-deep layout; covers all 497
+  historical blobs. Verified pre-push: 343 commits in/out, HEAD tree
+  bit-identical to original, zero `.bin` left anywhere in goldens
+  history, tags remapped.
+- C2: force-pushed main + branches + tags. Mirror backup at
+  `~/mlx-bun-mirror-backup-2026-07-02.tar.gz` (143 MB) — delete once
+  both boxes verified. M1 Max done: fetch --force + reset --hard,
+  surviving branches remapped, stale agent-session worktrees/branches
+  deleted (Josh: only main matters), stray `refs/codex`/`refs/original`
+  pruned, reflog expired, gc'd to 20 MB. **M4 Pro remaining (Josh)**:
+  `git fetch origin --tags --force --prune && git reset --hard
+  origin/main` (or fresh clone).
+- C3: `v0.0.10` resolves; release assets untouched (they live outside
+  git history); hygiene gate green; regen path proven live —
+  `bun scripts/regen-fused-sdpa-goldens.ts` rewrote
+  `goldens/apple-m1-max/fused-sdpa.{json,bin}`. The full suite could not
+  be run from the executing session (tooling restriction); run it once
+  per box as the final check.
+
+**Now ONE COMMAND**: `scripts/history-rewrite.sh` (dry-run by default,
+`MLX_BUN_REWRITE_GO=1` to execute) — it enforces every precondition below,
+writes the permanent mirror backup first, purges goldens `.bin` + the
+adapter safetensors from history, force-pushes, and prints the re-clone
+steps for the other laptop.
+
+One-time, destructive-by-design, ~15 min total. Preconditions: BOTH
+laptops fully pushed (verify `git log origin/main..HEAD` empty on each);
+no open PRs; a fresh `git clone --mirror` tarball kept as belt-and-
+suspenders until both machines are re-cloned.
+
+- `[x]` **C1. rewrite**: `git filter-repo --strip-blobs-bigger-than 200K
+  --path-glob 'goldens/**/*.bin' --invert-paths` (exact invocation to be
+  dry-run first with `--analyze`; target: drop all historical goldens
+  `.bin` while keeping every text file's history intact).
+- `[x]` **C2. force-push** `main` (+ tags), re-clone on BOTH laptops
+  (M1 Max + M4 Pro), re-link local-only assets (nothing tracked is lost —
+  the blobs being dropped are regenerable and already untracked at HEAD).
+- `[x]` **C3. verify**: `du -sh .git` (expect ~15–25 MB), `bash
+  scripts/test.sh` green, one `scripts/regen-*` smoke to prove the regen
+  path, tag `v0.0.10` still resolves + release assets untouched (GitHub
+  releases store assets outside git history).
+- NOT doing LFS: 290 MB of churning machine-specific artifacts is exactly
+  what LFS quotas punish, and the regen scripts make the artifacts
+  reproducible — a fixture server is overkill for a two-laptop project.
+- Exit: fresh clone < 30 MB ✓ (20 MB); M1 Max on rewritten history ✓;
+  M4 Pro pending its one-line reset (see notes above).
+
+## Phase D — docs debt (follow-the-map, 1–2 h, independent) — gate DONE 2026-07-02
+
+- `[~]` **D1.** PLAN.md is 139 KB — move closed phases older than Phase 17
+  into PLAN-archive.md (the established pattern; keep PLAN.md scannable).
+  - **Finding (2026-07-02):** no candidates currently. Every phase <17 in
+    PLAN.md (6, 7, 12, 13, 14, 15, 16) is marked `[~]` or `[ ]` — none are
+    `[x]`/CLOSED, and the established archive pattern archives only
+    `[x]`-marked phases. Phases 6 and 16 have *all* their checkboxes
+    checked but carry a deliberate `[~]`; reclassifying them to closed +
+    archiving is a Josh state-call (they may have ongoing narrative
+    threads). The closed `oMLX` phase (2026-07-02) and Phase 19 are
+    *newer* than 17, so they stay in PLAN.md as recent history per the
+    criterion. Revisit D1 when a <17 phase is confirmed closed.
+- `[x]` **D2.** `docs/investigations/curve-runs/` + two untracked HTML
+  strays in docs/investigations: gitignore or move under `reports/`.
+  - Already satisfied: `docs/investigations/curve-runs/` is gitignored
+    (`.gitignore`), and the four HTML strays (`curve-terrain.html`,
+    `hlg-ab.html`, `hlg-report.html`, `hlg-explorer.html`) are gitignored
+    via `docs/investigations/*.html`. No action needed.
+- `[x]` **D3.** The CLAUDE.md doc-map lists are hand-maintained; add a
+  `scripts/check-hygiene.ts` assertion that every `docs/**/*.md` appears in
+  the map (same guardrail spirit as B1).
+  - Implemented in `check-hygiene.ts` (`docs-map` check). First run caught
+    10 drifted docs (3 design, 1 investigation, 6 planning) — all added to
+    the CLAUDE.md map. Gate now green; CI enforces it on every push/PR.
+
+## Ordering & ownership
+
+A and D anytime (safe, agent-executable solo) — DONE 2026-07-02. B gate
+DONE 2026-07-02 (B2 adapter-untrack deferred to Josh — see B2 finding).
+C executed 2026-07-02 on Josh's explicit go — only remaining tail is the
+M4 Pro reset + deleting the backup tarball once both boxes are green.

package/docs/design/runtime-isolation.md

@@ -1,9 +1,25 @@
 # Runtime isolation — the AI may crash, the UI never may
 
-Status: **design** (not started — deferred future work; no scheduled PLAN.md phase)
+Status: **P1 LANDED 2026-07-05** (opt-in `--isolate`; default flip after
+bake). Phase 1 (yields) fully landed same day (the gateway onToken hop
+existed; prefill chunk yields added). **ARCHITECTURE DECISION (2026-07-05,
+deviates from the IPC sketch below):** the engine child is the ENTIRE
+existing server on a unix domain socket; the parent is a thin reverse
+proxy (src/serve/isolate.ts). Rationale: the gateway-as-IPC-client cut
+would serialize grammar controllers (WASM), vision embeddings (GPU
+arrays), sampler closures, and prompt-cache hooks over a message protocol
+— weeks of re-plumbing; HTTP-over-UDS reuses the whole serving stack
+verbatim, streams SSE for free, and cancellation is abort propagation.
+The inter-process API is the existing /v1 surface — no second protocol.
+**Measured (M1 Max, cpm5, paired)**: proxied 197.2 vs direct 197.9 tok/s
+(−0.4%, noise), TTFT +2 ms, SSE per-token granularity preserved (96 reads
+/ 96 tokens), parent answers in 0.6 ms mid-decode (criterion: <50 ms),
+child kill → 502 + respawn (gated model-free + E2E:
+tests/isolate-proxy.test.ts, tests/isolate-e2e.test.ts). Not proxied yet:
+/ws/chat (501 + pointer). P2 = child-per-model pool (multi-model switch by
+spawn-overlap); P3 = GPU-lease unification. The original design follows.
 Owner: serving layer / runtime
-Default today: inference runs **in-process**, on the server's event-loop
-thread. This doc proposes moving it to a **subprocess**.
+Default today: inference runs in-process unless `--isolate` is passed.
 
 ## Goal
 

package/docs/design/segmented-backward-training.md

@@ -184,6 +184,21 @@ quant. The trivial case to validate the machinery on a real model.
 
 ## 5. The segmentation strategy (the part to get right)
 
+> **2026-07-02 CORRECTION — this section's cost model is WRONG, measured.**
+> The full-attention isolation planner was built and A/B'd @8K on real chunk
+> rows (scripts/experiments/seg-isolation-smoke.ts, MLX_BUN_SEG_MEM_LOG=1
+> phase probes): ZERO peak win (18.09 vs 18.02 GB seg2). The premise "sliding
+> layers are O(L·window) in the backward" is false — mlx's sdpa BACKWARD
+> materializes the O(L²) scores for EVERY layer (~3.5 GB/layer @8K e4b; the
+> sliding window is only an additive mask; a sliding PAIR costs +7.1 GB ≈
+> sliding+full's +7.15). The worst segment is therefore set by LAYER COUNT
+> alone and `segment_size` is the whole knob: **seg1 = 14.59 GB @8K, +3% step
+> time, loss identical — fits the 24 GB M4 Pro today.** The head vjp adds
+> ~3 GB (full-[M,V] responseOnlyCe — kernel-review backlog #8). Going below
+> ~14 GB needs that head fix plus an O(L)-memory attention backward (flash
+> track), not smarter segmentation. Section kept for the (still-correct)
+> donor/boundary constraints; ignore its sliding-vs-full peak arithmetic.
+
 NOT a naive "divide by N". Compute the split points from the model:
 
 **Constraints & inputs (all readable from the model/config):**
@@ -234,8 +249,9 @@ returns the split indices + which donor K/V to save, computed from the above.
 - The chunk data needs ≥4096 (median 4092; 0% ≤2048; 98% ≤8192). Success = e4b
   trains at 8192 ≤ ~10 GB AND the chunk eval (`scripts/chunk-eval.ts`) shows a
   quality delta on e4b like MiniCPM5's.
-- Training requires `MLX_BUN_PERF_KERNEL=0 MLX_BUN_FUSED_GELU=0` (those fused
-  CustomKernels have no vjp).
+- Training needs no env-flag sanitization (the no-vjp fused CustomKernels that
+  once required `MLX_BUN_PERF_KERNEL=0 MLX_BUN_FUSED_GELU=0` were deleted
+  2026-07-05; see docs/design/unified-engine-frontier-plan.md §6).
 - MLX `Date.now()`/random caveats don't apply here; standard determinism via seed.
 
 ## 7. Key files

package/docs/design/ssd-kv-cold-tier.md

@@ -0,0 +1,288 @@
+# SSD cold tier for the prompt/KV cache
+
+Status: **LANDED P1–P3 (2026-07-02, same day)** — P4 hardening extras remain.
+**Layer-0 upgrade (2026-07-05, unified-engine plan):** D2 finished properly —
+the two-tier take/restore/trim dance moved INSIDE `PromptCache.take()`
+(structural `ColdTier` interface; the server binds SsdCacheStore + model into
+it), so EVERY consumer tiers automatically: the serial lane's hand-rolled
+block in runGeneration is gone, and the BATCH SCHEDULER restores prefixes
+from disk at admission (gated end-to-end: tests/batch-scheduler.test.ts
+"SSD tier through the batch lane"). `PromptCache.onPut` drives the debounced
+write-behind snapshot for both lanes' puts. NEW: **idle demotion**
+(`--ssd-demote-idle`, default 300 s with the tier on) — entries unused past
+the threshold spill to disk and free their GPU arrays; RAM drains between
+agent bursts while prefixes stay reachable via zero-copy restore. The sweep
+runs only when the gateway is fully idle (a runExclusive would otherwise
+drain a live batch). Rationale (Josh): prefill is the cost being avoided —
+12B prefill is 168 tok/s, so a 30k agent context is ~3 min of GPU compute vs
+a ~1 s NVMe restore; SSD-vs-RAM (40×) is the wrong comparison, SSD-vs-
+recompute (50–180×, growing with context) is the real one.
+
+**Measured (M1 Max, MiniCPM5, 13.7k-token prefix, kv_config quant):**
+- Restart TTFT: **12,083 ms (full prefill) → 236 ms** restored — 2% of full
+  prefill, 13,675 tokens back via a 3 ms zero-copy mmap. Beats oMLX's 1–3 s
+  claim and both exit criteria (<2.5 s, ≤10%).
+- Steady-state decode: **0% overhead** (192.0 vs 192.3 tok/s, flag off/on) —
+  nothing runs on the token loop, as designed (vs oMLX's measured ~20%).
+- Correctness: SSD-restored + trimmed continuation is **identical to the RAM
+  tier's cache-hit output** (control experiment); divergence from a fresh
+  full prefill is the pre-existing bf16 prefix-reuse property both tiers
+  share, not introduced by persistence.
+
+What landed: kv-store v2 (`src/kv-store.ts` — all five cache kinds incl.
+quantized + SSM, invalidation metadata, per-tensor hashes, atomic writes;
+`restoreState` added to both quantized cache classes), `src/ssd-cache.ts`
+(SsdCacheStore per D4), tier integration (`src/prompt-cache.ts` spill hook +
+retain thunks; `src/server.ts` tiered take with post-restore trim,
+debounced write-behind snapshots under the gateway lock, `/stats.ssd_cache`),
+flags `--ssd-cache <dir>` / `--ssd-cache-max <GB>` / `--ssd-cache-verify`.
+Tests: `tests/kv-store.test.ts` (fork-compare token-identical for bf16,
+quantized, SSM), `tests/ssd-cache.test.ts` (restart recovery, quarantine,
+corruption, eviction, supersede). One deviation from the plan below: find()
+returns PARTIAL prefix matches and the server trims after restore
+(divergent-tail conversations — full-entry-only missed the real pattern). Inspired by oMLX's paged SSD cache
+(Apache 2.0 — `omlx/cache/paged_ssd_cache.py`, `boundary_snapshot_store.py`;
+we port ideas, not code — add a `THIRD_PARTY_LICENSES.md` note if any code
+ends up derived). Their headline: coding-agent TTFT 30–90 s → 1–3 s by
+restoring long prefixes from SSD across requests AND restarts. Their
+measured costs (independent tests): ~2–3× cold TTFT and ~20% steady-state
+decode overhead — this design exists to get the win without those costs.
+Flag-gated, off by default.
+
+**Head start**: `src/kv-store.ts` already implements page-aligned KV
+persistence (`MLXBUNKV1`, 16 KiB-aligned tensors) with zero-copy COW mmap
+reload (`MmapFile.open(path, "cow")` → `MlxArray.fromPointer` — exactly
+the Metal page-alignment constraint, proven by expert-offload), including
+mid-rotation `RotatingKVCache` restore (`restoreState`,
+`src/model/gemma4-base.ts:880`). `scripts/experiments/cold-start.ts` is a
+working sub-1 s cold-restore harness. The SSD tier is wiring +
+quantized-cache support + an index/eviction/recovery layer.
+
+## Current state / gaps
+
+- `src/prompt-cache.ts`: byte-capped LRU RAM tier (default 2 GB,
+  `--prompt-cache`), prefix-entry based, adapter-namespaced, serial lane
+  only, race-free under the gateway lock. **A restart loses everything.**
+- `kv-store.ts` gaps: (1) no quantized kinds — `saveKvCache` throws for
+  `QuantizedKVCache`/`RotatingQuantizedKVCache` and neither has
+  `restoreState`, yet serving defaults to the model's `kv_config.json`
+  scheme, so the default production cache can't persist today; quantized
+  state is 6 arrays/layer (packed u32 + scales/biases bf16 × K/V);
+  (2) no invalidation metadata / checksums / atomic writes; (3) latent
+  bug: `loadKvCache` checks `caches.length !== model.layers.length` but
+  `makeCache()` returns donor layers only — wrong for KV-shared e4b
+  (verify in Phase 1).
+- Versioning primitive exists: `configFingerprint`
+  (`src/model/fingerprint.ts`) — sha256 over graph-shaping config incl.
+  `kvQuant`.
+
+## Design decisions
+
+**D1 — Whole-prefix-entry spill, not paged blocks.** oMLX pages for
+multi-user shared-prefix dedup; we are single-user with a prefix-entry
+cache that grows monotonically per conversation. Paging is exactly where
+their overheads come from: per-block SHA256 + save-queue bookkeeping
+*during decode* (their ~20% tax) and block-granular safetensors restore
+with copies (their 2–3× cold TTFT). Ours: **zero work on the token loop**
+(prefix tokens ARE the key), spill/persist only at request boundaries and
+evictions, restore = one zero-copy mmap with lazy page fault-in (~5 GB/s
+NVMe; 2 GB entry ≈ 0.4 s amortized into the first forward). Partial-prefix
+reuse works without paging: restore + `Cache.trim()`. Revisit paging only
+for multi-user batch dedup (v2, out of scope).
+
+**D2 — One store, two tiers.** `PromptCache` grows a cold tier behind the
+same `take`/`put` API:
+- RAM→SSD: `put()`'s evict loop spills (snapshot bytes → async writer →
+  dispose GPU arrays) instead of just disposing.
+- SSD→RAM: `take()` prefers a strictly-longer SSD prefix; restored entry
+  carries its `MmapFile`, unmapped on entry dispose (extensions copy into
+  fresh anonymous buffers via concat-grow anyway).
+- **Restart persistence needs write-behind at request completion**, not
+  just spill-on-evict: after `promptCache.put(...)` in `runGeneration`
+  (`src/server.ts:1019`), a debounced, coalesced-per-ns async snapshot.
+  Byte extraction (the only GPU-thread work) runs on the idle serial lane
+  between requests; file I/O is async. (oMLX's boundary-snapshot idea at
+  whole-entry granularity.)
+
+**D3 — Extend the existing format to `MLXBUNKV2`**, not safetensors
+(safetensors offsets aren't page-aligned; oMLX pays a copy via `mx.load`).
+Add: `"qkv"`/`"rotating-qkv"` kinds (six tensor slots + groupSize/bits;
+slice quantized buffers to `[.., :offset, :]` — don't persist 256-step
+slack), header fields `formatVersion`/`modelId`/`configFingerprint`/`ns`/
+`tokenizerHash`/`createdAt`, per-tensor checksums + header checksum.
+Integrity: header checksum always verified on scan; full verification
+opt-in (`--ssd-cache-verify`) because eager reads destroy lazy fault-in;
+default trusts atomic writes (tmp + fsync + rename). Corrupt/incompatible
+→ delete + log, never fatal. `loadKvCache` becomes model-generic
+(accept `{ makeCache(): Cache[] }`, donor-count check).
+
+**D4 — Files are the database.**
+`<ssd-dir>/<configFingerprint>/<nsHash>/<uuid>.mlxkv`. Startup recovery =
+header-only directory scan → in-memory `SsdIndex`
+(`{ns, tokens, path, bytes, mtimeMs}[]`); tokens live in the header JSON.
+LRU = file mtime (`utimes()` on hit); byte cap enforced at write time by
+oldest-mtime eviction; disk-full → drop write, warn once, keep serving.
+
+**D5 — Flags & interactions.** `--ssd-cache <dir>` +
+`--ssd-cache-max <GiB>` (default 32) → `ServerOptions.ssdCacheDir/MaxBytes`.
+Requires the RAM tier (`--prompt-cache 0` + `--ssd-cache` = startup
+error). `--memory-budget` unchanged (restored caches are clean file-backed
+pages until dirtied; admission already assumes worst-case bf16). `/status`
+gains `ssd: { entries, bytes, max_bytes, restores, spills, restore_ms_last }`.
+v1 is serial-lane only (where `PromptCache` lives); batch-lane interplay is
+docs/design/batching-perf-path.md P3's problem.
+
+## Phases
+
+- **P0 — Baseline bench (½ d)**: `scripts/experiments/ssd-cache-bench.ts`
+  grown from `cold-start.ts`: ≥16k-token prefix on the standard snapshot,
+  kv_config quant active; measure full-prefill TTFT / warm RAM-hit TTFT /
+  steady-state decode. Exit: baseline table committed.
+- **P1 — kv-store v2 (2–3 d, highest uncertainty)**: `restoreState` on
+  both quantized cache classes; all four kinds in save/load; header v2;
+  atomic writes; model-generic load. Exit: round-trip tests all kinds
+  (incl. wrapped ring) + **greedy decode parity** — N tokens continued
+  from a restored cache bit-exact vs never-persisted; `tsc` 0.
+- **P2 — `src/ssd-cache.ts` store (1–2 d)**: `SsdCacheStore
+  { scan, find(prompt, ns), store (async, extraction split from I/O),
+  evictToCap, remove, stats }`. Exit: unit tests — restart recovery,
+  incompatible-fingerprint quarantine, corrupt-header deletion, mtime cap
+  eviction, disk-full soft-fail.
+- **P3 — Tier integration + flags (2–3 d)**: cold tier in `PromptCache`,
+  debounced end-of-request snapshots, CLI/`/status`. **Exit (headline)**:
+  (1) restart TTFT on a 16k prefix **< 2.5 s and ≤ 10% of full-prefill**
+  (beat oMLX's 1–3 s; a true-cold miss runs the unchanged prefill path);
+  (2) steady-state decode overhead **< 2%** vs flag off (vs their ~20%);
+  (3) flag absent ⇒ zero behavior change.
+- **P4 — Hardening + docs (1–2 d)**: kill-during-write crash test,
+  adapter-ns isolation, kv-quant-scheme-change invalidation, README/help,
+  license note. Exit: suite green; on/off × cold/warm/restart matrix
+  recorded.
+
+Total ~7–10 working days.
+
+## Risks
+
+- **mmap lifetime vs mlx laziness**: unmap only in entry `dispose()` after
+  successor state is evaluated; COW mapping already covers buffer donation.
+- **Byte-extraction stall on multi-GB entries**: runs between requests on
+  the idle lane; measure in P3; chunk with yields if needed.
+- **COW dirty-page accounting**: extended restored caches migrate to
+  anonymous memory — probe with `phys_footprint` (expert-offload
+  discipline) before making footprint claims.
+- **Stale-token poisoning**: `configFingerprint` + `tokenizerHash` + ns;
+  chat-template drift degrades to a shorter matched prefix (safe).
+
+## Addendum 2026-07-07 — A7 closure: bounded memory on BOTH sides of the file
+
+The 07-06/07 serve benches flagged elevated ctx/restart-leg RSS on
+--ssd-cache arms ("A7"). Root-caused in three parts; the first two were
+real defects, both fixed in `src/kv-store.ts`:
+
+- **Write side** (residual after the v3 streaming writer of 4f9bad9): the
+  per-tensor `rawBytes()` readback ended in a JS-heap `.slice()`; under GC
+  lag the dead copies outlive the flush loop, re-spiking RSS toward the
+  whole entry. The writer now hashes and writes from a ZERO-COPY view of
+  the contiguous mlx buffer (`MlxArray.rawBytesView`) — no JS-heap copy
+  exists at any point; save-transient mlx memory measured 0 bytes for
+  contiguous sources (one tensor otherwise).
+- **Restore side** (never covered by A7's original scope): the zero-copy
+  restore (COW mmap + `fromPointer`) became a per-restore PROCESS-LIFETIME
+  mapping leak once the 2026-07-06 FFI-dtor fix removed the unmap signal,
+  and the exactly-offset-sized restored buffers made the first
+  post-restore decode step concat-copy the entire entry into fresh mlx
+  memory. Restore is now a **streamed copy** — each tensor is copied into
+  an mlx-owned leaf (`fromBytesCopy`, no dtor contract), its clean file
+  pages dropped (`MADV_DONTNEED`; tensor offsets are 16 KiB-aligned = the
+  arm64 page size), the mapping unmapped before `loadKvCache` returns, and
+  plain-KV tensors land in STEP-rounded capacity with ≥1 token of slack so
+  the first step updates in place. Measured (512 MB synthetic entry): mlx
+  peak during restore+first-step = live entry + ONE tensor (552 vs 520 MB
+  active); `vmmap` confirms no `.mlxkv` mapping survives; 12B real-model
+  cold cache-load → first token 277 ms (parity with the old zero-copy
+  ~240 ms claim — the old path paid the same whole-entry copy on the first
+  decode step anyway). Byte identity of the copy-restore is pinned for all
+  five cache kinds (save → load(verify) → re-save hash-identical;
+  tests/kv-store.test.ts "copy-restore byte identity") plus the real-model
+  bf16/quantized/SSM continuation suites.
+- **What was NOT a defect**: most of the benched leg delta is `ps` RSS
+  ACCOUNTING — the write-behind's hash+write CPU-touches the live KV
+  entry, which makes those already-allocated unified-memory pages visible
+  to `ps` (GPU-written buffers and python-arm KV never show). The bench
+  report now footnotes this instead of promising a pending fix.
+
+Supersedes the D2 line "restored entry carries its MmapFile, unmapped on
+entry dispose" and the D1 "restore = one zero-copy mmap with lazy page
+fault-in" mechanics above — the economics (SSD-vs-recompute) and the
+zero-work-on-token-loop property are unchanged.
+
+## Addendum 2026-07-07 — write-behind scheduling contract: flush only while idle
+
+The 07-07 serve bench showed decode@ctx losing to mlx-lm on the bun arms
+(e4b −9.3%, 12B −3.9%) while short-ctx decode won on every model — and the
+mixed arm (4× smaller flush bytes) BEAT the bf16 arms at decode@ctx. Root
+cause: "non-blocking" write-behind was only non-blocking at the event-loop
+level. Every per-tensor flush step is (1) `ops.contiguous` — a kernel on
+the SAME GPU stream decode uses — then (2) `rawBytesView()` → a synchronous
+`mlx_array_eval` that blocks the JS thread until the stream drains, then
+(3) a synchronous multi-MB `writeSync`. The `setImmediate` pacing
+interleaved those slices exactly between decode tokens, so a ~16k entry's
+flush overlapping the bench's cached ctx repeats taxed their 64-token
+decode windows (reproduced standalone: e4b 9.5k-token entry, repeat decode
+37.9 vs cold 47.1 tok/s; flat 44–46 across all three samples post-fix).
+mlx-lm runs no equivalent background work — the gap was pure self-inflicted
+contention.
+
+**The contract now:** the flush only progresses while the engine is idle.
+- `GenerationGateway.busy` covers BOTH lanes (mutex held/awaited — the
+  serial lane shows zero rows — OR batch rows active/pending);
+  `gateway.onIdle()` is the poll-based waiter (`generation-gateway.ts`).
+- `saveKvCacheAsync`/`SsdCacheStore.storeAsync` accept a per-step
+  `waitTurn` gate awaited before EVERY tensor step (including the first),
+  so a request arriving MID-flush pauses the remaining tensors until idle
+  again; the server passes `() => gateway.onIdle()` at both chain sites
+  (write-behind snapshots AND eviction/demotion spills). Gate failures are
+  swallowed — scheduling advice must never corrupt the write path.
+- `MLX_BUN_SSD_WRITEBEHIND=0` disables write-behind snapshots entirely
+  (kill switch + paired-A/B lever; restart survival then degrades to
+  spill-on-evict only).
+
+Accepted tradeoffs: durability waits for a quiet moment (single-user
+serving quiesces constantly; sustained hammering defers the flush AND the
+spill clones' GPU-memory release, bounded by the serial chain); one
+in-flight tensor step (~10–15 MB) can still land ahead of a just-arrived
+request; and the restart window tightens — the flush now runs in idle gaps
+instead of overlapping decodes, so the bench's 2.5 s pre-kill beat must
+absorb it (verified surviving: e4b 9.5k entry, restart cached=9575 with
+the gate on).
+
+## Addendum 2026-07-07 (later) — bounded retention + admission-safe snapshots
+
+The post-merge review found two operational holes in the contract above:
+
+1. **Snapshot timer could stall batch admission.** `scheduleSsdSnapshot`'s
+   debounced timer fired `gateway.runExclusive` unconditionally; the
+   exclusive registers a serial WAITER, which holds batch admission until
+   every running row drains naturally — a background durability timer
+   periodically freezing admission under `--batch N` sustained load. Fixed
+   with the same guard `demoteIdle` always had: the timer checks
+   `activeRows/pendingRows` first and RE-ARMS (5 s, unref'd) while busy —
+   entries are immutable, a late snapshot is equally valid.
+2. **Starved-gate clone retention was unbounded.** Every eviction/demotion/
+   snapshot chained GPU-pinning zero-copy clones onto a bare serial promise
+   chain, and every flush step awaits `onIdle()` — under continuous traffic
+   the chain starves and resident memory = prompt-cache cap + every queued
+   clone, defeating the byte cap exactly under the load causing evictions.
+   Now all three producers go through **`SpillQueue`** (kv-store.ts):
+   queued bytes are capped (default 2 GB = a quarter of the 8 GB RAM-cache
+   default; `MLX_BUN_SSD_SPILL_QUEUE_GB` overrides) — over cap the OLDEST
+   not-mid-store item drops with its clones disposed immediately. A dropped
+   spill is a future cache miss, never a wrong result: the contention-free
+   alternative to letting the flush cut into decode. `/stats.ssd_cache`
+   exposes `pending_spills` / `pending_spill_bytes` / `dropped_spills`.
+
+Explicitly accepted (documented, not built): **no shutdown flush** — a
+SIGINT/SIGTERM during (or within the debounce after) traffic loses queued
+write-behind entries; restart survival degrades to whatever already
+flushed. This is a best-effort cache tier; the cost of a lost entry is one
+re-prefill. `SpillQueue.drain()` exists if a shutdown hook is ever wanted.