mlx-bun 0.2.0 → 0.3.0

package/docs/design/parity-tier-dag.md

@@ -1,161 +0,0 @@
-# Parity-tier DAG → meaningful CLI flags
-
-**Status:** vision / roadmap (2026-06-21). The first artifact exists — a zoomable,
-tier-tagged map of the training + inference DAGs at
-[`docs/dag/training-inference-map.html`](../dag/training-inference-map.html). This
-doc records where it goes and *why*: to turn a sprawling pile of CLI flags into a
-small, coherent, tier-organized surface.
-
-## The framework (recap)
-
-Every node in the compute graph sits in the **lowest parity tier whose oracle it
-matches bit-for-bit**:
-
-- **L1** — drop-in for **mlx-lm** (its op is the oracle; bit-exact or it's a bug).
-- **L2** — drop-in for **mlx-optiq** (optiq's behavior is the oracle).
-- **the Lab** (formerly the `--l3` tier — deleted as a product mode 2026-07-05,
-  [unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)) — **our
-  originals** (flash-CCE head, ORPO loss, prefix-share, segmented backward, HLG
-  sampler, expert offload). **No oracle ships**, so correctness is gated by
-  finite-difference / KL / quality, *not* parity. Output-changing inference
-  experiments live here as env-flag items with a bench + expiry, until one beats
-  the L1 baseline in a paired A/B.
-
-**The gate is the oracle.** A node can only live in a lower tier if that oracle
-exists AND it matches bit-exactly; otherwise it bubbles up. Note the subtlety the
-map already shows: an *optimization* that still matches the oracle stays low —
-`compiled-decode` is ours but is L1 because it replays the same ops bit-for-bit.
-Only nodes with **no** oracle (flash-CCE) are *forced* into the Lab. (Background +
-post-mortem: [orpo-flash-cce-pin-leak.md](../archive/investigations/orpo-flash-cce-pin-leak.md).)
-
-## Two axes: parity gates performance
-
-Every optimization has **two independent coordinates**, and conflating them is what
-made the flag surface unreadable:
-
-1. **Parity** — *which reference does it reproduce bit-for-bit?* This sets the
-   **lowest level** it may live in: matches mlx-lm → L1; matches mlx-optiq but not
-   mlx-lm → L2; matches neither → no-oracle, Lab-gated (KL + tests).
-2. **Performance** — *is it the fastest correct way?* This sets whether it's the
-   **default (on)** within its level, or a kept-but-**off** opt-in.
-
-**Parity gates performance.** A faster kernel that breaks mlx-lm parity is simply
-*not in L1* — it bubbles up to whatever level its parity allows. You never trade the
-guarantee for speed silently; you change levels, and the user opts in. The level is
-your floor of trust; the default within it is the best way to honor that floor.
-
-### Decision procedure for any new optimization
-
-1. **Measure its parity** (run it both ways, compare — "one flag, one measurement
-   per lever"). The lowest oracle it matches bit-for-bit is its level ceiling.
-2. **Measure its performance** vs that level's current default:
-   - **faster AND holds the guarantee** → it becomes the **default (on)** for that level.
-   - **slower but still correct** → **keep it as a documented default-off flag**
-     (optionality for A/B; never the default, never deleted).
-   - **breaks the guarantee but offers a real tradeoff** (e.g. 2× tok/s at KL
-     0.0015) → a **Lab item** (env-flag, bench + expiry); it earns a place only
-     by beating the L1 baseline in a paired A/B.
-
-Worked examples:
-- *5% less memory, still bit-for-bit with mlx-lm* → **default-on in L1.**
-- *2× tok/s, but KL 0.0015 to mlx-lm* → **Lab item** (proves itself or expires).
-- *another way, 3% slower, still correct* → **kept, off in every level.**
-
-### Our decode levers under this rule (today)
-
-| lever | parity | perf | placement |
-|---|---|---|---|
-| `compiled-decode` | bit-exact w/ uncompiled (`compiled-decode.test`) | faster | **default-on, every level** |
-| `perf-kernel` | envelope-gated, no bit-exact oracle (was gated ≥56/64 teacher-forced argmax vs our own frozen compat trajectory) | regressed e4b 0.62–0.93× | **DELETED 2026-07-05, [unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)** (kernel + frozen-oracle tests removed) |
-| `fused-sdpa` | matches optiq bit-for-bit, not mlx-lm (`fused-sdpa.test` tier-a goldens) | faster | **default follows `--kv-quant`** (on for `config` — the L2 bridge; off for uniform/bf16) |
-| `fused-decode` | (no oracle) | ~1.00× (`optimization_plan.md`) | **DELETED 2026-07-05, [unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)** |
-
-So `--l1/--l2` is the **parity** axis (the guarantee), and the defaults within
-each tier are the **performance** axis (the fastest kernel that still holds it).
-2026-07-05 outcome ([unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)):
-no output-changing decode kernel beat the L1 baseline in a paired A/B, so `--l3`
-was deleted as a product mode (the flag now hard-errors) along with its kernels
-(perf-kernel, fused-decode). The no-oracle territory — HLG sampler, expert
-offload, batched mixed-precision — is now the **Lab**: env-flag experiments with
-a bench + expiry, gated by KL + quality, off the decode axis.
-
-> History: commit `f1bf5cc` put the perf kernel in the L2 preset claiming
-> "perf-kernel-oracle.test asserts the fused kernel == the frozen optiq goldens".
-> Both halves were wrong — that golden is frozen from mlx-bun's own compat engine,
-> and the gate is argmax-agreement, not equality — so 2026-07-01 restored the tier
-> rule: the bare tier is the guarantee; perf kernels are opt-ins within it.
-> Deletion outcome: the perf kernel never beat the L1 baseline in a paired A/B
-> (regressed e4b 0.62–0.93×; only win 12B@16k +6% with a KL WARN), so it was
-> DELETED 2026-07-05 with its frozen-oracle tests
-> ([unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)).
-
-## Why this matters for CLI flags
-
-We are accumulating a **lot** of flags (`--grad-accum`, `--grad-clip`, `--seg`,
-`--no-flash`, `--no-prefix`, `--no-segment`, `--lambda`, `--val-size`, kv-quant
-knobs, sampler knobs, …). The problem isn't any one flag — it's the **combinatorics**
-and the fact that **some are never really a choice** (always-on guards, or the only
-viable route). The tier-tagged DAG makes each flag legible by asking: **what route
-does this flag push the computation down, and what tier is that route?**
-
-A flag falls into exactly one of three buckets:
-
-| flag selects… | tier effect | what it IS | action |
-|---|---|---|---|
-| the only viable route (no alternative node) | — | not a choice | **always-on default; drop the flag** |
-| between two routes at the SAME tier / same oracle | none | cosmetic / safety guard | **default it; expose only if it ever fails** |
-| between a **lower-tier** route and a **no-oracle (Lab)** route | **L1/L2 ↔ Lab** | a real **parity ⇄ optimization** knob | **keep, document as such, group by tier** |
-
-So the DAG lets us answer "**what flags push down THAT route**" for any path, and
-from that:
-
-1. **Prune always-on flags to defaults** — anything whose alternative route is
-   never selected, or whose toggle doesn't change the tier, shouldn't be a flag.
-2. **Surface the real axis.** The flags that genuinely matter all toggle
-   *bit-exact-lower-tier ⇄ our-no-oracle-optimization*. Those could collapse into a small
-   number of intent-level switches (e.g. a single `--exact` that forces every node
-   down to its lowest tier, vs the optimized default) instead of N independent
-   kernel toggles the user has to understand individually.
-3. **Group the rest by what they trade** — memory (`--seg`, all bit-exact → not a
-   parity choice, a memory↔compute knob), hyperparameters (`--lambda`, `--lr` →
-   training knobs, orthogonal to tier), etc.
-
-### Worked classification (current training flags)
-
-- `--grad-clip 1.0` → standard guard, L1, **always-on default** (not a route choice).
-- `--seg N` / `--no-segment` → memory↔compute, **bit-exact either way** → a *memory*
-  knob, **not** a parity choice. Should read as "memory budget," not "exactness."
-- `--grad-accum N` → changes the accumulation route (L1; smoother grads) → a real
-  *training-quality* knob.
-- `--no-flash` → **no-oracle flash-CCE → L2 MLX fused head** → a genuine *parity ⇄
-  perf* fallback (the one place the head leaves the no-oracle class).
-- `--no-prefix` → **no-oracle prefix-share → bit-exact two-forward** → a *perf* knob
-  with no numerical cost.
-- `--lambda` → ORPO hyperparameter (pure no-oracle) → a *training* knob.
-
-The pattern: most flags are **memory/training knobs or always-on guards**; only a
-couple are true **parity ⇄ optimization** toggles. Today they're a flat list the
-user must reason about individually; tier-tagging lets us present them as what they
-actually are.
-
-## Roadmap to make it executable
-
-The map is hand-authored today. To make the framework *enforce* itself:
-
-1. **Derive the graph from code** — extract the call graph + per-node op/oracle tags
-   so it stays in sync and is **queryable** ("every no-oracle node", "paths that
-   cross into the Lab", "the pure-L1 subgraph", "flags that select route X").
-2. **CI-enforce the gate** — a node tagged L1 must have a passing bit-exact parity
-   test vs mlx-lm; if it can't, it's a bug or must be re-tagged no-oracle. "The
-   oracle is the gate" becomes a check, not a principle.
-3. **Shrink the no-oracle surface** — audit whether anything in the Lab island is
-   actually bit-exact-able and just unproven (e.g. the prefix-shared *forward* is
-   already bit-exact — `prefix-shared-parity.ts` — so only the encode-once *fold*
-   is truly no-oracle). Minimizing it minimizes what needs KL/quality gates.
-4. **Generate the flag surface from the graph** — once flags map to routes, the CLI
-   help + defaults can be derived: always-on routes have no flag, parity⇄opt routes
-   collapse to intent switches, knobs group by what they trade.
-
-End state: instead of a growing flat list of flags the user memorizes, a small set
-of intent-level switches whose meaning is exactly "which tier route does the compute
-take," backed by a DAG that's checked in CI.

package/docs/design/repo-cleanup-plan.md

@@ -1,231 +0,0 @@
-# Repo cleanup plan
-
-Status: PLANNED (2026-07-02, written post-v0.0.10). The audit behind it:
-`.git` was 179 MB, driven by 497 machine-specific golden `.bin` blobs
-(~290 MB working tree, every regen compounding history); root has ~30
-entries of mixed tracked/untracked provenance; `repro/` was undocumented.
-Already done in v0.0.10: `.bin` goldens untracked + gitignored +
-`goldens/README.md` policy; CLAUDE.md root-layout map. This plan is the
-rest, phased so each step is safe, reversible (except C1, which is the
-point), and independently shippable.
-
-Principle: the hygiene discipline IS the product edge — every phase ends
-with a guardrail so the mess can't re-form, not just a one-time sweep.
-
-## What can actually be lost? Nothing. (Read this before re-litigating.)
-
-Every binary this plan untracks or purges survives in MULTIPLE independent
-places, so "we might lose the goldens/fixtures" is not a reason to stall:
-
-1. **Both dev laptops' working trees** — `git rm --cached` and `.gitignore`
-   never touch disk; the files sit exactly where they were.
-2. **Git history** — until Phase C runs, every previously-committed byte is
-   still in `.git` and recoverable with `git show <rev>:<path>`.
-3. **The Phase-C mirror backup** — `scripts/history-rewrite.sh` refuses to
-   rewrite before writing `~/mlx-bun-history-backup-<date>.tar` (a full
-   pre-rewrite mirror, kept forever). Even after the purge, every historical
-   blob is one `tar -x` away.
-4. **Regen scripts** — goldens regenerate from `scripts/regen-*` + the
-   oracle venv; the universal-rope fixtures from their gen script.
-5. **Pinned-hash fetch** — the two LoRA adapter fixtures are sha256-pinned
-   in `scripts/fetch-test-fixtures.sh`, sourced from the `test-fixtures-v1`
-   GitHub release (one command to create — see the script header) or
-   either laptop.
-
-And no test can break: every suite that reads a binary fixture gates on
-FILE PRESENCE and skips cleanly when it's absent (goldens via
-tests/goldens.ts, LoRA via `haveAdapters`, universal-rope via its own
-check). Untracked ≠ deleted ≠ unrunnable.
-
-## Phase A — root consolidation (30 min, zero risk) — DONE 2026-07-02
-
-- `[x]` **A1. `lab/`**: `git mv repro lab/repro && git mv spikes lab/spikes`.
-  One home for "code that proves a point but isn't the product": upstream
-  bug reproductions and feasibility probes. Update the CLAUDE.md layout map
-  + the two docs that reference `repro/` (grep first: `grep -rn "repro/"
-  docs/ *.md`).
-  - Also updated every path reference (CLAUDE.md, PLAN.md, PLAN-archive.md,
-    docs/archive/investigations/lab-build-journal.md, docs/design/docs-reorg-plan.md,
-    and the `repro/`-pointing comments in src/mlx/{ffi,array,autograd,ops}.ts
-    + tests/{ffi-jit,train-autograd}.test.ts + the repro's own ISSUE.md).
-- `[x]` **A2. root strays** (all untracked; no git surgery): delete
-  `train.log`; move `benchmarks-e4b-flag-matrix-2026-06-15.json` into
-  `reports/`; confirm `.DS_Store` is gitignored globally.
-  - `train.log` deleted; benchmark json moved to `reports/`; `.DS_Store`
-    already globally gitignored (`.gitignore`).
-- `[x]` **A3. `archive/` → `docs/archive/`**: the two frozen HTML reports
-  are docs, not a root concern. `git mv`, update CLAUDE.md map.
-- Exit: root = code (`src tests scripts`), product surfaces (`docs website
-  packaging bin extensions`), fixtures (`goldens fixtures lab`), meta
-  (README/PLAN/STATUS/CLAUDE + configs), and gitignored working dirs. Every
-  tracked root entry appears in the CLAUDE.md layout map.
-
-## Phase B — tracked-content policy + guardrail (1 h) — DONE 2026-07-02 (B2 closed same day)
-
-- `[x]` **B1. Binary-in-git gate**: `scripts/check-hygiene.ts` — FAILS on
-  any tracked file that is (a) >1 MB, or (b) matches `*.bin|*.safetensors|
-  *.dylib|*.gguf|*.metallib` — outside an explicit, size-capped, justified
-  allowlist. Wired into `scripts/test.sh` (runs before tests) AND CI
-  (`.github/workflows/ci.yml` `hygiene` job, gates typecheck). Supports
-  `--staged` for pre-commit use. This is the "can't re-form" guarantee.
-- `[x]` **B2. fixtures/ audit**: 13 MB, 2 tracked `.safetensors`.
-  Decision per file:
-  - `fixtures/adapters/{upper,french}/adapters.safetensors` (6.6 MB each):
-    KEEP tracked, on the gate's explicit allowlist (cap 7 MB). They are
-    stable one-time-trained LoRA adapter INPUTS (trained 2026-06-10, never
-    churned) for the opt-in `MLX_BUN_TEST_LORA` hot-swap test, which also
-    requires the e4b base. The tracked `adapter_config.json` + `data-*/`
-    jsonl fully describe how to reproduce them; a bit-exact regen script
-    (trainer-of-record: optiq `lora train` vs mlx-lm — the saved config is
-    ambiguous) is a Josh task to confirm + smoke. NOT untracked now because
-    an untested untrack would break the lora hot-swap parity test, and these
-    do not churn (the goldens-bloat problem was *churning* artifacts). This
-    is a deliberate allowlist exception, not the default `fixtures/ ≤ 2 MB`
-    policy.
-  - **CLOSED later the same day**: untracked + gitignored after pinning the
-    bytes by sha256 in `scripts/fetch-test-fixtures.sh` (fetch from the
-    `test-fixtures-v1` release / either laptop / pre-rewrite history; the
-    gated LoRA test skips cleanly when absent — verified). The regen-
-    ambiguity concern was real but orthogonal: preservation is by pinned
-    original bytes, not by retraining. Allowlist entry removed; gate green
-    with ZERO multi-MB binaries tracked.
-- `[x]` **B3. tests/ binary audit**: 27 tracked `.bin` under
-  `tests/fixtures/universal-rope/` (4–8 KB each, ~108 KB total). All are
-  model-free CI-load-bearing bit-exact oracle fixtures (used by
-  `tests/universal-rope.test.ts`, which skips cleanly if absent),
-  regenerated by `scripts/gen-universal-rope-fixtures.py`. STAY, on the
-  allowlist (cap 16 KB each). Also allowlisted: `tests/fixtures/qwen-delta-golden.json`
-  (1.08 MB text JSON golden, model-free, regen script exists).
-- Exit: `git ls-files | grep -E '\.(bin|safetensors|dylib)$'` returns only
-  allowlisted entries; the gate enforces it forever.
-
-## Phase C — history rewrite (the 179 MB) — DONE 2026-07-02 (Josh go)
-
-Executed 2026-07-02 on Josh's explicit go. Result: `.git` **182 MB →
-20 MB**. Notes on what actually ran (vs the sketch below):
-
-- C1: dry-run `--analyze` confirmed goldens `.bin` = the entire bloat.
-  Actual invocation, narrower than sketched (no size threshold needed):
-  `git filter-repo --invert-paths --path-glob 'goldens/*.bin'
-  --path-glob 'goldens/*/*.bin'` — both globs needed because git
-  pathspec `**` doesn't span the flat + one-deep layout; covers all 497
-  historical blobs. Verified pre-push: 343 commits in/out, HEAD tree
-  bit-identical to original, zero `.bin` left anywhere in goldens
-  history, tags remapped.
-- C2: force-pushed main + branches + tags. Mirror backup at
-  `~/mlx-bun-mirror-backup-2026-07-02.tar.gz` (143 MB) — delete once
-  both boxes verified. M1 Max done: fetch --force + reset --hard,
-  surviving branches remapped, stale agent-session worktrees/branches
-  deleted (Josh: only main matters), stray `refs/codex`/`refs/original`
-  pruned, reflog expired, gc'd to 20 MB. **M4 Pro remaining (Josh)**:
-  `git fetch origin --tags --force --prune && git reset --hard
-  origin/main` (or fresh clone).
-- C3: `v0.0.10` resolves; release assets untouched (they live outside
-  git history); hygiene gate green; regen path proven live —
-  `bun scripts/regen-fused-sdpa-goldens.ts` rewrote
-  `goldens/apple-m1-max/fused-sdpa.{json,bin}`. The full suite could not
-  be run from the executing session (tooling restriction); run it once
-  per box as the final check.
-
-**Now ONE COMMAND**: `scripts/history-rewrite.sh` (dry-run by default,
-`MLX_BUN_REWRITE_GO=1` to execute) — it enforces every precondition below,
-writes the permanent mirror backup first, purges goldens `.bin` + the
-adapter safetensors from history, force-pushes, and prints the re-clone
-steps for the other laptop.
-
-One-time, destructive-by-design, ~15 min total. Preconditions: BOTH
-laptops fully pushed (verify `git log origin/main..HEAD` empty on each);
-no open PRs; a fresh `git clone --mirror` tarball kept as belt-and-
-suspenders until both machines are re-cloned.
-
-- `[x]` **C1. rewrite**: `git filter-repo --strip-blobs-bigger-than 200K
-  --path-glob 'goldens/**/*.bin' --invert-paths` (exact invocation to be
-  dry-run first with `--analyze`; target: drop all historical goldens
-  `.bin` while keeping every text file's history intact).
-- `[x]` **C2. force-push** `main` (+ tags), re-clone on BOTH laptops
-  (M1 Max + M4 Pro), re-link local-only assets (nothing tracked is lost —
-  the blobs being dropped are regenerable and already untracked at HEAD).
-- `[x]` **C3. verify**: `du -sh .git` (expect ~15–25 MB), `bash
-  scripts/test.sh` green, one `scripts/regen-*` smoke to prove the regen
-  path, tag `v0.0.10` still resolves + release assets untouched (GitHub
-  releases store assets outside git history).
-- NOT doing LFS: 290 MB of churning machine-specific artifacts is exactly
-  what LFS quotas punish, and the regen scripts make the artifacts
-  reproducible — a fixture server is overkill for a two-laptop project.
-- Exit: fresh clone < 30 MB ✓ (20 MB); M1 Max on rewritten history ✓;
-  M4 Pro pending its one-line reset (see notes above).
-
-## Phase D — docs debt (follow-the-map, 1–2 h, independent) — gate DONE 2026-07-02
-
-- `[~]` **D1.** PLAN.md is 139 KB — move closed phases older than Phase 17
-  into PLAN-archive.md (the established pattern; keep PLAN.md scannable).
-  - **Finding (2026-07-02):** no candidates currently. Every phase <17 in
-    PLAN.md (6, 7, 12, 13, 14, 15, 16) is marked `[~]` or `[ ]` — none are
-    `[x]`/CLOSED, and the established archive pattern archives only
-    `[x]`-marked phases. Phases 6 and 16 have *all* their checkboxes
-    checked but carry a deliberate `[~]`; reclassifying them to closed +
-    archiving is a Josh state-call (they may have ongoing narrative
-    threads). The closed `oMLX` phase (2026-07-02) and Phase 19 are
-    *newer* than 17, so they stay in PLAN.md as recent history per the
-    criterion. Revisit D1 when a <17 phase is confirmed closed.
-- `[x]` **D2.** `docs/archive/investigations/curve-runs/` + two untracked HTML
-  strays in docs/investigations: gitignore or move under `reports/`.
-  - Already satisfied: `docs/archive/investigations/curve-runs/` is gitignored
-    (`.gitignore`), and the four HTML strays (`curve-terrain.html`,
-    `hlg-ab.html`, `hlg-report.html`, `hlg-explorer.html`) are gitignored
-    via `docs/archive/investigations/*.html`. No action needed.
-- `[x]` **D3.** The CLAUDE.md doc-map lists are hand-maintained; add a
-  `scripts/check-hygiene.ts` assertion that every `docs/**/*.md` appears in
-  the map (same guardrail spirit as B1).
-  - Implemented in `check-hygiene.ts` (`docs-map` check). First run caught
-    10 drifted docs (3 design, 1 investigation, 6 planning) — all added to
-    the CLAUDE.md map. Gate now green; CI enforces it on every push/PR.
-
-## Ordering & ownership
-
-A and D anytime (safe, agent-executable solo) — DONE 2026-07-02. B gate
-DONE 2026-07-02 (B2 adapter-untrack deferred to Josh — see B2 finding).
-C executed 2026-07-02 on Josh's explicit go — only remaining tail is the
-M4 Pro reset + deleting the backup tarball once both boxes are green.
-
-## Phase D — work-artifact hygiene + rules (2026-08-18) — DONE
-
-Trigger: Josh — "this is a software project; we should not be adding all
-these random work things into it." Compared against ml-explore/mlx-lm's
-root (9 files, 4 dirs, zero work artifacts).
-
-- **D1 root sweep**: six tracked `benchmarks-serve-2026-07-0*.md` dumps
-  git-rm'd (they leaked past a gitignore pattern that only covered
-  `benchmarks-h2h-*`); seven untracked root bench artifacts moved to
-  `reports/bench-archive/`; `tmp/` deleted. `.gitignore` now blocks
-  `/benchmarks-*`, `/bench-*`, `/tmp/`, `/*.log` at root.
-- **D2 guardrail**: `scripts/check-hygiene.ts` gained a third gate —
-  tracked ROOT files must be on an explicit allowlist (16 meta files).
-- **D3 CONTRIBUTING.md**: the rules (what-goes-where table; no dated
-  artifacts; models → `~/models`; docs lifecycle; PLAN/STATUS discipline;
-  external oracle venv is not a repo).
-- **D4 docs archive**: all 20 `docs/investigations/` → 
-  `docs/archive/investigations/` (investigations document finished work);
-  5 release-notes + journal + 2 chat-ui visions + 2 pass-plans →
-  `docs/archive/planning/`; `hlg-sampling.md` (self-declared superseded) →
-  `docs/archive/`. All inbound links rewritten; docs-map gate green.
-- **D5 journal trim (rule 4)**: STATUS.md 1648 → 106 lines (Colibri
-  gate-history + all "where we were" + the stale July ranked backlog →
-  PLAN-archive under "STATUS archive 2026-08-18"); PLAN.md 3815 → 2679
-  (8 closed `[x]` phase blocks → PLAN-archive with one-line pointers).
-- Model artifacts moved out of the tree entirely (`~/models/`) earlier
-  the same day.
-
-Remaining (curated, not mechanical — needs judgment per file):
-- `docs/design/` still holds 46 docs; many describe LANDED work and
-  should archive under rule 3 (candidates: optimization_plan,
-  faithful-base superseded pair, diffusion/audio/omlx/dspark docs whose
-  phases closed). Do this per-doc with link fixes, not as a bulk move.
-- `docs/planning/` leftovers (chunk-task-roadmap, curve-sampler-research-
-  plan, memory-docs-and-dag-plan, pi-integration-review) — keep-or-archive
-  per Josh.
-- `scripts/experiments/` = 199 files; prune ones whose findings are fully
-  recorded in archived docs (keep regen/oracle scripts).
-- `runs/` is 32 GB machine-local (Colibri evidence + quant lab arms) —
-  delete lab arms once the TQ/GPTQ program closes; Colibri evidence dirs
-  are cited from the archive and stay.

package/docs/design/runtime-isolation.md

@@ -1,263 +0,0 @@
-# Runtime isolation — the AI may crash, the UI never may
-
-Status: **P1 LANDED 2026-07-05** (opt-in `--isolate`; default flip after
-bake). Phase 1 (yields) fully landed same day (the gateway onToken hop
-existed; prefill chunk yields added). **ARCHITECTURE DECISION (2026-07-05,
-deviates from the IPC sketch below):** the engine child is the ENTIRE
-existing server on a unix domain socket; the parent is a thin reverse
-proxy (src/serve/isolate.ts). Rationale: the gateway-as-IPC-client cut
-would serialize grammar controllers (WASM), vision embeddings (GPU
-arrays), sampler closures, and prompt-cache hooks over a message protocol
-— weeks of re-plumbing; HTTP-over-UDS reuses the whole serving stack
-verbatim, streams SSE for free, and cancellation is abort propagation.
-The inter-process API is the existing /v1 surface — no second protocol.
-**Measured (M1 Max, cpm5, paired)**: proxied 197.2 vs direct 197.9 tok/s
-(−0.4%, noise), TTFT +2 ms, SSE per-token granularity preserved (96 reads
-/ 96 tokens), parent answers in 0.6 ms mid-decode (criterion: <50 ms),
-child kill → 502 + respawn (gated model-free + E2E:
-tests/isolate-proxy.test.ts, tests/isolate-e2e.test.ts). Not proxied yet:
-/ws/chat (501 + pointer). P2 = child-per-model pool (multi-model switch by
-spawn-overlap); P3 = GPU-lease unification. The original design follows.
-Owner: serving layer / runtime
-Default today: inference runs in-process unless `--isolate` is passed.
-
-## Goal
-
-One `mlx-bun` process plays three or four roles at once:
-
-1. **Serves the web UI** (static assets, the chat page).
-2. **Serves the API** (`/v1/*` for the web UI, the CLI, and external
-   agents).
-3. **Runs the AI** (weight load, prefill, decode — the heavy GPU work).
-
-One executable doing all of this is the product's whole charm, and for
-most people, most of the time, it is perfect. It breaks in one specific
-way: when the AI is working hard — say an agentic pipeline classifying a
-photo library — the **web UI lags**. Page refreshes stall, `/status`
-polls hang, chat messages queue behind the GPU.
-
-Waiting for a *chat reply* is fine and expected (local inference takes
-time; hosted services take up to a minute too). A **laggy UI is neither
-normal nor acceptable.** The UI must stay responsive no matter what the
-AI is doing — and if the AI process dies (an uncatchable GPU OOM), the UI
-must survive and report it, not go down with it.
-
-## Root cause — blocking work on the event-loop thread
-
-The server is a single Bun event loop. The governing principle of an
-event loop is: **the loop thread may only ever do non-blocking work.**
-We violate it by running the synchronous MLX FFI eval loop *on* that
-thread.
-
-Concretely:
-
-- **The per-token GPU read is a blocking FFI call.** Each decode step
-  ends with `ops.itemUint32(cur)` → `a.eval()` → `mlx_eval`
-  (`src/mlx/ops.ts:471`), which parks the single JS thread until the GPU
-  returns. Compute is dispatched async (`asyncEvalAll` + decode
-  pipelining, `src/generate.ts:386`) — good — but reading the value back
-  over `bun:ffi` is a hard, synchronous wait on the main thread. There is
-  no thread handoff.
-- **The loop yields only *microtasks*, not *macrotasks*.** Bun services
-  I/O — accepting connections, answering `/status`, serving the page —
-  only at macrotask boundaries. A bare `yield` in the async generator
-  plus an `await` of an already-resolved value drains the microtask
-  queue without ever reaching the I/O phase.
-- **Streaming partially hides this; non-streaming does not.** The SSE
-  path's `onToken` returns `new Promise(r => setImmediate(r))` ~every
-  25 ms (`src/server.ts:1341`) — a real macrotask yield, so the page
-  breathes. The **non-streaming** path's `onToken` returns `undefined`
-  every token (`src/server.ts:1407`), so the whole completion is one
-  blocking-FFI-plus-microtask chain and the loop is starved start to
-  finish.
-- **Prefill never yields at all.** The chunked prompt eval calls
-  `ops.evalAll(...)` synchronously with no `await` between chunks
-  (`src/generate.ts:294`). A long prompt freezes the page for the full
-  prefill *even in streaming mode*.
-
-### Why yielding alone is not the fix
-
-A `setImmediate` yield can interleave I/O *between* tokens, but it
-**cannot interrupt an in-flight `mlx_eval`** — that C call is atomic from
-JS's view. Each token blocks for tens of ms; a prefill blocks one big
-chunk (hundreds of ms to seconds) with no yield point possible inside the
-kernel. So perfect yielding gives the UI thin inter-token slices, not
-isolation. Adequate for "chat plus occasional"; inadequate for "agentic
-pipeline running flat-out while I refresh the page." For true isolation
-the blocking work must leave the thread.
-
-## Decision — subprocess, not worker thread
-
-Move the AI runtime off the event-loop thread. Two ways:
-
-- **Worker thread** (Bun `Worker`): shares the process, cheapest IPC
-  (a `SharedArrayBuffer` token ring). But a GPU OOM / SIGTRAP — which our
-  own notes flag as **uncatchable** (CLAUDE.md "async GPU exec OOM is
-  uncatchable") — takes down the whole process, **including the UI
-  server**.
-- **Subprocess** (`Bun.spawn`): crash isolation. An MLX OOM kills the
-  inference process; the UI/API server survives, reports it, and
-  respawns. Heavier IPC, but on a single-user local box a pipe/socket is
-  trivially fast next to token cadence.
-
-**Decision: subprocess.** The product rule is "the AI may crash, the UI
-never may," and only address-space isolation delivers that against an
-uncatchable fault. It also reuses machinery the repo has already proven.
-
-## We already built the blueprint — for jobs
-
-`src/jobs/runner.ts` already does exactly this for the *other* heavy GPU
-consumers (quantize / finetune):
-
-- `submitSubprocess()` creates a job row and `Bun.spawn`s the binary as a
-  child (`src/jobs/runner.ts:210`, entry `src/jobs/job-entry.ts`).
-- A single global **GPU lease** (`gpuLeaseHolder`) serializes GPU access;
-  `isGpuBusy()` / `currentGpuJob()` (`src/jobs/runner.ts:159`) expose it,
-  **and the server already gates token generation on it** so
-  quantize/finetune don't fight live inference.
-- IPC is a file-backed SQLite job store + stdout/stderr line pumping into
-  a log + an SSE tail (`src/jobs/sse.ts`).
-
-So the pattern — heavy GPU work → its own process → server thread stays
-free, serialized by one lease — is established and tested. **Inference is
-the one heavy consumer still running inline on the event-loop thread.**
-It is the exception, and it is the one people hit on every message.
-
-## The seam already exists
-
-Request handlers do not call `generate()` directly — they call
-`gateway.run(promptIds, options, onToken, vision, shape)`
-(`src/serve/generation-gateway.ts:108`, call sites `src/server.ts:1331`
-and `:1407`). The `GenerationGateway` is the abstraction boundary between
-the HTTP layer and execution. **We replace what is behind it** (inline
-`generate()` → an IPC client that talks to the inference subprocess); the
-HTTP/SSE layer, the `onToken` contract, and the `false`-halts-generation
-semantics are unchanged.
-
-## Target architecture
-
-- **Main process = pure reactor.** `Bun.serve` does static UI + the API
-  surface + routing + tokenization/templating/SSE encoding (all cheap,
-  non-blocking JS). **Zero MLX calls.** Once generation is off-thread,
-  nothing else on the main thread blocks — static serves and `/status`
-  are already async; they lag today only because the loop is
-  monopolized. So this alone makes the UI instant.
-- **Inference runtime = a persistent subprocess** owning the model +
-  Metal context, doing prefill/decode, streaming tokens back. Persistent
-  (not spawn-per-request): weight load is the cost, can't pay it per
-  request. One process, one model, one GPU.
-- **One GPU lease governs all consumers** — live inference, agentic API
-  calls, quantize, finetune. The GPU is the single serialized resource
-  (matching the single-user reality: chat waiting behind the photo
-  pipeline is the *acceptable* kind of waiting), while UI/API I/O is
-  fully parallel and never waits.
-
-```
-┌──────────────────────────────────────────────┐
-│  main process (event loop — never blocks)     │
-│  Bun.serve: static UI · /v1/* API · routing   │
-│  tokenize · template · SSE encode             │
-│        │  gateway.run() = IPC client          │
-└────────┼──────────────────────────────────────┘
-         │  request ↓        ↑ token stream
-┌────────┼──────────────────────────────────────┐
-│  inference subprocess (may block freely)       │
-│  owns model + Metal · prefill · decode         │
-│  holds the GPU lease while generating          │
-└────────────────────────────────────────────────┘
-   (crash here ⇒ parent respawns; UI untouched)
-```
-
-## IPC sketch
-
-- **Transport:** Bun supports a structured `ipc` channel on `Bun.spawn`
-  (parent ⇄ child message passing); a unix domain socket is the
-  fallback. Tokens are slow relative to message passing, so per-token
-  messages are fine; batch a few if needed. (Jobs use stdout-line + the
-  SQLite store, which is right for fire-and-forget long jobs but too
-  coarse for low-latency per-token streaming — inference wants the
-  message channel.)
-- **Request → child:** `{ promptIds, options, shape }`. Tokenization and
-  template rendering stay in the parent (cheap); the child receives
-  token ids.
-- **Child → parent:** a stream of `{ token }` messages, then a final
-  `{ stats }`. The parent's gateway client invokes the handler's existing
-  `onToken` per message — per-row SSE fan-out keeps working unchanged.
-- **Cancellation / backpressure:** `onToken` returning `false` (stop
-  sequence) or a client disconnect must abort the in-flight generation —
-  send a cancel message; the child checks it at the decode-loop boundary
-  (the natural place `generate()` already yields).
-- **Crash handling:** child `exited` non-zero ⇒ fail any in-flight
-  request with a clean 5xx, respawn the child, reload the model. The UI
-  server never noticed.
-
-## Lease reconciliation (the one genuinely new problem)
-
-Jobs are *transient* (spawn → run → exit, lease released on exit). The
-inference worker is *persistent* (resident, holds the model in RAM). They
-must share one GPU coherently:
-
-- A quantize/finetune job needs the GPU while the inference worker is
-  resident. Options: (a) the inference worker **drains and yields** the
-  lease for the job's duration (finish current gen, pause admission,
-  release), then resumes; (b) the persistent worker **also runs the
-  jobs** (one GPU owner, no cross-process lease at all). (b) is cleaner
-  long-term but couples job lifecycle into the inference process; (a)
-  keeps jobs as-is. **Open — pick during phase 3.**
-- On 24 GB, the resident model already dominates memory; a concurrent
-  finetune that also needs multi-GB is an OOM footgun regardless of
-  process layout. The lease must remain *exclusive*, never truly
-  concurrent on the GPU.
-
-## What this unlocks / simplifies
-
-- **Delete the yield hacks.** Once the loop blocks only in the child, the
-  `setImmediate` throttle (`src/server.ts:1341`) and any prefill
-  chunk-yielding (`src/generate.ts:294`) are no longer needed for UI
-  responsiveness — the hot path gets cleaner and marginally faster.
-- **Folds into model hot-swap** (see PRODUCT_ROADMAP "Model hot-swap from
-  the UI"): the model lives in the worker, so `loadContext` and swap
-  happen there while the main thread serves a live "swapping…" state.
-  These two items converge on "the model is not on the main thread."
-- **Flags the next blocker:** vision preprocessing (`Bun.Image` decode,
-  `src/vision/preprocess.ts`) is also main-thread CPU work and should
-  ride into the worker, or it becomes the next source of lag.
-
-## Staging (incremental, not big-bang)
-
-- **Phase 1 — stop the bleeding (in-process).** Macrotask-yield the
-  non-streaming `onToken` (`src/server.ts:1407`) and the prefill chunk
-  loop (`src/generate.ts:294`), mirroring what streaming already does.
-  Trivial change; kills the acute hang for the common case. Does **not**
-  isolate the UI under sustained load — that's Phase 2.
-- **Phase 2 — inference subprocess behind the gateway.** Stand up the
-  persistent inference child + IPC; make `GenerationGateway` an IPC
-  client. Main process makes zero MLX calls. The HTTP layer is untouched.
-  This is the real fix.
-- **Phase 3 — unify the GPU lease.** Bring inference under the same lease
-  as quantize/finetune so all GPU consumers schedule coherently; resolve
-  the persistent-vs-transient reconciliation above.
-
-## Exit criteria
-
-- Under a sustained generation load (e.g. a tight `/v1` loop), a parallel
-  web-UI page load and `/status` poll return in **< 50 ms** (today: stalls
-  for the full completion in non-streaming, inter-token in streaming).
-- Killing the inference process mid-generation leaves the UI server up; a
-  fresh page load and `/status` still succeed, and the in-flight request
-  returns a clean error rather than a hang.
-- No throughput regression vs the in-process path on the single-stream
-  decode benchmark (token IPC overhead is below noise).
-
-## Open questions
-
-- Bun `ipc` channel vs unix socket vs `SharedArrayBuffer` ring for the
-  token stream — measure latency/overhead before committing.
-- One inference worker only (single GPU, single model) — confirmed by the
-  single-user model. Multiple models = future, and pairs with hot-swap.
-- Lease reconciliation strategy (a) drain-and-yield vs (b) worker runs
-  jobs — decide in Phase 3.
-- Where does prompt-cache live — in the worker (it holds KV bytes, must be
-  byte-capped per CLAUDE.md). Almost certainly the worker.
-- Startup UX: parent up and serving the UI immediately; worker model load
-  happens behind a "loading…" state the parent reports.