mlx-bun 0.0.10 → 0.0.11

package/docs/design/repo-cleanup-plan.md

@@ -0,0 +1,189 @@
+# Repo cleanup plan
+
+Status: PLANNED (2026-07-02, written post-v0.0.10). The audit behind it:
+`.git` was 179 MB, driven by 497 machine-specific golden `.bin` blobs
+(~290 MB working tree, every regen compounding history); root has ~30
+entries of mixed tracked/untracked provenance; `repro/` was undocumented.
+Already done in v0.0.10: `.bin` goldens untracked + gitignored +
+`goldens/README.md` policy; CLAUDE.md root-layout map. This plan is the
+rest, phased so each step is safe, reversible (except C1, which is the
+point), and independently shippable.
+
+Principle: the hygiene discipline IS the product edge — every phase ends
+with a guardrail so the mess can't re-form, not just a one-time sweep.
+
+## What can actually be lost? Nothing. (Read this before re-litigating.)
+
+Every binary this plan untracks or purges survives in MULTIPLE independent
+places, so "we might lose the goldens/fixtures" is not a reason to stall:
+
+1. **Both dev laptops' working trees** — `git rm --cached` and `.gitignore`
+   never touch disk; the files sit exactly where they were.
+2. **Git history** — until Phase C runs, every previously-committed byte is
+   still in `.git` and recoverable with `git show <rev>:<path>`.
+3. **The Phase-C mirror backup** — `scripts/history-rewrite.sh` refuses to
+   rewrite before writing `~/mlx-bun-history-backup-<date>.tar` (a full
+   pre-rewrite mirror, kept forever). Even after the purge, every historical
+   blob is one `tar -x` away.
+4. **Regen scripts** — goldens regenerate from `scripts/regen-*` + the
+   oracle venv; the universal-rope fixtures from their gen script.
+5. **Pinned-hash fetch** — the two LoRA adapter fixtures are sha256-pinned
+   in `scripts/fetch-test-fixtures.sh`, sourced from the `test-fixtures-v1`
+   GitHub release (one command to create — see the script header) or
+   either laptop.
+
+And no test can break: every suite that reads a binary fixture gates on
+FILE PRESENCE and skips cleanly when it's absent (goldens via
+tests/goldens.ts, LoRA via `haveAdapters`, universal-rope via its own
+check). Untracked ≠ deleted ≠ unrunnable.
+
+## Phase A — root consolidation (30 min, zero risk) — DONE 2026-07-02
+
+- `[x]` **A1. `lab/`**: `git mv repro lab/repro && git mv spikes lab/spikes`.
+  One home for "code that proves a point but isn't the product": upstream
+  bug reproductions and feasibility probes. Update the CLAUDE.md layout map
+  + the two docs that reference `repro/` (grep first: `grep -rn "repro/"
+  docs/ *.md`).
+  - Also updated every path reference (CLAUDE.md, PLAN.md, PLAN-archive.md,
+    docs/investigations/lab-build-journal.md, docs/design/docs-reorg-plan.md,
+    and the `repro/`-pointing comments in src/mlx/{ffi,array,autograd,ops}.ts
+    + tests/{ffi-jit,train-autograd}.test.ts + the repro's own ISSUE.md).
+- `[x]` **A2. root strays** (all untracked; no git surgery): delete
+  `train.log`; move `benchmarks-e4b-flag-matrix-2026-06-15.json` into
+  `reports/`; confirm `.DS_Store` is gitignored globally.
+  - `train.log` deleted; benchmark json moved to `reports/`; `.DS_Store`
+    already globally gitignored (`.gitignore`).
+- `[x]` **A3. `archive/` → `docs/archive/`**: the two frozen HTML reports
+  are docs, not a root concern. `git mv`, update CLAUDE.md map.
+- Exit: root = code (`src tests scripts`), product surfaces (`docs website
+  packaging bin extensions`), fixtures (`goldens fixtures lab`), meta
+  (README/PLAN/STATUS/CLAUDE + configs), and gitignored working dirs. Every
+  tracked root entry appears in the CLAUDE.md layout map.
+
+## Phase B — tracked-content policy + guardrail (1 h) — DONE 2026-07-02 (B2 closed same day)
+
+- `[x]` **B1. Binary-in-git gate**: `scripts/check-hygiene.ts` — FAILS on
+  any tracked file that is (a) >1 MB, or (b) matches `*.bin|*.safetensors|
+  *.dylib|*.gguf|*.metallib` — outside an explicit, size-capped, justified
+  allowlist. Wired into `scripts/test.sh` (runs before tests) AND CI
+  (`.github/workflows/ci.yml` `hygiene` job, gates typecheck). Supports
+  `--staged` for pre-commit use. This is the "can't re-form" guarantee.
+- `[x]` **B2. fixtures/ audit**: 13 MB, 2 tracked `.safetensors`.
+  Decision per file:
+  - `fixtures/adapters/{upper,french}/adapters.safetensors` (6.6 MB each):
+    KEEP tracked, on the gate's explicit allowlist (cap 7 MB). They are
+    stable one-time-trained LoRA adapter INPUTS (trained 2026-06-10, never
+    churned) for the opt-in `MLX_BUN_TEST_LORA` hot-swap test, which also
+    requires the e4b base. The tracked `adapter_config.json` + `data-*/`
+    jsonl fully describe how to reproduce them; a bit-exact regen script
+    (trainer-of-record: optiq `lora train` vs mlx-lm — the saved config is
+    ambiguous) is a Josh task to confirm + smoke. NOT untracked now because
+    an untested untrack would break the lora hot-swap parity test, and these
+    do not churn (the goldens-bloat problem was *churning* artifacts). This
+    is a deliberate allowlist exception, not the default `fixtures/ ≤ 2 MB`
+    policy.
+  - **CLOSED later the same day**: untracked + gitignored after pinning the
+    bytes by sha256 in `scripts/fetch-test-fixtures.sh` (fetch from the
+    `test-fixtures-v1` release / either laptop / pre-rewrite history; the
+    gated LoRA test skips cleanly when absent — verified). The regen-
+    ambiguity concern was real but orthogonal: preservation is by pinned
+    original bytes, not by retraining. Allowlist entry removed; gate green
+    with ZERO multi-MB binaries tracked.
+- `[x]` **B3. tests/ binary audit**: 27 tracked `.bin` under
+  `tests/fixtures/universal-rope/` (4–8 KB each, ~108 KB total). All are
+  model-free CI-load-bearing bit-exact oracle fixtures (used by
+  `tests/universal-rope.test.ts`, which skips cleanly if absent),
+  regenerated by `scripts/gen-universal-rope-fixtures.py`. STAY, on the
+  allowlist (cap 16 KB each). Also allowlisted: `tests/fixtures/qwen-delta-golden.json`
+  (1.08 MB text JSON golden, model-free, regen script exists).
+- Exit: `git ls-files | grep -E '\.(bin|safetensors|dylib)$'` returns only
+  allowlisted entries; the gate enforces it forever.
+
+## Phase C — history rewrite (the 179 MB) — DONE 2026-07-02 (Josh go)
+
+Executed 2026-07-02 on Josh's explicit go. Result: `.git` **182 MB →
+20 MB**. Notes on what actually ran (vs the sketch below):
+
+- C1: dry-run `--analyze` confirmed goldens `.bin` = the entire bloat.
+  Actual invocation, narrower than sketched (no size threshold needed):
+  `git filter-repo --invert-paths --path-glob 'goldens/*.bin'
+  --path-glob 'goldens/*/*.bin'` — both globs needed because git
+  pathspec `**` doesn't span the flat + one-deep layout; covers all 497
+  historical blobs. Verified pre-push: 343 commits in/out, HEAD tree
+  bit-identical to original, zero `.bin` left anywhere in goldens
+  history, tags remapped.
+- C2: force-pushed main + branches + tags. Mirror backup at
+  `~/mlx-bun-mirror-backup-2026-07-02.tar.gz` (143 MB) — delete once
+  both boxes verified. M1 Max done: fetch --force + reset --hard,
+  surviving branches remapped, stale agent-session worktrees/branches
+  deleted (Josh: only main matters), stray `refs/codex`/`refs/original`
+  pruned, reflog expired, gc'd to 20 MB. **M4 Pro remaining (Josh)**:
+  `git fetch origin --tags --force --prune && git reset --hard
+  origin/main` (or fresh clone).
+- C3: `v0.0.10` resolves; release assets untouched (they live outside
+  git history); hygiene gate green; regen path proven live —
+  `bun scripts/regen-fused-sdpa-goldens.ts` rewrote
+  `goldens/apple-m1-max/fused-sdpa.{json,bin}`. The full suite could not
+  be run from the executing session (tooling restriction); run it once
+  per box as the final check.
+
+**Now ONE COMMAND**: `scripts/history-rewrite.sh` (dry-run by default,
+`MLX_BUN_REWRITE_GO=1` to execute) — it enforces every precondition below,
+writes the permanent mirror backup first, purges goldens `.bin` + the
+adapter safetensors from history, force-pushes, and prints the re-clone
+steps for the other laptop.
+
+One-time, destructive-by-design, ~15 min total. Preconditions: BOTH
+laptops fully pushed (verify `git log origin/main..HEAD` empty on each);
+no open PRs; a fresh `git clone --mirror` tarball kept as belt-and-
+suspenders until both machines are re-cloned.
+
+- `[x]` **C1. rewrite**: `git filter-repo --strip-blobs-bigger-than 200K
+  --path-glob 'goldens/**/*.bin' --invert-paths` (exact invocation to be
+  dry-run first with `--analyze`; target: drop all historical goldens
+  `.bin` while keeping every text file's history intact).
+- `[x]` **C2. force-push** `main` (+ tags), re-clone on BOTH laptops
+  (M1 Max + M4 Pro), re-link local-only assets (nothing tracked is lost —
+  the blobs being dropped are regenerable and already untracked at HEAD).
+- `[x]` **C3. verify**: `du -sh .git` (expect ~15–25 MB), `bash
+  scripts/test.sh` green, one `scripts/regen-*` smoke to prove the regen
+  path, tag `v0.0.10` still resolves + release assets untouched (GitHub
+  releases store assets outside git history).
+- NOT doing LFS: 290 MB of churning machine-specific artifacts is exactly
+  what LFS quotas punish, and the regen scripts make the artifacts
+  reproducible — a fixture server is overkill for a two-laptop project.
+- Exit: fresh clone < 30 MB ✓ (20 MB); M1 Max on rewritten history ✓;
+  M4 Pro pending its one-line reset (see notes above).
+
+## Phase D — docs debt (follow-the-map, 1–2 h, independent) — gate DONE 2026-07-02
+
+- `[~]` **D1.** PLAN.md is 139 KB — move closed phases older than Phase 17
+  into PLAN-archive.md (the established pattern; keep PLAN.md scannable).
+  - **Finding (2026-07-02):** no candidates currently. Every phase <17 in
+    PLAN.md (6, 7, 12, 13, 14, 15, 16) is marked `[~]` or `[ ]` — none are
+    `[x]`/CLOSED, and the established archive pattern archives only
+    `[x]`-marked phases. Phases 6 and 16 have *all* their checkboxes
+    checked but carry a deliberate `[~]`; reclassifying them to closed +
+    archiving is a Josh state-call (they may have ongoing narrative
+    threads). The closed `oMLX` phase (2026-07-02) and Phase 19 are
+    *newer* than 17, so they stay in PLAN.md as recent history per the
+    criterion. Revisit D1 when a <17 phase is confirmed closed.
+- `[x]` **D2.** `docs/investigations/curve-runs/` + two untracked HTML
+  strays in docs/investigations: gitignore or move under `reports/`.
+  - Already satisfied: `docs/investigations/curve-runs/` is gitignored
+    (`.gitignore`), and the four HTML strays (`curve-terrain.html`,
+    `hlg-ab.html`, `hlg-report.html`, `hlg-explorer.html`) are gitignored
+    via `docs/investigations/*.html`. No action needed.
+- `[x]` **D3.** The CLAUDE.md doc-map lists are hand-maintained; add a
+  `scripts/check-hygiene.ts` assertion that every `docs/**/*.md` appears in
+  the map (same guardrail spirit as B1).
+  - Implemented in `check-hygiene.ts` (`docs-map` check). First run caught
+    10 drifted docs (3 design, 1 investigation, 6 planning) — all added to
+    the CLAUDE.md map. Gate now green; CI enforces it on every push/PR.
+
+## Ordering & ownership
+
+A and D anytime (safe, agent-executable solo) — DONE 2026-07-02. B gate
+DONE 2026-07-02 (B2 adapter-untrack deferred to Josh — see B2 finding).
+C executed 2026-07-02 on Josh's explicit go — only remaining tail is the
+M4 Pro reset + deleting the backup tarball once both boxes are green.

package/docs/design/runtime-isolation.md

@@ -1,9 +1,25 @@
 # Runtime isolation — the AI may crash, the UI never may
 
-Status: **design** (not started — deferred future work; no scheduled PLAN.md phase)
+Status: **P1 LANDED 2026-07-05** (opt-in `--isolate`; default flip after
+bake). Phase 1 (yields) fully landed same day (the gateway onToken hop
+existed; prefill chunk yields added). **ARCHITECTURE DECISION (2026-07-05,
+deviates from the IPC sketch below):** the engine child is the ENTIRE
+existing server on a unix domain socket; the parent is a thin reverse
+proxy (src/serve/isolate.ts). Rationale: the gateway-as-IPC-client cut
+would serialize grammar controllers (WASM), vision embeddings (GPU
+arrays), sampler closures, and prompt-cache hooks over a message protocol
+— weeks of re-plumbing; HTTP-over-UDS reuses the whole serving stack
+verbatim, streams SSE for free, and cancellation is abort propagation.
+The inter-process API is the existing /v1 surface — no second protocol.
+**Measured (M1 Max, cpm5, paired)**: proxied 197.2 vs direct 197.9 tok/s
+(−0.4%, noise), TTFT +2 ms, SSE per-token granularity preserved (96 reads
+/ 96 tokens), parent answers in 0.6 ms mid-decode (criterion: <50 ms),
+child kill → 502 + respawn (gated model-free + E2E:
+tests/isolate-proxy.test.ts, tests/isolate-e2e.test.ts). Not proxied yet:
+/ws/chat (501 + pointer). P2 = child-per-model pool (multi-model switch by
+spawn-overlap); P3 = GPU-lease unification. The original design follows.
 Owner: serving layer / runtime
-Default today: inference runs **in-process**, on the server's event-loop
-thread. This doc proposes moving it to a **subprocess**.
+Default today: inference runs in-process unless `--isolate` is passed.
 
 ## Goal
 

package/docs/design/segmented-backward-training.md

@@ -249,8 +249,9 @@ returns the split indices + which donor K/V to save, computed from the above.
 - The chunk data needs ≥4096 (median 4092; 0% ≤2048; 98% ≤8192). Success = e4b
   trains at 8192 ≤ ~10 GB AND the chunk eval (`scripts/chunk-eval.ts`) shows a
   quality delta on e4b like MiniCPM5's.
-- Training requires `MLX_BUN_PERF_KERNEL=0 MLX_BUN_FUSED_GELU=0` (those fused
-  CustomKernels have no vjp).
+- Training needs no env-flag sanitization (the no-vjp fused CustomKernels that
+  once required `MLX_BUN_PERF_KERNEL=0 MLX_BUN_FUSED_GELU=0` were deleted
+  2026-07-05; see docs/design/unified-engine-frontier-plan.md §6).
 - MLX `Date.now()`/random caveats don't apply here; standard determinism via seed.
 
 ## 7. Key files

package/docs/design/ssd-kv-cold-tier.md

@@ -1,6 +1,22 @@
 # SSD cold tier for the prompt/KV cache
 
 Status: **LANDED P1–P3 (2026-07-02, same day)** — P4 hardening extras remain.
+**Layer-0 upgrade (2026-07-05, unified-engine plan):** D2 finished properly —
+the two-tier take/restore/trim dance moved INSIDE `PromptCache.take()`
+(structural `ColdTier` interface; the server binds SsdCacheStore + model into
+it), so EVERY consumer tiers automatically: the serial lane's hand-rolled
+block in runGeneration is gone, and the BATCH SCHEDULER restores prefixes
+from disk at admission (gated end-to-end: tests/batch-scheduler.test.ts
+"SSD tier through the batch lane"). `PromptCache.onPut` drives the debounced
+write-behind snapshot for both lanes' puts. NEW: **idle demotion**
+(`--ssd-demote-idle`, default 300 s with the tier on) — entries unused past
+the threshold spill to disk and free their GPU arrays; RAM drains between
+agent bursts while prefixes stay reachable via zero-copy restore. The sweep
+runs only when the gateway is fully idle (a runExclusive would otherwise
+drain a live batch). Rationale (Josh): prefill is the cost being avoided —
+12B prefill is 168 tok/s, so a 30k agent context is ~3 min of GPU compute vs
+a ~1 s NVMe restore; SSD-vs-RAM (40×) is the wrong comparison, SSD-vs-
+recompute (50–180×, growing with context) is the real one.
 
 **Measured (M1 Max, MiniCPM5, 13.7k-token prefix, kv_config quant):**
 - Restart TTFT: **12,083 ms (full prefill) → 236 ms** restored — 2% of full
@@ -156,3 +172,117 @@ Total ~7–10 working days.
   discipline) before making footprint claims.
 - **Stale-token poisoning**: `configFingerprint` + `tokenizerHash` + ns;
   chat-template drift degrades to a shorter matched prefix (safe).
+
+## Addendum 2026-07-07 — A7 closure: bounded memory on BOTH sides of the file
+
+The 07-06/07 serve benches flagged elevated ctx/restart-leg RSS on
+--ssd-cache arms ("A7"). Root-caused in three parts; the first two were
+real defects, both fixed in `src/kv-store.ts`:
+
+- **Write side** (residual after the v3 streaming writer of 4f9bad9): the
+  per-tensor `rawBytes()` readback ended in a JS-heap `.slice()`; under GC
+  lag the dead copies outlive the flush loop, re-spiking RSS toward the
+  whole entry. The writer now hashes and writes from a ZERO-COPY view of
+  the contiguous mlx buffer (`MlxArray.rawBytesView`) — no JS-heap copy
+  exists at any point; save-transient mlx memory measured 0 bytes for
+  contiguous sources (one tensor otherwise).
+- **Restore side** (never covered by A7's original scope): the zero-copy
+  restore (COW mmap + `fromPointer`) became a per-restore PROCESS-LIFETIME
+  mapping leak once the 2026-07-06 FFI-dtor fix removed the unmap signal,
+  and the exactly-offset-sized restored buffers made the first
+  post-restore decode step concat-copy the entire entry into fresh mlx
+  memory. Restore is now a **streamed copy** — each tensor is copied into
+  an mlx-owned leaf (`fromBytesCopy`, no dtor contract), its clean file
+  pages dropped (`MADV_DONTNEED`; tensor offsets are 16 KiB-aligned = the
+  arm64 page size), the mapping unmapped before `loadKvCache` returns, and
+  plain-KV tensors land in STEP-rounded capacity with ≥1 token of slack so
+  the first step updates in place. Measured (512 MB synthetic entry): mlx
+  peak during restore+first-step = live entry + ONE tensor (552 vs 520 MB
+  active); `vmmap` confirms no `.mlxkv` mapping survives; 12B real-model
+  cold cache-load → first token 277 ms (parity with the old zero-copy
+  ~240 ms claim — the old path paid the same whole-entry copy on the first
+  decode step anyway). Byte identity of the copy-restore is pinned for all
+  five cache kinds (save → load(verify) → re-save hash-identical;
+  tests/kv-store.test.ts "copy-restore byte identity") plus the real-model
+  bf16/quantized/SSM continuation suites.
+- **What was NOT a defect**: most of the benched leg delta is `ps` RSS
+  ACCOUNTING — the write-behind's hash+write CPU-touches the live KV
+  entry, which makes those already-allocated unified-memory pages visible
+  to `ps` (GPU-written buffers and python-arm KV never show). The bench
+  report now footnotes this instead of promising a pending fix.
+
+Supersedes the D2 line "restored entry carries its MmapFile, unmapped on
+entry dispose" and the D1 "restore = one zero-copy mmap with lazy page
+fault-in" mechanics above — the economics (SSD-vs-recompute) and the
+zero-work-on-token-loop property are unchanged.
+
+## Addendum 2026-07-07 — write-behind scheduling contract: flush only while idle
+
+The 07-07 serve bench showed decode@ctx losing to mlx-lm on the bun arms
+(e4b −9.3%, 12B −3.9%) while short-ctx decode won on every model — and the
+mixed arm (4× smaller flush bytes) BEAT the bf16 arms at decode@ctx. Root
+cause: "non-blocking" write-behind was only non-blocking at the event-loop
+level. Every per-tensor flush step is (1) `ops.contiguous` — a kernel on
+the SAME GPU stream decode uses — then (2) `rawBytesView()` → a synchronous
+`mlx_array_eval` that blocks the JS thread until the stream drains, then
+(3) a synchronous multi-MB `writeSync`. The `setImmediate` pacing
+interleaved those slices exactly between decode tokens, so a ~16k entry's
+flush overlapping the bench's cached ctx repeats taxed their 64-token
+decode windows (reproduced standalone: e4b 9.5k-token entry, repeat decode
+37.9 vs cold 47.1 tok/s; flat 44–46 across all three samples post-fix).
+mlx-lm runs no equivalent background work — the gap was pure self-inflicted
+contention.
+
+**The contract now:** the flush only progresses while the engine is idle.
+- `GenerationGateway.busy` covers BOTH lanes (mutex held/awaited — the
+  serial lane shows zero rows — OR batch rows active/pending);
+  `gateway.onIdle()` is the poll-based waiter (`generation-gateway.ts`).
+- `saveKvCacheAsync`/`SsdCacheStore.storeAsync` accept a per-step
+  `waitTurn` gate awaited before EVERY tensor step (including the first),
+  so a request arriving MID-flush pauses the remaining tensors until idle
+  again; the server passes `() => gateway.onIdle()` at both chain sites
+  (write-behind snapshots AND eviction/demotion spills). Gate failures are
+  swallowed — scheduling advice must never corrupt the write path.
+- `MLX_BUN_SSD_WRITEBEHIND=0` disables write-behind snapshots entirely
+  (kill switch + paired-A/B lever; restart survival then degrades to
+  spill-on-evict only).
+
+Accepted tradeoffs: durability waits for a quiet moment (single-user
+serving quiesces constantly; sustained hammering defers the flush AND the
+spill clones' GPU-memory release, bounded by the serial chain); one
+in-flight tensor step (~10–15 MB) can still land ahead of a just-arrived
+request; and the restart window tightens — the flush now runs in idle gaps
+instead of overlapping decodes, so the bench's 2.5 s pre-kill beat must
+absorb it (verified surviving: e4b 9.5k entry, restart cached=9575 with
+the gate on).
+
+## Addendum 2026-07-07 (later) — bounded retention + admission-safe snapshots
+
+The post-merge review found two operational holes in the contract above:
+
+1. **Snapshot timer could stall batch admission.** `scheduleSsdSnapshot`'s
+   debounced timer fired `gateway.runExclusive` unconditionally; the
+   exclusive registers a serial WAITER, which holds batch admission until
+   every running row drains naturally — a background durability timer
+   periodically freezing admission under `--batch N` sustained load. Fixed
+   with the same guard `demoteIdle` always had: the timer checks
+   `activeRows/pendingRows` first and RE-ARMS (5 s, unref'd) while busy —
+   entries are immutable, a late snapshot is equally valid.
+2. **Starved-gate clone retention was unbounded.** Every eviction/demotion/
+   snapshot chained GPU-pinning zero-copy clones onto a bare serial promise
+   chain, and every flush step awaits `onIdle()` — under continuous traffic
+   the chain starves and resident memory = prompt-cache cap + every queued
+   clone, defeating the byte cap exactly under the load causing evictions.
+   Now all three producers go through **`SpillQueue`** (kv-store.ts):
+   queued bytes are capped (default 2 GB = a quarter of the 8 GB RAM-cache
+   default; `MLX_BUN_SSD_SPILL_QUEUE_GB` overrides) — over cap the OLDEST
+   not-mid-store item drops with its clones disposed immediately. A dropped
+   spill is a future cache miss, never a wrong result: the contention-free
+   alternative to letting the flush cut into decode. `/stats.ssd_cache`
+   exposes `pending_spills` / `pending_spill_bytes` / `dropped_spills`.
+
+Explicitly accepted (documented, not built): **no shutdown flush** — a
+SIGINT/SIGTERM during (or within the debounce after) traffic loses queued
+write-behind entries; restart survival degrades to whatever already
+flushed. This is a best-effort cache tier; the cost of a lost entry is one
+re-prefill. `SpillQueue.drain()` exists if a shutdown hook is ever wanted.