mlx-bun 0.4.0 → 1.0.0

package/docs/archive/investigations/dspark-handoff.md

@@ -1,224 +0,0 @@
-# DSpark — handoff
-
-**State (2026-07-06): the DSpark paper is CODE-COMPLETE and serve-integrated.**
-Architecture proven correct (overfit τ=3.24, paper-range); serve-integrated
-behind `--draft-model` (Phase 1 — the `DraftSource` seam is KV-borrowing-ready;
-sibling AssistantSource verified bit-lossless on real weights,
-`tests/spec-serve-assistant.test.ts`); and every remaining paper component
-landed in Phase 2 the same day: Alg-1 confidence-scheduled draft-length pruning
-(checkpoint-driven, zero change for uncalibrated checkpoints), STS calibration
-(§3.2.1, `calibration.ts` + `scripts/dspark-calibrate.ts`), the RNN sequential
-head (Eq 6, `--seq-head rnn`, init-equivalent to Markov), the tightened draft
-loop (on-device token chaining + deferred conf reads + `collectLogits:false`
-on serve — bit-identity pinned), and the `dspark` variant rename + central
-loader. Suites: smoke 21/21 · dspark tests 17/17 · adjacent server suites
-green. **What remains is exactly the GPU work below** — data scale + 12B
-retarget + train + calibrate + measure; e4b not netting a speedup is a
-data/target-speed problem, not architecture. Full design:
-[docs/design/dspark-speculative-decoding.md](../design/dspark-speculative-decoding.md).
-
-## PATH A (preferred, no training) — DeepSeek's published 12B drafter
-
-DeepSeek released a TRAINED drafter for our exact target:
-`deepseek-ai/dspark_gemma4_12b_block7` (6.86 GB bf16, γ=7; DeepSpec repo is
-the MIT reference impl + the oracle — see the "DeepSpec ground truth" section
-of the design doc). Once the deepspec-variant port lands:
-
-```sh
-# 1. download the trained drafter (~6.9 GB)
-mlx-bun get deepseek-ai/dspark_gemma4_12b_block7
-# 2. oracle gate: dump their reference trace (temp 0 = RNG-free deterministic)
-#    in a torch venv, then compare ours round-for-round (scripts staged)
-# 3. measure + serve (auto-detected via config.json architectures field)
-mlx-bun serve --model gemma-4-12B-it-OptiQ-4bit --draft-model <snapshot-of-dspark_gemma4_12b_block7>
-```
-
-### Oracle protocol (step 2, expanded)
-
-Two staged scripts implement the temp-0 deterministic trace protocol against
-DeepSpec's own reference implementation (github.com/deepseek-ai/DeepSpec, MIT).
-At temperature 0 their leaky-rejection verify degenerates to exact argmax
-token-match — the whole round trace is RNG-free and reproducible, so a
-plain round-for-round diff is a real bit-exact gate (same discipline as the
-D2 diffusion oracle, minus RNG parity entirely — see the design doc's
-"DeepSpec ground truth" section).
-
-```sh
-# a. dump the reference trace — needs its OWN torch venv (NOT the mlx-lm
-#    oracle venv at /Users/joshrossi/Code/mlx-lm/.venv, which has no
-#    torch/deepspec and shouldn't); see the script header for the exact
-#    venv-setup + pip install steps.
-.venv-deepspec/bin/python scripts/oracle-dspark-deepspec.py \
-    --target <snapshot-dir-of-google/gemma-4-12B-it> \
-    --drafter <snapshot-of-dspark_gemma4_12b_block7> \
-    --data prompts.jsonl --n 8 --max-new-tokens 128 \
-    --temperature 0 --confidence-threshold 0 \
-    --out goldens/dspark-deepspec/trace-thr0.jsonl
-
-# b. compare our port round-for-round
-bun scripts/dspark-deepspec-compare.ts \
-    --fixture goldens/dspark-deepspec/trace-thr0.jsonl \
-    --target gemma-4-12B-it-bf16 --drafter <snapshot-of-dspark_gemma4_12b_block7>
-```
-
-Run the pair twice — once at `--confidence-threshold 0` (no pruning), once at
-`0.5` — to gate both the base draft loop and the confidence-truncation path.
-
-Fixture location: `goldens/dspark-deepspec/*.jsonl` follows the house
-manifest convention (goldens/README.md) — these are machine-independent
-**JSON manifests** (prompts + token-id traces, no tensor blobs), so they are
-the **tracked** kind, not the `.bin` machine-specific kind. Regenerate by
-rerunning step (a); there is no `scripts/regen-*` wrapper for this family yet
-since the oracle run needs its own GPU-hosting torch venv, not the mlx-lm
-oracle venv this repo otherwise standardizes on.
-
-**The OptiQ-4bit caveat is load-bearing, not a footnote**: the fixture above
-is generated against the bf16 HF target. A `dspark-deepspec-compare.ts` run
-with `--target` pointing at our bf16 12B is the TRUE bit-exact gate. A run
-with `--target` pointing at the OptiQ-4bit-quantized 12B we actually serve is
-a **separate acceptance-rate measurement** — tapped hiddens differ
-numerically at 4-bit, so round-for-round divergence there is expected, not a
-bug; pass `--acceptance-only` on that arm so the script reports an aggregate
-accept-length reading instead of a PASS/FAIL assert.
-
-Caveat: trained against the bf16 HF target; our serving 12B is OptiQ-4bit —
-tapped hiddens differ numerically, so measure acceptance rather than assuming
-their 60–85%.
-
-**FIRST LIVE RUN (2026-07-07, loaded machine — directional only):** τ ≈ 2.8
-committed/target-forward (200 tok / 72 verify rounds; per-token acceptance
-26–33% — the predicted OptiQ-4bit degradation), output coherent, losslessness
-holding. **But wall-clock LOST ~3.4×** (spec 14.6 vs serial 49.8 agg tok/s;
-the 20 s bench TTFT is conc-4 queueing on the serial-forced spec lane, not
-prefill). The tax is the drafter itself. **The full step-by-step program for
-the next pass is
-[docs/design/dspark-serving-program.md](../design/dspark-serving-program.md)**
-(phases, gates, dependency graph; pickup at TurboQuant merge, only its
-Phase 5 waits for it). Lever summary:
-1. **Quantize the drafter** — the big one, and the ideal quantization target
-   PERIOD: drafter numerics only move ACCEPTANCE, never correctness
-   (losslessness is structural), so aggressive bpw is safe and the gate is a
-   single acceptance A/B. 6.9 GB bf16 → ~1.8 GB @4-bit kills both the
-   ~3 GB/round bandwidth tax (2 GB lm_head + 7×134 MB markov_w2 reads) and
-   the 14 GB two-model memory pressure on 24 GB. Uniform 4-bit (mlx-native)
-   first as the baseline; **TurboQuant (Phase 13), when it lands, competes
-   on acceptance-per-byte against that baseline** — the drafter is its
-   lowest-risk first customer. Enabler either way: quantized-weight support
-   in deepspec-module's forward (quantized_matmul, the house pattern).
-2. **Tighten deepspec draftBlock** — ~14 host syncs/round (7 argmax + 7
-   confidence reads); replay the on-device chaining + deferred-conf
-   tightening already done for our own module (threshold=0 needs NO conf
-   reads).
-3. **captureLayer in the GENERATED gemma forwards** — tapped verifies
-   currently fall back to the monolith (correct-but-unspecialized); teach
-   scripts/gen-model.ts to emit the tap so spec verifies keep the fast path.
-4. **γ sweep** — γ=7 at ~30% acceptance wastes drafts;
-   `--num-draft-tokens 2..3` may net better wall-clock free of any code.
-
-## PATH B — train our own module (custom targets / research)
-
-The payoff target is **12B** (27B is memory-infeasible to train on 24 GB; the
-pipeline stays dim-generic so 27B remains a future recipe). All three scripts
-take `--model` and (new, 2026-07-06) `--tap-layers` — the retarget is now a
-pure parameterization, no code edits. **`--tap-layers` MUST be identical across
-regen and train** (the shard feature dim is `m*H`, `m = tapLayers.length`).
-
-12B tap set: use **`5,17,29,41,46`** — DeepSeek's actual trained
-`target_layer_ids` for this model (their layer-output convention matches our
-tapLayers indexing; audited 2026-07-06). Our earlier `24,37,47,48` guess is
-superseded.
-
-```sh
-# 0. one-time: cache the 12B target + a prompt/topic corpus (thousands of
-#    on-distribution topics — the data-scale lever; 160 only reached per-pos ~0.17).
-mlx-bun get mlx-community/gemma-4-12B-it-OptiQ-4bit
-# (12B target may need the serve process stopped to free memory for regen.)
-
-# 1. regen multi-layer training shards from the 12B target's own generations
-bun scripts/dspark-regen-dflash.ts \
-  --model gemma-4-12B-it-OptiQ-4bit --tap-layers 5,17,29,41,46 \
-  --topics <thousands-of-topics.txt> --out data/dspark-12b --max-resp 320
-
-# 2. train the drafter against the 12B (warm-start survivable via --resume;
-#    GPU runs kept getting killed ~step 5000 pre-resume). Optional A/B:
-#    --seq-head rnn trains the Eq-6 RNN head instead of Markov (Eq 5).
-bun scripts/dspark-train-dflash.ts \
-  --model gemma-4-12B-it-OptiQ-4bit --tap-layers 5,17,29,41,46 \
-  --data data/dspark-12b --out ckpt/dspark-12b \
-  --iters 8000 --batch 8 [--seq-head rnn] [--ddraft 2560] [--resume ckpt/dspark-12b]
-
-# 3. STS-calibrate (§3.2.1): fit per-position confidence thresholds on real
-#    verify outcomes and stamp them into dspark.json — this ACTIVATES the
-#    Alg-1 confidence scheduler at decode (uncalibrated = fixed-γ, no change).
-bun scripts/dspark-calibrate.ts \
-  --model gemma-4-12B-it-OptiQ-4bit \
-  --drafter ckpt/dspark-12b --data <prompts.jsonl> [--target 0.5]
-
-# 4. measure live τ + tok/s vs vanilla 12B decode (clean machine — dirty
-#    numbers are garbage). Run BOTH pre- and post-calibration for the
-#    scheduler's own A/B (MLX_BUN_DSPARK_MINCONF=<c> = manual uniform knob).
-bun scripts/dspark-measure-dflash.ts \
-  --model gemma-4-12B-it-OptiQ-4bit \
-  --drafter ckpt/dspark-12b --data <prompts.jsonl>
-
-# 5. serve it (auto-detected as kind=dspark via dspark.json; γ pinned to cfg.gamma)
-mlx-bun serve --model gemma-4-12B-it-OptiQ-4bit --draft-model ckpt/dspark-12b
-```
-
-Decision gate (write it down, RESULTS.md only if clean-machine): does 12B live
-τ net a wall-clock win? If the draft loop dominates, do the loop-tightening
-(per-position host syncs + double 262K LM-head) first — see the open items in
-the design doc. Memory guard (`server.ts`) caps context to fit; a client
-sending huge `max_tokens` trips admission — lower it.
-
-## What happened (short version)
-- Built v1 first as a single-vector shortcut — it dropped the paper's core KV
-  injection and was never really DSpark. Chunk (τ≈1.26) and article (τ≈1.16)
-  runs on it were misleading.
-- Rebuilt the **faithful** DFlash (multi-layer H_ctx injected into every draft
-  layer, 5 layers) in parallel `*-dflash.ts` files (v1 kept as baseline; repo
-  stays green).
-- Found + fixed an **off-by-one in the p^t target** (TV loss trained against a
-  distribution shifted one token — present in v1 too, so all prior τ were bogus).
-- **Overfit test settled it:** 3 articles, eval on same → per-pos ~0.75, τ=3.24.
-  The architecture reaches the paper's numbers; the plateau at ~0.17 on 160
-  articles is data-starvation, not capacity (confirmed: width 1024 vs 2560 both
-  plateau ~0.17).
-- **Target-speed is the other half.** Measured decode: e4b 45.9, 12b 27.5, 27B
-  ~15 tok/s. The fixed draft overhead only amortizes on a slow target — so e4b
-  is ~worst-case for spec decode; the **27B agentic workload is the real target**
-  (τ≈3 there could plausibly net ~2–3× → 15 → ~35–45 tok/s).
-
-## Verified / working
-- Parity-safe `hiddenTap` multi-layer extraction (`gemma4.ts`), no-op when off.
-- `trim(n, bypass=true)` physical-slice rollback past the sliding window (`gemma4-base.ts`) — lossless gate held on 6K context.
-- CPU smoke 16/16 (`dspark-dflash-smoke.ts`): KV-injection forward, autograd, inference, multi-layer variable-context data round-trip.
-- Training `--resume` (warm-start from checkpoint) — the GPU runs kept getting killed ~step 5000; resume survives that.
-
-## Next (in priority order)
-1. **Retarget to 12B + scale data + train + calibrate + measure** — the GPU
-   recipe above is the complete remaining work. (27B stays a future recipe:
-   memory-infeasible to train on the 24 GB M4 Pro.)
-2. ~~Tighten the draft inference loop~~ **DONE 2026-07-06** (on-device token
-   chaining, deferred conf reads, `collectLogits:false`; bit-identity pinned).
-3. ~~Finish the paper~~ **DONE 2026-07-06**: RNN head (Eq 6, `--seq-head rnn`),
-   STS calibration (§3.2.1), Alg-1 confidence-scheduled draft-length pruning
-   (single-user form). ⚠ Eq 6 + §3.2.1 shapes are design-doc-faithful readings
-   (paper PDF absent) — verify against the paper when available.
-4. ~~Rename `dflash`→`dspark`~~ **DONE 2026-07-06** (canonical variant +
-   central loader `src/spec/dspark/loader.ts`; legacy stamps still load).
-
-## Key files
-`src/spec/dspark/{module,data,generate}-dflash.ts`, `{loss,sample,calibration,loader}.ts`;
-`src/spec/{source,serve-loop,dflash-source,assistant-source}.ts` (the serve seam);
-`src/model/gemma4.ts` (hiddenTap), `gemma4-base.ts` (trim bypass);
-`scripts/dspark-{regen,train,measure}-dflash.ts`, `dspark-dflash-smoke.ts`.
-
-## Memory / gotchas
-- `dspark-drafter-built-v1`, `dont-inflate-scope-build-the-paper` (the lesson:
-  build the paper's mechanism first — the faithful build was ~1hr; the wrong
-  builds cost the session).
-- e4b hidden_size = **2560** (not 2816 — a different config block). Module reads dims live.
-- Memory admission guard (`server.ts:1538`) rejects `prompt+max_tokens > maxSafeContext`
-  because a mid-gen GPU OOM SIGKILLs the server. 27B/32GB caps ~22.5k; a client
-  sending `max_tokens=64k` trips it — lower max_tokens (or make it a runtime stop).

package/docs/archive/investigations/expert-offload-single-user-moe.md

@@ -1,292 +0,0 @@
-# Expert offload for single-user local MoE — findings + design sketch
-
-*2026-06-14. Research spike. Question: can Apple's AFM-3 "flash-resident
-experts" idea (Instruction-Following Pruning) be applied to the MoE models
-we already run, to stop the inactive experts from squatting in RAM? Sources:
-Apple AFM-3 blog + IFP (arXiv 2501.02086) + LLM-in-a-flash (2312.11514);
-MoE-offload prior art (Mixtral-offloading 2312.17238, Fiddler 2402.07033,
-KTransformers SOSP'25, llama.cpp `--n-cpu-moe`). Grounded against PLAN
-Phase 6 (MoE bring-up, measured), Phase 8 (LoRA hot-swap, shipped),
-Phase 14 (Qwen 35B-A3B fit-table line), Phase 18 (slots).*
-
----
-
-## 0. TL;DR
-
-A trained MoE already activates a tiny fraction of its weights per token, but
-we currently hold **all** experts resident — measured at 14.09 GB of expert
-pool to use ~0.9 GB/token on `gemma-4-26B-A4B` (PLAN Phase 6). Those cold
-experts are read-only and rarely touched; the KV cache is hot, dirty, and
-un-offloadable. The hardware is telling us which one goes to disk.
-
-The catch with Apple's approach — it needs **task locality** (you can't swap
-experts per token across the flash bus, only per prompt/period) — is the exact
-property **single-user local AI gives us for free**: one human works on one
-kind of thing for minutes-to-hours at a time. A multi-tenant server can't
-promise that; a personal appliance can. So this is a feature that fits *our*
-product identity specifically.
-
-Two payoffs from one residency manager:
-1. **Shrink a model that already fits**: 26B-A4B from ~16.4 GB resident →
-   ~4–6 GB, returning ~10 GB to KV-context-length and to the user's other
-   apps. (Today the 26B leaves ~0 GB for anything else — see §2.)
-2. **Admit a model that doesn't fit**: e.g. a 35B-A3B class model on 24 GB,
-   flipping PLAN Phase 14's "larger hardware only" line into "fits, with a
-   domain-switch warm-up cost."
-
-**What's already done (de-risks this a lot):** the MoE forward is bit-exact
-today (Phase 6, `gather_qmm`, `tests/moe-ops.test.ts`). The missing piece is
-*only* the memory-tiering layer — not a new architecture.
-
----
-
-## 0.5 Empirical findings — the mechanism is de-risked (2026-06-14)
-
-Two probes on the M4 Pro (`scripts/probe-expert-residency.ts`,
-`scripts/probe-mmap-gather.ts`) turned the design from a sketch into a verified
-mechanism. Three results:
-
-1. **Disposing MLX *device* buffers does NOT reliably return RAM to the OS.**
-   In the controlled case, allocating a 512 MB array and then calling
-   `dispose()` *and* `clearCache()` moved `rss` not one byte; `cache_memory`
-   read 0 throughout. So a **fixed device-buffer slot pool is the wrong
-   mechanism** — it can hold *less*, but it can't give RAM *back* when idle,
-   which is the whole point.
-2. **The mmap clean-page path returns RAM to the OS deterministically.**
-   `munmap` of a mapped region dropped `rss` by exactly the mapped size
-   (−1 GB, and −2 GB at scale). This is the elastic-eviction substrate, and
-   it's exactly what expert weights are: read-only files.
-3. **The GPU gathers straight from a page-aligned mmap, bit-exact.** GPU
-   `gather_qmm` reading a quantized expert directly from a 16 KB-page-aligned
-   mmap (at a realistic non-zero offset) was **bit-exact** vs the same weights
-   resident (`max|diff| = 0`, no NaN). The Phase-1 "GPU reads garbage from
-   unaligned wrapped pointers" hazard is avoided purely by page alignment — so
-   we need **no copy into (sticky) device buffers** at all.
-
-**Revised design (supersedes the "materialize into resident MLX buffers / slot
-pool" framing in §5):** bit-exact transparent offload where expert weights live
-as **page-aligned, mmap-backed clean file pages** — a miss *maps* the region
-(GPU gathers from it directly), eviction *unmaps* it (`munmap`/
-`madvise(DONTNEED)`). Idle footprint collapses toward the core; a cold expert
-is just an unmapped clean region re-faulted from SSD when routing asks. The one
-requirement this forces: a one-time conversion to an **offload-ready expert
-file** with each expert's weight/scales/biases starting on a page boundary
-(safetensors packs them contiguously; the registry already scans the per-expert
-byte offsets).
-
----
-
-## 1. What Apple actually does — and what's portable
-
-Apple's AFM-3 "Core Advanced" is **two separable things**, and only one is
-bolt-on:
-
-- **Portable (inference-time runtime):** store the model in flash, stream only
-  the needed experts into DRAM, manage a resident working set. This is a
-  memory-management strategy — adoptable.
-- **NOT portable (trained architecture):** the *per-prompt routing* that makes
-  it work is **Instruction-Following Pruning** — a predictor co-trained
-  end-to-end with the base weights so the model is *prunable by prompt*. The
-  base weights are updated during training. You cannot bolt IFP onto a
-  finished model, and trained-MoE experts are **not** human-interpretable by
-  task ("expert 7 = Python" is a myth). LLM-in-a-flash additionally needs a
-  ReLU-sparse FFN (a relufication fine-tune for SiLU/GeGLU models).
-
-**Why per-prompt, not per-token:** flash→DRAM is ~1 GB/s on a phone vs ~100
-GB/s DRAM — a standard token-routed MoE would thrash the bus, so Apple decides
-the expert set once per prompt and freezes it. We don't need to copy the
-*mechanism* (we have unified memory + a fast SSD, not a 1 GB/s phone bus); we
-need the *insight* — amortize expert movement over a span where the working
-set is stable.
-
----
-
-## 2. The memory problem, measured
-
-On the M4 Pro (24 GB), a real desktop commits memory before the model loads:
-macOS baseline ~4–6 GB, plus a working set of Chrome + IDE + Slack + Claude
-Desktop + Mail + WhatsApp + Calendar ≈ 6–10 GB. That is ~10–16 GB gone before
-inference.
-
-`gemma-4-26B-A4B-OptiQ-4bit`, measured (PLAN Phase 6):
-- **16.4 GB resident**, of which **14.09 GB is the expert pool**.
-- **~0.9 GB of experts read per token** (top_k/num_experts of the pool). The
-  other ~13 GB is resident-but-idle on any given token — the squat.
-- KV is *not* the hog for this model: 5/30 global layers, 2 global KV heads,
-  Gemma sliding window ⇒ max safe context ~17.6k at bf16 KV, gated by the
-  weight footprint, not the cache. (KV bites harder on longer contexts or
-  models with fatter global attention; the tiering principle below still
-  holds, but for *this* model the experts are unambiguously the target.)
-
-So today the 26B leaves ~0 GB for apps and limits context to ~17.6k — and that
-is the *good* case (a model that fits at all).
-
-**The tiering principle.** Rank everything that wants RAM by access-frequency ×
-mutability:
-
-| What | Touched | Mutable | Belongs |
-|---|---|---|---|
-| KV cache | every token | read **+ write**, per-session dirty | resident — cannot offload (only quantize) |
-| Attention / embeddings / shared experts | every token | read-only | resident |
-| Active routed experts | ~top_k/N per token | read-only | resident working set + LRU |
-| Cold routed experts | rarely, this token | read-only, **clean file pages** | **SSD** — the natural victim |
-
-The current layout is backwards: it spends the fast tier on the coldest data.
-Cold experts are read-only and clean (cheap to evict / re-read); KV is dirty
-and hot (must stay). Two orthogonal levers compose along this split — quantized
-KV shrinks the hot tier (Phase 9, shipped), expert offload shrinks the cold
-tier (this note).
-
----
-
-## 3. The personal-AI insight: task locality is the unlock
-
-Per-prompt/period expert selection only avoids thrash if the working set is
-stable across that span. That is false for a topic-hopping chat and false for a
-multi-tenant server — but **true for one human doing one job**: two hours of
-coding, then a session reorganizing a 5,000-doc wiki. Within a task the hot
-expert set is stable; you pay a re-warm only at task boundaries.
-
-This is structurally the **same shape as the LoRA adapter system we already
-ship** (Phase 8): one resident base model, the calling app declares intent
-("use the coding fine-tune") via `/v1/adapters` + `adapterScoped()` in
-`src/lora.ts` / `src/generate.ts`. Expert offload reuses that surface: the app
-declares a *domain*, and the residency manager warms the experts that domain
-has historically hit. The difference from LoRA: a hint, not a hard mount —
-misses fall through to SSD (see §4), so correctness never depends on the hint.
-
-**Scope boundary (important):** this is a single-user / single-active-task
-feature. Under Phase 18 concurrent slots with diverse requests the locality
-guarantee disappears — in that mode experts should stay resident (or accept
-thrash). Don't combine the two naively; keep offload files separate from the
-batch/slots work, per the slots discipline.
-
----
-
-## 4. The correctness fork
-
-Two designs, very different parity implications:
-
-- **A — Transparent offload (BIT-EXACT).** Cold experts on SSD; materialize
-  whichever experts routing picks into a resident buffer before `gather_qmm`;
-  hot LRU cache. Output is identical to today's all-resident path — same
-  `gather_qmm`, same tokens — only latency varies on a miss. Stays L1/L2. This
-  is the foundation.
-- **B — Task-pinned routing (LOSSY).** Restrict routing to a pinned set and
-  skip cold experts → zero misses, lower latency, but the computation changed →
-  L3, KL-gated, default-off. This is the closest off-the-shelf analog to IFP's
-  select-and-freeze, and it's quality-risky on weights trained for per-token
-  routing.
-
-**Recommendation:** ship A as the always-correct floor; layer the §3 domain
-hint as a *prefetch* on top of A (warms the likely set, still bit-exact because
-misses fall to SSD). Offer B only as an opt-in mode for users who'll trade a
-little quality to eliminate switch latency — and keep it as a documented
-default-off flag, never the only path (per the "don't delete optionality"
-principle).
-
----
-
-## 5. Proposed architecture
-
-Layered, each layer independently useful and independently gated:
-
-1. **Offload-ready expert file (page-aligned).** A one-time conversion stores
-   each expert's packed weight/scales/biases starting on a 16 KB page boundary
-   (safetensors packs them contiguously; the registry already scans the per-
-   expert byte offsets). This is what lets the GPU gather straight from the
-   mmap — confirmed bit-exact in §0.5.
-2. **Residency manager (mmap-backed).** Per (layer, expert) state: `unmapped`
-   (clean on SSD) / `mapped`. A miss maps the expert's page-aligned region
-   (`MlxArray.fromView` over the mmap) and the GPU gathers from it directly —
-   **no copy into device buffers** (§0.5 showed those don't reliably return to
-   the OS). Eviction = `munmap`/`madvise(DONTNEED)`, which returns RAM to the
-   OS deterministically. Attention + embeddings + shared experts stay pinned;
-   LRU over the mapped expert set under a byte budget. Default = everything
-   mapped (inert, like the slots flag), opt-in via `--expert-offload`.
-3. **Domain prefetch (the LoRA-shaped surface).** Per-session/request `domain`
-   hint → prefetch that domain's profiled hot-set. Profiles are per-user,
-   persisted, updated online from observed routing (ties into the memory
-   flywheel). No hint ⇒ pure adaptive cache (still works, just pays re-warm at
-   task boundaries).
-4. **(Optional) Pinned mode (B).** Hard-restrict routing to the warm set; L3
-   gated; default-off.
-
-Correctness is preserved at layers 1–3 by construction: we always compute with
-the experts routing actually selected; offload changes *where weights live and
-when they arrive*, not *what is computed*.
-
----
-
-## 6. The two payoffs, quantified where we can
-
-- **Shrink the 26B (already fits):** resident floor after offload ≈ non-expert
-  (16.4 − 14.09 ≈ 2.3 GB) + active working set + a hot cache. With a ~2–3 GB
-  hot cache that's ~5–6 GB resident vs 16.4 GB today → ~10 GB returned to KV
-  context length and to the user's apps. Decode cost is the unknown to measure
-  (§7).
-- **Admit a 35B-A3B (doesn't fit):** characterize as a fit-table row today
-  (Phase 14), then show it runs usable with offload; the domain-switch warm-up
-  is the price. This *changes* the Phase 14 line.
-
-All tok/s figures here are **unmeasured** until §7 runs on a cleared machine —
-treat as hypotheses, not promises.
-
----
-
-## 7. Tradeoffs and the load-bearing unknown
-
-- **The #1 thing to measure: per-task expert skew.** Does a coding session
-  actually concentrate its routing on a stable subset, and how big is it? If
-  the hot ~50% of experts covers ≥90% of activations and is stable within a
-  task, the whole idea works; if routing is near-uniform per task, it doesn't.
-  This is empirical and unknown — measure before building anything else.
-
-  **E0 result (scripted, 2026-06-14 — `scripts/run-expert-trace.ts`, 8 prompts
-  × coding/writing/chat on the 26B): gate PASSES all three.** 90% of
-  activations land on ~40–47% of experts (51/53/60 of 128; uniform ≈ 90%) —
-  concentrated but moderate. Within-task stability (hot-set Jaccard over 4
-  windows) 0.63–0.70. Unique experts touched over ~1.2k tokens: 81–85% of the
-  3840 instances → the *touched* working set is large (~12 GB) even though
-  *activations* concentrate. Cross-task: coding vs prose 0.42–0.44
-  (specialised — domain prefetch pays), writing vs chat 0.68 (similar). Net:
-  bit-exact offload realistically frees ~6–7 GB (resident ~9–10 GB vs 16.4),
-  keeping the 90% hot set resident and re-faulting the rare tail — the exact
-  footprint-vs-miss-rate curve is E1's measurement, not an E0 promise. Caveat:
-  scripted short sessions; a real long focused session may tighten or broaden
-  the curve — worth a confirming real-session trace before locking a budget.
-- **Switch cost.** = (bytes that must become newly resident) / SSD bandwidth
-  (~5–7 GB/s) + dequant + buffer setup. Pure I/O floor for a few-GB working set
-  is single-digit seconds; the conservative "30–60 s" budget leaves room for
-  dequant/repack and a near-total set change. Measure the real number; it
-  depends on how much the hot set overlaps between domains.
-- **Miss latency on the bit-exact path.** A cold expert miss is a synchronous
-  SSD read on the decode critical path; bounded by ~7 GB/s × miss-rate. Hot
-  cache + domain prefetch are what keep miss-rate low.
-- **Not for multi-tenant serving** (§3 scope boundary).
-
----
-
-## 8. Phased plan (proposed — for promotion into PLAN.md)
-
-- **E0 — Measurement spike (no runtime risk).** Instrument the existing MoE
-  forward to log per-(layer,expert) selection over real coding / writing / chat
-  sessions. Produce the coverage curve (% experts covering 90/95/99% of
-  activations), within-task stability, and cross-task set shift. **Gate to
-  proceed:** hot-set coverage + stability good enough that offload pays. This
-  is the make-or-break, and it's pure observation.
-- **E1 — Transparent offload (bit-exact).** Expert store + residency manager +
-  on-demand materialization + hot LRU, behind `--expert-offload` (default off /
-  inert). **Parity gate:** bit-exact vs current all-resident (L1/L2 unchanged).
-  Measure resident footprint + decode tok/s vs miss-rate on a cleared machine.
-- **E2 — Domain prefetch (LoRA-shaped).** Per-session `domain` hint reusing the
-  adapter surface; per-user profile, learned online. Still bit-exact. Measure
-  cold-start vs warm latency and task-boundary switch cost.
-- **E3 — Admit a non-fitting model.** Bring a 35B-A3B-class model up under
-  offload; show usable serving on 24 GB; flip the Phase 14 fit-table line.
-- **E4 — (Optional) Pinned mode.** L3 KL-gated, default-off flag; only where the
-  quality delta is acceptable.
-
-**Exit criterion (overall):** 26B-A4B served at bit-exact parity with resident
-footprint cut to a measured target, leaving a usable machine, with a quantified
-domain-switch cost — promoted into `benchmarks/RESULTS.md`.