mlx-bun 0.0.9 → 0.0.10

package/docs/investigations/kernel-perf-review-2026-07.md

@@ -12,7 +12,7 @@ area reviews + full mechanisms): `reports/kernel-perf-review-2026-07-01.json`
   vendored MLX steel machinery, exact-by-default backward with autograd + FD
   parity — marred by decision drift: the PLAN-decided 1.35× filter win was
   never landed and the in-code comments argued three contradictory positions
-  (comments fixed 2026-07-01; the FLIP itself is still backlog #1).
+  (comments fixed 2026-07-01; the FLIP LANDED 2026-07-02 — see backlog #1).
 - **Training flash-attention kernel**: faithful port with exemplary post-bug
   discipline, correctly opt-in — but it copies a naive oracle (~30× slower)
   and its tests covered none of the production configs (extended 2026-07-01:
@@ -47,35 +47,109 @@ evalSftLoss error swallowing · tokens_per_sec metric · inverted comments
 (gemma4.ts:298, flash-cce filter trio) · silent steel fallback (now warns;
 H%128 throw narrowed) · flash-attn test blindness.
 
-STILL OPEN: **FUSED_DECODE=1 inside whole-graph compiled decode** bakes the
-tile loop at trace-time N — q-cat adapters grow N per step under shapeless
-replay, so the newest KV rows are silently never attended (gemma4-base.ts
-quantizedSdpaTiled; deferred for file-ownership at fix time — guard the combo
-or make the tile loop trace-safe).
+FIXED 2026-07-02: **FUSED_DECODE=1 inside whole-graph compiled decode** baked
+the tile loop at trace-time N — q-cat adapters grow N per step under shapeless
+replay, so the newest KV rows were silently never attended. REPRODUCED on e4b
+(42 layers traced tile=true at N=17; diverges within 3 tokens then
+repetition-loops). The 12B was never affected — its segmented form runs concat
+layers as uncompiled js layers. Fix: generate() refuses to compile under
+MLX_BUN_FUSED_DECODE=1 (explicit opt-in wins, like LoRA/MoE) + quantizedSdpa
+throws if the tiled path is ever selected inside a trace (lands in the
+transactional fallback, never silent). Regression in
+tests/compiled-decode.test.ts, whose e4b block also now uses the dynamic
+snapshot resolver — the old hard-coded hash silently skipped it on this
+machine, which is how the combo went unexercised.
+
+NEW BUG (found 2026-07-02 by the backlog-#8 gate test; FIXED in mlx-bun,
+UPSTREAM in mlx): **mx.quantized_matmul(transpose=false) returns wrong values
+at row count 2 or 3** — relnorm ~0.33-0.39 on our dylib, ~0.98-1.14 on stock
+mlx 0.31.2 (oracle venv repro); M=1 and M>=4 exact, transpose=true exact at
+every M. mlx's autograd computes dh for a transpose=true head qmm via an
+internal transpose=false call at the cotangent's row count, so EVERY training
+head silently produced wrong dh for 2-3-token responses, 2-3-token tail
+chunks, and tiny prompt spans (f32 ground truth: dh relnorm up to 0.99 at the
+affected positions, cos ~0.2). Serving/decode unaffected (transpose=true).
+Fix: (a) loss.ts `logitsFromHiddenPadM` pads differentiated head spans to 4
+rows (pad cotangents are exactly zero) — applied at all seven head sites;
+(b) ops.quantizedMatmul pads direct transpose=false calls (the fused head's
+analytic backward). Verified vs f32-dequant ground truth (chunk-4 dh
+0.47→0.006) + M-sweeps. Worth filing upstream with the 10-line python repro.
 
 ## The ranked optimization backlog (task #12)
 
-1. **[S] Land the coeff filter at eps~1e-5** (+ measure blockMax skip on real
-   data) — 1.35× ORPO head backward, ALREADY MEASURED
-   (flash-cce-filter-realdata.ts, PLAN.md "filter-on-real-data"); gates:
-   E4B=1 rerun + teacher-forced grad fidelity + eps=0 byte-identical.
-2. **[M] planSegments full-attention isolation** — e4b@8K ~17.5→10 GB;
-   unblocks 8K training on the 24 GB M4 Pro.
-3. **[S] Auto-dispatch the training head by M** — exact fused QM head for
-   short-M (~1.9×: 481 ms vs 934 ms at e4b M=512), flash only when memory
-   demands; both heads already share the fusedRespLogpMean interface.
-4. **[S] Fused decode full-capacity KV buffers + activeN** — kills 6
-   ensureRowContiguous copies/layer/step; 1-3% @8k, ~4× that at 32k.
+1. ~~**[S] Land the coeff filter at eps~1e-5**~~ **LANDED 2026-07-02** — BOTH
+   skips default 1e-5 (coeff filter + blockMax block skip; env=0 restores exact).
+   All gates run on the M1 Max with real chunk-ORPO data
+   (flash-cce-filter-realdata.ts, extended: E4B=1 + blockEps sweep + Vjp
+   full-logits fidelity): filter CPM5 0.343%/1.41×, e4b 0.158%/1.70×; blockMax
+   skip ~lossless (≤0.004%) and real text DOES go cold (the synthetic probe was
+   the artifact) — **combined 1.71× (CPM5) / 3.16× (e4b)**; teacher-forced
+   fidelity cosine ≥0.99993, filter adds ≤0.07% over the eps=0 reassociation
+   floor (standing test in tests/train-orpo-fused-ce.test.ts). The "eps=0
+   byte-identical" gate resolved structurally: eps=0 compiles the identical
+   pre-flip kernel, but run-to-run dh is never byte-stable (atomic-add
+   reassociation) — byte-replay was the wrong spelling of that gate.
+2. ~~**[M] planSegments full-attention isolation**~~ **REFUTED BY MEASUREMENT
+   2026-07-02** — built, A/B'd @8K on real chunk rows: ZERO peak win (18.09 vs
+   18.02 GB; scripts/experiments/seg-isolation-smoke.ts + MLX_BUN_SEG_MEM_LOG
+   phase probes). The §5 cost-model premise is false: mlx's sdpa BACKWARD
+   materializes O(L²) scores for EVERY layer (~3.5 GB/layer @8K e4b — the
+   sliding window is just an additive mask; sliding pair +7.1 GB ≈ full+sliding
+   +7.15). Worst segment = layer count alone, so **segment_size is the whole
+   knob: seg1 measures 14.59 GB @8K (+3% step time, loss identical) — which
+   DOES fit the 24 GB M4 Pro today, no code needed.** Inter-segment clearCache
+   also measured no-effect (watermark is live memory). Real next levers: the
+   head vjp's ~3 GB [M,V] spike (backlog #8) and an O(L)-memory attention
+   backward (the flash-attention track).
+3. ~~**[S] Auto-dispatch the training head by M**~~ **LANDED 2026-07-02** —
+   fusedRespLogpMean routes requested-flash batches with M <
+   MLX_BUN_FLASH_MIN_M (default 1024) to the exact fused head. Fresh crossover
+   sweep (head-dispatch-sweep.ts, M1 Max — the review's 1.9× @M=512 was
+   pre-filter-flip): fused wins e4b M=64 1.6× / M=512 1.13×, tie @1024; the
+   #1-flip made flash FASTER at long M too (e4b ≥2048; 8K 10.7 vs 13.2 s) on
+   top of its memory win. CPM5: fused faster at every M — flash stays a pure
+   memory lever there (≥1024 honors the caller's intent). Dispatch proven by
+   bit-identical-to-fused loss in tests/train-orpo-fused-ce.test.ts.
+4. ~~**[S] Fused decode full-capacity KV buffers + activeN**~~ **REFUTED BY
+   MEASUREMENT 2026-07-02** — the variant was built (an int32 activeN input,
+   capacity-strided rows, fetch attaches full buffers), gated BYTE-IDENTICAL
+   on all three recorded dispatch shapes and perf-oracle-green — and then
+   measured SLOWER end-to-end: interleaved in-process A/B on the 12B uniform
+   kv4 (tokens identical across arms, M1 Max) gave 24.64 vs 26.18 tok/s @8k
+   (0.94) and 23.07 vs 23.83 @22k (0.97) — the "~4× at 32k" scaling is
+   refuted too. Two holes in the estimate: the 12B's full-attention dispatch
+   shape is KV=1 (H16 D512), where the fetch view is a CONTIGUOUS PREFIX and
+   the six-copies model never applied; and kernel-only micro-A/Bs proved
+   unstable (±50% run-to-run) — the numbers behind "1-3%" were never
+   end-to-end. REVERTED; evidence + re-application recipe:
+   scripts/experiments/fused-decode-activen-ab.ts.
 5. **[M] Collapse per-token host syncs in the spec/DSpark loop** — ~2γ+1
    round-trips → ~2 per verify cycle; directly attacks DSpark's fixed draft
    overhead.
 6. **[M] Pipeline the batched decode loop + clearCache cadence** — folded into
    the batching-v2 wave (docs/design/batching-v2-plan.md).
 7. **[S] Enable the 26B perf kernel** (open PLAN Phase E box; oracle frozen).
-8. **[S] Bound the SFT segmented head** — replace full-[M,V] responseOnlyCe in
-   the segmented head vjp; ~2-4 GB peak on long-response e4b SFT.
-9. **[S] Segmented-step overhead pass** — detachLeaf single-copy, one evalAll
-   barrier/segment, memoize the CPM5 prefix block-sparse mask.
+8. ~~**[S] Bound the SFT segmented head**~~ **LANDED 2026-07-02** — both SFT
+   segmented classes now use boundedSftCe (token-chunked Checkpoint head, the
+   proven ORPO plumbing; MLX_BUN_SEG_HEAD/MLX_BUN_SEG_HEAD_CHUNK knobs).
+   Head-only A/B at e4b M=6000: whole-vocab head 16.60 GB peak vs bounded
+   6.60 GB (**~10 GB head-vjp transient eliminated**), loss identical, dh
+   relnorm 0.00000; head time +45% (Checkpoint recompute, small share of a
+   step). The @8K smoke's short responses were never head-bound (peak set by
+   the per-layer sdpa backward — see #2), so its wall peak moved little; the
+   win is exactly the long-response regime the review named. Gate test:
+   tests/train-orpo-fused-ce.test.ts "bounded SFT head".
+   **Landing this exposed a NEW upstream mlx bug (see ledger below).**
+9. ~~**[S] Segmented-step overhead pass**~~ **LANDED 2026-07-02** — all three
+   fixes: MlxArray.detachCopy (ONE mlx→mlx copy; the old rawBytes+
+   fromBytesCopy route was two copies via the JS heap), detachLeaves (ONE
+   evalAll barrier per segment's whole vjp set, applied in all six segmented
+   backward loops), and a step-shared block-sparse-mask memo for the CPM5
+   prefix classes (prefixSharedCaches — makeMask hands out slice views).
+   Gate: loss + grads BYTE-IDENTICAL on CPM5-SFT / CPM5-ORPO-prefix / e4b-SFT
+   fixed steps. Step time (short-seq fixtures, M1 Max): CPM5 ORPO-prefix
+   238→158 ms (−34%), e4b SFT 2065→1272 ms (−38%); e4b @8K flat (GPU-bound,
+   as predicted) with peak unchanged (14.49 GB).
 10. **[M] e4b prefill gap** — profile-first; NOTE: the decode look-again could
     NOT reproduce RESULTS.md's 304-vs-373 row on the M1 Max (no eval-DB
     backing) — re-bench on the M4 Pro before building anything.
@@ -85,8 +159,13 @@ or make the tile loop trace-safe).
 The review's framing assumed the megakernel post-mortem's "M=1 decode is at
 the floor". The follow-up investigation (docs/investigations/
 decode-roofline-lookagain.md) OVERTURNED that: only the 12B is at the wall
-(~92-93% of measured roofline); CPM5 58-64%, e4b 64-70%, 26B 60-62%, with the
-host JS graph build as the #1 recoverable term. Its ranked fixes (CPM5
-compiled-decode port; build-ahead graph overlap spike; 26B gather-qmm
-bandwidth; e4b dispatch batching) COMPOSE with the backlog above and should be
-sequenced first where they overlap.
+(~92-93% of measured roofline); CPM5 58-64%, e4b 64-70%, 26B 60-62%.
+**2026-07-02 second correction (lookagain §7):** the "host JS graph build =
+#1 recoverable term" part of that verdict was itself wrong — the overlap
+spike (decode-overlap-probe.ts, spin-injection + serial anchor) proved the
+pipelined loop ALREADY hides the build; wall = GPU step time. The lookagain's
+host-side fixes (1a–1d: CPM5 compiled-decode port, build-ahead spike, SharedKv
+segmented-compile, 12B concat-phase compile) are refuted or de-prioritized;
+its GPU-side fixes (26B gather-qmm bandwidth, e4b dispatch batching, CPM5
+KV-path, backlog #4 contiguity, spec decode) are now the ENTIRE recoverable
+gap and still compose with the backlog above.

package/docs/planning/release-notes-v0.0.10.md

@@ -0,0 +1,57 @@
+# mlx-bun v0.0.10 — batching parity, SSD KV cache, sharper serving
+
+## Highlights
+
+**Concurrent serving now matches the fastest MLX batching servers.** With
+`--batch 4`, aggregate throughput at 4 parallel requests is at parity with
+(or ahead of) oMLX on every shared model we benchmarked on an M1 Max —
+MiniCPM5-1B 349 vs 339 tok/s, gemma-4-e4b within 3%, Qwen3.5-4B within 1% —
+with 2–3× better time-to-first-token under load, and per-token SSE
+streaming (no burst batching of your tokens).
+
+- **Qwen3.5 joins the batch lane.** Hybrid gated-DeltaNet (SSM) caches grew
+  the dynamic-batching ops; the batched decode path is verified
+  token-for-token identical against mlx-lm's B=2 batched oracle.
+- **Sampler features batch too.** Repetition/presence/frequency penalties,
+  `logit_bias`, `min_p`, and XTC now run per-row inside the batch instead
+  of forcing requests onto the serial path. (Qwen3.5 ships a default
+  repetition penalty — before this fix it silently serialized every
+  request under `--batch N`.)
+
+**SSD cold tier for the prompt cache** (`--ssd-cache <dir>`). Long-context
+prefix KV now spills to disk on eviction, is snapshotted after requests
+settle, and **survives server restarts**: re-attaching a 13.7k-token
+coding-agent conversation went from a 12.1 s full re-prefill to a 0.24 s
+zero-copy mmap restore in our measurements — with 0% steady-state decode
+overhead (nothing runs on the token loop). Entries are keyed by model
+fingerprint + KV scheme + tokenizer hash + adapter namespace; corrupt or
+incompatible files self-quarantine; the tier is always safe to delete.
+Companion flags: `--ssd-cache-max <GB>` (default 32), `--ssd-cache-verify`.
+
+## Fixes
+
+- `serve`/`benchmark` now honor `--model <path-or-query>` as an explicit
+  override (it was previously accepted but ignored — auto-pick could
+  silently load a different model). A directory containing `config.json`
+  loads directly from that path.
+- The server stays responsive during long serial generations: `/stats`,
+  `/health`, and new connections previously stalled until the generation
+  finished (measured 2.5 s on a 512-token run); they now answer in tens of
+  milliseconds, with no measurable decode cost.
+
+## Internals
+
+- KV persistence format v2: all cache kinds round-trip (bf16, per-layer
+  quantized, sliding-window, quantized sliding-window, SSM), with
+  integrity hashes, invalidation metadata, and atomic writes.
+- Golden `.bin` parity fixtures are no longer tracked in git (they are
+  machine-specific and regenerable); JSON manifests remain.
+- New design docs: `docs/design/omlx-adoption-map.md` (feature adoption
+  scoreboard), `docs/design/batching-perf-path.md`,
+  `docs/design/ssd-kv-cold-tier.md`.
+
+Full details and measured numbers: STATUS.md and the design docs above.
+Credit where due: the SSD-tier and batching comparisons were driven by
+studying [oMLX](https://github.com/jundot/omlx) (Apache 2.0) — a strong
+project whose ideas we benchmarked and adapted where they made mlx-bun
+faster.

package/docs/reference/server-config.md

@@ -23,6 +23,9 @@ the design rationale is in
 | `--port` | n | `8080` | Listen port. A pre-flight probe refuses to start if the port is already serving. |
 | `--memory-budget` | GB | machine RAM × 0.75 | Admission ceiling. Requests whose `prompt + max_tokens` exceed the budget's max safe context are **rejected with 400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. Also caps the mlx allocator (`mlx_set_memory_limit`) as defense in depth. **Decimal GB (×10⁹).** |
 | `--prompt-cache` | GB | `2` (≈2e9 bytes) | Byte cap for the prompt (prefix-reuse KV) cache. **Binary GiB (×2³⁰)** on input. Unset ⇒ the 2 GB default; an explicit `--prompt-cache 0` **disables** the cache (maxBytes 0 evicts every entry). |
+| `--ssd-cache` | dir | off | SSD cold tier under the prompt cache ([docs/design/ssd-kv-cold-tier.md](../design/ssd-kv-cold-tier.md)): prefix KV spills to disk on RAM eviction, is snapshotted after requests settle (debounced 1 s), and **survives restarts** — a long-context agent re-attach restores via zero-copy mmap instead of re-prefilling (measured 13.7k-token prefix: 12 s → 0.24 s TTFT; 0% decode overhead). Entries are keyed by model fingerprint **+ effective kv scheme** + tokenizer hash + adapter ns; incompatible/corrupt files self-quarantine. Serial lane only. Requires the RAM cache. |
+| `--ssd-cache-max` | GB | `32` | SSD tier byte cap (binary GiB); oldest-mtime entries evicted at the cap. |
+| `--ssd-cache-verify` | (bool) | off | Verify every tensor hash on restore. Reads all bytes eagerly (defeats lazy page fault-in) — integrity paranoia only; the header hash is always verified. |
 | `--batch` | n | `1` (serial) | Max concurrent requests batched through the mlx-lm-parity engine. `>1` switches the **whole server** into bf16 continuous batching — a *mode*, not a load fallback. See [Execution modes](#execution-modes-serial-vs---batch-n). `--decode-concurrency` is accepted for drop-in compatibility, but the semantics differ: in mlx_lm.server it caps per-BatchGenerator decode parallelism (default 32); in mlx-bun it enables continuous batching with this cap (mlx-bun's default is the optimized serial path). |
 | `--kv-quant` | `config`\|`off`\|`4`\|`8` | `config` serial / `off`(bf16) under `--batch N` | KV-cache quantization. `config` = per-layer `kv_config.json` (optiq parity); `off` = bf16; `4`/`8` = uniform bits (group 64, start 0). Under `--batch N`, an explicit value routes those requests to the **serial** lane (batched is bf16-only). |
 | `--adapter` | dir | none | Mount a LoRA adapter at startup (same machinery as `POST /v1/adapters`; the adapter id is the directory's basename) and make it the **default** for requests that send no `adapter` field. A request's explicit `adapter` — including `"none"` — always wins, and hot-swap via `/v1/adapters` is unchanged. `--adapter-path` is accepted as the mlx_lm.server-named alias. A bad adapter fails startup loudly rather than silently serving the base model. This is the flag `mlx-bun train`'s completion message points at. |

package/docs/reference/training.md

@@ -132,6 +132,11 @@ piece independently toggled and each falling back cleanly:
 - **`orpo_flash_ce`** — the [M,vocab]-free flash-CCE Metal head (steel GEMM fwd
   **and** bwd; e4b backward 754 ms, peak 0.93 GB flat at M=8192). Implies the
   fused head. Falls back to the MLX fused head when the tiling isn't clean.
+  **Auto-dispatch by M** (2026-07-02): batches with M below `MLX_BUN_FLASH_MIN_M`
+  (default 1024) take the exact MLX fused head instead — it is faster at short
+  M (e4b M=64 1.6×, M=512 1.13×; CPM5 at every M), while flash wins memory
+  everywhere and time at long M on e4b (M≥2048; 8K: 10.7 vs 13.2 s with the
+  filter/blockMax defaults). `MLX_BUN_FLASH_MIN_M=0` always honors flash.
 - **`segment_size`** — segmented backward (gradient-checkpointed layer
   activations) for long context; the head is ~free in memory, the activations
   are the wall.
@@ -184,7 +189,7 @@ are `DEFAULT_TRAIN_CONFIG` (trainer.ts:89).
 | `orpo_lr_schedule` | `constant` \| `cosine` | `cosine` | ORPO LR schedule (orpo only) |
 | `orpo_chunk_size` | int | `0` (off) | Token-chunk the response head, bounding the `[M,vocab]` logits to `[chunk,vocab]` (defaults to 512 when a fused head is on). Exact (orpo only) |
 | `orpo_fused_ce` | bool | `false` | Fused linear-CE head: one CustomVjp with an analytic softmax−onehot backward (MLX `quantizedMatmul` both ways) — no autograd through the head, no retained `[M,vocab]`. Exact; `[chunk,vocab]` transient (orpo only) |
-| `orpo_flash_ce` | bool | `false` | Route the fused head through the **flash-CCE Metal kernel** (verbatim MLX steel GEMM + ORPO epilogue, fwd **and** bwd): neither `[M,vocab]` nor a dequantized head touches HBM → `[M,vocab]`-free, fastest on large vocab (e4b bwd 754 ms, 0.93 GB flat @ M=8192). Implies the fused head. The Apple-CCE coeff filter / blockMax skip are opt-in via `MLX_BUN_CCE_BWD_FILTER_EPS` / `_BLOCK_EPS` (default off — exact). B=1 (orpo only) |
+| `orpo_flash_ce` | bool | `false` | Route the fused head through the **flash-CCE Metal kernel** (verbatim MLX steel GEMM + ORPO epilogue, fwd **and** bwd): neither `[M,vocab]` nor a dequantized head touches HBM → `[M,vocab]`-free (e4b 0.93 GB flat @ M=8192). Implies the fused head. The Apple-CCE coeff filter + blockMax skip default ON at 1e-5 (2026-07-02; combined bwd 1.71× CPM5 / 3.16× e4b vs exact — `MLX_BUN_CCE_BWD_FILTER_EPS=0` / `_BLOCK_EPS=0` restore exact). Batches with M < `MLX_BUN_FLASH_MIN_M` (default 1024) auto-take the exact fused head (faster at short M; `=0` always honors flash). B=1 (orpo only) |
 | `orpo_prefix_shared` | bool | `false` | Shared prompt-prefix: one forward over `[prompt; chosen; rejected]` with a block-sparse mask + block-wise RoPE, so the shared prompt is encoded **once** (a big win when the prompt dominates). Falls back to two-forward per row on prompt mismatch. Composes with the flash head and the segmented backward (MiniCPM5 + e4b). B=1 (orpo only) |
 | `sft_scope` | `full` \| `response` | `full` | Scope of ORPO's chosen-NLL term. `full` (paper/TRL-faithful): token-mean CE over the **full prompt+response** (only padding excluded), from the same chosen forward. `response`: `L_NLL = −ℓ_w` — reproduces pre-2026-07 runs bit-exactly. The odds-ratio `ℓ_w/ℓ_r` are response-only in both modes. Applies to every ORPO path (naive/chunked/fused/flash/prefix/segmented) (orpo only). CLI spelling: `mlx-bun train … --sft-scope full\|response` |
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mlx-bun",
-  "version": "0.0.9",
+  "version": "0.0.10",
   "description": "Native MLX inference for Bun on Apple Silicon — local LLM server (OpenAI/Anthropic-compatible) and TypeScript library. No Python.",
   "type": "module",
   "license": "MIT",

package/src/cli.ts

@@ -9,6 +9,8 @@
 //   mlx-bun evals                         recent benchmark runs
 //   mlx-bun harness pi [--base-url <url>] [--remove]   connect your own pi to the local server
 
+import { existsSync } from "node:fs";
+import { basename, join, resolve } from "node:path";
 import { Registry } from "./registry";
 import { loadModelConfig } from "./config";
 import { fit, skuMatrix, thisMachine } from "./fit";
@@ -66,6 +68,10 @@ Examples:
   mlx-bun get mlx-community/gemma-4-12B-it-OptiQ-4bit`;
 
 const SERVER_FLAGS = `Server options:
+  --model <path|query>      Explicit model override (mlx_lm.server spelling):
+                            a directory containing config.json loads from that
+                            path; anything else resolves like the positional
+                            query. Wins over the positional and auto-pick.
   --host <addr>             Interface to bind  [default: 127.0.0.1, loopback
                             only; use 0.0.0.0 to expose on your network]
   --port <n>                Listen port  [default: 8080]
@@ -74,6 +80,14 @@ const SERVER_FLAGS = `Server options:
                             GPU  [default: machine RAM × 0.75, check-only]
   --prompt-cache <GB>       Prompt (KV) cache byte cap  [default: 2 GB;
                             --prompt-cache 0 disables the cache]
+  --ssd-cache <dir>         SSD cold tier for the prompt cache: prefix KV
+                            spills to disk on eviction and persists across
+                            restarts (long-context agent TTFT drops from a
+                            full re-prefill to a zero-copy mmap restore).
+                            Off unless set.
+  --ssd-cache-max <GB>      SSD tier byte cap  [default: 32 GB]
+  --ssd-cache-verify        Verify tensor hashes on every restore (reads all
+                            bytes eagerly — integrity paranoia only)
   --batch <n>               Max concurrent requests batched through the
                             mlx-lm-parity engine  [default: 1 = serial].
                             >1 opts the whole server into bf16 continuous
@@ -285,6 +299,9 @@ and runs the mlx-lm/optiq comparison legs.
                        positional)  [default: auto-pick]
 
 Options:
+  --model <path|query> Explicit model override: a directory containing
+                       config.json loads from that path; anything else
+                       resolves like the positional query. Wins over both.
   --tokens <n>         Tokens to decode per run  [default: 256]
   --runs <n>           Runs (median reported)  [default: 3]
   --prompt-tokens <n>  Pad the prompt to ~n tokens (long-context decode)
@@ -785,6 +802,14 @@ function serverRuntimeFlags(): { port: number; serverOptions: import("./server")
   if (budgetGB > 0) serverOptions.memoryBudgetBytes = budgetGB * 1e9;
   const pcRaw = opt("prompt-cache"); // null if absent; an explicit value (incl. 0) wins → `--prompt-cache 0` DISABLES the cache
   if (pcRaw !== null) serverOptions.promptCacheBytes = Math.max(0, Number(pcRaw)) * 2 ** 30;
+  // SSD cold tier (docs/design/ssd-kv-cold-tier.md) — off unless a dir is given.
+  const ssdDir = opt("ssd-cache");
+  if (ssdDir !== null) {
+    serverOptions.ssdCacheDir = ssdDir;
+    const maxRaw = opt("ssd-cache-max");
+    if (maxRaw !== null) serverOptions.ssdCacheMaxBytes = Math.max(1, Number(maxRaw)) * 2 ** 30;
+    if (flag("ssd-cache-verify")) serverOptions.ssdCacheVerify = true;
+  }
   // --batch N: max concurrent requests batched through the mlx-lm-parity
   // engine (N=1 = today's serial path). --decode-concurrency is accepted for
   // drop-in compatibility with mlx_lm.server, but the semantics differ: there
@@ -893,8 +918,21 @@ function runtimeSummary(o: import("./server").ServerOptions): string {
 
 /** Shared model resolution: explicit query wins; otherwise e4b (the default
  *  model everywhere) if it's downloaded, else the largest supported model that
- *  fits this machine. Downloads the recommended model first on a fresh install. */
+ *  fits this machine. Downloads the recommended model first on a fresh install.
+ *  A query that is a DIRECTORY containing config.json loads directly via
+ *  scanSnapshot (no registry dependency) — the `--model <path>` bench/CI shape. */
 async function resolveModelAuto(query: string | null): Promise<{ m: import("./registry").ModelRecord; picked: boolean }> {
+  if (query && existsSync(join(query, "config.json"))) {
+    const { scanSnapshot } = await import("./registry");
+    const dir = resolve(query);
+    // Recover the repo id from an HF-cache path (…/models--org--name/snapshots/<rev>);
+    // otherwise the directory basename is the display id.
+    const hf = /models--([^/]+)--([^/]+)\/snapshots\//.exec(dir);
+    const repoId = hf ? `${hf[1]}/${hf[2]}` : basename(dir);
+    const m = await scanSnapshot(dir, repoId);
+    if (!m) throw new Error(`${dir} has config.json but no weights (*.safetensors)`);
+    return { m, picked: false };
+  }
   const reg = new Registry();
   if (reg.list().length === 0) await reg.scan();
   if (query) return { m: reg.resolve(query), picked: false };
@@ -1220,7 +1258,12 @@ switch (cmd) {
         process.exit(1);
       }
     }
-    const { m, picked } = await resolveModelAuto(positional(0) ?? opt("query"));
+    // --model <path-or-query> is the explicit override (mlx_lm.server's
+    // spelling — previously accepted-but-IGNORED here, which silently
+    // auto-picked the wrong model). A directory with config.json loads
+    // straight from that path; anything else resolves like the positional
+    // fuzzy query. Precedence: --model > positional > --query > auto-pick.
+    const { m, picked } = await resolveModelAuto(opt("model") ?? positional(0) ?? opt("query"));
     const sFit = step("assessing fit");
     const config = await loadModelConfig(m.path);
     const report = fit(config, m.sizeBytes, 8192, undefined, undefined, m.expertsBytes);
@@ -1385,7 +1428,7 @@ switch (cmd) {
     const { banner, step, box, style } = await import("./tui");
     banner(pkg.version);
     serverRuntimeFlags(); // env levers (--compiled-decode, --perf-kernel, ...) apply to the run
-    const { m, picked } = await resolveModelAuto(positional(0) ?? opt("query"));
+    const { m, picked } = await resolveModelAuto(opt("model") ?? positional(0) ?? opt("query"));
     const tokens = Number(opt("tokens", "256"));
     const runs = Number(opt("runs", "3"));
     const promptTokens = Number(opt("prompt-tokens", "0"));
@@ -1573,13 +1616,14 @@ switch (cmd) {
     // (src/pi-terminal.ts) — pi is bundled, nothing to install. Users who
     // already run their own pi connect it with `mlx-bun harness pi`.
     const OURS_VAL = new Set([
-      "--query", "-q", "--port", "--host", "--memory-budget", "--prompt-cache", "--kv-quant",
+      "--query", "-q", "--model", "--port", "--host", "--memory-budget", "--prompt-cache",
+      "--ssd-cache", "--ssd-cache-max", "--kv-quant",
       "--batch", "--decode-concurrency", "--adapter", "--adapter-path",
       "--compiled-decode", "--perf-kernel", "--fused-decode", "--fused-sdpa", "--thinking",
       "--temperature", "--temp", "--top-p", "--top-k", "--max-tokens",
       "--hlg-sampling", "--hlg-width", "--hlg-shoulder", "--hlg-toe", "--hlg-pivot-offset",
     ]);
-    const OURS_BOOL = new Set(["--force-wire", "--expert-offload", "--no-open", "--l1", "--l2", "--l3"]);
+    const OURS_BOOL = new Set(["--force-wire", "--expert-offload", "--no-open", "--ssd-cache-verify", "--l1", "--l2", "--l3"]);
     const passthrough: string[] = [];
     for (let i = 1; i < argv.length; i++) {
       const a = argv[i]!;

package/src/generate.ts

@@ -508,8 +508,16 @@ async function* generateInner(
     // 0.6.0, and shapeless replay re-infers the whole tape whenever the
     // growing attention windows change shape (= every step). Remove this
     // when upstream implements GatherQMM::output_shapes.
+    // MLX_BUN_FUSED_DECODE=1 (the N-tiled fused decode experiment) cannot
+    // compose with compiled decode: the tile loop's bounds bake at trace-time
+    // N, so growing quantized caches silently stop attending their newest
+    // rows under shapeless replay (kernel-perf-review 2026-07; reproduced on
+    // e4b). The explicit opt-in wins over the default optimization — same
+    // policy as LoRA and MoE below. quantizedSdpa also throws on the combo
+    // as a backstop.
     let compiled =
       process.env.MLX_BUN_COMPILED_DECODE !== "0" &&
+      process.env.MLX_BUN_FUSED_DECODE !== "1" &&
       !options.adapters?.length &&
       model.config.modelType.startsWith("gemma4") &&
       !model.config.text.enableMoeBlock