mlx-bun 0.0.10 → 0.0.11

package/docs/reference/server-config.md

@@ -22,18 +22,28 @@ the design rationale is in
 | `--host` | addr | `127.0.0.1` | Interface to bind. Loopback-only by default (mlx_lm.server parity); pass `--host 0.0.0.0` to expose the server on your network. |
 | `--port` | n | `8080` | Listen port. A pre-flight probe refuses to start if the port is already serving. |
 | `--memory-budget` | GB | machine RAM × 0.75 | Admission ceiling. Requests whose `prompt + max_tokens` exceed the budget's max safe context are **rejected with 400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. Also caps the mlx allocator (`mlx_set_memory_limit`) as defense in depth. **Decimal GB (×10⁹).** |
-| `--prompt-cache` | GB | `2` (≈2e9 bytes) | Byte cap for the prompt (prefix-reuse KV) cache. **Binary GiB (×2³⁰)** on input. Unset ⇒ the 2 GB default; an explicit `--prompt-cache 0` **disables** the cache (maxBytes 0 evicts every entry). |
-| `--ssd-cache` | dir | off | SSD cold tier under the prompt cache ([docs/design/ssd-kv-cold-tier.md](../design/ssd-kv-cold-tier.md)): prefix KV spills to disk on RAM eviction, is snapshotted after requests settle (debounced 1 s), and **survives restarts** — a long-context agent re-attach restores via zero-copy mmap instead of re-prefilling (measured 13.7k-token prefix: 12 s → 0.24 s TTFT; 0% decode overhead). Entries are keyed by model fingerprint **+ effective kv scheme** + tokenizer hash + adapter ns; incompatible/corrupt files self-quarantine. Serial lane only. Requires the RAM cache. |
+| `--prompt-cache` | GB | `8` | RAM prefix-KV cache (byte-capped LRU; `0` disables). **PREFIX SHARING (2026-07-05): serves are NON-CONSUMING** — a hit hands out zero-copy clones and leaves the donor entry intact, so N agents (or a brand-new session) sharing a system prompt all reuse ONE prefill without destroying each other's conversation entries (the old consume-and-trim take cannibalized the donor). Extended entries supersede their prefix-ancestors (when trimmable), so a conversation stays one entry. Works on BOTH lanes and tiers over the SSD store (Layer 0). |
+| `--isolate` | (bool) | off | Runtime isolation ([runtime-isolation.md](../design/runtime-isolation.md)): the inference engine runs as a CHILD process on a unix socket; this process stays a pure UI/API reverse proxy — instant under any GPU load, survives engine crashes (auto-respawn, in-flight requests get a clean 502). Measured cost ≈0 (−0.4% tok/s, +2 ms TTFT, per-token SSE granularity preserved). `/ws/chat` is not proxied yet (501). `/engine` reports child pid/restarts. |
+| `--model-pool` | n | `1` | With `--isolate`: max RESIDENT model engines. Requests naming another model (**exact `/v1/models` id** — fuzzy strings keep mlx-lm's ignored-field semantics) spawn/route to that model's own engine child (SPAWN-OVERLAP: the new model loads while the old one keeps serving; nobody's stream is interrupted). Over the cap, the LRU engine drains, **demotes its prompt cache to the SSD tier**, and exits — switching back respawns it with state restored from disk. Measured (M1 Max, cpm5⇄qwen0.8b, pool 1): switch 1.5 s, switch-back 1.2 s with `cached_tokens` 103/104 — the conversation survived the eviction. The pool drives this eviction via `POST /admin/drain` — unix-socket-only (never exposed on the public TCP listener) — which quiesces the gateway and demotes the whole prompt cache to the SSD tier before eviction, responding `{ drained: true, demotions: N }`. |
+| `--ssd-cache` | dir | off | SSD cold tier under the prompt cache ([docs/design/ssd-kv-cold-tier.md](../design/ssd-kv-cold-tier.md)): prefix KV spills to disk on RAM eviction AND idle demotion, is snapshotted after requests settle (debounced 1 s; the flush is idle-gated per tensor so it never taxes an active decode — `MLX_BUN_SSD_WRITEBEHIND=0` disables), and **survives restarts** — a long-context agent re-attach restores via a bounded streamed copy instead of re-prefilling (measured 13.7k-token prefix: 12 s → ~0.25 s TTFT; 0% decode overhead; restore transient = live entry + one tensor, no mapping outlives the load). Entries are keyed by model fingerprint **+ effective kv scheme** + tokenizer hash + adapter ns; incompatible/corrupt files self-quarantine. **Layer 0 (2026-07-05): the tiering lives inside PromptCache.take(), so BOTH lanes restore from disk — batch-scheduler joiners included — and every put (batch rows too) schedules the write-behind snapshot.** Requires the RAM cache. |
 | `--ssd-cache-max` | GB | `32` | SSD tier byte cap (binary GiB); oldest-mtime entries evicted at the cap. |
-| `--ssd-cache-verify` | (bool) | off | Verify every tensor hash on restore. Reads all bytes eagerly (defeats lazy page fault-in) — integrity paranoia only; the header hash is always verified. |
-| `--batch` | n | `1` (serial) | Max concurrent requests batched through the mlx-lm-parity engine. `>1` switches the **whole server** into bf16 continuous batching — a *mode*, not a load fallback. See [Execution modes](#execution-modes-serial-vs---batch-n). `--decode-concurrency` is accepted for drop-in compatibility, but the semantics differ: in mlx_lm.server it caps per-BatchGenerator decode parallelism (default 32); in mlx-bun it enables continuous batching with this cap (mlx-bun's default is the optimized serial path). |
-| `--kv-quant` | `config`\|`off`\|`4`\|`8` | `config` serial / `off`(bf16) under `--batch N` | KV-cache quantization. `config` = per-layer `kv_config.json` (optiq parity); `off` = bf16; `4`/`8` = uniform bits (group 64, start 0). Under `--batch N`, an explicit value routes those requests to the **serial** lane (batched is bf16-only). |
+| `--ssd-cache-verify` | (bool) | off | Verify every tensor hash on restore before it is copied in (roughly doubles restore reads) — integrity paranoia only; the header hash is always verified. |
+| `--ssd-demote-idle` | sec | `300` (with `--ssd-cache`) | Idle demotion (Layer 0): prompt-cache entries unused this long spill to the SSD tier and **free their GPU memory** — unified memory drains between agent bursts while every prefix stays reachable (the next hit restores via the bounded streamed copy, ~0.25 s for a 13.7k-token entry vs a 12 s re-prefill). Swept only when the engine is fully idle (never drains a running batch). `0` disables. |
+| `--batch` | n | `8` | Max concurrent requests decoded together. **Default flipped 1→8 (2026-07-05)** after GATE-B1-SPEED: a LONE request through the unified engine IS the serial engine — adopted serial-class caches, compiled decode, prompt cache + SSD restore, byte-identical output, 0.992–0.996 paired decode ratios — so the cap only changes behavior when concurrent requests actually arrive (the agentic sub-agent workload: 4–8 coding agents against one local server). `--batch 1` pins strict serial (arrival-independent numerics). See [Execution modes](#execution-modes-serial-vs---batch-n). `--decode-concurrency` is accepted for drop-in compatibility (mlx_lm.server's cap, default 32). |
+| `--kv-quant` | `config`\|`off`\|`4`\|`8`\|`turbo[:k<bits>v<bits>]` | `off` (bf16) | KV-cache quantization. **Default flipped to bf16 2026-07-05** (naked = L1): quantized KV measured 5–20% slower decode than bf16 at ≤16k on every model — on mlx-lm too — so it pays only in **memory headroom** (e.g. ~1.3 GB on the 12B @16k) and is an explicit opt-in (`--kv-quant …` or `--l2`, whose preset passes `config`). `off` = bf16 (L1). `4`/`8` = **uniform** bits (group 64, start 0) — the scheme mlx-lm exposes as `--kv-bits`/`--kv-group-size`/`--quantized-kv-start`. In the L1 kernel set (`--l1 --kv-quant 8` = fused-sdpa off) our unfused quantized SDPA is **op-for-op identical** to mlx-lm's `quantized_scaled_dot_product_attention`, so uniform-quant is **bit-exact L1** (the fused-sdpa-ON path is optiq-aligned). `config` = **per-layer mixed-precision** from `kv_config.json` — optiq-only, no mlx-lm analog → **L2**. `turbo[:k<bits>v<bits>]` = **TurboQuant** ([docs/design/turboquant-kv.md](../design/turboquant-kv.md)): rotation-based KV quantization (asymmetric-affine int8/int4/int2/int5 keys, FWHT + Lloyd-Max values) — default `k8v3` (2.56× compression at head_dim 128); `kBits` ∈ {2,4,5,8}, `vBits` ∈ {2,3,4,5,8}. A separate axis from the scheme above (mutually exclusive with `config`/`4`/`8`); v1 is dequantize-on-fetch (stock `ops.sdpa`, no fused kernel) and **full-attention layers only** — sliding-window layers stay bf16 (one-time warning, never a throw). Under `--batch N`, a per-layer `config` scheme **batches** — full-attention layers since Phase 3.1 (MiniCPM5), rotating/sliding layers since milestone 2 (gemma's whole kv_config), so every shipped `kv_config.json` now batches. The scheduler applies the mixed scheme per row, gated bit-exact per row vs the serial composition (`tests/batched-kv-quant-parity.test.ts` — compositions no other stack ships). Uniform `4`/`8` still routes those requests serial (quantizedKvStart threshold semantics). **`turbo` is solo-only, unconditionally** — `TurboQuantKVCache` is a novel `Cache` implementation (no merge/filter/temporalView), so it's excluded from the batch scheduler by construction; `GenerationGateway.willBatch` also refuses it explicitly (belt + braces — both layers exist on purpose). |
+| `--paged-kv` | (bool) | off | **OPTIONAL vLLM-style paged KV cache** ([docs/design/paged-kv-cache.md](../design/paged-kv-cache.md)): full-attention layers store K/V in fixed-size **block pools** (host-side block table, `ops.takeAxis` gather back to contiguous before the stock SDPA — no new attention math, no fused kernel in v1). Env: `MLX_BUN_PAGED_KV=1`. **Gated bit-exact vs the plain path** (`tests/paged-kv-parity.test.ts`: identical single-forward logits + identical 48-token greedy trajectory on the 12B). **v1 scope: serial batch=1, Gemma4-family, bf16** — with no explicit `--batch` the CLI pins `--batch 1`; the server **refuses** (fails startup, never silently downgrades) `--batch N>1`, any `--kv-quant`, and `--draft-model`; sliding-window layers keep the rotating scheme (mixed paged-full + rotating-sliding is the supported shape). Media (vision/audio) and LoRA-adapter **requests** are scoped OUT per request — they run the plain cache path (v1 non-goal cells), never a 400. Paged requests **bypass the prompt cache** (fresh caches per request; `--ssd-cache` sees nothing) and run **uncompiled decode** (a data-dependent block-list length can't shapeless-replay). Expect a small decode cost at batch=1 — the per-step gather copy buys nothing until the batched/CoW follow-ups; this ships the correctness-proven abstraction, honestly labeled. Pool sized from prompt+`max_tokens` at request setup; exhaustion is a typed error (accounting tripwire), never truncation. |
+| `--paged-kv-block-size` | n | `256` | Tokens per KV block (`--paged-kv` only). 256 = the plain cache's growth step, so v1's allocation granularity is a permutation of today's into reusable slots, not a new tuning axis. |
 | `--adapter` | dir | none | Mount a LoRA adapter at startup (same machinery as `POST /v1/adapters`; the adapter id is the directory's basename) and make it the **default** for requests that send no `adapter` field. A request's explicit `adapter` — including `"none"` — always wins, and hot-swap via `/v1/adapters` is unchanged. `--adapter-path` is accepted as the mlx_lm.server-named alias. A bad adapter fails startup loudly rather than silently serving the base model. This is the flag `mlx-bun train`'s completion message points at. |
-| `--thinking` | `true`\|`false` | model's own (false for CPM) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM and Qwen3.5 hybrid reasoning). A request's `chat_template_kwargs.enable_thinking` overrides it. |
+| `--draft-model` | path/query | none | **Speculative decoding**: a drafter proposes tokens the main model verifies in one forward — exact results, faster decode when drafts land. Resolves like the main model. The artifact's **kind is auto-detected**: a full same-tokenizer model (mlx_lm.server parity — L1: token-for-token vs mlx-lm's spec path; tokenizer-family mismatch fails startup, upstream silently accepts ~0%), a Gemma `-assistant` KV-borrowing drafter (L2 vs optiq spec_generate; ~1.09× γ=1 on 12B, no training), a locally-trained **DSpark** checkpoint (`dspark.json`), or one of **DeepSeek's released DSpark drafters** (DeepSpec `Gemma4DSparkModel` config stamp, e.g. `deepseek-ai/dspark_gemma4_12b_block7` for the 12B — oracle: DeepSpec's reference at temperature 0, token-for-token). All four share ONE serve loop. A confidence-calibrated DSpark checkpoint additionally self-schedules its draft length per round (threshold pruning — fewer wasted verify positions; uncalibrated checkpoints draft fixed-length). Mounting a draft routes **every** request to the serial lane (upstream `is_batchable = draft is None`). Pays on slow targets (12B+); fast small models lose to the draft overhead. Composes with structured output (the constrained verify walk). Prompt-cache reuse is bypassed on the spec path (v1). Telemetry: `usage.speculation` (`drafted`/`accepted`/`targetCalls`). |
+| `--draft-kind` | `two-model`\|`assistant`\|`dspark`\|`deepspec`\|`ngram` | auto | Override the draft-artifact kind detection (`dspark.json` → dspark; `Gemma4DSparkModel` architecture stamp → deepspec; `*_assistant` config → assistant; else two-model). **`ngram` is never auto-detected — it has no artifact**: MODEL-FREE prompt lookup (drafts copied from the request's own prompt+generation when its trailing k-gram occurred earlier in the stream; port of prompt-lookup decoding / vLLM's `ngram` proposer). Mount it ALONE — `--draft-kind ngram` with a `--draft-model` is refused, as is any other kind without one. Zero weights/memory, lossless by the same verify (gated: serve-loop output token-identical to non-spec greedy on e4b, tests/spec-ngram.test.ts); a no-match round degrades to one plain target step. Best on agentic/RAG/code-edit traffic that re-emits context spans. |
+| `--num-draft-tokens` | n | `3` (`ngram`: `10`) | Drafts per verify round (mlx_lm.server's default; `mlx_lm.generate`'s is 2). A DSpark draft pins this to its trained block width (`cfg.gamma`); ngram drafting is free so its default is the reference implementation's 10. |
+| `--ngram-max` / `--ngram-min` | k | `3` / `1` | `--draft-kind ngram` only: longest/shortest trailing k-gram searched for a match (the reference values — longest first, first occurrence wins). |
+| `--thinking` | `true`\|`false` | model's own (false for CPM) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM and Qwen3.5 hybrid reasoning). Full precedence: an explicit request `chat_template_kwargs.enable_thinking` always wins; else `reasoning_effort` (`"none"` → off, any other level → on) if the request sent one; else this server default; else the model's own default. |
 | `--temperature` | n ∈ [0,5] | `generation_config.json` | Server-wide sampling default. Per-request `temperature` still wins; the browser chat (sends none) inherits this. `--temp` is accepted as an alias (mlx_lm.server compat); explicit `--temperature` wins if both are given. **Migration note:** mlx_lm.server's `--temp` *default* is `0.0` (unset-temperature requests are greedy there); mlx-bun falls back to the model's `generation_config.json`, then `0.7` — pass `--temp 0` for mlx-lm's behavior. |
 | `--top-p` | n ∈ [0,1] | `generation_config.json` | Server-wide top-p default (per-request `top_p` wins). |
 | `--top-k` | n ∈ [0,1e6] | `generation_config.json` | Server-wide top-k default (per-request `top_k` wins). |
 | `--max-tokens` | n | `65536` chat / `512` raw completion | Completion cap when a request omits `max_tokens` (mlx_lm.server flag; its default there is 512 — `--max-tokens 512` reproduces mlx_lm.server exactly). |
+| `--allow-private-media` | (bool) | off | Let `image_url`/`audio_url` content parts fetch from **private/loopback/link-local** hosts (a NAS, another LAN box). Off by default: a request's URL is attacker-controlled input, so remote media fetches (`src/media-fetch.ts`) refuse non-http(s) schemes and destinations in private/loopback/link-local/CGNAT ranges — including hosts that *resolve* there and every **redirect hop** (each hop re-passes the whole policy) — which keeps the server from being steered at cloud metadata (`169.254.169.254`) or LAN-internal services (SSRF). Independent of the flag, every remote fetch has a **10 s wall-clock timeout** and a **64 MB response cap** (streaming-enforced, not just Content-Length); violations surface as clean `400`s (`prompt build failed: …`). `data:` URLs decode locally and are never policy-checked. |
 | `--no-open` | (bool) | off | Skip the automatic browser open on start. By default an interactive terminal session opens `http://<host>:<port>/#/chat` once the server is ready; pass this flag to suppress it (e.g. headless or non-TTY environments already skip it). |
 | `--hlg-sampling` | `on`\|`off` | off | Piecewise tone-curve (HLG) sampling: rolls off the top-token region, boosts the mids, gentles the tail. The overall gain folds from `--temperature`. See [docs/design/hlg-sampling.md](../design/hlg-sampling.md). |
 | `--hlg-width` | nats | `4` | HLG mid-region half-width (nats). Only meaningful with `--hlg-sampling on`. |
@@ -41,32 +51,34 @@ the design rationale is in
 | `--hlg-toe` | nats | `6` | HLG shadow rolloff scale (nats). Only meaningful with `--hlg-sampling on`. |
 | `--hlg-pivot-offset` | nats | `6` | HLG pivot point: nats below the top token. Only meaningful with `--hlg-sampling on`. |
 | `--expert-offload` | (bool) | off | **MoE models only.** Serve experts from a page-aligned file mmap (built on first use). Keeps the model out of memory pressure — physical footprint ≈ active params. Ignored with a warning on dense models. Bit-exact with the resident path. |
-| `--l1` | (bool) | — | **Parity tier preset:** bit-for-bit IDENTICAL to mlx-lm (bf16 KV, unfused SDPA, no perf kernel). Sets the whole decode route; any explicit per-fork flag (`--kv-quant`/`--perf-kernel`/…) overrides the preset. See [docs/design/parity-tier-dag.md](../design/parity-tier-dag.md). |
-| `--l2` | (bool) | — | **Parity tier preset:** bit-for-bit IDENTICAL to mlx-optiq (quantized KV per `kv_config.json` + fused N-tiled prefill SDPA + **stock unfused decode** — the composition the optiq goldens track, `scripts/regen-kvq-goldens.ts`). The perf kernel stays **off**: it is envelope-gated, not bit-exact (see `--l3`), and opting into it (`--l2 --perf-kernel on`) is an explicit choice that leaves the bare-tier guarantee. |
-| `--l3` | (bool) | — | **Parity tier preset:** best performance, no bit-exact oracle (KL + test gated). On the decode path L3 = L2 + the envelope-gated perf kernel; L3 also owns the no-oracle features (HLG sampler, expert offload). No tier given ⇒ the per-flag defaults below, which equal `--l3`. |
+| `--l1` | (bool) | — | **Parity tier ALIAS:** bit-for-bit IDENTICAL to mlx-lm (bf16 KV, compiled decode + compiled activations — the faithful `@mx.compile` geglu/swiglu). **No tier given ⇒ `--l1`** (decided 2026-07-05: the L1 faithful kernel set matches mlx-lm 1.00× on every model and no output-changing lever has beaten that baseline in a paired A/B — each is opt-in until it does). Expands to the fastest set of per-fork flags that still holds the guarantee; any explicit per-fork flag (`--kv-quant`/`--fused-sdpa`/`--compiled-decode`/`--compiled-activations`) overrides one. See [docs/design/faithful-l1-consolidation.md](../design/faithful-l1-consolidation.md) and [parity-tier-dag.md](../design/parity-tier-dag.md). |
+| `--l2` | (bool) | — | **Parity tier preset:** bit-for-bit IDENTICAL to mlx-optiq (quantized KV per `kv_config.json` + fused N-tiled prefill SDPA + **stock unfused decode** — the composition the optiq goldens track, `scripts/regen-kvq-goldens.ts`). |
 | `--compiled-decode` | on\|off | on | Replay the per-step decode graph in C++ (`MLX_BUN_COMPILED_DECODE`). Bit-exact A/B lever. **Serial lane only** (see note below). **Gemma4-dense only** — LoRA, MoE, and non-Gemma4 models (MiniCPM5 / Qwen3.5) run eager; an unsupported step falls back to eager for the rest of that generation. |
-| `--perf-kernel` | on\|off | **on** | Fused quantized-KV decode-SDPA Metal kernel (`MLX_BUN_PERF_KERNEL`), the perf side of the compat A/B. **Not bit-exact** — envelope-gated (≥56/64 teacher-forced argmax vs the frozen compat trajectory, `tests/perf-kernel-oracle.test.ts`), so it is an **L3 lever**: on in `--l3`/no-tier default, **off in bare `--l1`/`--l2`**. Engages on quantized caches at decode. **Serial lane only.** |
-| `--fused-decode` | on\|off | off | Experimental: tile the quantized decode SDPA (`MLX_BUN_FUSED_DECODE`). **Serial lane only.** |
-| `--fused-sdpa` | on\|off | on | Fused SDPA path for quantized prefill/continuation (inverted env `MLX_BUN_NO_FUSED_SDPA`). **Serial lane only.** |
+| `--compiled-activations` | on\|off | on | Route the geglu/swiglu activation through mlx-lm's `@mx.compile` closure (`MLX_BUN_COMPILED_GEGLU` + `MLX_BUN_COMPILED_SWIGLU`) — the **faithful** kernel: same libmlx graph as mlx-lm → **bit-exact** AND one dispatch instead of ~9. `off` = the uncompiled composition (same L1 parity, slower). Toggles gemma geglu + MiniCPM5 swiglu; qwen3/qwen3.5/universal compile unconditionally. |
+| `--fused-sdpa` | on\|off | follows `--kv-quant` | Fused SDPA path for quantized prefill/continuation (inverted env `MLX_BUN_NO_FUSED_SDPA`). Defaults to the composition its oracle uses: **on** under `--kv-quant config` (the optiq-golden composition), **off** under uniform `4`/`8` (mlx-lm's `quantized_scaled_dot_product_attention`, the bit-exact L1-eligible scheme) and bf16 (no-op there). **Serial lane only.** |
 | `--force-wire` | (bool) | off | Wire weights into memory for the whole generation (`MLX_BUN_FORCE_WIRE`). Near-ceiling models (e.g. 26B) need it. **Serial lane only.** |
 
 The default host/port (`127.0.0.1:8080`) match `mlx_lm.server`, so running
 mlx-bun alongside the Python reference server needs an explicit `--port`.
 
-The five performance levers are A/B knobs whose defaults are the measured
-winners; flip them to compare. They are set as `MLX_BUN_*` env vars
-before the model loads, so they apply to `mlx-bun pi` too. They affect
-the **serial** decode path (`generate()`); the batched scheduler calls
-`model.forwardHidden` directly and so is unaffected by all of them — see
+The remaining levers are bit-exact A/B knobs; **the naked default is the
+L1 set** (2026-07-05 decision: an output-changing lever earns a default
+only by beating the L1 faithful baseline in a paired A/B, and none did —
+the 2026-07-05 pass measured the custom fused-decode at 1.00×, fused-gelu
+at +0–1%, the flash perf-kernel at 0.62–0.93× on e4b, and quantized KV
+5–20% *slower* decode than bf16 at ≤16k on both stacks; those losing
+kernels and their flags were **deleted** the same day — Phase 1 of
+[unified-engine-frontier-plan.md](../design/unified-engine-frontier-plan.md).
+New output-changing experiments live in the **Lab**: env-flagged, with a
+bench + expiry, until one beats the L1 baseline in a paired A/B). Flip
+the survivors to compare. They are set as `MLX_BUN_*` env vars before the
+model loads, so they apply to `mlx-bun pi` too. They affect the
+**serial** decode path (`generate()`); the batched scheduler calls
+`model.forwardHidden` directly and is unaffected by all of them except
+`--compiled-decode`, which also covers a lone batch-lane request (Phase
+3.2) — see
 [Levers that don't reach the batched lane](#--batch-n-is-compat-mode--perf-flags-dont-apply-by-design).
 
-> **Note — `--perf-kernel` default.** The code defaults it **on**
-> (`perfKernelEnabled()` returns true unless `MLX_BUN_PERF_KERNEL=0`, and
-> the ready card / `/stats` report it on). `STATUS.md` still lists a
-> "default flip" as pending; treat that as stale — the engaged default
-> today is on. (If the intent is genuinely off-until-clean-machine-pass,
-> the *code* is what needs changing, not this doc.)
-
 ## Per-request overrides
 
 Most quality knobs can be set per request in the chat body and override
@@ -88,44 +100,61 @@ These ride all three protocol surfaces (`/v1/chat/completions`,
 ## Environment levers
 
 The CLI flags above set these; you can also export them directly (e.g.
-for `bun scripts/serve.ts` or paired A/B harnesses). One has no CLI flag.
+for paired A/B harnesses).
 
 | Env var | CLI flag | Default | Effect |
 | --- | --- | --- | --- |
 | `MLX_BUN_COMPILED_DECODE` | `--compiled-decode` | on (`!=="0"`) | Compiled decode graph replay. |
-| `MLX_BUN_PERF_KERNEL` | `--perf-kernel` | on (`!=="0"`) | Fused quantized-KV decode kernel (not bit-exact; perf A/B). |
-| `MLX_BUN_FUSED_DECODE` | `--fused-decode` | off (`==="1"`) | Tile quantized decode SDPA. |
-| `MLX_BUN_NO_FUSED_SDPA` | `--fused-sdpa` (inverted) | fused on | `=1` forces the stock unfused SDPA everywhere. |
-| `MLX_BUN_FUSED_GELU` | *(none)* | on (`!=="0"`) | Fused GeGLU MLP kernel. Bit-exact, so it stays on both lanes; env-only opt-out. |
+| `MLX_BUN_NO_FUSED_SDPA` | `--fused-sdpa` (inverted) | follows `--kv-quant` | `=1` forces the stock unfused SDPA everywhere. |
+| `MLX_BUN_COMPILED_GEGLU` | `--compiled-activations` | on (`!=="0"`) | Gemma geglu via mlx-lm's `@mx.compile` closure — the faithful default (bit-exact vs mlx-lm, one kernel). `=0` → uncompiled composition (same parity, slower). |
+| `MLX_BUN_COMPILED_SWIGLU` | `--compiled-activations` | on (`!=="0"`) | `mx.compile`'d SwiGLU (`silu(gate)·up` → one kernel) on MiniCPM5 decode (M=1), porting mlx-lm's `activations.py`. Bit-exact (passes the exact logit-parity gate), both lanes. +5.5% CPM5 decode. (qwen3/qwen3.5/universal compile swiglu unconditionally, independent of this flag.) |
 | `MLX_BUN_FORCE_WIRE` | `--force-wire` | off (`==="1"`) | Wire weights for the generation. |
+| `MLX_BUN_PAGED_KV` | `--paged-kv` | off (`==="1"`) | Paged KV cache (see the flag row above — v1: serial, Gemma4, bf16; refusals + prompt-cache bypass apply identically). |
+| `MLX_BUN_ALLOW_PRIVATE_MEDIA` | `--allow-private-media` | off (`==="1"`) | Permit `image_url`/`audio_url` fetches to private/loopback/link-local hosts (the SSRF guard's LAN escape hatch — see the flag row above; timeout + size cap still apply). |
+| `MLX_BUN_PREFILL_TAIL_SPLIT` | — | on (`!=="0"`) | Oracle prefill convention (mlx-lm `generate_step` and its server's batched engine): drain the prompt only to len−1, then compute step-0 logits from a separate **L=1 forward of the last prompt token**. Both lanes. **The SPEC lane (`--draft-model`) follows its own oracle's shape under the same flag** (mlx-lm `speculative_generate_step`, re-anchored 2026-07-07): both target AND draft drain to len−1 and there is **no separate step-0 at all** — the un-drained last prompt token heads the first verify window. Gated live: token-for-token vs the oracle venv incl. a knife-edge cell (tests/spec-serve.test.ts "L1 knife-edge"). (The standalone optiq-oracled assistant loop `specGenerate` deliberately keeps FULL-prompt prefill — optiq's own convention.) `=0` restores the pre-2026-07-07 full-final-chunk convention everywhere (A/B lever + kill switch) — that convention is ulp-different in bf16 at step 0 AND in the last prompt token's stored KV, which flips near-tie greedy streams vs mlx-lm (the 2026-07-07 12B completion-probe divergence). |
+| `MLX_BUN_GRAMMAR_JUMP` | — | **off** (`==="1"`) | Jump-forward decoding for structured output (SGLang's technique via xgrammar's `findJumpForwardString`): when the grammar forces a unique continuation (JSON punctuation/keys; longest under `any_whitespace:false` fixed formatting), the SERIAL lane emits its retokenized ids with ONE multi-token forward instead of one masked forward per token. Lossless in STRING space and always grammar-valid, but the token stream (and, conditioning on it, content after a forced span) can legally differ from an unjumped run — no oracle, hence opt-in. Excluded automatically for `logprobs` requests; SentencePiece-family tokenizers whose raw encode can't reproduce a mid-stream span simply never jump (guard degrades to normal decode). The batch lane's per-row grammar (`#stepGrammar`) does not jump yet. Tests: tests/grammar-jump.test.ts (contract + Llama-3.2-1B e2e). |
+| `MLX_BUN_SSD_WRITEBEHIND` | — | on (`!=="0"`) | `--ssd-cache`'s debounced write-behind snapshot (restart survival). `=0` disables it entirely — the paired-A/B lever + kill switch for the 2026-07-07 idle-gating fix (eviction/demotion spills still write). The flush is **idle-gated**: every per-tensor step (blocking GPU sync + `writeSync`) waits for the engine to go idle and pauses when a request arrives mid-flush, so durability work never taxes an active decode (pre-fix: a ~16k entry's flush overlapping cached ctx repeats cost e4b ~9% decode@ctx vs mlx-lm). The snapshot timer also never holds batch admission — while rows are active it re-arms instead of grabbing the exclusive. |
+| `MLX_BUN_SSD_SPILL_QUEUE_GB` | — | 2 | Byte cap on the write-behind queue's PENDING spill clones (they pin the evicted entries' GPU memory until the idle-gated flush runs). Over cap, the oldest queued spill drops — its clones free immediately and the entry becomes a future cache miss (never a wrong result). `=0` keeps only the newest + in-flight clone pinned (the soft cap never drops the item just enqueued). `/stats.ssd_cache` reports `pending_spills` / `pending_spill_bytes` / `dropped_spills`. Note: there is deliberately **no shutdown flush** — exiting under traffic loses queued write-behind entries (best-effort tier; cost = one re-prefill). |
+| `MLX_BUN_BATCH_NO_PIPELINE` | — | off (`==="1"`) | Disables the batch lane's pipelined token readback (the A/B lever for the shipped pipelining, like its peers `MLX_BUN_BATCH_EXTEND` / `MLX_BUN_BATCH_SSM` / `MLX_BUN_BATCH_VEC_SAMPLE`). Numerically equivalent, slower steps. |
+| `MLX_BUN_DSPARK_MINCONF` | — | checkpoint-driven | Overrides the DSpark/DeepSpec draft scheduler's minimum-confidence threshold (Alg-1 draft-length pruning). Spec lane only; no CLI flag — a calibration/A/B lever (see [dspark-speculative-decoding.md](../design/dspark-speculative-decoding.md)). |
+| `MLX_BUN_EXPERT_OFFLOAD` | `--expert-offload` | off (`==="1"`) | Env mirror of the flag for direct/library runs (scripts that never parse serve flags). |
+
+(`MLX_BUN_PERF_KERNEL`, `MLX_BUN_FUSED_GELU`, `MLX_BUN_FUSED_DECODE`,
+`MLX_BUN_FUSED_SWIGLU*`, and `MLX_BUN_CPM5_FAITHFUL` were deleted
+2026-07-05 with their kernels — see
+[unified-engine-frontier-plan.md](../design/unified-engine-frontier-plan.md);
+exporting them now does nothing.)
 
 ## Execution modes: serial vs. `--batch N`
 
-`--batch 1` (default) is the serialized single-queue path: one GPU, one
-generation at a time, prompt-cache prefix reuse, mixed-precision KV by
-default. Untouched and unconditional.
-
-`--batch N` (N>1) opts the **whole server** into a continuous-batching
-engine that is **bit-parity with `mlx_lm.server` at `B=N`**. It is a
-*mode switch*, not a load-dependent fallback (results must not depend on
-concurrency).
-
-Parity is the guarantee, so the batched lane runs in **compat mode**: it
-exposes **none** of the serial lane's perf flags and runs the plain
-bit-exact forward — the same path mlx-lm runs, never the optional
-parity-breaking kernels. This is the mechanism of the guarantee, not a
-missing feature (see
-[compat mode](#--batch-n-is-compat-mode--perf-flags-dont-apply-by-design)).
-Because mlx-lm's batched path is bf16 (its quantized batching is NYI),
-bf16 continuous batching **is** the drop-in:
-
-- **KV quant unset ⇒ bf16** so the batch path engages out of the box
-  ("Option B"). The serial default stays mixed-precision (optiq parity).
-- **Explicit `--kv-quant config|4|8` ⇒** those requests route to the
-  serial lane (batched is bf16-only; a startup warning is printed). With
-  an explicit `--kv-quant`, *every* request carries a quant scheme, so
-  **nothing batches** — `--batch N --kv-quant config` is effectively
-  serial-with-quant. Omit `--kv-quant` to actually batch.
+Batching is **concurrency-driven** (default cap 8, flipped 2026-07-05): a
+lone request runs the exact serial engine — its caches are ADOPTED
+serial-class objects, compiled decode replays, the prompt cache and SSD
+tier serve it — and only a second concurrent request causes a batch
+layout to exist. "How many requests you send" is the batching decision;
+the flag is just the cap.
+
+`--batch 1` pins the strict serialized single-queue path: one generation
+at a time, arrival-independent numerics (a request's bits never depend on
+what else was in flight). Pin it for golden regeneration and
+reproducibility work. Note the trade is real but small: batched rows
+carry bf16 left-pad reduction-order noise vs their solo runs (calibrated
+per model in the gated suites) — a request that never shares a step is
+bit-identical either way.
+
+The batched engine is **bit-parity with `mlx_lm.server` at `B=N`** per
+row for bf16, and per-row oracle-gated for the quantized compositions
+(Phase 3.1 + milestone 2 — compositions mlx-lm does not ship):
+
+- **KV quant unset ⇒ bf16** — the batch path engages out of the box
+  (bf16 is the serial default too, since 2026-07-05).
+- **`--kv-quant config`** (per-layer mixed precision) **batches** on
+  every shipped model — full-attention layers since Phase 3.1 (cpm5),
+  rotating layers since milestone 2 (gemma) — applied per row, gated
+  bit-exact/KL-0 for unpadded rows vs the serial composition.
+- **`--kv-quant 4|8`** (uniform bits) routes those requests to the
+  serial lane (quantizedKvStart threshold semantics; a startup warning
+  is printed).
 
 ### The lane picker (`GenerationGateway.willBatch`)
 
@@ -140,25 +169,27 @@ concurrently with each other.
 | --- | --- |
 | vision (image parts) | ❌ serial — needs offset-0 single-seq prefill + bidirectional image mask |
 | LoRA `adapter` (resolves to ≥1) | ❌ serial — `loraState.active` is one per-generation field; per-row adapters unsupported |
-| `repetition_penalty` | ❌ serial — per-row logits processors are a later refinement |
-| `min_p` / `xtc_*` / `logit_bias` / `presence_penalty` / `frequency_penalty` | ❌ serial — one gate for the whole sampler/processor family (safe v1; min_p/XTC are per-row samplers and could batch later) |
+| `logprobs` / `top_logprobs` | ❌ serial — the batched sampler doesn't capture logprob arrays yet |
 | explicit `seed` | ❌ serial — reproducibility ⇒ solo (matches mlx-lm) |
-| KV quant active (explicit `--kv-quant`) | ❌ serial — batched is bf16-only in v1 |
-| hybrid-cache model (Qwen3.5 gated-DeltaNet) | ❌ serial — model-level cache-capability gate (see below) |
+| KV quant active (explicit `--kv-quant`) | ✅ batches for per-layer `config` schemes — full-attention layers (Phase 3.1) AND rotating layers (milestone 2: gemma's kv_config; unpadded row gated KL-0 vs serial); uniform bits → serial |
+| `--kv-quant turbo` (TurboQuant) | ❌ serial, unconditionally — solo-only in v1 (novel `Cache` class, no batched merge/filter/temporalView; belt-and-braces refusal in `willBatch` on top of the automatic capability-gate exclusion) |
+| `--draft-model` mounted | ❌ serial, server-wide — speculation is a B=1 latency mode (upstream `is_batchable = draft is None`) |
+| `repetition_penalty` / `min_p` / `xtc_*` / `logit_bias` / presence+frequency penalties | ✅ batches — per-row logits processors over a per-row device-side history (since 2026-07-02; some models — Qwen3.5 — ship a *default* repetition penalty, which used to route everything serial) |
+| structured output (`response_format` / `guided_*`) | ✅ batches — per-row grammar matchers driven by the scheduler (`MLX_BUN_GRAMMAR_BATCH=0` forces serial) |
 | `temperature` / `top_p` / `top_k` | ✅ batches (each row samples with its own seed) |
 | `stop` sequences | ✅ batches (per-row `StopMatcher` in the onToken closure) |
 | `tools` / `tool_choice` | ✅ batches (per-row tool router; decode-layer parse) |
 | `--thinking` / `enable_thinking` | ✅ batches (template-render concern, lane-independent) |
-| multi-turn / long prompt | ✅ batches, but **no prompt-cache reuse** (`cached_tokens=0`) |
+| multi-turn / long prompt | ✅ batches **with prompt-cache reuse** (Phase 3.2): a joiner restores the longest usable cached prefix and prefills only the suffix (`cached_tokens` reported); a request that finishes without ever sharing the batch puts its caches back. **Prefix sharing (2026-07-05): serves are non-consuming clones**, so concurrent agents with a shared system prompt all reuse one prefill and nobody's entry is destroyed. A row that merges with others is not re-put (its entry ages out) — heavy concurrency reduces hit rate, never correctness |
 
-Full-attention (CPM) and sliding-window (Gemma) models batch — the
-scheduler assembles each layer's cache by attention type. Hybrid
-gated-DeltaNet models (Qwen3.5) **route serial**: their `SSMCache` has no
-dynamic-B merge/filter, so the gateway's cache-capability gate (the
-mirror of mlx-lm server.py's all-caches-have-`merge` check) sends every
-request down the serial lane with the standard drain behavior. A batched
-SSM path is planned (mlx-lm's `ArraysCache` is the oracle) — see
-[batching-v2-plan.md](../design/batching-v2-plan.md) item (h).
+**Which models batch:** full-attention (CPM), sliding-window (Gemma),
+hybrid gated-DeltaNet (Qwen3.5 — the SSM batched path, token-exact vs the
+mlx-lm B=2 oracle, landed 2026-07-02; `MLX_BUN_BATCH_SSM=0` reverts it to
+serial routing), and **plain full-attention Tier-0 universal archs**
+(Llama etc. — per-row RoPE gated token-exact vs mlx-lm B=2, 2026-07-03).
+Still serial by the model-level capability gate: gemma2-family and
+sliding-window *universal* archs (unvalidated cells) and DiffusionGemma
+(non-autoregressive).
 
 A non-batchable request **drains** the batch: while it waits, the
 scheduler stops admitting new rows, finishes the running ones, and
@@ -181,19 +212,23 @@ inside it per the table above).
 | Option | serial (`--batch 1`) | `--batch N` (N>1) |
 | --- | --- | --- |
 | `--kv-quant config`/`4`/`8` | ✅ applied to all requests | ⚠️ applied, but forces **all** requests to the serial lane (no batching) |
+| `--kv-quant turbo[:k<bits>v<bits>]` | ✅ applied to all requests | ⚠️ applied, but forces **all** requests to the serial lane (no batching) |
 | `--kv-quant off` | ✅ bf16 | ✅ bf16 (same as the implicit batch default) |
-| *(kv-quant unset)* | mixed-precision `config` | **bf16** (Option B) — incl. serial-lane fallback requests |
+| *(kv-quant unset)* | bf16 (the L1 default) | **bf16** (Option B) — incl. serial-lane fallback requests |
 | `--memory-budget` | ✅ per-request admission | ✅ per-request admission — but **not aggregate** across rows (see limitations) |
-| `--prompt-cache` | ✅ prefix reuse | ⚠️ bypassed for batched requests (`cached_tokens=0`); serial-lane requests still reuse |
+| `--prompt-cache` | ✅ prefix reuse | ✅ prefix reuse on BOTH lanes (Phase 3.2): batch joiners `take()` at admission; lone never-merged rows `put()` back on finish (merged rows' entries age out) |
 | `--temperature`/`--top-p`/`--top-k` | ✅ | ✅ (per-row) |
 | `--thinking` | ✅ | ✅ |
 | vision request | ✅ | ✅ via serial lane (in bf16 under Option B) |
 | LoRA `adapter` | ✅ | ✅ via serial lane |
-| `repetition_penalty` | ✅ | ✅ via serial lane |
-| `min_p` / `xtc_*` / `logit_bias` / presence+frequency penalties | ✅ | ✅ via serial lane |
+| `repetition_penalty` | ✅ | ✅ (batches — per-row logits processors, since 2026-07-02) |
+| `min_p` / `xtc_*` / `logit_bias` / presence+frequency penalties | ✅ | ✅ (batches — same per-row processors) |
 | `seed` | ✅ | ✅ via serial lane |
 | `tools` / `stop` | ✅ | ✅ (batches) |
-| `--compiled-decode`/`--perf-kernel`/`--fused-*`/`--force-wire` | ✅ (serial perf tree) | **n/a — compat mode, no perf flags by design** |
+| structured output (`response_format`/`guided_*`) | ✅ (mask in the decode loop) | ✅ (batches; per-row matchers) |
+| `--draft-model` | ✅ spec decode (grammar composes) | ⚠️ mounts, but routes **every** request serial — spec and batching are different modes |
+| `--compiled-decode` | ✅ (serial decode route) | ✅ at **B=1 only** (Phase 3.2): a lone request's adopted serial-class caches replay the same compiled step, same kill switch; B>1 steps run the plain graph |
+| `--fused-sdpa`/`--force-wire` | ✅ (serial decode route) | **n/a — compat mode, no perf flags by design** |
 
 ### `--batch N` is compat mode — perf flags don't apply by design
 
@@ -206,77 +241,171 @@ model through `forwardHidden`/`logitsFromHidden` directly (not
 optional, parity-breaking ones. Flagging the batched lane would defeat
 the guarantee, so it's intentionally not wired.
 
-- **`--perf-kernel` / `--fused-decode` / `--compiled-decode`** — never
-  engage in the batched lane. They diverge from the -O0 reference (or, for
-  the quantized-KV kernels, are moot since batched is bf16). They're the
-  L3 perf row, a separate KL-gated path — see the validation matrix in
-  [parallel-slots.md](../design/parallel-slots.md).
+- **`--compiled-decode`** — engages at **B=1 only** (Phase 3.2,
+  adopt-don't-copy): a lone request's caches stay serial-class, so the
+  scheduler replays the serial engine's compiled step — bit-exact by the
+  same gate, `MLX_BUN_COMPILED_DECODE=0` is the same kill switch. The
+  moment a second row joins, steps run the plain batched graph.
+- **`--fused-sdpa`** — never engages in the batched lane (the fused
+  quantized-KV SDPA is an L2 serial-lane composition). See the
+  validation matrix in [parallel-slots.md](../design/parallel-slots.md).
 - **`--force-wire`** — doesn't wire (the scheduler bypasses `generate()`'s
   wired scope). A model that needs wiring for speed (e.g. 26B: 8.6 → 32.3
   tok/s wired) runs at mlx-lm-equivalent unwired speed under `--batch N`;
   in practice such a model has no headroom for B>1 KV anyway.
-- **Always-on bit-exact kernels still run.** Fused GeGLU is bit-exact with
-  the spelled-out MLP, so it stays on in both lanes without breaking
+- **Always-on bit-exact kernels still run.** The compiled activations
+  (mlx-lm's own `@mx.compile` geglu/swiglu) are bit-exact with the
+  spelled-out MLP, so they stay on in both lanes without breaking
   parity — "compat mode" means *no parity-breaking optionality*, not
   -O0.
 
-The serial lane is where the perf/optimization tree lives: `--perf-kernel`
-/ `--fused-*` / `--compiled-decode` engage there, and the
-mixed-precision-KV (optiq) default applies there.
+The serial lane is where the decode route lives: `--compiled-decode` /
+`--fused-sdpa` / `--kv-quant` (and the `--l2` optiq composition) engage
+there.
 
 ## Known limitations under `--batch N`
 
 These are deliberate v1 scope, not bugs — but they change behavior, so
 know them:
 
-1. **Prompt cache bypassed.** Batched requests solo-prefill every row;
-   `cached_tokens=0`. Wiring `PromptCache` into the scheduler is a
-   follow-up.
-2. **Admission is per-request, not aggregate.** `--memory-budget` checks
-   each request against single-sequence max-safe-context, but N
-   concurrent rows can collectively exceed the budget (the `B×S_max`
-   KV-budget admission is TODO). With a tight budget and several
-   large-context requests, the allocator cap is the only backstop — and
-   a true GPU OOM here is uncatchable. Size the budget with headroom for
-   N rows, or keep N small.
+1. **Prompt cache on the SPEC path is bypassed.** A `--draft-model`
+   server re-prefills every request; the target+draft cache-entry
+   composition is designed in mlx-lm-tool-parity-plan §7.6, not built.
+   (Batched rows DO reuse the prompt cache since Phase 3.2 — see the
+   lane table above and `/stats` `submitted_rows`.)
+2. **Aggregate admission is opt-in.** `--memory-budget` checks each
+   request against single-sequence max-safe-context; the AGGREGATE cap
+   across N concurrent rows is `--kv-budget <GB>` (landed 2026-07-03):
+   over-budget joiners queue until rows finish, a request over the budget
+   alone rejects. Without `--kv-budget`, N large-context rows can still
+   collectively exceed memory (uncatchable GPU OOM) — set it when running
+   `--batch N` near the machine's limit.
 3. **Short-context only.** Verified pre-ring-wrap (rows < the 1024
    sliding window). Long-context (context > window) batched decode is a
    separate validation.
-4. **bf16 only — by contract.** mlx-lm's batched path *is* bf16, so
-   bf16-only batching is exactly what mlx-lm-parity means (not a
+4. **bf16 by default — by contract; mixed-KV batching beyond it.**
+   mlx-lm's batched path *is* bf16, so bf16 batching is exactly what
+   mlx-lm-parity means. Phase 3.1 adds batched per-layer MIXED KV for
+   all-full-attention configs — a beyond-mlx-lm composition, verified
+   per row against the optiq oracle instead (not a
    shortcoming). Going further — batched + mixed-precision KV quant — is
    novel territory with no mlx-lm/optiq oracle, so it's a deferred,
    KL-gated extension.
-5. **`extend` join not yet used.** A joining request re-merges the whole
-   batch (O(B·S)); mlx-lm's keep-the-running-batch `extend` is a later
-   optimization (numerically equivalent).
+5. **`extend` join (landed 2026-07-03).** A joining request appends to
+   the running batch's full-attention KV in one pad+concat (mlx-lm
+   `BatchKVCache.extend` semantics, token-exact vs its oracle);
+   `MLX_BUN_BATCH_EXTEND=0` reverts to the old whole-batch re-merge
+   (numerically equivalent, O(B·S)). Sliding-window layers still re-merge
+   on join (rotating-extend is a follow-up).
+
+## Fidelity tiers and the decode route (`--l1` / `--l2`)
+
+The tiers are **correctness contracts**, and each flag is an alias for a
+decode-route preset (`applyDecodeRoute()` in [src/cli.ts](../../src/cli.ts));
+any per-fork flag (`--kv-quant`, `--compiled-decode`,
+`--compiled-activations`, `--fused-sdpa`)
+overrides its tier's preset.
+
+| Tier | Contract | KV | Kernels | Verified against |
+| --- | --- | --- | --- | --- |
+| `--l1` | mlx-lm **bit-for-bit** | bf16 | compiled-decode + compiled activations (faithful geglu/swiglu) | mlx-lm goldens (per machine) |
+| `--l2` | mlx-optiq **bit-for-bit** | mixed-precision (`kv_config.json`) | + fused-SDPA (matches optiq exactly) | optiq goldens |
+| *(none)* | **= `--l1`** | bf16 | compiled-decode + compiled activations | mlx-lm goldens |
+
+Compiled-decode is on in **every** tier (proven bit-exact with uncompiled —
+free speed, not a fidelity trade).
+
+**There is no `--l3` tier anymore** (removed 2026-07-05; passing `--l3`
+errors out with a pointer). The tier had no content: every kernel it
+gated lost its paired A/B against the L1 baseline and was deleted. Its
+role is replaced by the **Lab** — output-changing experiments run as env
+flags with a bench + expiry, and one graduates only by beating the L1
+baseline in a paired A/B. See
+[docs/design/unified-engine-frontier-plan.md](../design/unified-engine-frontier-plan.md).
+
+**KV precision is a separate axis from the tier's kernels.** Each tier row shows its
+*default* KV scheme, but KV is not what defines the tier. mlx-lm supports **uniform**
+quantized KV (`--kv-bits`), and `--l1 --kv-quant 8` is a **bit-exact L1** config: with
+fused-sdpa off (the L1 default) our quantized decode runs `quantizedSdpaUnfused`,
+which is op-for-op identical to mlx-lm's `quantized_scaled_dot_product_attention`
+(`mlx_lm/models/base.py`) — same `mx.quantized_matmul` ×2 + `mx.softmax(precise=True)`
++ `where(…, finfo.min)`. bf16 is just the simplest L1 default. The **fused / N-tiled**
+quantized path (fused-sdpa ON — the `--kv-quant config` / `--l2` composition) is the
+**optiq**-aligned one. Only the **per-layer mixed-precision** `config` scheme is
+optiq-only (→ L2).
+
+**Where each feature sits:** batching = L1-class (mlx-lm B=N parity);
+speculative decoding = L1-class (token-for-token vs mlx-lm's spec path);
+structured output = L2-class (oMLX oracle; masking doesn't touch the
+numerics of valid tokens, so it composes with any tier); grammar × spec
+together, HLG sampling, expert offload, and (future) batched quantized KV
+are **no-oracle (Lab-gated)** — no ancestor runtime does them, so they're
+gated by validity/KL/quality instead of bit-parity.
+
+## Performance characteristics & recipes
+
+Reference numbers are from this project's dev machines (loaded-machine
+numbers are directional only; `benchmarks/RESULTS.md` holds the quotable
+set, and `scripts/bench-feature-matrix.ts` measures the six composition
+cells in one run).
+
+- **Serial is the fastest single stream** — prompt cache, mixed-precision
+  KV, compiled decode. The batch lane at ONE live row is ~25% slower
+  (cpm5 ~149 vs 193 tok/s: no prompt cache, no kv-quant, wrapper
+  overhead), which is why serial stays the default.
+- **`--batch N` wins under concurrency** — cpm5 `--batch 4`: ~349 tok/s
+  aggregate vs ~173 serial-queued, TTFT 2–3× better; Llama-3B at just
+  B=2: 1.7× aggregate, TTFT 765→162 ms.
+- **Spec pays only on slow targets.** One accepted draft = one skipped
+  target forward, so value scales with target step cost: 12B ≈ 1.09× at
+  γ=1 (measured); fast small targets LOSE (e4b 0.78×; 3B smoke: spec
+  61.5 vs serial 81.5 tok/s). Draft for 12B+, skip below.
+- **Structured output is ~free** (<1% serial; the batch lane drops to
+  read-before-build scheduling while a grammar row is live — bounded,
+  ~0.1 ms/step class).
+- **`--ssd-cache` has 0% decode overhead** — pure TTFT/restart win; no
+  reason to leave it off on a persistent server.
+- **Quantized-KV prefill pays a scheme-intrinsic tax at long context** —
+  chunked prefill converts each chunk's KV at the chunk boundary, so
+  every later chunk attends against the already-quantized prefix
+  (quantize-on-write; the same streaming conversion optiq serve uses —
+  `maybeQuantizeKv` in `src/generate.ts` is the port). That cost is by
+  design, not an implementation gap: ~30% prefill throughput vs bf16 at
+  ~16k where the config quantizes every cache (the 1B starter measured
+  −33% in the 2026-07-06 serve pass), single-digit % on the
+  sliding-window gemmas (rotating caches stay window-bounded). The lever
+  is upstream — mlx's `quantized_matmul` kernels (the split-K work,
+  [ml-explore/mlx#3120](https://github.com/ml-explore/mlx/pull/3120);
+  status in [decode-speed-program.md](../design/decode-speed-program.md)
+  lever 2) — not this codebase.
+
+**Recipes:**
+
+- *Single-user agent/chat (the default use):*
+  `mlx-bun serve <model> --ssd-cache <dir>` — L1 serial, prompt cache +
+  SSD tier. On 12B+ add `--draft-model <small-same-tokenizer>`.
+- *Several clients at once (throughput):*
+  `mlx-bun serve <model> --batch 4 --ssd-cache <dir>` — don't set
+  `--kv-quant` (it would just un-batch everything), don't mount a draft.
+- *Reproducibility:* bare / `--l1` (≡ mlx-lm), `--l1 --batch N`
+  (≡ mlx-lm B=N), `--l2` (≡ optiq).
+- *Memory-tight big model:* serial + `--kv-quant config|4|8` +
+  `--memory-budget <GB>` + `--ssd-cache <dir>`; MoE adds
+  `--expert-offload`. Inherently the serial recipe until batched
+  quantized KV lands.
+
+**The two exclusions to remember:** batching ⊕ kv-quant (batch is bf16 by
+contract; the resolver is a Lab-gated follow-up) and spec ⊕ prompt
+cache (v1 bypass; the §7.6 composition is the fix).
 
 ## Observability — `GET /stats`
 
-The live config and batch state:
-
-```jsonc
-{
-  "server":  { "owner": "serve" | "pi-session" | "embedded", "model": "...", "started_at": 0 },
-  "prompt_cache":  { "entries": 0, "bytes": 0, "max_bytes": 2000000000, "hits": 0, "misses": 0 },
-  "response_store": { "entries": 0, "bytes": 0, "max_bytes": 33554432, "ttl_ms": 3600000 },
-  "kv_quant": {
-    "mode": "mixed (kv_config.json)" | "uniform-kv8" | "bf16",
-    "layers": { "kv4": 8, "bf16": 40 },
-    "attention": { "global": 10, "sliding_window": 38 }
-  },
-  "admission": {
-    "max_safe_context": 0,          // requests above this 400
-    "memory_budget_bytes": null,    // explicit budget, or null = machine default
-    "usable_bytes": 0, "weights_bytes": 0
-  },
-  "batch": {
-    "configured": 1,                // the --batch N value
-    "batched": false,               // batching enabled (N>1) for this server
-    "active_rows": 0                // rows currently decoding in the batch
-  }
-}
-```
+The live config and batch state. The canonical field-by-field snippet
+lives in [server-api.md](server-api.md#get-stats) (kept in lockstep with
+the code — this section only explains how to read it); highlights:
+`prompt_cache` (8 GB default cap), `kv_quant.mode` (incl. `turbo kXvY`),
+the conditional `ssd_cache` block, and `batch`
+(`active_rows`/`pending_rows`/`submitted_rows`/`kv_bytes`/`kv_budget_bytes`).
 
 `batch.batched` reflects only whether `--batch N` (N>1) is configured;
 with an explicit `--kv-quant` it can read `true` while `active_rows`