mlx-bun 0.0.13 → 0.3.0

package/docs/design/unified-engine-frontier-plan.md

@@ -1,178 +1,141 @@
-# Unified engine + the frontier program — the reorganization plan
-
-Status: **PLAN** (2026-07-05, from the flag-architecture discussion with Josh
-after the naked-=-L1 decision). This is the design that makes the library
-"make sense": one engine, one honest flag surface, per-scheme oracles, and a
-Lab where optimization experiments live until they earn defaults.
-
-## 0. The goals (Josh, 2026-07-05 — the two-part product)
-
-1. **A drop-in replacement for mlx-lm: support EVERYTHING they do, with
-   bit-exact parity.** This is the default. Parity is only DEFINED over
-   outputs mlx-lm can produce — you cannot ask for "mlx-lm's bits under
-   mixed-precision KV" because mlx-lm cannot output anything under it.
-2. **Beyond mlx-lm — features you opt into by deliberately trading the
-   parity guarantee for something you value MORE**, each answering to its
-   own oracle (whoever already ships it):
-   - **multi-model switching** (oracle: optiq per-request switching / oMLX
-     EnginePool) — ❌ missing today; the newest backlog item (layer 4).
-   - **mixed-precision KV cache** (oracle: optiq) — ✅ serial, bit-exact;
-     ✅ **under batching for all-full-attention configs (LANDED 2026-07-05
-     — Phase 3.1 P1: cpm5 end-to-end, per-row bit-exact vs the optiq
-     composition; no other stack ships this)**; rotating-layer configs
-     (gemma) = milestone 2.
-   - **prefix sharing** (oracle: vLLM/SGLang) — ⚠️ whole-prefix reuse only;
-     true cross-request sharing (one system prompt's KV, N rows) missing.
-   - **prompt caching** (oracle: mlx-lm's LRUPromptCache) — ✅ byte-capped
-     + SSD cold tier beyond anyone; ❌ batched rows → Phase 3.2.
-   - **structured outputs** (oracle: oMLX/xgrammar) — ✅ serial + batched,
-     byte-identical vs oMLX; composes with spec decode.
-
-Supersedes: batching-v2-plan.md's standing decision that `--batch N` is a
-mode switch (reversal documented in §4, with the new rationale — the old
-rejection's determinism argument is answered, not ignored). Builds on:
-faithful-l1-consolidation.md (the L1 baseline, now the naked default),
-parity-tier-dag.md (per-scheme oracles), omlx-adoption-map.md (survey),
-batching-perf-path.md (the B=1 gap numbers this plan must close).
-
+---
+status: active
+axis: ON
+canonical-for: engine-architecture
+plan-anchor: "Interface-based engine refactor `[~]`"
+last-verified: 2026-08-23
 ---
 
-## 1. North star: beyond the Pareto frontier
+# Engine architecture — four layers, per-scheme oracles, one request path
+
+This is the canonical architecture doc for mlx-bun's engine. It owns:
+
+- the **four layers** (L1 graph / L2 KV+quant schemes / L3 engine / L4 surface)
+  and the rule that features decompose along them;
+- the **fidelity tiers** — L1 = mlx-lm bit-exact, L2 = mlx-optiq bit-exact,
+  Lab = no-oracle experiments gated by KL/eval and a paired A/B before any
+  default — and the decision procedure that places every optimization;
+- the **naked default IS L1** decision (2026-07-05);
+- the **engine** decision that concurrency is the batch size;
+- the **request path** as it exists in `src/` today, the single-path rule, and
+  the known deviations from it;
+- the **flag end-state** and how it maps onto the layers.
+
+The v2 refactor is in [§12](#12-interface-based-engine-refactor).
+Josh requested replaceable implementations behind interfaces on 2026-09-04.
+That section defines the target and migration, with implementation status below.
+The portable session and native implementation ports are documented in the
+library API; existing model-class exports remain supported.
+Earlier sections retain the current architecture and numerical contracts.
+`last-verified` above dates the earlier inventory, not a whole-document reaudit.
+
+Folded in (2026-08-23): `docs/design/unified-engine-frontier-plan.md` (tier framing, two axes, flag
+classification) and `docs/archive/investigations/faithful-l1-consolidation.md` (the 2026-07-04 faithful→L1
+consolidation and the 2026-07-05 deletion pass, now the History section).
+Status and changelog prose lives in PLAN.md — see "Decision: naked default =
+--l1; levers must beat the baseline (2026-07-05, Josh)" for the decision
+record and "Serving architecture consolidation" for the live S0–S4 work. Every
+`src/` claim below was re-read against the tree on 2026-08-23; where an older
+draft of this doc disagreed with the code, the code wins and the note says so.
+
+Companions: `docs/design/batching.md` (B=1 gap numbers), `docs/design/batching.md` and
+`docs/design/batching.md` (scheduler mechanics — the mode-switch decision there is
+superseded by §5), `docs/design/unified-engine-frontier-plan.md` (the S0–S3 seam),
+`decode-speed-program.md` (the only doc that ranks speed levers; it also
+carries the oMLX port ledger that used to be `docs/design/decode-speed-program.md`).
 
-The claim we are building toward: **"the best way to run local AI on a Mac."**
-Formally: every (model, workload) point we ship should sit on or beyond the
-current local-inference Pareto frontier in this space:
+---
 
-| axis | direction | measured by |
-|---|---|---|
-| memory | less is better | generation peak (Metal), fit-model predictions |
-| decode tok/s | more | bench-h2h (stable cells only) |
-| prefill / TTFT | faster | bench-h2h, server TTFT |
-| throughput | higher | batched aggregate tok/s (bench-serving-load) |
-| intelligence | higher | frozen eval suite (MMLU/GSM8K/IFEval/HumanEval/BFCL/HashHop, evals.sqlite) |
-
-**THE CONTRACT (Josh, 2026-07-05):** two rules, and they compose:
-
-1. **The drop-in promise: we must produce the SAME BITS as mlx-lm** — the
-   original oracle. How the bits are produced (storage, kernels,
-   scheduling, engine, language) is the space we compete in.
-2. **The oracle for ANYTHING is whoever already does that thing.** Want
-   mlx-lm's decode → mlx-lm is the oracle. Want optiq's mixed KV → optiq.
-   Want vLLM's paged attention → **vLLM is the oracle**: read their
-   implementation, copy what they did, verify against them, then optimize.
-   This is the standing house method (FaithfulMiniCPM5 = copy mlx-lm;
-   mixed KV = copy optiq; batching = copy BatchGenerator; SSD tier = copy
-   oMLX). Never frame a capability someone ships as oracle-less. "No
-   oracle" means NOBODY does the thing — genuine research — and that, and
-   only that, is what the Lab is for.
-
-Verification form follows the oracle's framework: same-framework oracles
-(mlx-lm, optiq — same libmlx) admit bit-exact gates; cross-framework
-oracles (vLLM — CUDA/PyTorch) anchor the ALGORITHM and behavior (block
-semantics, sharing correctness, utilization), while rule 1 still binds the
-final output bits in the default regime. "Faithful" kernel-set identity is
-the easiest route to same-bits, not the contract; the one constraint
-bit-exactness puts on implementations is math/reduction order (the perf
-kernel died exactly there).
+## 1. The two-part product and the contract
+
+1. **A drop-in replacement for mlx-lm: everything they do, bit-exact.** This is
+   the default. Parity is only *defined* over outputs mlx-lm can produce — you
+   cannot ask for "mlx-lm's bits under mixed-precision KV" because mlx-lm
+   cannot output anything under it.
+2. **Beyond mlx-lm — features you opt into by deliberately trading the parity
+   guarantee for something you value more**, each answering to its own oracle:
+   mixed-precision KV (optiq), structured output (oMLX/xgrammar), speculative
+   decoding (mlx-lm's `speculative_generate_step`; optiq for the KV-borrowing
+   assistant drafter), prompt caching (mlx-lm's `LRUPromptCache`), prefix
+   sharing and paged KV (vLLM/SGLang), multi-model switching (optiq
+   per-request switching / oMLX EnginePool).
+
+**THE CONTRACT (Josh, 2026-07-05) — two rules that compose:**
+
+1. **The drop-in promise: the same BITS as mlx-lm.** How the bits are produced
+   (storage, kernels, scheduling, engine, language) is the space we compete in.
+2. **The oracle for anything is whoever already does that thing.** Want
+   mlx-lm's decode → mlx-lm. optiq's mixed KV → optiq. vLLM's paged attention
+   → vLLM: read their implementation, copy what they did, verify against them,
+   then optimize. "No oracle" means *nobody* does the thing — genuine research
+   — and that, and only that, is what the Lab is for.
+
+Verification form follows the oracle's framework: same-libmlx oracles (mlx-lm,
+optiq) admit bit-exact gates; cross-framework oracles (vLLM, CUDA/PyTorch)
+anchor the algorithm and behavior while rule 1 still binds the output bits in
+the default regime. "Faithful" kernel-set identity is the easiest route to
+same-bits, not the contract; the one constraint bit-exactness puts on an
+implementation is math/reduction order.
 
 Two consequences drive everything below:
 
-- **The baseline must be even and proven before optimizing.** We now have it:
-  the L1 faithful path is bit-exact with mlx-lm at 1.00× speed on every
-  model. Anything that claims to beat it proves it with a paired A/B on a
-  stable pass — otherwise it is noise or a regression (2026-07-05 evidence:
-  every output-changing lever failed this bar).
-- **Frontier-shifting work is usually a *quality-per-byte* play, not a raw
-  kernel play.** Mixed-precision quantization (weights AND KV) moves memory
-  down while holding or raising intelligence — a genuine beyond-the-frontier
-  move, unlike micro-kernel wins that trade along it. This is why quantized
-  KV stays first-class even though it lost the speed A/B: it buys context
-  headroom (12B@16k: −1.3 GB, growing linearly) and mixed schemes buy
-  intelligence-per-byte.
-
-## 2. Survey — what the other engines got right (and wrong for us)
-
-- **mlx-lm** (`server.py`/`generate.py`): the original oracle and THE
-  drop-in target — and the project most similar to ours: it spans layers
-  1+2+3(+4), shipping the reference graphs, native bf16 + uniform 4/8-bit
-  quantized KV (`QuantizedKVCache`/`--kv-bits` — layer 2 is theirs too),
-  the batched engine, and the server. Its server is **concurrency-driven
-  by default** — it auto-creates a `BatchGenerator` (decode-concurrency
-  cap 32, prefill concurrency, chunked `prefill_step_size`) and routes any
-  batchable request into it; only non-batchable requests (draft models
-  etc.) serve sequentially. There is no "batching mode" flag — the cap is
-  the only knob. Count-capped prompt cache (the OOM footgun we already
-  fixed by byte-capping); no quantization on the batched path.
-  → **Adopt: the concurrency-driven default is our drop-in-parity behavior.**
-- **vLLM**: continuous batching + PagedAttention — the KV cache is block-paged
-  with a block table per sequence, which is what makes batching, long context,
-  and prefix sharing *compose* instead of conflict. No batch flag; `max_num_seqs`
-  caps. Chunked prefill interleaves with decode so TTFT doesn't starve running
-  streams. → **Adopt (long-term): block-paged KV as the structure that unifies
-  batch + quantized KV + prompt cache + SSD tier. Adopt (near-term): the
-  no-mode-flag scheduler and prefill/decode interleaving policy.**
-- **llama.cpp** (server): slot-based (`-np` = slot count), one shared KV
-  buffer partitioned by sequence — concurrency-driven, no mode flag. KV cache
-  default is **f16; quantized KV (q8_0/q4_0) is opt-in** — independent
-  confirmation of our bf16-default decision. Its imatrix (importance-matrix)
-  quantization is the ecosystem's calibration-driven intelligence-per-byte
-  play — the same family as oQ and our knapsack `--target-bpw`.
-  → **Adopt: nothing structural we don't have; validates KV defaults; imatrix
-  is prior art for the §7 quant program.**
-- **oMLX**: scheduler on mlx-lm's BatchGenerator + product appliances. Already
-  systematically mapped (omlx-adoption-map.md): batching parity + SSD tier
-  ported and won; burst decode ported and REFUTED (Python-GIL medicine, our
-  disease doesn't exist); oQ quant + DFlash queued. → **Keep mining the queue;
-  its engine architecture is not ahead of ours.**
-- **mlx-optiq**: the L2 oracle — per-layer mixed KV (`kv_config.json`), fused
-  quantized SDPA, vision sidecar. Not a batching engine. → **Its role is the
-  oracle for mixed-precision schemes (§5) and the source of the next quant
-  ideas (TurboQuant per Josh).**
-
-## 3. The architecture (four layers, each with one job)
+- **The baseline must be even and proven before optimizing.** The L1 path is
+  bit-exact with mlx-lm and at decode parity on every model (PLAN.md, the
+  2026-07-05 decision). Anything claiming to beat it proves it with a paired
+  A/B on a stable pass; otherwise it is noise or a regression.
+- **Frontier-shifting work is usually a quality-per-byte play, not a raw kernel
+  play.** Mixed-precision quantization (weights and KV) moves memory down while
+  holding or raising intelligence. Quantized KV stays first-class even though
+  it lost the speed A/B: it buys context headroom.
+
+The frontier axes we measure: memory (generation peak, fit-model), decode
+tok/s (paired h2h, stable cells), prefill/TTFT, batched aggregate throughput,
+and intelligence (the frozen eval suite in evals.sqlite). Curated numbers live
+in `docs/reference/benchmarks.md`.
+
+## 2. The four layers
 
 ```mermaid
 flowchart TB
     subgraph clients["CLIENTS"]
         direction LR
         PI["Pi agent<br/>chat UI + terminal"]
-        CLI["mlx-bun CLI<br/>generate · eval · train"]
-        API["any OpenAI client"]
+        CLI["mlx-bun CLI"]
+        API["any OpenAI / Anthropic / Responses client"]
         LIB["TS/Bun library import"]
     end
-    subgraph surface["SERVING SURFACE — src/server.ts"]
-        V1["OpenAI /v1 API"]
-        ADMIT["admission control<br/>--memory-budget · --kv-budget"]
+    subgraph surface["L4 SERVING SURFACE — src/server.ts + src/serve/*"]
+        V1["/v1 endpoints → prepareCompletion → CompletionExecutor"]
+        ADMIT["admission: --memory-budget · --kv-budget"]
     end
-    subgraph engine["ENGINE — one loop (Phase 2: concurrency IS the batch size)"]
-        SCHED["continuous-batching scheduler<br/>--batch N = cap (default 32 after B=1 gate)"]
-        B1["B=1 fast path<br/>pipelined + compiled step (today's serial loop)"]
-        BN["B>=2 batched step<br/>extend-join · bit-exact vs mlx-lm at same composition"]
-        FEAT["shared, implemented ONCE: prompt cache ·<br/>SSD KV tier · spec decode · grammar"]
+    subgraph engine["L3 ENGINE — GenerationGateway.place() → serial | continuous"]
+        SCHED["BatchScheduler (--batch N cap, default 8)<br/>B=1 adopted-cache fast path · B≥2 step"]
+        SERIAL["strict serial executor (generate.ts)<br/>--batch 1 and not-yet-batchable compositions"]
+        FEAT["shared: prompt cache · SSD tier · grammar · samplers"]
     end
-    subgraph models["MODEL LAYER — fingerprint-routed, NEVER flag-routed"]
-        OPT["optimized-verified graphs<br/>gemma4 12B/e4b/26B · MiniCPM5 · Qwen3/3.5"]
-        UNI["universal graph — any HF model, best effort"]
+    subgraph models["L1 MODEL GRAPH — profile-routed, never flag-routed"]
+        OPT["dedicated / generated graphs<br/>gemma4 · MiniCPM5 · Qwen3/3.5/3.8 · Qwen3-MoE · GLM-5.2"]
+        UNI["universal dense graph — any supported HF arch, best effort"]
     end
-    subgraph numerics["NUMERICS / KV SCHEMES — per-scheme oracles"]
+    subgraph numerics["L2 KV / QUANT SCHEMES — per-scheme oracles"]
         BF["bf16 (default) — oracle: mlx-lm"]
         UKV["uniform kv 4/8 — oracle: mlx-lm --kv-bits"]
         MKV["mixed kv_config — oracle: mlx-optiq"]
-        LAB["LAB: TurboQuant · mixed-bpw weights ·<br/>new kernels — KL + eval gates"]
+        LAB["Lab: TurboQuant KV · rotation-folded weights · new kernels — KL + eval gates"]
     end
     HW["libmlx (Metal) via bun:ffi — the only runtime link"]
     PI --> V1
     CLI --> V1
     API --> V1
-    LIB --> SCHED
+    LIB --> SERIAL
     V1 --> ADMIT --> SCHED
-    SCHED -->|1 active| B1
-    SCHED -->|2+ active| BN
+    ADMIT --> SERIAL
     SCHED -.-> FEAT
-    B1 --> OPT
-    BN --> OPT
-    B1 --> UNI
-    BN --> UNI
+    SERIAL -.-> FEAT
+    SCHED --> OPT
+    SERIAL --> OPT
+    SCHED --> UNI
+    SERIAL --> UNI
     OPT --> BF
     OPT --> UKV
     OPT --> MKV
@@ -183,592 +146,1733 @@ flowchart TB
     MKV --> HW
 ```
 
-How a model becomes servable, and the promotion path:
-
-```mermaid
-flowchart LR
-    Q["mlx-bun serve QUERY"] --> REG["registry (local HF cache)"]
-    REG -->|not downloaded| GET["mlx-bun get"]
-    GET --> REG
-    REG --> FP["config.json fingerprint"]
-    FP -->|optimized-verified| OPTF["custom graph for THAT model"]
-    FP -->|anything else| UNIF["universal graph"]
-```
-
-```mermaid
-flowchart LR
-    A["runs on universal"] --> B["custom per-model graph<br/>(port the oracle's exact ops)"]
-    B --> C["bit-exact goldens vs scheme oracle"]
-    C --> D["h2h bench >= 1.00x mlx-lm, stable pass"]
-    D --> E["OPTIMIZED-VERIFIED badge"]
-    E -.-> F["Lab experiments on top<br/>paired A/B + expiry"]
-```
-
-**The model-layer philosophy (Josh's, verbatim intent):** the universal layer
-is the promise that anything from Hugging Face *runs*; the optimized-verified
-layer is the promise that the models we care about run *better than anywhere
-else*. Promotion from universal → optimized is a defined checklist: custom
-graph file, bit-exact parity goldens vs the scheme oracle, h2h bench entry at
-≥1.00× vs mlx-lm, eval-suite scores recorded. "Optimized and verified" is a
-badge with teeth, and the badge list is the marketing claim.
-
-### Layer 0 — the SSD spill substrate (Josh, 2026-07-05)
-
-Every stage that tried it found the same result: **it is almost always
-faster to cache to SSD and mmap it back than to regenerate, and spilling
-lets RAM be used as a cache instead of the only tier.** Three independent
-implementations already prove the pattern: the SSD KV cold tier (restart
-TTFT 12.1 s → 0.24 s, 0% decode overhead), MoE expert offload (page-aligned
-mmap, phys footprint ≈ active params), and the byte-capped prompt cache.
-Decision: extract the pattern into ONE generic substrate — content-addressed
-keys (model fingerprint + scheme + adapter + producer version), zero-copy
-mmap restore, byte-capped LRU eviction, corruption self-quarantine — that
-layers 1/2/4 consume. Future clients: vision encoder features (oMLX
-adoption-map #6), compiled xgrammar grammars, quantized-conversion results,
-TurboQuant artifacts. `src/ssd-cache.ts` is the seed to generalize.
-
-### The batching workload (Josh, 2026-07-05)
-
-The motivating case is THE USER'S AGENT HARNESS: someone points pi / a
-Claude-Code-style coding agent at their local server and spins up
-SUB-AGENTS to work different parts of a project. Serial, each agent queues
-behind all the others — the Nth agent's first token waits for N−1 full
-generations and every agent's loop stalls. Batched, 4–8 agents decode
-simultaneously: each stream is somewhat slower, but nobody waits, every
-agent starts instantly, and the fleet's wall-clock collapses (cpm5: 345
-aggregate at B=4 vs 267 solo → four jobs finish in ~3.1×T with zero queue
-latency, vs 4×T serial with the last agent idle through three
-generations). That is the entire reason a single-person laptop runtime
-carries a continuous-batching engine. Consequences: concurrency-driven
-batching (sub-agents don't coordinate arrivals); **prefix sharing gains
-priority** (sub-agents share long system prompts — one shared KV prefix ×
-N rows); per-slot adapters (different agents, different specializations);
-and batching × quantized KV first (agent fleets want long contexts).
-"Single-user" in this project means single-user, many-agents.
-
-### Layer responsibilities (the contract, 2026-07-05)
-
-1. **Model graph** — one forward step given hidden + caches; attention
-   dispatches on the CACHE TYPE it is handed (bf16 → ops.sdpa, quantized →
-   quantizedSdpa); shape-generic in B; selected by fingerprint, never flags.
-   *Responsibility: numerics of a step, for any B, for whatever cache it's
-   given.*
-2. **KV scheme (cache classes)** — the memory format and its semantics:
-   bf16 / uniform / mixed per-layer; conversion timing (bf16 prefill →
-   quantize populated rows — the optiq hook semantics); in batched form the
-   multi-row layout (padding, per-row rope, merge/extract/extend/filter).
-   *Responsibility: the scheme PER ROW — a row's bytes and math identical
-   solo or batched. This is where each scheme's oracle binds.*
-3. **Engine (scheduler)** — rows: admission, solo prefill, join/merge, the
+### Layer responsibilities (the contract)
+
+1. **Model graph (L1)** — one forward step given hidden + caches; attention
+   dispatches on the *cache type it is handed* (bf16 → `ops.sdpa`, quantized →
+   the quantized SDPA); shape-generic in B; selected by model profile, never by
+   flags. *Responsibility: the numerics of a step, for any B, for whatever
+   cache it is given.* The contract ENDS AT LOGITS.
+2. **KV scheme (L2, cache classes)** — the memory format and its semantics:
+   bf16 / uniform / mixed per-layer / TurboQuant; conversion timing (bf16
+   prefill → quantize populated rows, optiq's hook semantics); in batched form
+   the multi-row layout (padding, per-row rope, merge/extract/extend/filter).
+   *Responsibility: the scheme PER ROW — a row's bytes and math identical solo
+   or batched. This is where each scheme's oracle binds.* In code: `KvScheme`
+   (`src/kv-scheme.ts`, kinds `bf16 | affine-uniform | affine-config | turbo`),
+   the cache classes in `src/model/gemma4-base.ts`, and the batched variants in
+   `src/model/batched-*.ts`.
+3. **Engine (L3, scheduler)** — rows: admission, solo prefill, join/merge, the
    step loop, per-row sampling, eviction, pipelining, the concurrency cap.
-   Never implements scheme math; calls layer-2 operations.
-   *Responsibility: how many rows and when — never what a row's bytes mean.*
-4. **Serving surface** — endpoints, per-request features (adapters,
-   grammar, sampling, spec policy), prompt cache (keyed fingerprint +
-   effective scheme), admission budgets. *Responsibility: requests, not
-   tensors.*
-
-**Features decompose ALONG the layers (that's the model working, not
-leaking).** Per-request LoRA is the worked example: layer 1 = the math (a
-parameterization — `Wx + scale·B(Ax)` — oracle-gated vs mlx-lm's tuner);
-layer 3 = the policy (which rows run which adapter; today one active
-adapter per batch, per-slot adapters are the matrix item); layer 4 = the
-surface (`adapter` field, mount/hot-swap, discovery); plus one sideways
-obligation — KV under adapter A ≠ base KV, so the prompt cache keys by
-adapter (the SSD tier already does). When placing a feature, name its
-component at each layer and each component's gate.
-
-**Sampling's placement (and the debranching invariant).** The model
-graph's contract ENDS AT LOGITS. Sampling is *specified* at layer 4 (the
-request's params become a per-row sampler closure) and *executed* at
-layer 3 (each step, on each row's [1,V] slice; vectorized when all-greedy)
-— it never enters layer 1. This is the load-bearing placement behind the
-layer-1 debranching goal (Josh: the generated file pre-decides all ops and
-shapes; the path is a serial line with no branchable checks): everything
+   Never implements scheme math; calls layer-2 operations. *Responsibility: how
+   many rows and when — never what a row's bytes mean.* In code:
+   `src/serve/generation-gateway.ts` (placement + the single mutual-exclusion
+   domain), `src/serve/batch-scheduler.ts`, `src/generate.ts` (the strict
+   serial loop), `src/spec/serve-loop.ts` (the verify loop).
+4. **Serving surface (L4)** — endpoints, per-request features (adapters,
+   grammar, sampling, spec policy), prompt cache (keyed by fingerprint +
+   effective scheme + adapter), admission budgets. *Responsibility: requests,
+   not tensors.* In code: `src/server.ts`, `src/serve/request-plan.ts`,
+   `src/serve/completion-executor.ts`, `src/serve/completion-sink.ts`.
+
+### Model selection is a declared profile, not a flag
+
+`resolveModelProfile()` (`src/model/profile.ts`) freezes, before construction,
+the artifact/config identity, the fidelity target, the required engine
+capabilities, and the loader/graph/loop composition. Exact artifact profiles
+outrank family profiles; dedicated/generated graphs outrank the universal dense
+fallback; an exact profile whose config or capabilities mismatch refuses
+instead of downgrading. Profiles declare construction only — they cannot
+rewrite MTP, KV scheme, adapters, grammar, or sampling (those are request
+methods resolved independently). Promotion universal → dedicated is a
+checklist: custom graph file, bit-exact goldens vs the scheme oracle, an h2h
+entry at ≥1.00× vs mlx-lm on a stable pass, eval-suite rows recorded.
+
+Note: `profile.ts` keeps `FidelityTier = "l1" | "l2" | "l3"`, where `l3` means
+`oracle: null, claim: "measured"` — the Lab, named by its old letter. There is
+no `--l3` on the CLI (it hard-errors; §7).
+
+### Features decompose ALONG the layers
+
+Per-request LoRA is the worked example: L1 = the math (`Wx + scale·B(Ax)`,
+oracle-gated vs mlx-lm's tuner); L3 = the policy (which rows run which
+adapter; today one active adapter per batch, and adapter requests run on the
+serial mechanism); L4 = the surface (`adapter` field, mount/hot-swap,
+discovery); plus one sideways obligation — KV under adapter A ≠ base KV, so the
+prompt cache keys by adapter namespace (`cacheNs` in `runGeneration`). When
+placing a feature, name its component at each layer and each component's gate.
+
+### Sampling's placement and the debranching invariant
+
+Sampling is *specified* at L4 (request params → a per-row `StepSampler`,
+`src/sampler.ts`) and *executed* at L3 (each step, on each row's `[1,V]`
+slice; vectorized when all-greedy) — it never enters L1. Everything
 per-request-variable is either POST-GRAPH (sampling, grammar masks, stop
-logic) or DATA the fixed graph consumes (adapter weights, caches). Mixed
-adapter/no-adapter batches stay branch-free via vLLM's Punica/SGMV pattern
-— every row runs the adapter math, non-adapter rows at scale 0; data, not
-control flow (vLLM = the oracle for per-slot LoRA batching). Debranching
-backlog: the per-token runtime checks still in generated files (`L > 1`,
-`fusedSdpaRuntimeOk`) are per-GENERATION-decidable → hoist them out of the
-token path; compiled decode already achieves the fully-debranched line
-dynamically (trace once, replay).
-
-**The option surface IS the four layers.** Users adjust: (1) how the model
-graph works — universal vs optimized file, compiled decode, future
-unrolled/fused variants; (2) how the quantization works — weights bpw,
-KV scheme, future TurboQuant; (3) how the scheduler works — concurrency
-cap, KV budget, prefill chunking, spec policy; (4) the serving surface —
-endpoints, adapters, structured output, sampling defaults, caches,
-multi-model. Flags, help text, and reference docs should be ARRANGED in
-these four groups.
-
-**Composition truth (2026-07-05, source-verified to the line AND
-empirically proven live):** "does mixed-precision KV work with batched
-requests?" — optiq serve as shipped: **NO** (install_mixed_kv hooks
-stream_generate, the serial path; mlx_lm.server 0.31.3 contains zero
-quantization in its 1,904 lines and BatchKVCache has no to_quantized —
-batchable requests decode bf16). **Live 2×2 proof** (optiq serve, cpm5,
-9,132-token prompt, 128-tok decode, fresh server per arm; seeded requests
-route serial per mlx-lm's `_is_batchable`): batched path decode is
-IDENTICAL with and without --kv-config (120.5 vs 120.6 tok/s — the config
-is inert there), while the serial path drops 148.7 → 32.7 tok/s when the
-mixed hook engages (4.5× slower at this context in the shipped python
-pipeline). So optiq's composed reality today: batched requests silently
-bf16; quantized requests pay a heavy serial penalty. Phase 3.1's batched
-mixed-KV at full engine decode speed is an uncontested frontier point. mlx-bun
-today: **NO** (willBatch routes shape.kvQuant serial). mlx-bun after Phase
-3.1: **YES — the build**, and a first on this stack. In layer terms the
-limitation is purely layer 2: the quantized schemes only have single-row
-implementations, so layer 3 has nothing to call — not an engine limitation,
-not a model limitation. Phase 3.1 = give layer 2 batched quantized caches
-(copy the serial per-row ops that already bit-match optiq), teach layer 3 to
-call them at join/evict, hand layer 1 what it already dispatches on, then
-delete `shape.kvQuant` from willBatch — that deletion IS the feature.
-Bonus finding from the same review: optiq's Mac-safe concurrency default is
-**8** (they warn mlx-lm's 32 OOMs unified memory under burst) — adopt 8 as
-the `--batch` default, superseding the earlier 32.
-
-## 4. The engine: concurrency IS the batch size
+logic) or DATA the fixed graph consumes (adapter weights, caches). This is
+what keeps the generated graph a straight line. Debranching backlog, still
+open: the generated gemma files carry per-token runtime checks (`L > 1 &&
+fusedSdpaRuntimeOk(q, mask)` in `src/model/generated/gemma4-*.ts`) that are
+per-generation-decidable and should be hoisted out of the token path; compiled
+decode already achieves the debranched line dynamically (trace once, replay).
+
+### Layer 0 — the SSD spill substrate
+
+It is almost always faster to cache to SSD and mmap it back than to
+regenerate, and spilling lets RAM be a cache instead of the only tier. Three
+implementations prove it: the SSD KV cold tier (`src/ssd-cache.ts`, a
+`ColdTier` inside `PromptCache.take()` so both mechanisms restore prefixes
+from disk at admission), MoE expert offload (page-aligned mmap), and the
+byte-capped prompt cache. Decision: one generic substrate — content-addressed
+keys (fingerprint + scheme + adapter + producer version), zero-copy mmap
+restore, byte-capped LRU, corruption self-quarantine — that layers 1/2/4
+consume. Future clients: vision encoder features, compiled grammars,
+quantization results, TurboQuant artifacts. Economics (Josh): the SSD
+competes with RECOMPUTE, not RAM — a long agent context is minutes of prefill
+vs about a second of NVMe restore, and the ratio grows with context.
+
+### The option surface IS the four layers
+
+Users adjust: (1) how the model graph works — profile, compiled decode; (2)
+how quantization works — weight bpw, KV scheme, TurboQuant; (3) how the
+scheduler works — concurrency cap, KV budget, drafting policy; (4) the serving
+surface — endpoints, adapters, structured output, sampling defaults, caches,
+multi-model. Flags, help text, and reference docs should be arranged in these
+four groups (§7).
+
+## 3. Fidelity: per-scheme oracles, two axes, one decision procedure
+
+The tier ladder is not a product mode. It is a **map from numeric scheme to
+its verification oracle**:
+
+| scheme | oracle | verification |
+|---|---|---|
+| bf16 (default), any batch B | **mlx-lm** at the same composition | bit-exact goldens (serial + B=2) |
+| uniform kv 4/8 | **mlx-lm** `--kv-bits` (+ optiq's rotating cache class where upstream is NYI) | bit-exact (`kv-quant` tests) — `quantizedSdpaUnfused` is op-for-op mlx-lm's `quantized_scaled_dot_product_attention` |
+| mixed per-layer kv (`kv_config.json`) | **mlx-optiq** `install_mixed_kv` | bit-exact (`mixed-kv-parity` test; batched rows per-row vs the serial oracle) |
+| TurboQuant KV, rotation-folded weights, mixed-bpw weights, original kernels, batch-invariant kernels | **none exists** → Lab gates | KL vs our own reference path + frozen eval suite + envelope tests + kill switch |
+
+`--l1`/`--l2` therefore name the first three rows' oracles. Mixed-precision
+quantization is NOT demoted by the naked-=-L1 decision — it is the flagship of
+the row that mlx-lm cannot oracle.
+
+### The two axes (parity gates performance)
+
+Every optimization has two independent coordinates, and conflating them is
+what made the flag surface unreadable:
+
+1. **Parity** — which reference does it reproduce bit-for-bit? This sets the
+   *lowest* tier it may live in: matches mlx-lm → L1; matches optiq but not
+   mlx-lm → L2; matches neither → no oracle, Lab-gated.
+2. **Performance** — is it the fastest correct way? This sets whether it is
+   the **default (on)** within its tier or a kept-but-off opt-in.
+
+**Parity gates performance.** A faster kernel that breaks mlx-lm parity is not
+"a faster L1" — it bubbles up to whatever tier its parity allows, and the user
+opts in. An *optimization* that still matches the oracle stays low:
+compiled decode is ours but is L1 because it replays the same ops bit-for-bit.
+Only nodes with no oracle are forced into the Lab.
+
+### Decision procedure for any new optimization
+
+1. **Measure its parity** (run both ways, compare — one flag, one measurement
+   per lever). The lowest oracle it matches bit-for-bit is its tier ceiling.
+2. **Measure its performance** vs that tier's current default on a stable
+   pass:
+   - faster AND holds the guarantee → becomes the **default (on)** for the tier;
+   - slower but still correct → a documented **default-off kill switch**
+     (useful for A/B; never the default);
+   - breaks the guarantee but offers a real trade-off → a **Lab item**
+     (env flag, bench script, expiry); earns a default only by beating the L1
+     baseline in a paired A/B (+ KL PASS if output-changing).
+
+Worked: *5% less memory, still bit-for-bit* → default-on in L1. *2× tok/s at
+KL 0.0015* → Lab item. *Another way, 3% slower, still correct* → kept, off.
+
+### Composition rule: a composition inherits its scheme's oracle
+
+Batching, prompt cache, spec decode, adapters — none change a scheme's
+numerics, so composing them with a scheme verifies against THAT scheme's
+oracle, per row. Batched bf16 anchors to mlx-lm B=N goldens; batched mixed-KV
+anchors to optiq's per-row mechanics (our serial `maybeQuantizeKv` is the
+verified op-for-op copy) and gates on per-row bit-exactness of the unpadded
+row plus a calibrated envelope for padded rows (their noise is pre-existing
+bf16 reduction order under grid snapping, measured at the pinned join step).
+Do NOT invent Lab-style KL/teacher-forced gates for a composition of an
+oracle-backed scheme; the Lab is only for schemes with no oracle at all.
+
+### Flag classification (from the tier framing)
+
+A flag falls into exactly one of three buckets:
+
+| flag selects… | tier effect | what it IS | action |
+|---|---|---|---|
+| the only viable route | — | not a choice | always-on default; no flag |
+| between two routes at the same tier / same oracle | none | a kill switch (A/B, debugging) | default the fast one; keep the slow one documented |
+| between an oracle-backed route and a no-oracle route | tier ↔ Lab | a real parity ⇄ optimization knob | keep, document as such, group by layer |
+
+Training flags classify the same way (verified in `src/cli.ts` train help):
+`--grad-clip` is an always-on guard; `--seg N` / `--no-segment` is a
+memory↔compute knob, bit-exact either way; `--grad-accum` and `--lambda` are
+training-quality/hyperparameter knobs; `--no-flash` (flash-CCE head → the
+MLX fused head) and `--no-prefix` (prefix-shared forward → two-forward) are
+the only true parity ⇄ optimization toggles on that surface. The 2026-06-21
+"parity-tier DAG" HTML map that motivated this classification
+(`docs/dag/training-inference-map.html`) no longer exists in the tree; the
+roadmap it proposed — derive the graph from code, CI-enforce "an L1-tagged
+node has a passing bit-exact test", shrink the no-oracle surface, generate the
+flag surface from the graph — remains an idea, not a phase.
+
+### Lab lifecycle rule (the anti-flag-pile law)
+
+Every Lab experiment ships with (a) a hypothesis in frontier-axis terms, (b) a
+paired A/B bench script, (c) an expiry review date. Outcomes: promote to
+default (with stable-pass numbers) or delete (with a breadcrumb in the design
+doc). No third state. "Off in every regime with no promotion path" is a
+deletion that hasn't happened yet.
+
+## 4. Decision: the naked default IS L1 (2026-07-05, Josh)
+
+The 2026-07-05 h2h pass showed the L1 kernel set at decode parity with mlx-lm
+on every model while every output-changing lever failed to beat it in a paired
+A/B, and quantized KV measured slower than bf16 at ≤16k on both stacks (it
+buys memory headroom only). The record with numbers is PLAN.md "Decision:
+naked default = --l1".
+
+- **Naked = `--l1`.** `applyDecodeRoute` (`src/cli.ts`) defaults the tier to
+  `l1`: bf16 KV, compiled decode on, compiled activations on, fused SDPA off.
+- **Prior perf-optimization work is untrusted until re-proven** from this
+  baseline (the losers were deleted the same day — §11 History).
+- **The bar for any lever to earn a default back:** a paired A/B win vs L1 on
+  a stable pass (no `unstable` tag), plus KL PASS if output-changing.
+- **Explicit `--kv-quant` picks its oracle's composition**: `config` → optiq's
+  fused prefill SDPA + stock unfused decode (the L2 golden composition);
+  uniform `4|8` → unfused (mlx-lm's algorithm, L1-eligible); `turbo[:k.v.]` →
+  dequantize-on-fetch through stock `ops.sdpa` (Lab). `--fused-sdpa` still
+  overrides either.
+- **`--l1`/`--l2` are pure aliases.** Each expands to a fixed set of per-fork
+  values (`TIERS` in `applyDecodeRoute`), installed through
+  `configureRuntime()` into the immutable runtime snapshot before model modules
+  load; a per-fork flag overrides one value. Two invariants: a user can
+  reproduce any tier byte-for-byte from individual flags, and no preset may set
+  anything that isn't also an individual flag.
+
+## 5. The engine: concurrency IS the batch size
 
 **Decision (Josh, 2026-07-05): batching is determined by how many concurrent
-requests are in flight, not by a flag.** One request = batch of 1 at full
-serial speed; N requests = continuous batching; requests join/leave running
-batches (the extend-join machinery already landed and is token-exact vs
-mlx-lm B=2).
-
-This **reverses** batching-v2-plan.md's standing decision (`--batch N` as a
-mode switch, auto-batching rejected for determinism). The old objection was
-"an idle-vs-loaded server produces different numerics for the same request."
-It gets a real answer, not a shrug:
-
-1. **The L1 oracle itself behaves this way.** mlx_lm.server auto-batches by
-   default; "drop-in for mlx-lm" now *requires* concurrency-driven batching.
-   Our parity contract becomes: **bit-exact to mlx-lm at the same batch
-   composition** — already golden-verified at B=1 and B=2.
-2. Anyone who needs load-independent numerics pins the cap to 1 (see below).
-3. **Batch-invariant kernels** (identical numerics regardless of B — the
-   Thinking-Machines-style program) go to the Lab as a research item; if that
-   ever lands, determinism-under-load comes back for free and it's an
-   arXiv-lens result on Apple Silicon.
-
-**Flag decision (Josh, 2026-07-05): the flag stays `--batch` — no rename.**
-Its *semantics* change from mode switch to cap: `--batch N` = maximum
-concurrent rows (mlx-lm's `--decode-concurrency` twin), **default 32**, and
-`--batch 1` is the force-serial / determinism pin. TIMING: the 32 default
-lands **with Phase 2, not before** — flipping it while the batched lane is
-still 1.8× slower at B=1 would regress every single-request user ~44%. Until
-GATE-B1-SPEED passes, `--batch` keeps today's default (1 = the serial lane).
-
-**The crux gate — the B=1 gap.** Today the batched lane at B=1 runs cpm5 at
-~149 tok/s vs 267 serial (batching-perf-path.md): the unified engine is
-**1.8× off** at the composition that matters most for a single-user runtime.
-
-**Measured 2026-07-05 (paired in-process A/B, cpm5, M1 Max): mlx-lm's own
-`BatchGenerator` at B=1 runs 256.5 tok/s vs 264.6 for its `stream_generate`
-— a 3.3% tax on the most host-tax-sensitive model we have.** Two
-consequences: (1) the unified design is *proven achievable* — a continuous-
-batching loop can serve a lone request at ~0.97× of the best serial loop,
-so our lane's 1.8× gap is an implementation artifact, not an inherent cost
-of the architecture; (2) our h2h 1.00× parity numbers were measured against
-`stream_generate` (mlx-lm's FASTEST single-stream path), so the baseline we
-match is not batched-handicapped — and `mlx_lm.server` actually serves a
-single request ~3% below the number we already match. The serial lane
-cannot be deleted until:
-
-- `GATE-B1-SPEED`: unified engine at B=1 ≥ 99% of today's serial decode tok/s
-  and TTFT on cpm5 + e4b + 12B (stable bench pass), and
-- `GATE-B1-PARITY`: unified engine at B=1 bit-exact vs the L1 goldens (full
-  logprobs, not just greedy).
-
-How to close 1.8×: the serial lane's wins are known and portable — the
-pipelined decode loop (async-eval overlap), compiled per-step graph replay,
-and no per-token scheduler hops. The likely landing: **the scheduler owns
-admission and join/leave; when exactly one sequence is active it executes the
-same pipelined/compiled step the serial path runs today** (B=1 specialization
-inside one engine — a fast path, not a second lane), falling back to the
-general batched step at B≥2. Same public machinery, no gateway routing rules,
-no feature×lane matrix.
-
-**COMPOSITION IS THE PRODUCT (Josh, 2026-07-05).** The goal state, verbatim
-intent: *"you start with the server, it serves an optimized version of the
-model, you add on top of that mixed-precision quantized KV cache, as well as
-LoRA adapters, and you add on top of that DSpark speculative decoding, and
-structured output potentially, and then you also can switch the sampling
-method and values."* Every serving capability is a STACKABLE layer on one
-engine — never a lane-routing condition, never mutually exclusive. The
-current gateway is the anti-pattern: `willBatch` routes vision / adapters /
-logprobs / seeds / kv-quant / drafts to a different lane, which is exactly
-the "batching OR performance" choice being killed.
-
-The composition matrix the unified engine must satisfy (each row = a layer;
-any subset of rows must stack):
-
-| layer | today | end state |
+requests are in flight, not by a flag.** One request = a lone row at serial
+speed; N requests = continuous batching; rows join and leave running batches.
+This reversed `docs/design/batching.md`'s "`--batch N` is a mode switch" decision.
+The old objection — "an idle vs loaded server produces different numerics for
+the same request" — gets a real answer:
+
+1. **The L1 oracle itself behaves this way.** `mlx_lm.server` auto-batches;
+   "drop-in for mlx-lm" *requires* concurrency-driven batching. The parity
+   contract is **bit-exact to mlx-lm at the same batch composition**.
+2. Anyone who needs load-independent numerics pins `--batch 1`.
+3. **Batch-invariant kernels** (identical numerics regardless of B) are a Lab
+   research item; if they land, determinism-under-load returns for free.
+
+**The flag stays `--batch`** (no rename); its semantics are the cap on
+concurrent rows (`--decode-concurrency` is accepted as the mlx_lm.server
+alias). **Default 8** (`serverOptions.batch ?? 8` in `src/server.ts` and
+`src/cli.ts`) — optiq's Mac-safe concurrency default and the 4–8-sub-agent
+workload, superseding the earlier 32. `--batch 1` pins the strict serial
+executor. An older draft of this doc and its diagram said 32; the code says 8.
+
+**The workload this is for**: a user's agent harness pointing sub-agents at
+one local server. Serial, the Nth agent's first token waits for N−1 full
+generations; batched, every agent starts instantly and the fleet's wall-clock
+collapses. Consequences: concurrency-driven batching (sub-agents don't
+coordinate arrivals), prefix sharing gains priority (shared system prompts),
+per-slot adapters, and batching × quantized KV first (long contexts).
+"Single-user" in this project means single user, many agents.
+
+### The B=1 fast path (the crux gate, met)
+
+The unified engine could not replace the serial lane until a lone request
+through the scheduler matched serial decode speed and bits. mlx-lm's own
+`BatchGenerator` at B=1 runs within a few percent of its `stream_generate`
+(paired A/B, PLAN.md), so the design was proven achievable; our gap was an
+implementation artifact, closed in two moves (both are now design contracts,
+§8): readbacks in a pipelined loop must not create ops, and a lone row keeps
+bare serial-class caches so the step dispatches the same graph the serial loop
+builds. Today `BatchScheduler` runs, for `B === 1 && unpadded && supports(inners)`
+with a uint32 pipeline register, the serial engine's `CompiledDecode` step
+(adopt-don't-copy: a row joining an empty batch keeps its solo caches as the
+inners; the merge copy runs only when a second row actually joins). The
+recorded scheduler/serial paired decode ratios are 0.992–0.996 (PLAN.md
+"Serving architecture consolidation"), and `GATE-B1-PARITY` (bit-exact vs the
+L1 goldens through the scheduler) is green on the curated probes.
+
+### Composition is the product
+
+Every serving capability is a STACKABLE layer on one engine — never a
+lane-routing condition, never mutually exclusive. The matrix the engine must
+satisfy, with the state of `GenerationGateway.#supportsContinuous` as read on
+2026-08-23:
+
+| layer | today (src) | end state |
 |---|---|---|
-| optimized per-model graph | ✅ both lanes | ✅ (fingerprint-routed, unchanged) |
-| mixed-precision quantized KV | serial only | one active scheme per server, applies at any B |
-| LoRA adapters (hot-swap, per-request select) | serial only (compiled decode also skips them) | per-slot adapter state in the batched step |
-| speculative decoding (two-model + assistant + DSpark, 2026-07-06) | serial only, forces ALL requests serial; the `DraftSource` seam is now KV-borrowing-ready (target donor-KV / anchor-hidden / tapped H_ctx flow through it — the per-slot substrate is in place) | per-slot drafting behind the `DraftSource` seam |
-| structured output (grammar) | ✅ both lanes (B1 per-row matchers) | ✅ (already composes; keep the conformance gate) |
-| sampling method + values (temp/top-p/top-k/min-p/XTC/penalties/HLG/seed) | per-row samplers batch; seeds force serial | fully per-row, seeds included |
-| prompt cache / SSD tier | serial only | prefix reuse for batched rows |
-
-Ordering (by value): quantized KV under batching first (the
-agent-fan-out-with-long-context composition), then prompt cache, then
-adapters, then spec decode per slot, then chunked prefill interleaving
-(vLLM policy) so a 16k prefill doesn't stall running streams.
-`scripts/bench-modes.ts` is already the composition scoreboard (its cell
-matrix spans kv × batch × grammar × cache × spec) — every layer added to
-the batched engine gets a cell there and a parity gate.
-
-## 5. Fidelity: per-SCHEME oracles (fixes "L1 can't oracle mixed")
-
-The tier ladder stops being a product mode and becomes what it always really
-was — a **map from numeric scheme to its verification oracle**:
+| dedicated per-model graph | both mechanisms (profile-routed) | unchanged |
+| mixed-precision KV (`affine-config`) | continuous when every configured layer is a plain or rotating KVCache the scheme can convert (`KvScheme.batchable` + the gateway's cache probe); uniform `kvBits`, SSM-layer configs, and TurboQuant route serial | one active scheme per server, any B |
+| LoRA adapters | serial only (`shape.hasAdapters`) | per-slot adapter state (vLLM Punica/SGMV pattern: every row runs the adapter math, non-adapter rows at scale 0) |
+| speculative decoding (two-model · assistant · dspark · deepspec · native MTP · ngram) | serial only; a mounted draft routes every request serial (`shape.hasDraft`, mlx_lm.server's `is_batchable = draft is None`); GLM-5.2 MTP is on by default and pins the serial verify lane (`--mtp off` restores batching) | per-slot drafting behind the `DraftSource` seam |
+| structured output (grammar) | both (per-row matchers; `MLX_BUN_GRAMMAR_BATCH=0` forces serial) | unchanged; keep the conformance gate |
+| sampling (temp/top-p/top-k/min-p/XTC/penalties/HLG/seed) | per-row `StepSampler` batches, including repetition penalty and logits extras; a user-fixed `seed` forces serial (mlx-lm's `_is_batchable`) | fully per-row, seeds included |
+| logprobs capture | serial only (`shape.wantsLogprobs`) | per-row readback in the scheduler |
+| vision / audio media prompts | serial only (`shape.hasVision`; embeddings-prefill bypasses the prompt cache) | batched media prefill |
+| prompt cache / SSD tier | both: scheduler joiners `take()` the longest usable prefix at admission, never-merged rows `put()` back on finish; the cold tier restores inside `take()` | block-granular sharing across rows |
+| paged KV (`--paged-kv`) | serial only, Gemma4 bf16, refuses `--batch N>1`, `--kv-quant`, `--draft-model`; bypasses the prompt cache | block-level CoW prefix sharing (vLLM oracle) |
+
+Ordering of the remaining rows (by value): adapters per slot, then spec per
+slot, then logprobs, then chunked prefill interleaving (vLLM policy) so a long
+prefill doesn't stall running streams. `scripts/bench-matrix.ts modes` is the
+composition scoreboard — every layer added to the scheduler gets a cell and a
+parity gate.
+
+## 6. Request path today
+
+Modules in order, as read in `src/` on 2026-08-30:
+
+1. **The request pipeline** (`src/serve/`). A request is data; each stage
+   is one program with a declared input and output, composed in order by
+   `src/server.ts` (which otherwise only assembles the server — KV scheme,
+   prompt cache + SSD tier, gateway, admission — and dispatches routes):
+   `new ChatRequest(body)` (`chat-request.ts`; validation at construction,
+   mlx-lm's 400 messages) → `ChatStage.run` (`chat-stage.ts`; template render
+   + prompt ids, media embeddings by family, grammar, adapters, sampling
+   defaults via `request-prep.ts`) → `InferenceRequest` (`inference-request.ts`)
+   → `InferenceStage.admit` (memory ceiling → reject or clamp; seals the plan)
+   → `InferenceStage.run` → `InferenceResult` → wire. `/v1/completions` is
+   `TextCompletionStage` into the same inference stage; `/v1/messages` and
+   `/v1/responses` reuse `ChatStage` and format the result themselves. One
+   JSON/SSE writer (`http.ts`) serves every surface; the protocol object
+   (`openai-wire.ts`, `src/anthropic.ts`, `src/responses.ts`) is the only
+   thing that differs.
+2. **`src/serve/completion-executor.ts` `prepareCompletion()`** → **`src/serve/request-plan.ts` `planRequest()`** — admission (memory budget → reject or clamp `maxTokens`), capture options, adapter selection, and the `RequestShape` are derived ONCE for chat and raw-text, streaming and non-streaming. The result is an opaque, single-use `PreparedCompletion`; adapters cannot inspect or rewrite the plan. Owned resources (media arrays, grammar) dispose on rejection.
+3. **`CompletionExecutor.execute()`** — owns one completion attempt: asks the engine for placement, records the lane (`src/serve/lane-registry.ts`), builds the `CompletionSink` (semantic events: content / reasoning / tool calls / stop), collects logprobs, transfers resource ownership exactly once, runs, and settles the terminal summary (finish reason, usage, cached tokens, final lane, speculation stats). Adapters only format that summary.
+4. **`src/serve/generation-gateway.ts` `GenerationGateway.place(shape)`** — freezes one `GenerationPlacement { shape, mechanism: "serial" | "continuous" }`. `#supportsContinuous` is a capability check only (§5 matrix); it never rewrites MTP, KV scheme, TurboQuant, grammar, adapters, or sampling. `run()` rejects a placement made for another shape. One `AsyncMutex` is the single mutual-exclusion domain: a serial run holds it; the scheduler holds it for its whole active period; a waiting serial request drains the batch (mlx-lm's `drain_batch`).
+5. **`continuous` →** `src/serve/batch-scheduler.ts` — admission under `--kv-budget`, chunked interleaved solo prefill, join/merge/extend/filter over per-layer cache types, the pipelined step loop, per-row `StepSampler`, eviction. B=1 takes the adopted-cache fast path (§5). **`serial` →** `runGeneration` in `src/server.ts` → `src/generate.ts` (prompt-cache take/put, snapshot boundary, the chunked prefill, the pipelined compiled decode loop) or, when a draft is mounted and the request is spec-eligible, `src/spec/serve-loop.ts`.
+
+**The single-path rule.** Options are *declarations* carried through
+placement, never forks above the scheduler. A request's resolved settings
+(KV scheme, TurboQuant, drafting, grammar, adapters, sampling) are authoritative
+from `planRequest` onward; `place()` may only answer "does the continuous
+mechanism implement exactly this composition?" — it may not substitute a
+cheaper composition to make a request batchable (the optiq bug class: silently
+serving bf16 under a quantized scheme). The scheduler independently refuses a
+scheme it cannot execute at construction, and `KvScheme` freezes its per-layer
+entries so a caller cannot mutate conversion or accounting after resolution.
+
+**Known deviations from the single-path rule (verified 2026-08-23):**
+
+1. **Two prefill loops.** `src/generate.ts` (the chunk loop with the
+   `snapshotAt` split, the mlx-lm tail-split convention
+   `min(prefillChunkSize, remaining-1)`, per-chunk `clearCache`) and
+   `BatchScheduler.#prefillChunk` in `src/serve/batch-scheduler.ts` (the same
+   conventions re-implemented over `PrefillState` for interleaved admission).
+   Both are gated bit-exact, but a convention change must land in two places.
+2. **Spec eligibility is decided after placement.** `planRequest` only knows
+   `hasDraft = !!ctx.draft` (server-level), which routes every request serial
+   while a draft is mounted. The real per-request eligibility — text-only, no
+   adapters, no logprobs capture, bf16 KV (no `kvBits`/`kvConfig`/`turboQuant`),
+   not paged — is evaluated inside `runGeneration` after `place()`; ineligible
+   requests silently fall through to plain serial decode, and the lane label is
+   corrected from `serial+spec` to `serial` only when `stats.spec` is absent
+   (`finalLane` in the executor). Resolving this means lifting the eligibility
+   predicate into `RequestShape` so placement and the lane label are decided
+   once.
+
+Other seams worth knowing: `runGeneration` also decides the prompt-cache
+bypass (media, paged) and the adapter cache namespace; `--isolate` wraps the
+ENTIRE server as an engine child behind a proxy (`src/serve/isolate.ts`,
+`ModelPool` for `--model-pool` LRU residency and model switching by the
+request's `model` field) — the inter-process API is the /v1 surface itself, no
+second protocol.
+
+## 7. The flag surface (src truth, grouped by layer)
+
+Read from `SERVER_FLAGS` and `applyDecodeRoute` in `src/cli.ts`. Every
+`MLX_BUN_*` value is read through the immutable runtime snapshot
+(`src/runtime-config.ts` `runtimeValue/runtimeFlag`, `src/flags.ts` `flagOn`);
+feature code never reads `process.env`, and the CLI installs tier/fork
+overrides with `configureRuntime()` before model modules load.
+
+**Parity tier (aliases):** `--l1` (default) · `--l2`. `--l3` hard-errors with a
+pointer to this doc.
+
+**L2 — quantization:** `--kv-quant config|off|4|8|turbo[:k<bits>v<bits>]`
+(default off). An older draft of this doc listed only `config|4|8|off`; the
+`turbo` axis is real and mutually exclusive with the affine modes.
+
+**L3 — scheduler / decode policy:** `--batch <n>` (default 8;
+`--decode-concurrency` alias) · `--kv-budget <GB>` · `--draft-model <query>` ·
+`--draft-kind two-model|assistant|dspark|deepspec|mtp|ngram` ·
+`--num-draft-tokens` · `--ngram-max` / `--ngram-min` · `--mtp on|off`
+(GLM-5.2 native MTP, on by default) · `--context-length` (GLM-5.2 reservation)
+· `--paged-kv` / `--paged-kv-block-size` (env `MLX_BUN_PAGED_KV=1`).
+
+**L4 — serving surface:** `--model` · `--host` · `--port` · `--memory-budget`
+· `--prompt-cache <GB>` · `--ssd-cache <dir>` · `--ssd-cache-max` ·
+`--ssd-demote-idle` · `--ssd-cache-verify` · `--isolate` · `--model-pool` ·
+`--unix` (internal) · `--no-open` · `--allow-private-media` · `--adapter`
+(`--adapter-path` alias) · `--thinking` · `--temperature`/`--temp` · `--top-p`
+· `--top-k` · `--max-tokens` · the HLG sampler family (`--hlg-sampling`,
+`--hlg-width`, `--hlg-shoulder`, `--hlg-toe`, `--hlg-pivot-offset`).
+
+**Kill switches (bit-exact A/B levers, each selects a slower same-parity
+path):** `--compiled-decode on|off` (`MLX_BUN_COMPILED_DECODE`) ·
+`--compiled-activations on|off` (`MLX_BUN_COMPILED_GEGLU` +
+`MLX_BUN_COMPILED_SWIGLU`; only gemma geglu and MiniCPM5 swiglu have an
+uncompiled form — qwen3/qwen3.5/universal compile unconditionally) ·
+`--fused-sdpa on|off` (`MLX_BUN_NO_FUSED_SDPA`, inverted; default follows
+`--kv-quant`) · `--force-wire` (`MLX_BUN_FORCE_WIRE`) · `--expert-offload`
+(`MLX_BUN_EXPERT_OFFLOAD`). An older draft said these were env-only; the three
+decode kill switches are CLI flags as well.
+
+**Lab / diagnostic env flags (no CLI flag; live with a bench and an expiry):**
+`MLX_BUN_GRAMMAR_JUMP=1` (jump-forward decoding, serial non-spec loop),
+`MLX_BUN_GRAMMAR_BATCH=0` (grammar back to serial), `MLX_BUN_BATCH_SSM=0`
+(SSM caches back to serial), `MLX_BUN_BATCH_NO_PIPELINE`,
+`MLX_BUN_BATCH_STEP_TRACE`, `MLX_BUN_BATCH_VEC_SAMPLE`, `MLX_BUN_BATCH_EXTEND`,
+`MLX_BUN_PREFILL_TAIL_SPLIT`, `MLX_BUN_FLASH_MIN_M`, the `MLX_BUN_CCE_*`
+training-kernel switches, and the memory/trace loggers. Anything
+output-changing is Lab by definition.
+
+**Deleted (2026-07-05, §11):** `--fused-decode`, `--fused-gelu`,
+`--perf-kernel`, `--l3`, and their env twins. None exist in `src/`.
+
+## 8. Design invariants proven en route (durable, not changelog)
+
+- **Readbacks in a pipelined loop must not create ops.** `toFloat32()` on the
+  int token array enqueued a cast BEHIND the next dispatched step, stalling the
+  "overlapped" read a full GPU step per token. Use `MlxArray.toIntTokens()`.
+- **A lone row keeps bare serial-class caches.** `KVCache.makeMask(1)` is the
+  empty mask and scalar rope; the batched step then dispatches the same graph
+  the serial loop builds. The per-layer `BatchedDecodeMaskCache` wrapper is
+  only for padded batches.
+- **Rope-array step-stability contract.** A batched cache's `ropeOffsetArr`
+  must be stable within a decode step and refresh only at `releaseRopeArr()`;
+  re-reading it mid-update ropes K and Q one position apart.
+- **Generated-file guards accept batched subclasses.** The generated forwards'
+  per-layer `instanceof` guards pass any batched cache that subclasses a
+  serial class, so an all-quant gemma batch decodes through the generated
+  fast path — a feature (B=1 proved bit-exact) but a contract: batched caches
+  must behave serially under re-reads. Test with FULL configs; one bf16 layer
+  anywhere fails the guard and drops to the monolith.
+- **Quantization packs along HEAD_DIM**, so token-axis batch surgery over
+  (packed, scales, biases) triples is byte-safe; solo rows convert at serial
+  chunk boundaries (bit-exact by construction).
+- **A scheme-less path must refuse, never silently drop quantization.** The
+  gateway only threads a `KvScheme` to the scheduler when it is batchable; the
+  scheduler refuses an unsupported scheme at construction.
+- **Prefix sharing is non-consuming.** `PromptCache.take()` serves zero-copy
+  clones (ref-counted retain so a demoted donor never unmaps pages a clone
+  reads); the donor stays put; `put()` supersedes same-namespace prefix
+  ancestors when the new entry is trimmable. Kills the cannibalization flaw
+  (agent B consuming agent A's entry). v1 shares COMPUTE and durability;
+  concurrent rows still hold separate physical KV — one shared physical prefix
+  across rows is the block-KV frontier item, and whole-entry duplication in
+  the disk tier is the block-granularity revisit trigger.
+- **Prompt-boundary snapshot.** The prompt+generation entry is untrimmable
+  past a wrapped ring or under quantized KV, so every substantial request also
+  snapshots a strict prompt prefix (cap `len-1`), the mlx-lm
+  `insert_segments` invariant; a re-rendered next turn always matches.
+- **Never a JS callback as an mlx buffer destructor** — last-ref `Data` dtors
+  run on the Metal completion thread (native `dlsym(free)` dtor, process-pinned
+  mmaps).
+- **Isolation proxies the whole server**, not the gateway: grammar WASM,
+  vision arrays, and sampler closures don't serialize; the /v1 surface is the
+  IPC.
+
+## 9. The frontier program (order matters)
 
-| scheme | oracle | verification |
+Ranked by expected frontier shift (memory ↓ / intelligence ↑ first, then
+speed). Each item is a Lab program with its own doc; `decode-speed-program.md`
+ranks the speed levers and is the only doc that does.
+
+1. **Mixed-precision weights** — knapsack `--target-bpw` (ours; OptiQ-style
+   sensitivity port in `src/quantize/sensitivity.ts`) and rotation-folded
+   quantization (`docs/design/turboquant.md`, `--rotate-weights`). Gate:
+   perplexity + frozen 6-task eval at equal bpw.
+2. **TurboQuant KV** (`docs/design/turboquant.md`, landed v1) — orthogonal to
+   allocation: mixed precision is the ALLOCATION axis, TurboQuant the
+   QUANTIZER axis; they compose. Solo-only in v1.
+3. **Speculative decoding depth** — DFlash/DSpark, native MTP, behind the
+   `DraftSource` seam (`docs/design/speculative-decoding.md`).
+4. **Per-model graph work from the baseline** — unroll a model's flat DAG,
+   find fusion the compiler misses, prove with kernel-trace diffs, promote per
+   model.
+5. **Batch-invariant kernels** (research) — restores determinism under load.
+
+## 10. Decisions log and open items
+
+1. **Cap default 8** — flipped 2026-07-05 after the B=1 gates; `--batch 1`
+   pins arrival-independent numerics; kv-budget admission keeps 8 safe on
+   small boxes.
+2. **Flag stays `--batch`**; semantics = cap.
+3. **Perf kernel deleted**; a future flash-decode kernel re-derives from the L1
+   baseline in the Lab — no resurrection.
+4. **`--l1`/`--l2` remain user-facing CLI aliases** (`src/cli.ts` documents
+   them under "Parity tier"). The 2026-07-05 intent that they become
+   bench/test vocabulary only has not been executed; if it is, `serve --help`,
+   `docs/reference/cli.md`, and `server-config.md` change in the same commit.
+5. **Block-paged KV**: an optional serial-only v1 exists (`--paged-kv`,
+   `docs/design/kv-cache.md`); padded-batch waste removal and block CoW prefix
+   sharing are its follow-ups, triggered when prefix sharing under batching
+   becomes the bottleneck.
+6. **Prefix sharing scope**: must extend to the disk tier; first-class cases
+   are new-session spin-up and server restart — the shared prefix is a durable
+   object, not a property of one conversation's entry.
+7. **Open composition rows** (§5): adapters, spec, logprobs, media on the
+   continuous mechanism; chunked prefill interleaving as policy.
+8. **Open single-path deviations** (§6): unify the prefill loops; move
+   prompt/media/grammar construction behind the seam; lift spec eligibility
+   into `RequestShape`.
+9. **Debranching backlog** (§2): hoist the generated files' per-token
+   `L > 1 && fusedSdpaRuntimeOk` checks.
+
+## 11. History
+
+**2026-07-04 — faithful → L1 consolidation.** An audit found three
+overlapping "match mlx-lm" mechanisms that disagreed: the `--l1/--l2/--l3`
+tier presets, an `MLX_BUN_FAITHFUL` preset that forced the gemma monolith and
+set choices no flag could reach, and a family of unwired `Faithful*` model
+subclasses. The decisive rule that collapsed them: **"compiled" kernels
+(`@mx.compile` geglu/swiglu, compiled decode) go through the same libmlx as
+mlx-lm → bit-exact and faster → L1 defaults; custom "fused" Metal kernels were
+mlx-bun originals with a proven residual → not bit-exact.** Landed in four
+phases: compiled activations default-on everywhere (qwen3, qwen3.5, universal
+dense, gemma geglu via `MLX_BUN_COMPILED_GEGLU`, MiniCPM5 swiglu), verified
+`maxDiff === 0` vs mlx-lm; `--compiled-activations` and `--fused-gelu` added as
+per-fork flags so `--l1` became a pure alias; `MLX_BUN_FAITHFUL`, `faithful.ts`
+(→ `flags.ts`, only `flagOn` kept), and the four unwired subclasses deleted;
+docs + the "cost of removing each faithful kernel" bench matrix. The only
+family that had it right from the start was `qwen3_moe`: the production class
+IS the faithful port. Bit-exact-to-mlx-lm *wants* compiled activations (mlx-lm
+`@mx.compile`s them), so the earlier `--l1` "unfused" help text was backwards.
+
+**2026-07-05 — the Phase-1 deletion pass** (same day as the naked-=-L1
+decision; ~50 files touched, 23 deleted, suite green). Josh: "we will end up
+redoing all the work we have currently done given that we now have a different
+starting point." Deleted, one funeral each:
+
+- `--fused-decode` / `MLX_BUN_FUSED_DECODE` — 1.00×, forced uncompiled decode,
+  a silent-wrongness footgun.
+- `--fused-gelu` / `MLX_BUN_FUSED_GELU` and `MLX_BUN_FUSED_SWIGLU` (+
+  `fused-mlp-kernel.ts`, `steel-linear-kernel.ts`, the swiglu/geglu/steel
+  one-off scripts) — +0–1%; the compiled closures already own the fusion win
+  bit-exactly.
+- `--perf-kernel` / `MLX_BUN_PERF_KERNEL` (the fused online-softmax decode
+  SDPA) with its frozen-oracle scaffolding (`perf-kernel-oracle.test.ts`,
+  `freeze-perf-oracle.ts`) — regressed e4b in the paired A/B; its one win
+  carried a KL WARN. Its earlier placement in the L2 preset (commit `f1bf5cc`)
+  had claimed a bit-exact optiq oracle; the golden was frozen from mlx-bun's
+  own engine and the gate was argmax agreement, so 2026-07-01 had already
+  restored the rule that the bare tier is the guarantee.
+- `MLX_BUN_CPM5_FAITHFUL` + `FaithfulMiniCPM5` (`minicpm5-faithful.ts`) — the
+  default IS the faithful path; the op-for-op A/B reference served its purpose
+  (it is how the unfused-swiglu dispatch tax was found).
+- `--l3` as a product mode — the Lab replaces it; the flag hard-errors.
+- Training-side flag sanitization (`MLX_BUN_PERF_KERNEL=0` /
+  `MLX_BUN_FUSED_GELU=0` in trainers, launchers, recipes) — obsolete.
+
+**2026-07-05 — engine unification** (Phases 0–3.2 of the original migration
+plan; numbers in PLAN.md): the B=1 batch-lane gap measured, root-caused to the
+two host bugs in §8, closed; batched mixed-KV for full-attention and rotating
+layers; adopt-don't-copy + compiled decode at B=1 + prompt cache on the batch
+lane; SSD tier moved inside `PromptCache.take()`; non-consuming prefix sharing;
+`--isolate` P1 (proxy) and P2 (`ModelPool`, multi-model switching); `--batch`
+default 8.
+
+**2026-08-21 — serving architecture consolidation S0–S3** (live in PLAN.md):
+`PreparedCompletion` + `CompletionExecutor` (S0), immutable
+`GenerationPlacement` (S1), declared model profiles (S2), `serial | continuous`
+as the placement vocabulary with the scheduler's B=1 fast path as the default
+lone-request route (S3). S4 (land + post-merge verify) is the open box.
+
+## 12. Interface-based engine refactor
+
+Design requested 2026-09-04. Initial implementation is recorded in §12.13. The R0–R10
+checkboxes live in PLAN.md under "Interface-based engine refactor". This
+section owns the contracts, migration details, and acceptance criteria.
+
+### 12.1 Outcome and scope
+
+The product objective is to push the Pareto frontier of **speed, quality, and
+size for local AI on a Mac**. Josh's goal is that a user sees the result and
+asks how that quality, performance, and model size are possible on their
+laptop. Interface consistency is useful when it makes those gains easier to
+build, compare, compose, and ship. It is not an independent reason to accept
+slower execution or a larger working set.
+
+The current target is **Josh's designated Qwen3.8-27B quants**. They are on a
+drive unavailable to this session. Artifact access is needed for target-specific
+numerical and performance validation; it does not block the refactor.
+The locally cached published flagship and TQ snapshots are not those targets
+(Josh clarified this on 2026-09-04); availability must not select the product
+baseline. Quant recipes and quality findings remain in [TurboQuant](turboquant.md),
+including rejected variants and raw-EOS regressions. Interfaces must preserve
+exact packing, rotation, per-layer precision, recurrent-state, and kernel
+choices. Other artifacts/families provide compatibility coverage only.
+
+A downloaded quant supplies weights, configuration, and tokenizer data. The
+engine owns the hyperoptimized implementations and the rules for identifying
+which quant each one supports. Resolve artifact identity and configuration at
+load time, select registered engine code, then retain that binding for execution.
+The quant does not supply a graph, kernel, method, plugin, or executable manifest.
+An unknown artifact may use a compatible family implementation. An exact
+declaration with missing code or incompatible configuration must fail explicitly.
+Sessions remain independent of the selected implementation.
+
+Aggressive specialization for a particular quantization is expected. A
+quant-specific layout, graph, fused kernel, cache scheme, compiled schedule,
+or complete execution method is a legitimate production implementation.
+Several specialized implementations can be worth maintaining when each wins
+in a useful regime. Shared orchestration must not force the fastest numerical
+implementation through an inefficient generic operator path.
+A model implementation may expose several supported execution methods. Loading
+the weights does not permanently choose one method; request planning selects
+among the compatible methods. Their private graphs, state and numerical loops
+can differ while the session contract remains the same.
+
+A session must work without knowing the model class, graph implementation,
+decoding algorithm, cache layout, or scheduling mechanism. Every replaceable
+component exposes an interface. Immutable configuration, events, and results
+remain data records; implementations keep private helpers private.
+
+There are three distinct replacement operations:
+
+| Replacement | Integration change | What must remain compatible |
 |---|---|---|
-| bf16 (default), any batch B | **mlx-lm** at same composition | bit-exact goldens (serial + B=2 landed) |
-| uniform kv 4/8 | **mlx-lm** `--kv-bits` (+ optiq's rotating cache class where upstream is NYI) | bit-exact (kv-quant.test) |
-| mixed per-layer kv (`kv_config.json`) | **mlx-optiq** `install_mixed_kv` | bit-exact (mixed-kv-parity.test, landed 2026-07-05, maxDiff 0) |
-| mixed-precision weights (knapsack/oQ/TurboQuant), original kernels, batch-invariant kernels | **none exists** → Lab gates | KL vs our own reference path + frozen eval suite (intelligence axis) + envelope tests + kill switch |
-
-So: mixed-precision quantization is NOT demoted by the naked-=-L1 decision —
-it is the flagship of the row that mlx-lm cannot oracle, verified against
-optiq where optiq reaches (KV) and against the eval suite where no oracle
-exists (weights). `--l1/--l2` survive as *bench/test vocabulary* for the
-first three rows; they stop being user-facing product modes.
-
-**Composition rule (Josh, 2026-07-05): a composition inherits its scheme's
-oracle.** Batching, prompt cache, spec decode, adapters — none of them
-change a scheme's numerics, so composing them with a scheme verifies
-against THAT SCHEME'S oracle, per row. Batched bf16 anchored to mlx-lm
-(B=N token-exact goldens); batched mixed-KV anchors to the L2 oracle the
-same way: copy optiq's per-row quantization mechanics (`install_mixed_kv`'s
-per-layer map + streaming `to_quantized` + RotatingQuantizedKVCache — our
-serial `maybeQuantizeKv` is already the verified op-for-op copy), then
-gate on per-row bit-exactness against it (row cache contents; B=1 through
-the engine vs the mixed-KV golden; B>1 trajectories at the standing
-long-prefix bar). Do NOT invent Lab-style KL/teacher-forced gates for a
-composition of an oracle-backed scheme — the Lab is only for schemes with
-no oracle at all (new quant methods, original kernels).
-
-## 6. The flag surface (end state — every flag with its reason)
-
-User-facing serve/generate flags after the deletion pass:
-
-| flag | why it exists |
+| Dedicated graph → generated, compiled, or quant-specialized graph | Replace one graph registration in a model bundle | Graph ABI, backend, input/output shapes, state ABI, numerical contract |
+| Autoregressive → speculative or diffusion method | Select a method registration and its compatible graph bundle | Session lifecycle, committed-output events, cancellation, accounting |
+| MLX implementation → another runtime | Register a complete backend bundle with its graphs, state, sampling, and resource services | Engine/session contracts; a numerical claim requires separate evidence |
+
+The first operation must require no session, scheduler-policy, route, or
+application changes. The second must require no method-specific session or
+protocol branches. A different numerical runtime cannot reuse MLX tensor
+handles merely because the TypeScript interface matches.
+
+Preserve the existing APIs and numerical defaults during migration. A new
+internal interface does not implicitly expand supported feature combinations.
+Keep one repository and the single-binary distribution. Remote compute, new
+model families, a universal tensor library, a generic graph compiler, and true
+B-wide prefill are separate projects. Training gets its own task interface;
+it does not pretend to be a text-generation session.
+
+This scope boundary does not defer quant-specific optimization. The refactor
+must accommodate the existing quantization campaign and future optimized
+implementations throughout migration. New numerical work has separate paired
+validation, so structural changes cannot conceal a quality or performance
+tradeoff. §12.11 defines the shared frontier scorecard.
+
+Future design question (2026-09-05): consider a monorepo with the inference
+engine kept together as one package, and application modules such as memory
+and the web UI in separate packages. CLI/TUI and training boundaries can be
+considered in that discussion. This is an unplanned idea, not part of the
+current optimization campaign. Before choosing boundaries, map dependencies,
+native assets, builds, tests, and releases; preserve in-process execution and
+the single-binary distribution. Package boundaries should follow demonstrated
+ownership and dependency needs without adding abstractions to the decode path.
+
+### 12.2 Dependency structure
+
+```mermaid
+flowchart TD
+    Apps["CLI / HTTP / Pi / memory / evaluation"] --> Client["CompletionClient and task interfaces"]
+    Client --> Prep["Prompt preparation and request planning"]
+    Prep --> Engine["InferenceEngine"]
+    Engine --> Session["GenerationSession lifecycle"]
+    Engine --> Schedule["Scheduler policy and resource admission"]
+    Session --> Method["InferenceMethod"]
+    Schedule --> Group["SchedulingAdapter / ExecutionGroup"]
+    Group --> Method
+    Method --> Graph["Graph interface"]
+    Method --> State["State and checkpoint interfaces"]
+    Method --> Sample["Sampling / constraints / draft interfaces"]
+    Graph --> Backend["Backend runtime and resource leases"]
+    State --> Backend
+    Sample --> Backend
+    Compose["Composition root and implementation registry"] -.-> Engine
+    Compose -.-> Method
+    Compose -.-> Graph
+    Compose -.-> Backend
+```
+
+Interfaces belong to the consumer's contract layer. Composition roots import
+implementations and supply them explicitly. No service locator inside a token
+loop, import-time self-registration, dependency injection framework, or
+reflection-based constructor discovery is needed.
+
+Proposed homes, introduced only as their migration phase lands:
+
+| Home | Responsibility and permitted dependencies |
 |---|---|
-| `--kv-quant config\|4\|8\|off` | the ONE performance trade-off a user makes: decode speed vs context headroom (+ intelligence-per-byte with mixed). Default off (bf16). Composition (fused vs unfused quantized SDPA) is DERIVED from the scheme's oracle, not chosen. |
-| `--batch N` | cap on concurrent rows (mlx-lm's `--decode-concurrency` twin). Default 32 once Phase 2's B=1 gate passes; `--batch 1` = force-serial / determinism pin. Name kept (Josh 2026-07-05) — semantics change from mode to cap. |
-| `--draft-model`, `--num-draft-tokens` | speculative decoding (mlx-lm parity) |
-| `--adapter`, `--memory-budget`, `--kv-budget`, `--prompt-cache`, `--ssd-cache`, `--expert-offload`, `--force-wire` | capacity/serving features, not numerics levers |
-| sampling flags (`--temperature` etc., HLG family) | request-default sampling |
-
-Everything else becomes either a **kill switch** (env-only, documented for
-debugging: `MLX_BUN_COMPILED_DECODE=0`, `MLX_BUN_COMPILED_GEGLU=0`,
-`MLX_BUN_NO_FUSED_SDPA=1`) or a **Lab flag** (env-only, lives with its bench
-script and expiry — §7). Kill switches are bit-exact by definition (they
-select a slower same-parity path); anything output-changing is Lab.
-
-**Deleted — EXECUTED 2026-07-05** (one funeral each; ~50 files touched, 23
-deleted, suite green):
-- `--fused-decode` / `MLX_BUN_FUSED_DECODE` — 1.00×, forces uncompiled,
-  silent-wrongness footgun. Delete kernel path + flag + backstop throw.
-- `--fused-gelu` / `MLX_BUN_FUSED_GELU` + `MLX_BUN_FUSED_SWIGLU` — +0–1%;
-  the compiled closures already own the fusion win bit-exactly. Delete both
-  custom Metal kernels + flags (git history keeps the source).
-- `--perf-kernel` / `MLX_BUN_PERF_KERNEL` — **DELETE** (Josh 2026-07-05:
-  "we are going to start a full optimization program anyway… we will end up
-  redoing all the work we have currently done given that we now have a
-  different starting point"). The kernel, its frozen-oracle test scaffolding
-  (perf-kernel-oracle.test.ts, freeze-perf-oracle.ts), and the flag all go;
-  git history + the h2h evidence (+6% 12B@16k, −38% e4b@16k, KL WARN) are
-  the breadcrumb. If a flash-decode kernel returns, it's re-derived from the
-  L1 baseline in the Lab under the §7.4 program, not resurrected.
-- `MLX_BUN_CPM5_FAITHFUL` + `FaithfulMiniCPM5` — the default IS the faithful
-  path now; the A/B reference served its purpose.
-- `--l3` as a product mode — the Lab replaces it; the flag now hard-errors
-  with a pointer here. `--l1`/`--l2` stay as documented aliases.
-- Also deleted with the fused-swiglu family: `fused-mlp-kernel.ts` and
-  `steel-linear-kernel.ts` (only reachable through the deleted gates) and
-  the ~17 `scripts/experiments/swiglu-*`/geglu/steel one-off scripts.
-- Training flag sanitization (`MLX_BUN_PERF_KERNEL=0` / `MLX_BUN_FUSED_GELU=0`
-  in trainers/launchers/recipes) — obsolete; the no-vjp kernels it guarded
-  against no longer exist.
-
-**Lab lifecycle rule** (the anti-flag-pile law): every Lab experiment ships
-with (a) a hypothesis stated in frontier-axis terms, (b) a paired A/B bench
-script, (c) an expiry review date. Outcomes: promote to default (with the
-stable-pass numbers) or delete (with a breadcrumb in the design doc). No
-third state. "Off in every regime with no promotion path" — the fused-decode
-condition — is a deletion that hasn't happened yet.
-
-## 7. The frontier program (AFTER stability — the order matters)
-
-Phase gate: none of this starts until §8 phases 0–2 are done. "Now we
-finally have a baseline that is even and proven; investigate THAT."
+| `src/contracts/` | Portable requests, events, results, session/client/task interfaces. No Bun, DOM, MLX, concrete model, or application imports. |
+| `src/engine/` | Session lifecycle, planner, scheduler policy, admission. Depends on contracts and injected ports. |
+| `src/inference/` | Method and graph contracts; reusable AR orchestration, verification, and scheduling adapters. Tensor-facing code is parameterized by a backend binding. |
+| `src/backends/mlx/` | MLX runtime, typed bindings, resource owners, execution groups, state adapters. Wrap existing numerics before relocating files. |
+| `src/model/` | Existing model math behind graph adapters, artifact profiles, loaders. Concrete implementations may use MLX internally. |
+| `src/serve/` | Protocol parsing/framing and transport; calls the completion client. No concrete graph selection or cache surgery. |
+| `src/app/` | Composition roots, runtime bootstrap, server/application assembly. The intentional location for concrete imports. |
+| Existing application directories | Memory, Pi, web, eval, train, quantize consume their declared client/task ports. Move root modules only when their dependencies have been separated. |
+
+Use TypeScript projects plus an AST/import-resolution boundary check. The check
+must follow type-only imports, re-exports, aliases, and literal dynamic imports.
+New forbidden edges fail; temporary legacy edges have a named R-phase owner and
+a deletion gate. Browser contracts cannot import declarations out of pi-web.ts.
+No new concrete model checks in engine/session/planner code. Model-specific
+checks remain legitimate inside the model's own adapter.
+
+### 12.3 Interface inventory
+
+These are responsibility boundaries, not a mandate for one class per row.
+Small stateless ports may be implemented by functions or object literals.
+
+| Interface | Owns | Must not own |
+|---|---|---|
+| `CompletionClient` | Semantic completion request, event stream, terminal result, cancellation | Tensor/state handles, model class selection |
+| `PromptPreparer` | Template/tokenizer policy, token IDs, validated media descriptors, stable-prefix boundary | GPU scheduling or request-global model mutation |
+| `ExecutionPlanner` | Effective immutable policy, implementation selection, capability negotiation, fallback reasons, plan identity | Starting generation or inventing new feature support |
+| `InferenceEngine` | Session creation, runtime coordination, shutdown | Per-family forward math |
+| `GenerationSession` | Lifecycle, committed output, result, cancellation and close | Token-at-a-time assumption, KV offsets, concrete method branches |
+| `InferenceMethod` | AR/speculative/diffusion algorithm and private per-run state | HTTP framing or application sessions |
+| `SchedulingAdapter` / `ExecutionGroup` | Method-specific join/leave and bounded execution work | Queue priority policy or independent memory admission |
+| `Scheduler` | Fairness, active-row cap, queue limits, safe-point dispatch | Cache merge math, sampler math, model-family checks |
+| `ModelBundle` / `GraphFactory` | Artifact construction, graph selection, compatible backend/state services | Rewriting requested sampling or KV policy |
+| `AutoregressiveGraph` | Forward-to-hidden and selective logits projection | Token selection, prefill loop, stopping, persistence |
+| `DenoisingGraph` | Canvas/noise-step operations required by diffusion | Pretend AR cache or fake next-token API |
+| `SpecializedExecutionFactory` | Quant/artifact/device-specific fused execution behind the same method/group contract | Changing quality policy or observable session semantics without a declared plan |
+| `SamplingPolicy` / `SamplingKernel` | Shared processor/mask/logprob/sample/history semantics and backend execution | Transport, independent request defaults |
+| `ConstraintProvider` / `DraftProvider` | Per-run constraint state or proposals under declared capabilities | Mutating target model fields outside a bound context |
+| `StateFactory` and capability facets | Owned state, borrowing, snapshots, batch operations, rollback where supported | Assuming every state is trimmable token KV |
+| `PrefixStore` / `CheckpointStore` | Lookup, retention, integrity, atomic persistence and version checks | Reconstructing arbitrary concrete cache classes itself |
+| `ResourceManager` / `ResourceLease` | Reservations, backing memory lifetime, device completion, release | Model math or hiding synchronization |
+| `ArtifactResolver` / `TokenizerProvider` | Pinned artifact identity, tokenizer/template provenance and loading | Request execution |
+| `TelemetrySink` / `Clock` | Structured observations and time | Scheduling or output decisions |
+| `EngineHost` / `TaskRunner` | Direct or isolated runtime lifecycle; train/quantize task execution | Sending live tensor handles across a process boundary |
+
+Existing `CompletionExecutor`, `CompletionSink`, `DraftProvider`, `DraftSource`,
+model profiles, `KvScheme`, and the cache codec table are starting points.
+Adapt or evolve them; do not retain a second permanent wrapper hierarchy.
+
+### 12.4 Session and graph contracts
+
+Illustrative signatures below fix the intended separation. Supporting record
+types are specified by the semantics in this section and finalized in R1.
+They are not implemented public exports yet.
+
+```ts
+interface InferenceEngine {
+  open(request: GenerationRequest, control: RunControl): Promise<GenerationSession>;
+  close(): Promise<void>;
+}
+
+interface GenerationSession {
+  readonly id: string;
+  readonly events: AsyncIterable<GenerationEvent>;
+  readonly result: Promise<GenerationOutcome>;
+  cancel(reason: CancelReason): Promise<void>;
+  close(): Promise<void>;
+}
+
+type GenerationEvent =
+  | { type: "committed"; sequence: number; tokenIds: readonly number[] }
+  | { type: "progress"; usage: UsageSnapshot };
+
+interface InferenceMethod<Binding> {
+  readonly descriptor: MethodDescriptor;
+  createRun(plan: ExecutionPlan, binding: Binding): Promise<MethodRun<Binding>>;
+}
+
+// Tensor-facing contracts exist only inside a compatible backend bundle.
+interface AutoregressiveGraph<Tensor, State, Inputs> {
+  readonly descriptor: GraphDescriptor;
+  forwardHidden(inputs: Inputs, state: State): Tensor;
+  projectLogits(hidden: Tensor, selection: LogitSelection): Tensor;
+}
+
+interface GraphFactory<Graph, Binding> {
+  readonly descriptor: GraphDescriptor;
+  open(artifact: ResolvedArtifact, binding: Binding): Promise<Graph>;
+}
+```
 
-Ranked by expected frontier shift (memory ↓ / intelligence ↑ first, then
-speed):
-
-1. **Mixed-precision weights** — knapsack `--target-bpw` (ours) vs oQ
-   (calibration-driven sensitivity, omlx-adoption-map §4) vs llama.cpp
-   imatrix as prior art. Gate: perplexity + frozen 6-task eval at equal bpw.
-   The purest beyond-the-frontier play we know: same memory, more
-   intelligence. (arXiv-lens candidate.)
-2. **TurboQuant KV** (Josh) — ORTHOGONAL to mixed precision and composes
-   with it: mixed precision is the ALLOCATION axis (how many bits per
-   transformer layer, optiq's sensitivity map), TurboQuant is the
-   QUANTIZER axis (how a given bit budget encodes each K/V vector, within
-   a layer — rotation-based, online, calibration-free). Composition =
-   sensitivity allocation × better quantizer: more fidelity at the same
-   KV bytes. Architecture: a layer-2 cache format + a layer-1 attention
-   kernel keyed by cache type (same shape as optiq's scheme). Oracle: the
-   paper + its reference implementation anchor the ALGORITHM (read them
-   first, per the oracle rule); the output is genuinely novel (no serving
-   stack ships it) → eval-suite gates. Composes with Phase 3.1's batched
-   quantized buffers.
-3. **Speculative decoding depth** — DFlash/DSpark behind the existing
-   `DraftSource` seam; decode tok/s at zero quality cost when drafts land.
-4. **Per-model graph work from the baseline** — the optimized-verified
-   layer's whole point: unroll a model's flat DAG, find fusion the compiler
-   misses, prove with kernel-trace diffs, promote per model. The perf-kernel
-   root-cause lives here.
-5. **Batch-invariant kernels** (research) — restores determinism-under-load;
-   novel on Apple Silicon.
-
-## 8. Migration phases (each with a hard gate)
-
-- **Phase 0 — measure the gap** — **DONE 2026-07-05** (M1 Max, same server
-  harness both lanes, median-of-5, bf16 KV):
-
-  | model | serial B=1 | batch-lane B=1 | ratio | extra host ms/token | TTFT serial→batch |
-  |---|---|---|---|---|---|
-  | cpm5 | 281.4 tok/s | 128.8 | 0.46× | +4.2 ms | 44 → 55 ms |
-  | e4b  | 62.1 | 44.9 | 0.72× | +6.2 ms | 54 → 121 ms |
-  | 12B  | 29.8 | 25.6 | 0.86× | +5.5 ms | 82 → 251 ms |
-
-  The overhead is a roughly CONSTANT ~4–6 ms per decode step, independent of
-  model size → per-step host tax, not GPU work. cpm5 (which never uses
-  compiled decode) pays it too, so compiled-decode's absence is not the
-  story. Suspects (in dispatch-site order): the `setImmediate` macrotask hop
-  per drive-loop iteration, per-step per-layer `BatchedDecodeMaskCache`
-  construction + mask materialization at B=1 (serial passes caches bare with
-  an empty mask), and the per-token emit path. TTFT gap = solo-prefill +
-  merge + admission hops. Closure worklist = profile these three, in order.
-- **Phase 1 — the deletion pass** (§6): dead kernels, flag surface, docs,
-  Lab scaffolding (move perf-kernel + its bench + oracle tests). Gate: full
-  suite green; naked defaults byte-identical to today's L1 route.
-- **Phase 2 — engine unification** — **DECODE GAP CLOSED 2026-07-05.** The
-  ~4–6 ms/step host tax decomposed into exactly two bugs, both fixed:
-  1. **The pipelined token read enqueued an `astype` kernel** —
-     `prev.toFloat32()` on the int token array created a NEW cast op that
-     queued BEHIND the entire just-dispatched next step on the GPU stream,
-     stalling the "overlapped" read for a FULL step of GPU work every
-     token (this alone was the bulk of the gap; zero host/GPU overlap,
-     proven by NO_PIPELINE ≈ pipelined). Fix: `MlxArray.toIntTokens()` —
-     eval + raw uint32/int32 buffer read, no cast (mirrors serial's
-     `itemUint32`).
-  2. **Per-layer per-step mask/rope wrapper churn**: `#step` rebuilt a
-     `BatchedDecodeMaskCache` + array mask for every full layer every
-     token even when no row had left padding. Fix: the unpadded fast path —
-     bare caches (KVCache.makeMask(1) = the empty mask, scalar rope) →
-     the batch step dispatches the SAME graph the serial loop builds.
-
-  Measured after (B=1 through the batch lane, same SSE harness as Phase 0):
-  cpm5 129→**264** (serial 281 SSE / in-process ratio **0.994**), e4b
-  45→**57.6** (0.93 — remainder = compiled decode, serial-only today),
-  12B 25.6→**29.7** (**1.00**). Batched parity oracles green (11/11
-  applicable; the CPM extend-join golden failure PRE-EXISTS this work —
-  proven by stash/rerun — and is logged as an open item).
-  **Remaining before the `--batch` 32 default flip**: compiled decode
-  inside the B=1 step (e4b's last 7%), prompt cache for batched rows
-  (TTFT: serial hits the cache on repeat prompts, batch re-prefills —
-  e4b 126 vs 54 ms, 12B 246 vs 82 ms), then GATE-B1-SPEED ≥99% on all
-  three + GATE-B1-PARITY (bit-exact vs L1 goldens) + B=2 goldens intact.
-- **Phase 3 — composition**: quantized KV under batching — **P1 LANDED
-  2026-07-05**: src/model/batched-quant.ts (merge/extend/filter over
-  triples + BatchedQuantDecodeMaskCache), scheduler solo-conversion at the
-  serial chunk boundaries, gateway kv-batchability memo (all-full-attention
-  kvConfig batches; uniform/rotating stays serial; scheme-less gateway
-  refuses — never silently drop quantization, the optiq bug class). Gates
-  green: B=1 through the scheduler BIT-EXACT vs the cpm5 optiq golden
-  (decode steps, maxDiff 0); B=2 dynamic join — unpadded row BIT-EXACT vs
-  solo at every step, padded row within the calibrated envelope (5e-2;
-  bf16 same-harness shows ~9e-3 — the pre-existing reduction-order noise,
-  amplified by grid snapping; scripts/experiments/batched-quant-kl-profile
-  .ts). E2E: --batch 2 --kv-quant config on cpm5 = /stats active_rows 2,
-  400 tok @ 240 tok/s aggregate, coherent output. NEXT: milestone 2
-  (batched rotating-quant for gemma configs), prompt-cache reuse for
-  batched rows, chunked prefill interleaving. Gate per feature: parity +
-  no aggregate-throughput regression at B=4.
-- **Phase 3.2 — lone-request = serial (LANDED 2026-07-05)**, three moves:
-  1. **Adopt-don't-copy**: a row joining an EMPTY batch keeps its solo
-     caches as the inners (pointer handoff; the merge copy now runs only
-     when a second row actually joins — incl. a new merge branch that
-     treats an adopted serial RotatingKVCache as the first row). The
-     lone row's caches stay SERIAL-CLASS, which unlocks the next two.
-  2. **Compiled decode at B=1**: the scheduler replays the serial
-     engine's CompiledDecode step when B=1 ∧ unpadded ∧
-     supports(inners) ∧ uint32 pipeline register (same runner, same
-     traces, same MLX_BUN_COMPILED_DECODE kill switch; falls back +
-     disables on error like generate()). Gate: free-running greedy B=1
-     == serial generate() token-for-token on 12B with
-     CompiledDecode.stepsExecuted advancing (tests/batch-scheduler).
-  3. **Prompt cache on the batch lane**: joiners take() the longest
-     usable prefix at admission (suffix-only prefill — multi-turn chat
-     TTFT); rows finishing NEVER-MERGED put() back (exact prompt+fed
-     accounting, offset-checked; reject/batch-drop paths poisoned; SSD
-     retain hooks ride every disposal path). Gate: second identical
-     request reports cached_tokens=prompt-1, step-0 argmax anchored,
-     KL≤1e-2 (tests/batch-scheduler "prompt-cache reuse").
-  **Measured (apple-m1-max/32GB, paired in-process A/B, best-of):
-  B=1-through-batch-lane / serial = cpm5 0.996, e4b 0.992 (was 0.93 —
-  compiled closed the last 7%), 12B 0.993 → GATE-B1-SPEED decode met on
-  all three.** All gated suites green (scheduler CPM+12B, quant gates 1+2,
-  full per-file model-free suite 0 fail).
-  Findings: (a) gate 2's padded-row KL is JOIN-STEP dependent (grid-snap
-  bin flips compound: K=5→4.5e-2, 6→3.5e-2, 7→1.5e-1, 8→1.3e-1; bf16
-  ≤9e-3 at every K; identical on pre-3.2 main; extend vs re-merge
-  byte-identical) — the gate now PINS the join step and calibrates there;
-  padded-row KL is an envelope, the contract is the unpadded row's
-  bit-exactness + model-free triple-surgery byte-identity. (b) v1 gap:
-  an entry take()n by a row that later MERGES is consumed, not re-put —
-  fix is a zero-copy view snapshot at merge (serial's boundary-snapshot
-  trick); noted for milestone 2. (c) monolithic `bun test` can jetsam on
-  a busy 32GB box (largestProcess=bun) — per-file loop is the fallback
-  gate; not a repo defect.
-  **Remaining before the `--batch` default flip (Josh's call)**:
-  milestone 2 batched rotating-quant, then flip.
-- **Phase 3 milestone 2 — batched ROTATING-quantized KV (LANDED
-  2026-07-05)**: src/model/batched-rotating-quant.ts —
-  BatchedRotatingQuantCache subclasses RotatingQuantizedKVCache (attention
-  dispatch untouched) and runs the mlx-lm BatchRotatingKVCache ring
-  mechanics over (packed, scales, biases) triples; RotatingQuantizedKVCache
-  gains temporalView(); scheduler #quantizeSolo converts rotating layers
-  (maybeQuantizeKv dispatch) and #mergeJoiner grows a rot-quant twin
-  branch; gateway #kvBatchable accepts configs naming rotating layers →
-  **gemma's whole kv_config batches** (every shipped kv_config now does).
-  Gates: model-free byte-identity per row vs the serial oracle through
-  ring wrap, B=1+B=2+filter (tests/batched-rotating-quant.test.ts); gemma
-  12B B=2 dynamic join through the real scheduler — unpadded row KL-0 at
-  EVERY step, padded row ≤4e-3 (tests/batched-kv-quant-parity.test.ts).
-  **Findings (hard-won, 2026-07-05)**: (a) THE ROPE-ARRAY STEP-STABILITY
-  CONTRACT — a batched cache's ropeOffsetArr must be STABLE within a
-  decode step and refresh only at releaseRopeArr(): the monolith CAPTURES
-  it pre-update and ropes Q post-update (in-update dispose = use-after-
-  dispose), while the GENERATED specializations RE-READ it for Q
-  post-update (in-update refresh = K and Q roped one position apart →
-  O(10) KL garbage). (b) GENERATED-FILE GUARDS ACCEPT BATCHED SUBCLASSES:
-  the generated forwards' per-layer `instanceof` cache guards pass any
-  batched cache that subclasses a serial class — an all-quant gemma batch
-  therefore decodes through the GENERATED fast path (a feature — B=1
-  twin proved BIT-EXACT through it — but a contract: batched caches must
-  behave serially under re-reads). This is why the bug only appeared when
-  EVERY layer was configured: one bf16 layer anywhere fails the guard and
-  drops to the monolith. Diagnosis artifacts:
-  scripts/experiments/gemma-rotquant-kl-profile.ts (CONFIG_FILTER/
-  ROT_KEEP/ROT_DROP/ROT_IDX bisection) and gemma-twin-b1-repro.ts
-  (TWIN_MASK/TWIN_ROPE toggles — rope isolated in two runs once the
-  B=1 repro existed).
-- **Layer 0 — the SSD tier becomes a property of the store (LANDED
-  2026-07-05)**: PromptCache gains a structural ColdTier and runs the
-  two-tier take (RAM peek vs cold find → zero-copy restore + trim)
-  INSIDE take(), so the batch scheduler restores prefixes from disk at
-  admission with zero scheduler changes (gated E2E on cpm5: demote →
-  disk → batch-lane restore, cached=prompt−1, KL 1.5e-3); onPut drives
-  the write-behind snapshot for both lanes; NEW idle demotion
-  (--ssd-demote-idle, default 300 s with the tier) spills idle entries
-  and FREES their GPU memory — RAM drains between agent bursts, every
-  prefix stays reachable. Also fixes a latent flaw: the old serial flow
-  trimmed the RAM entry before deciding the SSD copy won. The economics
-  (Josh): the SSD competes with RECOMPUTE, not RAM — 12B prefill 168
-  tok/s ⇒ a 30k agent context is ~3 min of prefill vs ~1 s of NVMe
-  restore, and the ratio grows with context. Follow-ups noted: extract
-  the generic blob-store mechanics (fingerprint dirs, atomic writes,
-  scan/recovery, LRU cap) for a second client when one lands (adapters,
-  expert offload); memory-pressure-triggered demotion (vs idle-only).
-- **Prefix sharing v1 (LANDED 2026-07-05)**: PromptCache.take() is
-  NON-CONSUMING — a hit serves zero-copy CLONES (cloneKvCaches views,
-  trimmed to the matched prefix, retain ref-counted so a demoted/evicted
-  donor can never unmap pages a clone still reads) and the donor entry
-  stays put. Safe by mlx's functional cache updates (updateAndFetch
-  reassigns fresh arrays; donation needs refcount 1 and the donor holds a
-  ref), so clones can never mutate donors. put() supersedes same-ns
-  prefix-ancestors AND exact duplicates when the new entry is trimmable
-  (untrimmable extensions preserve their prompt-boundary snapshots — the
-  drift-proof server). This kills the CANNIBALIZATION flaw: agent B
-  borrowing agent A's 2k system prompt used to consume-and-trim A's 10k
-  entry to serve it. Gates: 22 unit tests (donor persistence, clone
-  independence, supersede rules, ref-count ordering) + real-model
-  scheduler gate: B served from A's entry (cached=|SYS|, KL 1.8e-3), A's
-  NEXT TURN still hits its full entry. New-session and restart sharing
-  fall out of the (already non-consuming) SSD tier. **Scope, honestly**:
-  v1 shares the COMPUTE (one prefill serves N agents) and the durability;
-  concurrent batch rows still hold separate physical KV copies — one
-  shared physical prefix across rows is the paged/block-KV frontier item
-  (§9.4), and per §9.5 the disk tier's whole-entry duplication of shared
-  prefixes is the block-granularity revisit trigger.
-- **Runtime isolation P1 (LANDED 2026-07-05, opt-in `--isolate`)**: the
-  engine child = the ENTIRE existing server on a unix domain socket; the
-  parent = a thin reverse proxy (src/serve/isolate.ts) — zero MLX calls,
-  crash isolation (502 + auto-respawn with crash-loop backoff), abort
-  propagation, GET-retry across respawn. Decision recorded in
-  runtime-isolation.md: proxy-the-whole-server instead of the original
-  gateway-as-IPC-client sketch (grammar WASM/vision arrays/sampler
-  closures don't serialize; the inter-process API is the /v1 surface
-  itself — no second protocol). Measured paired (M1 Max, cpm5): proxied
-  197.2 vs direct 197.9 tok/s (−0.4%, noise), TTFT +2 ms, SSE per-token
-  granularity preserved (96 reads/96 tokens), parent 0.6 ms mid-decode
-  (criterion <50 ms). Gates: tests/isolate-proxy.test.ts (model-free fake
-  engine: passthrough, granularity, abort, crash→respawn) +
-  tests/isolate-e2e.test.ts (real cpm5 child). Prefill chunk loops also
-  gained macrotask yields (the last isolation-Phase-1 item). Not proxied
-  v1: /ws/chat (501). NEXT: P2 = child-per-model pool — multi-model
-  switching by SPAWN-OVERLAP (new child loads while the old one serves;
-  drain → demote-to-SSD → park; route by the request `model` field);
-  P3 = GPU-lease unification with quantize/finetune jobs.
-- **Runtime isolation P2 — CHILD-PER-MODEL POOL (LANDED 2026-07-05; task
-  #14 multi-model switching)**: ModelPool in src/serve/isolate.ts — LRU
-  residency over engine children (--model-pool, default 1), request
-  routing by the JSON `model` field on the generation endpoints (STRICT
-  exact-/v1/models-id resolution: fuzzy strings like "gpt-4" keep
-  mlx-lm's ignored-field drop-in semantics), spawn-overlap switching (the
-  new model health-gates while the old serves), and graceful lossless
-  eviction: POST /admin/drain (engine-mode-only endpoint: gateway
-  quiesce + demoteIdle(0) → the whole prompt cache to that model's SSD
-  fingerprint dir) then exit. engineArgvForModel pins --model per child.
-  **Measured (M1 Max, cpm5⇄qwen0.8b, pool 1): switch 1.5 s · switch-back
-  1.2 s · cached_tokens 103/104 on return** — the conversation's KV
-  survived full model eviction and restored from disk instead of
-  re-prefilling (tests/isolate-switch-e2e.test.ts). Model-free pool gates
-  (routing, drop-in ignore, cap-1 drain→demote→evict→respawn):
-  tests/isolate-proxy.test.ts. Defaults unaffected: without --isolate no
-  pool code exists in the request path.
-- **Phase 4 — the frontier program opens** (§7 order).
-
-## 9. Decisions log (was: open questions)
-
-1. **Cap default 8 — FLIPPED 2026-07-05** (Josh; earlier "32" note
-   superseded by the sub-agents rationale, 4–8 agents + optiq's Mac-safe
-   8). Landed after Phase 3 met every gate: lone-request = serial engine
-   (bits + speed + TTFT + prompt/SSD cache), quantized compositions
-   batch. `--batch 1` pins strict arrival-independent numerics;
-   kv-budget admission keeps 8 safe on small boxes.
-2. **Flag stays `--batch`** — no rename; semantics become the cap;
-   `--batch 1` = force-serial pin.
-3. **Perf kernel: deleted in Phase 1** (§6) — the optimization program
-   re-derives from the L1 baseline; no resurrection.
-4. Still open: PagedAttention-style block KV — Phase 3 follow-up or its own
-   design doc when prefix sharing under batching becomes the bottleneck.
-5. **Prefix sharing scope (Josh 2026-07-05)**: sharing must extend to the
-   DISK tier too — the whole-entry cold store duplicates a shared system
-   prompt across N agent conversations (blocks would dedupe; that is the
-   revisit trigger for block granularity). First-class use cases beyond
-   concurrent rows: SPINNING UP A NEW SESSION (fresh conversation, same
-   system prompt → instant prefix) and SERVER RESTART (re-attach without
-   re-prefill) — the shared prefix should be a durable object, not a
-   property of any one conversation's entry.
+`GenerationSession` is one completion attempt, distinct from a Pi conversation
+or application thread. The engine coordinates independent sessions sharing a
+resident model. The method's private run owns prompt position, sampler history,
+pending device work, and caches/canvas; those do not become session fields.
+`RunControl` uses a portable cancellation port; HTTP adapters translate their
+AbortSignal into it. Contract declarations do not pull in browser/server modules.
+
+All methods publish only irrevocably committed token spans. A span can contain
+one token, an accepted speculative group, or a whole completed diffusion
+result. Tentative diffusion canvas revisions and rejected draft tokens are
+private. A non-streaming method may publish nothing until its final output.
+The existing completion sink converts committed tokens to semantic text,
+reasoning, tools, stops, and protocol usage. Tokenizer identity is fixed at
+session open. Task types that cannot provide this token contract use a separate
+task interface rather than fabricated tokens or usage.
+
+Lifecycle is `created → preparing → queued → running → settling → terminal`.
+Terminal outcome is exactly one of completed, cancelled, or failed. Each state
+accepts cancellation. Early iterator return initiates cancellation. Only one
+consumer may read events; progress observation cannot steal output. A bounded
+queue propagates backpressure to scheduling between safe work units. Never
+silently drop output or let a slow consumer retain unlimited device state.
+
+`result` settles once on every path, including no consumer, worker death,
+preparation failure, and cancellation. It contains a discriminated outcome
+with final usage/error data. Events already emitted remain a partial result
+on failure. GenerationRequest declares streaming or collecting consumption at
+open. Streaming starts on demand and has a configured idle-consumer deadline;
+an abandoned/unread stream cancels when that deadline expires. Collecting runs
+without an event reader, reserves output storage against the admitted maximum,
+and returns collected tokens with the outcome. Insufficient collection capacity
+fails admission; it never silently truncates output. R1 fixes the queue/deadline
+defaults with tests. Completed sessions reject further advancement;
+cancel/close are idempotent. Close cancels live work and awaits safe release.
+
+Graph calls remain direct device operations. The interface does not insert
+an await, readback, tensor wrapper, or synchronization for each operator.
+`forwardHidden` plus selective `projectLogits` preserves the existing ability
+to avoid a full prompt-length × vocabulary logits allocation. Compile caches
+belong to the implementation and key on the graph/state/request parameters
+that affect their trace. A graph factory owns and closes resident weights;
+individual runs borrow them through leases.
+
+The selective-hidden/logits interface is the reusable AR path. A specialization
+may instead implement a coarse execution contract that fuses graph, head,
+sampling, or state updates and never materializes those intermediate tensors.
+It must preserve the declared sampler/constraint semantics, accounting, safe
+points, and resource ownership, and pass the same conformance suite. Do not
+require a fused kernel to synthesize hidden/logits objects solely to satisfy
+an abstraction. The planner binds this alternative once through the registry;
+the session and scheduler policy remain unchanged.
+
+R1 must specify graph ABI dimensions, dtypes, mutation rules, position/mask
+inputs, output ownership, and evaluation semantics. Include media inputs,
+hidden taps, and adapter bindings as typed graph capabilities. Move mRoPE,
+adapter selection, and hidden taps out of mutable model-global request fields
+into bound per-run contexts. While legacy models still mutate those fields,
+their adapter requires the exclusive lease and restores them on every exit.
+
+The registration bundle binds graph, tensor type, state factory, sampler,
+device runtime, and optional capabilities together. Use generics and opaque
+handles internally; do not erase them with `any` or public casts. Validate
+backend ID, state ABI, graph ABI, and artifact identity at construction and
+restore. TypeScript compatibility alone does not establish numerical parity.
+
+Specialization keys may include exact artifact fingerprint, weight quant/layout
+and packing version, layer/shape geometry, KV scheme, Mac GPU capabilities,
+context regime, and batch regime. Dispatch is planned or bound at a safe shape
+transition, with a validated compatible fallback outside the optimized range.
+It must not become a collection of per-token environment checks. Fused paths
+may retain optional features only when implemented; capability negotiation
+selects another faithful implementation when the requested composition needs it.
+
+### 12.5 Capability negotiation and planning
+
+Replace scattered feature booleans with versioned capability descriptors and
+typed capability interfaces. Obtain a complete optional facet once during
+planning, such as rollback or batch join. The core never probes a handful of
+optional methods and hopes they form a complete implementation.
+
+Evaluate the whole composition: artifact/profile, graph, method, KV scheme,
+sampler, seed, constraints, adapters, media, checkpointing, and batching.
+Two independently supported features do not prove their combination works.
+Each supported composition links to its conformance/oracle test cell.
+
+Within a compatible composition, selection may use a measured execution recipe
+for the artifact/quant/device/workload. A recipe binds the entire implementation
+bundle, its quality contract, and resource model. It can choose a quant-specific
+fused method rather than a generic graph. Record provenance and the evidence
+behind automatic selection. Never optimize by changing the requested artifact,
+quant, or quality tier without an explicitly permitted product policy.
+
+Planning returns supported, refused, or an explicit policy-permitted fallback
+with reasons. Distinguish required request behavior from preferred acceleration.
+Retain existing intentional fallback behavior during migration and record it
+in the plan. Never silently change a required scheme, constraint, seed, or
+adapter to gain batching. Preserving existing support also means retaining
+explicit refusals. Fallback ordering is deterministic and tested.
+
+The plan contains artifact and tokenizer identities, implementation IDs/ABIs,
+oracle tier, actual algorithm, resolved sampling/stop policy, state layout,
+cache/checkpoint policy, resource estimates, and allowed execution shapes.
+Dynamic batch membership is runtime state within those allowed shapes. Record
+it in traces; do not freeze a future arrival schedule into admission.
+
+Normalize environment/CLI/request configuration at the boundary. Capture an
+immutable effective snapshot per engine/run. No runtime configuration lookups
+inside forward/decode after binding. Experiment commands construct separate
+bindings with explicit snapshots. Diagnostics may observe a running request
+but cannot change its numeric policy halfway through execution.
+
+Admission is two-stage: validate metadata and reserve a conservative preparation
+budget before compiling grammar or running media towers; then refine the
+reservation using prepared sizes before execution. Count shared resident
+weights once, per-run state, draft resources, media/preparation transients,
+compiled state, pending output, and retained snapshot/write-behind buffers.
+Reservation ownership transfers across stages and releases on every failure.
+Keep prompt/context clamping semantics and aggregate KV limits consistent with
+current policy. A rejected prompt must not first allocate its full GPU payload.
+
+### 12.6 State, ownership, and persistence
+
+Resource rules apply to success, partial construction, eviction, cancellation,
+rollback, shutdown, and native failure:
+
+- A resource has one owner, with explicit borrowed views or retained leases.
+  Ownership transfer is a checked operation; a resource cannot be disposed by
+  both the old and new owner. Close is idempotent and attempts every release
+  even if one cleanup fails, preserving the original execution error.
+- Device handles and host/mmap backing memory have distinct lifetimes. A native
+  completion fence governs final backing release. JS scope exit or array
+  disposal alone is insufficient. Never restore a JS callback destructor on
+  a Metal completion thread. Finalization remains a diagnostic/backstop.
+- Replace `stateNeedsDispose` with an explicit owned state-view lease. Snapshot
+  creation returns a stable owner; later writes to the donor cannot change it.
+  Prefix lookup returns retained state and its exact logical coverage.
+- State capability facets include snapshot, prefix reuse, trimming, batch
+  merge/extract/filter, and speculative transaction. Recurrent state supports
+  exact restore/replay where proven; diffusion state can expose a different
+  snapshot without claiming a token prefix or trim operation.
+- A speculative transaction includes every participating layer and draft-side
+  state. Commit or rollback leaves one consistent accepted prefix. Partial
+  failure invalidates the run; it cannot return uncertain state to PrefixStore.
+- Storage policy is separate from layout codecs. Provider-owned codecs bind
+  formats to a compatible implementation. Keep streamed tensor writes,
+  corruption checks, atomic publication, byte caps, and bounded spill queues.
+  Do not replace the current copy-restore with unsafe mmap borrowing for style.
+
+Prefix identity and generation-resume identity are separate records. Prefix
+identity covers artifact content, tokenizer/template interpretation, adapters
+and their revisions, effective numeric scheme, graph/state compatibility, and
+media content when reuse is supported. Generation identity additionally covers
+the complete normalized request, stop strings, sampling/seed policy, method,
+constraints, generation limit, and resume schema. Canonicalize map keys while
+preserving semantically ordered arrays. Store the descriptor as well as its
+digest, and validate it on restore.
+
+A resume snapshot includes consumed input coverage, committed completion count,
+pending token/device work in a restorable form, RNG/history position, and any
+method/constraint state needed for exact continuation. Physical cache offset,
+committed output, and bytes delivered to a client are different quantities.
+Snapshots are legal only at declared safe points. Unsupported exact-resume
+compositions remain refused or disabled as documented.
+
+Retries must distinguish a new generation from a resumed attempt. Preserve the
+current replay contract through its compatibility adapter; explicitly design
+any future idempotency/resume field before changing the public API. Never claim
+exactly-once delivery across a broken stream or automatically replay tool side
+effects. Cold-prefix misses may recompute; an incompatible exact-resume request
+must report the mismatch rather than invent a continuation. Version the format
+and define read-old/write-new compatibility or explicit invalidation per change.
+
+### 12.7 Scheduling and inference methods
+
+Keep policy independent from mechanics. Scheduler chooses eligible work,
+fairness, queue bounds, active-row limits, and resource reservations. A method's
+scheduling adapter creates execution groups and implements join/leave/advance
+over its own state. Native batch merge and filter stay inside the MLX adapter.
+Scheduler must not import KV classes or perform a per-layer tensor loop.
+
+An advancement is a bounded safe unit, not necessarily a token: one prefill
+chunk, a pipelined decode step, one verify round, or one denoising iteration.
+Budgets describe admitted resources and work hints separately. A synchronous
+kernel cannot be interrupted; cancellation is honored at the next safe point.
+Keep pending MLX work between advancements to preserve decode overlap. No
+mandatory GPU synchronize or full logits readback at the scheduling boundary.
+
+An execution group owns its resident run handles exclusively. Join/leave
+validates compatibility before state mutation. Group-wide failure either
+proves unaffected rows safe or fails the entire group; it cannot publish
+possibly corrupted snapshots. Slow readers stop receiving dispatch while their
+reservations remain accounted for; cancellation releases them after pending
+device work is safe. Queued cancellation removes the waiter promptly so a dead
+serial request cannot keep batch admission in drain mode.
+
+Migrate these existing algorithms behind the contracts:
+
+| Method/component | Preserved behavior |
+|---|---|
+| AR method | Shared prefill program, existing tail split/snapshot boundaries, sampler order, pending-token pipeline |
+| Speculative method | Existing verifier and DraftSource variants, actual proposal length, target taps, per-step RNG, recurrent rollback, source-specific prefill convention |
+| Fill/grammar acceleration | Explicit assert/verify proposal policy and constraint state; preserve current exclusions and fidelity gates |
+| Diffusion method | Existing denoising behavior with private canvas state and committed output only; no fabricated AR state |
+| Continuous execution adapter | B=1 adopted-cache path; existing merge/extract/filter operations and per-row sampler history |
+| Quant-specific execution | Fused or precompiled work units with the same observable semantics and explicit numerical tier; no forced intermediate allocations |
+
+Shared AR prefill does not imply full-prompt forwarding for every method.
+Native GLM MTP already declares a different prefill convention. Preserve it as
+data in the method plan. Likewise, unifying solo prefill code does not establish
+bit identity with B-wide prefill or broaden the batch support matrix.
+
+The shared prefill program is the canonical policy for segmentation, snapshots,
+and cancellation. A measured specialization can execute that program through
+a different kernel schedule while satisfying its numerical and resource
+contract. Share the decisions; do not prohibit a faster executor merely because
+its implementation cannot reuse a helper function.
+
+### 12.8 Applications, library, and process boundaries
+
+Keep the existing token-to-semantic completion pipeline. `CompletionClient`
+is the application/transport boundary; `GenerationSession` is the worker's
+method-neutral token boundary. An ordinary OpenAI-compatible remote endpoint
+can implement CompletionClient without exposing token IDs or local checkpoints.
+It cannot masquerade as a fully capable local GenerationSession.
+
+Provide direct and isolated CompletionClient/EngineHost implementations. Evolve
+the existing `src/serve/isolate.ts` EngineChild, ModelPool, health gating,
+restart policy, and HTTP-over-Unix-socket proxy. Isolation already exists.
+Do not build a competing supervisor or change the wire protocol merely to
+match internal interfaces. Current whole-server isolation remains supported
+while CPU application services are separated from worker-owned compute.
+
+Worker requests contain portable policy and media descriptors/bytes. Grammar
+controllers, media tensors, sampler closures, state handles, and device leases
+are created in the worker. A crash fails in-flight completion attempts, resets
+worker-owned residency, and preserves parent health/application state. Existing
+GET/HEAD retry behavior does not authorize automatic generation POST replay.
+Cross-process resume requires the explicit checkpoint/replay contract above.
+
+One resource coordinator covers inference and managed training/quantization
+jobs that share a runtime host. Preserve the existing jobs subprocess design
+and persist terminal job state. Do not describe this coordinator as controlling
+unrelated external Python processes or a separately launched training run.
+
+The high-level library opens a runtime asynchronously before importing/loading
+native implementations. Root high-level imports must work without installed
+MLX. Keep advanced native/model access in explicit low-level exports with clear
+ownership and bootstrap requirements. Compatibility wrappers preserve current
+exports until a deliberate versioned removal; they may not maintain duplicate
+generation loops. Validate the packaged binary and native sidecars, not only
+source execution, before release.
+
+### 12.9 Current-to-target migration map
+
+| Current code | Target and removal condition |
+|---|---|
+| `src/server.ts` runGeneration | Method/cache orchestration moves to inference/engine services; server retains composition and route dispatch |
+| `src/generate.ts` | Compatibility entry around the AR method/session; one shared prefill and pending-token implementation |
+| `src/serve/batch-scheduler.ts` | Split queue policy from MLX execution-group mechanics; delete duplicate prefill after parity gates |
+| `src/serve/generation-gateway.ts` | Planner plus scheduling/resource coordination; remove late feature eligibility and concrete model probes |
+| `src/serve/{request-plan,completion-executor,completion-sink}.ts` | Evolve into the request/session boundary and semantic adapter; retain tested accounting/framing behavior |
+| `src/model/{factory,profile}.ts` and concrete graph files | Model bundle registration and graph adapters; replace the central concrete RuntimeModel union in generic consumers |
+| `src/model/gemma4-base.ts` cache classes | Shared state contracts leave the Gemma module; implementations remain behind MLX state adapters until moved independently |
+| `src/{prompt-cache,kv-store,ssd-cache}.ts` | Prefix/checkpoint policies use provider codecs and explicit retained snapshots |
+| `src/{runtime-config,mlx/array,mlx/ffi}.ts` | Bound immutable configuration, explicit runtime opening, native resource owners |
+| `src/spec/` and `src/fill/` | Typed method/draft/constraint capabilities; existing algorithms and oracle fixtures retained |
+| `src/pi-web.ts`, Pi provider files, `src/web/src/` | Shared portable protocol declarations, client adapters, one provider-wiring implementation |
+| `src/memory/`, `src/eval/`, `src/train/`, `src/quantize/`, `src/jobs/` | Completion/task clients and explicit artifact/resource services; preserve task-specific semantics |
+| `src/serve/isolate.ts`, `src/index.ts`, `src/cli.ts` | Host/client adapters and composition roots; async high-level bootstrap and compatibility exports |
+
+File moves follow responsibility extraction. Every temporary adapter has a
+deletion criterion. Do not combine renames, numerical rewrites, and protocol
+changes in one patch. Preserve failure reproduction fixtures through migration.
+
+### 12.10 Implementation stages and dependencies
+
+Each stage is one or more reviewable PRs. No calendar promise is made before R0
+establishes the fixture and machine prerequisites. Checkboxes are in PLAN.md.
+
+**R0. Stabilize and record the baseline.** Pin the compiler and align local/CI
+typechecks. Reproduce and fix checkpoint stop-policy identity and serial
+cancellation with targeted tests. Diagnose the recorded Qwen cache-reuse crash
+before migrating its state implementation. Record exact revisions, existing
+support/refusal cells, public defaults, ABI/schema versions, and resource
+ownership. Capture quiet-machine baselines using existing bench tooling.
+Structural refactoring and model-free validation proceed while target artifacts,
+quiet hardware, or crash diagnosis are unavailable. Preserve existing numerical
+and state behavior; changing the affected state implementation or promoting a
+default still requires its relevant correctness and performance gates.
+Exit: fixes independently verified, baseline manifest available, missing
+machine/oracle cells explicitly listed. No unverified previous-review claim
+becomes a passing gate merely by entering this plan.
+
+**R1. Land contracts and prove method independence.** Depends on R0's inventory.
+Create portable interfaces and dependency checks. Implement scripted AR-like
+and final-output-only fake methods with different private state. Run the exact
+same session lifecycle suite against both, including cancellation, failure,
+backpressure, no-consumer behavior, and close. Introduce a legacy adapter around
+the current executor so existing APIs still run. Exit: engine/session modules
+import no concrete graph/native module; switching fake methods edits registration
+only. Finalize the illustrative signatures through these executable contracts.
+
+**R2. Bind real graphs and backend capabilities.** Depends on R1. Wrap existing
+models with GraphFactory/ModelBundle bindings, beginning with one dedicated
+graph and its existing generated counterpart. Bind sampler/state/runtime types,
+resource estimates, artifact profile, media inputs, taps, and adapter context.
+Exit: both real graphs run through the same method/session without consumer
+edits; direct forward logits match their pinned oracle; backend/state mismatch
+fails before allocation. Keep old models reachable through adapters until their
+family-specific tests have migrated.
+
+Also adapt an existing quant-specific path through its bundle and verify that
+the interface adds no obligatory dequantization, tensor copy, intermediate
+allocation, or synchronization. Validate the coarse fused-execution contract
+before standardizing the reusable graph ABI.
+
+**R3. Establish resource and state ownership.** Depends on R2 and the relevant
+R0 crash fix. Introduce explicit state views, retained snapshots, device-backed
+leases, and complete rollback facets. Evolve the codec table into provider-bound
+codecs. Validate prefix and resume identities, including stop policy and adapter
+revision. Exit: injected failures release all owners; donor/clone isolation,
+ring wrap, recurrent rollback, cold restore, and restart fixtures pass. No new
+GC-dependent lifetime or unsafe mmap release is accepted.
+
+**R4. Resolve plans and reservations once.** Depends on R1–R3. Move speculation,
+fill, compilation, cache bypass, and capability selection into one planner.
+Use exact resolved plan data for execution, metrics, and checkpoint identity.
+Move native media/constraint allocation behind preparation admission. Capture
+configuration per binding. Exit: table-driven tests cover every shipped
+composition and refusal; lane reports agree before/after execution; engine code
+does not reread mutable global configuration. Existing fallbacks stay explicit.
+
+**R5. Share AR prefill and session execution.** Depends on R2–R4. Extract a
+step-able prefill program preserving chunk/eval/quantization/snapshot/final-token
+order. Route serial execution and current solo batch-prefill through it. Move
+pending-token, history, and checkpoint-safe-point state into the AR method run.
+Thread cancellation through queue, preparation, prefill, and decode. Exit:
+duplicate prefill loops are removed, phase-specific abort tests release the
+lease, and cold/warm/restored logits plus streamed results match the baseline.
+This is the Maintainability prefill item; Phase 18 S1a remains separate.
+
+**R6. Adapt other methods and extensions.** Depends on R3–R5. Move the existing
+speculative verifier and all retained DraftSource implementations behind typed
+target views. Adapt fill/grammar acceleration and diffusion without expanding
+their support. Exit: the same session suite passes for each; accepted/rejected
+trace, pending-token accounting, stop handling, and recurrent rollback match
+their method oracle. No session branch identifies AR/spec/diffusion by name.
+
+**R7. Separate scheduler policy from batch execution.** Depends on R5–R6.
+Move merge/extract/filter and B=1 adoption into execution groups. Centralize
+fairness, drain policy, queue bounds, reservations, and cancellation. Exit:
+dynamic join/leave, uneven rows, slow consumers, group failure, mixed-KV,
+recurrent state, and ring-wrap cells pass wherever supported; unsupported
+compositions retain their recorded placement/refusal. Preserve pipeline overlap.
+Do not merge true B-wide prefill or new concurrency features into this stage.
+
+**R8. Move applications onto client/task interfaces.** Depends on R1 and R4;
+runtime cutover follows R5–R7. Separate shared web/Pi protocol declarations,
+consolidate Pi provider configuration, and route memory/eval through clients.
+Train/quantize retain task ports and dedicated numerical graphs. Add async
+high-level library bootstrap and compatibility exports. Exit: CPU-only clients
+and high-level package imports need no MLX; API/CLI/UI fixtures preserve current
+behavior; task progress/cancellation and artifact semantics remain intact.
+
+**R9. Adapt isolation and resource coordination.** Depends on R7–R8. Implement
+EngineHost over existing isolation/model-pool code, separate parent application
+state from worker compute, and coordinate managed GPU jobs. Exit: worker startup
+failure, death mid-stream, bounded restart, model eviction, disconnect, and
+shutdown tests pass; no automatic POST replay or native object serialization;
+direct/isolated clients share the semantic contract. Compare transport overhead
+before any default change. Existing `--isolate` behavior remains available.
+
+**R10. Cut over, remove duplication, and close.** Depends on R0–R9. Default to
+the new engine only after the full supported matrix and performance gates pass.
+Delete legacy executor branches and temporary import exceptions. Review generated
+twins, diffusion, curve/HLG, expert offload, paged KV, DSpark variants, compiled
+decode, and Pi wiring using the existing D6 requirement: demonstrated use or
+measured value, otherwise an explicit deletion decision. Retained experiments
+register through declared interfaces. Update reference docs and package exports
+in the same change as any public difference. Exit: replacement exercise below
+passes, no duplicate engine remains, binary packaging works, and open-phase
+blocks close without archiving this still-canonical architecture document.
+
+### 12.11 Validation and release gates
+
+**Contract tests on every PR.** Run without native MLX: scripted alternate
+methods, lifecycle, ownership transfer, capability composition/refusal, stable
+identity, queue fairness, resource accounting, event ordering/backpressure,
+partial result/error behavior, and dependency boundaries. Add graph adapter
+type-tests that reject incompatible backend/state bindings. Existing protocol
+and UI tests remain their behavioral reference.
+
+**Numerical and state matrix on available Apple Silicon machines.** Use the
+pinned oracle and existing machine-layered goldens. Cover dedicated/generated
+graphs, dense/MoE/recurrent/rotating state, bf16/uniform/mixed/TurboQuant where
+supported, cold/warm/SSD/resume, B=1 and matching-B concurrent composition,
+constraints, seeds, logprobs, adapters, media, speculation, and fill. Use targeted
+pairwise cases plus every known failure reproduction; do not imply every
+Cartesian combination is supported. Diffusion keeps its distinct oracle gate.
+Exact cells remain exact. Existing tolerance cells require their recorded
+justification; a refactor cannot loosen a gate to get green.
+
+**Lifecycle sequences.** Exercise cancel before preparation, during admission,
+between prefill chunks, after dispatch but before readback, during verify, during
+checkpoint write, and on slow/broken streams. Check one terminal outcome and
+eventual reservation release after device completion. Restore-and-continue must
+equal uninterrupted execution for the exact supported resume policy. Test
+incompatible/corrupt checkpoints and failed cleanup. Measure steady memory
+after synchronization across repeated warm and aborted requests.
+
+**Frontier scorecard.** Speed, quality, and size are independent axes. Keep a
+Pareto set rather than collapsing them into one weighted score that hides a
+regression. Every optimization names its target Mac memory budget, workload,
+context, and concurrency before measurement.
+
+| Axis | Required evidence |
+|---|---|
+| Speed | Short/long-context TTFT, sustained decode, inter-token latency, concurrent throughput; include startup/compile and cold/warm behavior separately |
+| Quality | Oracle identity for unchanged numerics; for changed quants/numerics, KL as a screen plus frozen task evaluations, raw-prompt/EOS checks, and representative multi-turn agent/tool outcomes |
+| Size | Actual artifact bytes and effective bpw including scales/codebooks/sidecars; peak unified-memory footprint at the declared context, including KV, draft model, workspaces and retained snapshots |
+| Practical experience | Largest useful context that fits, responsiveness during long work, completion correctness, restart behavior, and a reproducible local task with visible user benefit |
+
+A frontier improvement improves at least one axis without a meaningful loss on
+the others at the declared measurement precision. A useful tradeoff can also be
+a new non-dominated point; describe the lost/gained quantities explicitly and
+apply the intended quality floor. A smaller model that causes an EOS/task
+failure is not a quality-preserving win because its KL or aggregate benchmark
+looked similar. Neither faster kernels alone nor smaller on-disk weights alone
+establish a better product experience.
+
+Compare to the best relevant measured recipe, including the current shipping
+bundle and credible same-budget alternatives, not an intentionally weak
+baseline. Publish artifact provenance, input corpus, evaluator, uncertainty,
+machine and memory accounting. Keep training/tuning examples separate from the
+held-out promotion set. Include ordinary laptop use and cold-start experience
+in product evaluation while keeping quotable paired microbenchmarks quiet.
+
+Optimization cycle: choose a bottleneck and target budget → propose a quant,
+layout, graph/kernel or execution recipe → establish its numerical/quality
+contract → measure the full scorecard → retain the non-dominated recipes →
+register validated selectors for the appropriate Macs and workloads. Negative
+findings belong in the topic doc; one-off research scripts are then removed.
+The existing TurboQuant, decode-speed, KV and speculative programs own their
+experiments. This architecture supplies the common registration and evaluation
+contract without duplicating their backlogs.
+
+Pure refactor PRs may be frontier-neutral. V2 product promotion requires at
+least one reproducible frontier advance plus a representative local workload
+that demonstrates the benefit. Until measured, it is an objective, not a claim.
+Supporting multiple useful quant-specialized implementations is intentional;
+remove duplicated lifecycle policy, not a demonstrated numerical advantage.
+
+**Performance regression guard.** Record host/chip/RAM, revisions, artifact, effective plan,
+concurrency, context, cache warmth and native runtime. Compare paired old/new
+runs on the same quiet machine with existing tooling. Track startup, short/long
+TTFT, inter-token latency, decode throughput, concurrent throughput/fairness,
+peak memory, cancellation delay, and checkpoint stall. Predeclare a 3% median
+regression investigation trigger for TTFT/decode on each representative cell, with
+enough repetitions to distinguish it from noise; noisy results are unresolved.
+Even smaller repeatable losses require a fix or an explicit measured benefit
+elsewhere. No unexplained growing memory retention or disabled pipeline overlap.
+The trigger prioritizes investigation; it is not an acceptable abstraction tax,
+does not redefine a Pareto improvement, and does not permit quality degradation.
+Report on both dev machines before final cutover; do not compare bits across
+their different metallibs. Curated numbers go only in benchmarks.md.
+
+**Replacement exercise.** Before closure, replace a real compatible graph with
+another registered graph, run a quant-specific coarse/fused implementation
+without adding session branches, then swap the method for a final-output-only test
+implementation, then run the client through direct and isolated hosts. Session,
+scheduler-policy, protocol, and application code must remain unchanged. Changes
+belong only to the implementation/bundle registration and its tests. This is the
+acceptance test for Josh's "swap one piece" requirement.
+
+### 12.12 Rollout rules and remaining design decisions
+
+Each stage routes a bounded set of existing configurations through a temporary
+implementation selector in the composition root. Preserve the old implementation
+only for paired comparison and rollback while that stage is open. Do not add
+permanent user modes for intermediate architecture. Rollback selects a coherent
+bundle and compatible persistence schema; never mix old graph state with a new
+codec because both happen to implement an interface.
+
+Decisions still requiring implementation evidence have explicit owners:
+
+| Decision | Owner / decision gate |
+|---|---|
+| Exact event buffering and collect-mode limits | R1 contract tests; no-consumer and slow-consumer behavior must be bounded |
+| Smallest graph ABI that covers existing taps/media/compiled paths | R2 real graph replacement plus per-family adapters |
+| Fence/lease implementation and old checkpoint compatibility | R3 native lifetime tests and restore/restart fixtures |
+| Which batch mechanics can share a method adapter | R7 parity and pipeline-overlap measurements |
+| Whether finer worker separation should become a default | R9 crash containment, packaging, and transport measurements |
+| Retirement of each experimental/duplicate implementation | R10 D6 usage or paired measurement; no deletion by interface preference |
+| Which optimized recipe becomes the automatic choice for a Mac/workload | R10 full frontier scorecard and held-out quality gate; no universal winner assumed |
+
+R0 fixes are small independent changes. R1–R4 establish and validate the
+contracts. R5–R7 perform the execution migration. R8–R10 complete applications,
+isolation, and deletion. Contract drafting must lead to the first real graph
+replacement in R2; it cannot grow indefinitely without that working proof.
+
+### 12.13 Implementation status (2026-09-05)
+
+Branch: `refactor/interface-engine-v2`. The comparison baseline is
+`6d45ca1e17e825c48af9b246a37486380117f640` (`origin/main`, fetched again
+2026-09-05). Its reference docs and tests freeze the pre-refactor public
+defaults, supported combinations and refusals; use `git show <revision>:<path>`
+to inspect that baseline without consulting a moving branch.
+
+| Baseline dimension | Frozen source / compatibility rule |
+|---|---|
+| Flags, defaults and routes | Baseline `docs/reference/{server-config,server-api,cli}.md`; no new architecture mode becomes a default in this phase |
+| Families, quants and feature support | Baseline `docs/reference/models.md`, model profiles and parity tests; the locally available published Qwen artifacts supply compatibility evidence only |
+| Native oracle and machine | Baseline `docs/reference/environment.md` plus its machine-layered golden manifests; all branch numerical comparisons reported here use the M1 Max 32 GB and the pinned venv |
+| Toolchain | Bun 1.4.0 (`34cbb9a40`), TypeScript pinned to 6.0.3; Python MLX/Metal 0.31.2, mlx-lm 0.31.3, mlx-optiq 0.2.15 |
+| Graph/state representation | Existing MLX hidden `[batch, positions, width]` and legacy cache arrays; newly declared `mlx-hidden-bsh-v1` / `legacy-cache-array-v1` describe those representations without converting them. Denoising uses its distinct `mlx-denoising-v1` graph ABI |
+| Persistence | SSD prefix format 3 is retained; legacy codec identity remains readable. Generation resume identity advances from the baseline's unversioned policy hash to version 4, including stops, adapter content, implementation and resolved policy |
+| Ownership | Resident weights remain borrowed; request caches, rollback scratch, native preparation and retained prefix backing release at their existing device-safe boundaries, now expressed through owners and leases |
+| Performance | Baseline/candidate paired diagnostics exist for available artifacts; quiet-machine preflight did not pass, so none establishes a quotable regression result or frontier advance |
+
+Outstanding evidence is explicit: Josh's designated quant artifacts, the M4 Pro
+24 GB matrix, quiet paired performance, unavailable diffusion/drafter weights,
+the Qwen 4B B=2/persistence oracle fixtures, and reconciliation of the reported
+other-machine Metal-memory fix. The crash is diagnosed as a native allocation
+failure; this branch preserves the affected cache math. These missing cells
+do not prevent interface work. They keep the full frontier scorecard and
+production acceptance gate open after the branch implementation is complete.
+
+R0 fixes pin TypeScript 6.0.3 for
+local/CI checks, add stop strings and a version to generation-checkpoint
+identity, remove cancelled serial waiters immediately, and check cancellation
+between AR/spec target-prefill chunks and committed-token boundaries.
+
+`src/contracts/generation.ts` defines portable planner, method/run, session,
+output, result, cancellation, and timer ports. `src/engine/` implements lifecycle
+and delivery without model or runtime imports. The dedicated TypeScript project
+has ES2022 only and no ambient runtime types; an AST/resolved-import test also
+checks type imports, re-exports, and dynamic imports against dependency layers.
+
+Streaming starts on the first reader demand. Defaults are 32 queued events,
+256 queued token IDs, 20 top-logprob entries per token, and a 30-second
+idle-consumer deadline. Large committed spans split under backpressure.
+Collecting reserves a Uint32 buffer before method creation, with an aggregate
+65,536-token active-collection ceiling; terminal output ownership transfers to
+the caller. Collection currently returns IDs only; logprobs require streaming.
+Cancellation and early iterator return wait for run cleanup. Failure outcomes
+retain the execution error if cleanup also fails. These are internal defaults,
+not new server flags.
+
+HTTP completion execution now uses the shared session by default.
+`createSessionCompletionEngine` delivers tokens directly to the existing
+stop/tool parser through callback output, avoiding a second queue. The callback
+is awaited and can request a normal stop; cleanup completes before the result
+settles. Streaming/collection library consumers retain their bounded delivery.
+AR, speculative and denoising methods honor callback stops. Denoising now settles
+its emitted-token count when delivery closes before the completed canvas ends.
+`src/backends/mlx/text-engine.ts` is the data-only token API. It snapshots requests
+before demand-start and shares `createCompletionMethod` with HTTP. Native media,
+adapters and grammar still require prepared inputs; their HTTP requests now run
+through the same session lifecycle. The temporary text-only legacy engine is gone.
+
+`ModelServingBinding` owns the gateway binding, serial execution construction,
+prompt/media preparation, SSD restore, signal evaluation, embedding execution and
+model diagnostics. The server consumes model metadata and this binding.
+`MlxGatewayBinding` owns compatibility decisions and batch-group construction;
+method IDs are implementation-owned strings. A model can expose several methods
+with unrelated private state. Existing classes enter through the MLX binder;
+replacement implementations supply the port directly. `loadContext` selects an
+engine-owned host implementation before either default weight loader runs.
+The HTTP replacement test selects a synthetic exact artifact registration and
+runs two custom methods with no concrete model, forward function or cache factory.
+This proves the construction-to-session path, not quant quality or performance.
+
+`src/inference/graph.ts` declares the backend-bound graph ABI.
+`src/backends/mlx/graph.ts` binds synchronous/streamed forwards once and selects
+hidden positions before vocabulary projection. `src/backends/mlx/autoregressive.ts`
+defines one `MlxAutoregressiveBinding` for the graph, cache construction, media
+forwarding, adapter state, memory accounting, and optional compiled/fused decode.
+`generateAutoregressive()` consumes that interface without requiring a
+`RuntimeModel`. The existing `generate()` API adapts AR models into it, preserving
+the serial loop's sampling, prefill chunks, lazy evaluation, and decode pipeline.
+No tensor wrapper, device readback, or synchronization was introduced.
+
+Concrete Gemma compiled-decode selection and rollback-based fallback now live
+in the legacy binding. A replacement supplies its own decoder; the loop never
+infers one from the model family. A declined step must leave state unchanged.
+Unrecovered errors propagate without retry. Bindings must declare the MLX
+hidden-state ABI and legacy `Cache[]` ABI; mismatches fail before cache allocation.
+
+The host now captures AR/denoising and speculative bindings once. Compiled
+replay permission and grammar jumping are resolved alongside placement and
+included in checkpoint identity. Serial execution consumes that policy; B=1
+batch replay also honors it, with dynamic cache geometry still allowed to
+decline replay transactionally. A lazily created batch group inherits the
+gateway's runtime snapshot. Composition and snapshot tests cover later config
+changes, adapters, paged/media fallback, speculation, denoising and logprobs.
+Legacy kernel flag reads now resolve through an execution-local runtime snapshot.
+AR/denoising generator resumes and cleanup, the speculative verifier and its
+memory scope, and each batch driver retain that snapshot across awaits. A later
+host configuration change cannot alter a running binding's kernel policy.
+Nested/concurrent scope tests and native binding tests cover early close,
+verification callbacks and lazy batch construction. The scope adds no tensor
+copy, evaluation or device fence; its performance remains part of the paired gate.
+Rejected initial state releases owned caches while preserving borrowed caches.
+Decoder cleanup completes before cache disposal, including early iterator return.
+If execution and decoder cleanup both fail, the aggregate retains both errors.
+The legacy descriptor is not a persistence identity. Rich media remains a native
+preparation input; tap capabilities and provider-bound persistence are described
+below. A different cache ABI requires its own compatible method/backend binding.
+
+Serial cache lookup, checkpoint replay/persistence and prompt-boundary snapshots
+now live in `src/backends/mlx/serial-executor.ts`. The HTTP server injects a bound
+generator/verifier, cache constructor, prefix/checkpoint stores, clone function
+and adapter namespace service. Native Qwen media context is installed/restored
+by its compatibility binder. The executor has no concrete model dispatch;
+the existing gateway still owns scheduling and exclusive execution.
+Native input ownership starts before cache lookup/construction. A failed lookup,
+cache constructor or trace hook releases prepared grammar/media and any acquired
+cache state; one failed destructor does not skip sibling releases or replace
+the execution error. Checkpoint replay checks cancellation before each saved
+token and retains its durable checkpoint when interrupted.
+Failed prefix trimming releases acquired clone/restored state. Prefix insertion
+validates cache metadata before adopting ownership; eviction cleanup errors are
+reported after adoption without inviting the caller to dispose the live stored
+entry. Clear detaches entries and attempts cleanup for every sibling. Backing
+retention is released only after successful cache disposal. Rejected optional
+prompt-boundary snapshots release their clones while generation retains its
+live state. These are failure-path ownership changes, with no cache-math change.
+
+Model-free binding tests exercise an independent graph, a replacement fused
+decoder, declined steps, unrecovered errors, and unsupported ABI/media/adapter
+requests. The native `generated-parity` gate runs dedicated and generated Gemma
+bindings through the same gateway method and portable session consumer. Tokens
+and cache coverage match, and its counter verifies the generated path executed.
+This is a compatibility gate using available weights, not a product speed claim.
+The binding migration also passes all eight compiled-decode parity/recovery
+checks and the Qwen3.8-27B TQ smoke below on the M1 Max 32 GB.
+
+`src/model/implementation.ts` defines the backend-parameterized
+`ModelImplementation` construction port and immutable registration table.
+`ModelExecutionComposition.implementation` lets an exact engine-owned artifact
+profile name its code. Existing identity/config guards perform the matching;
+the registry does not introduce another artifact matcher. `src/model/factory.ts`
+now registers the existing resident constructors, including generated Gemma,
+and selects one before construction. `openModel` checks the binding before
+opening weights; callers can compose profiles and implementations through its
+options. `createModel` accepts a registry for already-open weights. No selection
+is added to the token loop, and no model file format changes.
+Both factories preserve a supplied registry's return interface; replacement
+implementations need not join the concrete `RuntimeModel` union. Construction
+does not select the request's execution method. An implementation can support
+several methods, including quant-specific paths, while the session consumes
+the selected method's execution, output and cleanup contract. The shared
+contract does not require a shared numerical loop.
+
+Unknown artifacts retain existing family selection. Missing implementations,
+graph/loader/loop mismatches, duplicate IDs, and inconsistent exact identities
+fail explicitly. Construction failures propagate without trying another graph.
+Model-free tests exercise the factory with a synthetic exact quant identity and
+an alternate constructor; this establishes selection, not numerical or speed
+claims. No registration pretends to identify Josh's inaccessible quants.
+This step covers resident graph construction. Colibri retains its existing
+streamed opener and refuses supplied resident registries or named overrides
+until its resident loader binding exists. Existing class-based library exports
+remain available through compatibility binders. The server consumes the
+model-owned serving port; host implementations can own resident or streamed
+loading. Specialized inference methods remain.
+
+Artifact-parameterized native smoke gate:
+`MLX_BUN_QWEN_QUANT_PATH=/path/to/our/artifact bun test tests/parity/qwen-quant-engine.test.ts`.
+Run one artifact per process. Compatibility-only checks on the M1 Max 32 GB
+pass for the locally cached published `mjriii/Qwen3.8-27B` and
+`mjriii/Qwen3.8-27B-TQ`: direct-versus-bound logits, legacy-versus-session greedy
+output/cache coverage, and repeated recurrent-prefix borrowing. These are not
+Josh's current target quants. Target validation awaits access to those artifacts;
+implementation continues independently. The test is a bounded smoke, not a full model oracle or the
+recorded turn-8 repro. Gemma compiled-decode checks are secondary coverage too.
+
+R1 is complete. Incremental and final-only producers run the same lifecycle
+cases for collection/streaming, blocked-consumer cancellation, partial failure,
+and no-consumer/pre-cancelled settlement. Additional tests cover preparation
+cancellation, admission refusal, metadata bounds, and cleanup failures.
+
+R3 now gives request resources one checked owner. Transfer invalidates the
+previous owner, close is idempotent even after a destructor failure, and cleanup
+attempts all releases while preserving the original execution error. MLX state
+views expose a lease; prefill and prefix-cache accounting close it without
+knowing whether handles are borrowed or temporary. Only the compatibility
+adapter reads the legacy ownership marker. No tensor copy, evaluation or device
+synchronization is added by these leases. Fault-injection tests cover transfer,
+partial view acquisition and throwing destructors. Existing native fences remain
+in place; codec binding and persistence identity are described below.
+
+R4 now resolves method, scheduling mechanism, paged-KV fallback, prompt-cache
+bypass, fill eligibility and checkpoint eligibility in one portable planner.
+The gateway passes that immutable decision to serial execution; lane accounting
+uses the actual method, including AR fallback with a configured draft. Request
+policy arrays/maps are copied and frozen at admission. AR runtime settings and
+batch work flags use a binding snapshot. Native grammar/media preparation takes
+the gateway execution lease before allocation; text rendering remains outside
+that lease. Cancellation before/during preparation is covered with allocation
+and cleanup assertions.
+Native AR/speculative/denoising method creation now uses the same policy snapshot
+as HTTP admission. Scalar settings and nested policy arrays/maps cannot change
+between registration and the first reader demand. A binding without an explicit
+runtime captures the host default at method creation; kernels execute under
+that snapshot. Native caches, controllers and callbacks retain their declared
+ownership instead of being cloned with policy data.
+
+Checkpoint request identity is now canonical SHA-256 over sampling/stop policy,
+resolved execution and the artifact/implementation/state ABI. Object insertion
+order is ignored; array order remains meaningful. Version-4 keys include the
+resolved compiled/grammar policy; version-2/3 generation keys do
+not resume through this version; ordinary prefix-store format is unchanged.
+Nine native scheduling gates pass after planner integration, including rotating
+dynamic joins, row failure containment, serial drain, prefix sharing, SSD restore
+and compiled B=1. Preparation reservations now remain owned through execution; see below.
+
+R3 state persistence now uses `CacheCodecProvider`, bound through the server's
+model context, RAM cloner, batch snapshots and SSD store. A provider supplies
+clone/snapshot/restore/trimmability together; ambiguous matches refuse. Persisted
+provider identity is checked before cache allocation or mmap. Earlier v3 files
+implicitly use the legacy provider; a different provider must identify itself.
+Restore still copies one tensor at a time and unmaps only after those copies,
+so this change introduces no new device/backing lifetime assumption. Failed
+clone and snapshot construction release intermediate owned views.
+
+Adapter namespaces now hash the weight file plus scale/rank policy once at
+mount. Remounting the same name with different bytes or scale cannot reuse its
+old RAM prefix or generation checkpoint. The shared server cache uses that
+namespace for both lookup and persistence. Codec provider fault/round-trip tests
+pass; native persistence passes five cells with one unavailable golden skipped,
+and dedicated/generated Gemma passes six cells, including direct logit identity.
+
+R5 now shares a portable prefill program between serial AR and solo batch
+admission. Its MLX executor preserves drain evaluation, KV maintenance, allocator
+clears, stable-boundary snapshots, and the separate final-token forward. Batch
+admission retains its short-tail scheduling behavior. Six native scheduler
+checks pass, including dynamic rows, prompt reuse, SSD restore and compiled B=1;
+the dedicated/generated same-session test also passes on the M1 Max 32 GB.
+The native AR method run owns its generator's pending tokens, history and
+checkpoint boundary. Phase-specific tests now cover cancellation before any
+cache allocation, after a prefill chunk, after decode dispatch/before emission,
+and while an asynchronous checkpoint borrows live state. Cancellation waits for
+the checkpoint to finish before releasing state. Current-token logprob arrays
+remain owned until readback succeeds, including forward/grammar failure;
+cancelled execution stops before emission or a subsequent checkpoint write.
+
+R6 now runs the speculative verifier through `MlxSpeculativeBinding`. Target
+forwards, projection, tap capture, draft construction and kernel pinning bind
+outside the verifier. Legacy tap contexts restore their previous value on exit.
+The complete rollback facet validates recurrent begin/commit/replay together;
+partial layer failures invalidate the transaction and the run discards its state.
+Ring-capacity fallback retains its strict pre-write boundary. The verifier owns
+input/hidden scratch on failure and attempts every final release.
+
+AR and speculative methods now implement `InferenceMethod` directly. Their shared
+host adapter translates cancellation and waits for native cleanup. The same
+session tests exercise collection, streaming, early reader closure and forward
+failure for both. Four native Qwen3.5-0.8B gates pass, including recurrent
+rejections and identical speculative acceptance traces through a portable
+session. This smaller available model supplies compatibility coverage only.
+Denoising now implements that same method/session contract through a separate
+`DenoisingGraph<Tensor, State>`. Its test binding uses a position-only state
+record, with no AR cache or token-prefix API. The existing synchronous diffusion
+entry and cooperative generation drain one denoising program. Cooperative calls
+yield between denoising steps and publish only the final output; cancellation
+releases canvas, feedback, embedding weights and graph state. Generation holds
+the existing exclusive runtime lease, including the diffusion global RNG.
+
+Scripted regression tests cover both samplers, RNG-dependent canvases across
+multiple blocks, sync/async identity, preparation failure, cancellation between
+steps and repeated-run native memory accounting. The existing soft-embedding
+lifetime test passes. Diffusion weights are absent on this machine, so its
+weight/oracle gates remain unrun. Draft sources now consume declared target
+ports for assistant donor views/scaled embeddings, Gemma hidden taps/output
+projection and Qwen MTP embeddings/output projection. The target view contains
+no concrete model or cache array; the MLX compatibility binder owns family
+checks and donor layout. Native GLM providers compare an opaque target identity
+before borrowing their own model-owned weights. Both retained DSpark variants
+use the projection interface. Missing extensions refuse before draft allocation.
+An independent assistant target fixture proves construction and draft cleanup
+without a model class; donor acquisition releases partial views on failure.
+The locally cached published Qwen3.8-27B and its bundled MTP head pass the
+same-artifact greedy comparison after this migration. This does not establish
+performance or compatibility for Josh's unavailable exact quants.
+
+R7 now separates the portable scheduling driver from `MlxBatchExecutionGroup`.
+The driver sees queue/active counts, admission readiness and bounded work units;
+it imports no model, cache, tensor or runtime implementation. Native admission,
+merge/extract/filter, sampling and pipeline state live in `src/backends/mlx/`.
+A compatibility re-export preserves existing batch-scheduler imports.
+
+The driver retains short-admission grouping, long-prefill interleaving, serial
+drain and the 25 ms single-row responsiveness budget. Six native scheduler
+checks and two native failure-containment/serial-drain checks pass. Policy tests cover those ordering rules, execution failure and
+lease release; backend fault tests cover a request removed from the queue when
+admission fails, retained-prefix cleanup, shutdown and submission after close.
+Closing a group stops at a safe boundary, rejects pending/active requests and
+releases owned state. Bounded request and preparation queues now reject overflow explicitly. The full
+mixed-state concurrency/performance matrix remains open.
+
+R8 now shares Pi/browser messages, history, sampling data and route IDs in
+`src/contracts/pi.ts`. Compatibility re-exports preserve server-side imports.
+The browser TypeScript project has no Bun ambient types; Bun UI tests remain
+covered by the root project. The resolved-import gate prevents browser modules
+from importing server implementations, including type-only imports.
+
+R4/R7 admission now uses a portable FIFO `AdmissionPool`. The gateway bounds
+active requests and queued waiters; cancelled waiters do not release another
+request's capacity. Media and grammar preparation reserve capacity before native
+allocation. The request owner retains that reservation after transferring native
+resources to execution, then releases it on success, failure or cancellation.
+The direct batch group also bounds pending submissions. Overload is explicit in
+the existing HTTP/SSE error contract. Nine native scheduling checks pass after
+this integration, including rotating joins, failure containment, prefix/SSD reuse
+and compiled B=1. The gateway's shared execution lease covers preparation,
+generation and managed GPU jobs.
+
+R8 is complete: completion/batch/task contracts are portable. Memory calls accept an injected
+client; their existing native implementation moved behind a lazy backend import.
+Eval generation accepts a replacement completion client and retains its separate
+bit-exact greedy oracle path in the MLX backend. Training and quantization keep
+their numerical runners and artifact results; both job execution modes now use
+the task adapter. Progress delivery is synchronous and unbuffered; cooperative
+cancellation checks before work and at progress boundaries. Native work already
+dispatched still completes at its safe boundary.
+
+Pi's embedded registry and generated extension use one model-definition builder.
+Job and Pi/browser messages now share portable declarations. New `mlx-bun/engine`
+and `mlx-bun/client` exports import without MLX; `initializeMlx()` explicitly
+bootstraps the compatibility API. The existing root import remains compatible.
+The JSON completion client uses either HTTP or an `EngineHost` without changing
+callers, preserves abort, and never retries completion POSTs.
+
+R9 adapts the existing `EngineChild` into `EngineHost`, bounds restart attempts,
+cancels restart backoff on close, kills failed startup, and checks disconnect
+while waiting for readiness. Model-pool startup failures and shutdown close their
+owned children; an evicted default can be resolved again for an absent/unknown
+model request. The fake-host proxy suite now runs in the model-free CI tier.
+Real MiniCPM JSON and incremental SSE requests pass through the isolated host.
+
+Managed GPU subprocess jobs reserve the same gateway lease as inference: the
+current batch drains before spawn, and the job retains the lease through child
+exit, including crashes. Host shutdown cancels admission/queued jobs and terminates
+its active job before releasing the lease. Fault tests cover rejected admission,
+spawn failure, child death and shutdown. The isolation parent now owns job/settings routes and their store. A portable
+shared/exclusive coordinator covers every worker's startup, restart, response
+body and eviction. Before GPU job spawn, live UDS lease responses reserve each
+worker's gateway after durability flush; disconnect releases the worker lock.
+Those connections remain owned through subprocess exit, not merely until a
+client HTTP disconnect. Native MiniCPM checks verify that generation waits for
+the parent lease and resumes after release. Worker death does not remove parent
+job records. In-process task cancellation waits for runner cleanup before closing
+the store. The parent serves the same embedded web assets and owns Responses
+conversation history; direct and isolated serving share conversation-resolution
+policy. Workers skip duplicate history retention for parent-owned requests. A
+real worker replacement preserves previous-response continuation and streamed
+metadata. WebSocket chat remains an explicit unsupported isolation cell. No
+process-host default has changed. R9's direct/isolated transport comparison is
+complete on the M1 Max using the same MiniCPM artifact and CLI/request cells:
+completion and chat probes each matched 64 greedy tokens; cold, warm, SSD
+restart and concurrent legs completed. The run is diagnostic because the quiet
+gate failed. The optional `mlx-bun-isolated` benchmark arm accounts for parent
+and worker RSS; raw reports stay machine-local. R9 is complete for the retained
+opt-in host; promotion and the other machine's scorecard remain R10 gates.
+
+The current model-free suite passes; root, browser and portable typechecks pass.
+Native state checks cover mixed-KV oracle logits, wrapped-ring conversion,
+speculative recurrent rollback and persistence. The Qwen B=2 oracle and one
+Qwen persistence fixture are absent and skip. The binary builds and its CLI/Pi
+asset smoke passes. Managed jobs now self-exec the packaged binary; literal
+runner imports retain quantize/train code in the bundle, and a compiled
+subprocess smoke verifies progress and terminal persistence. These checks do not replace the unavailable artifact/machine
+cells or the final production acceptance gate.
+
+The Qwen replay now has a native exception diagnosis: Metal reports insufficient
+memory from its completion handler. Details remain in the existing turn-8 repro.
+Josh reports a possible fix on the other machine; reconcile it before changing
+the affected state path. The 2026-09-05 benchmark preflight found 4 GB of existing swap and high
+background CPU load. Measurements taken without that gate are diagnostic only.
+Quiet-machine baselines remain open. Remaining R0–R10
+work is tracked in PLAN.md; no speed/quality/size improvement is claimed yet.
+
+The first per-file native matrix at `a75cd39` ran 84 local parity files with
+batch-decode opt-in: 338 passes, 44 skips and seven failed assertions. Five
+audio failures were a fixture path left behind by the test-directory move.
+The two numerical failures reproduced on baseline `6d45ca1`: the local CPM
+mixed-KV blobs did not match their manifest hashes, and CPM extend-join fell
+back to the M4 reference because no M1 override existed. The pinned Python
+oracle regenerated the mixed-KV manifest's exact hashes and an M1 extend-join
+trajectory identical to the implementation. The corrected batch suites pass
+15 tests with one unavailable e4b oracle skip. Tests now check blob hashes;
+the mixed-KV generator writes blobs and manifests to the same resolved output
+directory. A second per-file run enables the local train/quantize/LoRA/generated
+and model-family gates. Two rotating-quantized continuation failures also
+reproduced on the baseline: the test compared `generate()`'s bf16-head/quantize/
+tail prefill with a fully quantized single-forward oracle. The generator now
+records both procedures separately. Existing mechanics, single-forward logits
+and trajectories regenerated unchanged; the matching serving trajectories pass
+the original prefix threshold. All six rotating tests pass. Missing new
+continuation fixtures skip explicitly on other machines until regeneration.
+A process that exits successfully without registering tests is not a passing
+matrix cell; absent large-model/drafter fixtures remain open.
+The production test runner now uses a fresh process for each native test file,
+preserving model residency and runtime isolation between cells. The complete
+default runner passed at `2a14021`, including hygiene,
+all TypeScript projects, the model-free suite and available parity/research
+tests. Optional cells are reported separately; a green default run does not
+imply unavailable weights or opt-in cases ran.
+That run reports 2,344 passes, 98 skips and no failures, with 109 native files
+each receiving a fresh process.
+The expanded runner at `ab0db74` enabled the locally applicable model-family,
+training, quantization, LoRA and batch-decode gates: 2,427 passes, 44 skips and
+no failures across the model-free tier and 109 native files. Ten native files
+registered no tests because their configured model/drafter fixtures were absent;
+those are unavailable cells, not passes. At `b9b56aa`, the model-free suite
+passed 1,718 tests with 10 skips. All TypeScript projects, hygiene and compiled
+CLI/Pi/assets/job smokes passed. The subsequent focused native run passed 44
+tests covering batch lifecycle, SSD state, generated graphs, ngram verification,
+both available published Qwen graph/session bindings and the published Qwen MTP
+target/draft pair. Explicit MTP paths supplied one previously unavailable cell.
+These results do not close the target-artifact, second-machine or default-cutover
+gates.
+
+The sequential Gemma e4b repeat compares baseline `6d45ca1` with `5e3b92e`
+using default and serial HTTP arms, the same artifact, a 4K context and 96
+requested tokens. Both arms preserve the completion/chat greedy probes and
+durable restart state. The earlier long-context prefill slowdown and default-arm
+RSS increase did not reproduce; warm-cache latency still varies across runs.
+Background load and existing swap keep these reports diagnostic. This repeat
+does not establish a speed win or satisfy the quiet-machine frontier gate.
+
+R10's implementation-retention review currently records these decisions. None
+of the open measurement cells changes an existing numerical default:
+
+| Implementation | Current disposition and remaining evidence |
+|---|---|
+| Pi provider wiring | Consolidated into one builder; generated-extension and binary asset tests pass. Duplicate policy is removed |
+| Dedicated/generated Gemma | Both register through the engine and pass the same session/oracle gates. Keep both during paired comparison; numerical equivalence alone does not choose a winner |
+| Compiled decode | Retained behind the declared decoder capability and resolved eligibility. Existing recovery/parity gates pass; final paired regression measurements remain required |
+| Diffusion | One denoising program and method interface replace duplicate orchestration. Real model/oracle fixtures remain unavailable; no promotion or deletion decision is implied by scripted tests |
+| Assistant/DSpark/DeepSpec/MTP | Typed draft target ports replace concrete target dispatch. Available Qwen MTP and ngram verification pass; source-specific drafter validation and the DSpark wall-clock decision remain open in the speculative program |
+| Curve/HLG and paged KV | Existing opt-in placement and refusals remain. Their experiment-specific quality/performance gates remain open; this refactor does not make them defaults |
+| Expert offload | Existing streamed GLM loading/task semantics remain behind native bindings; model-free streamed/provider tests and packaged task entry pass. The larger-model value/footprint decision remains in its own program |
+The final server cutover is implemented at `015f56d`. The model-free suite passes
+1,732 tests with 10 skips. Fresh native processes run all 109 parity/research
+files with the locally applicable opt-ins, including the published Qwen target
+and MTP paths: 720 passes, 33 skips, no failures. Nine fixture-dependent files
+register no tests and remain unavailable cells. The second published Qwen TQ
+artifact also passes the graph/session test. The public replacement and import/
+docs boundary checks pass after the final exports. Commit `d1abd60` also puts
+embeddings under the shared GPU lease. Its new ownership test, the public
+replacement test and both native embedding-route tests pass. All three
+TypeScript projects pass, as does hygiene. The standalone binary built at
+`d1abd60` passes CLI, Pi asset and managed-job subprocess smoke checks.
+
+The final HTTP comparison uses the frozen `6d45ca1` baseline and this branch on
+the M1 Max 32 GB, with MiniCPM5 and Gemma e4b, default and serial arms, a 4k
+context and 96 generated tokens. The first candidate run overlapped a repository
+hygiene scan and showed slower e4b long prefill and SSD restart. A sequential
+e4b repeat without other checks removed the long-prefill gap, but restart
+latency still varied. A further candidate-then-baseline default-arm pair removed
+that restart gap too. Completion and chat parity probes match between default
+and serial arms; every restart restores a durable prefix with no failed or
+dropped spills. These loaded-machine diagnostics do not establish a quiet
+performance acceptance result. Raw reports remain local in
+`/tmp/mlx-bun-cutover-{baseline,candidate}-report.md`,
+`/tmp/mlx-bun-cutover-repeat-{baseline,candidate}-report.md` and
+`/tmp/mlx-bun-cutover-reverse-{baseline,candidate}-report.md`.
+
+R1–R9 implementation and R10 code migration are complete. Existing model-class
+exports use backend compatibility binders; there is one default session lifecycle
+and no alternate legacy HTTP executor. R10's retained implementations keep their
+existing placement and numerical defaults. Their independent experiment programs
+still own promotion/deletion decisions. Quiet performance, exact target quants,
+unavailable oracle/drafter fixtures and the second machine remain acceptance
+work, not missing interface code. No frontier advance is claimed by this refactor.