mlx-bun 0.0.9 → 0.0.11

package/docs/design/paged-kv-cache.md

@@ -0,0 +1,214 @@
+# Optional paged KV cache (vLLM-style) — v1 design
+
+Status: **v1 in progress** (this PR)
+Default: **off** (`--paged-kv` unset ⇒ today's `KVCache`/`RotatingKVCache`
+path, byte-identical — the flag-off proof is that zero new code executes).
+
+Plan produced by a multi-agent design pass (4 subsystem readers +
+prior-art scout → 3 competing designs → judged synthesis), then vetted
+against the code by hand. Judge's pick: minimal-seam (8.25) over
+serving-first (6.5) and future-proof (7.25); build-order discipline
+grafted from future-proof.
+
+## Motivation — what paged buys here, honestly
+
+vLLM's PagedAttention (Kwon et al., SOSP 2023, arXiv 2309.06180) solves
+contiguous worst-case per-sequence KV reservation + allocator
+fragmentation. mlx-bun is a single process on unified memory with
+byte-budget admission already shipped (parallel-slots.md rung 2), so the
+fragmentation story mostly doesn't transfer. What does:
+
+- **Padded-batch waste removal** — today's batched buffer width tracks
+  the LONGEST live row (`mergeKVRows`/`extendKVRows`,
+  src/model/batched-mask.ts): a 50-token row batched with a 4000-token
+  row pays ~4000 tokens of KV while they cohabit. Block-granular
+  allocation removes this. **A batching feature; v1 does NOT implement
+  it** (non-goal — follow-up PR).
+- **Block-level prefix sharing / copy-on-write** — PromptCache clones at
+  entry granularity; block CoW is vLLM's headline win for concurrent
+  agents sharing a system prompt. **Not in v1.**
+- **At batch=1, paging buys nothing** — gather-before-attend is pure
+  bandwidth tax on a bandwidth-bound decode. Measured and reported
+  plainly, never spun.
+
+v1 therefore ships the correctness-proven block-manager abstraction at
+serial (batch=1) scope, expecting a small decode regression and zero
+density win — scaffolding for the follow-ups, not a standalone perf win.
+parallel-slots.md already names paged KV "rung 3 / S3+ deferred";
+unified-engine-frontier-plan.md names block-paged KV the long-term
+unifier. This PR is the first, narrowest slice.
+
+Prior-rejection reconciliation (the record already says "no paging"
+twice, for different axes, and both stand): the 2026-07-07 vLLM-mining
+pass rejected paged KV as a PROMPT-CACHE substitute (take()'s zero-copy
+entry clones already share physically), and ssd-kv-cold-tier.md D1
+rejected paged BLOCKS as the SSD-spill granularity (per-block hashing on
+the token loop = oMLX's ~20% tax). This PR is neither: it's the rung-3
+ALLOCATION abstraction (parallel-slots.md), explicitly requested as an
+optional flag, with the padded-batch/CoW payoff deferred to follow-ups.
+
+**No external oracle exists**: mlx-lm's cache.py has no paged/block
+cache (verified in the oracle venv). The correctness gate is mlx-bun's
+own plain `KVCache` — valid because v1's claim is storage-layout
+equivalence (same bytes, different physical arrangement), not new
+arithmetic. Bit-exactness (tol 0) is the bar.
+
+## v1 scope
+
+- Serial (non-batched) single-sequence decode; Gemma4-family models
+  (modelType `gemma4*`), plain full-attention layers only. Sliding
+  (`RotatingKVCache`) layers keep today's scheme untouched — mixed
+  per-layer paged-full + rotating-sliding is the supported shape.
+- bf16 only. No `--kv-quant`/TurboQuant composition.
+- Strategy: **block pool + gather-to-contiguous, then the unchanged
+  `ops.sdpa()`** (gemma4.ts Attention.forward's plain-cache `else`
+  branch consumes any `Cache` whose updateAndFetch returns contiguous
+  [1,H,S,D] views). No new attention math, no new kernel, no new branch
+  in the forward.
+- Fixed pool per request, sized up front from prompt length + maxTokens
+  (capacity is known at generate() time) — exhaustion is a typed
+  `PagedPoolExhausted` error, never silent truncation or OOM.
+
+## Non-goals (v1)
+
+- No `--batch N` integration (`--paged-kv` + `--batch N>1` refused at
+  startup). No `LayerInner` union change in batch-scheduler.ts.
+- No sliding-window paging; no quantized blocks; no CoW/prefix sharing;
+  no MoE/SSM/universal/CPM/Diffusion models; no vision/audio; no
+  LoRA/adapters; no speculative decode (spec eligibility excludes paged).
+- No fused paged-attention Metal kernel (follow-up once the gather tax
+  is measured to matter; metal-kernel.ts infra + optiq's
+  sdpa_2pass_paged are the port sources).
+- No compiled-decode composability: a data-dependent block-list length
+  is the exact shape shapeless replay already broke on (tiled quantized
+  SDPA precedent). `CompiledDecode.supports()` excludes PagedKVCache
+  automatically (it's not one of the four supported classes); the
+  serve/CLI layer also refuses the explicit combination.
+- No PromptCache integration: paged requests bypass take/put (the
+  vision precedent in server.ts runGeneration) and dispose caches on
+  completion. `state()` exists only for evalCacheState.
+
+## Design
+
+New file `src/model/paged-kv.ts`:
+
+- **`BlockPool`** — per-layer arena. K and V pool tensors shaped
+  `[numBlocks, H_kv, blockSize, headDim]`, allocated via `ops.zeros`
+  (mlx-owned end to end — no host-pointer alignment/dtor hazards).
+  `alloc()` pops the free list or throws `PagedPoolExhausted`;
+  `free(idx)` pushes back. Incoming K/V write via the existing
+  `ops.sliceUpdate` at `[block, 0, within, 0]` — the incoming
+  `[1,H,l,D]` piece matches the destination slice directly, no
+  transpose.
+- **`PagedKVCache implements Cache`** — standalone implementation
+  (deliberately NOT a `KVCache` subclass, same reasoning as
+  `TurboQuantKVCache`: every instanceof gate — CompiledDecode.supports,
+  gateway #modelCachesBatchable, generated `#matches()` — then excludes
+  it automatically and routes to the monolith forward).
+  - `updateAndFetch(k, v)`: split the incoming L along block boundaries,
+    sliceUpdate each piece into pool blocks (allocating from the free
+    list as the tail block fills), then fetch: `ops.takeAxis(pool,
+    blockTable, 0)` → `[nb,H,bs,D]`, transpose `[1,0,2,3]` →
+    `[H,nb,bs,D]`, reshape `[1,H,nb*bs,D]` (the gather copy), slice to
+    offset. **Uses only existing bound ops** — the plan's proposed
+    `mlx_gather` FFI binding is unnecessary (`takeAxis` gathers whole
+    blocks along axis 0), which deletes the bun:ffi stack-arg ABI risk
+    outright.
+  - `makeMask`: same logic as `KVCache.makeMask` (reads only offset).
+  - `trim(n)`: rewind offset, free now-unoccupied tail blocks. Stale
+    bytes past offset are never read (fetch slices to offset; writes
+    cover exact ranges) — same invariant as KVCache's padding.
+  - Block table is host-side `number[]` — no device indirection.
+  - All intermediates bound to locals and disposed
+    (mlx-inline-slice-leak-pattern discipline).
+
+**Wiring** — mirrors the verified `maybeQuantizeKv` precedent
+(post-construction in-place mutation of the cache array,
+generate.ts:153; no makeCache() signature change): `maybePageKv(cache,
+options, capacityTokens)` called ONCE at generation setup (unlike
+quantization there is no "convert when populated" trigger — paged
+replaces EMPTY plain `KVCache` entries, offset 0 only, before prefill).
+`GenerateOptions.pagedKv?: { blockSize?: number }`.
+
+Gates:
+- server startup (explicit refuse, not silent downgrade): `--paged-kv` +
+  any of `--batch N>1`, `--kv-quant`≠off, TurboQuant, `--draft-model`,
+  non-gemma4 model ⇒ exit with a clear message. With no explicit
+  `--batch`, the CLI pins `--batch 1` (default is 8) and says so.
+- request scope (documented v1 non-goal CELLS, scoped per request rather
+  than 400'd): media (vision/audio) and LoRA-adapter requests strip the
+  flag and run the plain cache path — one effective `pagedKv` value in
+  runGeneration keeps the strip and the prompt-cache bypass coherent.
+- serve serial lane: paged requests skip promptCache.take/put and the
+  boundary snapshot; caches dispose on completion. Spec-decode
+  eligibility gains `!options.pagedKv` (belt — the startup refusal
+  already excludes the combination).
+- batch lane: unreachable by construction (startup refusal pins serial),
+  so no RequestShape change — noted here so the next increment knows the
+  gate to add when batched paging lands.
+
+Flag surface:
+- CLI: `mlx-bun serve --paged-kv [--paged-kv-block-size N]` (default
+  block size 256 = `KVCache.STEP`, so v1's growth granularity is a
+  permutation of today's into reusable slots, not a new tuning axis).
+- Env: `MLX_BUN_PAGED_KV=1` via `flagOn()` (explicit env wins).
+- ServerOptions: `pagedKv?: { blockSize?: number }`, doc-commented like
+  the neighboring kvQuant/turboQuant fields.
+
+## Implementation order
+
+1. **BlockPool + PagedKVCache + model-free unit tests**
+   (tests/paged-kv.test.ts, FAST tier): block-boundary crossing,
+   free-list reuse after trim, pool-exhaustion typed error, gathered
+   fetch vs a plain-KVCache reference fed identical synthetic k/v
+   (bit-equal every step), dispose-doesn't-leak. Exit: green with no
+   weights.
+2. **Wiring**: `maybePageKv` in generate.ts; GenerateOptions/
+   ServerOptions/CLI flags; startup refusals; gateway shape exclusion;
+   prompt-cache/spec bypass. Exit: flag parses, default-off inert;
+   flag-on serves a completion end-to-end (gemma4); incompatible combos
+   fail fast.
+3. **Parity gate** (tests/paged-kv-parity.test.ts, weights-gated,
+   skipIf pattern): bit-exact (tol 0) full greedy trajectory paged-on
+   vs paged-off, incl. a ≥2-block prompt and a trim/rollback case.
+   Exit: bit-exact; any divergence documented, never a loosened
+   tolerance.
+4. **Perf disclosure**: paged-on vs paged-off decode/prefill on a quiet
+   machine via the preflight-gated harness (expected: small decode
+   regression at M=1; report plainly). Numbers land in
+   benchmarks/RESULTS.md when run on the reference machine.
+5. **Docs, same commit**: server-config.md, cli.md flag rows;
+   parallel-slots.md S3+ status flip (v1 shipped: serial, bf16,
+   full-attention-only, gather-then-sdpa; batched + CoW + kernel still
+   deferred); STATUS.md handoff; this doc.
+
+## Risks
+
+- Gather pays a full K/V copy per decode step once >1 block is occupied
+  — pure tax at M=1 (disclosed, measured; the fused-kernel follow-up is
+  the cure if it matters).
+- Fixed pool sizing from prompt+maxTokens: a trim past a block boundary
+  followed by regrowth must re-alloc from the free list correctly —
+  dedicated test.
+- trim/rollback against a block table is more failure-prone than
+  KVCache's offset rewind — dedicated test, not incidental coverage.
+- Every caller that could receive a PagedKVCache it can't handle
+  (cloneKvCaches, prompt-cache put, compiled decode, batch scheduler)
+  is excluded by gates above; the ones reached only through excluded
+  paths throw typed errors if ever reached anyway.
+
+## Follow-ups (out of scope, in dependency order)
+
+1. **Batched integration** — PagedKVCache into the scheduler's
+   LayerInner union; block-count admission replacing byte-projection;
+   #mergeJoiner/#applyFilter/#extractAndPut allocate/free blocks
+   instead of pad+concat. This is where the padded-batch-waste win
+   actually lives.
+2. **Block-level CoW prefix sharing** — refcounted block table
+   (retain/release), fork-on-divergent-write; extends PromptCache's
+   entry-level ref-counting down to blocks.
+3. **Quantized paged blocks** — dtype-parametric block descriptor
+   (packed/scales/biases pools sharing one block-index space).
+4. **Fused paged-attention Metal kernel** — port optiq's
+   sdpa_2pass_paged via the copy-verbatim methodology; kills the
+   per-step gather copy.

package/docs/design/parallel-slots.md

@@ -1,6 +1,8 @@
 # Parallel slots — configurable batched serving
 
-Status: **S0 + S1b SHIPPED** (S2 not started; S3+ deferred)
+Status: **S0 + S1b + S2 SHIPPED** (S2's continuous inject/evict shipped
+with the scheduler; its byte-budget admission half landed 2026-07-03 as
+`--kv-budget`; S3+ deferred)
 Owner: serving layer
 Default: **off** (`--batch 1` = today's serialized path, untouched)
 
@@ -27,6 +29,16 @@ Direct/embedding mode stays batch=1.
 > onToken failure containment, drain-on-serial-waiter (no starvation), and
 > the engine-hygiene transplant (pipelined decode via asyncEval, clearCache
 > 1/256 steps, chunked admit prefill interleaved with decode steps).
+> Landed since: **SSM batched path** (Qwen3.5 batches, token-exact vs
+> mlx-lm B=2 — 2026-07-02, batching-perf-path.md P5); **per-row logits
+> processors** (repetition/presence/frequency penalties + logit_bias +
+> min_p/XTC batch — 2026-07-02, same doc); **grammar under batching** (B1
+> per-row matchers — structured-output.md); **KV-budget admission**
+> (`--kv-budget`, queue-don't-OOM, `/stats.batch` fields — 2026-07-03,
+> grammar-spec-batching-integration.md Phase D); **Tier-0 universal
+> coverage** (plain full-attention archs, e.g. Llama, batch via per-row
+> RoPE, token-exact vs mlx-lm B=2 — 2026-07-03, same doc's execution log;
+> maskArray + sliding universal archs stay serial).
 
 ## STATUS (2026-06-14) — LIVE for full-attention models
 
@@ -62,8 +74,8 @@ default stays mixed-precision (optiq parity); an explicit `--kv-quant` under
 novel extension: a memory-density win (batching × 4-bit KV compound), gated by
 KL + 6-task quality (per the tree framing), deferred. Other follow-ups (all
 optional polish): the `extend` join op (today joins re-merge), KV-budget
-admission, prompt-cache reuse under batching, and **paged KV** (zero-padding-waste
-allocation).
+admission (**LANDED 2026-07-03** — `--kv-budget`), prompt-cache reuse under
+batching, and **paged KV** (zero-padding-waste allocation).
 
 ## Why it helps (and when it doesn't)
 
@@ -222,54 +234,58 @@ base library; **mlx-optiq and mlx-bun are both optimizations on top of it**
 (optiq = mixed-precision per-layer KV quant, "drop-in for mlx_lm.server", no
 batching; mlx-bun = per-model fused kernels + batched serving). Bit-parity with
 an ancestor is just an **easy correctness test where we overlap one** — not a
-porting goal. The "L1/L2/L3" labels below are **parity anchors / correctness
+porting goal. The "L1/L2" labels below are **parity anchors / correctness
 checkpoints**, not a strict stack: L1 = "matches mlx-lm (bf16)", L2 = "matches
-optiq (mixed-precision quant)", L3 = "our own optimizations, no ancestor does
-them → no oracle". A combo no ancestor does (e.g. **batched mixed-precision
+optiq (mixed-precision quant)"; beyond them is the no-oracle **Lab** class ("our
+own optimizations, no ancestor does them → no oracle" — formerly labeled "L3";
+the `--l3` product tier was deleted 2026-07-05,
+docs/design/unified-engine-frontier-plan.md §6). A combo no ancestor does (e.g. **batched mixed-precision
 serving** — mlx-lm can't do mixed-precision, optiq can't batch) is genuinely new
 territory: gate it by KL + the 6-task quality + benchmarks, never bit-exactness.
 The anchors are still useful — we can always dial back to a bit-exact-vs-ancestor
 config as a correctness checkpoint and build outward.
 
 - **L1 — mlx-lm parity (standard KV).** bf16 KV, **bit-exact vs mlx-lm**. The
-  foundation oracle. Path: monolith `Gemma4Model` / `MiniCPM5Model`, `ops.sdpa`,
-  perf kernels off.
+  foundation oracle (and, since 2026-07-05, the naked default). Path: monolith
+  `Gemma4Model` / `MiniCPM5Model`, `ops.sdpa`.
 - **L2 — mlx-optiq parity (mixed-precision KV quant).** quant/mixed KV,
-  **bit-exact vs optiq**. Path: `QuantizedKVCache` + `quantizedSdpa` *unfused*
-  (`MLX_BUN_PERF_KERNEL=0`).
-- **L3 — our perf mode.** generated unrolled handlers + fused/perf kernels;
-  **deliberately beyond the oracle — there is NO bit-exact reference for L3**, by
-  design (these optimizations go past what mlx-lm/optiq do). The gate is **KL vs
-  the compatible upstream + the 6 eval-task mean (`src/eval/tasks/`) + perf
-  numbers** — never bit-exactness. Path: generated handler, perf-kernel +
-  fused-sdpa + fused-gelu on (the defaults). compiled-decode is inert here
-  (CustomKernel has no `output_shapes`) — a compat/benchmark artifact, not a
-  production mode. **L3's freedom is bounded by graceful degradation**: it may
-  diverge only because L2/L1 stay reachable and provably bit-exact.
-
-Degradation levers: `--perf-kernel off` / `--fused-* off` drops L3→L2; bf16 KV
-drops L2→L1. Each layer must stand alone and stay reachable (don't-delete-
+  **bit-exact vs optiq**. Path: `QuantizedKVCache` + `quantizedSdpa` (fused-sdpa
+  follows `--kv-quant`: on for `config` — the optiq golden composition; off →
+  *unfused* for uniform/bf16).
+- **L3 — DELETED as a product mode 2026-07-05**
+  (docs/design/unified-engine-frontier-plan.md §6): no output-changing kernel
+  (perf-kernel, fused-decode, fused-gelu) beat the L1 baseline in a paired A/B,
+  so the kernels were deleted and `--l3` hard-errors. Its no-oracle territory is
+  now the **Lab**: output-changing experiments are env-flag items with a bench +
+  expiry, gated by **KL vs the compatible upstream + the 6 eval-task mean
+  (`src/eval/tasks/`) + perf numbers** — never bit-exactness — until one earns
+  promotion.
+
+Degradation lever: bf16 KV drops L2→L1. The surviving kill switches
+(`--compiled-decode`, `--compiled-activations`, `--fused-sdpa`) are per-kernel
+escapes. Each layer must stand alone and stay reachable (don't-delete-
 optionality as a fallback ladder).
 
 So a feature like batched decode (`--batch`) is "done" only when green at
 **every (model × layer)** cell — and each cell may need its own fix:
 
-| Model · path | L1 (bf16 / mlx-lm) | L2 (quant / optiq) | L3 (perf / KL) |
+| Model · path | L1 (bf16 / mlx-lm) | L2 (quant / optiq) | ~~L3~~ (tier deleted 2026-07-05) |
 | --- | --- | --- | --- |
-| CPM (MiniCPM5) | ✅ **oracle-verified** (= mlx-lm B=2) | ☐ quant + unfused-mask | ☐ |
-| Gemma 12B | ✅ **oracle-verified** (= mlx-lm B=2, short-ctx) | ☐ generated + sliding-window | ☐ (gen rope fix) |
-| Gemma e4b | ✅ **oracle-verified** (= mlx-lm B=2, short-ctx) | ☐ generated + same | ☐ (gen rope fix) |
-| Gemma 26B | ✅ **oracle-verified** (= mlx-lm B=2, short-ctx) | ☐ generated + MoE | ☐ (gen rope fix) |
-| *(Qwen, future)* | *new path* | *new path* | *new path* |
+| CPM (MiniCPM5) | ✅ **oracle-verified** (= mlx-lm B=2) | ☐ quant + unfused-mask | — |
+| Gemma 12B | ✅ **oracle-verified** (= mlx-lm B=2, short-ctx) | ☐ generated + sliding-window | — |
+| Gemma e4b | ✅ **oracle-verified** (= mlx-lm B=2, short-ctx) | ☐ generated + same | — |
+| Gemma 26B | ✅ **oracle-verified** (= mlx-lm B=2, short-ctx) | ☐ generated + MoE | — |
+| *(Qwen, future)* | *new path* | *new path* | — |
 
 **🎯 L1 batched decode COMPLETE (2026-06-14d): the entire L1 row is green** —
 all 4 model paths bit-parity with mlx-lm B=2 (CPM full-attn, Gemma 12B
 sliding+full, e4b per-layer-input+KV-share, 26B MoE). Only one per-path fix was
 needed (e4b's `[1,L,…]` per-layer-input hardcode); sliding (RotatingKVCache),
 KV-sharing, and MoE were all already B-generic. Caveat: short-context (pre-wrap)
-— ring-wrap (>window) is the one remaining L1 follow-up. Next rows: **L2**
-(quantized KV → `quantizedSdpaUnfused` + 4-D mask, vs optiq) and **L3** (perf
-kernels under batching, KL+quality).
+— ring-wrap (>window) is the one remaining L1 follow-up. Next row: **L2**
+(quantized KV → `quantizedSdpaUnfused` + 4-D mask, vs optiq). (The former
+"L3: perf kernels under batching" row is void — those kernels were deleted
+2026-07-05, docs/design/unified-engine-frontier-plan.md §6.)
 
 Harness: `tests/batched-decode-parity.test.ts` has two paths — (a)
 `runBatchedDecodeParity` = the internal-consistency KL check (B=N vs our B=1,
@@ -329,15 +345,15 @@ Batched-decode landmines found by the modes analysis (2026-06-14):
   breaks; plus KV-sharing (sharer `SharedKv`).
 - **Gemmas**: `RotatingKVCache` sliding-window ring — per-row decode mask + ring
   write (the hard one). 26B adds MoE (`GatherQMM`/`SwitchGLU`) batched routing.
-- **Generated handlers (L3, all 3 Gemmas)** repeat the K/Q rope timing trap —
+- **Generated handlers (all 3 Gemmas)** repeat the K/Q rope timing trap —
   they read `cache.ropeOffsetArr` twice around `updateAndFetch`
   (generated/gemma4-12b.ts:63 & :74). The fix goes in the GENERATOR
   `scripts/gen-model.ts`, then regenerate (the files are DO-NOT-EDIT).
-- **Quant + batched**: the `[B,1,N,S]` array mask disqualifies the fused decode
-  kernel (`mask.mode===""` required) and the tiled path (2-D causal-equiv
-  required), so quant batched decode falls to `quantizedSdpaUnfused` — correct
-  but the perf kernel is bypassed (perf debt; a batched-aware fused decode kernel
-  is a later item). `quantizedSdpaUnfused` must be validated with a 4-D bool mask.
+- **Quant + batched**: quant batched decode goes through `quantizedSdpaUnfused`
+  — correct. (The old "perf kernel bypassed under batching" debt is resolved by
+  deletion: the perf/fused-decode kernels were deleted 2026-07-05,
+  docs/design/unified-engine-frontier-plan.md §6 — no batched-aware fused decode
+  kernel is planned.) `quantizedSdpaUnfused` must be validated with a 4-D bool mask.
 - **All cells**: compiled-decode forced off under `--batch>1`; LoRA batches only
   within one adapter group (else drain).
 
@@ -363,7 +379,7 @@ most" fidelity and cost:
    concurrent rows per GB. This is what "dynamically allocate based on
    who needs it most" actually means. Cost: a custom attention kernel +
    a block manager. Feasible — the project already ships custom Metal
-   kernels (`src/mlx/metal-kernel.ts`, `src/model/fused-decode-kernel.ts`).
+   kernel infrastructure (`src/mlx/metal-kernel.ts`).
 
 Plan: ship rung 2 (S1–S2) for the throughput win + dynamic admission
 with no new kernel; build rung 3 (S3+) as the memory-density upgrade.
@@ -376,8 +392,16 @@ growing in 256-token steps, width = longest live sequence, left-pad
 `BatchScheduler` + `GenerationGateway` (S1b SHIPPED). But the
 **budget/admission half of rung 2 is NOT built** (no `B × S_max`
 projection, no KV budget enforcement) — that's scheduler work (S2, NOT
-STARTED). And **rung 3 (paged) is a deferred spike**. So today: rung-2
-*allocation + scheduling wired*, no *budget/admission control* yet.
+STARTED). *(Since LANDED 2026-07-03: `--kv-budget` aggregate projection
+gate in the scheduler — grammar-spec-batching-integration.md Phase D.)*
+And **rung 3 (paged): the v1 CACHE CLASS shipped** —
+`PagedKVCache`/`BlockPool` behind default-off `--paged-kv`
+(docs/design/paged-kv-cache.md): serial batch=1, Gemma4, bf16,
+gather-then-stock-sdpa (no custom kernel yet), gated bit-exact vs the
+plain path. The rung-3 PAYOFF (batched block allocation replacing the
+pad+concat merge/extend, block-level CoW prefix sharing, the fused paged
+kernel) is still the deferred follow-up — the shipped slice is the block
+manager + correctness proof, not the density win.
 
 KV quantization is a force multiplier: 4-bit KV (`generate.ts:67`,
 already supported) is ~4× smaller, so the same budget buys ~4× the slots
@@ -456,7 +480,10 @@ server, concurrent + streaming + drain). No serial regression (server 17/17,
 tools 13/13).
 (3) **DONE 2026-06-14** — `_is_batchable` gate is the `GenerationGateway.willBatch`
 predicate (full-attention model + no vision/adapter/repetition-penalty/user-seed
-→ batch; else serial). `B×S_max` memory admission is still TODO.
+→ batch; else serial). *(The predicate has since widened: repetition
+penalty + logits extras now batch via per-row processors, 2026-07-02.)*
+`B×S_max` memory admission is still TODO — **DONE 2026-07-03**
+(`--kv-budget`, see the v2-plan pointer above).
 
 ## Fallbacks (must degrade, never break)
 
@@ -478,9 +505,9 @@ predicate (full-attention model + no vision/adapter/repetition-penalty/user-seed
 
 ## Config surface
 
-- `ServerOptions.batch` (default 1); later `kvBudgetBytes`,
-  `promptConcurrency`.
-- CLI: `--batch N` (shipped; `--slots N` was the original design name, `--decode-concurrency` accepted as mlx_lm.server alias); later `--kv-budget <GB>`.
+- `ServerOptions.batch` (default 1); `kvBudgetBytes` (shipped 2026-07-03);
+  later `promptConcurrency`.
+- CLI: `--batch N` (shipped; `--slots N` was the original design name, `--decode-concurrency` accepted as mlx_lm.server alias); `--kv-budget <GB>` (shipped 2026-07-03).
 - `batch === 1` → today's serialized promise chain, untouched.
   `batch > 1` → scheduler.
 
@@ -502,11 +529,17 @@ predicate (full-attention model + no vision/adapter/repetition-penalty/user-seed
   **Teacher-forced gate**: a 2-row batch must produce per-row logits matching
   two solo runs within bf16 tolerance (the train-batch-e2e methodology, now
   applied to decode).
-- **S2 — N-wide + continuous injection/eviction. (NOT STARTED)** Rows retire on EOS;
-  queued requests prefill into freed rows. Dynamic byte-budget admission
-  (`B × S_max` projection, KV budget enforcement) still TODO.
-- **S3+ — paged KV (DEFERRED)** (custom paged-attention kernel + block manager),
-  KV-quant under batch, LoRA-group batching.
+- **S2 — N-wide + continuous injection/eviction. (DONE)** Rows retire on EOS;
+  queued requests prefill into freed rows (shipped with the scheduler).
+  Dynamic byte-budget admission landed 2026-07-03 as `--kv-budget`
+  (aggregate projection gate, queue-don't-OOM —
+  grammar-spec-batching-integration.md Phase D).
+- **S3+ — paged KV (v1 cache class SHIPPED; density payoff DEFERRED)** —
+  the block manager (`src/model/paged-kv.ts`) shipped behind default-off
+  `--paged-kv` at serial scope, bit-exact-gated (paged-kv-cache.md); the
+  batched block allocation, block-level CoW, and the custom
+  paged-attention kernel remain the follow-ups. KV-quant under batch
+  landed separately (Phase 3.1 + milestone 2); LoRA-group batching open.
 
 Every phase ships default-off behind `batch=1`; the serialized path is
 never removed (only bypassed).

package/docs/design/parity-tier-dag.md

@@ -13,15 +13,19 @@ matches bit-for-bit**:
 
 - **L1** — drop-in for **mlx-lm** (its op is the oracle; bit-exact or it's a bug).
 - **L2** — drop-in for **mlx-optiq** (optiq's behavior is the oracle).
-- **L3** — **our originals** (flash-CCE head, ORPO loss, prefix-share, segmented
-  backward, HLG sampler, expert offload). **No oracle ships**, so correctness is
-  gated by finite-difference / KL / quality, *not* parity.
+- **the Lab** (formerly the `--l3` tier — deleted as a product mode 2026-07-05,
+  [unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)) — **our
+  originals** (flash-CCE head, ORPO loss, prefix-share, segmented backward, HLG
+  sampler, expert offload). **No oracle ships**, so correctness is gated by
+  finite-difference / KL / quality, *not* parity. Output-changing inference
+  experiments live here as env-flag items with a bench + expiry, until one beats
+  the L1 baseline in a paired A/B.
 
 **The gate is the oracle.** A node can only live in a lower tier if that oracle
 exists AND it matches bit-exactly; otherwise it bubbles up. Note the subtlety the
 map already shows: an *optimization* that still matches the oracle stays low —
 `compiled-decode` is ours but is L1 because it replays the same ops bit-for-bit.
-Only nodes with **no** oracle (flash-CCE) are *forced* to L3. (Background +
+Only nodes with **no** oracle (flash-CCE) are *forced* into the Lab. (Background +
 post-mortem: [orpo-flash-cce-pin-leak.md](../investigations/orpo-flash-cce-pin-leak.md).)
 
 ## Two axes: parity gates performance
@@ -31,7 +35,7 @@ made the flag surface unreadable:
 
 1. **Parity** — *which reference does it reproduce bit-for-bit?* This sets the
    **lowest level** it may live in: matches mlx-lm → L1; matches mlx-optiq but not
-   mlx-lm → L2; matches neither → L3 (gated by KL + tests, no oracle).
+   mlx-lm → L2; matches neither → no-oracle, Lab-gated (KL + tests).
 2. **Performance** — *is it the fastest correct way?* This sets whether it's the
    **default (on)** within its level, or a kept-but-**off** opt-in.
 
@@ -49,11 +53,12 @@ your floor of trust; the default within it is the best way to honor that floor.
    - **slower but still correct** → **keep it as a documented default-off flag**
      (optionality for A/B; never the default, never deleted).
    - **breaks the guarantee but offers a real tradeoff** (e.g. 2× tok/s at KL
-     0.0015) → an **L3 opt-in**; the user chooses exact vs fast.
+     0.0015) → a **Lab item** (env-flag, bench + expiry); it earns a place only
+     by beating the L1 baseline in a paired A/B.
 
 Worked examples:
 - *5% less memory, still bit-for-bit with mlx-lm* → **default-on in L1.**
-- *2× tok/s, but KL 0.0015 to mlx-lm* → **L3 opt-in** (user picks exact or 2×).
+- *2× tok/s, but KL 0.0015 to mlx-lm* → **Lab item** (proves itself or expires).
 - *another way, 3% slower, still correct* → **kept, off in every level.**
 
 ### Our decode levers under this rule (today)
@@ -61,23 +66,28 @@ Worked examples:
 | lever | parity | perf | placement |
 |---|---|---|---|
 | `compiled-decode` | bit-exact w/ uncompiled (`compiled-decode.test`) | faster | **default-on, every level** |
-| `perf-kernel` | **envelope-gated, no bit-exact oracle** — mlx-bun-original kernel; gate is ≥56/64 teacher-forced argmax vs *our own* frozen compat trajectory (`perf-kernel-oracle.test`, `freeze-perf-oracle.ts`), and the optiq kv-quant goldens track the **stock unfused L=1 decode** (`regen-kvq-goldens.ts`) | faster | **L3 / explicit opt-in only** (off in bare `--l2`; on in `--l3` and the no-tier default) |
-| `fused-sdpa` | matches optiq bit-for-bit, not mlx-lm (`fused-sdpa.test` tier-a goldens) | faster | **default-on from L2** (the L2 bridge) |
-| `fused-decode` | (would be L3) | ~3% slower (`optimization_plan.md`) | **kept, off everywhere** |
-
-So `--l1/--l2/--l3` is the **parity** axis (the guarantee), and the defaults within
-each tier are the **performance** axis (the fastest kernel that still holds it). On
-the *decode path* L2 and L3 now differ by exactly one node — the perf kernel (bare
-`--l2` = the optiq-golden composition: fused N-tiled prefill + stock unfused L=1
-decode; `--l3` adds the envelope-gated fused decode kernel). L3's remaining
-territory is the no-oracle **features** — HLG sampler, expert offload, batched
-mixed-precision — which live off the decode axis and are gated by KL + quality.
+| `perf-kernel` | envelope-gated, no bit-exact oracle (was gated ≥56/64 teacher-forced argmax vs our own frozen compat trajectory) | regressed e4b 0.62–0.93× | **DELETED 2026-07-05, [unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)** (kernel + frozen-oracle tests removed) |
+| `fused-sdpa` | matches optiq bit-for-bit, not mlx-lm (`fused-sdpa.test` tier-a goldens) | faster | **default follows `--kv-quant`** (on for `config` — the L2 bridge; off for uniform/bf16) |
+| `fused-decode` | (no oracle) | ~1.00× (`optimization_plan.md`) | **DELETED 2026-07-05, [unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)** |
+
+So `--l1/--l2` is the **parity** axis (the guarantee), and the defaults within
+each tier are the **performance** axis (the fastest kernel that still holds it).
+2026-07-05 outcome ([unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)):
+no output-changing decode kernel beat the L1 baseline in a paired A/B, so `--l3`
+was deleted as a product mode (the flag now hard-errors) along with its kernels
+(perf-kernel, fused-decode). The no-oracle territory — HLG sampler, expert
+offload, batched mixed-precision — is now the **Lab**: env-flag experiments with
+a bench + expiry, gated by KL + quality, off the decode axis.
 
 > History: commit `f1bf5cc` put the perf kernel in the L2 preset claiming
 > "perf-kernel-oracle.test asserts the fused kernel == the frozen optiq goldens".
 > Both halves were wrong — that golden is frozen from mlx-bun's own compat engine,
 > and the gate is argmax-agreement, not equality — so 2026-07-01 restored the tier
 > rule: the bare tier is the guarantee; perf kernels are opt-ins within it.
+> Deletion outcome: the perf kernel never beat the L1 baseline in a paired A/B
+> (regressed e4b 0.62–0.93×; only win 12B@16k +6% with a KL WARN), so it was
+> DELETED 2026-07-05 with its frozen-oracle tests
+> ([unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)).
 
 ## Why this matters for CLI flags
 
@@ -94,7 +104,7 @@ A flag falls into exactly one of three buckets:
 |---|---|---|---|
 | the only viable route (no alternative node) | — | not a choice | **always-on default; drop the flag** |
 | between two routes at the SAME tier / same oracle | none | cosmetic / safety guard | **default it; expose only if it ever fails** |
-| between a **lower-tier** route and an **L3** route | **L1/L2 ↔ L3** | a real **parity ⇄ optimization** knob | **keep, document as such, group by tier** |
+| between a **lower-tier** route and a **no-oracle (Lab)** route | **L1/L2 ↔ Lab** | a real **parity ⇄ optimization** knob | **keep, document as such, group by tier** |
 
 So the DAG lets us answer "**what flags push down THAT route**" for any path, and
 from that:
@@ -102,7 +112,7 @@ from that:
 1. **Prune always-on flags to defaults** — anything whose alternative route is
    never selected, or whose toggle doesn't change the tier, shouldn't be a flag.
 2. **Surface the real axis.** The flags that genuinely matter all toggle
-   *bit-exact-lower-tier ⇄ our-L3-optimization*. Those could collapse into a small
+   *bit-exact-lower-tier ⇄ our-no-oracle-optimization*. Those could collapse into a small
    number of intent-level switches (e.g. a single `--exact` that forces every node
    down to its lowest tier, vs the optimized default) instead of N independent
    kernel toggles the user has to understand individually.
@@ -117,11 +127,11 @@ from that:
   knob, **not** a parity choice. Should read as "memory budget," not "exactness."
 - `--grad-accum N` → changes the accumulation route (L1; smoother grads) → a real
   *training-quality* knob.
-- `--no-flash` → **L3 flash-CCE → L2 MLX fused head** → a genuine *parity ⇄ perf*
-  fallback (the one place the head leaves L3).
-- `--no-prefix` → **L3 prefix-share → bit-exact two-forward** → a *perf* knob with no
-  numerical cost.
-- `--lambda` → ORPO hyperparameter (pure L3) → a *training* knob.
+- `--no-flash` → **no-oracle flash-CCE → L2 MLX fused head** → a genuine *parity ⇄
+  perf* fallback (the one place the head leaves the no-oracle class).
+- `--no-prefix` → **no-oracle prefix-share → bit-exact two-forward** → a *perf* knob
+  with no numerical cost.
+- `--lambda` → ORPO hyperparameter (pure no-oracle) → a *training* knob.
 
 The pattern: most flags are **memory/training knobs or always-on guards**; only a
 couple are true **parity ⇄ optimization** toggles. Today they're a flat list the
@@ -133,15 +143,15 @@ actually are.
 The map is hand-authored today. To make the framework *enforce* itself:
 
 1. **Derive the graph from code** — extract the call graph + per-node op/oracle tags
-   so it stays in sync and is **queryable** ("every L3 node", "paths that cross into
-   L3", "the pure-L1 subgraph", "flags that select route X").
+   so it stays in sync and is **queryable** ("every no-oracle node", "paths that
+   cross into the Lab", "the pure-L1 subgraph", "flags that select route X").
 2. **CI-enforce the gate** — a node tagged L1 must have a passing bit-exact parity
-   test vs mlx-lm; if it can't, it's a bug or must be re-tagged L3. "The oracle is
-   the gate" becomes a check, not a principle.
-3. **Shrink the L3 surface** — audit whether anything in the L3 island is actually
-   bit-exact-able and just unproven (e.g. the prefix-shared *forward* is already
-   bit-exact — `prefix-shared-parity.ts` — so only the encode-once *fold* is truly
-   L3). Minimizing L3 minimizes what needs KL/quality gates.
+   test vs mlx-lm; if it can't, it's a bug or must be re-tagged no-oracle. "The
+   oracle is the gate" becomes a check, not a principle.
+3. **Shrink the no-oracle surface** — audit whether anything in the Lab island is
+   actually bit-exact-able and just unproven (e.g. the prefix-shared *forward* is
+   already bit-exact — `prefix-shared-parity.ts` — so only the encode-once *fold*
+   is truly no-oracle). Minimizing it minimizes what needs KL/quality gates.
 4. **Generate the flag surface from the graph** — once flags map to routes, the CLI
    help + defaults can be derived: always-on routes have no flag, parity⇄opt routes
    collapse to intent switches, knobs group by what they trade.