mlx-bun 0.2.0 → 0.3.0

package/docs/design/batching-v2-plan.md

@@ -1,529 +0,0 @@
-# Batching v2 — state map, debt, and the gap plan
-
-Status: **PLAN, largely EXECUTED** (written 2026-07-01 as a read-only
-audit; §1's state map is that day's snapshot). Landed since: steps 1–3
-(hotfix bundle, failure/fairness, engine hygiene — 2026-07-01); step 8
-SSM batched path (2026-07-02, superseded by batching-perf-path.md P5 —
-token-exact vs mlx-lm B=2); step 5 KV-budget admission (2026-07-03,
-executed as grammar-spec-batching-integration.md Phase D:
-`--kv-budget`, queue-don't-OOM, `/stats.batch` fields,
-tests/batch-kv-budget.test.ts — the `--prompt-concurrency` half still
-open); step 7's e1+e3 (min_p/XTC + penalties/logit_bias batch via
-per-row logits processors, 2026-07-02, batching-perf-path.md header
-item 2; e2 logprobs still routes serial). Still open: step 4 (extend
-join), step 6 (prompt-cache reuse), 7's e2, 9 (rolling), 10 (deferred).
-Scope: the `--batch N` continuous-batching engine (`src/serve/batch-scheduler.ts`,
-`src/serve/generation-gateway.ts`, `src/model/batched-mask.ts`,
-`src/model/batched-rotating.ts`, the `server.ts` wiring).
-Predecessor docs: [parallel-slots.md](parallel-slots.md) (the design, still
-accurate on the primitives), PLAN.md Phase 18 (history),
-[server-config.md](../reference/server-config.md) (user-facing modes doc).
-
-Mandate (Josh): batching is "kinda implemented, but is not great… make sure we
-have all the code really well done, and documented and exposed via the flags."
-
-**Standing decisions honored throughout (do not relitigate):**
-- `--batch N` is a **mode switch**, not a load-dependent fallback. Josh decided
-  this explicitly (determinism: results must not depend on concurrency, and the
-  drop-in-for-`mlx_lm.server` promise needs a predictable path). Auto-batching
-  "when >1 request arrives" was considered and **rejected** — an idle-vs-loaded
-  server would produce different numerics for the same request. This plan keeps
-  `--batch` as the UX and documents the rejection.
-- The batched lane is **compat mode** (bit-parity with mlx-lm B=N, bf16, no
-  perf flags) — see server-config.md "compat mode" section. All work below
-  preserves that guarantee; anything that can't be oracle-gated is L3 and
-  deferred (item f).
-
-**In-flight work this plan routes around (2026-07-01):**
-- **Same-ms seed collision fix** — `server.ts:1120` defaults
-  `seed: req.seed ?? (Date.now() & 0xffffffff)`; two batched rows admitted in
-  the same millisecond sample identical RNG streams. A fix is IN FLIGHT by
-  another agent. Noted as context; **not planned here**.
-- **`src/model/gemma4-base.ts` is modified in another agent's working tree**
-  (and `src/model/universal/` is a new in-flight subsystem). Items below that
-  would naturally touch cache classes are designed to live in NEW files
-  (`BatchedKVCache` beside `BatchedRotatingCache`) precisely so they don't
-  collide; coordinate before touching gemma4-base.ts.
-
----
-
-## 1. State map (verified against code, not docs)
-
-### 1.1 Architecture as built
-
-```
-handler (server.ts)                         GenerationGateway (generation-gateway.ts)
-  promptIdsFor → options → RequestShape ──▶   willBatch(shape)?
-  onToken closure (StopMatcher +               ├─ no  → AsyncMutex → serialRun
-    tool router + thinking splitter +          │        (= runGeneration: PromptCache
-    SSE send / collector)                      │         take/put + generate())
-                                               └─ yes → BatchScheduler.submit
-                                                          (batch-scheduler.ts)
-BatchScheduler #drive loop (detached, Bun-async, no threads):
-  acquire mutex (held first-admit → batch-empty)
-  ├─ #admit each pending row: SOLO prefill (one forward, whole prompt)
-  │    → sample token 0 → re-MERGE whole batch per layer
-  │    (full layers: temporalView + per-row slice + mergeKVRows → new KVCache;
-  │     rot layers: BatchedRotatingCache.merge)
-  ├─ #step: forward [B,1] (full layers wrapped per-step in
-  │    BatchedDecodeMaskCache; rot caches persistent), per-row sample on
-  │    [1,V] slices, SYNC readback [B], clearCache()
-  ├─ #emit per row (EOS → "stop" w/o onToken; onToken false → "stop";
-  │    generated ≥ maxTokens → "length")
-  └─ #applyFilter(keep): full layers filterKVRows → new KVCache;
-       rot layers filter() in place
-```
-
-- **Scheduler loop shape**: iteration-level (continuous) scheduling — admit
-  between steps, evict after each step. Correct design; the loop-body
-  *mechanics* carry the debt (§2).
-- **Admission today**: count-only — `#running.length < #maxBatch`
-  (batch-scheduler.ts:138). No byte projection, no KV budget. The server's
-  per-request `context_over_budget` 400 (server.ts:~1780, from `fit()`) checks
-  each request against **single-sequence** max-safe-context; N rows can jointly
-  exceed the budget (documented as limitation #2 in server-config.md).
-- **Eviction**: per-step `#applyFilter` (batch-scheduler.ts:327-355) —
-  full-attention layers rebuild a fresh `KVCache` from `filterKVRows`
-  (takeAxis on B); rotating caches filter in place including padding
-  reduction. Matches mlx-lm `filter` semantics (cache.py:1016-1033) except
-  ours doesn't do the `min_left_pad` shift-left for full layers (mlx-lm
-  reduces padding after filter; ours keeps stale left pad until the next
-  merge — correct but wasteful width).
-- **Join**: `#admit` re-merges the ENTIRE batch (batch-scheduler.ts:193-254):
-  per layer, temporalView of the running batch, B per-row slices, re-concat
-  with the new row. O(B·S) traffic per join **plus** ~2B slice kernels per
-  layer. mlx-lm instead `extend`s (cache.py:1035-1078): pad self+other once,
-  one concat — and prefills joiners as a right-padded *batch* (see item a/d).
-
-### 1.2 The `willBatch` gate — exact predicate today
-
-`generation-gateway.ts:113-127`: batches iff **all** of
-
-```
-batch > 1
-AND model is not DiffusionGemmaModel          (non-AR)
-AND !hasVision                                 (offset-0 prefill + bidir mask)
-AND !hasAdapters                               (loraState is one global field)
-AND !hasRepetitionPenalty                      (no per-row logits processors)
-AND !hasLogitsExtras                           (min_p ∥ xtc_probability ∥ logit_bias
-                                                ∥ presence ∥ frequency — one family gate)
-AND !wantsLogprobs                             (non-stream logprobs/top_logprobs capture)
-AND !userSeed                                  (body.seed !== undefined)
-AND !kvQuant                                   (request carries kvConfig/kvBits —
-                                                i.e. any explicit --kv-quant)
-```
-
-`RequestShape` is built **twice**, near-identically: chat (server.ts:1813-1826)
-and completions (server.ts:2085-2096, `hasVision: false`). mlx-lm's gate for
-comparison (server.py:371-374, 685-686): `draft_model is None AND all caches
-hasattr 'merge' AND args.seed is None`. **We have no cache-capability check**
-— see debt D1.
-
-### 1.3 Which caches support which dynamic-B ops
-
-| Cache | merge | filter | extend | extract | batched mask | verified vs |
-| --- | --- | --- | --- | --- | --- | --- |
-| `KVCache` (full attn) | ✅ `mergeKVRows` (batched-mask.ts:171) | ✅ `filterKVRows` (:196) | ❌ (join = re-merge) | ✅ implicit (per-row slice in `#admit`) | ✅ `BatchedDecodeMaskCache` wrapper | mlx-lm `BatchKVCache` dynamic golden (gated) |
-| `RotatingKVCache` (sliding) | ✅ `BatchedRotatingCache.merge` (batched-rotating.ts:273) | ✅ `.filter` (:233) | ❌ | ✅ `.temporalView` (:207) | ✅ built-in `makeMask` incl. ring-roll | mlx-lm `BatchRotatingKVCache` model-free golden (FAST, incl. ring-wrap) |
-| `SSMCache` (Qwen3.5 gated-DeltaNet, qwen3-delta.ts:184) | ❌ | ❌ | ❌ | ❌ (no `temporalView`) | ❌ (`makeMask` returns empty; comment says "B=1 single-stream") | — |
-| `QuantizedKVCache` / rotating-quant | n/a — excluded by the `kvQuant` gate (bf16-only v1) | | | | | no oracle exists (L2/L3, item f) |
-
-**⇒ D1 (bug): a Qwen3.5 server with `--batch N` errors every batched
-request.** The scheduler's constructor classifies `SSMCache` as `"full"`
-(batch-scheduler.ts:105, `instanceof RotatingKVCache ? "rot" : "full"`), then
-`#admit` casts it to `KVCache` and calls `temporalView()`
-(batch-scheduler.ts:200-201) → TypeError; `#drive` catches and rejects the row,
-so **every** batched request 500s. `willBatch` has no family/capability gate
-(only DiffusionGemma is excluded). And **server-config.md:151-153 claims the
-opposite** ("full-attention (CPM), sliding-window (Gemma), and hybrid
-gated-DeltaNet (Qwen3.5) — all batch"). The doc claim is false today. Note an
-oracle for the real fix exists: mlx-lm's `ArraysCache` implements
-`filter/extend/extract/finalize/merge` (cache.py:632-702), which is how mlx-lm
-batches Mamba-family state.
-
-### 1.4 Test coverage matrix
-
-| Behavior | Test | Tier | Gate quality |
-| --- | --- | --- | --- |
-| decode mask math (`buildBatchedDecodeMask`), `mergeKVRows`/`filterKVRows` pure | `tests/batched-decode-mask.test.ts` | FAST (no model) | unit, exact |
-| rotating merge/decode/mask/filter incl. **ring-wrap** | `tests/batched-rotating.test.ts` | FAST (fixture) | **oracle-gated** — replays `scripts/gen-rotating-golden.py` vs mlx-lm `BatchRotatingKVCache`, per-step mask + keys |
-| batched prefill+decode forward, 4 models | `tests/batched-decode-parity.test.ts` | GATED `MLX_BUN_TEST_BATCH_DECODE=1` + weights | **oracle-gated** — bit-exact vs mlx-lm B=2 goldens (`scripts/gen-batched-golden.py`) |
-| dynamic-B join/leave ({A,B}→+C→−A) | same file, CPM | GATED | **oracle-gated** — token-for-token vs `scripts/gen-batched-dynamic-golden.py` (mlx-lm merge/extract/filter) |
-| scheduler orchestration (admit/step/evict/join) | `tests/batch-scheduler.test.ts` | GATED | **KL-gated** teacher-forced (tol 1e-2) CPM; Gemma 12B greedy vs mlx-lm B=2 golden |
-| server wiring: Option-B engage, `/stats`, concurrent completion, SSE fan-out, serial-lane coexist (user-seed drain, no deadlock) | `tests/batch-serving.test.ts` | GATED (live in-process server) | **smoke** — coherence assertions, no numeric oracle |
-| prefill batched mask (training side) | `tests/train-batch{,-e2e}.test.ts` | FAST / gated | oracle (per-row loss == solo) |
-
-**Not tested anywhere:** row-failure containment (one row's error vs the
-batch), serial-lane starvation/fairness under sustained batchable load,
-KV admission (doesn't exist), batched Qwen3.5 (would fail, §1.3),
-ring-wrap **end-to-end** on a real Gemma forward (model-free math is gated;
-the long-context forward golden is the known open L1 follow-up),
-batched-lane usage-timing stats (currently hardwired zeros).
-
-### 1.5 What the flags/docs currently say (accurate except one claim)
-
-`--batch <n>` (cli.ts:750-757; `--decode-concurrency` accepted as alias),
-`--kv-quant` interplay (Option B: unset+batch ⇒ bf16; explicit ⇒ those requests
-serial, warned at server.ts:1065-1070), `/stats.batch`
-{configured, batched, active_rows} (server.ts:1437-1441). server-config.md's
-"Execution modes", lane-picker table, compatibility matrix, compat-mode
-rationale, and known-limitations list are **verified accurate against code**
-— except the Qwen3.5 batching claim (D1) and it doesn't mention the curve/
-signal endpoints' lock bypass (D3, which is a bug to fix, not to document).
-
----
-
-## 2. Code-quality assessment
-
-**Verdict:** the *primitives* are excellent — oracle-gated, well-commented,
-disposal-disciplined (`batched-mask.ts`, `batched-rotating.ts` are among the
-cleanest files in the repo). The *orchestration* (`batch-scheduler.ts`,
-~360 lines, readable) is correct-by-gate but **naive as an engine**: it has
-none of the serial loop's hard-won loop hygiene, its failure domain is the
-whole batch, and the server grew a second mutual-exclusion domain that two
-endpoints silently don't participate in. Specific debt, ranked:
-
-- **D1 — Qwen3.5 batched = guaranteed request failure + false doc claim.**
-  batch-scheduler.ts:105 + :200-201 vs qwen3-delta.ts:184 (`SSMCache`, no
-  `temporalView`); server-config.md:151-153. Fix now: a cache-capability gate
-  (mirror mlx-lm server.py:371-374) in `willBatch` — batchable iff every
-  `makeCache()` element is `KVCache | RotatingKVCache` — plus the doc
-  correction. The real SSM batch path is item (h), later. *(confidence 95)*
-  **RESOLVED**: gate landed 2026-07-01; the full SSM batch path landed
-  2026-07-02 (batching-perf-path.md P5) — server-config's Qwen3.5 claim is
-  true again.
-
-- **D2 — `#admit` materializes full-prompt logits.**
-  batch-scheduler.ts:173-174 runs `logitsFromHidden(h)` on the **whole**
-  `[1,Lp,H]` hidden, then slices the last position. The lm_head matmul
-  computes `[1,Lp,V]` at eval — at Gemma's V=262k and an 8k prompt that is a
-  ~4.3 GB bf16 transient *per admit* (plus the wasted GEMM). The serial path
-  slices `hLast` FIRST (generate.ts:466-468) precisely to avoid this. One-line
-  reorder + the gated suites re-run. *(confidence 95)*
-
-- **D3 — two mutual-exclusion domains; curve `/generate` and `/signal` bypass
-  the gateway.** The old serialized queue (`enqueue`, server.ts:946-951) used
-  to be THE GPU lock; generation moved to the gateway's `AsyncMutex`, but
-  `/generate` (curve designer, server.ts:1663 — `enqueue(() =>
-  runGeneration(...))`) and `/signal` (server.ts:1608 — `enqueue` + raw
-  `model.forward`) still run GPU work (and `PromptCache.take/put`) under
-  `enqueue` only ⇒ they can run **concurrently with batched decode steps**
-  (GPU contention + the prompt-cache "single owner" invariant from
-  parallel-slots.md broken). Adapter mount/unmount (server.ts:1578, 1589) sit
-  on `enqueue` too — mount is registry-level, but unmount disposing adapter
-  arrays concurrently with a serial-lane generation using them is the same
-  class of race. The `runGeneration` doc comment "Must be called inside the
-  queue" (server.ts:960) is stale. Fix: one lock — expose
-  `gateway.runExclusive(fn)` and route curve/signal/adapter-unmount through
-  it; fix the comment. *(confidence 90 on curve//signal; 80 on unmount)*
-
-- **D4 — the batched loop lacks the serial loop's pipelining + clearCache
-  cadence** (= reports/kernel-perf-review-2026-07-01.json backlog #6, area 7
-  verdict). Three sub-items, all in `#step`/`#admit`:
-  1. **No `asyncEval` pipelining**: `#step` builds one step's graph then
-     synchronously reads all B tokens (`toFloat32`, batch-scheduler.ts:286)
-     — every step eats a full dispatch+readback bubble. Serial overlaps step
-     n's readback with step n+1's compute (generate.ts:482, 539); mlx-lm's
-     `GenerationBatch._step` does exactly this batched (`mx.async_eval`
-     next, `mx.eval` current — generate.py:1367-1373).
-  2. **`clearCache()` EVERY step** (batch-scheduler.ts:293) — trashes the
-     buffer pool each token, so every step re-allocates its transients.
-     Serial clears every 256 tokens (generate.ts:571); mlx-lm batched every
-     512 steps (generate.py:1778-1779).
-  3. **No prefill chunking in `#admit`** — the whole prompt goes through one
-     forward (batch-scheduler.ts:172), so a long prompt is an unbounded
-     transient (serial chunks at `prefillChunkSize` with per-chunk eval +
-     clearCache, generate.ts:440-456; mlx-lm at `prefill_step_size=2048`).
-     Worse, admits run to completion before the next `#step`, so **every
-     running row stalls for the entire prefill of every joiner**
-     (head-of-line blocking; mlx-lm interleaves one decode step with one
-     prefill chunk per loop iteration — generate.py:1760-1844).
-  *(confidence 90; perf claims need bench numbers per house rules)*
-
-- **D5 — whole-batch failure domain.** Any throw in `#step` — including a
-  row's own `onToken` (awaited inline, batch-scheduler.ts:299; an SSE
-  controller error qualifies) — is caught in `#drive` (:153-158) and
-  **rejects every row and drops the entire batch KV**. mlx-lm removes the
-  offending uid and keeps the batch (server.py:913-919). Also: `#admit` has
-  no try/finally around the prefill, so `solo` caches + `ids` leak if the
-  forward throws (:170-173); onToken should be wrapped per-row so one
-  consumer's exception evicts one row. *(confidence 90)*
-
-- **D6 — serial-lane starvation under sustained batchable load.** The
-  scheduler holds the mutex while `running ∪ pending ≠ ∅` and **keeps
-  admitting** new batchable arrivals (batch-scheduler.ts:135-145); a queued
-  serial-lane request (vision, adapter, seed…) waits for the batch to empty,
-  which under continuous traffic is *never*. mlx-lm's drain sets
-  `drain_batch=True` and **stops admitting** until the incompatible request
-  has run (server.py:826-830). Fix: gateway signals "serial waiter present" →
-  scheduler pauses admission (finish running rows, release, FIFO mutex hands
-  over, then resume). Single-user appliance softens this, but agent fan-out
-  is exactly the batching use case, so it's real. *(confidence 85)*
-
-- **D7 — zeroed stats + duplicated request-shape/sampling logic.**
-  - Batched stats hardwire `prefillMs/decodeMs/…Tps = 0`
-    (generation-gateway.ts:170) → SSE `usage` timing and any logging report
-    zeros for batched requests; `cachedTokens` 0 is documented, the timings
-    are just missing accounting.
-  - `RequestShape` construction duplicated chat vs completions
-    (server.ts:1813 vs 2085) — extract one `shapeFor()` helper.
-  - `maxTokens` fallback drift: gateway `?? 512` (generation-gateway.ts:160),
-    chat finish `?? 1024` (server.ts:1904), completions finish `?? 512`
-    (server.ts:2098) — all dead in practice (server defaults 65_536 at
-    :1116) but a trap for library callers; unify.
-  - The `#emit` EOS/length/onToken contract mirrors `generate()` by comment
-    only (batch-scheduler.ts:306-316) — drift risk; a shared contract test
-    (same fixture through both lanes) would pin it.
-  - Full-layer `filter` keeps stale left-padding (no `min_left_pad`
-    shift-left, cf. cache.py:1027-1033) — width waste, not correctness.
-
-None of this contradicts the gates that exist: the numerics ARE oracle-proven.
-The debt is engine mechanics, failure domains, and the second lock.
-
----
-
-## 3. Gap plan (implementation-ready, per item)
-
-### (a) `extend` join op — stop re-merging the running batch
-
-**Port target:** mlx-lm `BatchKVCache.extend` (cache.py:1035-1078) and
-`BatchRotatingKVCache.extend` (cache.py:1369-1416). Semantics: right-justify
-both caches to a common `_idx`/width (one `mx.pad` each), concat on B, sum
-left-padding adjustments. One pad+concat per layer per join, vs today's
-per-row slice storm + full rebuild.
-
-**Design:** introduce **`BatchedKVCache`** (new file or in batched-mask.ts) —
-the full-attention twin of `BatchedRotatingCache`: owns
-`keys/values/leftPad/offset`, methods `merge` (subsumes `mergeKVRows`),
-`filter` (subsumes `filterKVRows` + adds the min-left-pad shift),
-**`extend`**, `extract(row)` (the per-row slice `#admit` currently inlines),
-`makeMask` + `ropeOffsetArr` (subsumes the per-step `BatchedDecodeMaskCache`
-wrapper rebuild and the scheduler's shared `#fullLeftPad` field —
-batch-scheduler.ts:93, a state smell). `BatchedRotatingCache.extend` follows
-the same pad-to-common-idx recipe in temporal order (both sides are kept
-temporal by our merge, so no dynamic-roll needed for the N=1-update scope —
-verify against the golden, don't assume). Scheduler `#admit` becomes:
-solo-prefill → `inners[layer].extend(soloRow)` — no re-merge, no wrapper churn.
-**Does not touch gemma4-base.ts** (in-flight ownership).
-
-**Oracle:** extend `scripts/gen-batched-dynamic-golden.py` to drive the join
-through mlx-lm's real `extend` (it already drives merge/extract/filter);
-assert the same {A,B}→+C→−A per-row trajectories token-for-token, CPM L1 +
-a Gemma variant for the rotating extend. Existing gated suites must stay
-green unchanged (numerically `extend ≡ re-merge`; the golden proves it).
-
-**Effort: M.**
-
-### (b) Prompt-cache reuse under batching
-
-**mlx-lm behavior (the model to follow):** on insert,
-`prompt_cache.fetch_nearest_cache` → pass the pre-warmed per-sequence cache
-into the batch (server.py:751-767); on row finish, extract the row's cache and
-`insert_cache` it back keyed by all tokens (server.py:900-907).
-
-**Design (parallel-slots.md's sole-owner rule, kept):** all `PromptCache`
-take/put happen **inside the scheduler loop** (plus the serial lane, which
-already does — the two are mutex-exclusive, and D3's fix closes the remaining
-holes).
-- `#admit`: `promptCache.take(promptIds, "")` (batched lane is always
-  adapter-less, so namespace ""); on hit, the entry's caches ARE the solo
-  caches — forward only the uncached tail (the trim/`cachedTokens` logic
-  exists in generate.ts; lift the prefix-trim helper rather than reimplement —
-  memory: eval-reuse-serving-prompt-path). `BatchStats.cachedTokens` becomes
-  real.
-- Row finish (`#finish`) or eviction: `extract(row)` (from item a) →
-  `restoreState` into fresh solo `KVCache`/`RotatingKVCache`s →
-  `promptCache.put(rowTokens, caches, "")`. Cost: O(S_row) copy per finished
-  row, off the step hot path (do it after `#applyFilter`).
-- Byte interplay with (c): mirror mlx-lm's
-  `prompt_cache.trim_to(total − activeBatchBytes)` (server.py:795-799) so the
-  LRU + live batch share one ceiling.
-
-**Gate:** batch-serving test asserting `cached_tokens > 0` on an identical
-repeat under `--batch 2`; parity assertion that a warm-start row's trajectory
-equals the cold row's (same-B, bit-exact — prefix KV is identical by
-construction); serial suite untouched.
-
-**Effort: M–L** (the prefix-trim + rotating-cache restore corners are where
-the time goes; RotatingKVCache entries past the window are non-trimmable —
-mirror the serial path's existing handling).
-
-### (c) KV-budget admission (B×S_max) + `--prompt-concurrency`
-
-**Fact-check first:** mlx-lm does NOT byte-budget admission — it caps counts
-(`--decode-concurrency` 32, `--prompt-concurrency` 8), optionally swaps KVCache
-→ RotatingKVCache via `max_kv_size`, and byte-trims only the *prompt cache*
-against `--prompt-cache-bytes` minus live batch bytes. So byte-budget admission
-is OUR addition (rung 2 of parallel-slots.md), consistent with the byte-capped
-`PromptCache` precedent and the existing `fit()` admission (server.ts:1012).
-
-**Design:**
-- Projected row bytes = Σ_layers 2·bytes_bf16(H_kv·D·S_cap) where
-  S_cap = min(promptLen+maxTokens, window) for rotating layers and
-  promptLen+maxTokens for full layers (a `projectKvBytes(model, promptLen,
-  maxTokens)` helper next to `fit()`, which already knows per-layer shapes).
-- Budget = explicit `--kv-budget <GB>` else derived: `fit()`'s usable bytes −
-  weights − prompt-cache cap − headroom. Admission: admit while
-  Σ projected(running+candidate) ≤ budget; otherwise leave in `#pending`
-  (FIFO — no reordering, keeps fairness and determinism of arrival order).
-  `maxTokens` default 65_536 makes naive projection admit ~1 row — so
-  projection uses a decode-allowance term (e.g. min(maxTokens, 4096) beyond
-  prompt) with the true backstop being re-check-on-growth: on each STEP-chunk
-  cache growth, re-evaluate; if over budget, stop admitting (never evict a
-  running row). Document the estimate honestly.
-- **`--prompt-concurrency`: accept for drop-in** (we already accept
-  `--decode-concurrency`), mapped to "max joiners admitted per loop
-  iteration" — which becomes meaningful exactly when (d)'s interleaved
-  chunked admit lands. Until then accept + warn like `--decode-concurrency`'s
-  semantic note in cli.ts:744-748. `/stats.batch` grows
-  `{pending_rows, kv_bytes, kv_budget_bytes}`.
-
-**Gate:** unit tests on the projection math (pure); a gated live test that
-M oversized requests are admitted ≤ budget-implied width and all complete
-(queueing, not OOM). Perf numbers only via bench-serving-load on a clean
-machine (Josh-gated).
-
-**Effort: M.**
-
-### (d) Pipelining + clearCache cadence + chunked/interleaved admit
-(the backlog-#6 transplant — pairs with fixing D2, D4)
-
-1. **Slice-before-head in `#admit`** (D2): `hLast = h.slice(last)` →
-   `logitsFromHidden(hLast)`. One-line; do first.
-2. **Chunked prefill in `#admit`**: loop `prefillChunkSize` (reuse the serial
-   constant) with `evalAll(cache.state)` + `clearCache()` per chunk
-   (generate.ts:440-456 pattern).
-3. **Interleave admit with steps**: `#drive` admits at most one prefill
-   *chunk* (per joiner, up to `--prompt-concurrency` joiners) between decode
-   steps instead of running each admit to completion — bounds running-row
-   stall to one chunk. (This restructures `#admit` into a resumable
-   per-joiner state; mlx-lm's `_currently_processing` is the reference,
-   generate.py:1760-1844.)
-4. **Pipelined decode**: mirror mlx-lm's `GenerationBatch._step`
-   (generate.py:1320-1373): keep step n's sampled `[B]` token array on
-   device, `asyncEvalAll` it, build step n+1's graph from the **unread**
-   array, then read step n. Eviction moves after the read: rows evicted at
-   step n have already contributed a step-n+1 KV write — harmless, `filter`
-   drops the row (mlx-lm does exactly this). EOS rows' extra write is also
-   how mlx-lm behaves, so parity holds by construction.
-5. **clearCache cadence**: every 256 steps (serial's constant; mlx-lm uses
-   512 — either is defensible, pick 256 for house consistency and comment
-   the divergence).
-
-**Gate:** numerics unchanged ⇒ the teacher-forced KL suite and the mlx-lm
-goldens must pass **unchanged** (pipelining is scheduling, not math);
-batch-serving smoke; A/B throughput via `scripts/bench-serving-load.ts`
-(quotable numbers clean-machine only).
-
-**Effort: M** (item 3 is the bulk; 1/2/5 are S alone).
-
-### (e) Gate widening — which serial-only features can batch safely
-
-Ranked by value/risk. Each needs per-row state listed; each lands with its own
-oracle before the `willBatch`/`hasLogitsExtras` split changes.
-
-1. **min_p + XTC — S, low risk, do first.** Pure per-row *samplers*; the
-   batched lane already builds a per-request sampler closure
-   (generation-gateway.ts:150-156 → `makeSampler(options)`), which already
-   implements min_p/XTC for serial. Per-row state: none beyond the closure
-   (XTC needs `xtcSpecialTokens`, already in options). Change: move
-   `minP`/`xtc*` out of `hasLogitsExtras` into the batchable set. Gate:
-   teacher-forced determinism (same seed ⇒ same tokens, batched == serial,
-   per row — sampling happens on the `[1,V]` row slice so it's the SAME code
-   path); note the seed-collision fix must land first for two same-ms hot
-   rows to be meaningfully independent (in flight).
-2. **logprobs / top_logprobs — M, low-medium risk.** Non-stream only
-   (server.ts:1797-1805 keeps streams capture-free). The scheduler's sampler
-   already computes `toLogprobs` per row; extend `RowSampler` to optionally
-   return extras (the serial `sampleStep` extras shape, generate.ts) and have
-   `#emit` pass them to `onToken` — the handler's `LogprobsCollector` then
-   works unchanged. Per-row state: none (readback per emitted token, rows
-   that asked for it only). mlx-lm computes logprobs for every batched row
-   natively, so behavior-parity is clean.
-3. **penalties (repetition/presence/frequency) + logit_bias — M, medium
-   risk.** Needs per-row token history + per-row logits processors applied to
-   the `[1,V]` slice before sampling — exactly mlx-lm's per-row
-   `logits_processors` loop (generate.py:1338-1349, TokenBuffer). We have the
-   processors (sampler.ts, serial); per-row state: a bounded history array
-   (context sizes 20 by default) per row, filtered on eviction. logit_bias is
-   stateless (easiest of the family) — could ship with tier 1 if the family
-   gate is split per-feature. Gate: teacher-forced vs serial with identical
-   forced trajectories (processors are deterministic given history).
-4. **LoRA same-adapter-set batching — L, defer.** Design already decided
-   (group-by-adapter + drain, parallel-slots.md Fallbacks); needs adapter-set
-   as a batch-compatibility KEY in the scheduler, not just a boolean gate.
-   Not before (a)–(d).
-
-Never batch: vision (offset-0 bidirectional prefill), user seed
-(reproducibility ⇒ solo — matches mlx-lm), explicit kv-quant (item f),
-DiffusionGemma (non-AR).
-
-### (f) Quantized-KV batched — L3 novel extension (design sketch ONLY, deferred)
-
-No ancestor does it (mlx-lm's `to_quantized` raises on batch caches; optiq
-doesn't batch) ⇒ no bit-exact oracle, per the three-level fidelity tree this
-is KL + 6-task + benchmark gated. Sketch: `BatchedQuantizedKVCache` mirroring
-(a)'s `BatchedKVCache` over (packed, scales, biases) triples — merge/filter/
-extend are the same B-axis surgery per component; attention falls to
-`quantizedSdpaUnfused` with the `[B,1,N,S]` bool mask (the fused decode kernel
-requires `mask.mode===""` and the tiled path 2-D causal — both disqualified,
-known perf debt from the modes analysis). Value: memory-density (4-bit KV ×
-batching compound, ~4× rows per GB) on the 24 GB machine. **Do not start**
-until (a)–(d) are landed and a concrete density need exists; then it begins
-with a KL harness (batched-quant vs batched-bf16 at same B), never a
-trajectory diff.
-
-### (g) Docs + flags
-
-- **server-config.md**: fix the Qwen3.5 batching claim (D1) — with the
-  capability gate landed the honest row is "hybrid-cache models (Qwen3.5)
-  route to the serial lane; batched SSM is a later port (mlx-lm ArraysCache
-  is the oracle)". Update the known-limitations list as (a)–(e) land (each
-  item literally deletes a numbered limitation: #1 prompt cache, #2
-  admission, #5 extend). Add `--kv-budget`, `--prompt-concurrency`,
-  `/stats.batch.{pending_rows,kv_bytes}` when they exist. Add one line
-  documenting the auto-batch rejection ("mode, not load-fallback —
-  determinism", the standing decision).
-- **parallel-slots.md**: refresh the STATUS block (S2 admission half, extend,
-  prompt-cache reuse) to point HERE for the v2 execution plan; it remains the
-  primitive-design record.
-- **`--batch` stays the UX.** No smarter default. `--decode-concurrency`
-  alias kept; `--prompt-concurrency` accepted per (c). No new mode flags.
-- **STATUS.md / PLAN.md Phase 18**: check off items as they land (house
-  rule: write conclusions down).
-
-### (h) (surfaced by this audit, not in the original list) SSM batched path
-
-Port mlx-lm's ArraysCache batch ops for `SSMCache` (conv/recurrent state are
-`[B,…]` tensors; merge/filter/extend are trivial B-axis concat/take — no
-padding subtleties since there's no sequence axis; the mask is the harder bit:
-Qwen3.5's linear layers need an `ssm_mask` under left-padding). Oracle exists
-(mlx-lm batches these models). Schedule after (a); until then the capability
-gate (D1 fix) keeps Qwen servers correct-but-serial.
-
----
-
-## 4. Sequencing
-
-Order minimizes risk (correctness first, then the engine, then features),
-respects in-flight work (seed fix in flight — untouched; gemma4-base.ts owned
-elsewhere — all new cache code goes in new/batched-* files), and gives every
-step a gate.
-
-| # | Item | Effort | Test gate | Notes |
-| --- | --- | --- | --- | --- |
-| 1 | ✅ **DONE 2026-07-01** — **Hotfix bundle**: D2 slice-before-head; D1 capability gate + server-config.md correction; D3 route curve/`/signal`(/adapter-unmount) through `gateway.runExclusive`; stale comment server.ts:960 | **S** | gated batch suites unchanged; fast suite; new unit: willBatch(Qwen)=false | no numerics change anywhere |
-| 2 | ✅ **DONE 2026-07-01** — **D5/D6 failure + fairness**: per-row onToken containment (evict row, keep batch); `#admit` try/finally; drain-on-serial-waiter | **S–M** | new gated test: throwing onToken evicts one row, siblings complete; drain test (serial completes under sustained batchable load) | mlx-lm `remove` semantics |
-| 3 | ✅ **DONE 2026-07-01** — **(d) engine hygiene**: chunked+interleaved admit, pipelined decode, clearCache cadence | **M** | KL suite + mlx-lm goldens UNCHANGED; bench-serving-load A/B (clean machine, Josh-gated for quotables) | closes perf-review backlog #6 |
-| 4 | **(a) `BatchedKVCache` + `extend`** (full + rotating) — **still open** | **M** | extended dynamic golden (mlx-lm `extend` driven); existing suites green | new files only — no gemma4-base.ts conflict |
-| 5 | ✅ **DONE 2026-07-03** (admission half; `--prompt-concurrency` still open) — **(c) KV-budget admission** — executed as grammar-spec-batching-integration.md Phase D | **M** | projection unit tests; gated queueing test (tests/batch-kv-budget.test.ts); `/stats` fields | depends on 3 (interleaved admit) for prompt-concurrency semantics |
-| 6 | **(b) prompt-cache reuse under batching** — **still open** | **M–L** | `cached_tokens>0` gated test; warm==cold row parity; serial suite | depends on 4 (`extract`) + 5 (byte interplay) |
-| 7 | **(e) gate widening**: ✅ e1 min_p/XTC + e3 penalties/logit_bias **DONE 2026-07-02** (per-row logits processors, batching-perf-path.md header item 2); e2 logprobs **still open** (routes serial) | **S+M+M** | per-feature: batched==serial teacher-forced/deterministic-seed tests | e1 waits for the seed fix to merge |
-| 8 | ✅ **DONE 2026-07-02** — **(h) SSM batched path** (Qwen3.5) — landed as batching-perf-path.md P5, token-exact vs mlx-lm B=2 | **M–L** | mlx-lm B=N golden for a Qwen3.5 model (ArraysCache oracle) | un-does the item-1 gate for Qwen; doc row flips back |
-| 9 | **(g) docs pass** | **S** | doc-vs-code re-audit (this file's §1 as the checklist) | rolling — update per landing, final sweep here |
-| 10 | **(f) quantized-KV batched** | **L** | KL + 6-task + density benchmark — design doc first | DEFERRED — needs a concrete density case |
-
-D7's small items (shared `shapeFor()`, `maxTokens` fallback unification, real
-batched timing stats, filter min-left-pad shift, shared emit-contract test)
-slot into whichever of steps 2–4 touches the neighboring code; none justify a
-standalone step.