mlx-bun 0.2.0 → 0.3.0

package/docs/design/dspark-speculative-decoding.md

@@ -1,318 +0,0 @@
-# DSpark speculative decoding for mlx-bun
-
-Implementation of DeepSeek's **DSpark** ("Confidence-Scheduled Speculative
-Decoding with Semi-Autoregressive Generation", `DSpark_paper.pdf`) against this
-repo's models. DSpark = a **DFlash** parallel backbone (KV-injected, all γ draft
-tokens in one pass) **+ a lightweight sequential head** (Markov, Eq 5, or RNN,
-Eq 6) that adds intra-block dependency (the *semi-autoregressive* part) **+ a
-confidence head → STS calibration → hardware-aware prefix scheduler**. Decode
-becomes draft → verify → accept; output is lossless (bit-identical to the target
-at any temperature); only decode speed (τ) changes.
-
-> **DFlash vs DSpark:** DFlash is *only* the parallel backbone (prior work) and
-> suffers suffix decay. DSpark adds the sequential head + confidence-scheduled
-> verification. The sequential Markov head is what makes this DSpark.
-
-## Two implementations (flag-selected, both in `src/spec/dspark/`)
-
-| | `module.ts` (v1) | `module-dflash.ts` (v2, **the real one**) |
-|---|---|---|
-| context | **single vector** (final hidden at the anchor) fused as one token | faithful **multi-layer KV injection** (Eq 2–3): H_ctx = RMSNorm(W_c·[H^{l1};…;H^{lm}]) over the FULL context, re-projected into every draft layer's K/V |
-| layers | 2 | 5 |
-| status | **superseded** — dropped the paper's core idea (kept as baseline) | the actual DSpark model (Markov variant) |
-
-v1 was a shortcut that removed the load-bearing mechanism; it never tested the
-paper. v2 is faithful. Checkpoints stamp `variant` in `dspark.json`; the runtime
-loads the matching module. (`variant:"dflash"` should be renamed to `"dspark"` —
-see Open items.)
-
-## The mechanism (v2, paper Eq 2–3)
-
-- **Eq 2 (context):** tap the target's hidden states at layers `{l1…lm}` (e4b
-  default {20,31,41,42}, the last = post-finalNorm), concat on the feature axis,
-  project to the draft width: `H_ctx ∈ [L_ctx, d]`. Extraction via a parity-safe
-  `model.hiddenTap` (no-op when off) added to `gemma4.ts` `forwardLayers`.
-- **Eq 3 (injection):** every draft layer forms `K_i=[W_i^K H_ctx; W_i^K H_d]`,
-  `V_i` likewise, `Q_i=W_i^Q H_d`; the γ-block queries attend bidirectionally
-  over `[context ++ block]`. Context is read-only memory re-projected fresh each
-  layer. The draft projects the target's HIDDEN STATES with its OWN Wk/Wv (not
-  the target's K/V — that's the separate `GemmaAssistantDrafter` baseline).
-- **Markov head (Eq 5):** `B_k = W1[x_{k-1}]·W2`, r=256, W2=0 init → starts as
-  pure DFlash, τ climbs as intra-block dependency is learned.
-- **Confidence head (Eq 7):** `c_k = σ(wᵀ[h_k; W1[x_{k-1}]])`.
-- **Loss (Eq 9–12):** `0.1·CE + 0.9·TV + 1.0·BCE`, position-weighted.
-- **Lossless verify** (`generate-dflash.ts`): greedy = longest-prefix match;
-  temp>0 = `min(1,p/q)` accept + residual `norm(relu(p−q))` resample. Growing
-  multi-layer H_ctx buffer; rollback via `trim(n, bypass=true)` (physically
-  slices the rejected tips past the sliding window — see `gemma4-base.ts`).
-
-## Results & findings (measured, M1 Max 32GB, e4b OptiQ-4bit)
-
-**The architecture is CORRECT — proven by an overfit test.** Trained on 3
-articles, eval on the same 3: per-position acceptance **~0.75**, τ **3.24** —
-squarely paper-range (0.6–0.9 / τ 3–4). So the faithful DFlash, correctly
-implemented, reaches the paper's numbers.
-
-**Getting there required finding a real bug — an off-by-one in the p^t target
-alignment** (the TV loss, weight 0.9). Block position k predicts `x_{t+1+k}`,
-whose target distribution is `softmax(LM_head(h_{t+k}))`; the data gathered the
-hidden at `t+k+1` (shifted forward one). Present in **both v1 and v2** — so every
-τ number before the fix was trained against a shifted target. Fixed in
-`data-dflash.ts` (`blockIdx = g+k`, k=0..γ-1). Before fix: per-pos ~0.08. After:
-~0.17 generalizing, ~0.75 overfit.
-
-**Two things stand between "correct architecture" and "net speedup," now
-separated:**
-1. **Data.** 160 articles → generalizes only to per-pos ~0.17 (plateaus, doesn't
-   climb); overfit on 3 proves capacity. The paper uses 1.3M samples × 10 epochs.
-   → needs real data scale.
-2. **Target speed (the decisive one).** Spec decode amortizes the fixed draft
-   overhead only when the *target forward* is expensive. Measured decode:
-   **e4b 45.9 tok/s, 12b 27.5 tok/s, 27B ~15 tok/s.** On fast e4b the draft
-   overhead dominates → even τ≈3 nets ~0.4–0.5× (SLOWER). On a slow target (27B,
-   ~67 ms/forward) that overhead is a small fraction → the same τ≈3 could land
-   ~2–3× (15 → ~35–45 tok/s). **e4b is close to the worst case for spec decode;
-   the 27B agentic workload is the ideal target and the real motivation.**
-
-Also confirmed: width is NOT the ceiling (`dDraft=1024` and `2560` both plateau
-~0.17 pre-scale); the parity-safe `hiddenTap` doesn't change target numerics;
-long-context spec rollback works via the `trim(n, bypass)` physical-slice.
-
-## Serve integration (Phase 1, 2026-07-06)
-
-DSpark is now serve-loadable behind the same `--draft-model` seam as every
-other drafter — one verify/accept executor, no lane routing. The seam
-(`src/spec/source.ts`) was extended from its two-model-only shape to carry the
-target state KV-borrowing drafters need (it previously could NOT host these,
-despite the header claiming so):
-
-- `DraftProvider.open({sampler, target:{model,caches}})` — the source gets the
-  target model + its live caches.
-- `DraftSource.tapLayers?` — when set (DSpark), the serve loop taps the
-  target's prefill + every verify forward and hands the captured `[1,L,m*H]`
-  context to `prefill(ids, ctxML)` / `commit(n, kAccept, vCtxML)` (ownership
-  transfers). `draft(feed, n, stepBase, anchorHidden?)` — the assistant borrows
-  the target anchor hidden; DSpark drafts from its growing `H_ctx`.
-
-Provider selection is by artifact kind (`src/server.ts` `detectDraftKind`):
-`dspark.json` → **DflashProvider** (`src/spec/dflash-source.ts`), a `*_assistant`
-config → **AssistantProvider** (`src/spec/assistant-source.ts`, the optiq
-KV-borrowing Gemma drafter — ships a real ~1.09× γ=1 decode win with no
-training), else **TwoModelProvider**. `--draft-kind` overrides. The server pins
-`--num-draft-tokens` to the DSpark checkpoint's trained block width
-(`cfg.gamma`) so the serve loop never asks for more positions than were trained.
-
-**Unified-engine alignment (per-slot readiness).** This is the
-`DraftSource` seam the unified-engine plan names as the substrate for its
-frontier "per-slot drafting behind the `DraftSource` seam" row
-([docs/design/unified-engine-frontier-plan.md](unified-engine-frontier-plan.md)).
-Spec still forces the serial lane today (upstream `is_batchable = draft is
-None`); Phase 1 did NOT change that — it made the seam *carry per-source target
-state* (each source binds its own `TargetView{model,caches}`, anchor hidden,
-and tapped H_ctx), which is exactly what a batched executor would need to open
-one `DraftSource` per row. So the seam is forward-compatible with per-slot
-drafting; the hard part that remains is the executor, not the seam — variable
-accept-length per slot breaks a uniform-B step (see
-[spec-decode-larger-targets.md](spec-decode-larger-targets.md) caveats), so
-per-slot spec is a real batched-scheduler project, not free composition.
-
-**v1 acceptance caveat:** the serve loop verifies with TOKEN-MATCH acceptance
-(mlx-lm/optiq style, lossless at any temperature), NOT the paper's
-distribution-level rejection-sampling verify. Greedy is identical; temp>0 is
-still lossless but lower-acceptance. The richer rejection-sampling verify stays
-in the standalone `generate-dflash.ts` (the measure script). See
-[[dspark-seam-kv-borrowing]].
-
-## Files
-- `src/spec/source.ts` — the extended DraftSource/DraftProvider/TargetView seam.
-- `src/spec/assistant-source.ts` / `dflash-source.ts` — the two KV-borrowing providers.
-- `src/spec/serve-loop.ts` — the shared verify/accept executor (now tap-aware).
-- `src/model/gemma4.ts` — `hiddenTap`/`captureLayer` (parity-safe multi-layer tap).
-- `src/model/gemma4-base.ts` — `trim(n, bypass)` on rotating caches (spec rollback past the window).
-- `src/spec/dspark/module-dflash.ts` — faithful DFlash+Markov+confidence module.
-- `src/spec/dspark/data-dflash.ts` — multi-layer shards, variable-length prefix context, left-pad masking.
-- `src/spec/dspark/generate-dflash.ts` — growing-H_ctx lossless decode loop.
-- `src/spec/dspark/{loss,sample}.ts` — loss (Eq 9–12) + temp>0 sampling (shared with v1).
-- `src/spec/dspark/{module,data,generate}.ts` — v1 single-vector (superseded baseline).
-- `scripts/dspark-regen-dflash.ts` / `dspark-train-dflash.ts` (`--resume`) / `dspark-measure-dflash.ts`.
-- `scripts/dspark-dflash-smoke.ts` — CPU smoke (16/16). `scripts/dspark-smoke.ts` — v1 smoke (33/33).
-
-## Run sequence (faithful DFlash; GPU = Josh runs)
-```
-bun scripts/dspark-regen-dflash.ts --topics <topics.txt> --out <data> --max-resp 320
-bun scripts/dspark-train-dflash.ts --data <data> --out <ckpt> --iters 8000 --batch 8 [--resume] [--ddraft 2560]
-bun scripts/dspark-measure-dflash.ts --drafter <ckpt> --data <prompts.jsonl>   # τ + tok/s vs vanilla
-```
-
-## Paper components — Phase 2 (2026-07-06): the paper is CODE-COMPLETE
-
-All remaining components landed the same day (multi-agent build, adversarially
-reviewed; smoke 21/21, dspark test files 17/17, real-weights serve gate 3/3):
-
-- **Alg 1, single-user form — confidence-scheduled draft-length pruning.**
-  `forwardInfer` drops position k (and truncates the block) when
-  `c_k < thresholds[k] ?? minConf`; position 0 always survives. Activation is
-  checkpoint-driven: STS thresholds in `dspark.json` (`cfg.sts`) or the
-  `MLX_BUN_DSPARK_MINCONF` env override — **uncalibrated checkpoints draft
-  fixed-γ exactly as before** (zero behavior change). Losslessness invariant:
-  pruning changes how many positions the target verifies, never what's emitted
-  (gated: truncation-never-redraw, prefix-identical to the unpruned block).
-- **Variable-length draft contract.** `DraftSource.draft()` returns 1..n —
-  the RETURN LENGTH is authoritative; the serve loop and `dflashGenerate`
-  verify/trim/re-feed over `d = drafts.length`. Two-model/assistant return
-  exactly n (unchanged).
-- **STS calibration (§3.2.1).** `src/spec/dspark/calibration.ts`
-  (`fitStsThresholds`: per position, smallest τ whose Laplace-smoothed
-  `P(accepted | conf ≥ τ)` ≥ target; pos 0 → 0, under-sampled → 0 (don't
-  prune on thin evidence), unreachable target → 1.0) +
-  `scripts/dspark-calibrate.ts` (GPU: greedy unpruned rounds via
-  `dflashGenerate`'s `onRound` hook → fit → patch `config.sts` into
-  dspark.json in place). ⚠ estimator shape is our reading — paper PDF absent.
-- **RNN sequential head (Eq 6 — ⚠ design-doc-faithful shape, paper PDF
-  absent, flagged in code).** `cfg.seqHead: "markov"|"rnn"`; Elman recurrence
-  `s_k = tanh(s_{k-1}·wH + E[x_{k-1}] + bH)`, `B_k = s_k·wO`, `wO` zero-init
-  (starts as pure DFlash, like markov.w2); shares `markov.w1` as the token
-  embedding (also feeds the confidence head). Same loss, autograd-gated;
-  init-equivalence gate: rnn ≡ markov token-for-token at init.
-  Train with `scripts/dspark-train-dflash.ts --seq-head rnn`.
-- **Draft loop TIGHTENED** (handoff item 3): greedy tokens chain ON-DEVICE
-  (one concat + one host read after the loop, was γ syncs); confidence reads
-  deferred+batched when pruning is inactive (per-position only when Alg 1
-  needs the answer mid-loop — inherent); `collectLogits: false` on the serve
-  path skips the per-position draft-logits materialization entirely
-  (`draftLogits` now optional). Gate: bit-identical to the pre-tightening
-  loop (pinned reference tokens, tests/dspark-infer-loop.test.ts).
-- **Rename + central loader.** Canonical variant is now `"dspark"`
-  (`save()` stamps it; `load()` accepts legacy `"dflash"`);
-  `src/spec/dspark/loader.ts` `loadDsparkDrafter()` dispatches by variant and
-  refuses v1 single-vector checkpoints with a pointer at the v2 trainer. The
-  serve path and the bench (`bench-feature-matrix.ts` via the server's
-  exported `detectDraftKind`) both go through it.
-
-## DeepSpec ground truth (2026-07-06 audit — paper arXiv:2607.05147 + source)
-
-DeepSeek open-sourced the reference implementation (**DeepSpec**, MIT,
-github.com/deepseek-ai/DeepSpec) + trained drafters incl.
-**`deepseek-ai/dspark_gemma4_12b_block7`** (for google/gemma-4-12B-it — our
-primary target; 6.86 GB bf16 single safetensors, γ=7). Two audit agents
-verified our build against the paper AND the source. Verdicts:
-
-- **MATCH the paper exactly:** Eq 2-3 (context + KV injection), the
-  anchor++masks block, Eq 5 Markov (r=256), Eq 7 confidence (we add a
-  zero-init bias — superset), **Eq 8-12 loss to the letter** (0.1/0.9/1.0,
-  w_k=exp(−(k−1)/γ), analytic c*). Our defaults = their production config
-  (γ=5, 5 layers, Markov head).
-- **Eq 6 RNN differs:** paper is a gated cell over
-  `z_k=[s_{k−1}; W₁[x_{k−1}]; h_k]`, fused (2r+d)→3r projection,
-  `B_k=W₂ᵀtanh(W_o z_k)` (conditioned on the backbone hidden). Ours is
-  ungated Elman without h_k — kept as variant `"rnn"` (paper calls the gains
-  "marginal"; production uses Markov); DeepSpec also ships an RNNHead of the
-  gated form if we ever want the faithful one.
-- **§3.2.1 STS + Alg 1: the paper's machinery is NOT in the released code.**
-  STS = Sequential Temperature Scaling (per-position temperatures by ECE grid
-  search on cumulative survival products); Alg 1 = expected-throughput
-  maximization over survival products with a profiled SPS(B) cost table —
-  both live in their UNRELEASED production serving layer. The released
-  reference ships **per-position threshold truncation on sigmoid confidence
-  (default 0 = off), ℓ=0 allowed** — i.e. the released reference is shaped
-  like OUR scheduler. Our threshold fit (`calibration.ts`) is therefore a
-  reasonable calibrator for the reference-shaped scheduler, NOT the paper's
-  STS; renamed framing in code comments. Paper-faithful STS/Alg-1 = a future
-  Lab item if the survival-product scheduler ever earns a default.
-- **The reference drafter is a DIFFERENT module than ours:** a full
-  Gemma4-shaped 5-layer transformer — hidden 3840, 16 Q heads × head_dim 512,
-  **1 KV head with K≡V (`attention_k_eq_v`, no v_proj tensor; scale-less
-  v_norm)**, attention scale 1.0 (QK-norm), RoPE partial_rotary 0.25 θ=1e6,
-  per-layer learned `layer_scalar`, scaled embed (×√3840, not stored), UNTIED
-  lm_head, final softcap 30, and an **incremental context-KV cache** (stores
-  projected target-context K/V; crops the block's noise K/V each round).
-  Sequential Markov sampling + confidence head (input [h; w1[prev]] = 4096)
-  as in the paper. Checkpoint: 74 tensors, bf16, **no key prefix**;
-  config.json `architectures: ["Gemma4DSparkModel"]` (loader detection key).
-  **12B tap layers: `target_layer_ids [5,17,29,41,46]`** (their layer_id+1
-  output convention == our tapLayers index; no final-norm sentinel).
-- **THE ORACLE:** their verify is leaky rejection sampling that at
-  temperature 0 **degenerates to exact argmax token-match — RNG-free,
-  token-for-token deterministic**. Their eval emits per-round
-  proposal/accepted/next-token traces. Protocol: run their eval at temp 0
-  (+ a threshold arm) on fixed prompts → JSONL round fixtures → our port
-  must match round-for-round. Same discipline as the D2 diffusion oracle,
-  minus RNG parity entirely.
-
-**Consequence:** DSpark is no longer oracle-less L3. The port of their module
-(the `deepspec` variant) + their checkpoint gives a bit-exact-gateable path to
-a REAL 12B speedup with zero training. Our own trainable module remains the
-research path for custom targets. Training recipe reference: open-perfectblend
-regenerated by the target, 10 epochs, batch 512, lr 6e-4, position decay
-exp(−pos/4), ~38 TB cached hiddens — i.e. our "thousands of generations" data
-estimate was directionally right and their scale is far beyond a laptop run.
-
-## 27B program (2026-08-19): the TQ quant unlocks the real target
-
-The turboquant-weights campaign shipped rot-sens48 Qwen3.8-27B at 17 GB
-(docs/design/turboquant-weights.md) — and it changes both DSpark blockers
-this doc recorded:
-
-- **Target speed:** measured ~11.8 tok/s decode. This is the slow-target
-  regime where the doc's own analysis says τ≈3 nets 2–3× (≈25–35 tok/s) —
-  "the ideal target and the real motivation," now actually runnable.
-- **Training feasibility:** "27B memory-infeasible to train on 24 GB" (open
-  items below) assumed the bf16 trunk (54 GB). With the quantized trunk
-  FROZEN (inference-only: hidden taps + teacher logits) and only the small
-  drafter (~5 layers) holding grads/optimizer state in bf16, 27B-target
-  drafter training fits a 32 GB machine. The premise is dead; the line in
-  the gated item is superseded by this section.
-- **Rollback risk retired:** the stock Qwen-MTP spec path already serves
-  this model losslessly (68% per-token acceptance), so verify/rollback
-  across the 48 DeltaNet recurrent-state layers is proven engine behavior,
-  not new work.
-
-Two tracks, cheap-first:
-
-- **Track A — confidence-scheduled verification on the stock MTP head (no
-  drafter training).** Dynamic per-step block length instead of fixed
-  `--num-draft-tokens`: v0 confidence = the draft's own token probability
-  (no trained head), mapped through STS calibration
-  (`scripts/dspark-calibrate.ts` pattern) to expected acceptance; the
-  scheduler stops drafting when marginal expected-accept drops below the
-  verify amortization point. Calibration needs thousands of
-  (confidence, accepted?) pairs from served traffic — hours, not weeks.
-  Gate: paired A/B decode tok/s vs fixed-γ MTP on the same prompts;
-  lossless by construction (verify unchanged).
-- **Track B — true DSpark drafter for the 27B (multi-token single-pass).**
-  The faithful v2 (KV injection, Markov head) retargeted: `hiddenTap` added
-  to `qwen3_5.ts` forwardLayers (gemma4 has it; parity-safe no-op when
-  off), tap-layer selection for the 64-layer trunk (bias toward the 16
-  full-attention layers + post-finalNorm), data generated by TEACHER-FORCED
-  passes over corpus text with the frozen quantized trunk (every position
-  of a forward yields a sample — data scale is forward-passes over tokens,
-  not generate calls). Open idea: distill from / initialize against the
-  shipped MTP head to shortcut the acceptance floor. Gate: generalizing
-  per-position acceptance must beat the MTP head's 0.68 or Track B loses to
-  Track A + MTP and gets dropped.
-
-Sequencing: after the benchmark campaign closes (GPU is owned by the
-certification suite). Track A first — it's measurable in a day and its
-calibration data collection is also Track B's baseline instrumentation.
-Adapters/DSpark drafts trained on the TQ artifact are married to its
-rotation basis (fine for our bundle; note on the card if published).
-
-## Open items / next
-
-- [x] Serve integration (Phase 1) · [x] Alg-1-shaped scheduler (matches the
-  RELEASED reference; paper Alg-1 unreleased) · [x] threshold calibration ·
-  [x] RNN head (Elman variant; paper's gated cell differs — documented above)
-  · [x] loop tightening · [x] `dspark` rename + loader —
-  **everything buildable without a GPU is DONE.**
-- [ ] **DeepSpec-variant port** (`Gemma4DSparkModel`) + loader detection +
-  oracle scripts — the no-training path to the 12B speedup (in progress).
-- [ ] **Josh-gated GPU (the payoff):** real data scale (thousands of
-  on-distribution generations) + **retarget to 12B** (`--model` +
-  `--tap-layers 5,17,29,41,46` (DeepSeek's trained taps); ~~27B memory-infeasible to train on 24 GB~~ (superseded — the 17 GB TQ trunk makes it feasible; see "27B program" above), kept
-  dim-generic) + train (`--seq-head` A/B optional) + **calibrate**
-  (`scripts/dspark-calibrate.ts`) + live-τ measure. Recipe:
-  [docs/archive/investigations/dspark-handoff.md](../archive/investigations/dspark-handoff.md).
-- [ ] Verify the Eq 6 / §3.2.1 shapes against the actual paper when the PDF
-  is available (both flagged in code).
-
-Full session handoff: `docs/archive/investigations/dspark-handoff.md`.

package/docs/design/grammar-spec-batching-integration.md

@@ -1,301 +0,0 @@
-# Grammar × spec decode × batching — integration plan + feature-matrix benchmark
-
-Status: **ALL PHASES EXECUTED** (A/B/C/E 2026-07-03; D completed
-2026-07-04: KV-budget admission + vectorized homogeneous sampling +
-extend-join, each oracle/A-B gated — see the execution log). Remaining
-follow-ups live in "Later / parked" with their triggers.
-Execution log at the bottom of this doc. Successor plan tying together
-three workstreams so they compose deliberately instead of colliding:
-
-- **Structured output** — SHIPPED on main (serial + batched lanes;
-  docs/design/structured-output.md). Open debt: B2 model-gated batch
-  tests + bench, F4 compiler cache, F5 regex, F6 escaping, F7 structural
-  tags, U1/U2 engine upgrades.
-- **Batching perf path** — P5 done; P0–P4 open
-  (docs/design/batching-perf-path.md).
-- **Speculative decoding** — research proven (assistant drafter bit-exact
-  vs optiq, 12B γ=1 ≈ 1.09× win, docs/design/spec-decode-larger-targets.md;
-  DSpark architecture verified, docs/archive/investigations/dspark-handoff.md) but
-  **not served**: no `--draft-model`. Build plan already written:
-  docs/design/mlx-lm-tool-parity-plan.md §7 (the `DraftSource` interface,
-  two-model L1 source, serve loop, admission, prompt-cache composition).
-
-This doc does NOT restate those plans — it sequences them, defines the
-composition rules where they meet, and specifies the benchmark that
-measures every combination. The per-feature designs stay canonical in
-their own docs.
-
-## The composition matrix (v1 target state)
-
-`GenerationGateway.willBatch` is the single routing point; every rule
-below is one predicate there. "Serial" routing is a v1 rule, not a
-limitation to hide — upstream mlx-lm routes spec serial too
-(`is_batchable = draft_model is None`, server.py:371).
-
-| Request shape | Lane | Mechanism | Status |
-| --- | --- | --- | --- |
-| plain | batch (or serial) | shipped engine | DONE |
-| grammar only | batch (or serial) | B1 per-row matchers / serial mask loop | DONE (B2 gates open) |
-| spec only | **serial** | `hasDraft` routes serial (upstream parity) | Phase B |
-| grammar + spec | **serial** | constrained verify walk (novel — §Phase C) | Phase C |
-| batch + spec | serial per request | `hasDraft` wins over batch | Phase B (routing) |
-| batch + grammar + spec | serial per request | `hasDraft` wins | falls out of B+C |
-
-Non-goals (explicitly out of scope, revisit only on measured demand):
-per-row spec inside the batch scheduler (variable accept-lengths break
-uniform-B; a research project, not a feature gap), DSpark as a served
-source before its 27B retrain/data-scale lands, grammar on the
-`/v1/responses` shim beyond what chat inheritance already gives.
-
-## Phases
-
-Ordering rationale: A is small and closes the trust gap on what already
-shipped. B is the flagship parity feature and lays the `DraftSource`
-seam C needs. C is the novel composition. D runs parallel to B/C
-(scheduler files vs serial-loop files — near-zero overlap). E needs all
-of them and produces the numbers.
-
-### Phase A — structured-output debt (B2 + F4)  [S, 2–3 days]
-
-1. **B2 model-gated scheduler tests** (spec'd in structured-output.md):
-   mixed batch mask isolation; **all-grammar B=4 with four different
-   schemas** (THE bug-class gate — row↔matcher misalignment after
-   eviction/join); early termination + churn + joiner; max_tokens
-   mid-JSON truncation under batch; prefill-terminated row never merges.
-2. **F4 compiler cache**: one `GrammarCompiler` per `TokenizerInfo`
-   (both vocab-structural; `ownsCompiler=false` path already exists).
-   Measure compile time cold vs cached on a real JSON schema — this is
-   the agentic-replay win and a benchmark row in Phase E.
-3. Opportunistic: F6 choice escaping (tiny), F3 explicit padded-vocab
-   comment. F5/F7 stay parked for the structural-tag pass (see Later).
-
-Exit: gated suites green on cpm5 + e4b; compile-cache A/B number
-recorded.
-
-### Phase B — `serve --draft-model` (two-model, serial lane)  [M, ~1 wk]
-
-Execute mlx-lm-tool-parity-plan.md §7 as written (it is already the
-hand-to-an-implementer plan). Integration deltas this plan adds:
-
-- **`RequestShape.hasDraft`** beside `hasGrammar`; `willBatch` returns
-  false for it. Lane-debug line includes it. (The B0-grammar routing
-  pattern, reused verbatim.)
-- **The serve verify loop lands grammar-aware from day one**: the loop
-  emits bursts of ≤ n+1 tokens; the stop-sequence/detokenizer audit in
-  §7 ("risky parts") must treat the grammar mask hook as a first-class
-  participant even while it's unused in this phase — i.e. the per-round
-  walk is structured as `for pos in verifyWindow: sample(logits[pos],
-  maskHook?)` so Phase C is a hook implementation, not a loop rewrite.
-- **Kill switch** `MLX_BUN_SPEC=0` (degrade: ignore the flag, log once),
-  house style.
-
-Oracle: L1 spec-vs-spec token equality with `mlx_lm.server
---draft-model` from the oracle venv (§7.8). Acceptance telemetry
-(`drafted/accepted/targetCalls`) in stats — Phase E consumes it.
-
-Exit: greedy token-for-token vs mlx-lm spec on a fixed pair; rotating
-wrap falls back non-spec (no 500); `--batch 4` + `--draft-model` routes
-spec requests serial and batches the rest; admission accounts for draft
-weights + KV.
-
-### Phase C — grammar × spec: the constrained verify walk  [M, ~1 wk]
-
-Novel territory — **no oracle serves grammar+spec** (mlx-lm has no
-grammar; oMLX has no spec). No-oracle (Lab-class) gating per the
-parity-tier contract: equivalence + validity gates, not bit-parity to an
-ancestor.
-
-Design (v1 = verify-side masking, drafter free-running):
-
-1. **Drafter is unconstrained.** Drafts are proposals; correctness never
-   depends on them (a grammar-invalid draft is simply rejected at
-   verify). No matcher state touches the draft path.
-2. **The verify walk applies the mask sequentially.** One batched target
-   forward over `[pending, ...drafts]` (n+1 positions) as in Phase B;
-   then the host-side accept walk goes position by position:
-   `mask_i = matcher.bitmask` → apply to `logits[i]` → sample → if
-   sampled == draft_i: `matcher.accept(draft_i)`, continue; else emit
-   the masked correction token, `matcher.accept(correction)`, stop.
-   The matcher only ever advances on **emitted** tokens — no rollback
-   needed in v1. Sequential bitmask fills cost 0.004–0.19 ms each
-   (measured, structured-output.md); at γ≤3 that's ≤0.8 ms/round against
-   a 12B target step of ~40 ms — noise.
-3. **Grammar termination mid-burst** truncates the accepted prefix at
-   the terminating token (same all-`-inf` guarantee as serial/batched:
-   nothing is sampled past termination), finish_reason "stop".
-4. **The measured fork** ([[flag-and-try-both-on-forks]]): on
-   grammar-tight spans (e.g. `guided_choice`, closing braces) a
-   free-running drafter's acceptance may crater — every draft off-grammar
-   is a wasted round. If Phase E shows that, the alternative is built
-   behind `MLX_BUN_SPEC_GRAMMAR_DRAFT=1`: mask the drafter's logits too
-   (matcher advances per draft, `rollBack(n − kAccept)` on rejection —
-   the WASM API exposes rollBack; this is its use case). Build the flag
-   only if the default's acceptance number demands it; don't pre-build.
-
-Gates (no external oracle):
-- **Equivalence**: greedy grammar+spec long-prefix-equal to greedy
-  grammar-only serial on fixed prompts (exact-match acceptance means
-  spec must reproduce the non-spec masked-greedy stream, modulo the
-  documented batched-verify-head knife-edges — same caveat class as L1
-  spec-vs-spec).
-- **Validity**: every grammar+spec output parses / conforms to its
-  schema (100%, no exceptions), matcher never rejects an emitted token.
-- **Composition regression fence**: grammar-only and spec-only suites
-  byte-unchanged.
-
-### Phase D — batching P0 (+ admission slice of P3)  [M, ~1 wk, parallel with B/C]
-
-From batching-perf-path.md, the subset that the benchmark needs and that
-doesn't collide with B/C's serial-loop work:
-
-1. **P0**: `scripts/bench-serving-load.ts` (N concurrent vs live server,
-   aggregate tok/s + TTFT p50/p95 at the SSE first-token event) —
-   Phase E builds on this harness; `extend` join (kill the O(B·S)
-   re-merge); vectorized homogeneous-sampler fast path.
-   - `extend` + grammar rows: the B1 read-before-build step must hold
-     across an extend-join — add one B2-style churn test with a joiner
-     landing while a grammar row is live.
-2. **P3 admission slice only**: `projectKvBytes` + `--kv-budget`
-   queueing (the OOM backstop the combined benchmark needs before it
-   pushes B=4 × long prompts on 24 GB). Prompt-cache-under-batching,
-   adapters, and the default flip stay in batching-perf-path P3.
-
-P1 (quantized KV at B>1) is NOT in this plan's critical path — it's the
-next wave after the matrix benchmark exists, and the benchmark is exactly
-what will size its payoff. (P2 — perf kernel at B>1 — is OBSOLETE: the
-kernel was deleted 2026-07-05, docs/design/unified-engine-frontier-plan.md §6.)
-
-### Phase E — the feature-matrix benchmark  [S–M, 3–4 days]
-
-One harness answering "what does each feature — and each legal
-combination — cost or buy, per model?" Extends P0's
-`bench-serving-load.ts`; preflight-gated like benchmark.sh (dirty-machine
-numbers are garbage; quotable rows need the clean-machine protocol).
-
-**Cells** (skip-if-unavailable, like the model-gated suites):
-
-- Lanes/features: `serial`, `batch4`, `serial+grammar`, `batch4+grammar`,
-  `serial+spec`, `serial+grammar+spec` — the six legal shapes from the
-  matrix. (batch+spec cells route serial by design; the harness records
-  them as routing checks, not perf rows.)
-- Models: cpm5 + e4b (fast movers, both lanes), 12B (+ its assistant
-  drafter — the measured spec win), Qwen3.5 (batch + thinking-model
-  grammar canary; grammar cells expected to expose F7 until structural
-  tags land — record, don't hide).
-- Workloads: short-chat (128 tok), JSON-extraction (a real schema,
-  grammar cells), long-form (512 tok). 4 concurrent for batch cells,
-  1 for serial cells; N=3 median.
-
-**Metrics per cell**: aggregate + per-stream decode tok/s · TTFT
-p50/p95 (SSE first token, wall-clock) · spec acceptance
-(drafted/accepted/targetCalls) · grammar overhead vs the same cell
-without grammar (%) · schema-conformance rate (**must be 100% in every
-grammar cell — a conformance failure fails the run, it's a correctness
-gate wearing a benchmark costume**) · peak RSS.
-
-**Denominator discipline** (spec-decode-larger-targets.md): every
-feature cell is compared to the SAME model+lane baseline cell from the
-SAME run. No cross-model or cross-run speedup claims.
-
-**Output**: eval-DB rows tagged `feature-matrix[<host>]` + a dated
-markdown artifact (gitignored, like benchmark.sh's); promoted rows land
-in benchmarks/RESULTS.md under a new "composition" table. Josh-gated
-clean-machine run for anything quotable.
-
-Exit: the matrix runs end-to-end on this machine; a first
-(loaded-machine, unquotable) sweep sanity-checks every cell; the
-clean-machine run is queued in STATUS Josh-gated.
-
-## Later / parked (pointers, so they aren't lost)
-
-- **F7 + F5 structural-tag pass** (thinking-model grammar + real regex)
-  — try the installed WASM package's `fromStructuralTag`/`RegexFormat`
-  first; U1 rebuild only if its Dec-2025 core predates the semantics
-  (structured-output.md addendum). Qwen3.5's grammar cells in Phase E
-  are the forcing function.
-- **U2 native XGrammar-2 via TVM-FFI** — trigger conditions unchanged
-  (B≫8, spec+grammar draft-tree masking via `traverse_draft_tree`, or
-  compile time visible despite F4). Phase C's v1 walk deliberately
-  avoids needing it.
-- **Batching P1** (quant-KV at B>1) — next wave, sized by the matrix.
-  (P2, perf kernel at B>1, is obsolete — kernel deleted 2026-07-05.)
-- **P4 device-side step chaining** — orthogonal serial-decode work; the
-  matrix's serial baselines will show its headroom.
-- **Assistant/DSpark sources behind `--draft-model`** —
-  mlx-lm-tool-parity-plan §7.9; DSpark stays research until 27B
-  retrain + data scale (dspark-handoff.md).
-- **Spec inside the batch lane** — research question, not scheduled.
-
-## Execution log (2026-07-03)
-
-- **Phase A** ✅ — B2 gates in `tests/batch-grammar.test.ts` (all five
-  scenarios + a sixth found later, below); F4 compiler cache went through a
-  bring-up fix (single-flight rebuild — concurrent compiles after a
-  tokenizer switch double-disposed the previous compiler); F6, F3 landed.
-- **Phase B** ✅ — `serve --draft-model`/`--num-draft-tokens`;
-  `src/spec/{source,two-model,serve-loop}.ts`. **L1 gate: 48/48
-  token-for-token vs mlx-lm spec** (Llama 3B+1B greedy, 65% acceptance,
-  `scripts/oracle-spec-two-model.py`). Deviations as designed (ring-wrap
-  degrades pre-pollution via an offset+n+1 gate — upstream's post-hoc check
-  would already have polluted the ring; prompt cache bypassed v1).
-- **Phase C** ✅ — the constrained verify walk in `samplePos`; gates green
-  (100% validity; 12/12 token-identical to grammar-only serial; choice
-  terminates mid-spec). Drafter-masking flag NOT built (acceptance 6/15 on
-  a tight schema — usable; revisit only if Phase E's clean run says so).
-- **Phase E** ✅ (harness) — `scripts/bench-feature-matrix.ts` (the planned
-  bench-serving-load extension became its own script: that file already
-  exists as a client-only stack-vs-stack tool and stays one). Six cells
-  green end-to-end on Llama 3B+1B; `usage.speculation` telemetry on all
-  four API usage blocks. Clean-machine run for RESULTS.md is Josh-gated.
-- **The conformance gate paid for the whole phase in one smoke run** —
-  three real bugs: (1) snake_case wire fields never reached the grammar
-  resolver (structured output was DEAD over HTTP since it landed);
-  (2) `#flushPipeline` didn't advance matchers → stale masks on mid-decode
-  joins (regression test added); (3) **UniversalDenseModel batched RoPE
-  uses the scalar cache.offset** → uneven-row batches decode joiners at
-  wrong positions — latent for ALL Tier-0 archs since v0.0.9. **Fixed
-  same day**: `UniversalRope.applyDynamic` (+ `ops.ropeScaledDynamic`)
-  routes the wrapper's per-row `ropeOffsetArr`; gated **token-exact vs
-  mlx-lm B=2 on Llama-3.2-3B** — static uneven rows AND the dynamic
-  join/leave protocol (tests/batched-decode-parity.test.ts "Llama 3B
-  Tier-0"). Plain full-attention universal archs now batch; maskArray
-  (gemma2-family, pad-blind causal mask in forwardLayers) and
-  sliding-window universal archs remain serial (unvalidated cells).
-  Lesson, per [[per-model-quant-specialization]]: the gateway's
-  capability gate admitted a model family no batched-parity suite ever
-  validated.
-
-### Phase D completion (2026-07-04)
-
-- **KV-budget admission** — `--kv-budget <GB>`: aggregate projected-KV gate
-  in the scheduler (queue-don't-OOM, oversized-alone rejects,
-  `/stats.batch.{pending_rows,kv_bytes,kv_budget_bytes}`;
-  tests/batch-kv-budget.test.ts).
-- **Vectorized homogeneous sampling** — all-plain-greedy batches sample via
-  ONE log-softmax+argmax over [B,V]; gateway flags `plainGreedy`; BIT-equal
-  A/B vs the per-row path (`MLX_BUN_BATCH_VEC_SAMPLE=0` kill switch;
-  tests/batch-vec-sample.test.ts).
-- **Extend-join** — `extendKVRows` (mlx-lm `BatchKVCache.extend` semantics:
-  one pad + one B-axis concat, running rows untouched, pads grow) replaces
-  the O(B·S) re-merge for full-attention layers in `#mergeJoiner`
-  (`MLX_BUN_BATCH_EXTEND=0` = re-merge). **Own oracle** (extend's pad
-  layout ≠ re-merge's): scripts/gen-batched-extend-golden.py →
-  token-for-token on CPM AND Llama-3B (tests/batched-decode-parity.test.ts
-  "EXTEND-JOIN ORACLE"); full scheduler suite incl. the grammar-churn join
-  test green through the extend path AND the kill-switch path.
-  Rotating-layer extend (Gemma joins re-merge their sliding layers, extend
-  their full layers) is a follow-up.
-
-## Risks
-
-- **Grammar+spec has no oracle** — mitigated by the three-gate stack
-  (equivalence to grammar-only greedy, 100% validity, regression fence)
-  and by keeping the drafter out of the grammar state in v1.
-- **Multi-token bursts through stop-sequence/detokenizer state** — the
-  known §7 risky part, now load-bearing for two phases; land the burst
-  audit with Phase B tests, not after.
-- **Memory at 12B+drafter / B=4 on 24 GB** — Phase D's admission slice
-  is the backstop; benchmark cells must go through it, not around it.
-- **Scope creep via the matrix** — the benchmark measures what exists;
-  it must not become a reason to build P1/P2/drafter-masking
-  speculatively. Each parked item has its trigger written above.