mlx-bun 0.0.10 → 0.0.11

package/docs/design/dspark-speculative-decoding.md

@@ -78,7 +78,55 @@ Also confirmed: width is NOT the ceiling (`dDraft=1024` and `2560` both plateau
 ~0.17 pre-scale); the parity-safe `hiddenTap` doesn't change target numerics;
 long-context spec rollback works via the `trim(n, bypass)` physical-slice.
 
+## Serve integration (Phase 1, 2026-07-06)
+
+DSpark is now serve-loadable behind the same `--draft-model` seam as every
+other drafter — one verify/accept executor, no lane routing. The seam
+(`src/spec/source.ts`) was extended from its two-model-only shape to carry the
+target state KV-borrowing drafters need (it previously could NOT host these,
+despite the header claiming so):
+
+- `DraftProvider.open({sampler, target:{model,caches}})` — the source gets the
+  target model + its live caches.
+- `DraftSource.tapLayers?` — when set (DSpark), the serve loop taps the
+  target's prefill + every verify forward and hands the captured `[1,L,m*H]`
+  context to `prefill(ids, ctxML)` / `commit(n, kAccept, vCtxML)` (ownership
+  transfers). `draft(feed, n, stepBase, anchorHidden?)` — the assistant borrows
+  the target anchor hidden; DSpark drafts from its growing `H_ctx`.
+
+Provider selection is by artifact kind (`src/server.ts` `detectDraftKind`):
+`dspark.json` → **DflashProvider** (`src/spec/dflash-source.ts`), a `*_assistant`
+config → **AssistantProvider** (`src/spec/assistant-source.ts`, the optiq
+KV-borrowing Gemma drafter — ships a real ~1.09× γ=1 decode win with no
+training), else **TwoModelProvider**. `--draft-kind` overrides. The server pins
+`--num-draft-tokens` to the DSpark checkpoint's trained block width
+(`cfg.gamma`) so the serve loop never asks for more positions than were trained.
+
+**Unified-engine alignment (per-slot readiness).** This is the
+`DraftSource` seam the unified-engine plan names as the substrate for its
+frontier "per-slot drafting behind the `DraftSource` seam" row
+([docs/design/unified-engine-frontier-plan.md](unified-engine-frontier-plan.md)).
+Spec still forces the serial lane today (upstream `is_batchable = draft is
+None`); Phase 1 did NOT change that — it made the seam *carry per-source target
+state* (each source binds its own `TargetView{model,caches}`, anchor hidden,
+and tapped H_ctx), which is exactly what a batched executor would need to open
+one `DraftSource` per row. So the seam is forward-compatible with per-slot
+drafting; the hard part that remains is the executor, not the seam — variable
+accept-length per slot breaks a uniform-B step (see
+[spec-decode-larger-targets.md](spec-decode-larger-targets.md) caveats), so
+per-slot spec is a real batched-scheduler project, not free composition.
+
+**v1 acceptance caveat:** the serve loop verifies with TOKEN-MATCH acceptance
+(mlx-lm/optiq style, lossless at any temperature), NOT the paper's
+distribution-level rejection-sampling verify. Greedy is identical; temp>0 is
+still lossless but lower-acceptance. The richer rejection-sampling verify stays
+in the standalone `generate-dflash.ts` (the measure script). See
+[[dspark-seam-kv-borrowing]].
+
 ## Files
+- `src/spec/source.ts` — the extended DraftSource/DraftProvider/TargetView seam.
+- `src/spec/assistant-source.ts` / `dflash-source.ts` — the two KV-borrowing providers.
+- `src/spec/serve-loop.ts` — the shared verify/accept executor (now tap-aware).
 - `src/model/gemma4.ts` — `hiddenTap`/`captureLayer` (parity-safe multi-layer tap).
 - `src/model/gemma4-base.ts` — `trim(n, bypass)` on rotating caches (spec rollback past the window).
 - `src/spec/dspark/module-dflash.ts` — faithful DFlash+Markov+confidence module.
@@ -96,11 +144,125 @@ bun scripts/dspark-train-dflash.ts --data <data> --out <ckpt> --iters 8000 --bat
 bun scripts/dspark-measure-dflash.ts --drafter <ckpt> --data <prompts.jsonl>   # τ + tok/s vs vanilla
 ```
 
+## Paper components — Phase 2 (2026-07-06): the paper is CODE-COMPLETE
+
+All remaining components landed the same day (multi-agent build, adversarially
+reviewed; smoke 21/21, dspark test files 17/17, real-weights serve gate 3/3):
+
+- **Alg 1, single-user form — confidence-scheduled draft-length pruning.**
+  `forwardInfer` drops position k (and truncates the block) when
+  `c_k < thresholds[k] ?? minConf`; position 0 always survives. Activation is
+  checkpoint-driven: STS thresholds in `dspark.json` (`cfg.sts`) or the
+  `MLX_BUN_DSPARK_MINCONF` env override — **uncalibrated checkpoints draft
+  fixed-γ exactly as before** (zero behavior change). Losslessness invariant:
+  pruning changes how many positions the target verifies, never what's emitted
+  (gated: truncation-never-redraw, prefix-identical to the unpruned block).
+- **Variable-length draft contract.** `DraftSource.draft()` returns 1..n —
+  the RETURN LENGTH is authoritative; the serve loop and `dflashGenerate`
+  verify/trim/re-feed over `d = drafts.length`. Two-model/assistant return
+  exactly n (unchanged).
+- **STS calibration (§3.2.1).** `src/spec/dspark/calibration.ts`
+  (`fitStsThresholds`: per position, smallest τ whose Laplace-smoothed
+  `P(accepted | conf ≥ τ)` ≥ target; pos 0 → 0, under-sampled → 0 (don't
+  prune on thin evidence), unreachable target → 1.0) +
+  `scripts/dspark-calibrate.ts` (GPU: greedy unpruned rounds via
+  `dflashGenerate`'s `onRound` hook → fit → patch `config.sts` into
+  dspark.json in place). ⚠ estimator shape is our reading — paper PDF absent.
+- **RNN sequential head (Eq 6 — ⚠ design-doc-faithful shape, paper PDF
+  absent, flagged in code).** `cfg.seqHead: "markov"|"rnn"`; Elman recurrence
+  `s_k = tanh(s_{k-1}·wH + E[x_{k-1}] + bH)`, `B_k = s_k·wO`, `wO` zero-init
+  (starts as pure DFlash, like markov.w2); shares `markov.w1` as the token
+  embedding (also feeds the confidence head). Same loss, autograd-gated;
+  init-equivalence gate: rnn ≡ markov token-for-token at init.
+  Train with `scripts/dspark-train-dflash.ts --seq-head rnn`.
+- **Draft loop TIGHTENED** (handoff item 3): greedy tokens chain ON-DEVICE
+  (one concat + one host read after the loop, was γ syncs); confidence reads
+  deferred+batched when pruning is inactive (per-position only when Alg 1
+  needs the answer mid-loop — inherent); `collectLogits: false` on the serve
+  path skips the per-position draft-logits materialization entirely
+  (`draftLogits` now optional). Gate: bit-identical to the pre-tightening
+  loop (pinned reference tokens, tests/dspark-infer-loop.test.ts).
+- **Rename + central loader.** Canonical variant is now `"dspark"`
+  (`save()` stamps it; `load()` accepts legacy `"dflash"`);
+  `src/spec/dspark/loader.ts` `loadDsparkDrafter()` dispatches by variant and
+  refuses v1 single-vector checkpoints with a pointer at the v2 trainer. The
+  serve path and the bench (`bench-feature-matrix.ts` via the server's
+  exported `detectDraftKind`) both go through it.
+
+## DeepSpec ground truth (2026-07-06 audit — paper arXiv:2607.05147 + source)
+
+DeepSeek open-sourced the reference implementation (**DeepSpec**, MIT,
+github.com/deepseek-ai/DeepSpec) + trained drafters incl.
+**`deepseek-ai/dspark_gemma4_12b_block7`** (for google/gemma-4-12B-it — our
+primary target; 6.86 GB bf16 single safetensors, γ=7). Two audit agents
+verified our build against the paper AND the source. Verdicts:
+
+- **MATCH the paper exactly:** Eq 2-3 (context + KV injection), the
+  anchor++masks block, Eq 5 Markov (r=256), Eq 7 confidence (we add a
+  zero-init bias — superset), **Eq 8-12 loss to the letter** (0.1/0.9/1.0,
+  w_k=exp(−(k−1)/γ), analytic c*). Our defaults = their production config
+  (γ=5, 5 layers, Markov head).
+- **Eq 6 RNN differs:** paper is a gated cell over
+  `z_k=[s_{k−1}; W₁[x_{k−1}]; h_k]`, fused (2r+d)→3r projection,
+  `B_k=W₂ᵀtanh(W_o z_k)` (conditioned on the backbone hidden). Ours is
+  ungated Elman without h_k — kept as variant `"rnn"` (paper calls the gains
+  "marginal"; production uses Markov); DeepSpec also ships an RNNHead of the
+  gated form if we ever want the faithful one.
+- **§3.2.1 STS + Alg 1: the paper's machinery is NOT in the released code.**
+  STS = Sequential Temperature Scaling (per-position temperatures by ECE grid
+  search on cumulative survival products); Alg 1 = expected-throughput
+  maximization over survival products with a profiled SPS(B) cost table —
+  both live in their UNRELEASED production serving layer. The released
+  reference ships **per-position threshold truncation on sigmoid confidence
+  (default 0 = off), ℓ=0 allowed** — i.e. the released reference is shaped
+  like OUR scheduler. Our threshold fit (`calibration.ts`) is therefore a
+  reasonable calibrator for the reference-shaped scheduler, NOT the paper's
+  STS; renamed framing in code comments. Paper-faithful STS/Alg-1 = a future
+  Lab item if the survival-product scheduler ever earns a default.
+- **The reference drafter is a DIFFERENT module than ours:** a full
+  Gemma4-shaped 5-layer transformer — hidden 3840, 16 Q heads × head_dim 512,
+  **1 KV head with K≡V (`attention_k_eq_v`, no v_proj tensor; scale-less
+  v_norm)**, attention scale 1.0 (QK-norm), RoPE partial_rotary 0.25 θ=1e6,
+  per-layer learned `layer_scalar`, scaled embed (×√3840, not stored), UNTIED
+  lm_head, final softcap 30, and an **incremental context-KV cache** (stores
+  projected target-context K/V; crops the block's noise K/V each round).
+  Sequential Markov sampling + confidence head (input [h; w1[prev]] = 4096)
+  as in the paper. Checkpoint: 74 tensors, bf16, **no key prefix**;
+  config.json `architectures: ["Gemma4DSparkModel"]` (loader detection key).
+  **12B tap layers: `target_layer_ids [5,17,29,41,46]`** (their layer_id+1
+  output convention == our tapLayers index; no final-norm sentinel).
+- **THE ORACLE:** their verify is leaky rejection sampling that at
+  temperature 0 **degenerates to exact argmax token-match — RNG-free,
+  token-for-token deterministic**. Their eval emits per-round
+  proposal/accepted/next-token traces. Protocol: run their eval at temp 0
+  (+ a threshold arm) on fixed prompts → JSONL round fixtures → our port
+  must match round-for-round. Same discipline as the D2 diffusion oracle,
+  minus RNG parity entirely.
+
+**Consequence:** DSpark is no longer oracle-less L3. The port of their module
+(the `deepspec` variant) + their checkpoint gives a bit-exact-gateable path to
+a REAL 12B speedup with zero training. Our own trainable module remains the
+research path for custom targets. Training recipe reference: open-perfectblend
+regenerated by the target, 10 epochs, batch 512, lr 6e-4, position decay
+exp(−pos/4), ~38 TB cached hiddens — i.e. our "thousands of generations" data
+estimate was directionally right and their scale is far beyond a laptop run.
+
 ## Open items / next
-- **Real data scale** (thousands of on-distribution generations, not 160) to lift generalizing τ toward the overfit ~0.75.
-- **Retarget to a slow model (27B/12b)** where the τ≈3 architecture actually nets a speedup — the whole point. Needs the drafter sized to that model's H/layers + regen+train there; 27B on 32GB is memory-tight (17.75GB weights + KV budget).
-- **Tighten the draft inference loop** (per-position host syncs, double 262K LM-head) so τ translates to wall-clock.
-- **Remaining paper components:** RNN head (Eq 6), STS calibration (§3.2.1), hardware-aware prefix scheduler (Alg 1 — single-user form = confidence-scheduled draft-length pruning).
-- **Rename** `dflash`→`dspark` (the faithful module IS DSpark; v1 is the legacy single-vector variant).
+
+- [x] Serve integration (Phase 1) · [x] Alg-1-shaped scheduler (matches the
+  RELEASED reference; paper Alg-1 unreleased) · [x] threshold calibration ·
+  [x] RNN head (Elman variant; paper's gated cell differs — documented above)
+  · [x] loop tightening · [x] `dspark` rename + loader —
+  **everything buildable without a GPU is DONE.**
+- [ ] **DeepSpec-variant port** (`Gemma4DSparkModel`) + loader detection +
+  oracle scripts — the no-training path to the 12B speedup (in progress).
+- [ ] **Josh-gated GPU (the payoff):** real data scale (thousands of
+  on-distribution generations) + **retarget to 12B** (`--model` +
+  `--tap-layers 5,17,29,41,46` (DeepSeek's trained taps); 27B memory-infeasible to train on 24 GB, kept
+  dim-generic) + train (`--seq-head` A/B optional) + **calibrate**
+  (`scripts/dspark-calibrate.ts`) + live-τ measure. Recipe:
+  [docs/investigations/dspark-handoff.md](../investigations/dspark-handoff.md).
+- [ ] Verify the Eq 6 / §3.2.1 shapes against the actual paper when the PDF
+  is available (both flagged in code).
 
 Full session handoff: `docs/investigations/dspark-handoff.md`.

package/docs/design/faithful-l1-consolidation.md

@@ -0,0 +1,310 @@
+# Faithful → L1: consolidating the "match mlx-lm" path
+
+> **2026-07-05 addendum:** the Phase-1 deletion pass executed —
+> fused-gelu/fused-swiglu/perf-kernel/fused-decode/CPM5_FAITHFUL deleted, `--l3`
+> removed ([unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md)).
+> The flag tables below are the historical decision record.
+
+**Status:** PLANNED (2026-07-04). Written after the 2026-07-04 audit of the
+`MLX_BUN_FAITHFUL` work (commit `fd8ff5f`). The faithful kernels are correct and
+proven byte-identical to mlx-lm — but they were wired as a *parallel base-preset
+axis* rather than folded into the L1 parity regime, which (a) abandons the
+model-specific optimized files and (b) leaves the preset incomplete. This doc is
+the contract for fixing that. Companion memory:
+`faithful-is-the-l1-implementation`. Framing: [parity-tier-dag.md](parity-tier-dag.md),
+`three-level-fidelity-tree-model` (memory).
+
+## The governing framing (two orthogonal axes)
+
+1. **Parity** — a *guarantee about which oracle you reproduce bit-for-bit*, full
+   logprobs (not just greedy argmax). **L1 = mlx-lm, L2 = mlx-optiq, L3 = no
+   oracle.** A config *is* L1 by definition iff its output matches mlx-lm bit-for-bit.
+   Nothing about parity concerns speed or methodology.
+2. **Performance / implementation** — orthogonal. Within a regime there are many
+   valid implementations (faster/slower, more/less memory). For a given regime we
+   **default-serve the fastest implementation that still holds the guarantee**; a
+   user overrides individual kernels with extra flags (`--l1 --fused-geglu`).
+
+**Where "faithful" fits:** it is *the fastest known implementation of L1* — it runs
+mlx-lm's own exact kernels, so it's bit-exact, and per the 2026-07-03 h2h it is
+faster than our previous L1 path (the old "perf" path was net-SLOWER than mlx-lm:
+12B 0.88×, cpm5 0.95×, perf-kernel HURT e4b 0.62×). So faithful should become
+**`--l1`'s default implementation**, not a new naked default and not a third
+selection system.
+
+**Naked `mlx-bun` (no flags) is unchanged:** best experience for local AI on a Mac —
+fastest / most-intelligent / most-memory-efficient, no parity worries. The user who
+needs the mlx-lm drop-in guarantee reads the docs and passes `--l1`.
+
+> **SUPERSEDED 2026-07-05 — naked default is now `--l1`.** The full h2h pass
+> (benchmarks-h2h-2026-07-05, M1 Max 32GB) showed the L1 faithful kernel set at
+> exact speed parity with mlx-lm (1.00×) on cpm5/e4b/12B while every
+> output-changing lever failed to beat it in a paired A/B: fused-decode 1.00×,
+> fused-gelu +0–1%, the perf arm 0.62–0.93× on e4b (its one win, 12B @16k +6%,
+> carried a KL WARN), and quantized KV 5–20% slower decode than bf16 at ≤16k on
+> BOTH stacks (mlx-lm's own kv8 trails its bf16). "Best experience" and "L1"
+> turned out to be the same thing today. Decision: naked = the l1 preset
+> (`applyDecodeRoute` defaults the tier), quantized KV and every non-faithful
+> kernel are opt-in (`--l2`/`--l3`/per-fork flags), and the L1 baseline is the
+> base all future optimization must beat in a paired A/B to earn a default.
+
+**`--l1/--l2/--l3` are pure ALIASES, not modes.** Each expands to a fixed set of the
+individual per-fork flags — [`applyDecodeRoute`, cli.ts:752](../../src/cli.ts) just
+`set()`s env vars, and a per-fork flag overrides the alias. Two invariants follow,
+and they are the real design bar:
+
+- **A user can reproduce any tier byte-for-byte by passing the individual flags.**
+  `--l1` is nothing but "the fastest parity-holding value of each flag."
+- **No preset may set anything that isn't also an individual flag.** There is exactly
+  one bundle per tier; nothing bypasses the flag layer.
+
+`MLX_BUN_FAITHFUL` violates both — it is a *second bundle* that sets choices
+(compiled-geglu, `compiledLogitSoftcap`) which are **not exposed as flags at all**, so
+`--l1` can neither reproduce nor reach them. Corollary: bit-exact-to-mlx-lm actually
+*wants* compiled activations (mlx-lm itself `@mx.compile`s geglu/swiglu), so the
+current `--l1` "unfused" description is backwards — **compiled is the L1 default**
+(bit-exact + faster); unfused is a slower, same-parity opt-in.
+
+## What is actually wrong today
+
+Three overlapping "match mlx-lm" mechanisms exist and disagree:
+
+- **Tier system** `--l1/--l2/--l3` ([`applyDecodeRoute`, cli.ts:752](../../src/cli.ts))
+  — the correct axis, but `--l1` keeps `compiled-decode` on, leaves
+  `MLX_BUN_FUSED_GELU` on (custom metal geglu, divergent), and does **not** route to
+  the faithful compiled-geglu. So today `--l1` is not actually bit-exact for gemma.
+- **`MLX_BUN_FAITHFUL` preset** ([faithful.ts](../../src/faithful.ts)) — flips a
+  *different, smaller* set of defaults and **forces the gemma monolith**
+  ([factory.ts:51](../../src/model/factory.ts)), abandoning the generated
+  `gemma4-12b/e4b/26b` specialization. It also leaves `MLX_BUN_PERF_KERNEL` on
+  (fires on every quantized-KV/OptiQ model), leaves kv-quant on, and never sets
+  `MLX_BUN_CPM5_FAITHFUL` — so `=1` is neither a real L1 guarantee on OptiQ models
+  nor faithful for MiniCPM5.
+- **`Faithful*` subclasses** — `FaithfulMiniCPM5` (wired), and `FaithfulGemma4` /
+  `FaithfulQwen3` / `FaithfulQwen35` / `FaithfulUniversalDense` (**unwired** — their
+  headers claim factory dispatch that does not exist; `MLX_BUN_UNIVERSAL_FAITHFUL`
+  lives only in a comment).
+
+Two of these violate stated principles: **forcing the monolith** runs an optimized
+model through the generic path (rejected), and the **preset is incomplete**.
+
+Only `qwen3_moe` did it right: the production class **is** the faithful port, so it's
+bit-exact-by-construction with no flag. That is the pattern to generalize.
+
+### Current-state flag reference
+
+| Flag | Default | Diverges from mlx-lm? | Notes |
+|---|---|---|---|
+| `MLX_BUN_PERF_KERNEL` | **ON** | **yes** — original kernel, not bit-exact | fires only on quantized KV; output-changing → L3-only |
+| kv-quant | **config** for OptiQ | uniform=mlx-lm has it; mixed=optiq-only | off (bf16)=L1; uniform 4\|8=L1 in the L1 set (unfused SDPA == mlx-lm base.py); config (mixed)=L2 |
+| `MLX_BUN_COMPILED_DECODE` | ON | no (whole-decode fusion, bit-exact) | structural; off in faithful preset |
+| `MLX_BUN_FUSED_GELU` | ON | **yes** — custom metal geglu ≠ compiled geglu | output-changing vs mlx-lm dispatch |
+| `MLX_BUN_FUSED_DECODE` | off | yes | L>1 only |
+| `MLX_BUN_NO_FUSED_SDPA` | fused **on** | no — `ops.sdpa` is mlx's own kernel | faithful-aligned |
+| `MLX_BUN_COMPILED_SWIGLU` | ON | no — matches mlx-lm `@mx.compile` | faithful-aligned |
+
+The genuinely output-changing, must-be-off-for-L1 levers: **perf-kernel, kv-quant,
+custom fused-gelu, fused-decode.** Compiled activations and `ops.sdpa` are
+faithful-aligned and should stay on.
+
+**Fork coverage gap.** `applyDecodeRoute` exposes only four forks —
+`--compiled-decode`, `--perf-kernel`, `--fused-decode`, `--fused-sdpa`. The
+activation-fusion choice (compiled-geglu vs custom fused-gelu; compiled-swiglu /
+`useCompiledActivation`) has **no CLI fork** and is not in the TIERS presets. So the
+alias invariant is broken: `--l1` cannot select the faithful compiled activation, and
+you cannot reproduce the faithful setup with individual flags — only the out-of-band
+`MLX_BUN_FAITHFUL` reaches it. Closing this gap is the crux of the fix.
+
+## The decisive rule: "compiled" = L1, custom "fused" = L3
+
+The whole flag pile collapses to one distinction, verified in-source 2026-07-04:
+
+- **"compiled" kernels** (`@mx.compile` geglu/swiglu, `compiled-decode`) go through
+  the **same libmlx** as mlx-lm → same graph → same kernel → **bit-exact**. These are
+  the faithful kernels. L1-valid AND faster (they remove per-op host tax).
+- **custom "fused" Metal kernels** are mlx-bun ORIGINALS, each with a proven residual
+  vs mlx-lm → **NOT bit-exact → L3 only**:
+  - `fused-gelu-kernel.ts` header: *"Still a perf kernel (not bit-exact)… the only
+    residual is the pow/tanh math-lib difference"* (bit-exact vs our OWN unfused
+    path, kl=0 — but not vs mlx-lm's math lib).
+  - `fused-swiglu-kernel.ts`: MATCH variant *"Only residual: the metal exp() vs MLX's
+    math lib"*; F32 variant *"NOT bit-exact → L3"*. Default off.
+  - `fused-decode-kernel.ts` (`PERF_KERNEL`): online-softmax, envelope-gated, quant-KV
+    only. Not bit-exact by construction.
+- **kv-quant** is a cache-precision axis, not a kernel, and it is NOT a clean
+  L1↔L2 boundary. **off (bf16) → L1**. **uniform `4|8` → L1** in the L1 kernel set
+  (`--l1 --kv-quant 8` = fused-sdpa off + perf off): mlx-bun's `quantizedSdpaUnfused`
+  (gemma4-base.ts) is **op-for-op identical** to mlx-lm's
+  `quantized_scaled_dot_product_attention` (`mlx_lm/models/base.py` — same
+  `mx.quantized_matmul` ×2 + `mx.softmax(precise=True)` + `where(…, finfo.min)` +
+  `queries *= scale`), and mlx-lm exposes exactly this scheme via `--kv-bits`. So
+  uniform-quant is **bit-exact L1 by construction** (a uniform-kv logit golden would
+  confirm the measured number; `bench.ts --baseline-kv 8` runs the stock-mlx-lm side
+  for the perf comparison). The FUSED / N-tiled prefill path (fused-sdpa ON — the
+  naked default and `--l2`) is the **optiq-aligned** one, not mlx-lm. **config
+  (per-layer mixed-precision) → L2** (optiq-only; mlx-lm has no per-layer analog).
+
+**Decision table (serve decode flags):**
+
+| Flag | Breaks mlx-lm parity? | L1 | L3 / naked | Relation to faithful |
+|---|---|---|---|---|
+| `MLX_BUN_COMPILED_SWIGLU` | no | ON | on | **is** faithful swiglu |
+| compiled-geglu *(needs a fork)* | no | ON | on | **is** faithful geglu |
+| `MLX_BUN_COMPILED_DECODE` | no (on==off) | ON | on | orthogonal free speed |
+| `MLX_BUN_FUSED_SDPA` | no (`ops.sdpa`=mlx kernel) | on (bf16 no-op) | on | faithful-aligned; L2 lever |
+| `MLX_BUN_FORCE_WIRE` / `EXPERT_OFFLOAD` | no (memory only) | user | user | orthogonal |
+| `MLX_BUN_FUSED_GELU` | **yes** (pow/tanh) | **OFF** | if faster, KL-gated | competes w/ faithful geglu |
+| `MLX_BUN_FUSED_SWIGLU`(`_F32`) | **yes** (exp/f32) | **OFF** | off (unproven) | competes w/ faithful swiglu |
+| `MLX_BUN_PERF_KERNEL` | **yes** (online softmax) | **OFF** | on w/ kv-quant | unrelated original |
+| `MLX_BUN_FUSED_DECODE` | **yes** | **OFF** | off | unrelated experiment |
+| `--kv-quant` | off=L1; uniform `4\|8`=L1 in the L1 set (unfused SDPA==mlx-lm); config=L2 | **off (bf16)**, uniform ok | config | cache axis, not kernel (see kv note above) |
+| `MLX_BUN_FAITHFUL` | — | **delete** | delete | illegal 2nd bundle → flags |
+| `MLX_BUN_CPM5_FAITHFUL` | no | fold into `--l1` | — | A/B tool; likely retire |
+
+So **L1 = all "compiled" flags on, all custom-"fused" flags off, kv-quant off** —
+which is exactly what the `--l1` alias should expand to. After Phase 1 gives
+compiled-geglu a fork, "faithful" stops being a separate concept: it's just the L1
+flag values.
+
+## Per-model target (the 13 supported models)
+
+"Targeted" = a dedicated optimized file exists and MUST be used in every regime.
+"Generic" = universal-dense fallback.
+
+| Model | Dedicated file | Faithful-L1 today | Gap |
+|---|---|---|---|
+| MiniCPM5-1B-OptiQ | `minicpm5.ts` (+`-faithful`, `CPM5_FAITHFUL`) | ~yes (compiled-swiglu default) | `--l1` ⇒ kv off + perf off; confirm default == faithful (may retire `-faithful`) |
+| Qwen3-Embedding-4B | `qwen3.ts` | ❌ unfused swiglu | make compiled-swiglu default; retire `FaithfulQwen3` |
+| Qwen3.5-4B / 9B-OptiQ | `qwen3_5.ts` | ❌ `useCompiledActivation=false` | default it true; retire `FaithfulQwen35` |
+| Qwen3.6-27B-OptiQ | `qwen3_5.ts` | ❌ same | same |
+| Qwen3-30B-A3B | `qwen3-moe.ts` | ✅ by default | gate perf/kv by regime only |
+| gemma-4-e4b/12B/26B-OptiQ | `generated/gemma4-*.ts` | ❌ faithful forces monolith | drop force-monolith; make compiled-geglu/softcap default |
+| Qwen2.5-0.5B, Llama-3.2-1B/3B | `universal/dense.ts` | ❌ unfused swiglu | default compiled-swiglu; retire `FaithfulUniversalDense` |
+| gemma-2-2b-it | `universal/dense.ts` | ✅ (gemma2 geglu not `@mx.compile`d upstream) | none |
+
+## Plan
+
+### Phase 1 — Global bit-exact wins (no output change) ✅ LANDED 2026-07-04
+
+Landed on branch `faithful-decode-parity` (M1 Max, 32 GB). Changes: qwen3-dense
+(`qwen3.ts`) and universal-dense (`universal/dense.ts` `swigluFn` default) now use
+the shared `compiledSwiglu`; gemma `faithfulGegluActive` defaults ON via
+`MLX_BUN_COMPILED_GEGLU` (kill-switch `=0`); custom `MLX_BUN_FUSED_GELU` metal geglu
+demoted to explicit opt-in (`fusedGeluEnabled` default off); factory force-monolith
+branch removed (gemma keeps its generated specialization). qwen3.5 MLP already
+compiled (`useCompiledActivation` fields were vestigial). **Verified bit-exact
+(`maxDiff===0`) vs mlx-lm:** `universal-parity` (Qwen2.5-0.5B, Llama-3.2-1B/3B,
+gemma-2-2b) 6/6; `generated-parity` (12B + e4b generated vs monolith) 5/5;
+`parity.test` (gemma-4-12B monolith greedy+logits vs mlx-lm goldens) 1/1. tsc 0,
+hygiene green. qwen3-dense (Qwen3-Embedding) not on this box → verified by analogy
+to the identical qwen3.5 MLP pattern. The already-global micro-fixes (native
+`mlx_arange`, host-side bf16 scalars) need no change — they were never
+`faithfulMode`-gated.
+
+These change *dispatched kernels only*, not output, so they apply in **every**
+regime (naked default included) and need no regime gate. Each is bit-exact + faster.
+
+1. **Compiled activation default-on** for the dedicated classes that still run it
+   unfused: `qwen3.ts`, `qwen3_5.ts` (`useCompiledActivation` → default true), and
+   `universal/dense.ts` swiglu kinds. (qwen3_moe, minicpm5 already default-on.)
+2. **Decouple compiled-geglu + `compiledLogitSoftcap` from `faithfulMode()`** in
+   `gemma4.ts` (`faithfulGegluActive`) → make them the default; retire the custom
+   `MLX_BUN_FUSED_GELU` metal geglu to explicit opt-in. This makes gemma bit-exact on
+   the activation axis under `--l1` *and* naked default, in the generated files (which
+   inherit the base MLP) without regenerating them.
+3. **Drop the "force monolith under faithfulMode" branch** in `factory.ts` — the
+   generated `gemma4-*` specialization keeps running; it already gates
+   perf-kernel/fused-decode on flags, so `--l1` (perf/fused off) yields the L1 path
+   while staying model-specific.
+4. Confirm the already-global bit-exact micro-fixes (native `mlx_arange`, host-side
+   bf16 scalars) apply on all paths.
+
+**Verify per model:** bit-exact tokens **and** logprobs vs the mlx-lm oracle venv
+(`/Users/joshrossi/Code/mlx-lm/.venv`), plus an xctrace shader-list diff where a
+kernel-set claim is made (`.gputrace` isn't parseable — use the shader-list method).
+
+### Phase 2 — Make `--l1` a pure alias over complete forks ✅ LANDED 2026-07-04
+
+Landed: added `--compiled-activations on|off` (drives `MLX_BUN_COMPILED_GEGLU` +
+`MLX_BUN_COMPILED_SWIGLU`) and `--fused-gelu on|off` (`MLX_BUN_FUSED_GELU`, the
+custom L3 kernel) as per-fork flags in `applyDecodeRoute`; the `TIERS` presets now
+set `compiledAct: true, fusedGelu: false` for l1/l2/l3, so `--l1` = { bf16 KV,
+perf-kernel off, fused-decode off, **compiled activations on, custom fused-gelu
+off** } and is byte-reproducible from individual flags. Fixed the `--l1` help text
+(compiled activations, not "unfused"); documented both forks in `serve --help`, the
+ready-card lever summary, `server-config.md`, and `cli.md`. Verified: tsc 0, help
+renders the flags, and gemma `parity.test` is bit-exact vs mlx-lm under BOTH
+`MLX_BUN_COMPILED_GEGLU=1` (default) and `=0` (uncompiled) — proving
+`--compiled-activations off` is a valid same-parity opt-in and the env plumbing name
+is honored. (qwen3/qwen3.5/universal compile swiglu unconditionally, so
+`--compiled-activations off` only reverts gemma geglu + MiniCPM5 swiglu — documented.)
+
+Original plan for reference:
+
+First **close the fork-coverage gap**: expose the activation-fusion choice as an
+individual per-fork flag (e.g. `--geglu compiled|fused|unfused` and the swiglu
+equivalent, or booleans `--compiled-geglu` / `--fused-gelu` / `--compiled-swiglu`),
+reading the same env vars the model files already check. Then wire it into the
+`TIERS` presets in `applyDecodeRoute` so the `l1` alias sets it to the compiled
+(bit-exact-fastest) value.
+
+Result: `--l1` ⇒ { kv-quant off (bf16), perf-kernel off, fused-decode off,
+custom-fused-gelu off, **compiled activations on**, model-specific file retained } —
+and every one of those is an individual flag, so `--l1` is byte-reproducible by hand
+and any single choice is overridable (`--l1 --fused-gelu on`). For MiniCPM5, `--l1`
+implies whatever makes its decode bit-exact (its default may already qualify once
+kv/perf are off — decide whether `FaithfulMiniCPM5` is still needed or retire it).
+
+Fix the `--l1` help text: it is compiled activations + bf16 KV (not "unfused").
+
+### Phase 3 — Retire the parallel axis ✅ LANDED 2026-07-04
+
+Deleted `MLX_BUN_FAITHFUL`/`faithfulMode`: `src/faithful.ts` → `src/flags.ts` (keeps
+only the generic `flagOn` helper); `generate.ts` compiled-decode default is now a
+plain `flagOn(…, true)`. Deleted the four unwired subclasses (`gemma4-faithful`,
+`qwen3-faithful`, `qwen3_5-faithful`, `universal/dense-faithful`) and their exclusive
+hooks: `setFaithfulGeglu`/`_faithfulGeglu` gone (`faithfulGegluActive` → the simpler
+`compiledGegluActive` = `flagOn("MLX_BUN_COMPILED_GEGLU", true)`), the dead
+`useCompiledActivation` fields gone (qwen3.5 already compiles all three activation
+sites unconditionally). **Kept `FaithfulMiniCPM5`** (`MLX_BUN_CPM5_FAITHFUL`) as the
+one wired, tested, complete op-for-op A/B reference. Verified: tsc 0, repo-wide sweep
+for the deleted concepts is clean, gemma `parity.test` + `minicpm5-faithful-parity`
+green. Net: "faithful" is no longer a concept in the code — it's just the default
+kernels, selected (or not) by the individual flags.
+
+### Phase 4 — Docs + benchmark ✅ LANDED 2026-07-04 (numbers Josh-gated)
+
+Docs updated: `features-matrix.md` (new `--compiled-activations` + `--fused-gelu`
+rows), `server-config.md` + `cli.md` (Phase 2). Reworked `scripts/bench-faithful-matrix.ts`
++ `benchmark.sh` comparison 0 for the new model — since faithful is now the default,
+the matrix measures what **removing** each faithful kernel costs and what the L3
+custom kernels add on top (`− compiled-decode`, `− compiled activations`,
+`+ custom fused-gelu`), all vs mlx-lm; tsc 0. **The actual numbers are Josh-gated**
+(clean/rebooted machine — `./benchmark.sh --redo`, or
+`bun scripts/bench-faithful-matrix.ts`); loaded-machine absolutes are noise. Promote
+quotable rows to `benchmarks/RESULTS.md` after the clean run.
+
+## Status: consolidation complete
+
+Phases 1–4 landed on `faithful-decode-parity`. The three overlapping "match mlx-lm"
+mechanisms are now one: the individual flags, with `--l1/l2/l3` as pure aliases.
+Faithful kernels are the default; `--l1` is the bit-exact-mlx-lm implementation;
+naked default is best-experience. Only follow-ups: (a) live qwen3-dense parity on a
+box that has Qwen3-Embedding; (b) the clean-machine bench numbers.
+
+## Open decisions
+
+- **`FaithfulMiniCPM5`**: retire (if `MiniCPM5Model` default is already bit-exact
+  mlx-lm under `--l1`) or keep as the A/B harness?
+- **Keep one A/B backend** per family for regression proofing, or delete all and
+  lean on the oracle parity tests?
+- **`MLX_BUN_FAITHFUL`**: delete, or alias to `--l1`?
+
+## Risks
+
+- Compiled activation must be **autograd-safe** on the training path (mlx-lm's
+  `@mx.compile` threads the VJP; verified for cpm5/dense earlier — re-verify per
+  family before defaulting on).
+- Making compiled-geglu the naked default changes gemma's *default* dispatched
+  kernels; confirm no downstream (vision, batching, generated specializations)
+  assumes the unfused/custom path.

package/docs/design/generic-model-support.md

@@ -22,10 +22,10 @@ This doc generalizes that shape to the whole mlx-lm zoo:
 - **Tier-0 GENERIC** — one config-driven "universal" module per architecture
   *family* (dense llama-family first, MoE second). Runs any covered checkpoint
   at **L1 parity** (bit-exact vs mlx-lm on this machine's GPU). Monolith path
-  only: no perf kernels, no compiled decode, no kv-quant. Slow, never broken.
+  only: no compiled decode, no kv-quant. Slow, never broken.
 - **Tier-1 TARGETED** — a model *graduates*: dedicated/generated forward,
-  optiq-quant **L2** parity, **L3** original kernels, benchmark entries.
-  Exactly what gemma4 / minicpm5 / qwen3_5 already are.
+  optiq-quant **L2** parity, Lab experiments where warranted, benchmark
+  entries. Exactly what gemma4 / minicpm5 / qwen3_5 already are.
 
 The fidelity contract is unchanged (three-level tree): **L1 = mlx-lm
 bit-exact is the gate for ANY new model**, generic or targeted. A model the
@@ -337,8 +337,11 @@ vs a dedicated path that's worth the maintenance (number in
    `Gemma4Model` specifically — generalizing it to emit from
    `UniversalDense` is Phase 4 work, and *optional*: hand ports are proven
    and cheap ("faithful port took ~1 hr, not the huge effort claimed").
-3. **L3 kernels**: fused decode / compiled decode / kv-quant / flash paths,
-   each behind its existing flag, each frozen-oracle-gated.
+3. **Perf levers**: compiled decode / kv-quant / fused-sdpa, each behind its
+   existing flag. (The former "L3 kernels" step — fused decode / flash paths
+   with frozen-oracle gates — was deleted 2026-07-05; output-changing
+   experiments are now Lab items, see
+   [unified-engine-frontier-plan.md §6](unified-engine-frontier-plan.md).)
 4. RESULTS.md + docs entries; `mlx-bun models` label flips to `targeted`.
 
 Nothing about graduation removes the generic path — it stays as the