mlx-bun 0.2.0 → 0.3.0

package/docs/design/turboquant-weights.md

@@ -1,467 +0,0 @@
-# TurboQuant weights — rotation-folded quantization (design)
-
-Status: active. Fold recipes and artifact-pipeline integration are implemented;
-quality/release gates remain in PLAN.md "TurboQuant weights". Companion to
-docs/design/turboquant-kv.md (the landed KV leg).
-
-## What it is
-
-Fold an orthogonal rotation R offline into producer/consumer weight
-pairs so every weight matrix is quantized in a rotated basis where
-outlier channels are smeared into near-Gaussian marginals — then
-quantize with mlx's EXISTING formats (affine / mxfp4 / nvfp4). Zero
-runtime cost, zero new kernels, output loadable by stock mlx-lm. The
-QuaRot/SpinQuant family; composes with (does not replace) per-layer
-allocation (OptiQ-style) and calibration.
-
-Target: `Qwen3.8-27B-MTP-turbo` beating OptiQ-4bit / plain 4bit at
-equal effective bpw (gate: ppl + frozen 6-task eval, eval DB rows).
-
-## Oracle chain
-
-"The oracle is whoever ships it": the folding mechanics come from the
-reference repos, read 2026-08-17 (agent fan-out, full-file reads):
-
-- **QuaRot** (arXiv:2404.00456, github spcl/QuaRot):
-  `fake_quant/rotation_utils.py` (fuse_ln_linear, fuse_layer_norms,
-  rotate_model), `fake_quant/hadamard_utils.py` (get_hadK, matmul_hadU,
-  random_hadamard_matrix), `fake_quant/model_utils.py`.
-- **SpinQuant** (arXiv:2405.16406, github facebookresearch/SpinQuant):
-  `utils/fuse_norm_utils.py` (a copy of QuaRot's γ-fold),
-  `eval_utils/rotation_utils.py` (R1/R2 folds), `optimize_rotation.py`
-  (Cayley-SGD learned R — deferred to W5).
-
-Weight layout: torch and MLX `nn.Linear` are both `[out, in]`,
-`y = x @ W.T` — their fold table transfers verbatim.
-
-## The recipe (weight-only, fully offline)
-
-Order matters: γ-fold FIRST (R1 commutes only with gain-free RMSNorm),
-then R1, then R2, then quantize.
-
-**0. Untie embeddings** if `tie_word_embeddings` (clone into a separate
-lm_head, set the flag false). Required: step 1 transforms lm_head but
-not embed_tokens. (SpinQuant `ptq.py` does exactly this.)
-
-**1. γ-fold (norm fusion)** — for each RMSNorm feeding linears,
-`W' = W · diag(γ)` on the INPUT dim (columns of [out,in]), then set that
-norm's γ to ones (keep the module + its eps — do NOT copy QuaRot's
-replacement RMSN, which hardcodes eps=1e-5; our subjects use 1e-6):
-- input_layernorm → q/k/v_proj
-- post_attention_layernorm → gate/up_proj
-- final norm → lm_head
-
-**2. R1 residual fold** — one hidden-size orthogonal R1:
-
-| matrix | transform | reason |
-|---|---|---|
-| embed_tokens | `W @ R1` | writes residual (rows are embeddings) |
-| q/k/v_proj | `W @ R1` | reads residual (input dim) |
-| o_proj (+bias) | `R1ᵀ @ W` | writes residual (output dim) |
-| gate/up_proj | `W @ R1` | reads residual |
-| down_proj (+bias) | `R1ᵀ @ W` | writes residual |
-| lm_head | `W @ R1` | reads residual |
-
-Q/K outputs untouched → RoPE unaffected. Input-side folds never touch
-biases (bias lives on the output).
-
-**3. R2 per-head v/o fold — SpinQuant's version, NOT QuaRot's.** Per
-layer, one head_dim-sized orthogonal R2 applied block-diagonally:
-v_proj output blocks get R2ᵀ, o_proj input blocks get R2; softmax
-weights are R2-invariant so it cancels exactly, incl. under GQA (same
-R2 for all heads; repeat_kv replicates whole heads). QuaRot's variant
-folds a FULL Hadamard into o_proj and needs an ONLINE cross-head
-Hadamard (`online_partial_had`) to cancel — folding its v-side alone
-breaks the network. Skip QuaRot's pairing entirely.
-
-**4. Quantize** the folded weights with the stock convert path
-(affine g64 / mxfp4 / nvfp4 arms).
-
-Implementation seam (2026-08-20): `WeightTransform` separates pure tensor-name
-planning from lazy tensor application. Llama, Qwen3.5 trunk, and Qwen MTP are
-adapters over that contract; `quantizeModelDir` applies the selected plan in its
-existing module walk. `mlx-bun convert --rotate-weights` selects the adapter,
-and `optiq_metadata.json` records the transform id, family, seed, and deviations.
-Mixed-precision calibration receives a `ProbeSource`; its default builds an
-8-bit probe through the lower-level writer instead of recursively re-entering
-the top-level quantizer.
-
-### Deviations from the references (decided, with reasons)
-
-- **No embedding mean-centering.** Their `fuse_layer_norms`
-  mean-centers embedding rows (SliceGPT LayerNorm→RMSNorm trick). For
-  RMSNorm models this is NOT an exact-invariance (RMSNorm doesn't
-  subtract means) — they do it anyway; we don't, because W0's exit
-  criterion is logit parity of the folded bf16 model.
-- **Delete the hidden R4 half-fold.** BOTH repos'
-  `rotate_mlp_output` unconditionally applies a full Hadamard to
-  down_proj's input dim right after the R1ᵀ fold — that is the offline
-  HALF of R4 and requires the runtime `online_full_had` on down_proj's
-  activation. Weight-only must skip both halves together (their line:
-  QuaRot rotation_utils.py:173 / SpinQuant eval_utils/rotation_utils.py
-  98-100). The `R1ᵀ @ W_down` part stays.
-- **No R3.** Post-RoPE Q/K rotation exists only for KV-cache quant
-  (their code registers it only when k_bits < 16). Our KV story is the
-  landed TurboQuant KV codec — online rotation remains KV-only.
-- **Fold precision: fp32, not fp64.** References fold in fp64 (fp16-era
-  caution). Our weights are bf16 (8-bit mantissa); fp32 accumulation at
-  n≲6k leaves ample headroom. Escape hatch if the W0 parity gate fails:
-  generate fold goldens in the oracle venv (mlx CPU float64).
-
-### R generation
-
-- Random-Hadamard R1 = `diag(±1 signs) · H_n / √n` (QuIP#-style).
-  Non-pow-2 n via Kronecker: n = K·2^m with hardcoded H_K, K ∈ {12, 20,
-  28, 36, 40, 44, 52, 60, 108, 140, 156, 172}; else n must be pow-2.
-  QR-of-Gaussian `random` mode works for any n (fold-time only, speed
-  irrelevant).
-- R2 default: plain Hadamard of head_dim (must be pow-2) or any
-  orthogonal.
-- Seed-pin and persist R1/R2 (sidecar JSON in the output snapshot) for
-  reproducible artifacts; R is never needed at runtime.
-- W5 (only if random-R wins first): SpinQuant Cayley-SGD learned R1 +
-  per-layer R2, folded identically afterward.
-
-## W1 — Qwen3.8 (qwen3_5) corridor map (2026-08-18, from src/model/qwen3_5.ts
-+ src/spec/qwen-mtp-source.ts + src/vision/qwen3vl-tower.ts + tensor dumps)
-
-**R2 is architecturally OFF for this family:** full attention computes
-`o_proj(attention_output · σ(gate))` — q_proj emits 2×head_dim per head and
-the gate multiplies the attention output ELEMENTWISE in head space. An
-elementwise gate does not commute with a per-head rotation, so only γ+R1
-apply. hidden 5120 = 20·256 — mlx's hadamard_transform takes it natively
-(m·2^k, m ∈ {1,12,20,28}; verified live).
-
-Trunk corridors (prefix `language_model.`):
-- readers (@R1 input dim, γ folded in): self_attn.q/k/v_proj,
-  linear_attn.in_proj_{qkv,z,b,a}, mlp.gate/up_proj, lm_head (γ = model.norm)
-- writers (R1ᵀ output dim): self_attn.o_proj, linear_attn.out_proj,
-  mlp.down_proj
-- vision: `vision_tower.merger.linear_fc2` weight+bias is the ONLY
-  vision→residual seam (deepstack is empty for qwen3_5 and not ported) —
-  output-dim fold; everything else in the tower passes through bf16
-- untouched internal bases: q_norm/k_norm (head-space), linear_attn.{norm,
-  A_log, conv1d, dt_bias} (post-projection), rotary
-- 27B trunk is untied; the 0.8B proof subject is tied (untie step exercised)
-
-MTP companion (separate artifact, same seed/R1 — shared residual basis):
-- `fc [H,2H]`: input block 0 = embedding stream (γ = pre_fc_norm_embedding),
-  block 1 = hidden stream (γ = pre_fc_norm_hidden) — per-block γ+@R1 — plus
-  an output-dim R1ᵀ fold; both pre-fc norms → ones
-- `layers.0`: standard full-attention corridor treatment
-- **final `norm` γ is DROPPED (→ ones):** it feeds the SHARED trunk lm_head,
-  which already carries the trunk's final γ. Draft logits therefore see
-  γ_trunk instead of γ_mtp — draft-quality-only (the verified target path is
-  exact); revisit lever = ship a private folded head inside the companion.
-
-Small-scale proof (0.8B, dequantized from OptiQ-4bit since no bf16 is
-published; fold-parity on dequant-vs-folded-dequant tests the fold exactly):
-teacher-forced logits through STOCK mlx-lm (also the W2 cross-stack check) —
-worst per-position KL 0.00353, argmax flips 2/42 positions BOTH at reference
-margin 0.0 (exact ties). scripts/experiments/{dequant-model,fold-qwen35}.ts +
-tq-fold-parity.py.
-
-Quantization packaging (CORRECTED 2026-08-18: OptiQ does NOT fully strip
-vision — its main weights omit it but `optiq/optiq_vision.safetensors`
-carries it, and `optiq/mtp.safetensors` bundles MTP; their artifact also
-publishes `optiq/sensitivity.json` — per-layer KL sensitivities for
-{4,8}-bit g64 against the SAME bf16 trunk, the ready-made allocation
-input for a GPTQ+sensitivity mixed pass. No kv_config.json is published.)
-Our packaging: language modules quantize uniformly (embed + lm_head
-included), `vision_tower.*` is predicate-excluded and stays bf16 both
-IN-MAIN (mlx-vlm compatible — OptiQ's sidecar-only vision is not) and as
-the `optiq/` sidecar, so one self-contained artifact serves text+vision
-across stacks.
-
-## Subject models
-
-- **W0/W3: mlx-community/Llama-3.2-1B-Instruct-bf16** (~2.5 GB; the
-  reference repos' literal target family; 4bit sibling already proven
-  in our engine; TIED embeddings → exercises step 0). MiniCPM5-1B has
-  no published mlx bf16 (would need a dtype-convert of openbmb's torch
-  weights) — alternate subject if Llama surprises us; its geometry is
-  fold-friendly too (model_type llama, untied, head_dim 128,
-  hidden 1536 = had12 ⊗ H128, eps 1e-6).
-- **W4: mlx-community/Qwen3.8-27B-bf16** (11 shards, 54.7 GB) +
-  `Qwen3.8-27B-MTP-bf16` companion (~850 MB, local). The companion
-  shares the trunk's residual basis → must fold consistently (W1
-  decision). DeltaNet linear-attention corridors: folding math not in
-  the papers; derive or fall back to allocation-only for those blocks
-  (community quants mark exactly those projections sensitive).
-
-## W3/W4 measured curve (2026-08-18, RTN affine g64, ppl via stock mlx-lm
-on the local UF-derived corpus, seq 512; M1 Max 32 GB)
-
-**Headline: rotation-only + RTN does NOT beat plain RTN at the 4-bit
-operating point; it wins decisively at 3-bit — the paper's law, reproduced.**
-Per-module function-space Frobenius error is a wash at 4-bit (~0.091 all
-arms, scripts/experiments/tq-quant-error.py — γs are tame, 0.7–2.7, no
-module class is the culprit); the 4-bit regression is the anisotropy story
-(isotropic rotated error vs activation-aligned plain error), which is
-exactly what GPTQ-style calibration fixes — the composition the paper and
-this doc already name as the real win.
-
-0.8B lab (dequant-OptiQ source; PAIRED arms; bf16 anchor 6.41, 48×512):
-
-| arm | bpw | ppl |
-|---|---|---|
-| plain 4-bit | 4.50 | 7.01 |
-| TQ(R1+γ) 4-bit | 4.50 | 7.39  ← regression, reproduces 27B |
-| plain 3-bit | 3.50 | 19.19 ← RTN collapse |
-| **TQ 3-bit** | 3.50 | **14.54** ← rotation −24% ppl |
-| plain mixed (3-bit MLP, 4-bit attn/embed) | 4.15 | 9.69 |
-| **TQ mixed (same profile)** | 4.24 | **9.22** ← paired rotation win |
-
-27B (32×512, same corpus): plain 4-bit 4.659 ±0.093 (4.50 bpw, 15.0 GB) ·
-TQ 4-bit 4.923 ±0.102 (+5.7%, consistent with the lab) · **TQ-mixed
-(4-bit attn/embed + 3-bit MLP) 4.932 ±0.098 at 3.86 bpw / 13.9 GB** —
-matches rotated-uniform-4-bit quality at −0.64 bpw; the artifact of
-record for the 14z M4-Pro fit lever.
-
-AWQ-style equalization spike (norm-carried per-channel scales, α=0.5,
-function-preservation verified at KL 0.0037/0 flips): TQ+eq 7.363 (≈no
-change), plain+eq 7.809 (HURT — per-channel scales inside a g64 group
-widen group ranges; real AWQ grid-searches a far gentler α and the real
-4-bit fix is GPTQ-style compensated rounding). Scripts kept:
-tq-collect-actstats.py + tq-equalize.ts.
-
-**End-to-end validation of the TQ-mixed 27B artifact (2026-08-18, M1
-Max, busy box — correctness only):** server chat with correct arithmetic
-reasoning (thinking → `reasoning` field); vision through the folded
-merger + sidecar correctly identifies the gradient fixture's colors over
-HTTP; MTP via the 14g harness with the same-seed folded companion: 71%
-acceptance, 2.40 tok/target-forward, arms token-identical (losslessness),
-ON/OFF 0.726× (consistent with the stock 0.821× correct-but-slower
-verdict; MTP stays opt-in).
-
-Consequences for the release recipe: do NOT ship rotated uniform-4-bit
-(loses to the trivial baseline); the shipping artifact is the TQ-mixed
-≤4 bpw band where rotation is decisively ahead (paired no-rotation
-control worse; uniform-3-bit −24% ppl). The 4.5-bpw flagship win waits
-for the calibration composition (GPTQ on ROTATED weights — the W5 leg).
-
-## W5a corrected-GPTQ matrix (2026-08-18; 0.8B lab, 4.5 bpw arms, 48×512
-ppl, anchor = dequant-OptiQ 6.41 — verified lossless: original OptiQ
-scores 6.4077)
-
-mlx-lm's shipped GPTQ (`mlx_lm/quant/gptq.py`) has TWO real defects that
-partially cancel: the in-loop update window is `k:k+j` (over-propagates
-past the block edge; must be `k:j` — the k term itself writes the
-quantized value since e·Hinv[k,k] = w−q) and `err[..., k:k+1]` indexes a
-group-local buffer with the GLOBAL k (mlx out-of-range slice assignment
-silently no-ops → cross-group propagation lost after block 0). Fixing
-only err DOUBLE-compensates [j, k+j) and REGRESSES below RTN (measured:
-7.22/7.50); fixing both = paper GPTQ. Worth an upstream report.
-Fork: scripts/experiments/tq-gptq.py (also restricts GPTQ+fallback to
-language modules — vision H is a zero scalar and stays bf16).
-
-| arm | ppl |
-|---|---|
-| **plain + GPTQ (fixed)** | **6.741** |
-| TQ(R1) + GPTQ (fixed) | 6.847 |
-| plain RTN | 7.010 |
-| TQ RTN | 7.390 |
-
-**Verdict: calibration is the main course; rotation SUBTRACTS at the
-4-bit band even under GPTQ on this family** (gap −1.6%; GPTQ closes half
-the RTN→anchor gap). Rotation remains the sub-4-bpw lever (−24% at
-3-bit). Flagship recipe = plain + GPTQ 4-bit; TQ-mixed stays the
-small-footprint variant. 27B needs a chunked Hessian/GPTQ driver (stock
-flow = whole bf16 model + all Hessians resident; down_proj H = 1.2 GB
-f32 each).
-
-## W5c 27B production results (2026-08-18)
-
-**v2 run (uniform GPTQ-4): BROKEN artifact (ppl 306k).** Root cause,
-evidence-backed: the compensation loop DIVERGED on layers with
-rank-deficient calibration Hessians (layer-0 down_proj group scales grew
-0.003 → 27 monotonically across columns, max 2.8e7; cosine ~0 vs source;
-neighbors healthy). Early layers see low-rank activations (layer-0 input
-= raw embeddings; rank ≤ distinct calibration tokens), so 5120 dims
-outran the standard 1% damping — the 0.8B (1024 dims) never hit it,
-which is why validation passed there.
-
-**v3 fixes (all landed):** divergence guard in `gptq_one_guarded`
-(GPTQ scales vs 4× the RTN scale ceiling per matrix; damping escalation
-1e-2→10; RTN fallback = never worse than RTN); 4× calibration (128×512);
-sensitivity-driven per-module bits from OptiQ's published
-sensitivity.json (semantics: `sensitivities[b]` = measured KL GAIN at b
-bits — first read had the sign flipped and allocated 0 modules); greedy
-benefit-per-param to `--target-bpw`. v3 27B: 169 modules @8-bit,
-4.80 bpw, 17.0 GB, ZERO guard triggers (the calibration bump fixed
-conditioning outright), 64 layers in 166 min, per-layer checkpoints.
-
-**27B ppl ladder (73×512, UF corpus, stock mlx-lm):** plain RTN-4
-4.570 ±0.060 · OptiQ 5.14 bpw 4.574 ±0.061 · GPTQ+sens 4.80 bpw
-4.618 ±0.061. **At 27B every sane ≥4.5-bpw recipe SATURATES this
-instrument** — OptiQ's +0.6 bpw buys nothing measurable, and GPTQ shows
-a small consistent deficit (likely calibration-domain mismatch:
-calibration_v5 vs chat-flavored eval text). This contradicts the 0.8B
-lab ordering (GPTQ clearly won there) — quantization robustness grows
-with scale. Decision moves to task benchmarks: MMLU-100 + GSM8K-50
-across {GPTQ-v3, plain4, OptiQ, TQ-mixed}, eval DB rows.
-
-## FINAL BOARD + DECISION (2026-08-18 night 2)
-
-Task evals (lean mlx-lm runner, paired items; in-engine sweep swap-thrashed
-the box — 73 GB swap at 0.7 GB resident — killed, gap recorded below):
-
-| arm | bpw | GB | ppl(73×512) | MMLU-100 | GSM8K-50 |
-|---|---|---|---|---|---|
-| **GPTQ+sens v3 (SHIPPED as mjriii/Qwen3.8-27B)** | 4.80 | 16.3 | 4.618 | 87 | **96** |
-| plain RTN 4-bit | 4.50 | 15.0 | **4.570** | 88 | 94 |
-| OptiQ published | 5.14 | 18.1 | 4.574 | **89** | 92 |
-| TQ-mixed (staged, small-footprint variant) | 3.86 | 13.0 | 4.93 | 82 | 96 |
-
-All ≥4.5-bpw arms statistically tied on every instrument — 27B quality
-SATURATES at 4-bit; recipes differentiate below 4 bpw and on
-completeness. THE artifact = v3 (never trails; protective recipe;
-one-repo vision+video+MTP; additive to the ecosystem where plain-4bit
-already exists publicly). Serve gauntlet on THE artifact through OUR
-engine: text ✓ (23×19), image ✓ ("Green and pink"), video ✓ (gradient
-motion described; AVFoundation chain), MTP harness 76% accept /
-2.53 tok-per-forward with an output-divergence flag vs plain greedy
-(consistent with verify-width reduction-order near-ties, 12B step-0
-class — margin analysis queued; NOT claimed lossless on the card).
-
-## Territory-mapping campaign (2026-08-18/19, Josh-directed: "map all the
-levers before concluding TQ-off")
-
-**sens3 arm (Josh's recipe: rotated 3-bit base + sensitivity-driven 8-bit,
-4.26 bpw / 15.25 GB incl. vision):** ppl 4.630 ±0.058 — TIED with the
-4.80-bpw flagship (4.618) at −0.54 bpw, and a full recovery over
-unprotected TQ-mixed (4.93). GSM8K 46/50 = 92% (plateau band). **MMLU
-83/100 — the 3-bit-base dip persists** (TQ-mixed scored 82): two
-independent 3-bit-base arms at 82–83 vs the 87–89 plateau → the
-knowledge-recall damage is SMEARED across the MLP long tail; KL-derived
-sensitivity shields fluency, not recall. Lever under test next: rotated
-4-bit g128 (4.25 bpw — same size, 4-bit granularity everywhere).
-
-Perplexity proxy calibration (from our own table): deltas <~0.1 ppl are
-task-invisible; ~0.3+ coincides with visible MMLU damage; ppl's
-remaining role is catastrophe detection + screening. mlx kernel bound:
-group_size ∈ {32, 64, 128} — 256+ needs custom kernels (non-goal).
-
-**Group-size lab (0.8B ppl) + 27B g128 arm:** rotation does NOT buy
-coarse-group tolerance at 4-bit (tax +4.7% rotated vs +4.2% plain); at
-3-bit the dividend is real but partial (+29% vs +47%; rotated 3.25 bpw ≈
-plain 3.5 bpw). 27B plain-4bit-g128 (4.25 bpw): ppl 4.714 (+3%), MMLU 88
-(plateau!), **GSM8K 62% — COLLAPSE via generation-behavior damage**
-(eyeballed: correct-but-rambling truncations + instant-EOS empty
-outputs; single-forward instruments blind to it). Lessons: (1) ppl deltas
-DO NOT bound task damage (+0.14 ppl ↔ −32 GSM8K pts) — the earlier
-"~0.3 ppl = visible damage" calibration is WRONG; every finalist needs
-generative tasks; (2) g128 disqualified for generative work.
-
-**Run B — the full suite (rotation + GPTQ + sensitivity, 4.80 bpw,
-17.0 GB, zero guard triggers):** ppl **4.5965** (best quantized arm,
-tied w/ plain-4bit) · MMLU 84 · raw-completion GSM8K **24% — and the
-collapse is a FORMAT CLIFF, not a reasoning failure**: when it answers
-it is flawless, but 3/4 raw few-shot prompts get INSTANT EOS; under the
-native chat template it always answers (3/5 correct at a tight token
-cap). g128's 62% was the same failure at lower frequency. **Methodology
-consequence: the raw-completion GSM8K column conflates math ability
-with EOS-cliff robustness** — different quant-noise realizations
-displace a razor-thin answer-vs-stop margin at token 0. The map now
-carries TWO generative columns: templated GSM8K (serving-path
-reasoning, re-run across all live arms) and raw-completion GSM8K
-(format-robustness probe). Neither ppl nor MMLU sees any of this.
-
-## THE FRONTIER (campaign complete, 2026-08-19 — map:
-reports/frontier-map-2026-08-19.html)
-
-| arm | bpw | GB | ppl | MMLU | tGSM | rawGSM | tok/s* | Pareto |
-|---|---|---|---|---|---|---|---|---|
-| **rot+sens48 RTN (Josh's recipe)** | 4.79 | 17.0 | 4.680 | **90** | 94 | 84 | **11.80** | **YES** |
-| GPTQ+sens (staged flagship) | 4.80 | 17.0 | 4.618 | 87 | 94 | **96** | 11.42 | — |
-| rot+GPTQ+sens (full suite) | 4.80 | 17.0 | **4.597** | 84 | 94 | 24 | 11.77 | — |
-| OptiQ published | 5.14 | 18.1 | 4.574 | 89 | 94 | 92 | 8.92 | — (dominated) |
-| **TQ-mixed** | 3.86 | 13.0 | 4.932 | 82 | 82 | 96 | 10.94 | **YES** |
-| bf16 ceiling (streamed eval) | 16 | 54.4 | 4.552 | 87 | n/a | n/a | n/a | ref |
-
-*busy-box, spreads 8–34% — directional until quiet benchmark.sh.
-
-Findings of record: (1) the ≥4.5-bpw band sits AT the bf16 ceiling (ppl
-within ~1%, MMLU within noise — a quant scoring 90 vs ceiling 87 proves
-±3 noise); (2) BOTH Pareto points are ROTATED — rotation at 4-bit is
-competitive exactly when paired with sensitivity allocation (the earlier
-uniform-rotation loss concentrated in the sensitive modules, as Josh
-hypothesized), and mandatory below 4 bpw; (3) OptiQ is dominated on all
-three axes (its extra 0.34 bpw buys nothing and costs ~25% decode);
-(4) GPTQ's measurable 27B contribution is ppl-only — it buys no task
-quality at this scale and (with rotation) costs raw-format robustness;
-(5) 3-bit kernels pay an unpack tax that eats their bandwidth win
-(TQ-mixed slower than 4-bit arms despite −4 GB); (6) cross-engine parity
-CERTIFIED (byte-identical greedy, ours vs stock mlx-lm, on the staged
-flagship). Remaining before publish: promote+package the chosen artifact,
-its serve gauntlet, quiet-box TPS, M4 Pro 24 GB cut (Josh-gated).
-
-## Known engine gaps found in passing (not TQ defects)
-
-- `mlx-bun perplexity` cannot score qwen3_5: it routes through
-  `trainForward`, whose cache stub lacks the DeltaNet `SSMCache.advance`
-  (`qwen3_5.ts:226` throws on BOTH plain and TQ arms identically).
-  27B ppl therefore runs through stock mlx-lm
-  (scripts/experiments/tq-ppl.py) — which doubles as the cross-stack
-  load check for the release artifact.
-- bf16 (unquantized) qwen3_5 trunks don't load in our engine:
-  `QuantizedEmbedding.load`/`QuantizedLinear.load` hard-require `.scales`.
-  The small-scale fold proof ran through stock mlx-lm instead.
-- `scripts/eval.ts` capability tasks at 27B swap-thrash a 32 GB box
-  (73 GB swapfiles, process 95% paged out) where the mlx-lm scoring path
-  is fine — investigate the in-engine eval memory profile.
-- Qwen MTP via the HTTP serve lane 500s with `[slice] Invalid number of
-  indices or strides for array with dimension 3` — REPRODUCES WITH STOCK
-  ARTIFACTS (plain-4bit trunk + original mlx-community MTP companion,
-  `serve --draft-model <companion> --draft-kind mtp`, any chat request),
-  so it is a pre-existing serving-path bug, not TQ. The 14g harness path
-  (specServeRun via scripts/experiments/qwen38-mtp-ab.ts) is the green
-  gate; TQ MTP was validated through that.
-
-## Non-goals
-
-Mirrors the PLAN.md phase: no custom weight format / new qmm kernels,
-no activation quant, no runtime weight rotation, no GGUF/AWQ export.
-
-## M4 Pro 24 GB speed row (2026-08-20, mlx-bun, winner artifact)
-
-Measured on the second reference machine (M4 Pro, 24 GB, ~273 GB/s) with
-`sysctl iogpu.wired_limit_mb=21504` and the DeltaNet leak fix (37de0d0):
-
-| prompt | prefill tok/s | TTFT | decode tok/s | notes |
-|---|---|---|---|---|
-| 1,024 | 127 | 8.1 s | 14.66 (1.1%) | chunk 2048 |
-| 8,192 | 124 | 66 s | 14.13 (1.3%) | chunk 2048 |
-| 32,768 | 114 | 288 s | 12.77 | chunk 1024 (peak 19.3 GB) |
-
-- **Compute-bound decode confirmed cross-machine:** 14.7 tok/s here vs
-  11.8 on the M1 Max (400 GB/s) — the machine with 68% of the bandwidth
-  is 25% FASTER. Decode is dequant-ALU-bound, not weight-streaming-bound
-  (supporting M1 telemetry: GPU 100% @ ~266 GB/s of 400). Kernel headroom
-  to the ~24 tok/s bandwidth roofline is a future perf program.
-- **Decode-at-depth is nearly flat** (−13% at 32× context): only 16 of 64
-  layers grow KV; DeltaNet state is constant.
-- **Prefill transient scales with chunk × offset** (~2.8 GB at 24k with
-  2048-chunks; measured via MLX_BUN_PREFILL_MEM_LOG): on 24 GB the 32k row
-  needs 1024-token chunks (−8% prefill). Engine follow-up: serve should
-  scale prefillChunkSize from fit.ts headroom automatically.
-- Prefill absolute rate (~125 tok/s) is the weak axis — suspected
-  DeltaNet per-token recurrence during prefill; compare against mlx-lm
-  same-box before treating as our bug (oracle-gap rule).
-
-### Compact sibling on the M4 (same protocol, chunk 1024 all rows)
-
-| prompt | prefill tok/s | TTFT | decode tok/s (spread) |
-|---|---|---|---|
-| 1,024 | 117 | 8.8 s | **17.14** (0.7%) |
-| 8,192 | 124 | 66 s | 16.05 (2.4%) |
-| 32,768 | 85 | 6.4 min | 13.80 (2.9%) |
-
-**The 3-bit speed verdict is GPU-generation-dependent:** on M1 Max the
-3-bit-heavy artifact decoded SLOWER than the 4/8 winner (10.94 vs 11.80 —
-unpack ALU dominates); on M4 Pro it decodes FASTER (17.14 vs 14.66 —
-newer ALU absorbs the unpack, fewer bytes win). Card guidance: M1-class →
-winner fastest; M4-class → compact is smaller AND faster. Compact 32k
-prefill (85 tok/s) is its weak axis. Peak not instrumented this run;
-estimated ~16 GB at 32k (likely fits the DEFAULT 24 GB wired limit —
-untested).