mlx-bun 0.0.8 → 0.0.10

package/README.md

@@ -1,10 +1,39 @@
 # mlx-bun
 
+**One command, chatting in under a minute** — a fully local AI on your
+Apple Silicon Mac's GPU, no Python, no API key:
+
+```sh
+curl -fsSL https://mlx-bun.dev/install.sh | sh
+mlx-bun
+```
+
+Docs: **[mlx-bun.dev](https://mlx-bun.dev)**
+
 Native MLX inference for Bun, and the beginning of a one-command local AI
 product for Apple Silicon Macs. `mlx-bun` detects the machine, starts a useful
 local assistant, exposes OpenAI/Anthropic-compatible APIs, and gives
 TypeScript apps direct access to MLX without Python or a sidecar server.
 
+Six goals, in one binary:
+
+1. **A drop-in replacement for `mlx_lm.server`** — same port, endpoints,
+   request fields, and flags; stop the Python server, start `mlx-bun serve`,
+   the same curl works ([site guide](https://mlx-bun.dev/guides/drop-in-mlx-lm/)).
+2. **A TypeScript library** — embed MLX generation in Bun/Tauri/Electron apps,
+   or ship it as a signed single-binary sidecar
+   ([library](./docs/reference/library-api.md), [embedding](./docs/reference/embedding.md)).
+3. **Bit-exact, fast, and memory-honest** — logit parity with the Python
+   reference as the correctness oracle, measured speed wins, and a
+   deterministic `fit` memory contract ([benchmarks](#benchmarks),
+   [correctness](#correctness)).
+4. **One command, under a minute** — the appliance path above; the default
+   experience needs zero configuration ([getting started](#getting-started)).
+5. **A personal memory** — a local, git-tracked wiki your assistant reads and
+   (via local synthesis) writes ([memory](#personal-memory)).
+6. **An AI lab** — training, speculative decoding, and sampling research kept
+   next to the runtime that serves them ([the lab](#the-lab)).
+
 The product bet is that local AI on Mac is a finite optimization problem, not
 an infinite configuration maze: Apple ships a known set of chip/RAM/bandwidth
 SKUs, the supported model list is curated, and the task mode is explicit. That
@@ -36,7 +65,7 @@ mlx-bun
 
 Self-contained (binary + MLX runtime) and notarized, so it runs without a
 Gatekeeper prompt. Installs to `~/.mlx-bun` by default — override with
-`MLX_BUN_INSTALL_DIR`, or pin a release with `MLX_BUN_VERSION=v0.0.4`.
+`MLX_BUN_INSTALL_DIR`, or pin a release with `MLX_BUN_VERSION=v0.0.8`.
 
 ### Homebrew
 
@@ -93,7 +122,7 @@ model named, it does everything for you:
 3. starts downloading `gemma-4-e4b` (the stronger 4B model) in the
    background — it becomes the default on your next `mlx-bun serve`;
 4. opens the chat UI in your browser at
-   [`http://localhost:8090/#/chat`](http://localhost:8090/#/chat)
+   [`http://localhost:8080/#/chat`](http://localhost:8080/#/chat)
    (pass `--no-open` to skip).
 
 Want a specific model instead? Name it — `mlx-bun serve e4b` (substring
@@ -105,7 +134,7 @@ your machine.
 **Prefer the API?** It's OpenAI-compatible — any OpenAI client works:
 
 ```sh
-curl http://localhost:8090/v1/chat/completions \
+curl http://localhost:8080/v1/chat/completions \
   -H 'Content-Type: application/json' \
   -d '{
     "messages": [{"role": "user", "content": "Hello!"}],
@@ -117,9 +146,13 @@ That's it. The server serves the one model it was started with (the
 request's `model` field is ignored and the loaded model id is echoed
 back in responses).
 
+Longer walkthrough on the site:
+[Installation](https://mlx-bun.dev/getting-started/installation/) and
+[Quickstart](https://mlx-bun.dev/getting-started/quickstart/).
+
 ## What mlx-bun is
 
-`mlx-bun` has four faces, all sharing one native runtime:
+`mlx-bun` has several faces, all sharing one native runtime:
 
 - **The local AI product** — run `mlx-bun` or `bunx mlx-bun` and get a
   working local chat UI plus a local server. The default path should feel like
@@ -128,25 +161,57 @@ back in responses).
 - **The app developer library** — embed local MLX inference inside Bun,
   Tauri, Electron, and other TypeScript applications without running Python in
   the background.
-- **The AI lab** — keep parity, evals, kernels, LoRA/ORPO training, adapter
-  routing, and performance experiments close to the runtime that will actually
-  serve them.
 - **The background agent runtime** — leave a local OpenAI-compatible endpoint
   available for scripts, recurring jobs, app integrations, and per-request
   adapter routing.
-- **The local memory layer** — `mlx-bun memory init` creates a git-tracked
-  Markdown wiki at `~/.mlx-bun/wiki`; `Reference/` is seeded with read-only
-  symlinks to mlx-bun's own docs, and the built-in pi agents read both those
-  docs and your articles quietly (`memory_search`, `memory_section`, etc.) and can open it in Obsidian for you
-  (`mlx-bun memory open [article]`). Chat-time memory is read-only; synthesis is
-  the separate writer path. The planned synthesis pipeline uses the same local
-  models with stage-specific LoRA adapters (chunk, bucket, synthesize, edit)
-  trained by the project creator and distributed from Hugging Face.
+- **The local memory layer** and **the AI lab** — each gets its own section
+  below.
 
 The default experience is intentionally opinionated. Power users can override
 models, budgets, KV modes, adapters, and flags; the average user should not
 need to know those knobs exist.
 
+## Personal memory
+
+`mlx-bun memory init` creates a git-tracked Markdown wiki at
+`~/.mlx-bun/wiki`; `Reference/` is seeded with read-only symlinks to
+mlx-bun's own docs, and the built-in pi agents read both those docs and your
+articles quietly (`memory_search`, `memory_section`, etc.) and can open it in
+Obsidian for you (`mlx-bun memory open [article]`). Chat-time memory is
+read-only; synthesis is the separate writer path: `mlx-bun memory synthesize`
+runs the built-in local pipeline (segment → extract → route → create/patch →
+reconcile → link → wikify) that turns your conversations into cross-linked
+subject articles, using the same local models with a fine-tuned chunking LoRA
+adapter.
+
+It's your data in your format: plain Markdown, git history, editable in any
+tool, never leaves the machine. Reference:
+[docs/reference/memory.md](./docs/reference/memory.md); synthesis design:
+[docs/design/memory-synthesis.md](./docs/design/memory-synthesis.md); on the
+site: [Personal memory](https://mlx-bun.dev/guides/memory/).
+
+## The lab
+
+Parity, evals, kernels, training, and sampling research live next to the
+runtime that serves them:
+
+- **LoRA fine-tuning on your Mac** (SFT / DPO / ORPO) — `mlx-bun train
+  <model> --data <dir>`, with a flash-CCE Metal head, segmented backward, and
+  prefix-sharing so long-context preference training fits laptop memory;
+  watch runs live with `mlx-bun train-watch`
+  ([training](./docs/reference/training.md),
+  [quickstart](./docs/reference/orpo-quickstart.md)).
+- **The curve designer** — HLG tone-curve sampling with an interactive
+  designer in the server UI at `/curves`
+  ([design](./docs/design/hlg-sampling.md)).
+- **Speculative decoding research (DSpark)** — a KV-injection drafter
+  ([design](./docs/design/dspark-speculative-decoding.md)).
+- The full research trail — including dead ends — is tracked in
+  [docs/design/](./docs/design/) and
+  [docs/investigations/](./docs/investigations/).
+
+On the site: [The lab](https://mlx-bun.dev/about/lab/).
+
 ## Supported models
 
 Scope is deliberate: a few model families held to **bit-exact** logit
@@ -164,7 +229,9 @@ Currently MiniCPM5, the Gemma-4 OptiQ quants, and Qwen3.5-4B:
 Not sure what fits your machine? `bun src/cli.ts fit <model> --ctx 8192`
 gives a deterministic answer (see below). The larger
 `Qwen3.6-27B-OptiQ-4bit` is still in bring-up — parity and serving polish
-remain (see [PLAN.md](./PLAN.md)).
+remain (see [PLAN.md](./PLAN.md)). Downloading, cache layout, and reclaiming
+disk: [docs/reference/models.md](./docs/reference/models.md); on the site:
+[Choosing a model](https://mlx-bun.dev/getting-started/models/).
 
 ## Why
 
@@ -204,8 +271,12 @@ bun src/cli.ts ls                   # list models (size, params, quant, capabili
 bun src/cli.ts ls --vision --max-size 10GB
 bun src/cli.ts fit gemma --ctx 32768          # memory contract: fits? max context? predicted tok/s
 bun src/cli.ts fit gemma --ctx 8192 --skus    # ...same, across the Apple Silicon lineup
-bun src/cli.ts serve gemma --port 8090        # OpenAI-compatible server
+bun src/cli.ts serve gemma --port 8080        # OpenAI-compatible server
 bun src/cli.ts serve gemma --memory-budget 18 # ...with admission control (GB)
+bun src/cli.ts generate gemma "a haiku about metal shaders"   # raw one-shot generation
+bun src/cli.ts train e4b --data ./pref-data --method orpo     # LoRA fine-tune (SFT/DPO/ORPO)
+bun src/cli.ts memory init          # git-tracked Markdown wiki at ~/.mlx-bun/wiki
+bun src/cli.ts memory synthesize    # conversations -> cross-linked wiki articles, all local
 bun src/cli.ts pi                   # built-in agentic coding CLI (pi's TUI, in-process)
 bun src/cli.ts evals                # recorded benchmark runs
 bun src/cli.ts harness pi           # connect your own pi install to the local server
@@ -223,11 +294,14 @@ MoE active-expert bytes for decode), calibrated prefill transient, and
 the machine's wired-memory ceiling. Predictions are recorded next to
 measured peaks in the eval DB.
 
+Every verb, with flags: [docs/reference/cli.md](./docs/reference/cli.md);
+on the site: [CLI reference](https://mlx-bun.dev/reference/cli/).
+
 ## HTTP API
 
 `serve` exposes an OpenAI-compatible API. Anything that speaks the
 OpenAI protocol can point at it — the OpenAI SDK
-(`baseURL: "http://localhost:8090/v1"`, any non-empty `apiKey`), or
+(`baseURL: "http://localhost:8080/v1"`, any non-empty `apiKey`), or
 agent CLIs like pi/OpenClaw via their provider config.
 
 - **`POST /v1/chat/completions`** — streaming (SSE) and non-streaming;
@@ -278,21 +352,30 @@ agent CLIs like pi/OpenClaw via their provider config.
   up front is the only defense. Ceiling visible at `GET /stats`.
 - **Anthropic Messages API** (`POST /v1/messages`, on by default) —
   point any Anthropic-SDK tool at the server
-  (`ANTHROPIC_BASE_URL=http://localhost:8090`); Claude Code runs
+  (`ANTHROPIC_BASE_URL=http://localhost:8080`); Claude Code runs
   against it as a fully local backend. Streaming event grammar, native
   tool_use/tool_result mapping, image blocks via the vision path.
 - **OpenAI Responses API** (`POST /v1/responses`) — the protocol
   Codex/Cursor/Continue speak; includes `previous_response_id`
   resumption backed by a TTL + byte-capped store (visible in /stats).
+- **OpenAI Embeddings API** (`POST /v1/embeddings`) — when the served
+  model is an embedding model (`mlx-bun serve Qwen3-Embedding-4B-4bit-DWQ`):
+  last-token pooled, L2-normalized vectors, bit-exact vs mlx-lm. Also a
+  one-shot CLI: `mlx-bun embed <model> --text "…"`.
 - **`GET /v1/models`**, **`GET /stats`** (cache hit rates, bytes,
   active KV scheme, response store), **`GET/POST/DELETE /v1/adapters`**,
   plus **`GET /library`**, **`GET /fit`**, **`GET /downloads`**, and
   **`GET /v1`** (a self-describing index of every endpoint).
 
+Guide with examples on the site:
+[The HTTP API](https://mlx-bun.dev/guides/http-api/); coming from
+`mlx_lm.server`:
+[Drop-in for mlx-lm](https://mlx-bun.dev/guides/drop-in-mlx-lm/).
+
 ## Library
 
 The server is one consumer of a library-first API. Published to npm as
-`mlx-bun` (current: 0.0.4) — import from the package (`from "mlx-bun"`)
+`mlx-bun` (current: 0.0.8) — import from the package (`from "mlx-bun"`)
 or `./src/index` in a clone; `bunx mlx-bun` runs the CLI without
 installing. Full reference:
 [docs/reference/library-api.md](./docs/reference/library-api.md). For shipping inside a
@@ -326,6 +409,9 @@ KV caches can be persisted to disk (page-aligned files that reload as
 zero-copy GPU-safe mmaps — see `src/kv-store.ts`) so a standard agent
 preamble prefills once, ever.
 
+On the site: [Using the library](https://mlx-bun.dev/guides/library/) and
+[Embedding in a Mac app](https://mlx-bun.dev/guides/embedding/).
+
 ## Benchmarks
 
 Head-to-head against the Python stacks (mlx-lm 0.31.3, mlx-optiq 0.2.1),
@@ -368,6 +454,9 @@ both served it from the same machine state). One further optiq cell
 (12B/kv=config) is blocked on an upstream `quantized_matmul` bug. Both
 are documented in the results file.
 
+On the site (with the honest negatives spelled out):
+[Benchmarks](https://mlx-bun.dev/about/benchmarks/).
+
 ## Correctness
 
 Logit parity with mlx-lm (same weights, Python reference) is the
@@ -387,6 +476,9 @@ bun test    # fast tier runs everywhere; model-loaded tests run only
             # when the reference snapshot is in your HF cache
 ```
 
+On the site: [Correctness](https://mlx-bun.dev/about/correctness/) and
+[How it compares](https://mlx-bun.dev/about/comparison/).
+
 ## Troubleshooting
 
 - **`dlopen` / `libmlxc.dylib` not found** — the MLX runtime

package/docs/design/batching-perf-path.md

@@ -0,0 +1,232 @@
+# Batching perf path — promote continuous batching out of compat mode
+
+Status: P5 + two gate fixes LANDED (2026-07-02, same day); rest PLANNED.
+
+**Headline result (M1 Max 32 GB, loaded machine, 4 concurrent × 128 tok,
+median of 3, wall-clock aggregate; oMLX 0.4.5.dev1, same OptiQ snapshots):**
+
+| Model | oMLX | mlx-bun `--batch 4` | mean TTFT (theirs/ours) |
+| --- | --- | --- | --- |
+| MiniCPM5-1B | 339.0 | **345.4** | 397 / **119 ms** |
+| gemma-4-e4b | **89.8** | 87.1 (−3%) | 815 / **397 ms** |
+| Qwen3.5-4B | **100.9** | 99.5 (−1%) | 848 / **368 ms** |
+
+What landed (all bit-gated, `tsc` 0, suites green):
+1. **P5 — SSM batched path** (`SSMCache.mergeRows`/`filter` in
+   `src/model/qwen3-delta.ts`; "ssm" kind in `batch-scheduler.ts` merge/wrap/
+   filter; per-row RoPE via `ropeOffsetArr` in `qwen3_5.ts`; gate relaxed in
+   `generation-gateway.ts`, kill switch `MLX_BUN_BATCH_SSM=0`). Oracle:
+   `tests/qwen35-batched-parity.test.ts` — solo-prefill+merge batched greedy
+   is **token-for-token identical** to mlx-lm B=2 left-padded batch
+   (`goldens/apple-m1-max/batched-golden-qwen35.json`). No ssm_mask needed:
+   rows solo-prefill unpadded, decode feeds one real token per row.
+2. **Logits processors batch** (`generation-gateway.ts`): the per-row sampler
+   folds `makeLogitsProcessors` over a per-row device-side token history
+   (generate()'s pushHistory), so repetition/presence/frequency penalties +
+   logit_bias + min_p/XTC no longer route serial. Load-bearing discovery:
+   **Qwen3.5 ships a default repetition penalty in generation_config.json**,
+   which routed EVERY request serial under `--batch N` — the SSM port alone
+   changed nothing over HTTP until this landed.
+3. Diagnostic breadcrumb: `MLX_BUN_LANE_DEBUG=1` logs per-request lane
+   decisions + shape (server.ts) — how (2) was found.
+
+Bench-method traps found while measuring — **both FIXED 2026-07-02**:
+- `serve`/`bench` accepted `--model` but IGNORED it (positional-only fuzzy
+  query; auto-pick silently loaded the wrong model). Now: `--model
+  <path-or-query>` is an explicit override — a directory with config.json
+  loads via `scanSnapshot` (no registry dependency), anything else resolves
+  like the positional. Precedence: `--model` > positional > `--query` >
+  auto-pick.
+- Event-loop starvation during generation was the SERIAL lane, not batch
+  (batch's drive loop yields per step — `/stats` answered in ~8 ms mid-
+  generation once measured properly): serial decode is an unbroken microtask
+  chain, so `/stats`//health/new accepts stalled for the whole generation
+  (measured 2.5 s on a 512-token run). Fixed with a ≥25 ms rate-limited
+  macrotask hop wrapped around onToken in `GenerationGateway.run`'s serial
+  branch — covers all four API surfaces, streaming and not. After: `/stats`
+  10–44 ms mid-generation, decode unchanged (cpm5 serial 193 tok/s). Motivated by the oMLX head-to-head (M1 Max,
+2026-07-02): at 4 concurrent requests oMLX's continuous batching gets
+1.46–1.89× our aggregate throughput (Qwen3.5-4B 97.3 vs 66.5 tok/s;
+MiniCPM5 328 vs 173; e4b 85 vs 52) and our mean TTFT queue-inflates to
+1.1–3.8 s. Single-stream we match or beat them (e4b 55.8 vs 53.7,
+Qwen3.5 66.3 vs 63.9, TTFT 3–7× better everywhere); the gap is entirely
+the serial lane going flat under parallel load. Full comparison + numbers:
+the 2026-07-02 oMLX report (Claude artifact) and `~/.omlx` bench scripts.
+
+Design stance: the batch lane grows the **same parity-tier DAG as the
+serial lane** (docs/design/parity-tier-dag.md). `--l1 --batch N` stays
+bit-exact mlx-lm-B=N compat mode forever (the oracle checkpoint); the
+default/`--l3` batch path becomes the optimized one. `--batch` remains a
+mode switch, not a load fallback.
+
+## 1. Current state
+
+`src/serve/batch-scheduler.ts` (`BatchScheduler`) is a real
+continuous-batching engine: iteration-level scheduling, pipelined decode
+(`#step` via `ops.asyncEvalAll`, mlx-lm `GenerationBatch._step`
+semantics), chunked joiner prefill (2048-token chunks) interleaved with
+decode, per-row eviction (`filterKVRows` / `BatchedRotatingCache.filter`),
+per-row failure containment, drain-on-serial-waiter, `clearCache` every
+256 steps. Bit-parity with mlx-lm B=N verified at B=2 across all 4 model
+paths (`tests/batched-decode-parity.test.ts`, `tests/batched-rotating.test.ts`,
+`tests/batch-scheduler.test.ts`).
+
+`src/serve/generation-gateway.ts` (`GenerationGateway.willBatch`) routes;
+one `AsyncMutex` is the GPU/`loraState` exclusion domain. The batch lane
+is deliberately compat mode: it calls `model.forwardHidden` /
+`logitsFromHidden` directly, bypassing `generate()` and every perf lever.
+
+### Levers the batch lane lacks, and why
+
+| Lever | Serial mechanism | Why batch can't use it today |
+| --- | --- | --- |
+| Quantized KV (`kv_config`) | `maybeQuantizeKv` (`src/generate.ts:101`) | `QuantizedKVCache` (`src/model/gemma4-base.ts:419`) has no dynamic-B ops (no merge/filter/extend/temporalView over the packed/scales/biases triples); `willBatch` hard-gates `kvQuant` → serial |
+| Perf-kernel fused decode SDPA | dispatched from `quantizedSdpa` (`gemma4-base.ts:1560`) | kernel M=1-specialized: `fused-decode-kernel.ts:262` rejects `B !== 1 \|\| L !== 1`, and requires empty mask (batched decode needs per-row left-pad validity) |
+| Fused N-tiled SDPA (`--fused-decode`) | `quantizedSdpaTiled` | needs 2-D mask; batched masks are 4-D `[B,1,1,S]` (`buildBatchedDecodeMask`); quantized-only anyway |
+| Compiled decode | `CompiledDecode.for()` (`generate.ts:518-525`) | trace is `[1,1]`; B changes every join/evict |
+| Prompt-cache prefix reuse | `PromptCache.take/put` in `runGeneration` | bypassed by design in v1; every batched row solo-prefills (`cachedTokens: 0`) |
+| Vectorized sampling | n/a | `#step` samples per row (B slice+sample+concat graphs per token) |
+
+Two more gap sources specific to the benchmark:
+
+- **Qwen3.5 never batches at all**: `SSMCache` (`src/model/qwen3-delta.ts`)
+  has no merge/filter, so the capability gate routes every request serial.
+  The Qwen3.5 concurrency loss is pure serial queueing. mlx-lm batches
+  Mamba-family state via `ArraysCache` — oracle exists.
+- Joins re-merge the whole batch (O(B·S) slice storm, `#mergeJoiner`)
+  instead of mlx-lm's `extend`.
+
+## 2. Gap analysis: quantized KV + fused kernels at B>1
+
+- `QuantizedKVCache` already stores `[B,H,S,D']` triples; missing is
+  dynamic-B surgery. New `src/model/batched-quant.ts`:
+  `BatchedQuantizedKVCache` = quantized twin of `BatchedRotatingCache`
+  (merge/extend/filter/extract/temporalView = the `mergeKVRows`/`filterKVRows`
+  B-axis ops from `src/model/batched-mask.ts:171,196`, applied to all three
+  triples). Left-pad must align to quantization group boundaries — pad in
+  whole 256-token STEP granules (both caches already grow in STEP=256).
+- Conversion point mirrors `maybeQuantizeKv`: row solo-prefills bf16,
+  `toQuantized` per layer per `kv_config` before merge.
+- **Scope cut**: shipped `kv_config.json` quantize full-attention layers
+  only ⇒ P1 = quantized full layers + bf16 `BatchedRotatingCache` sliding
+  layers (the serial `config` composition). Batched rotating-quant deferred.
+- Correctness fallback: `quantizedSdpaUnfused` (`gemma4-base.ts:1273`) is
+  already B-generic with `mask.mode === "array"`; needs the batched mask
+  broadcast as `[B,1,1,1,N]` for the 5-D GQA score reshape.
+- Perf kernel at B>1: (1) drop the `B !== 1` gate (keep `L !== 1`);
+  (2) grid gains a B axis (`[TG_THREADS, H·B, G]`), `SCORE_ROW` offsets
+  index `((b·KV + kv)·N + n)`; (3) replace the empty-mask requirement with
+  a per-row `leftPad` int32 `[B]` input — each row's n-loop starts at
+  `max(blockStart, leftPad[b])`, so pad bytes are never read.
+- RoPE per-row positions already solved (`ropeOffsetArr`, the S1b.1 fix).
+
+## 3. Phases
+
+### P0 — Baseline + engine quick wins (no new numerics)  [S–M, 2–4 days]
+- Write `scripts/bench-serving-load.ts` (N concurrent vs live server,
+  aggregate tok/s + TTFT p50/p95). Re-measure vs oMLX **with `--batch 4`
+  actually on** to size the true engine gap vs the serial-queueing gap.
+- Land `extend` join (full-attention twin + `BatchedRotatingCache.extend`);
+  oracle: extend `scripts/gen-batched-dynamic-golden.py` (mlx-lm `extend`).
+- Vectorize the homogeneous-sampler fast path in `#step` (all-greedy →
+  one `ops.argmaxAxis` on `[B,V]`); mixed rows keep the loop.
+- Exit: gated suites unchanged; B=4 baseline recorded; joins not O(B·S).
+
+### P1 — Quantized KV at B>1  [M–L, 1–1.5 wk]
+- `src/model/batched-quant.ts` + scheduler wiring (layer kind
+  `"full" | "rot" | "qfull"`); solo-prefill bf16 → `toQuantized` → extend.
+- Batched mask 5-D broadcast; validate `quantizedSdpaUnfused`.
+- `willBatch`: `kvQuant` stops being a serial gate when quant layers are
+  full-attention-only.
+- Exit (parity ladder; no external oracle — mlx-lm quant batching NYI):
+  (1) B=1-row batched-quant **bit-exact vs serial `--kv-quant config`**;
+  (2) row-isolation within the calibrated bf16-reduction envelope;
+  (3) KL vs batched-bf16 ≤ serial quant-vs-bf16 KL; 6-task eval unchanged;
+  (4) `--l1 --batch N` suites untouched.
+
+### P2 — Perf kernel at B>1 + fallback matrix  [M, ~1 wk]
+- Kernel edits per §2; dispatch when every row supported, else unfused.
+- Fallback matrix documented in server-config.md.
+- Batched frozen-oracle envelope (B=2/4 `freeze-perf-oracle` variant,
+  ≥56/64 per row, `tests/perf-kernel-oracle.test.ts` pattern).
+- Exit: dispatch counter proves engagement; envelope green; measured
+  uplift at B=4 vs P1 unfused on a clean machine; if <5%, ship the
+  fallback matrix only and close honestly.
+
+### P3 — Admission, prompt cache, adapters, defaults  [M–L, 1–1.5 wk]
+- KV-budget admission: `projectKvBytes(model, promptLen, maxTokens)`
+  beside `fit()`; `--kv-budget`, `--prompt-concurrency`;
+  `/stats.batch.{pending_rows,kv_bytes,kv_budget_bytes}`.
+- Prompt-cache reuse under batching: `take` at admit, forward uncached
+  tail; on finish/evict `extract(row)` → solo caches → `put`. Quantized
+  entries are non-trimmable mid-group — match on full-prefix or STEP
+  boundaries. Real `prefillMs/decodeMs`; `BatchStats.cachedTokens` real.
+- Single-adapter batches: adapter-spec = batch compatibility key
+  (join if match, drain if not; `loraState.active` set once per epoch).
+- Default review: if batch lane at 1 live row is within ~2% of serial,
+  propose flipping the shipped default to `--batch 4` (Josh decision,
+  numerics change vs serial default); else README recommends `--batch 4`
+  for agent workloads.
+- Exit: TTFT p50 <500 ms / p95 <1.5 s at 4 concurrent short prompts;
+  `cached_tokens>0` on repeats under `--batch 2`; oversized-request
+  queueing test (no OOM); B=4 aggregate within 15% of oMLX on cpm5 + e4b.
+
+### P4 — Burst decode  [REVISED after reading + porting oMLX's source]
+- **oMLX-style event-loop bursting: BUILT AND REFUTED (2026-07-02).** Read
+  their actual implementation (`omlx/engine_core.py _step_burst` — full
+  source ships in `/Applications/oMLX.app/Contents/Resources/omlx/`): it is
+  NOT a GPU trick — it runs K `scheduler.step()` calls per
+  `run_in_executor` hand-off with adaptive time budgets (100 ms single /
+  30 ms concurrent, 64-step cap) because each per-token hand-off ping-pongs
+  the GIL with asyncio/uvicorn (~1 ms/token; their comment: 80 vs 74 tok/s).
+  Ported faithfully to `BatchScheduler.#drive` and measured: **regression**
+  — cpm5 B=4 345→289, batch-lane B=1 149→121, TTFT +~100 ms (SSE flushes
+  wait out the budget). Bun's `setImmediate` hop costs microseconds — no
+  GIL to dodge — so the port kept the cost and lost the win. Reverted;
+  numbers restored (cpm5 B=4 349.1). Breadcrumb comment left at the drive
+  loop. This also explains their ~21-token SSE bursts (measurement trap in
+  the comparison report).
+- **Still open (different mechanism, not refuted): device-side step
+  chaining** — generalize `generateInner`'s pipeline to depth k (k chained
+  step graphs fed device-side, one host sync per k tokens). That attacks
+  OUR real per-token cost (FFI graph-build + sync readback), which is not
+  what oMLX's burst attacks. Exit unchanged: bit-exact vs depth-1, cpm5
+  single-stream gap vs oMLX (178 vs ~214) is the target; no regression at
+  k=1.
+
+### P5 — SSM batched path (Qwen3.5)  [DONE 2026-07-02 — see header]
+- Landed in one day, much cheaper than the ~1 wk estimate: `ssm_mask` under
+  left-padding turned out unnecessary (our join model solo-prefills unpadded,
+  so SSM state never sees a pad token; mlx-lm's masked-pad prefill is
+  equivalent — proven by the token-exact oracle match). Batch ops are plain
+  B-axis concat/take on the two state slots.
+- Bonus fix that the exit benchmark forced out: logits processors now batch
+  (per-row fold; Qwen3.5's default repetition penalty had been routing every
+  request serial). Exit numbers in the header table.
+
+Ordering: P1→P2 strictly ordered; P4 and P5 independent (P5 first if the
+Qwen3.5 headline matters most). Total ~5–6 wk serial, ~4 wk with P4/P5
+parallel.
+
+## 4. Risks
+
+- **No external oracle for quant-batched** — mitigated by the P1 parity
+  ladder; `--l1 --batch N` stays the always-reachable bit-exact checkpoint.
+- **Metal kernel at B>1**: leftPad start-index never touches pad bytes;
+  no atomics (deterministic); gate = batched frozen-oracle envelope +
+  dispatch assertions + A/B vs unfused.
+- **Memory**: B× quant caches + prompt cache + weights on 24 GB — P3
+  admission is the backstop; mirror mlx-lm's
+  `trim_to(total − activeBatchBytes)` so LRU and live batch share one
+  ceiling. GPU OOM is uncatchable — queueing test before default changes.
+- **Ring-wrap at B>1 short-context-verified only**: land a long-context
+  bf16 batched golden before any P3 default flip. Quantized rotating out
+  of scope.
+- **Default flip changes numerics** vs today's serial default — explicit
+  Josh decision with doc trail.
+- **Benchmark hygiene**: quotable numbers via `bench-serving-load.ts` on a
+  clean machine; TTFT at the SSE first-token event, not server-side
+  (oMLX's own server log over-reports by excluding ~180 ms/req cache
+  bookkeeping — measure them wall-clock only).
+- **Regression fence**: all existing gated batch suites + the serial suite
+  green unchanged at every phase.