mlx-bun 0.0.9 → 0.0.11

package/docs/planning/release-notes-v0.0.10.md

@@ -0,0 +1,57 @@
+# mlx-bun v0.0.10 — batching parity, SSD KV cache, sharper serving
+
+## Highlights
+
+**Concurrent serving now matches the fastest MLX batching servers.** With
+`--batch 4`, aggregate throughput at 4 parallel requests is at parity with
+(or ahead of) oMLX on every shared model we benchmarked on an M1 Max —
+MiniCPM5-1B 349 vs 339 tok/s, gemma-4-e4b within 3%, Qwen3.5-4B within 1% —
+with 2–3× better time-to-first-token under load, and per-token SSE
+streaming (no burst batching of your tokens).
+
+- **Qwen3.5 joins the batch lane.** Hybrid gated-DeltaNet (SSM) caches grew
+  the dynamic-batching ops; the batched decode path is verified
+  token-for-token identical against mlx-lm's B=2 batched oracle.
+- **Sampler features batch too.** Repetition/presence/frequency penalties,
+  `logit_bias`, `min_p`, and XTC now run per-row inside the batch instead
+  of forcing requests onto the serial path. (Qwen3.5 ships a default
+  repetition penalty — before this fix it silently serialized every
+  request under `--batch N`.)
+
+**SSD cold tier for the prompt cache** (`--ssd-cache <dir>`). Long-context
+prefix KV now spills to disk on eviction, is snapshotted after requests
+settle, and **survives server restarts**: re-attaching a 13.7k-token
+coding-agent conversation went from a 12.1 s full re-prefill to a 0.24 s
+zero-copy mmap restore in our measurements — with 0% steady-state decode
+overhead (nothing runs on the token loop). Entries are keyed by model
+fingerprint + KV scheme + tokenizer hash + adapter namespace; corrupt or
+incompatible files self-quarantine; the tier is always safe to delete.
+Companion flags: `--ssd-cache-max <GB>` (default 32), `--ssd-cache-verify`.
+
+## Fixes
+
+- `serve`/`benchmark` now honor `--model <path-or-query>` as an explicit
+  override (it was previously accepted but ignored — auto-pick could
+  silently load a different model). A directory containing `config.json`
+  loads directly from that path.
+- The server stays responsive during long serial generations: `/stats`,
+  `/health`, and new connections previously stalled until the generation
+  finished (measured 2.5 s on a 512-token run); they now answer in tens of
+  milliseconds, with no measurable decode cost.
+
+## Internals
+
+- KV persistence format v2: all cache kinds round-trip (bf16, per-layer
+  quantized, sliding-window, quantized sliding-window, SSM), with
+  integrity hashes, invalidation metadata, and atomic writes.
+- Golden `.bin` parity fixtures are no longer tracked in git (they are
+  machine-specific and regenerable); JSON manifests remain.
+- New design docs: `docs/design/omlx-adoption-map.md` (feature adoption
+  scoreboard), `docs/design/batching-perf-path.md`,
+  `docs/design/ssd-kv-cold-tier.md`.
+
+Full details and measured numbers: STATUS.md and the design docs above.
+Credit where due: the SSD-tier and batching comparisons were driven by
+studying [oMLX](https://github.com/jundot/omlx) (Apache 2.0) — a strong
+project whose ideas we benchmarked and adapted where they made mlx-bun
+faster.

package/docs/planning/release-notes-v0.0.11.md

@@ -0,0 +1,239 @@
+# v0.0.11 release notes
+
+Everything since v0.0.10. The big themes: mlx-bun grows a second modality
+(audio), a constrained-decoding stack (structured output / grammar),
+speculative decoding, a novel KV-cache quantization scheme (TurboQuant),
+completed dynamic batching, crash-isolated serving, and a ground-up
+rewrite of the web chat UI. The decode-default story also resets: after
+benchmarking every custom performance kernel in a paired A/B against the
+bit-exact baseline and finding none of them won, they were deleted —
+naked `mlx-bun serve` now runs the faithful, mlx-lm-parity kernel set by
+default.
+
+## Audio input (Gemma e4b)
+
+- OpenAI-style `input_audio` parts are served end-to-end: WAV/AAC
+  transcode, a USM mel-spectrogram extractor, and a Conformer tower
+  ported bit-exact (rel-RMSE 2.4e-8) against the oracle.
+- Build-out ran phase-by-phase (A0–A5): a `bun:ffi` conv2d binding for
+  the front-end, the multimodal prompt builder wired to the LM, and
+  capability discovery for audio surfaced consistently across
+  `/v1/models`, `/library`, and the pi websocket handshake — mirroring
+  how vision capability is advertised.
+
+## Structured output / grammar-constrained generation
+
+- `response_format` (`json_object`, `json_schema`, `text`) and
+  `guided_grammar`/`guided_regex`/`guided_choice`/`structured_outputs`
+  are wired into both `/v1/chat/completions` and `/v1/completions`, built
+  on `xgrammar`: a `GrammarController` masks logits inside the decode
+  loop, with a per-`TokenizerInfo` compiler cache and a `MLX_BUN_GRAMMAR=0`
+  kill switch.
+- The batched lane got its own per-row matcher path so grammar-constrained
+  requests don't force a request onto the serial lane, and grammar
+  composes with `--draft-model` (the verify walk respects the mask).
+- Two defects found and fixed during the build: the oMLX-parity
+  JSON-system-prompt degrade path (used when a client can't do function
+  calling) had gone unreachable behind a dropped hint, and the WASM
+  grammar controllers were leaking on every early-400 reject path
+  (SSRF-blocked media, vision/audio validation failures, bad
+  `logit_bias`, etc.) — both fixed, with disposal now covering all paths.
+
+## Speculative decoding
+
+- `--draft-model` lands in serve: a two-model, serial-lane speculative
+  decoding path verified token-for-token against mlx-lm's own
+  `--draft-model` oracle (`DraftSource` seam, rewind rule, batched-verify
+  lm-head).
+- An ngram / jump-forward draft kind ships alongside the model-based
+  drafter for cases with no small model available.
+- A DeepSpec (DSpark) drafter-quant research track is underway
+  (Phase 1 landed) targeting the larger models; per
+  `benchmarks/RESULTS.md`, the first live pair (12B target +
+  bf16 DeepSpec drafter, 26–33% acceptance) measured net **slower**
+  than serial on a loaded box — spec decoding stays **off by default**
+  pending a quantized-drafter pair that clears the 1.3× bar.
+
+## TurboQuant KV-cache quantization
+
+- `--kv-quant turbo[:kXvY]` adds a rotation-based per-group KV
+  quantization scheme (sign-FWHT + Lloyd-Max for values), proven
+  bit-exact against the vendored vllm-metal reference
+  (`goldens/turboquant.json`).
+- Per `benchmarks/RESULTS.md`, the k8v3 default holds a lower KL-vs-bf16
+  than affine uniform kv4 while compressing KV **2.56×** (measured on
+  MiniCPM5-1B, paired quality/KL harness). It's a memory/context lever,
+  not a speed lever — v1 dequant-on-fetch is slower per decode step at
+  long context, so it stays **opt-in, not default**, per the project's
+  "levers must earn defaults" rule.
+
+## Batching: Phase D completed
+
+- Aggregate `--kv-budget <GB>` admission with FIFO queuing, vectorized
+  homogeneous sampling (penalties/`logit_bias`/`min_p`/XTC all run
+  per-row in the batch), and extend-join so a request can join an
+  in-flight batch mid-flight instead of waiting for a full drain.
+- Per-row RoPE lands for Tier-0 universal architectures, lifting the
+  batching gate for those models (e.g. Llama).
+- Batched mixed-precision quantized KV — the first stack to compose
+  per-row KV quantization with batching.
+- `--batch` default changes from 1 to 8 (see Breaking changes below).
+
+## Runtime isolation
+
+- `--isolate` runs the engine as a crash-isolated child process behind a
+  reactor parent, restarting with backoff on a crash — the product rule
+  is **"the AI may crash, the UI never may"** (`docs/design/runtime-isolation.md`).
+- `--model-pool` extends this to a child-per-model pool: LRU residency,
+  spawn-overlap switching (no dead time swapping models), and
+  `POST /admin/drain` for graceful eviction.
+
+## Optional paged KV cache
+
+- `--paged-kv` adds a vLLM-style block-table paged KV cache
+  (`PagedKVCache` + `BlockPool`) feeding the unchanged stock SDPA via
+  block gather. Default off; v1 scope is serial `--batch 1`, the
+  Gemma4 family, bf16 — it refuses to silently combine with
+  `--batch N>1`, `--kv-quant`, or `--draft-model`.
+
+## Prompt cache overhaul
+
+- Prefix-sharing v1: `take()` now returns non-consuming, ref-counted
+  clones instead of consuming/trimming the donor entry, so multiple
+  agents or sessions can reuse one prefill without cannibalizing each
+  other's cache.
+- The SSD cold tier is promoted from an add-on to a first-class property
+  of the cache store, with `--ssd-demote-idle <seconds>` (default 300,
+  requires `--ssd-cache`) freeing GPU memory by demoting idle entries;
+  `/stats` gains `ssd_cache.demotions`.
+- A stable-boundary snapshot fix restores prompt-cache hits for
+  multi-turn agent traffic beyond the sliding window — this was
+  previously a silent cache miss on turn 2+ of long conversations.
+- The SSD write-behind flush is now idle-gated, so background disk
+  writes stop contending with decode at long context; the earlier A7
+  RSS regression on the ssd-cache path closed with zero-copy write views
+  and a streamed copy-restore.
+
+## Web chat UI: ground-up redesign (Phases 0–3)
+
+- Streaming render rewritten to be block-memoized (no more O(n²)
+  re-parse of the growing message), with a typed module split.
+- New surfaces: a memory panel, adapter routing, a system-prompt editor,
+  an approval gate, client-side RAG over chat history (BM25, no server
+  round-trip), a Model Hub, Canvas, per-message sampling controls,
+  self-healing tool calls, a command palette, full-text search, and
+  chat export.
+- An app-aware assistant (v1) that can see and navigate its own UI —
+  spotlight/selector resolution now treats a model-invented or malformed
+  selector as a miss instead of throwing inside the websocket handler.
+- PWA support, plus a round of visual-QA fixes: the phone nav band,
+  light-theme code legibility, a composer placeholder regression, memory
+  panel markdown chrome, the theme-toggle Auto button, a clipped status
+  pill, and tab flex-shrink pinned to zero on phones.
+
+## Security: media-fetch SSRF hardening
+
+- `image_url`/`audio_url` fetches now refuse private/loopback/link-local/
+  CGNAT destinations and non-`http(s)` schemes by default, re-validate
+  through up to 5 redirect hops, enforce a 10s wall-clock timeout, and
+  cap bodies at 64MB.
+- LAN use needs the new escape hatch: `--allow-private-media` /
+  `MLX_BUN_ALLOW_PRIVATE_MEDIA=1`.
+
+## Fixed
+
+- 12B `/v1/completions` probe parity: adopted the oracle's step-0
+  prefill convention (drain to len−1, step 0 from an `L=1` forward of the
+  last prompt token). Per `benchmarks/RESULTS.md`, live HTTP probes are
+  now byte-identical to `mlx_lm.server`/optiq serve for MiniCPM5-1B,
+  gemma-4-e4b, and gemma-4-12B, on completion and chat probes, in both
+  the `--batch 8` default lane and `--batch 1`.
+- BPE detokenization now mirrors mlx-lm's bare-space hold-back and
+  never-finalize semantics exactly.
+- Cache invariant enforced on every tier; an FFI destructor deadlock
+  fixed; write-behind made non-blocking; per-row batch cache extraction
+  fixed.
+- `generate()` now yields a macrotask between prefill chunks, keeping the
+  isolation path (and the server generally) responsive during long
+  prefills.
+- Benchmark harness: the `--batch 1` serial control arm now runs by
+  default (it had been skippable, which undermined validating the new
+  batch-8 default); `--engine` is additive instead of mutually exclusive
+  with the serve pass; the batch-path test now asserts cumulative
+  `/stats` `submitted_rows` instead of a stale `cachedTokens=0` premise.
+- `any_whitespace` grammar knob landed; batch-grammar gates re-anchored
+  to compact separators.
+
+## Infra / benchmark harness redesign
+
+- `benchmark.sh` + `scripts/bench-modes.ts` now spawn real CLI paths
+  instead of bench-local server wrappers, folding bit-parity and
+  peak-memory checks into the same serve cells (one pass instead of
+  several). New columns: cold-start and restart-survival; new workload:
+  `xl` (~4k prefix); a feature-matrix benchmark mode.
+- Repo cleanup: `lab/` consolidation, a binary-tracking hygiene gate, and
+  a git history rewrite that took the repo from 182MB to 20MB, removing
+  the last multi-MB binaries from the index.
+- New design docs: `unified-engine-frontier-plan`, `dspark-serving-program`
+  (the DeepSpec/DSpark spec-decode research program), `structured-output`,
+  `web-chat-redesign` (+ app-aware-assistant, superset-doctrine, and
+  visual-QA-bar notes), `paged-kv-cache`, `batching-perf-path` /
+  `batching-v2-plan`, `runtime-isolation`.
+- Doc drift swept: env-lever table entries (`MLX_BUN_DSPARK_MINCONF`,
+  `MLX_BUN_BATCH_NO_PIPELINE`, `MLX_BUN_EXPERT_OFFLOAD`), ngram
+  draft-kind coverage in README/server-api, doc-map sync, an openwiki
+  investigation write-up. STATUS.md was rewritten to current-state-only
+  several times through the release to keep handoff notes accurate.
+- A new ground rule was recorded in CLAUDE.md: changes to the served
+  surface must update the user-facing reference docs in the same commit.
+- A direct optiq mixed-KV bit-parity golden replaced an indirect
+  ours-vs-ours comparison; a bit-exact L1 parity harness was added for
+  the Qwen3-30B-A3B MoE port, with uniform quantized KV confirmed
+  bit-exact against mlx-lm's own `--kv-bits` and a stock-mlx-lm baseline
+  added to the bench.
+- Dependency/CI bumps: pi, `actions/checkout`, and the site build moved
+  off a deprecated Node runtime.
+
+## Breaking / behavior changes
+
+- **`--batch` default changes from 1 to 8.** Serving now batches by
+  default whenever concurrent requests arrive; pass `--batch 1` to force
+  strict serial (needed if you require arrival-order-independent,
+  bit-identical numerics).
+- **The default decode path changed.** Naked, no-flag serving now runs
+  the faithful, bit-exact-vs-mlx-lm kernel set — what `--l1` used to
+  mean. `--l1`/`--l2`/`--l3` are now pure aliases over per-feature flags
+  rather than distinct performance tiers. Every custom-kernel path that
+  didn't beat this baseline in a paired A/B was deleted in the same
+  pass: the fused-decode kernel, fused-gelu/fused-swiglu/fused-mlp/
+  steel-linear Metal kernels, the flash-decode "perf" kernel and its
+  frozen-oracle test scaffolding, `--l3`, and `FaithfulMiniCPM5` /
+  `MLX_BUN_CPM5_FAITHFUL`. None of these flags or code paths exist
+  anymore. `MLX_BUN_FAITHFUL` is also gone (superseded by the faithful
+  kernels being the default) — the server's unset `--kv-quant` now
+  defaults to bf16 instead of `kv_config`, so getting the old default
+  composition requires an explicit `--kv-quant`.
+- **New CLI/API surface:** `--kv-quant turbo[:kXvY]` (TurboQuant),
+  `--paged-kv` (off by default; refuses to combine with `--batch N>1`,
+  `--kv-quant`, or `--draft-model`), `--draft-model` (speculative
+  decoding), `--isolate` and `--model-pool` (runtime isolation),
+  `--kv-budget <GB>` (aggregate KV admission), `--ssd-demote-idle
+  <seconds>` (default 300, requires `--ssd-cache`),
+  `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1`, and the
+  `response_format` / `guided_grammar` / `guided_regex` / `guided_choice`
+  / `structured_outputs` request fields (kill switch:
+  `MLX_BUN_GRAMMAR=0`).
+- **Media fetches are restricted by default.** `image_url`/`audio_url`
+  now refuse private/loopback/link-local/CGNAT destinations and
+  non-`http(s)` schemes — previously they fetched anything. LAN media
+  needs the new escape hatch above.
+- **`PromptCache.take()` semantics changed.** Cache hits now return
+  non-consuming, ref-counted clones instead of consuming/trimming the
+  donor entry — code holding onto the old consuming behavior should
+  re-check callers.
+
+Full details and measured numbers: STATUS.md, `benchmarks/RESULTS.md`,
+and the design docs referenced above. Credit where due: TurboQuant's
+codec is verified bit-exact against the vendored vllm-metal reference,
+and the speculative-decoding path is verified token-for-token against
+mlx-lm's own `--draft-model` implementation.

package/docs/reference/cli.md

@@ -11,14 +11,15 @@ candidates — just make it more specific.
 Bare `mlx-bun` with no verb runs `serve` — the appliance path: first run
 downloads a starter model, serves it, and opens the chat UI.
 
-> Verbs marked **(v0.0.9)** ship in v0.0.9; everything else is in v0.0.8.
+> All verbs below ship in current releases; the release notes in
+> `docs/planning/` record when each verb first appeared.
 
 ## Serving & chat
 
 ### `serve` — run the server
 
-Start the OpenAI/Anthropic-compatible server plus the status page and browser
-chat UI.
+Start the OpenAI/Anthropic-compatible server plus the browser web app (chat,
+status, and model tools).
 
 ```sh
 mlx-bun serve                              # default model, http://localhost:8080
@@ -34,16 +35,32 @@ Common flags (full list in [server-config.md](server-config.md)):
 | `--host <addr>` | Interface to bind (default `127.0.0.1`, loopback only; `0.0.0.0` to expose) |
 | `--port <n>` | Listen port (default 8080) |
 | `--memory-budget <GB>` | Reject loads/requests that can't fit the budget |
-| `--kv-quant config\|off\|4\|8` | KV cache quantization: per-layer `kv_config.json`, bf16, or uniform bits (default `config`) |
+| `--kv-budget <GB>` | Aggregate KV budget across concurrent batch rows: over-budget joiners queue until rows finish; a request over the budget alone is rejected (off unless set) |
+| `--prompt-cache <GB>` | RAM prefix-KV cache, byte-capped LRU (default 8; `0` disables). Non-consuming prefix sharing: a hit clones instead of consuming the donor entry |
+| `--isolate` | Run the inference engine as a crash-isolated child process on a unix socket; this process becomes a pure reverse-proxy (instant under GPU load, auto-respawns on engine crash) — default off ([runtime-isolation.md](../design/runtime-isolation.md)) |
+| `--model-pool <n>` | With `--isolate`: cap on resident per-model engine children (default 1); over the cap the LRU engine drains its cache to the SSD tier and exits, respawning on demand |
+| `--ssd-cache <dir>` | SSD cold tier under the prompt cache: idle/evicted prefix-KV spills to disk and restores on reattach, surviving restarts (default off — [ssd-kv-cold-tier.md](../design/ssd-kv-cold-tier.md)) |
+| `--ssd-cache-max <GB>` | SSD tier byte cap, oldest-mtime evicted first (default 32; `--ssd-cache` only) |
+| `--ssd-cache-verify` | Verify every tensor hash on restore, not just the header (integrity paranoia; roughly doubles restore reads; `--ssd-cache` only) |
+| `--ssd-demote-idle <sec>` | Idle prompt-cache entries spill to the SSD tier and free GPU memory after this long (default 300 with `--ssd-cache`; `0` disables) |
+| `--kv-quant config\|off\|4\|8\|turbo[:k<bits>v<bits>]` | KV cache quantization: per-layer `kv_config.json`, bf16, uniform bits, or TurboQuant (rotation-based, default `k8v3` — [turboquant-kv.md](../design/turboquant-kv.md); full-attention layers only, solo-only/serial) (default `off` — quantized KV trades 5–20% decode speed for memory headroom, so it's opt-in) |
+| `--paged-kv` | OPTIONAL vLLM-style paged KV cache (block pool + gather before the stock SDPA; env `MLX_BUN_PAGED_KV=1`) — bit-exact with the plain path, default off. v1: serial only (pins `--batch 1` unless `--batch` is given; refuses `--batch N>1`, `--kv-quant`, `--draft-model`), Gemma4-family, bf16; bypasses the prompt cache and runs uncompiled decode. Expect a small decode cost at batch=1 — v1 ships the abstraction for batched/CoW follow-ups ([paged-kv-cache.md](../design/paged-kv-cache.md)) |
+| `--paged-kv-block-size <n>` | Tokens per KV block (default 256 = the plain cache's growth step; `--paged-kv` only) |
 | `--adapter <dir>` | Mount a LoRA adapter at startup (`--adapter-path` accepted as the mlx_lm.server alias) |
-| `--batch <n>` | Continuous-batched bf16 serving (default 1 = serial) |
+| `--draft-model <path\|query>` | Speculative decoding: a drafter proposes, the main model verifies — exact results, faster decode when drafts land. The artifact's kind is auto-detected: a full same-tokenizer model (mlx_lm.server parity), a Gemma `-assistant` KV-borrowing drafter, a DSpark checkpoint trained here, or one of DeepSeek's released DSpark drafters (DeepSpec `Gemma4DSparkModel`, e.g. `dspark_gemma4_12b_block7`). Serial lane only: with `--batch N` a mounted draft routes every request serial |
+| `--draft-kind <kind>` | Override draft-artifact detection: `two-model` \| `assistant` \| `dspark` \| `deepspec` \| `ngram` (default: auto). `ngram` is MODEL-FREE prompt lookup — no `--draft-model` (mounting one alongside it is refused): drafts are copied from the request's own prompt+generation whenever its trailing tokens have occurred earlier in that stream, verified by the target like any other draft, so results stay exact at any temperature. Zero weights, zero memory; best on agentic/RAG/code-edit traffic that re-emits spans already in context (prior art: prompt-lookup decoding / vLLM's `ngram` proposer) |
+| `--num-draft-tokens <n>` | Drafts per verify round (default 3; `ngram` defaults to 10 — drafting is free; a DSpark draft pins to its trained block width) |
+| `--ngram-max <k>` / `--ngram-min <k>` | Prompt-lookup match-window bounds, `--draft-kind ngram` only: the longest/shortest trailing k-gram searched (defaults 3 / 1, the reference values) |
+| `--batch <n>` | Continuous-batched serving cap, mlx-lm B=N parity (default 8; `--batch 1` pins the serial path) |
 | `--temperature` / `--top-p` / `--top-k` / `--max-tokens` | Server-wide sampling defaults (per-request fields still win) |
-| `--l1` / `--l2` / `--l3` | Parity tier: bit-exact to mlx-lm / bit-exact to mlx-optiq / best performance |
+| `--l1` / `--l2` | Parity tier alias: bit-exact to mlx-lm / bit-exact to mlx-optiq. **No tier = `--l1`** (the default since 2026-07-05 — output-changing levers are opt-in until they beat the L1 baseline in a paired A/B). Each expands to per-fork flags (`--compiled-decode`, `--compiled-activations`, `--fused-sdpa`, `--kv-quant`); a fork flag overrides one. `--l3` was removed 2026-07-05 and now errors (the Lab replaces it — [unified-engine-frontier-plan.md](../design/unified-engine-frontier-plan.md)). See [server-config.md](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2). |
+| `--allow-private-media` | Let `image_url`/`audio_url` parts fetch from private/loopback/link-local hosts (blocked by default — SSRF guard; the 10 s timeout and 64 MB cap on remote media apply either way) |
 | `--no-open` | Don't auto-open the chat UI |
 
 Endpoints: `/v1/chat/completions`, `/v1/completions`, `/v1/messages`,
 `/v1/responses`, `/v1/models`, `/v1/adapters`, `/health`, `/stats`, `/fit`,
-`/library`, `/downloads` — status page at `/`, browser chat at `/chat`. Full
+`/library`, `/downloads` — web app at `/` (lands on chat); `/chat`, `/status`,
+`/quantize`, `/finetune`, `/dataset` deep-link its views. Full
 schemas in [server-api.md](server-api.md).
 
 ### `pi` — built-in coding agent
@@ -73,7 +90,7 @@ mlx-bun harness pi --remove     # disconnect
 
 Load the model, generate, print, exit — no server. Renders through the chat
 template by default; `--raw` feeds the prompt verbatim. Sampling and
-decode-path levers mirror `serve` (`--temperature`, `--seed`, `--l1/--l2/--l3`,
+decode-path levers mirror `serve` (`--temperature`, `--seed`, `--l1/--l2`,
 `--kv-quant`, …).
 
 ```sh
@@ -81,7 +98,7 @@ mlx-bun generate gemma "a haiku about metal shaders"
 mlx-bun gen e4b --prompt "…" --max-tokens 512 --seed 42
 ```
 
-### `embed` — text embeddings **(v0.0.9)**
+### `embed` — text embeddings
 
 One-shot embeddings from a local Qwen3-Embedding model (last-token pooled,
 L2-normalized — bit-exact vs mlx-lm). One JSON vector per input line.
@@ -125,10 +142,12 @@ mlx-bun ls --vision --max-size 10GB # filter
 mlx-bun ls --all-revisions          # one row per cached snapshot
 ```
 
-### `gc` — reclaim disk **(v0.0.9)**
+### `gc` — reclaim disk
 
 Delete superseded snapshots and the blobs only they reference. Prints the plan
-by default; `--yes` actually deletes.
+by default; `--yes` actually deletes. Same planner (`src/registry.ts`
+`planGc`/`executeGc`) also backs `GET /api/gc/plan` / `POST /api/gc/execute`
+on a running `serve` — see [server-api.md](server-api.md#get-apigcplan--post-apigcexecute).
 
 ```sh
 mlx-bun gc          # dry-run: what would be reclaimed
@@ -170,7 +189,7 @@ Tails an adapter dir's `metrics.jsonl` and renders live loss / speed / memory.
 mlx-bun train-watch ~/.cache/mlx-bun/mlx-bun-finetunes/orpo-e4b
 ```
 
-### `fuse` — merge an adapter into the base **(v0.0.9)**
+### `fuse` — merge an adapter into the base
 
 Folds LoRA deltas into the base weights and writes a standalone snapshot; a
 quantized base keeps its exact quantization layout. Flag names match
@@ -181,7 +200,7 @@ mlx-bun fuse e4b --adapter ./adapters --save-path ./fused_model
 mlx-bun serve ./fused_model
 ```
 
-### `convert` — quantize a model **(v0.0.9)**
+### `convert` — quantize a model
 
 Quantize an HF model (local path, downloaded model, or repo id) into a local
 MLX snapshot — uniform 4/8-bit, or mixed-precision via `--target-bpw`
@@ -192,7 +211,7 @@ mlx-bun convert Qwen/Qwen3-4B -q                      # uniform 4-bit
 mlx-bun convert Qwen/Qwen3-4B --target-bpw 4.5        # mixed precision
 ```
 
-### `upload` — push to the Hugging Face Hub **(v0.0.9)**
+### `upload` — push to the Hugging Face Hub
 
 Native push-to-hub, no Python: creates the repo if needed, uploads via the
 git-LFS batch protocol, commits. Flag names match `mlx_lm.upload`.
@@ -201,7 +220,7 @@ git-LFS batch protocol, commits. Flag names match `mlx_lm.upload`.
 mlx-bun upload --path ./fused_model --upload-repo you/my-model
 ```
 
-### `perplexity` — evaluate on your data **(v0.0.9)**
+### `perplexity` — evaluate on your data
 
 `mlx_lm.perplexity` methodology exactly, over a **local** `.jsonl`/`.txt` file.
 
@@ -226,7 +245,7 @@ mlx-bun memory synthesize        # conversations → cross-linked articles, all
 mlx-bun memory schedule --at 03:00
 ```
 
-`setup` is an alias for `memory` **(v0.0.9)**.
+`setup` is an alias for `memory`.
 
 ## Measurement
 

package/docs/reference/distribution.md

@@ -156,8 +156,8 @@ manual escape hatch if offline.
 
 ## npm / bunx
 
-mlx-bun is published to npm as `mlx-bun` (current: 0.0.4). Install or
-run without a permanent install:
+mlx-bun is published to npm as `mlx-bun`. Install or run without a
+permanent install:
 
 ```sh
 # one-off (no install)
@@ -168,10 +168,17 @@ bun install -g mlx-bun
 mlx-bun --version
 ```
 
-The npm package wraps the same signed binary via a postinstall download
-step (it does not bundle the ~228 MB dylibs/metallib in the npm tarball
-itself). Publish to npm as part of a release:
+The npm package is a thin launcher (no postinstall hook): it imports
+`src/cli.ts` under Bun, which fetches the small (~52 MB compressed,
+sha256-verified) native runtime pack — `libmlxc`/`libmlx`/`libjaccl`/
+`mlx.metallib` — into `~/Library/Caches/mlx-bun/` on first invocation if
+no copy is already resolvable (env / beside the binary / cache /
+Homebrew). This is the same npm/sidecar mechanism described above, not
+the signed/notarized Homebrew bundle. `bun publish` is already run
+automatically and idempotently as step 4 of `PUBLISH=1
+./scripts/release-binary.sh`; run it standalone only as a manual
+fallback/retry:
 
 ```sh
-bun publish   # from repo root; package.json version must already be bumped
+bun publish   # manual retry; package.json version must already be bumped
 ```

package/docs/reference/embedding.md

@@ -19,15 +19,29 @@ Produces a relocatable directory:
 
 | file | what | size (arm64) |
 |---|---|---|
-| `mlx-bun` | the CLI/server, compiled with `bun build --compile` | ~61 MB |
+| `mlx-bun` | the CLI/server, compiled with `bun build --compile` | ~70-75 MB |
 | `libmlxc.dylib` | mlx-c, rewritten to load `@loader_path/libmlx.dylib` | ~0.7 MB |
 | `libmlx.dylib` | mlx core (+`@loader_path` rpath added for libjaccl) | ~15 MB |
 | `libjaccl.dylib` | mlx's distributed-comm dependency | ~0.6 MB |
 | `mlx.metallib` | Metal kernels — libmlx loads it from its own directory | ~150 MB |
 | `photon_rs_bg.wasm` | pi image codec — only the web chat's `read`-on-image path; resolved next to the executable | ~1.8 MB |
 
+`build-binary.sh` also sidecars a second set of assets for the embedded
+pi **terminal** (`mlx-bun pi`, `src/pi-terminal.ts` → `InteractiveMode`),
+none of which the headless web chat touches:
+
+| file | what |
+|---|---|
+| `theme/*.json` | TUI color themes (built-in dark/light fallback exists in code; shipped so custom theme reads and the schema resolve) |
+| `assets/*.png` | TUI startup art |
+| `export-html/template.html`, `export-html/vendor/*.js` | the `/export` command's HTML template + vendor JS |
+| `package.json`, `CHANGELOG.md` | version banner + startup changelog |
+| `native/darwin/prebuilds/<arch>/darwin-modifiers.node` | pi-tui's native modifier-key detection (degrades gracefully if absent) |
+
 Library resolution order (src/mlx/ffi.ts): `MLX_BUN_LIBMLXC` env var →
-`libmlxc.dylib` next to the executable → homebrew
+`libmlxc.dylib` next to the executable → the downloaded native-pack
+cache (`~/Library/Caches/mlx-bun/native-v<ver>-<arch>/`, populated on
+first run when no sidecar or homebrew install is found) → homebrew
 (`/opt/homebrew/lib`, `/usr/local/lib`). The whole directory can be
 renamed/moved; nothing references absolute paths after the build
 script's `install_name_tool` fixups.
@@ -60,21 +74,28 @@ crashing), so it is best-effort, not load-bearing.
 > The web chat itself is unaffected (its provider is text-only), and the
 > wasm is shipped so it will work once Bun's support lands.
 
-**Intentionally omitted** (TUI-only assets pi ships beside its own binary,
-never reached by the headless web chat):
+## Embedded pi terminal (`mlx-bun pi`)
+
+The compiled binary also embeds pi's full interactive TUI
+(`src/pi-terminal.ts` → `InteractiveMode`), reachable via `mlx-bun pi`.
+Unlike the headless web chat, the terminal *does* resolve theme JSON,
+startup art, the `/export` template, the version banner, and the
+pi-tui native modifier-key helper by path — the second table above is
+sidecar'd for exactly this mode. `src/pi-web.ts`'s session (the web
+chat) still builds with `noThemes`/`noSkills`/`noExtensions`/
+`noPromptTemplates`/`noContextFiles`, so none of that table is reached
+from `/ws/chat` — only `mlx-bun pi` uses it.
+
+**Omitted** (upstream pi ships these beside its own binary; neither
+mlx-bun mode reaches them):
 
 | omitted asset | what it's for |
 |---|---|
-| `theme/*.json` | TUI color themes (loaded via `initTheme`; we pass `noThemes`) |
-| `assets/*.png` | TUI announcement/interactive images |
-| `export-html/*` | the `/export-html` command's template + vendor JS |
-| `docs/`, `examples/`, `README.md`, `CHANGELOG.md` | TUI help/doc browsing |
-
-These are gated behind TUI code paths and the explicit `exportToHtml()`
-method; `createAgentSession`'s import graph (`core/sdk.js`) pulls in no
-theme/asset/export-html loaders, and `src/pi-web.ts` builds the session
-with `noThemes`/`noSkills`/`noExtensions`/`noPromptTemplates`/
-`noContextFiles`.
+| `docs/`, `examples/`, `README.md` | upstream's own help/doc browsing, not exposed by either mlx-bun entry point |
+
+`createAgentSession`'s import graph (`core/sdk.js`) pulls in no loaders
+for these; both `src/pi-web.ts` and `src/pi-terminal.ts` build sessions
+that never reference them.
 
 **Verifying it.** `build-binary.sh` runs `scripts/verify-binary-pi.ts` as
 a sibling compiled binary inside the bundle (so `process.execPath` points

package/docs/reference/features-matrix.md

@@ -0,0 +1,97 @@
+# Features matrix — everything mlx-bun serves, in one table
+
+The complete option inventory: every serving feature, its default, which
+execution lane it runs on, which fidelity tier verifies it, and its knob.
+This is the *what-exists* view; per-flag detail lives in
+[server-config.md](server-config.md), wire formats in
+[server-api.md](server-api.md), CLI verbs in [cli.md](cli.md).
+
+**Lanes:** `serial` = the default single-queue path; `batch` = the
+`--batch N` continuous-batching engine (bf16 compat mode, mlx-lm B=N
+parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
+· Lab = no external oracle; KL/eval-gated experiments (see
+[server-config.md → Fidelity tiers](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2)).
+
+## Serving & performance
+
+| Feature | Default | Lane | Tier | Knob |
+| --- | --- | --- | --- | --- |
+| OpenAI chat completions (`/v1/chat/completions`, SSE) | on | both | — | — |
+| Raw text completions (`/v1/completions`) | on | both | — | — |
+| Anthropic Messages (`/v1/messages`) | on | both | — | — |
+| OpenAI Responses shim (`/v1/responses`) | on | both | — | — |
+| Continuous batching (mlx-lm B=N parity) | **on** — cap 8 (default flipped 2026-07-05; a lone request runs the exact serial engine, so the cap engages only under real concurrency; `--batch 1` pins serial) | batch | L1 | `--batch <n>` |
+| Prompt cache (prefix KV reuse) | on, 8 GB | both | — | `--prompt-cache <GB>` (0 = off) |
+| **SSD KV cold tier** (cache survives eviction + restarts) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
+| **Runtime isolation** (engine runs as a crash-isolated CHILD process behind a reverse-proxy parent; UI stays instant under GPU load and survives engine crashes — [runtime-isolation.md](../design/runtime-isolation.md)) | off | both | — | `--isolate` |
+| **Model pool** (with `--isolate`: LRU-capped RESIDENT model engines; over the cap the least-recently-used engine demotes its cache to the SSD tier and exits, respawning on the next request) | 1 resident engine | both | — | `--model-pool <n>` (requires `--isolate`) |
+| Mixed-precision KV (`kv_config.json`, optiq's scheme) | **off** — opt-in (default flipped to bf16 2026-07-05; quantized KV trades 5–20% decode for memory headroom) | serial + **batch** (per-layer configs batch on every shipped model — full-attention layers Phase 3.1, rotating layers milestone 2; uniform bits stay serial) | L2 | `--kv-quant config\|off\|4\|8`, `--l2` |
+| **TurboQuant KV** (rotation-based: affine keys + FWHT/Lloyd-Max values) | off — opt-in ([turboquant-kv.md](../design/turboquant-kv.md); v1 dequantize-on-fetch, full-attention layers only, no speed claim) | serial only (solo-only, unconditionally) | Lab (codec is oracle-backed vs vllm-metal; the cache class is unvalidated) | `--kv-quant turbo[:k<bits>v<bits>]` (default `k8v3`) |
+| Compiled decode (bit-exact graph replay) | on, every tier | serial | L1/L2 | `--compiled-decode on\|off` |
+| Compiled activations (faithful geglu/swiglu — mlx-lm's `@mx.compile`) | **on**, every tier | both | L1 | `--compiled-activations on\|off` |
+| Fused SDPA (optiq-exact quantized-KV attention) | follows `--kv-quant`: on for `config`, off for uniform/bf16 | serial | L2 | `--fused-sdpa on\|off` |
+| **Paged KV** (vLLM-style block pool: host-side block table + gather before the stock SDPA, no fused kernel in v1 — [paged-kv-cache.md](../design/paged-kv-cache.md)) | off | serial (pins `--batch 1` unless `--batch` given) | gated bit-exact vs the plain `KVCache` (`tests/paged-kv-parity.test.ts`) | `--paged-kv`, `--paged-kv-block-size <n>` (default 256) — v1 scope Gemma4-family bf16; refuses `--batch N>1`, `--kv-quant`, `--draft-model` |
+| **Speculative decoding** (auto-detected drafter: two-model = mlx-lm parity/L1, Gemma `-assistant` = optiq/L2, DSpark local = Lab, DeepSpec released = DeepSpec-reference oracle) | off | serial (forces all-serial) | per-drafter oracle | `--draft-model <path\|query>`, `--draft-kind`, `--num-draft-tokens` |
+| Speculative decoding, **model-free prompt lookup** (drafts copied from the request's own context; port of prompt-lookup decoding / vLLM `ngram`; zero weights) | off | serial (forces all-serial) | lossless by verify (gated vs non-spec greedy, tests/spec-ngram.test.ts) | `--draft-kind ngram` (no `--draft-model`), `--ngram-max`, `--ngram-min`, `--num-draft-tokens` (default 10) |
+| Memory admission (refuse what can't fit; never GPU-OOM) | on (RAM × 0.75) | both | — | `--memory-budget <GB>` |
+| Aggregate KV admission for batch rows (queue, don't OOM) | off | batch | — | `--kv-budget <GB>` |
+| Expert offload (MoE cold experts on mmap) | off | serial | Lab | `--expert-offload` |
+| Extend-join (O(1) batch admission) | on | batch | L1 (own oracle) | `MLX_BUN_BATCH_EXTEND=0` |
+| Vectorized greedy batch sampling | on | batch | bit-equal A/B | `MLX_BUN_BATCH_VEC_SAMPLE=0` |
+
+## Request features
+
+| Feature | Default | Lane | Tier | Knob |
+| --- | --- | --- | --- | --- |
+| **Structured output** (`response_format` json_object/json_schema) | on | both | L2 (oMLX) | request field; `MLX_BUN_GRAMMAR=0` kills |
+| Structured output **jump-forward** (grammar-forced spans emitted without per-token forwards; SGLang technique via xgrammar) | **off** | serial only | Lab (string-lossless + validity gates, tests/grammar-jump.test.ts; token stream may differ — no oracle) | `MLX_BUN_GRAMMAR_JUMP=1` |
+| `guided_grammar` (EBNF) / `guided_regex`¹ / `guided_choice` / `structured_outputs` | on | both | L2 | request fields |
+| Structured output × speculative decoding | on (when both active) | serial | Lab (validity+equivalence) | — |
+| Quantized KV × speculative decoding (any axis: uniform / config / turbo) | KV scheme wins — drafted requests decode serially WITHOUT speculation (spec lane is bf16-KV-only in v1; startup warning) | serial | — | omit `--kv-quant` to speculate |
+| Tool calling (Gemma sentinel / CPM+Qwen XML) + `role:"tool"` loops | on | both | — | request `tools` |
+| Vision (`image_url` parts; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on (models with a tower); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial | L1/L2 | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
+| **Audio input** (`input_audio`/`audio`/`audio_url` parts; WAV native, mp3/m4a/flac/ogg/aiff via CoreAudio; ≤30 s / 750 tokens per clip; mixes with images) | on (models with `audio_config` + sidecar tower: e4b); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial³ | L2 (greedy stream EXACT vs the optiq internal model) | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
+| LoRA adapters (mount at start / hot-swap at runtime) | off | serial | — | `--adapter <dir>`, `POST /v1/adapters` |
+| Sampling: temperature / top-p / top-k / min-p / XTC / logit_bias / presence+frequency+repetition penalties | per-request | both | L1 (mlx-lm-faithful) | request fields / server defaults |
+| `logprobs` / `top_logprobs` (mlx-lm semantics) | off | serial | L1 | request fields |
+| Fixed `seed` reproducibility | off | serial (routes solo) | — | request field |
+| Thinking-mode control (hybrid-reasoning models) | model default | both | — | `--thinking`, `chat_template_kwargs` |
+| Stop sequences / streaming / usage accounting | on | both | — | request fields |
+| HLG tone-curve sampling | off | serial | Lab | `--hlg-sampling on` |
+| Spec-decode telemetry (`usage.speculation`) | on with a draft | serial | — | — |
+| Per-turn lane telemetry (`usage.lane`: serial / serial+spec / batched) | on | both | — | — |
+
+## Model coverage (per-model validated cells — no generic path is trusted untested)
+
+| Family | Serial | Batch | Notes |
+| --- | --- | --- | --- |
+| MiniCPM5 (cpm5) | ✅ L1/L2 | ✅ | the starter model |
+| Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
+| Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
+| DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
+| Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family/sliding universal → serial |
+
+## Beyond serving (the same binary)
+
+Training (`mlx-bun train`: ORPO/DPO/SFT, flash-CCE, segmented backward) ·
+quantization (`convert`, `fuse`, `--target-bpw`) · embeddings · local
+memory/wiki (`mlx-bun memory`) · pi agent integration (`mlx-bun pi`) ·
+model registry + `fit` planner + `gc` · HF `upload` · web chat Model Hub
+(`/api/hub/local` + `/search` + `/download`, per-row `/fit` verdict;
+`/api/hub/serve` is `restart_required`-only today — no in-process model-swap
+seam reaches `/ws/chat`, see [server-api.md](server-api.md#model-hub-apihub))
+· **PWA installability** (manifest + shell-only service worker —
+installable, instant static-shell paint; explicitly not offline chat, see
+[server-api.md](server-api.md#web-app-static-routes--pwa-installability)).
+See [cli.md](cli.md) and the per-topic reference docs.
+
+---
+¹ `guided_regex` accepts the regex∩EBNF subset today (no `\d`/anchors — those degrade to prompt injection); real regex support is tracked.
+² Gated token-exact vs mlx-lm B=2 (static + dynamic join/leave) on Llama-3.2-3B.
+³ Audio is a capability neither ancestor SERVES: mlx-lm strips the audio tower at load and 400s non-text parts; optiq ships the full machinery but never wires it into its serve frontend — mlx-bun exposes it. Serial lane by design (embeddings prefill; batched audio has no oracle → novel-territory gates would apply), prompt cache skipped.
+
+Composition rules and exclusions (what combines with what):
+[server-config.md → Compatibility matrix](server-config.md#compatibility-matrix).
+Performance per mode: `scripts/bench-modes.ts` (TTFT / prefill / decode /
+peak-memory across every cell above); feature-composition perf:
+`scripts/bench-feature-matrix.ts`.