mlx-bun 0.0.10 → 0.0.11

package/docs/reference/cli.md

@@ -11,14 +11,15 @@ candidates — just make it more specific.
 Bare `mlx-bun` with no verb runs `serve` — the appliance path: first run
 downloads a starter model, serves it, and opens the chat UI.
 
-> Verbs marked **(v0.0.9)** ship in v0.0.9; everything else is in v0.0.8.
+> All verbs below ship in current releases; the release notes in
+> `docs/planning/` record when each verb first appeared.
 
 ## Serving & chat
 
 ### `serve` — run the server
 
-Start the OpenAI/Anthropic-compatible server plus the status page and browser
-chat UI.
+Start the OpenAI/Anthropic-compatible server plus the browser web app (chat,
+status, and model tools).
 
 ```sh
 mlx-bun serve                              # default model, http://localhost:8080
@@ -34,16 +35,32 @@ Common flags (full list in [server-config.md](server-config.md)):
 | `--host <addr>` | Interface to bind (default `127.0.0.1`, loopback only; `0.0.0.0` to expose) |
 | `--port <n>` | Listen port (default 8080) |
 | `--memory-budget <GB>` | Reject loads/requests that can't fit the budget |
-| `--kv-quant config\|off\|4\|8` | KV cache quantization: per-layer `kv_config.json`, bf16, or uniform bits (default `config`) |
+| `--kv-budget <GB>` | Aggregate KV budget across concurrent batch rows: over-budget joiners queue until rows finish; a request over the budget alone is rejected (off unless set) |
+| `--prompt-cache <GB>` | RAM prefix-KV cache, byte-capped LRU (default 8; `0` disables). Non-consuming prefix sharing: a hit clones instead of consuming the donor entry |
+| `--isolate` | Run the inference engine as a crash-isolated child process on a unix socket; this process becomes a pure reverse-proxy (instant under GPU load, auto-respawns on engine crash) — default off ([runtime-isolation.md](../design/runtime-isolation.md)) |
+| `--model-pool <n>` | With `--isolate`: cap on resident per-model engine children (default 1); over the cap the LRU engine drains its cache to the SSD tier and exits, respawning on demand |
+| `--ssd-cache <dir>` | SSD cold tier under the prompt cache: idle/evicted prefix-KV spills to disk and restores on reattach, surviving restarts (default off — [ssd-kv-cold-tier.md](../design/ssd-kv-cold-tier.md)) |
+| `--ssd-cache-max <GB>` | SSD tier byte cap, oldest-mtime evicted first (default 32; `--ssd-cache` only) |
+| `--ssd-cache-verify` | Verify every tensor hash on restore, not just the header (integrity paranoia; roughly doubles restore reads; `--ssd-cache` only) |
+| `--ssd-demote-idle <sec>` | Idle prompt-cache entries spill to the SSD tier and free GPU memory after this long (default 300 with `--ssd-cache`; `0` disables) |
+| `--kv-quant config\|off\|4\|8\|turbo[:k<bits>v<bits>]` | KV cache quantization: per-layer `kv_config.json`, bf16, uniform bits, or TurboQuant (rotation-based, default `k8v3` — [turboquant-kv.md](../design/turboquant-kv.md); full-attention layers only, solo-only/serial) (default `off` — quantized KV trades 5–20% decode speed for memory headroom, so it's opt-in) |
+| `--paged-kv` | OPTIONAL vLLM-style paged KV cache (block pool + gather before the stock SDPA; env `MLX_BUN_PAGED_KV=1`) — bit-exact with the plain path, default off. v1: serial only (pins `--batch 1` unless `--batch` is given; refuses `--batch N>1`, `--kv-quant`, `--draft-model`), Gemma4-family, bf16; bypasses the prompt cache and runs uncompiled decode. Expect a small decode cost at batch=1 — v1 ships the abstraction for batched/CoW follow-ups ([paged-kv-cache.md](../design/paged-kv-cache.md)) |
+| `--paged-kv-block-size <n>` | Tokens per KV block (default 256 = the plain cache's growth step; `--paged-kv` only) |
 | `--adapter <dir>` | Mount a LoRA adapter at startup (`--adapter-path` accepted as the mlx_lm.server alias) |
-| `--batch <n>` | Continuous-batched bf16 serving (default 1 = serial) |
+| `--draft-model <path\|query>` | Speculative decoding: a drafter proposes, the main model verifies — exact results, faster decode when drafts land. The artifact's kind is auto-detected: a full same-tokenizer model (mlx_lm.server parity), a Gemma `-assistant` KV-borrowing drafter, a DSpark checkpoint trained here, or one of DeepSeek's released DSpark drafters (DeepSpec `Gemma4DSparkModel`, e.g. `dspark_gemma4_12b_block7`). Serial lane only: with `--batch N` a mounted draft routes every request serial |
+| `--draft-kind <kind>` | Override draft-artifact detection: `two-model` \| `assistant` \| `dspark` \| `deepspec` \| `ngram` (default: auto). `ngram` is MODEL-FREE prompt lookup — no `--draft-model` (mounting one alongside it is refused): drafts are copied from the request's own prompt+generation whenever its trailing tokens have occurred earlier in that stream, verified by the target like any other draft, so results stay exact at any temperature. Zero weights, zero memory; best on agentic/RAG/code-edit traffic that re-emits spans already in context (prior art: prompt-lookup decoding / vLLM's `ngram` proposer) |
+| `--num-draft-tokens <n>` | Drafts per verify round (default 3; `ngram` defaults to 10 — drafting is free; a DSpark draft pins to its trained block width) |
+| `--ngram-max <k>` / `--ngram-min <k>` | Prompt-lookup match-window bounds, `--draft-kind ngram` only: the longest/shortest trailing k-gram searched (defaults 3 / 1, the reference values) |
+| `--batch <n>` | Continuous-batched serving cap, mlx-lm B=N parity (default 8; `--batch 1` pins the serial path) |
 | `--temperature` / `--top-p` / `--top-k` / `--max-tokens` | Server-wide sampling defaults (per-request fields still win) |
-| `--l1` / `--l2` / `--l3` | Parity tier: bit-exact to mlx-lm / bit-exact to mlx-optiq / best performance |
+| `--l1` / `--l2` | Parity tier alias: bit-exact to mlx-lm / bit-exact to mlx-optiq. **No tier = `--l1`** (the default since 2026-07-05 — output-changing levers are opt-in until they beat the L1 baseline in a paired A/B). Each expands to per-fork flags (`--compiled-decode`, `--compiled-activations`, `--fused-sdpa`, `--kv-quant`); a fork flag overrides one. `--l3` was removed 2026-07-05 and now errors (the Lab replaces it — [unified-engine-frontier-plan.md](../design/unified-engine-frontier-plan.md)). See [server-config.md](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2). |
+| `--allow-private-media` | Let `image_url`/`audio_url` parts fetch from private/loopback/link-local hosts (blocked by default — SSRF guard; the 10 s timeout and 64 MB cap on remote media apply either way) |
 | `--no-open` | Don't auto-open the chat UI |
 
 Endpoints: `/v1/chat/completions`, `/v1/completions`, `/v1/messages`,
 `/v1/responses`, `/v1/models`, `/v1/adapters`, `/health`, `/stats`, `/fit`,
-`/library`, `/downloads` — status page at `/`, browser chat at `/chat`. Full
+`/library`, `/downloads` — web app at `/` (lands on chat); `/chat`, `/status`,
+`/quantize`, `/finetune`, `/dataset` deep-link its views. Full
 schemas in [server-api.md](server-api.md).
 
 ### `pi` — built-in coding agent
@@ -73,7 +90,7 @@ mlx-bun harness pi --remove     # disconnect
 
 Load the model, generate, print, exit — no server. Renders through the chat
 template by default; `--raw` feeds the prompt verbatim. Sampling and
-decode-path levers mirror `serve` (`--temperature`, `--seed`, `--l1/--l2/--l3`,
+decode-path levers mirror `serve` (`--temperature`, `--seed`, `--l1/--l2`,
 `--kv-quant`, …).
 
 ```sh
@@ -81,7 +98,7 @@ mlx-bun generate gemma "a haiku about metal shaders"
 mlx-bun gen e4b --prompt "…" --max-tokens 512 --seed 42
 ```
 
-### `embed` — text embeddings **(v0.0.9)**
+### `embed` — text embeddings
 
 One-shot embeddings from a local Qwen3-Embedding model (last-token pooled,
 L2-normalized — bit-exact vs mlx-lm). One JSON vector per input line.
@@ -125,10 +142,12 @@ mlx-bun ls --vision --max-size 10GB # filter
 mlx-bun ls --all-revisions          # one row per cached snapshot
 ```
 
-### `gc` — reclaim disk **(v0.0.9)**
+### `gc` — reclaim disk
 
 Delete superseded snapshots and the blobs only they reference. Prints the plan
-by default; `--yes` actually deletes.
+by default; `--yes` actually deletes. Same planner (`src/registry.ts`
+`planGc`/`executeGc`) also backs `GET /api/gc/plan` / `POST /api/gc/execute`
+on a running `serve` — see [server-api.md](server-api.md#get-apigcplan--post-apigcexecute).
 
 ```sh
 mlx-bun gc          # dry-run: what would be reclaimed
@@ -170,7 +189,7 @@ Tails an adapter dir's `metrics.jsonl` and renders live loss / speed / memory.
 mlx-bun train-watch ~/.cache/mlx-bun/mlx-bun-finetunes/orpo-e4b
 ```
 
-### `fuse` — merge an adapter into the base **(v0.0.9)**
+### `fuse` — merge an adapter into the base
 
 Folds LoRA deltas into the base weights and writes a standalone snapshot; a
 quantized base keeps its exact quantization layout. Flag names match
@@ -181,7 +200,7 @@ mlx-bun fuse e4b --adapter ./adapters --save-path ./fused_model
 mlx-bun serve ./fused_model
 ```
 
-### `convert` — quantize a model **(v0.0.9)**
+### `convert` — quantize a model
 
 Quantize an HF model (local path, downloaded model, or repo id) into a local
 MLX snapshot — uniform 4/8-bit, or mixed-precision via `--target-bpw`
@@ -192,7 +211,7 @@ mlx-bun convert Qwen/Qwen3-4B -q                      # uniform 4-bit
 mlx-bun convert Qwen/Qwen3-4B --target-bpw 4.5        # mixed precision
 ```
 
-### `upload` — push to the Hugging Face Hub **(v0.0.9)**
+### `upload` — push to the Hugging Face Hub
 
 Native push-to-hub, no Python: creates the repo if needed, uploads via the
 git-LFS batch protocol, commits. Flag names match `mlx_lm.upload`.
@@ -201,7 +220,7 @@ git-LFS batch protocol, commits. Flag names match `mlx_lm.upload`.
 mlx-bun upload --path ./fused_model --upload-repo you/my-model
 ```
 
-### `perplexity` — evaluate on your data **(v0.0.9)**
+### `perplexity` — evaluate on your data
 
 `mlx_lm.perplexity` methodology exactly, over a **local** `.jsonl`/`.txt` file.
 
@@ -226,7 +245,7 @@ mlx-bun memory synthesize        # conversations → cross-linked articles, all
 mlx-bun memory schedule --at 03:00
 ```
 
-`setup` is an alias for `memory` **(v0.0.9)**.
+`setup` is an alias for `memory`.
 
 ## Measurement
 

package/docs/reference/distribution.md

@@ -156,8 +156,8 @@ manual escape hatch if offline.
 
 ## npm / bunx
 
-mlx-bun is published to npm as `mlx-bun` (current: 0.0.4). Install or
-run without a permanent install:
+mlx-bun is published to npm as `mlx-bun`. Install or run without a
+permanent install:
 
 ```sh
 # one-off (no install)
@@ -168,10 +168,17 @@ bun install -g mlx-bun
 mlx-bun --version
 ```
 
-The npm package wraps the same signed binary via a postinstall download
-step (it does not bundle the ~228 MB dylibs/metallib in the npm tarball
-itself). Publish to npm as part of a release:
+The npm package is a thin launcher (no postinstall hook): it imports
+`src/cli.ts` under Bun, which fetches the small (~52 MB compressed,
+sha256-verified) native runtime pack — `libmlxc`/`libmlx`/`libjaccl`/
+`mlx.metallib` — into `~/Library/Caches/mlx-bun/` on first invocation if
+no copy is already resolvable (env / beside the binary / cache /
+Homebrew). This is the same npm/sidecar mechanism described above, not
+the signed/notarized Homebrew bundle. `bun publish` is already run
+automatically and idempotently as step 4 of `PUBLISH=1
+./scripts/release-binary.sh`; run it standalone only as a manual
+fallback/retry:
 
 ```sh
-bun publish   # from repo root; package.json version must already be bumped
+bun publish   # manual retry; package.json version must already be bumped
 ```

package/docs/reference/embedding.md

@@ -19,15 +19,29 @@ Produces a relocatable directory:
 
 | file | what | size (arm64) |
 |---|---|---|
-| `mlx-bun` | the CLI/server, compiled with `bun build --compile` | ~61 MB |
+| `mlx-bun` | the CLI/server, compiled with `bun build --compile` | ~70-75 MB |
 | `libmlxc.dylib` | mlx-c, rewritten to load `@loader_path/libmlx.dylib` | ~0.7 MB |
 | `libmlx.dylib` | mlx core (+`@loader_path` rpath added for libjaccl) | ~15 MB |
 | `libjaccl.dylib` | mlx's distributed-comm dependency | ~0.6 MB |
 | `mlx.metallib` | Metal kernels — libmlx loads it from its own directory | ~150 MB |
 | `photon_rs_bg.wasm` | pi image codec — only the web chat's `read`-on-image path; resolved next to the executable | ~1.8 MB |
 
+`build-binary.sh` also sidecars a second set of assets for the embedded
+pi **terminal** (`mlx-bun pi`, `src/pi-terminal.ts` → `InteractiveMode`),
+none of which the headless web chat touches:
+
+| file | what |
+|---|---|
+| `theme/*.json` | TUI color themes (built-in dark/light fallback exists in code; shipped so custom theme reads and the schema resolve) |
+| `assets/*.png` | TUI startup art |
+| `export-html/template.html`, `export-html/vendor/*.js` | the `/export` command's HTML template + vendor JS |
+| `package.json`, `CHANGELOG.md` | version banner + startup changelog |
+| `native/darwin/prebuilds/<arch>/darwin-modifiers.node` | pi-tui's native modifier-key detection (degrades gracefully if absent) |
+
 Library resolution order (src/mlx/ffi.ts): `MLX_BUN_LIBMLXC` env var →
-`libmlxc.dylib` next to the executable → homebrew
+`libmlxc.dylib` next to the executable → the downloaded native-pack
+cache (`~/Library/Caches/mlx-bun/native-v<ver>-<arch>/`, populated on
+first run when no sidecar or homebrew install is found) → homebrew
 (`/opt/homebrew/lib`, `/usr/local/lib`). The whole directory can be
 renamed/moved; nothing references absolute paths after the build
 script's `install_name_tool` fixups.
@@ -60,21 +74,28 @@ crashing), so it is best-effort, not load-bearing.
 > The web chat itself is unaffected (its provider is text-only), and the
 > wasm is shipped so it will work once Bun's support lands.
 
-**Intentionally omitted** (TUI-only assets pi ships beside its own binary,
-never reached by the headless web chat):
+## Embedded pi terminal (`mlx-bun pi`)
+
+The compiled binary also embeds pi's full interactive TUI
+(`src/pi-terminal.ts` → `InteractiveMode`), reachable via `mlx-bun pi`.
+Unlike the headless web chat, the terminal *does* resolve theme JSON,
+startup art, the `/export` template, the version banner, and the
+pi-tui native modifier-key helper by path — the second table above is
+sidecar'd for exactly this mode. `src/pi-web.ts`'s session (the web
+chat) still builds with `noThemes`/`noSkills`/`noExtensions`/
+`noPromptTemplates`/`noContextFiles`, so none of that table is reached
+from `/ws/chat` — only `mlx-bun pi` uses it.
+
+**Omitted** (upstream pi ships these beside its own binary; neither
+mlx-bun mode reaches them):
 
 | omitted asset | what it's for |
 |---|---|
-| `theme/*.json` | TUI color themes (loaded via `initTheme`; we pass `noThemes`) |
-| `assets/*.png` | TUI announcement/interactive images |
-| `export-html/*` | the `/export-html` command's template + vendor JS |
-| `docs/`, `examples/`, `README.md`, `CHANGELOG.md` | TUI help/doc browsing |
-
-These are gated behind TUI code paths and the explicit `exportToHtml()`
-method; `createAgentSession`'s import graph (`core/sdk.js`) pulls in no
-theme/asset/export-html loaders, and `src/pi-web.ts` builds the session
-with `noThemes`/`noSkills`/`noExtensions`/`noPromptTemplates`/
-`noContextFiles`.
+| `docs/`, `examples/`, `README.md` | upstream's own help/doc browsing, not exposed by either mlx-bun entry point |
+
+`createAgentSession`'s import graph (`core/sdk.js`) pulls in no loaders
+for these; both `src/pi-web.ts` and `src/pi-terminal.ts` build sessions
+that never reference them.
 
 **Verifying it.** `build-binary.sh` runs `scripts/verify-binary-pi.ts` as
 a sibling compiled binary inside the bundle (so `process.execPath` points

package/docs/reference/features-matrix.md

@@ -0,0 +1,97 @@
+# Features matrix — everything mlx-bun serves, in one table
+
+The complete option inventory: every serving feature, its default, which
+execution lane it runs on, which fidelity tier verifies it, and its knob.
+This is the *what-exists* view; per-flag detail lives in
+[server-config.md](server-config.md), wire formats in
+[server-api.md](server-api.md), CLI verbs in [cli.md](cli.md).
+
+**Lanes:** `serial` = the default single-queue path; `batch` = the
+`--batch N` continuous-batching engine (bf16 compat mode, mlx-lm B=N
+parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
+· Lab = no external oracle; KL/eval-gated experiments (see
+[server-config.md → Fidelity tiers](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2)).
+
+## Serving & performance
+
+| Feature | Default | Lane | Tier | Knob |
+| --- | --- | --- | --- | --- |
+| OpenAI chat completions (`/v1/chat/completions`, SSE) | on | both | — | — |
+| Raw text completions (`/v1/completions`) | on | both | — | — |
+| Anthropic Messages (`/v1/messages`) | on | both | — | — |
+| OpenAI Responses shim (`/v1/responses`) | on | both | — | — |
+| Continuous batching (mlx-lm B=N parity) | **on** — cap 8 (default flipped 2026-07-05; a lone request runs the exact serial engine, so the cap engages only under real concurrency; `--batch 1` pins serial) | batch | L1 | `--batch <n>` |
+| Prompt cache (prefix KV reuse) | on, 8 GB | both | — | `--prompt-cache <GB>` (0 = off) |
+| **SSD KV cold tier** (cache survives eviction + restarts) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
+| **Runtime isolation** (engine runs as a crash-isolated CHILD process behind a reverse-proxy parent; UI stays instant under GPU load and survives engine crashes — [runtime-isolation.md](../design/runtime-isolation.md)) | off | both | — | `--isolate` |
+| **Model pool** (with `--isolate`: LRU-capped RESIDENT model engines; over the cap the least-recently-used engine demotes its cache to the SSD tier and exits, respawning on the next request) | 1 resident engine | both | — | `--model-pool <n>` (requires `--isolate`) |
+| Mixed-precision KV (`kv_config.json`, optiq's scheme) | **off** — opt-in (default flipped to bf16 2026-07-05; quantized KV trades 5–20% decode for memory headroom) | serial + **batch** (per-layer configs batch on every shipped model — full-attention layers Phase 3.1, rotating layers milestone 2; uniform bits stay serial) | L2 | `--kv-quant config\|off\|4\|8`, `--l2` |
+| **TurboQuant KV** (rotation-based: affine keys + FWHT/Lloyd-Max values) | off — opt-in ([turboquant-kv.md](../design/turboquant-kv.md); v1 dequantize-on-fetch, full-attention layers only, no speed claim) | serial only (solo-only, unconditionally) | Lab (codec is oracle-backed vs vllm-metal; the cache class is unvalidated) | `--kv-quant turbo[:k<bits>v<bits>]` (default `k8v3`) |
+| Compiled decode (bit-exact graph replay) | on, every tier | serial | L1/L2 | `--compiled-decode on\|off` |
+| Compiled activations (faithful geglu/swiglu — mlx-lm's `@mx.compile`) | **on**, every tier | both | L1 | `--compiled-activations on\|off` |
+| Fused SDPA (optiq-exact quantized-KV attention) | follows `--kv-quant`: on for `config`, off for uniform/bf16 | serial | L2 | `--fused-sdpa on\|off` |
+| **Paged KV** (vLLM-style block pool: host-side block table + gather before the stock SDPA, no fused kernel in v1 — [paged-kv-cache.md](../design/paged-kv-cache.md)) | off | serial (pins `--batch 1` unless `--batch` given) | gated bit-exact vs the plain `KVCache` (`tests/paged-kv-parity.test.ts`) | `--paged-kv`, `--paged-kv-block-size <n>` (default 256) — v1 scope Gemma4-family bf16; refuses `--batch N>1`, `--kv-quant`, `--draft-model` |
+| **Speculative decoding** (auto-detected drafter: two-model = mlx-lm parity/L1, Gemma `-assistant` = optiq/L2, DSpark local = Lab, DeepSpec released = DeepSpec-reference oracle) | off | serial (forces all-serial) | per-drafter oracle | `--draft-model <path\|query>`, `--draft-kind`, `--num-draft-tokens` |
+| Speculative decoding, **model-free prompt lookup** (drafts copied from the request's own context; port of prompt-lookup decoding / vLLM `ngram`; zero weights) | off | serial (forces all-serial) | lossless by verify (gated vs non-spec greedy, tests/spec-ngram.test.ts) | `--draft-kind ngram` (no `--draft-model`), `--ngram-max`, `--ngram-min`, `--num-draft-tokens` (default 10) |
+| Memory admission (refuse what can't fit; never GPU-OOM) | on (RAM × 0.75) | both | — | `--memory-budget <GB>` |
+| Aggregate KV admission for batch rows (queue, don't OOM) | off | batch | — | `--kv-budget <GB>` |
+| Expert offload (MoE cold experts on mmap) | off | serial | Lab | `--expert-offload` |
+| Extend-join (O(1) batch admission) | on | batch | L1 (own oracle) | `MLX_BUN_BATCH_EXTEND=0` |
+| Vectorized greedy batch sampling | on | batch | bit-equal A/B | `MLX_BUN_BATCH_VEC_SAMPLE=0` |
+
+## Request features
+
+| Feature | Default | Lane | Tier | Knob |
+| --- | --- | --- | --- | --- |
+| **Structured output** (`response_format` json_object/json_schema) | on | both | L2 (oMLX) | request field; `MLX_BUN_GRAMMAR=0` kills |
+| Structured output **jump-forward** (grammar-forced spans emitted without per-token forwards; SGLang technique via xgrammar) | **off** | serial only | Lab (string-lossless + validity gates, tests/grammar-jump.test.ts; token stream may differ — no oracle) | `MLX_BUN_GRAMMAR_JUMP=1` |
+| `guided_grammar` (EBNF) / `guided_regex`¹ / `guided_choice` / `structured_outputs` | on | both | L2 | request fields |
+| Structured output × speculative decoding | on (when both active) | serial | Lab (validity+equivalence) | — |
+| Quantized KV × speculative decoding (any axis: uniform / config / turbo) | KV scheme wins — drafted requests decode serially WITHOUT speculation (spec lane is bf16-KV-only in v1; startup warning) | serial | — | omit `--kv-quant` to speculate |
+| Tool calling (Gemma sentinel / CPM+Qwen XML) + `role:"tool"` loops | on | both | — | request `tools` |
+| Vision (`image_url` parts; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on (models with a tower); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial | L1/L2 | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
+| **Audio input** (`input_audio`/`audio`/`audio_url` parts; WAV native, mp3/m4a/flac/ogg/aiff via CoreAudio; ≤30 s / 750 tokens per clip; mixes with images) | on (models with `audio_config` + sidecar tower: e4b); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial³ | L2 (greedy stream EXACT vs the optiq internal model) | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
+| LoRA adapters (mount at start / hot-swap at runtime) | off | serial | — | `--adapter <dir>`, `POST /v1/adapters` |
+| Sampling: temperature / top-p / top-k / min-p / XTC / logit_bias / presence+frequency+repetition penalties | per-request | both | L1 (mlx-lm-faithful) | request fields / server defaults |
+| `logprobs` / `top_logprobs` (mlx-lm semantics) | off | serial | L1 | request fields |
+| Fixed `seed` reproducibility | off | serial (routes solo) | — | request field |
+| Thinking-mode control (hybrid-reasoning models) | model default | both | — | `--thinking`, `chat_template_kwargs` |
+| Stop sequences / streaming / usage accounting | on | both | — | request fields |
+| HLG tone-curve sampling | off | serial | Lab | `--hlg-sampling on` |
+| Spec-decode telemetry (`usage.speculation`) | on with a draft | serial | — | — |
+| Per-turn lane telemetry (`usage.lane`: serial / serial+spec / batched) | on | both | — | — |
+
+## Model coverage (per-model validated cells — no generic path is trusted untested)
+
+| Family | Serial | Batch | Notes |
+| --- | --- | --- | --- |
+| MiniCPM5 (cpm5) | ✅ L1/L2 | ✅ | the starter model |
+| Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
+| Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
+| DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
+| Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family/sliding universal → serial |
+
+## Beyond serving (the same binary)
+
+Training (`mlx-bun train`: ORPO/DPO/SFT, flash-CCE, segmented backward) ·
+quantization (`convert`, `fuse`, `--target-bpw`) · embeddings · local
+memory/wiki (`mlx-bun memory`) · pi agent integration (`mlx-bun pi`) ·
+model registry + `fit` planner + `gc` · HF `upload` · web chat Model Hub
+(`/api/hub/local` + `/search` + `/download`, per-row `/fit` verdict;
+`/api/hub/serve` is `restart_required`-only today — no in-process model-swap
+seam reaches `/ws/chat`, see [server-api.md](server-api.md#model-hub-apihub))
+· **PWA installability** (manifest + shell-only service worker —
+installable, instant static-shell paint; explicitly not offline chat, see
+[server-api.md](server-api.md#web-app-static-routes--pwa-installability)).
+See [cli.md](cli.md) and the per-topic reference docs.
+
+---
+¹ `guided_regex` accepts the regex∩EBNF subset today (no `\d`/anchors — those degrade to prompt injection); real regex support is tracked.
+² Gated token-exact vs mlx-lm B=2 (static + dynamic join/leave) on Llama-3.2-3B.
+³ Audio is a capability neither ancestor SERVES: mlx-lm strips the audio tower at load and 400s non-text parts; optiq ships the full machinery but never wires it into its serve frontend — mlx-bun exposes it. Serial lane by design (embeddings prefill; batched audio has no oracle → novel-territory gates would apply), prompt cache skipped.
+
+Composition rules and exclusions (what combines with what):
+[server-config.md → Compatibility matrix](server-config.md#compatibility-matrix).
+Performance per mode: `scripts/bench-modes.ts` (TTFT / prefill / decode /
+peak-memory across every cell above); feature-composition perf:
+`scripts/bench-feature-matrix.ts`.

package/docs/reference/library-api.md

@@ -3,16 +3,19 @@
 The HTTP server is one consumer of a library-first API. Everything it
 does — generation, sampling, KV schemes, LoRA, vision, caching, memory
 admission — is importable directly into a Bun process. Published as
-`mlx-bun` on npm (0.0.4); `bunx mlx-bun` works out of the box, or
+`mlx-bun` on npm; `bunx mlx-bun` works out of the box, or
 `import … from "mlx-bun"` for programmatic use.
 
-One rule everywhere: **one generation at a time per process** (one GPU,
-batch=1). The server serializes through a queue; do the same.
+One rule everywhere: **one generation at a time per process** (one GPU).
+The server serializes through a queue (or, under `--batch N`, its
+continuous-batching scheduler — a server-lane feature, not a library
+surface); direct library callers must do the same.
 
 ## Quick start
 
 ```ts
 import {
+  ensureNativeRuntime,
   loadModelConfig,
   Weights,
   createModel,       // dispatches to Gemma4Model / MiniCPM5Model / Qwen35Model
@@ -21,6 +24,7 @@ import {
   generate,
 } from "mlx-bun";   // or "./src/index" in-repo
 
+await ensureNativeRuntime();
 const dir = "/path/to/hf-snapshot";          // mlx-bun ls prints these
 const config = await loadModelConfig(dir);
 const model = createModel(await Weights.open(dir), config);   // returns RuntimeModel
@@ -29,7 +33,7 @@ const template = await ChatTemplate.load(dir);
 // Gemma4Model and MiniCPM5Model can still be imported directly from "mlx-bun"
 // if you need the concrete type. Qwen35Model is NOT exported from the public
 // package — use createModel/RuntimeModel (preferred dispatch), or import from
-// "./src/model/qwen35" in-repo.
+// "./src/model/qwen3_5" in-repo.
 
 const ids = tok.encode(template.render([{ role: "user", content: "hi" }]));
 const promptIds = ids[0] === ids[1] && ids[0] === tok.bosTokenId ? ids.slice(1) : ids;
@@ -44,6 +48,14 @@ console.log(gen.stats);                          // set once iteration ends
 Weight loading is lazy (mmap + mlx native loader): construction is
 milliseconds; weights materialize on first forward.
 
+> **First run:** library consumers bypass the CLI's first-run step, so
+> call `await ensureNativeRuntime()` once before constructing a model on
+> a machine that may not have the MLX native runtime yet — it downloads
+> the sha256-verified native pack to `~/Library/Caches/mlx-bun/` and is
+> a no-op when the runtime is already present (beside the executable,
+> already cached, or installed via homebrew). `nativeRuntimeDir()`
+> returns the resolved directory, or `null` on a fresh machine.
+
 ## generate(model, promptTokens, options) → Generation
 
 `Generation` is an `AsyncIterable<GeneratedToken>` with a `stats`
@@ -64,13 +76,20 @@ after an early `break`, which cleanly cancels the in-flight step).
 | `kvConfig` | off | per-layer mixed precision from `kv_config.json` (`config.kvQuant`); overrides `kvBits`, start 0 — optiq serve semantics |
 | `adapters` | none | mounted LoRA adapter ids, applied for exactly this generation |
 | `promptEmbeddings` / `imageMask` | — | vision path (see `src/vision/`) |
+| `logprobs` / `topLogprobs` | off | per-token logprob capture (mlx_lm.server parity): `logprobs` captures the emitted token's log-probability, `topLogprobs` (0/unset = off) captures the top-k (id, logprob) pairs — same full-vocab log-softmax distribution mlx-lm's generate_step samples from, taken after logits processors and before the sampler |
+| `grammar` | off | a compiled `GrammarController` (`src/grammar.ts`) for structured output — masks invalid tokens to `-inf` each step (L2-class, oMLX oracle) |
+| `turboQuant` | off | TurboQuant KV scheme (docs/design/turboquant-kv.md), a rotation-based alternative to uniform `kvBits`; mutually exclusive with `kvBits`/`kvConfig` |
+| `pagedKv` | off | `{ blockSize? }` — OPTIONAL paged KV cache (docs/design/paged-kv-cache.md); v1 scope is serial batch=1 Gemma4-family bf16, mutually exclusive with `kvBits`/`kvConfig`/`turboQuant`/draft/compiled decode |
 
 `GenerateStats`: `promptTokens`, `cachedTokens`, `generatedTokens`,
 `prefillTps` / `prefillMs`, `decodeTps` / `decodeMs`, `cacheTokens`
 (the exact token sequence whose KV is in the cache — feed it to
 `PromptCache.put`). Timing semantics match mlx-lm: the prompt clock
 runs until the first token arrives (the prefill→decode boundary is
-prompt time), the decode clock from there.
+prompt time), the decode clock from there. An optional `spec` field
+carries speculative-decoding telemetry (serve `--draft-model` path
+only): `drafted`, `accepted`, `targetCalls`, and per-draft-position
+`draftedByPos` / `acceptedByPos` counts.
 
 ## Serving pieces
 
@@ -80,7 +99,7 @@ import { loadContext, createServer } from "mlx-bun";
 const ctx = await loadContext(dir, "my-model", { memoryBudgetBytes: 12e9 });
 const server = createServer(ctx, 8080, {
   promptCacheBytes: 2e9,        // byte-capped LRU (never count-capped)
-  kvQuant: undefined,           // default: kv_config.json; "off" | bits
+  kvQuant: undefined,           // default: kv_config.json; "off" | "config" | bits
   memoryBudgetBytes: 12e9,      // admission control — the only OOM defense
 });
 ```
@@ -135,11 +154,13 @@ under one adapter must not seed another's prefill).
 ```ts
 import { saveKvCache, loadKvCache } from "./src/kv-store";
 saveKvCache("/tmp/prefix.kv", tokens, caches);       // page-aligned file
-const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model as Gemma4Model);
+const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model);
 // reload is a zero-copy MAP_PRIVATE mmap straight to the GPU (~1 ms);
 // keep `mmap` referenced as long as the caches live
-// Note: loadKvCache is typed to accept Gemma4Model, but only uses model.layers.length
-// at runtime — a cast from RuntimeModel is safe for any model that has a layers array.
+// Note: loadKvCache accepts anything with makeCache(): Cache[] — it validates
+// the entry count against the cache list makeCache() returns (not
+// model.layers.length, which is wrong for KV-shared models like e4b, whose
+// makeCache() returns donors only). RuntimeModel qualifies directly, no cast.
 ```
 
 Quantized caches are not persistable yet (documented gap).

package/docs/reference/memory.md

@@ -42,18 +42,38 @@ mlx-bun memory unschedule
 
 ## Agent tools
 
-Both `mlx-bun pi` and the web chat expose these read-only tools:
+Both `mlx-bun pi` and the web chat expose these read-only tools, in the recommended call order — FIND the article, READ it small, follow the graph, fall back to reference docs and utilities, search only as a last resort:
 
-- `memory_search`
-- `memory_read`
-- `memory_toc`
-- `memory_section`
-- `memory_links`
+**FIND** (deterministic lookup, never a vector search):
+- `memory_resolve` — name/alias → the article
+- `memory_category` — category/type/series → members
+
+**READ** (TOC → one section, not a full-article dump):
+- `memory_read` — TOC + lead by default; `force=true` for the full article
+- `memory_section` — one section's body (the default granularity)
+
+**Follow the graph**:
+- `memory_links` — outbound/inbound wikilinks
+- `memory_infobox` — read an article's infobox fields (there is no infobox *query* tool — the infobox is content the model reads, not a filter)
+
+**Reference docs** (mlx-bun's own docs, mirrored read-only into the vault):
+- `reference_search`
+- `reference_read`
+- `reference_list`
+
+**Utility**:
 - `memory_list` — user articles plus read-only `Reference/*` docs
 - `memory_status` — vault path, setup state, article count, git state, last synthesis run (from the vault's git log), schedule state, recent changed articles
 
+**Last resort**:
+- `memory_search` — substring fallback when FIND failed
+
 The tools are always registered. If no vault exists, they return a setup message instead of disappearing. Search/read results prefixed with `Reference/` are mlx-bun docs, not user memory articles.
 
+## Web chat REST surface
+
+The web chat's Memory panel talks to a separate, HTTP-only `/api/memory/*` route surface (status/list/search/article/links/history/diff + a `POST /api/memory/init` consent-card endpoint) — thin wrappers over the same vault read path above, documented in [server-api.md](server-api.md#get-apimemory--post-apimemoryinit). The agent-tool surface above is unaffected by it.
+
 ## Obsidian
 
 ```bash

package/docs/reference/models.md

@@ -79,6 +79,8 @@ Schema (table `models`, one row **per snapshot**, keyed on `path`):
 | `quant_bits`, `quant_group_size`, `quant_mode` | quantization |
 | `has_vision_sidecar` | sidecar file present |
 | `vision_config_type` | `config.json`'s `vision_config.model_type` when it names a vision tower (`*_vision`) |
+| `has_audio_config` | `config.json` declares an `audio_config` block |
+| `has_audio_tower` | sidecar header names `audio_tower.*` tensors |
 | `has_kv_config`, `has_tool_template`, `license` | capabilities/terms |
 | `scanned_at` | freshness (canonical tie-break) |
 
@@ -89,6 +91,14 @@ Schema (table `models`, one row **per snapshot**, keyed on `path`):
 Presence of a `vision_config` key alone is **not** a signal — Qwen3.5
 nests a copy of its own text config there.
 
+**Audio capability** is `has_audio_config AND has_audio_tower`: both
+legs are required because the 12B OptiQ snapshot carries an
+`audio_config` but a stub sidecar whose only audio entry is
+`embed_audio.embedding_projection.weight` (no `audio_tower.*` Conformer
+tensors) — config presence alone would advertise audio that fails at
+request time. The check reads only the sidecar's safetensors header,
+never tensor bytes.
+
 ## Query resolution rules (per verb)
 
 Every verb resolves fuzzy queries the same way, with revision collapsing
@@ -98,7 +108,7 @@ so a stale snapshot never makes a repo "ambiguous":
   row per repo** (the canonical revision). `--all-revisions` shows the
   per-snapshot truth with the canonical marked `*`. Capabilities column
   labels the support tier: `supported (targeted)` (dedicated forward,
-  L2/L3 paths) vs `supported (generic)` (Tier-0 universal module).
+  L2/opt-in paths) vs `supported (generic)` (Tier-0 universal module).
 - **`fit` / `serve` / `train` / … (single-model verbs)** — `resolve()`:
   the query must match exactly one **repo**; multiple cached revisions of
   that repo collapse to the canonical snapshot (refs/main, else most

package/docs/reference/orpo-quickstart.md

@@ -14,9 +14,9 @@ mlx-bun train <model> --data <dir>
 - `<model>` is a query or snapshot path (e.g. `e4b`, `MiniCPM5`); omit it to auto-pick.
 - `--data <dir>` is a directory with `train.jsonl` (+ optional `valid.jsonl`); rows are
   `{"prompt": "...", "chosen": "...", "rejected": "..."}`.
-- The command **auto-detects e4b/Gemma** and sets its required training env flags
-  (`MLX_BUN_PERF_KERNEL=0`, `MLX_BUN_FUSED_GELU=0`) before loading the model — you don't
-  set them yourself. The full stack (flash head + prefix-share + segmented) is on by default.
+- The full stack (flash head + prefix-share + segmented) is on by default. No env-flag
+  setup is needed for any model (the old Gemma training-flag requirement is gone —
+  the no-vjp kernels were deleted 2026-07-05).
 - It prints a plan box (dataset counts, format, the resolved knobs), streams per-step loss
   + peak memory, and saves a **mountable** adapter (`mlx-bun serve <model> --adapter <dir>`).
 - `--dry-run` inspects the dataset and prints the plan without training. `mlx-bun help train`
@@ -59,7 +59,6 @@ mlx-bun train e4b --data ./prefs --resume ~/.cache/mlx-bun/mlx-bun-finetunes/orp
 MODEL=/path/to/snapshot DATA=/path/to/datadir bun scripts/train-orpo.ts
 ```
 - `DATA` is a directory with `train.jsonl` (+ optional `valid.jsonl`).
-- Auto-detects e4b/Gemma and sets the training env flags, same as the CLI.
 
 ### Preconfigured defaults (override via env)
 | env | default | meaning |