mlx-bun 0.0.10 → 0.0.12

package/docs/reference/embedding.md

@@ -7,7 +7,7 @@
 
 Ship local inference inside a Mac app (Tauri, Electron, or anything
 that can spawn a process) with zero user-visible dependencies: one
-executable + three native libraries, dropped into your app's resources.
+executable + four native libraries, dropped into your app's resources.
 
 ## Build the bundle
 
@@ -19,18 +19,35 @@ Produces a relocatable directory:
 
 | file | what | size (arm64) |
 |---|---|---|
-| `mlx-bun` | the CLI/server, compiled with `bun build --compile` | ~61 MB |
+| `mlx-bun` | the CLI/server, compiled with `bun build --compile` | ~70-75 MB |
 | `libmlxc.dylib` | mlx-c, rewritten to load `@loader_path/libmlx.dylib` | ~0.7 MB |
 | `libmlx.dylib` | mlx core (+`@loader_path` rpath added for libjaccl) | ~15 MB |
 | `libjaccl.dylib` | mlx's distributed-comm dependency | ~0.6 MB |
+| `libmlx_bun_expert_io.dylib` | bounded streamed-expert I/O used by direct Colibri GLM-5.2 | ~0.03 MB |
 | `mlx.metallib` | Metal kernels — libmlx loads it from its own directory | ~150 MB |
 | `photon_rs_bg.wasm` | pi image codec — only the web chat's `read`-on-image path; resolved next to the executable | ~1.8 MB |
 
+`build-binary.sh` also sidecars a second set of assets for the embedded
+pi **terminal** (`mlx-bun pi`, `src/pi-terminal.ts` → `InteractiveMode`),
+none of which the headless web chat touches:
+
+| file | what |
+|---|---|
+| `theme/*.json` | TUI color themes (built-in dark/light fallback exists in code; shipped so custom theme reads and the schema resolve) |
+| `assets/*.png` | TUI startup art |
+| `export-html/template.html`, `export-html/vendor/*.js` | the `/export` command's HTML template + vendor JS |
+| `package.json`, `CHANGELOG.md` | version banner + startup changelog |
+| `native/darwin/prebuilds/<arch>/darwin-modifiers.node` | pi-tui's native modifier-key detection (degrades gracefully if absent) |
+
 Library resolution order (src/mlx/ffi.ts): `MLX_BUN_LIBMLXC` env var →
-`libmlxc.dylib` next to the executable → homebrew
+`libmlxc.dylib` next to the executable → the downloaded native-pack
+cache (`~/Library/Caches/mlx-bun/native-v<ver>-<arch>/`, populated on
+first run when no sidecar or homebrew install is found) → homebrew
 (`/opt/homebrew/lib`, `/usr/local/lib`). The whole directory can be
 renamed/moved; nothing references absolute paths after the build
-script's `install_name_tool` fixups.
+script's `install_name_tool` fixups. The expert helper independently resolves
+`MLX_BUN_EXPERT_IO_DYLIB`, then the executable directory, then the same
+native-pack cache.
 
 ## Embedded pi web chat (`/ws/chat`)
 
@@ -60,21 +77,28 @@ crashing), so it is best-effort, not load-bearing.
 > The web chat itself is unaffected (its provider is text-only), and the
 > wasm is shipped so it will work once Bun's support lands.
 
-**Intentionally omitted** (TUI-only assets pi ships beside its own binary,
-never reached by the headless web chat):
+## Embedded pi terminal (`mlx-bun pi`)
+
+The compiled binary also embeds pi's full interactive TUI
+(`src/pi-terminal.ts` → `InteractiveMode`), reachable via `mlx-bun pi`.
+Unlike the headless web chat, the terminal *does* resolve theme JSON,
+startup art, the `/export` template, the version banner, and the
+pi-tui native modifier-key helper by path — the second table above is
+sidecar'd for exactly this mode. `src/pi-web.ts`'s session (the web
+chat) still builds with `noThemes`/`noSkills`/`noExtensions`/
+`noPromptTemplates`/`noContextFiles`, so none of that table is reached
+from `/ws/chat` — only `mlx-bun pi` uses it.
+
+**Omitted** (upstream pi ships these beside its own binary; neither
+mlx-bun mode reaches them):
 
 | omitted asset | what it's for |
 |---|---|
-| `theme/*.json` | TUI color themes (loaded via `initTheme`; we pass `noThemes`) |
-| `assets/*.png` | TUI announcement/interactive images |
-| `export-html/*` | the `/export-html` command's template + vendor JS |
-| `docs/`, `examples/`, `README.md`, `CHANGELOG.md` | TUI help/doc browsing |
-
-These are gated behind TUI code paths and the explicit `exportToHtml()`
-method; `createAgentSession`'s import graph (`core/sdk.js`) pulls in no
-theme/asset/export-html loaders, and `src/pi-web.ts` builds the session
-with `noThemes`/`noSkills`/`noExtensions`/`noPromptTemplates`/
-`noContextFiles`.
+| `docs/`, `examples/`, `README.md` | upstream's own help/doc browsing, not exposed by either mlx-bun entry point |
+
+`createAgentSession`'s import graph (`core/sdk.js`) pulls in no loaders
+for these; both `src/pi-web.ts` and `src/pi-terminal.ts` build sessions
+that never reference them.
 
 **Verifying it.** `build-binary.sh` runs `scripts/verify-binary-pi.ts` as
 a sibling compiled binary inside the bundle (so `process.execPath` points

package/docs/reference/features-matrix.md

@@ -0,0 +1,99 @@
+# Features matrix — everything mlx-bun serves, in one table
+
+The complete option inventory: every serving feature, its default, which
+execution lane it runs on, which fidelity tier verifies it, and its knob.
+This is the *what-exists* view; per-flag detail lives in
+[server-config.md](server-config.md), wire formats in
+[server-api.md](server-api.md), CLI verbs in [cli.md](cli.md).
+
+**Lanes:** `serial` = the default single-queue path; `batch` = the
+`--batch N` continuous-batching engine (bf16 compat mode, mlx-lm B=N
+parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
+· Lab = no external oracle; KL/eval-gated experiments (see
+[server-config.md → Fidelity tiers](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2)).
+
+## Serving & performance
+
+| Feature | Default | Lane | Tier | Knob |
+| --- | --- | --- | --- | --- |
+| OpenAI chat completions (`/v1/chat/completions`, SSE) | on | both | — | — |
+| Raw text completions (`/v1/completions`) | on | both | — | — |
+| Anthropic Messages (`/v1/messages`) | on | both | — | — |
+| OpenAI Responses shim (`/v1/responses`) | on | both | — | — |
+| Continuous batching (mlx-lm B=N parity) | **on** — cap 8 (default flipped 2026-07-05; a lone request runs the exact serial engine, so the cap engages only under real concurrency; `--batch 1` pins serial) | batch | L1 | `--batch <n>` |
+| Prompt cache (prefix KV reuse) | on, 8 GB | both | — | `--prompt-cache <GB>` (0 = off) |
+| **SSD KV cold tier** (cache survives eviction + restarts; v3 includes GLM compressed MLA/DSA state without full-K/V reconstruction) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
+| **Runtime isolation** (engine runs as a crash-isolated CHILD process behind a reverse-proxy parent; UI stays instant under GPU load and survives engine crashes — [runtime-isolation.md](../design/runtime-isolation.md)) | off | both | — | `--isolate` |
+| **Model pool** (with `--isolate`: LRU-capped RESIDENT model engines; over the cap the least-recently-used engine demotes its cache to the SSD tier and exits, respawning on the next request) | 1 resident engine | both | — | `--model-pool <n>` (requires `--isolate`) |
+| Mixed-precision KV (`kv_config.json`, optiq's scheme) | **off** — opt-in (default flipped to bf16 2026-07-05; quantized KV trades 5–20% decode for memory headroom) | serial + **batch** (per-layer configs batch on every shipped model — full-attention layers Phase 3.1, rotating layers milestone 2; uniform bits stay serial) | L2 | `--kv-quant config\|off\|4\|8`, `--l2` |
+| **TurboQuant KV** (rotation-based: affine keys + FWHT/Lloyd-Max values) | off — opt-in ([turboquant-kv.md](../design/turboquant-kv.md); v1 dequantize-on-fetch, full-attention layers only, no speed claim) | serial only (solo-only, unconditionally) | Lab (codec is oracle-backed vs vllm-metal; the cache class is unvalidated) | `--kv-quant turbo[:k<bits>v<bits>]` (default `k8v3`) |
+| Compiled decode (bit-exact graph replay) | on, every tier | serial | L1/L2 | `--compiled-decode on\|off` |
+| Compiled activations (faithful geglu/swiglu — mlx-lm's `@mx.compile`) | **on**, every tier | both | L1 | `--compiled-activations on\|off` |
+| Fused SDPA (optiq-exact quantized-KV attention) | follows `--kv-quant`: on for `config`, off for uniform/bf16 | serial | L2 | `--fused-sdpa on\|off` |
+| **Paged KV** (vLLM-style block pool: host-side block table + gather before the stock SDPA, no fused kernel in v1 — [paged-kv-cache.md](../design/paged-kv-cache.md)) | off | serial (pins `--batch 1` unless `--batch` given) | gated bit-exact vs the plain `KVCache` (`tests/paged-kv-parity.test.ts`) | `--paged-kv`, `--paged-kv-block-size <n>` (default 256) — v1 scope Gemma4-family bf16; refuses `--batch N>1`, `--kv-quant`, `--draft-model` |
+| **Speculative decoding** (auto-detected drafter: two-model = mlx-lm parity/L1, Gemma `-assistant` = optiq/L2, DSpark local = Lab, DeepSpec released = DeepSpec-reference oracle) | off | serial (forces all-serial) | per-drafter oracle | `--draft-model <path\|query>`, `--draft-kind`, `--num-draft-tokens` |
+| **GLM-5.2 checkpoint-native MTP** (bounded auxiliary expert tier, exact verify) | **on** for GLM-5.2 | serial (`usage.lane: serial+spec`) | Colibri/oracle trajectory + synthetic HTTP gate | `--mtp on\|off` (`off` enables ordinary GLM batching) |
+| Speculative decoding, **model-free prompt lookup** (drafts copied from the request's own context; port of prompt-lookup decoding / vLLM `ngram`; zero weights) | off | serial (forces all-serial) | lossless by verify (gated vs non-spec greedy, tests/spec-ngram.test.ts) | `--draft-kind ngram` (no `--draft-model`), `--ngram-max`, `--ngram-min`, `--num-draft-tokens` (default 10) |
+| Memory admission (refuse what can't fit; never GPU-OOM) | on (RAM × 0.75) | both | — | `--memory-budget <GB>` |
+| Aggregate KV admission for batch rows (queue, don't OOM) | off | batch | — | `--kv-budget <GB>` |
+| Expert offload (MoE cold experts on mmap) | off | serial | Lab | `--expert-offload` |
+| Extend-join (O(1) batch admission) | on | batch | L1 (own oracle) | `MLX_BUN_BATCH_EXTEND=0` |
+| Vectorized greedy batch sampling | on | batch | bit-equal A/B | `MLX_BUN_BATCH_VEC_SAMPLE=0` |
+
+## Request features
+
+| Feature | Default | Lane | Tier | Knob |
+| --- | --- | --- | --- | --- |
+| **Structured output** (`response_format` json_object/json_schema) | on | both | L2 (oMLX) | request field; `MLX_BUN_GRAMMAR=0` kills |
+| Structured output **jump-forward** (grammar-forced spans emitted without per-token forwards; SGLang technique via xgrammar) | **off** | serial only | Lab (string-lossless + validity gates, tests/grammar-jump.test.ts; token stream may differ — no oracle) | `MLX_BUN_GRAMMAR_JUMP=1` |
+| `guided_grammar` (EBNF) / `guided_regex`¹ / `guided_choice` / `structured_outputs` | on | both | L2 | request fields |
+| Structured output × speculative decoding | on (when both active) | serial | Lab (validity+equivalence) | — |
+| Quantized KV × speculative decoding (any axis: uniform / config / turbo) | KV scheme wins — drafted requests decode serially WITHOUT speculation (spec lane is bf16-KV-only in v1; startup warning) | serial | — | omit `--kv-quant` to speculate |
+| Tool calling (Gemma sentinel / CPM+Qwen XML / GLM `arg_key`+`arg_value`) + `role:"tool"` loops | on | both | — | request `tools` |
+| Vision (`image_url` parts; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on (models with a tower); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial | L1/L2 | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
+| **Audio input** (`input_audio`/`audio`/`audio_url` parts; WAV native, mp3/m4a/flac/ogg/aiff via CoreAudio; ≤30 s / 750 tokens per clip; mixes with images) | on (models with `audio_config` + sidecar tower: e4b); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial³ | L2 (greedy stream EXACT vs the optiq internal model) | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
+| LoRA adapters (mount at start / hot-swap at runtime) | off | serial | — | `--adapter <dir>`, `POST /v1/adapters` |
+| Sampling: temperature / top-p / top-k / min-p / XTC / logit_bias / presence+frequency+repetition penalties | per-request | both | L1 (mlx-lm-faithful) | request fields / server defaults |
+| `logprobs` / `top_logprobs` (mlx-lm semantics) | off | serial | L1 | request fields |
+| Fixed `seed` reproducibility | off | serial (routes solo) | — | request field |
+| Thinking-mode control (hybrid-reasoning models) | model default | both | — | `--thinking`, `chat_template_kwargs` |
+| Stop sequences / streaming / usage accounting | on | both | — | request fields |
+| HLG tone-curve sampling | off | serial | Lab | `--hlg-sampling on` |
+| Spec-decode telemetry (`usage.speculation`) | on with a draft | serial | — | — |
+| Per-turn lane telemetry (`usage.lane`: serial / serial+spec / batched) | on | both | — | — |
+
+## Model coverage (per-model validated cells — no generic path is trusted untested)
+
+| Family | Serial | Batch | Notes |
+| --- | --- | --- | --- |
+| MiniCPM5 (cpm5) | ✅ L1/L2 | ✅ | the starter model |
+| Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
+| Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
+| GLM-5.2 / Colibri (pre-release port gate) | ✅ chat/text, Messages, Responses, SSE, tools, grammar, logprobs, library/CLI | ✅ compressed MLA/DSA scheduler | native MTP defaults to serial+spec; `--mtp off` enables ordinary batching; artifact-aware CLI/API/status `fit` and live main/MTP expert-tier telemetry; embeddings, vision/audio, adapters, and training are explicitly unsupported; per-row batched MTP is post-release |
+| DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
+| Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family/sliding universal → serial |
+
+## Beyond serving (the same binary)
+
+Training (`mlx-bun train`: ORPO/DPO/SFT, flash-CCE, segmented backward) ·
+quantization (`convert`, `fuse`, `--target-bpw`) · embeddings · local
+memory/wiki (`mlx-bun memory`) · pi agent integration (`mlx-bun pi`) ·
+model registry + `fit` planner + `gc` · HF `upload` · web chat Model Hub
+(`/api/hub/local` + `/search` + `/download`, per-row `/fit` verdict;
+`/api/hub/serve` is `restart_required`-only today — no in-process model-swap
+seam reaches `/ws/chat`, see [server-api.md](server-api.md#model-hub-apihub))
+· **PWA installability** (manifest + shell-only service worker —
+installable, instant static-shell paint; explicitly not offline chat, see
+[server-api.md](server-api.md#web-app-static-routes--pwa-installability)).
+See [cli.md](cli.md) and the per-topic reference docs.
+
+---
+¹ `guided_regex` accepts the regex∩EBNF subset today (no `\d`/anchors — those degrade to prompt injection); real regex support is tracked.
+² Gated token-exact vs mlx-lm B=2 (static + dynamic join/leave) on Llama-3.2-3B.
+³ Audio is a capability neither ancestor SERVES: mlx-lm strips the audio tower at load and 400s non-text parts; optiq ships the full machinery but never wires it into its serve frontend — mlx-bun exposes it. Serial lane by design (embeddings prefill; batched audio has no oracle → novel-territory gates would apply), prompt cache skipped.
+
+Composition rules and exclusions (what combines with what):
+[server-config.md → Compatibility matrix](server-config.md#compatibility-matrix).
+Performance per mode: `scripts/bench-modes.ts` (TTFT / prefill / decode /
+peak-memory across every cell above); feature-composition perf:
+`scripts/bench-feature-matrix.ts`.

package/docs/reference/library-api.md

@@ -3,33 +3,37 @@
 The HTTP server is one consumer of a library-first API. Everything it
 does — generation, sampling, KV schemes, LoRA, vision, caching, memory
 admission — is importable directly into a Bun process. Published as
-`mlx-bun` on npm (0.0.4); `bunx mlx-bun` works out of the box, or
+`mlx-bun` on npm; `bunx mlx-bun` works out of the box, or
 `import … from "mlx-bun"` for programmatic use.
 
-One rule everywhere: **one generation at a time per process** (one GPU,
-batch=1). The server serializes through a queue; do the same.
+One rule everywhere: **one generation at a time per process** (one GPU).
+The server serializes through a queue (or, under `--batch N`, its
+continuous-batching scheduler — a server-lane feature, not a library
+surface); direct library callers must do the same.
 
 ## Quick start
 
 ```ts
 import {
-  loadModelConfig,
-  Weights,
-  createModel,       // dispatches to Gemma4Model / MiniCPM5Model / Qwen35Model
+  ensureNativeRuntime,
+  openModel,         // artifact-aware dispatch, including bounded GLM-5.2
   loadTokenizer,
   ChatTemplate,
   generate,
 } from "mlx-bun";   // or "./src/index" in-repo
 
+await ensureNativeRuntime();
 const dir = "/path/to/hf-snapshot";          // mlx-bun ls prints these
-const config = await loadModelConfig(dir);
-const model = createModel(await Weights.open(dir), config);   // returns RuntimeModel
+const model = await openModel(dir, {
+  // Used only by the direct Colibri GLM-5.2 artifact; other models ignore it.
+  contextTokens: 4096, maxGenerationTokens: 256, enableMtp: false,
+});
 const tok = await loadTokenizer(dir);
 const template = await ChatTemplate.load(dir);
 // Gemma4Model and MiniCPM5Model can still be imported directly from "mlx-bun"
 // if you need the concrete type. Qwen35Model is NOT exported from the public
-// package — use createModel/RuntimeModel (preferred dispatch), or import from
-// "./src/model/qwen35" in-repo.
+// package — use openModel/RuntimeModel (preferred artifact-aware dispatch), or import from
+// "./src/model/qwen3_5" in-repo.
 
 const ids = tok.encode(template.render([{ role: "user", content: "hi" }]));
 const promptIds = ids[0] === ids[1] && ids[0] === tok.bosTokenId ? ids.slice(1) : ids;
@@ -41,8 +45,18 @@ console.log(tok.decode(out, true));
 console.log(gen.stats);                          // set once iteration ends
 ```
 
-Weight loading is lazy (mmap + mlx native loader): construction is
-milliseconds; weights materialize on first forward.
+Ordinary safetensors loading is lazy (mmap + mlx native loader). GLM-5.2 first
+runs its header-only exact process equation, then opens the bounded Colibri
+resident/expert tiers; impossible plans fail before either tier is committed.
+
+> **First run:** library consumers bypass the CLI's first-run step, so
+> call `await ensureNativeRuntime()` once before constructing a model on
+> a machine that may not have the MLX native runtime yet — it downloads
+> the sha256-verified native pack (MLX plus the direct-Colibri expert-I/O
+> helper) to `~/Library/Caches/mlx-bun/` and is
+> a no-op when the runtime is already present (beside the executable,
+> already cached, or installed via homebrew). `nativeRuntimeDir()`
+> returns the resolved directory, or `null` on a fresh machine.
 
 ## generate(model, promptTokens, options) → Generation
 
@@ -64,13 +78,20 @@ after an early `break`, which cleanly cancels the in-flight step).
 | `kvConfig` | off | per-layer mixed precision from `kv_config.json` (`config.kvQuant`); overrides `kvBits`, start 0 — optiq serve semantics |
 | `adapters` | none | mounted LoRA adapter ids, applied for exactly this generation |
 | `promptEmbeddings` / `imageMask` | — | vision path (see `src/vision/`) |
+| `logprobs` / `topLogprobs` | off | per-token logprob capture (mlx_lm.server parity): `logprobs` captures the emitted token's log-probability, `topLogprobs` (0/unset = off) captures the top-k (id, logprob) pairs — same full-vocab log-softmax distribution mlx-lm's generate_step samples from, taken after logits processors and before the sampler |
+| `grammar` | off | a compiled `GrammarController` (`src/grammar.ts`) for structured output — masks invalid tokens to `-inf` each step (L2-class, oMLX oracle) |
+| `turboQuant` | off | TurboQuant KV scheme (docs/design/turboquant-kv.md), a rotation-based alternative to uniform `kvBits`; mutually exclusive with `kvBits`/`kvConfig` |
+| `pagedKv` | off | `{ blockSize? }` — OPTIONAL paged KV cache (docs/design/paged-kv-cache.md); v1 scope is serial batch=1 Gemma4-family bf16, mutually exclusive with `kvBits`/`kvConfig`/`turboQuant`/draft/compiled decode |
 
 `GenerateStats`: `promptTokens`, `cachedTokens`, `generatedTokens`,
 `prefillTps` / `prefillMs`, `decodeTps` / `decodeMs`, `cacheTokens`
 (the exact token sequence whose KV is in the cache — feed it to
 `PromptCache.put`). Timing semantics match mlx-lm: the prompt clock
 runs until the first token arrives (the prefill→decode boundary is
-prompt time), the decode clock from there.
+prompt time), the decode clock from there. An optional `spec` field
+carries speculative-decoding telemetry (serve `--draft-model` path
+only): `drafted`, `accepted`, `targetCalls`, and per-draft-position
+`draftedByPos` / `acceptedByPos` counts.
 
 ## Serving pieces
 
@@ -80,7 +101,7 @@ import { loadContext, createServer } from "mlx-bun";
 const ctx = await loadContext(dir, "my-model", { memoryBudgetBytes: 12e9 });
 const server = createServer(ctx, 8080, {
   promptCacheBytes: 2e9,        // byte-capped LRU (never count-capped)
-  kvQuant: undefined,           // default: kv_config.json; "off" | bits
+  kvQuant: undefined,           // default: kv_config.json; "off" | "config" | bits
   memoryBudgetBytes: 12e9,      // admission control — the only OOM defense
 });
 ```
@@ -90,6 +111,24 @@ budget (pre-GPU, mmap-only check). The server exposes OpenAI chat
 completions, Anthropic `/v1/messages`, OpenAI Responses, embeddings,
 adapters, and `/stats` — [server-api.md](./server-api.md).
 
+For GLM-5.2, pass its load-only resource controls under `glm`:
+
+```ts
+const ctx = await loadContext(dir, "glm-5.2", {
+  glm: {
+    contextTokens: 4096,
+    maxGenerationTokens: 128,
+    batchSize: 8,
+    enableMtp: true, // default; native MTP routes requests serial+spec
+  },
+});
+```
+
+`openModel()` and `loadContext()` both select the streamed Colibri runtime;
+callers do not use `Weights.open()` or `createModel()` for this artifact.
+Direct `generate()` supports the full sampling/grammar/logprobs contract but
+does not itself mount a drafter; native MTP is a serving-context feature.
+
 ## Text embeddings
 
 ```ts
@@ -135,14 +174,24 @@ under one adapter must not seed another's prefill).
 ```ts
 import { saveKvCache, loadKvCache } from "./src/kv-store";
 saveKvCache("/tmp/prefix.kv", tokens, caches);       // page-aligned file
-const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model as Gemma4Model);
-// reload is a zero-copy MAP_PRIVATE mmap straight to the GPU (~1 ms);
-// keep `mmap` referenced as long as the caches live
-// Note: loadKvCache is typed to accept Gemma4Model, but only uses model.layers.length
-// at runtime — a cast from RuntimeModel is safe for any model that has a layers array.
+const { tokens, caches } = loadKvCache("/tmp/prefix.kv", model, {
+  modelId, configFingerprint, tokenizerHash, verify: true,
+});
+// Restore streams one tensor at a time from a read-only mmap into MLX-owned
+// storage, drops each clean file-page range, and unmaps before returning.
+// Note: loadKvCache accepts anything with makeCache(): Cache[] — it validates
+// the entry count against the cache list makeCache() returns (not
+// model.layers.length, which is wrong for KV-shared models like e4b, whose
+// makeCache() returns donors only). RuntimeModel qualifies directly, no cast.
 ```
 
-Quantized caches are not persistable yet (documented gap).
+Format v3 supports plain, rotating, quantized, TurboQuant, SSM, and GLM-5.2
+checkpoint-native caches. GLM rows use `mla`, `mla-dsa`, and `mtp-mla` kinds and
+store only compressed latent/RoPE/DSA state—never reconstructed full K/V.
+`Glm52NativeMtpSource.clonePersistentCache()` returns an owned MTP snapshot;
+`restorePersistentCache()` adopts a validated restored `mtp-mla` row. Writes
+are page-aligned, hashed, atomic (`.tmp` + fsync + rename), and available in
+both synchronous and event-loop-yielding `saveKvCacheAsync` forms.
 
 ## Registry + fit (model discovery and memory math)
 

package/docs/reference/memory.md

@@ -42,18 +42,38 @@ mlx-bun memory unschedule
 
 ## Agent tools
 
-Both `mlx-bun pi` and the web chat expose these read-only tools:
+Both `mlx-bun pi` and the web chat expose these read-only tools, in the recommended call order — FIND the article, READ it small, follow the graph, fall back to reference docs and utilities, search only as a last resort:
 
-- `memory_search`
-- `memory_read`
-- `memory_toc`
-- `memory_section`
-- `memory_links`
+**FIND** (deterministic lookup, never a vector search):
+- `memory_resolve` — name/alias → the article
+- `memory_category` — category/type/series → members
+
+**READ** (TOC → one section, not a full-article dump):
+- `memory_read` — TOC + lead by default; `force=true` for the full article
+- `memory_section` — one section's body (the default granularity)
+
+**Follow the graph**:
+- `memory_links` — outbound/inbound wikilinks
+- `memory_infobox` — read an article's infobox fields (there is no infobox *query* tool — the infobox is content the model reads, not a filter)
+
+**Reference docs** (mlx-bun's own docs, mirrored read-only into the vault):
+- `reference_search`
+- `reference_read`
+- `reference_list`
+
+**Utility**:
 - `memory_list` — user articles plus read-only `Reference/*` docs
 - `memory_status` — vault path, setup state, article count, git state, last synthesis run (from the vault's git log), schedule state, recent changed articles
 
+**Last resort**:
+- `memory_search` — substring fallback when FIND failed
+
 The tools are always registered. If no vault exists, they return a setup message instead of disappearing. Search/read results prefixed with `Reference/` are mlx-bun docs, not user memory articles.
 
+## Web chat REST surface
+
+The web chat's Memory panel talks to a separate, HTTP-only `/api/memory/*` route surface (status/list/search/article/links/history/diff + a `POST /api/memory/init` consent-card endpoint) — thin wrappers over the same vault read path above, documented in [server-api.md](server-api.md#get-apimemory--post-apimemoryinit). The agent-tool surface above is unaffected by it.
+
 ## Obsidian
 
 ```bash

package/docs/reference/models.md

@@ -35,6 +35,11 @@ mlx-bun get <substring>          # no "/" = registry query, re-gets the match
 - Plain HTTPS resolve/CDN (no Xet), sequential files, `Range`-resume of
   partial blobs (`<blob>.incomplete`). Auth: `HF_TOKEN` env, then the
   token `hf auth login` writes. 401/403 answers get a "gated repo" hint.
+- Before the first payload request, mlx-bun totals the bytes still absent,
+  credits complete shared blobs and valid `.incomplete` prefixes, and checks
+  the target volume for that remainder plus a fixed 1 GiB safety reserve. An
+  impossible acquisition fails immediately with required/available GiB and a
+  rerun/resume instruction rather than filling the disk halfway through.
 - A substring argument (no `/`) resolves against already-downloaded
   repos — `mlx-bun get 12B` refreshes the 12B to upstream's latest. An
   unknown substring errors with a pointer to `mlx-bun ls <q>`.
@@ -51,14 +56,56 @@ mlx-bun get <substring>          # no "/" = registry query, re-gets the match
 Every blob is checksummed **while streaming** (a resume re-hashes the
 existing prefix in chunks — no whole-file allocation):
 
-- **LFS files** (weights): sha256 must equal the API's `lfs.oid` — which
-  is also the blob's filename.
+- **LFS files** (weights): sha256 must equal the API's current `lfs.sha256`
+  field (the older `lfs.oid` spelling is also accepted) — which is also the
+  blob's filename.
 - **Small files** (configs, tokenizer): git blob identity,
   `sha1("blob <size>\0" + content)`, must equal the API's `blobId`.
 
 A mismatch deletes the partial (never resume corrupt bytes); a short
 read keeps the `.incomplete` for resume.
 
+## GLM-5.2 on a 32 GB Mac
+
+The validated direct-container artifact is
+`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp` at revision
+`3cc8db99b1b13fc79325d987ba3c1c430766b3b8`. It is approximately 384 GB
+decimal / 357 GiB on disk, so choose the Hugging Face cache volume before the
+download. Fast internal NVMe is strongly preferred; an external volume also
+needs to sustain the expert-streaming workload after acquisition.
+
+```sh
+# Optional: choose a large fast volume before downloading.
+export HF_HUB_CACHE=/Volumes/FastSSD/huggingface/hub
+
+mlx-bun get mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
+  --revision 3cc8db99b1b13fc79325d987ba3c1c430766b3b8
+mlx-bun scan
+mlx-bun serve GLM-5.2-colibri \
+  --memory-budget 26.8435456 \
+  --context-length 4096 \
+  --max-tokens 128 \
+  --mtp on
+```
+
+`26.8435456` decimal GB is exactly 25 GiB. The validated plan reserves about
+19.9 GiB for the process and leaves the rest of the 32 GiB machine to macOS.
+Startup runs the exact header-only resource equation before opening resident
+weights or expert slabs. The server reports the resulting plan at
+`GET /stats` under `glm52`.
+
+If acquisition is interrupted, rerun the identical `get` command: mlx-bun
+rehashes each existing prefix in bounded chunks, sends a Range request for the
+remainder, and verifies the complete blob before publishing it. If the disk
+preflight refuses, free space or point `HF_HUB_CACHE` at another volume and
+rerun. No Python environment and no converted copy inside the mlx-bun checkout
+are involved.
+
+The artifact is an external MIT-licensed derivative of
+`zai-org/GLM-5.2-FP8`, converted with Colibri and extended with int8 MTP
+weights. Model weights are not bundled with mlx-bun; review the artifact model
+card before redistribution.
+
 ## scan + the registry
 
 `mlx-bun scan` walks `models--*/snapshots/*` reading only `config.json`
@@ -79,6 +126,8 @@ Schema (table `models`, one row **per snapshot**, keyed on `path`):
 | `quant_bits`, `quant_group_size`, `quant_mode` | quantization |
 | `has_vision_sidecar` | sidecar file present |
 | `vision_config_type` | `config.json`'s `vision_config.model_type` when it names a vision tower (`*_vision`) |
+| `has_audio_config` | `config.json` declares an `audio_config` block |
+| `has_audio_tower` | sidecar header names `audio_tower.*` tensors |
 | `has_kv_config`, `has_tool_template`, `license` | capabilities/terms |
 | `scanned_at` | freshness (canonical tie-break) |
 
@@ -89,6 +138,14 @@ Schema (table `models`, one row **per snapshot**, keyed on `path`):
 Presence of a `vision_config` key alone is **not** a signal — Qwen3.5
 nests a copy of its own text config there.
 
+**Audio capability** is `has_audio_config AND has_audio_tower`: both
+legs are required because the 12B OptiQ snapshot carries an
+`audio_config` but a stub sidecar whose only audio entry is
+`embed_audio.embedding_projection.weight` (no `audio_tower.*` Conformer
+tensors) — config presence alone would advertise audio that fails at
+request time. The check reads only the sidecar's safetensors header,
+never tensor bytes.
+
 ## Query resolution rules (per verb)
 
 Every verb resolves fuzzy queries the same way, with revision collapsing
@@ -98,7 +155,7 @@ so a stale snapshot never makes a repo "ambiguous":
   row per repo** (the canonical revision). `--all-revisions` shows the
   per-snapshot truth with the canonical marked `*`. Capabilities column
   labels the support tier: `supported (targeted)` (dedicated forward,
-  L2/L3 paths) vs `supported (generic)` (Tier-0 universal module).
+  L2/opt-in paths) vs `supported (generic)` (Tier-0 universal module).
 - **`fit` / `serve` / `train` / … (single-model verbs)** — `resolve()`:
   the query must match exactly one **repo**; multiple cached revisions of
   that repo collapse to the canonical snapshot (refs/main, else most

package/docs/reference/orpo-quickstart.md

@@ -14,9 +14,9 @@ mlx-bun train <model> --data <dir>
 - `<model>` is a query or snapshot path (e.g. `e4b`, `MiniCPM5`); omit it to auto-pick.
 - `--data <dir>` is a directory with `train.jsonl` (+ optional `valid.jsonl`); rows are
   `{"prompt": "...", "chosen": "...", "rejected": "..."}`.
-- The command **auto-detects e4b/Gemma** and sets its required training env flags
-  (`MLX_BUN_PERF_KERNEL=0`, `MLX_BUN_FUSED_GELU=0`) before loading the model — you don't
-  set them yourself. The full stack (flash head + prefix-share + segmented) is on by default.
+- The full stack (flash head + prefix-share + segmented) is on by default. No env-flag
+  setup is needed for any model (the old Gemma training-flag requirement is gone —
+  the no-vjp kernels were deleted 2026-07-05).
 - It prints a plan box (dataset counts, format, the resolved knobs), streams per-step loss
   + peak memory, and saves a **mountable** adapter (`mlx-bun serve <model> --adapter <dir>`).
 - `--dry-run` inspects the dataset and prints the plan without training. `mlx-bun help train`
@@ -59,7 +59,6 @@ mlx-bun train e4b --data ./prefs --resume ~/.cache/mlx-bun/mlx-bun-finetunes/orp
 MODEL=/path/to/snapshot DATA=/path/to/datadir bun scripts/train-orpo.ts
 ```
 - `DATA` is a directory with `train.jsonl` (+ optional `valid.jsonl`).
-- Auto-detects e4b/Gemma and sets the training env flags, same as the CLI.
 
 ### Preconfigured defaults (override via env)
 | env | default | meaning |