mlx-bun 0.0.10 → 0.0.12

package/docs/planning/release-notes-v0.0.11.md

@@ -0,0 +1,239 @@
+# v0.0.11 release notes
+
+Everything since v0.0.10. The big themes: mlx-bun grows a second modality
+(audio), a constrained-decoding stack (structured output / grammar),
+speculative decoding, a novel KV-cache quantization scheme (TurboQuant),
+completed dynamic batching, crash-isolated serving, and a ground-up
+rewrite of the web chat UI. The decode-default story also resets: after
+benchmarking every custom performance kernel in a paired A/B against the
+bit-exact baseline and finding none of them won, they were deleted —
+naked `mlx-bun serve` now runs the faithful, mlx-lm-parity kernel set by
+default.
+
+## Audio input (Gemma e4b)
+
+- OpenAI-style `input_audio` parts are served end-to-end: WAV/AAC
+  transcode, a USM mel-spectrogram extractor, and a Conformer tower
+  ported bit-exact (rel-RMSE 2.4e-8) against the oracle.
+- Build-out ran phase-by-phase (A0–A5): a `bun:ffi` conv2d binding for
+  the front-end, the multimodal prompt builder wired to the LM, and
+  capability discovery for audio surfaced consistently across
+  `/v1/models`, `/library`, and the pi websocket handshake — mirroring
+  how vision capability is advertised.
+
+## Structured output / grammar-constrained generation
+
+- `response_format` (`json_object`, `json_schema`, `text`) and
+  `guided_grammar`/`guided_regex`/`guided_choice`/`structured_outputs`
+  are wired into both `/v1/chat/completions` and `/v1/completions`, built
+  on `xgrammar`: a `GrammarController` masks logits inside the decode
+  loop, with a per-`TokenizerInfo` compiler cache and a `MLX_BUN_GRAMMAR=0`
+  kill switch.
+- The batched lane got its own per-row matcher path so grammar-constrained
+  requests don't force a request onto the serial lane, and grammar
+  composes with `--draft-model` (the verify walk respects the mask).
+- Two defects found and fixed during the build: the oMLX-parity
+  JSON-system-prompt degrade path (used when a client can't do function
+  calling) had gone unreachable behind a dropped hint, and the WASM
+  grammar controllers were leaking on every early-400 reject path
+  (SSRF-blocked media, vision/audio validation failures, bad
+  `logit_bias`, etc.) — both fixed, with disposal now covering all paths.
+
+## Speculative decoding
+
+- `--draft-model` lands in serve: a two-model, serial-lane speculative
+  decoding path verified token-for-token against mlx-lm's own
+  `--draft-model` oracle (`DraftSource` seam, rewind rule, batched-verify
+  lm-head).
+- An ngram / jump-forward draft kind ships alongside the model-based
+  drafter for cases with no small model available.
+- A DeepSpec (DSpark) drafter-quant research track is underway
+  (Phase 1 landed) targeting the larger models; per
+  `benchmarks/RESULTS.md`, the first live pair (12B target +
+  bf16 DeepSpec drafter, 26–33% acceptance) measured net **slower**
+  than serial on a loaded box — spec decoding stays **off by default**
+  pending a quantized-drafter pair that clears the 1.3× bar.
+
+## TurboQuant KV-cache quantization
+
+- `--kv-quant turbo[:kXvY]` adds a rotation-based per-group KV
+  quantization scheme (sign-FWHT + Lloyd-Max for values), proven
+  bit-exact against the vendored vllm-metal reference
+  (`goldens/turboquant.json`).
+- Per `benchmarks/RESULTS.md`, the k8v3 default holds a lower KL-vs-bf16
+  than affine uniform kv4 while compressing KV **2.56×** (measured on
+  MiniCPM5-1B, paired quality/KL harness). It's a memory/context lever,
+  not a speed lever — v1 dequant-on-fetch is slower per decode step at
+  long context, so it stays **opt-in, not default**, per the project's
+  "levers must earn defaults" rule.
+
+## Batching: Phase D completed
+
+- Aggregate `--kv-budget <GB>` admission with FIFO queuing, vectorized
+  homogeneous sampling (penalties/`logit_bias`/`min_p`/XTC all run
+  per-row in the batch), and extend-join so a request can join an
+  in-flight batch mid-flight instead of waiting for a full drain.
+- Per-row RoPE lands for Tier-0 universal architectures, lifting the
+  batching gate for those models (e.g. Llama).
+- Batched mixed-precision quantized KV — the first stack to compose
+  per-row KV quantization with batching.
+- `--batch` default changes from 1 to 8 (see Breaking changes below).
+
+## Runtime isolation
+
+- `--isolate` runs the engine as a crash-isolated child process behind a
+  reactor parent, restarting with backoff on a crash — the product rule
+  is **"the AI may crash, the UI never may"** (`docs/design/runtime-isolation.md`).
+- `--model-pool` extends this to a child-per-model pool: LRU residency,
+  spawn-overlap switching (no dead time swapping models), and
+  `POST /admin/drain` for graceful eviction.
+
+## Optional paged KV cache
+
+- `--paged-kv` adds a vLLM-style block-table paged KV cache
+  (`PagedKVCache` + `BlockPool`) feeding the unchanged stock SDPA via
+  block gather. Default off; v1 scope is serial `--batch 1`, the
+  Gemma4 family, bf16 — it refuses to silently combine with
+  `--batch N>1`, `--kv-quant`, or `--draft-model`.
+
+## Prompt cache overhaul
+
+- Prefix-sharing v1: `take()` now returns non-consuming, ref-counted
+  clones instead of consuming/trimming the donor entry, so multiple
+  agents or sessions can reuse one prefill without cannibalizing each
+  other's cache.
+- The SSD cold tier is promoted from an add-on to a first-class property
+  of the cache store, with `--ssd-demote-idle <seconds>` (default 300,
+  requires `--ssd-cache`) freeing GPU memory by demoting idle entries;
+  `/stats` gains `ssd_cache.demotions`.
+- A stable-boundary snapshot fix restores prompt-cache hits for
+  multi-turn agent traffic beyond the sliding window — this was
+  previously a silent cache miss on turn 2+ of long conversations.
+- The SSD write-behind flush is now idle-gated, so background disk
+  writes stop contending with decode at long context; the earlier A7
+  RSS regression on the ssd-cache path closed with zero-copy write views
+  and a streamed copy-restore.
+
+## Web chat UI: ground-up redesign (Phases 0–3)
+
+- Streaming render rewritten to be block-memoized (no more O(n²)
+  re-parse of the growing message), with a typed module split.
+- New surfaces: a memory panel, adapter routing, a system-prompt editor,
+  an approval gate, client-side RAG over chat history (BM25, no server
+  round-trip), a Model Hub, Canvas, per-message sampling controls,
+  self-healing tool calls, a command palette, full-text search, and
+  chat export.
+- An app-aware assistant (v1) that can see and navigate its own UI —
+  spotlight/selector resolution now treats a model-invented or malformed
+  selector as a miss instead of throwing inside the websocket handler.
+- PWA support, plus a round of visual-QA fixes: the phone nav band,
+  light-theme code legibility, a composer placeholder regression, memory
+  panel markdown chrome, the theme-toggle Auto button, a clipped status
+  pill, and tab flex-shrink pinned to zero on phones.
+
+## Security: media-fetch SSRF hardening
+
+- `image_url`/`audio_url` fetches now refuse private/loopback/link-local/
+  CGNAT destinations and non-`http(s)` schemes by default, re-validate
+  through up to 5 redirect hops, enforce a 10s wall-clock timeout, and
+  cap bodies at 64MB.
+- LAN use needs the new escape hatch: `--allow-private-media` /
+  `MLX_BUN_ALLOW_PRIVATE_MEDIA=1`.
+
+## Fixed
+
+- 12B `/v1/completions` probe parity: adopted the oracle's step-0
+  prefill convention (drain to len−1, step 0 from an `L=1` forward of the
+  last prompt token). Per `benchmarks/RESULTS.md`, live HTTP probes are
+  now byte-identical to `mlx_lm.server`/optiq serve for MiniCPM5-1B,
+  gemma-4-e4b, and gemma-4-12B, on completion and chat probes, in both
+  the `--batch 8` default lane and `--batch 1`.
+- BPE detokenization now mirrors mlx-lm's bare-space hold-back and
+  never-finalize semantics exactly.
+- Cache invariant enforced on every tier; an FFI destructor deadlock
+  fixed; write-behind made non-blocking; per-row batch cache extraction
+  fixed.
+- `generate()` now yields a macrotask between prefill chunks, keeping the
+  isolation path (and the server generally) responsive during long
+  prefills.
+- Benchmark harness: the `--batch 1` serial control arm now runs by
+  default (it had been skippable, which undermined validating the new
+  batch-8 default); `--engine` is additive instead of mutually exclusive
+  with the serve pass; the batch-path test now asserts cumulative
+  `/stats` `submitted_rows` instead of a stale `cachedTokens=0` premise.
+- `any_whitespace` grammar knob landed; batch-grammar gates re-anchored
+  to compact separators.
+
+## Infra / benchmark harness redesign
+
+- `benchmark.sh` + `scripts/bench-modes.ts` now spawn real CLI paths
+  instead of bench-local server wrappers, folding bit-parity and
+  peak-memory checks into the same serve cells (one pass instead of
+  several). New columns: cold-start and restart-survival; new workload:
+  `xl` (~4k prefix); a feature-matrix benchmark mode.
+- Repo cleanup: `lab/` consolidation, a binary-tracking hygiene gate, and
+  a git history rewrite that took the repo from 182MB to 20MB, removing
+  the last multi-MB binaries from the index.
+- New design docs: `unified-engine-frontier-plan`, `dspark-serving-program`
+  (the DeepSpec/DSpark spec-decode research program), `structured-output`,
+  `web-chat-redesign` (+ app-aware-assistant, superset-doctrine, and
+  visual-QA-bar notes), `paged-kv-cache`, `batching-perf-path` /
+  `batching-v2-plan`, `runtime-isolation`.
+- Doc drift swept: env-lever table entries (`MLX_BUN_DSPARK_MINCONF`,
+  `MLX_BUN_BATCH_NO_PIPELINE`, `MLX_BUN_EXPERT_OFFLOAD`), ngram
+  draft-kind coverage in README/server-api, doc-map sync, an openwiki
+  investigation write-up. STATUS.md was rewritten to current-state-only
+  several times through the release to keep handoff notes accurate.
+- A new ground rule was recorded in CLAUDE.md: changes to the served
+  surface must update the user-facing reference docs in the same commit.
+- A direct optiq mixed-KV bit-parity golden replaced an indirect
+  ours-vs-ours comparison; a bit-exact L1 parity harness was added for
+  the Qwen3-30B-A3B MoE port, with uniform quantized KV confirmed
+  bit-exact against mlx-lm's own `--kv-bits` and a stock-mlx-lm baseline
+  added to the bench.
+- Dependency/CI bumps: pi, `actions/checkout`, and the site build moved
+  off a deprecated Node runtime.
+
+## Breaking / behavior changes
+
+- **`--batch` default changes from 1 to 8.** Serving now batches by
+  default whenever concurrent requests arrive; pass `--batch 1` to force
+  strict serial (needed if you require arrival-order-independent,
+  bit-identical numerics).
+- **The default decode path changed.** Naked, no-flag serving now runs
+  the faithful, bit-exact-vs-mlx-lm kernel set — what `--l1` used to
+  mean. `--l1`/`--l2`/`--l3` are now pure aliases over per-feature flags
+  rather than distinct performance tiers. Every custom-kernel path that
+  didn't beat this baseline in a paired A/B was deleted in the same
+  pass: the fused-decode kernel, fused-gelu/fused-swiglu/fused-mlp/
+  steel-linear Metal kernels, the flash-decode "perf" kernel and its
+  frozen-oracle test scaffolding, `--l3`, and `FaithfulMiniCPM5` /
+  `MLX_BUN_CPM5_FAITHFUL`. None of these flags or code paths exist
+  anymore. `MLX_BUN_FAITHFUL` is also gone (superseded by the faithful
+  kernels being the default) — the server's unset `--kv-quant` now
+  defaults to bf16 instead of `kv_config`, so getting the old default
+  composition requires an explicit `--kv-quant`.
+- **New CLI/API surface:** `--kv-quant turbo[:kXvY]` (TurboQuant),
+  `--paged-kv` (off by default; refuses to combine with `--batch N>1`,
+  `--kv-quant`, or `--draft-model`), `--draft-model` (speculative
+  decoding), `--isolate` and `--model-pool` (runtime isolation),
+  `--kv-budget <GB>` (aggregate KV admission), `--ssd-demote-idle
+  <seconds>` (default 300, requires `--ssd-cache`),
+  `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1`, and the
+  `response_format` / `guided_grammar` / `guided_regex` / `guided_choice`
+  / `structured_outputs` request fields (kill switch:
+  `MLX_BUN_GRAMMAR=0`).
+- **Media fetches are restricted by default.** `image_url`/`audio_url`
+  now refuse private/loopback/link-local/CGNAT destinations and
+  non-`http(s)` schemes — previously they fetched anything. LAN media
+  needs the new escape hatch above.
+- **`PromptCache.take()` semantics changed.** Cache hits now return
+  non-consuming, ref-counted clones instead of consuming/trimming the
+  donor entry — code holding onto the old consuming behavior should
+  re-check callers.
+
+Full details and measured numbers: STATUS.md, `benchmarks/RESULTS.md`,
+and the design docs referenced above. Credit where due: TurboQuant's
+codec is verified bit-exact against the vendored vllm-metal reference,
+and the speculative-decoding path is verified token-for-token against
+mlx-lm's own `--draft-model` implementation.

package/docs/planning/release-notes-v0.0.12.md

@@ -0,0 +1,80 @@
+# v0.0.12 release notes
+
+mlx-bun 0.0.12 completes the native Colibri/GLM-5.2 program: a Python-free,
+direct-container runtime that can serve the approximately 357 GiB artifact on
+a 32 GiB Apple Silicon Mac by keeping a bounded working set in memory and
+streaming routed experts from fast storage.
+
+## GLM-5.2 direct-container serving
+
+- Loads `mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp` directly. No converted
+  copy and no Python process are involved in the serving path.
+- Preserves checkpoint precision and true top-8 routing. The port is gated
+  against pinned Colibri at tiny, production-component, real-model, MTP, cache,
+  and first-sparse DSA boundaries.
+- Native int8 MTP is enabled by default and uses the serial speculative lane.
+  It produced a measured 1.306x warm end-to-end win over MTP-off on the 32 GiB
+  reference machine while preserving all target tokens.
+- Compressed MLA/DSA/MTP cache persistence, restart restore, compressed-byte
+  admission, ordinary continuous batching, mixed-length join/leave/cancel, and
+  cross-row expert union are included. Per-row batched MTP remains a later
+  feature; MTP requests truthfully report `serial+spec`.
+- Chat completions, text completions, Anthropic Messages, OpenAI Responses,
+  SSE, tools, structured output, stops, sampling, usage, and serial logprobs
+  are supported. Embeddings, vision/audio, LoRA, and training are explicitly
+  reported unsupported for this model instead of being emulated.
+
+## Memory and acquisition contract
+
+- `mlx-bun get` now computes the exact remaining payload before the first
+  payload request, credits verified shared blobs and resumable prefixes, and
+  refuses an acquisition that cannot leave a fixed 1 GiB safety reserve.
+- The GLM header-only preflight runs before resident weights or expert slabs
+  open. With the stock DSA overlay installed, 4,096 context / 128 generation
+  tokens and MTP on produce a 19.89 GiB plan under a 25 GiB process ceiling,
+  leaving 5.11 GiB of process/macOS headroom on a 32 GiB machine.
+- `fit`, `/fit`, `/stats`, and the status page now distinguish the complete
+  artifact on disk from resident weights, main/MTP slabs, compressed KV, and
+  reserves. They also expose main/MTP residency, hit rate, SSD bytes, policy
+  telemetry, and repin events.
+
+## Performance: measured, not predicted
+
+- Warm quality-preserving throughput on the M1 Max 32 GiB reference machine is
+  0.1577 decode tok/s / 0.1487 end-to-end tok/s with MTP on. The same-machine
+  direct-Colibri control is approximately 0.27 tok/s.
+- The 2 tok/s figure remains an aspiration, not a release gate or `fit`
+  prediction. The measured warm end-to-end result is 13.45x below it.
+- Replicated telemetry measured 1,974,949,363,712 logical expert bytes per
+  128-token warm turn (15.429 decimal GB/token) and a 1.66% cache hit rate.
+  Startup auto-pin, live LFRU, PILOT hints, two-step prediction, and coupling
+  did not improve paired end-to-end performance and remain off by default.
+- The checkpoint's 21-full/57-shared DSA schedule is implemented and exact.
+  Paired 2K/8K tests did not establish an MTP-on product-speed win, so this
+  release makes no DSA speed claim. The public Colibri artifact does not ship
+  the stock DSA indexer overlay; without it the runtime follows the artifact's
+  available attention path.
+
+## Distribution and operator experience
+
+- First-run native pack v0.2.0 adds the bounded native expert-I/O helper. The
+  npm/bunx fresh-cache path downloads, verifies, and extracts all five runtime
+  files; the signed/Homebrew bundle includes the same helper beside the binary.
+- The pinned 32 GiB quickstart, volume selection, interrupted-download recovery,
+  model lineage, licenses, resource plan, measured performance, and limitations
+  are documented in README and the reference guides.
+- The interactive Atlas report reproduces Colibri's expert topic-affinity
+  result: 29/30 leave-one-prompt-out classifications (96.7%) and 1,065 strong
+  specialists among 13,236 replicated experts.
+
+## Stability fixes since v0.0.11
+
+- Closed pre-Colibri correctness and lifecycle findings, including expert-slab
+  post-close and lazy-graph reuse hazards, with forced native/MLX churn gates.
+- Fixed selected-logprob readback stalling and revalidated the audio splice
+  boundary against the oracle.
+- Corrected current Hugging Face LFS checksum-field handling and retained the
+  older spelling for compatibility.
+
+Full provenance and exact numbers are in `benchmarks/RESULTS.md`, `STATUS.md`,
+and `docs/design/colibri-glm52-port.md`.

package/docs/reference/cli.md

@@ -11,20 +11,22 @@ candidates — just make it more specific.
 Bare `mlx-bun` with no verb runs `serve` — the appliance path: first run
 downloads a starter model, serves it, and opens the chat UI.
 
-> Verbs marked **(v0.0.9)** ship in v0.0.9; everything else is in v0.0.8.
+> All verbs below ship in current releases; the release notes in
+> `docs/planning/` record when each verb first appeared.
 
 ## Serving & chat
 
 ### `serve` — run the server
 
-Start the OpenAI/Anthropic-compatible server plus the status page and browser
-chat UI.
+Start the OpenAI/Anthropic-compatible server plus the browser web app (chat,
+status, and model tools).
 
 ```sh
 mlx-bun serve                              # default model, http://localhost:8080
 mlx-bun serve e4b                          # pick a model by substring
 mlx-bun serve gemma --memory-budget 18     # ...with admission control (GB)
 mlx-bun serve e4b --no-open                # don't open the browser chat UI
+mlx-bun serve GLM-5.2 --context-length 4096 # bounded Colibri runtime, native MTP
 ```
 
 Common flags (full list in [server-config.md](server-config.md)):
@@ -34,16 +36,34 @@ Common flags (full list in [server-config.md](server-config.md)):
 | `--host <addr>` | Interface to bind (default `127.0.0.1`, loopback only; `0.0.0.0` to expose) |
 | `--port <n>` | Listen port (default 8080) |
 | `--memory-budget <GB>` | Reject loads/requests that can't fit the budget |
-| `--kv-quant config\|off\|4\|8` | KV cache quantization: per-layer `kv_config.json`, bf16, or uniform bits (default `config`) |
+| `--kv-budget <GB>` | Aggregate KV budget across concurrent batch rows: over-budget joiners queue until rows finish; a request over the budget alone is rejected (off unless set) |
+| `--prompt-cache <GB>` | RAM prefix-KV cache, byte-capped LRU (default 8; `0` disables). Non-consuming prefix sharing: a hit clones instead of consuming the donor entry |
+| `--isolate` | Run the inference engine as a crash-isolated child process on a unix socket; this process becomes a pure reverse-proxy (instant under GPU load, auto-respawns on engine crash) — default off ([runtime-isolation.md](../design/runtime-isolation.md)) |
+| `--model-pool <n>` | With `--isolate`: cap on resident per-model engine children (default 1); over the cap the LRU engine drains its cache to the SSD tier and exits, respawning on demand |
+| `--ssd-cache <dir>` | SSD cold tier under the prompt cache: idle/evicted prefix-KV spills to disk and restores on reattach, surviving restarts (default off — [ssd-kv-cold-tier.md](../design/ssd-kv-cold-tier.md)) |
+| `--ssd-cache-max <GB>` | SSD tier byte cap, oldest-mtime evicted first (default 32; `--ssd-cache` only) |
+| `--ssd-cache-verify` | Verify every tensor hash on restore, not just the header (integrity paranoia; roughly doubles restore reads; `--ssd-cache` only) |
+| `--ssd-demote-idle <sec>` | Idle prompt-cache entries spill to the SSD tier and free GPU memory after this long (default 300 with `--ssd-cache`; `0` disables) |
+| `--kv-quant config\|off\|4\|8\|turbo[:k<bits>v<bits>]` | KV cache quantization: per-layer `kv_config.json`, bf16, uniform bits, or TurboQuant (rotation-based, default `k8v3` — [turboquant-kv.md](../design/turboquant-kv.md); full-attention layers only, solo-only/serial) (default `off` — quantized KV trades 5–20% decode speed for memory headroom, so it's opt-in) |
+| `--paged-kv` | OPTIONAL vLLM-style paged KV cache (block pool + gather before the stock SDPA; env `MLX_BUN_PAGED_KV=1`) — bit-exact with the plain path, default off. v1: serial only (pins `--batch 1` unless `--batch` is given; refuses `--batch N>1`, `--kv-quant`, `--draft-model`), Gemma4-family, bf16; bypasses the prompt cache and runs uncompiled decode. Expect a small decode cost at batch=1 — v1 ships the abstraction for batched/CoW follow-ups ([paged-kv-cache.md](../design/paged-kv-cache.md)) |
+| `--paged-kv-block-size <n>` | Tokens per KV block (default 256 = the plain cache's growth step; `--paged-kv` only) |
 | `--adapter <dir>` | Mount a LoRA adapter at startup (`--adapter-path` accepted as the mlx_lm.server alias) |
-| `--batch <n>` | Continuous-batched bf16 serving (default 1 = serial) |
-| `--temperature` / `--top-p` / `--top-k` / `--max-tokens` | Server-wide sampling defaults (per-request fields still win) |
-| `--l1` / `--l2` / `--l3` | Parity tier: bit-exact to mlx-lm / bit-exact to mlx-optiq / best performance |
+| `--draft-model <path\|query>` | Speculative decoding: a drafter proposes, the main model verifies — exact results, faster decode when drafts land. The artifact's kind is auto-detected: a full same-tokenizer model (mlx_lm.server parity), a Gemma `-assistant` KV-borrowing drafter, a DSpark checkpoint trained here, or one of DeepSeek's released DSpark drafters (DeepSpec `Gemma4DSparkModel`, e.g. `dspark_gemma4_12b_block7`). Serial lane only: with `--batch N` a mounted draft routes every request serial |
+| `--draft-kind <kind>` | Override draft-artifact detection: `two-model` \| `assistant` \| `dspark` \| `deepspec` \| `ngram` (default: auto). `ngram` is MODEL-FREE prompt lookup — no `--draft-model` (mounting one alongside it is refused): drafts are copied from the request's own prompt+generation whenever its trailing tokens have occurred earlier in that stream, verified by the target like any other draft, so results stay exact at any temperature. Zero weights, zero memory; best on agentic/RAG/code-edit traffic that re-emits spans already in context (prior art: prompt-lookup decoding / vLLM's `ngram` proposer) |
+| `--num-draft-tokens <n>` | Drafts per verify round (default 3; `ngram` defaults to 10 — drafting is free; a DSpark draft pins to its trained block width) |
+| `--ngram-max <k>` / `--ngram-min <k>` | Prompt-lookup match-window bounds, `--draft-kind ngram` only: the longest/shortest trailing k-gram searched (defaults 3 / 1, the reference values) |
+| `--mtp on\|off` | GLM-5.2 checkpoint-native MTP (default on). Native MTP uses the exact serial verify lane; `off` enables ordinary continuous batching (batched per-row MTP is post-release) |
+| `--context-length <n>` | GLM-5.2 context reserved by the header-only resource equation (default 4096) |
+| `--batch <n>` | Continuous-batched serving cap, mlx-lm B=N parity (default 8; `--batch 1` pins the serial path) |
+| `--temperature` / `--top-p` / `--top-k` / `--max-tokens` | Server-wide sampling defaults (per-request fields still win; GLM-5.2 defaults to the planned 128 generated tokens) |
+| `--l1` / `--l2` | Parity tier alias: bit-exact to mlx-lm / bit-exact to mlx-optiq. **No tier = `--l1`** (the default since 2026-07-05 — output-changing levers are opt-in until they beat the L1 baseline in a paired A/B). Each expands to per-fork flags (`--compiled-decode`, `--compiled-activations`, `--fused-sdpa`, `--kv-quant`); a fork flag overrides one. `--l3` was removed 2026-07-05 and now errors (the Lab replaces it — [unified-engine-frontier-plan.md](../design/unified-engine-frontier-plan.md)). See [server-config.md](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2). |
+| `--allow-private-media` | Let `image_url`/`audio_url` parts fetch from private/loopback/link-local hosts (blocked by default — SSRF guard; the 10 s timeout and 64 MB cap on remote media apply either way) |
 | `--no-open` | Don't auto-open the chat UI |
 
 Endpoints: `/v1/chat/completions`, `/v1/completions`, `/v1/messages`,
 `/v1/responses`, `/v1/models`, `/v1/adapters`, `/health`, `/stats`, `/fit`,
-`/library`, `/downloads` — status page at `/`, browser chat at `/chat`. Full
+`/library`, `/downloads` — web app at `/` (lands on chat); `/chat`, `/status`,
+`/quantize`, `/finetune`, `/dataset` deep-link its views. Full
 schemas in [server-api.md](server-api.md).
 
 ### `pi` — built-in coding agent
@@ -73,15 +93,16 @@ mlx-bun harness pi --remove     # disconnect
 
 Load the model, generate, print, exit — no server. Renders through the chat
 template by default; `--raw` feeds the prompt verbatim. Sampling and
-decode-path levers mirror `serve` (`--temperature`, `--seed`, `--l1/--l2/--l3`,
+decode-path levers mirror `serve` (`--temperature`, `--seed`, `--l1/--l2`,
 `--kv-quant`, …).
 
 ```sh
 mlx-bun generate gemma "a haiku about metal shaders"
 mlx-bun gen e4b --prompt "…" --max-tokens 512 --seed 42
+mlx-bun generate GLM-5.2 --prompt "Explain DSA" --max-tokens 128
 ```
 
-### `embed` — text embeddings **(v0.0.9)**
+### `embed` — text embeddings
 
 One-shot embeddings from a local Qwen3-Embedding model (last-token pooled,
 L2-normalized — bit-exact vs mlx-lm). One JSON vector per input line.
@@ -101,13 +122,21 @@ Full guide: [models.md](models.md).
 ### `get` — download a model
 
 Resumable, checksum-verified download into the standard Hugging Face cache
-(plain HTTPS, no Xet — no 0% stalls).
+(plain HTTPS, no Xet — no 0% stalls). Before payload transfer, the downloader
+credits complete blobs and partial prefixes, then requires enough free space
+for the exact remainder plus a fixed 1 GiB reserve.
 
 ```sh
 mlx-bun get mlx-community/gemma-4-12B-it-OptiQ-4bit
 mlx-bun get 12B                  # no "/" = registry query; re-gets/refreshes the match
+mlx-bun get mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
+  --revision 3cc8db99b1b13fc79325d987ba3c1c430766b3b8
 ```
 
+The GLM artifact is roughly 357 GiB; select `HF_HUB_CACHE` on a sufficiently
+large fast volume before starting. Rerunning the same command resumes verified
+`.incomplete` prefixes. See the [32 GB GLM quickstart](models.md#glm-52-on-a-32-gb-mac).
+
 ### `scan` — index your cache
 
 Re-index the HF cache into the registry (config + safetensors headers only,
@@ -125,10 +154,12 @@ mlx-bun ls --vision --max-size 10GB # filter
 mlx-bun ls --all-revisions          # one row per cached snapshot
 ```
 
-### `gc` — reclaim disk **(v0.0.9)**
+### `gc` — reclaim disk
 
 Delete superseded snapshots and the blobs only they reference. Prints the plan
-by default; `--yes` actually deletes.
+by default; `--yes` actually deletes. Same planner (`src/registry.ts`
+`planGc`/`executeGc`) also backs `GET /api/gc/plan` / `POST /api/gc/execute`
+on a running `serve` — see [server-api.md](server-api.md#get-apigcplan--post-apigcexecute).
 
 ```sh
 mlx-bun gc          # dry-run: what would be reclaimed
@@ -146,6 +177,20 @@ mlx-bun fit gemma --ctx 32768          # for this machine
 mlx-bun fit gemma --ctx 8192 --skus    # across the Apple Silicon lineup
 ```
 
+The direct Colibri GLM-5.2 artifact is handled by its streamed-runtime
+contract, not by the generic resident-weights estimator. `fit GLM-5.2`
+defaults to the supported 4,096-token / 128-generation-token plan and reports
+the artifact's disk footprint separately from resident weights, the main and
+MTP expert slabs, KV, reserves, and the 25 GiB process ceiling. Its throughput
+line is the measured warm quality-preserving result; the direct Colibri oracle
+and the 2 tok/s aspiration are labeled separately. `--skus` explains that the
+generic resident-model matrix does not apply to this streamed artifact.
+
+```sh
+mlx-bun fit GLM-5.2
+mlx-bun fit GLM-5.2 --ctx 4096
+```
+
 ## Training & model creation
 
 Full reference: [training.md](training.md); guided walkthrough:
@@ -170,7 +215,7 @@ Tails an adapter dir's `metrics.jsonl` and renders live loss / speed / memory.
 mlx-bun train-watch ~/.cache/mlx-bun/mlx-bun-finetunes/orpo-e4b
 ```
 
-### `fuse` — merge an adapter into the base **(v0.0.9)**
+### `fuse` — merge an adapter into the base
 
 Folds LoRA deltas into the base weights and writes a standalone snapshot; a
 quantized base keeps its exact quantization layout. Flag names match
@@ -181,7 +226,7 @@ mlx-bun fuse e4b --adapter ./adapters --save-path ./fused_model
 mlx-bun serve ./fused_model
 ```
 
-### `convert` — quantize a model **(v0.0.9)**
+### `convert` — quantize a model
 
 Quantize an HF model (local path, downloaded model, or repo id) into a local
 MLX snapshot — uniform 4/8-bit, or mixed-precision via `--target-bpw`
@@ -192,7 +237,7 @@ mlx-bun convert Qwen/Qwen3-4B -q                      # uniform 4-bit
 mlx-bun convert Qwen/Qwen3-4B --target-bpw 4.5        # mixed precision
 ```
 
-### `upload` — push to the Hugging Face Hub **(v0.0.9)**
+### `upload` — push to the Hugging Face Hub
 
 Native push-to-hub, no Python: creates the repo if needed, uploads via the
 git-LFS batch protocol, commits. Flag names match `mlx_lm.upload`.
@@ -201,7 +246,7 @@ git-LFS batch protocol, commits. Flag names match `mlx_lm.upload`.
 mlx-bun upload --path ./fused_model --upload-repo you/my-model
 ```
 
-### `perplexity` — evaluate on your data **(v0.0.9)**
+### `perplexity` — evaluate on your data
 
 `mlx_lm.perplexity` methodology exactly, over a **local** `.jsonl`/`.txt` file.
 
@@ -226,7 +271,7 @@ mlx-bun memory synthesize        # conversations → cross-linked articles, all
 mlx-bun memory schedule --at 03:00
 ```
 
-`setup` is an alias for `memory` **(v0.0.9)**.
+`setup` is an alias for `memory`.
 
 ## Measurement
 

package/docs/reference/distribution.md

@@ -14,6 +14,7 @@ tarball (~73 MB compressed, ~228 MB installed):
 |---|---|
 | `mlx-bun` | the CLI/server (`bun build --compile`), ~61 MB |
 | `libmlxc.dylib`, `libmlx.dylib`, `libjaccl.dylib` | MLX native runtime |
+| `libmlx_bun_expert_io.dylib` | bounded native expert-streaming helper required by direct Colibri GLM-5.2 |
 | `mlx.metallib` | Metal kernels (~150 MB — the bulk) |
 | pi assets | `photon_rs_bg.wasm`, `theme/`, `assets/`, `export-html/`, `native/`, `package.json`, `CHANGELOG.md` |
 
@@ -156,8 +157,8 @@ manual escape hatch if offline.
 
 ## npm / bunx
 
-mlx-bun is published to npm as `mlx-bun` (current: 0.0.4). Install or
-run without a permanent install:
+mlx-bun is published to npm as `mlx-bun`. Install or run without a
+permanent install:
 
 ```sh
 # one-off (no install)
@@ -168,10 +169,40 @@ bun install -g mlx-bun
 mlx-bun --version
 ```
 
-The npm package wraps the same signed binary via a postinstall download
-step (it does not bundle the ~228 MB dylibs/metallib in the npm tarball
-itself). Publish to npm as part of a release:
+The npm package is a thin launcher (no postinstall hook): it imports
+`src/cli.ts` under Bun, which fetches the small (~52 MB compressed,
+sha256-verified) native runtime pack — `libmlxc`/`libmlx`/`libjaccl`/
+`mlx.metallib` plus `libmlx_bun_expert_io.dylib` — into
+`~/Library/Caches/mlx-bun/` on first invocation if
+no copy is already resolvable (env / beside the binary / cache /
+Homebrew). This is the same npm/sidecar mechanism described above, not
+the signed/notarized Homebrew bundle. `bun publish` is already run
+automatically and idempotently as step 4 of `PUBLISH=1
+./scripts/release-binary.sh`; run it standalone only as a manual
+fallback/retry:
 
 ```sh
-bun publish   # from repo root; package.json version must already be bumped
+bun publish   # manual retry; package.json version must already be bumped
 ```
+
+### Publishing the first-run native pack
+
+The npm/bunx path has an independent, versioned release asset. Whenever its
+file manifest changes, build and publish it before shipping code that names the
+new `NATIVE_PACK_VERSION`:
+
+```sh
+sh scripts/build-native-pack.sh 0.2.0 dist-native
+gh release create native-v0.2.0 \
+  dist-native/mlx-bun-native-v0.2.0-arm64.tar.gz \
+  dist-native/mlx-bun-native-v0.2.0-arm64.tar.gz.sha256 \
+  --title "mlx-bun native runtime v0.2.0" \
+  --notes "Adds the bounded native expert-I/O helper required by direct Colibri GLM-5.2."
+```
+
+For v0.2.0 the arm64 archive is 52,307,647 bytes with SHA-256
+`9bd3795c5ea8f52b18413501f2d68c32c264a20751302300a08dc04cd67df97c`.
+It contains `libmlxc.dylib`, `libmlx.dylib`, `libjaccl.dylib`,
+`mlx.metallib`, and `libmlx_bun_expert_io.dylib`. These values must match
+`src/native-pack.ts`; a fresh-cache extraction plus real MLX dlopen and native
+expert read is the pre-publish smoke.