mlx-bun 0.0.11 → 0.0.12

package/docs/investigations/indexshare-performance-spike.md

@@ -0,0 +1,357 @@
+# IndexShare / IndexCache performance spike
+
+**Date:** 2026-08-16
+**Status:** accepted research spike; paper/code audit complete; implementation and
+full-model measurement not started
+**Target:** GLM-5.2 DSA on the M1 Max 32 GB Colibri path
+
+## Recommendation
+
+Run the spike, but do not treat it as a new model feature. The GLM-5.2 path
+already implements the checkpoint's cross-layer IndexShare schedule: retained
+`full` layers run the DSA indexer and intervening `shared` layers reuse the most
+recent selection. The useful work is to make that correctness path production
+shaped, prove long-context parity, and measure whether the saved indexer work is
+visible behind expert I/O on Apple Silicon.
+
+The first prerequisite is the exact stock indexer overlay. The pinned public
+Colibri artifact has no `out-idx-*` files, so every full-model result recorded
+to date is DSA-off. The second prerequisite is a scalable indexer path: current
+code materializes a large `[heads, context, head_dim]` product, copies every
+score to the host, sorts the full score vector in JavaScript, and supports
+sparse DSA only for single-token decode. Those choices are good correctness
+scaffolding but cannot test the paper's long-prefill result.
+
+This spike must not change the trained `full`/`shared` pattern or expose it as a
+normal user knob. GLM-5.2 ships IndexShare as checkpoint architecture, not as an
+optional quality/performance preset.
+
+## Naming and source claims
+
+The linked paper calls the method **IndexCache**. The GLM-5.2 model card calls
+the deployed architecture **IndexShare**. They use the same core mechanism:
+cache the top-k positions produced by a retained DSA indexer and reuse them in
+subsequent shared layers.
+
+Primary sources:
+
+- [IndexCache paper, arXiv:2603.12201v1](https://arxiv.org/abs/2603.12201v1)
+- [authors' SGLang/vLLM reference patches](https://github.com/THUDM/IndexCache)
+- [pinned GLM-5.2 model card](https://huggingface.co/zai-org/GLM-5.2/blob/f6142f127a14b58dc602592e996cd7d8ff139351/README.md)
+- [pinned GLM-5.2 configuration](https://huggingface.co/zai-org/GLM-5.2/blob/f6142f127a14b58dc602592e996cd7d8ff139351/config.json)
+
+The sources establish three different claims that must not be blended:
+
+| Claim | Published evidence | What mlx-bun may say before local measurement |
+|---|---|---|
+| Cross-layer reuse | F layers compute top-k; S layers reuse the nearest preceding F layer | Implemented semantics, once parity is gated |
+| IndexCache speed | On a 30B DSA model on an 8x H100 node, 1/4 retention reached up to 1.82x prefill and 1.48x per-request decode at 200K context | Paper result only; not a Mac prediction |
+| GLM-5.2 IndexShare | The model card says one retained indexer per four sparse-attention layers reduces per-token FLOPs by 2.9x at 1M context | Vendor FLOP claim only; not end-to-end speed |
+
+The paper also reports that uniform 1/4 retention hurts a frozen 30B model's
+long-context average (50.2 to 43.0), while a loss-searched pattern recovers it
+to 49.9. Training-aware IndexCache makes uniform interleaving viable. Therefore
+mlx-bun must preserve GLM-5.2's trained schedule rather than search a new one
+during this spike.
+
+## GLM-5.2 schedule
+
+The pinned GLM-5.2 config has 78 layers, `index_topk=2048`,
+`index_topk_freq=4`, and `index_skip_topk_offset=3`. Its explicit schedule has
+21 full and 57 shared layers:
+
+```text
+FFFSSSFSSSFSSSF...FSSS
+```
+
+Relative to an all-full counterfactual, the checkpoint omits 57/78 = 73.1% of
+layer indexer invocations. It does not ship the missing shared-layer indexer
+weights, so a true same-checkpoint `IndexShare off` A/B is impossible. Any
+estimated all-full number must be labeled a component counterfactual, not an
+end-to-end model benchmark.
+
+`index_share_for_mtp_iteration` is a second, orthogonal optimization: reuse
+indices across speculative draft iterations. mlx-bun parses that field but does
+not currently consume it in the MTP runtime. Keep its measurement separate so
+cross-layer IndexShare and MTP-iteration sharing each receive an attributable
+result.
+
+## Current mlx-bun state
+
+Already present:
+
+- [`src/model/glm52-config.ts`](../../src/model/glm52-config.ts) parses explicit
+  `indexer_types` and the frequency/offset fallback.
+- [`src/model/glm52.ts`](../../src/model/glm52.ts) constructs DSA indexers only
+  for full layers and passes one request-local selection state through the
+  decoder stack.
+- [`src/model/glm52-dsa.ts`](../../src/model/glm52-dsa.ts) records a full
+  layer's exact dense/sparse selection and serves it to shared layers.
+- [`tests/glm52-config-container.test.ts`](../../tests/glm52-config-container.test.ts)
+  and [`tests/glm52-dsa.test.ts`](../../tests/glm52-dsa.test.ts) cover schedule
+  parsing, missing-owner refusal, context consistency, and exact reuse.
+
+Known blockers and likely bottlenecks:
+
+1. **No production DSA overlay.** The public Colibri artifact deliberately
+   disables DSA because every full-layer indexer tensor is absent. Producing the
+   exact overlay requires selective reads from 20 pinned source shards (about
+   99.90 GiB input, about 197 MB output). This is a Josh-run acquisition step,
+   not an agent-session download.
+2. **No sparse long prefill.** `Glm52DsaIndexer.projectAndSelect()` rejects a
+   multi-token prefill once context exceeds `index_topk`. The paper's prefill
+   headline is therefore out of scope until chunked/sparse DSA prefill exists.
+3. **Quadratic transient shape.** `glm52DsaScoresMlx()` forms the broadcast
+   product before reduction. At production geometry this scales as
+   `32 * context * 128` float elements rather than streaming/tiled score
+   accumulation.
+4. **GPU-to-host top-k.** The full score vector crosses to a `Float32Array`,
+   then `selectDsaThresholdTiesF32()` clones and sorts all positions. This is
+   `O(context log context)` host work in each full layer, unlike the production
+   GPU top-k assumed by the paper.
+5. **Shared-index copies.** Shared layers clone the 2,048-element JavaScript
+   positions list, and absorbed MLA uploads it again before each gather. The
+   paper's temporary index tensor is reused in place. Measure this separately;
+   remove copies only after ownership and asynchronous lifetime are gated.
+6. **The current short-context bottleneck is elsewhere.** At 128 tokens the
+   full model is limited by streamed expert delivery, and DSA cannot prune a
+   prefix smaller than 2,048. No short-context speedup should be expected.
+
+## Hypotheses
+
+- **H1 - component cost:** at the longest context that fits the 32 GB contract,
+  retained-indexer time is material and grows roughly linearly per decode token
+  with context length.
+- **H2 - sharing value:** avoiding 57 of 78 indexer invocations removes a
+  measurable amount of counterfactual work even after a tiled score/top-k
+  kernel is used.
+- **H3 - product value:** DSA plus the shipped IndexShare schedule improves
+  end-to-end long-context decode over DSA-off full attention, without token
+  drift from the official DSA runtime. This comparison measures the combined
+  sparse-attention product, not IndexShare in isolation.
+- **H4 - MTP interaction:** any win remains a win with native MTP enabled. A
+  decode-only win that disappears under the product MTP path is not a default
+  performance claim.
+
+## Bounded experiment
+
+### Stage 0 - artifact and parity gate
+
+1. Generate the immutable `out-idx-*` overlay from the pinned
+   `zai-org/GLM-5.2-FP8` revision. Record source revision, selected shard list,
+   tensor inventory, byte count, and SHA-256 values. Do not mutate the 357 GiB
+   serving snapshot.
+2. Header-validate exactly the 21 full-layer indexers required by the shipped
+   schedule and prove that shared-layer indexers are neither required nor
+   silently loaded.
+3. Add an official-runtime oracle at contexts above 2,048. Gate selected
+   positions, logits on tie-free rows, and greedy token IDs before taking any
+   performance number.
+
+#### Stage 0 result — pass (2026-08-17)
+
+The stock overlay was generated outside mlx-bun from
+`zai-org/GLM-5.2-FP8@ba978f7d347eaf65d22f1a86833408afdb953541`
+with `colibri@ecade075cfc2eae684097ea7de5570c3786ce199`. It contains 20
+`out-idx-*` files totaling 197,202,400 bytes. The existing header validator
+finds 118,646 tensors, 59,066 quantized tensors, 19,456 routed experts,
+complete DSA and MTP families, all 21 required full-layer indexers, and no
+shared-layer indexer requirement.
+
+The first-sparse gate prefills exactly 2,048 deterministic tokens, then sends
+one greedy token at context 2,049 with MTP off. The mlx-bun probe observes the
+21 configured full layers and exactly 2,048 selected positions per layer. A
+detached pinned Colibri build captures each official score row, selection,
+float32 logit vector, and token trajectory. Replaying the captured scores with
+`Glm52DsaSelectionState` gives:
+
+- 21/21 exact ordered position vectors;
+- 21/21 exact float32 thresholds;
+- 21/21 tie-free boundaries, minimum boundary gap 0.025123596;
+- exact greedy trajectory `[264, 264]` and the same sparse-step top-1 token;
+- 154,880-logit cosine 0.997645, RMSE 0.200109, maximum absolute drift
+  0.947218, with wide top-1 margins in both runtimes (9.571914 / 9.567580).
+
+The direct full-runtime selections are 10/21 ordered-exact and 14/21
+set-exact. Each of the seven set differences swaps only the single excluded
+position at this top-2,048-of-2,049 boundary. This is upstream numerical drift
+from the runtimes' different quantized-matmul accumulation before DSA, not a
+selector mismatch; replaying identical official scores proves the isolated
+DSA threshold/tie contract exactly. No performance claim is made from either
+correctness run.
+
+Reproduction is split between
+`scripts/probe-colibri-glm52-dsa-long.ts` and
+`scripts/check-colibri-glm52-dsa-long.ts`. The raw capture, external Colibri
+instrumentation patch, and SHA-256 inventory are machine-local at
+`~/.cache/mlx-bun/evidence/glm52-dsa-stage0-2026-08-17/`; manifest SHA-256 is
+`6ecb88d944b055146dd700a1714af97b61845d4218527282fcbea009663fe3b4`.
+
+### Stage 1 - production-shaped indexer kernel
+
+Build a model-free benchmark at exact geometry (`32` heads, head dimension
+`128`, top-k `2048`) for contexts `2,048`, `8,192`, and `32,768`:
+
+- tiled score accumulation that never materializes `[H,L,D]`;
+- on-device/native deterministic top-k with the current threshold and
+  lower-position tie contract;
+- one compact index buffer retained across F -> S layers;
+- direct gather consumption without a score-vector host copy or per-shared-
+  layer index re-upload.
+
+Compare the candidate against the existing reference composition for exact
+selected positions on random, tied, all-equal, and production-captured rows.
+Record kernel time, synchronization time, peak MLX allocation, process
+footprint, and bytes copied to/from the host. Negative results count.
+
+#### Stage 1 result — pass (2026-08-17)
+
+The production path now computes `[H,D] @ [D,L] -> [H,L]` and never forms the
+old `[H,L,D]` broadcast product. Deterministic device selection encodes the
+lexicographic `(score descending, position ascending)` rank into `uint64`, uses
+MLX `argpartition`, and then restores Colibri's observable two-scan ordering
+(strictly above threshold first, threshold ties second). A FULL layer retains
+only the resulting 2,048-entry, 8 KiB index buffer; SHARED layers borrow that
+same device array and MLA gathers from it directly. Normal inference performs
+no score/index host readback or per-SHARED-layer upload. The probe observer is
+the explicit diagnostic exception.
+
+Correctness gates pass for deterministic random rows, threshold ties,
+all-equal rows, the exact 2,049/2,048 boundary, borrowed MLA consumption, and
+all 21 production-captured Colibri score rows. The latter reproduce exact
+ordered positions and float32 thresholds on device. A fresh full-model run
+also reproduced all 21 prior mlx-bun position vectors and both greedy tokens;
+its complete 154,880-float decode-logit payload is byte-identical (SHA-256
+`d1f62be2de4877d2fc8ae8feab88e522958210252b50ef96eb786cb22581ba18`).
+Tiled accumulation moved nine diagnostic thresholds by at most
+`3.0517578125e-5`, but no selection boundary moved.
+
+The exact-geometry, warm synchronized component benchmark (seven repeats) is:
+
+| Context | Production behavior | Score median | Score + top-k median | Timed host copies | MLX peak |
+|---:|---|---:|---:|---:|---:|
+| 2,048 | dense boundary; production skips score/top-k | 0.448 ms (diagnostic only) | — | 0 B | 1.27 MiB |
+| 8,192 | sparse | 0.473 ms | 0.929 ms | 0 B | 5.41 MiB |
+| 32,768 | sparse | 0.748 ms | 1.269 ms | 0 B | 21.35 MiB |
+
+At those contexts the removed broadcast product would have been 32, 128, and
+512 MiB respectively; the tiled dot output is 0.25, 1, and 4 MiB. These are
+model-free component measurements, not end-to-end inference claims. The
+benchmark is `scripts/bench-glm52-dsa-indexer.ts`. Machine-local evidence is
+under `~/.cache/mlx-bun/evidence/glm52-dsa-stage1-2026-08-17/`; manifest
+SHA-256 is
+`600ec92c41d4c66735e1834a678a09f2ae55ecdddfab31c3dd1bf31ee9709fef`.
+
+### Stage 2 - decode matrix
+
+Use direct library inference, not a server. Run separate fresh processes and
+the standard quiet-machine preflight.
+
+| Axis | Cells |
+|---|---|
+| Context | 2,048; 8,192; 32,768; 64K only if the exact planner remains inside the 25 GiB process ceiling |
+| DSA | off/full attention; on with the checkpoint's 21F/57S schedule |
+| MTP | off to isolate DSA; on for the product decision |
+| Residency | the same fixed expert policy and seed profile in every paired arm |
+| Repeats | three fresh processes per default-eligible cell; cold and warm turn in each process |
+
+For each turn record TTFT, prefill wall time, decode tok/s, total wall time,
+DSA projection/score/top-k/gather time, MLA time, expert disk service/wait,
+target forwards, MTP acceptance, physical footprint, compression, swap, and
+exact token IDs.
+
+The DSA-off versus DSA-on result combines sparse core attention with
+IndexShare. To estimate IndexShare's isolated contribution, report the measured
+retained-indexer total and a clearly labeled all-full component counterfactual
+derived from the same per-layer exact-geometry benchmark. Do not present that
+counterfactual as a runnable checkpoint.
+
+#### Stage 2 result — pass, negative product-performance decision (2026-08-17)
+
+The fresh-process runner planned 36 cells. Twenty-four were eligible under the
+G5 25 GiB process ceiling and completed: contexts 2,048 and 8,192, four DSA/MTP
+arms, and three repeats. Every process ran cold and warm turns. Cold/warm,
+cross-repeat, and MTP-off/on greedy token vectors were exact; all emitted
+sixteen token IDs equal to `264`. The twelve 32K combinations were not run:
+the exact planner reports 29,334,150,784 bytes (27.320 GiB) MTP-off and
+30,644,281,216 bytes (28.540 GiB) MTP-on, both above the unchanged 25 GiB
+process limit. The initial refusal is retained in the evidence as an attempt
+record. This is contract ineligibility, not an allocation crash.
+
+Long prompts use an explicitly bounded benchmark seam: dense batched prompt
+construction ends at the exact prompt boundary, and every subsequent decode or
+MTP verification row uses exact DSA. These numbers characterize decode and
+whole-turn wall time only. They are not sparse-prefill measurements.
+
+Warm medians across three fresh processes are:
+
+| Context | DSA | MTP | Decode tok/s | Total wall |
+|---:|---|---|---:|---:|
+| 2,048 | off | off | 0.1363 | 260.9 s |
+| 2,048 | on | off | 0.1318 | 271.5 s |
+| 2,048 | off | on | 0.1312 | 281.7 s |
+| 2,048 | on | on | 0.0880 | 338.7 s |
+| 8,192 | off | off | 0.1292 | 722.1 s |
+| 8,192 | on | off | 0.1420 | 711.8 s |
+| 8,192 | off | on | 0.1587 | 711.8 s |
+| 8,192 | on | on | 0.1042 | 770.1 s |
+
+At 8K without MTP, the paired median DSA delta is +12.38% decode throughput
+and -1.89% total wall time. Expert I/O and prompt work hide most of the decode
+gain, leaving the whole-turn result below the required 5%. With MTP enabled,
+the paired median delta reverses to -34.33% decode throughput and +8.19% total
+wall time. The exact MTP trace explains part of the product result: DSA-off
+accepted ten drafts in five rounds (3.2 tokens/target forward, six target
+calls), whereas DSA-on accepted eight drafts in seven rounds (2.286
+tokens/target forward, eight target calls). Small verify batches now keep
+residual/MLP/expert work batched, but exact causal sparse attention still runs
+per verification row. Two of the three 8K DSA+MTP cells observed net swap
+growth, so the memory side of the win gate also does not close.
+
+The immutable external overlay retains 197,202,400 bytes of indexers. As a
+clearly labeled non-runnable component counterfactual, Stage 1's 8K
+score+top-k median implies 19.51 ms for the shipped 21 FULL layers versus
+72.48 ms for hypothetical 78 FULL layers, so IndexShare avoids about 52.97 ms
+of selection work per target row. At 32K the corresponding component-only
+figures are 26.64 ms, 98.95 ms, and 72.31 ms avoided. The checkpoint lacks the
+57 missing indexers, and these sums exclude the rest of model execution.
+
+The product decision is negative: retain the exact 21F/57S implementation as
+model semantics and land its correctness/scalability work, but make no local
+end-to-end speed claim. Decode is conclusive, so Stage 3 remains a separate
+deferred prefill experiment rather than a G6R exit requirement. Evidence is
+under `~/.cache/mlx-bun/evidence/glm52-dsa-stage2-2026-08-17/`; manifest
+SHA-256 is
+`90b3fe4ed53714604b7a747991b3bb1b87aedbf57a139915065f5b4be42cda38`.
+
+### Stage 3 - deferred prefill follow-up
+
+Only after decode is characterized, implement chunked sparse prefill with exact
+causal selection semantics and run the same context ladder. This is the stage
+that can test whether the paper's growing prefill benefit translates to MLX;
+the existing single-token path cannot support that claim.
+
+## Decision rules
+
+- Preserve the shipped 21F/57S schedule regardless of outcome; it is model
+  semantics.
+- Call the work a local performance win only when three fresh MTP-on processes
+  are token-identical and the paired median end-to-end wall time improves by at
+  least 5% at a context that fits the 32 GB contract, without new swap or a
+  footprint regression outside the existing G5 bounds.
+- If the kernel saves DSA time but expert I/O hides the wall win, land only the
+  correctness/scalability work and record the negative end-to-end result.
+- Do not quote the paper's H100 speedups or GLM-5.2's 1M-token FLOP reduction as
+  mlx-bun speed. Curated claims require this machine's `benchmark.sh`-quality
+  evidence and belong in `benchmarks/RESULTS.md` only after the gate passes.
+- Evaluate MTP-iteration index sharing as a separate follow-up with its own
+  on/off control and acceptance trace.
+
+## Exit
+
+The spike closes when the DSA overlay is provenance-pinned, long-context
+IndexShare matches an official runtime, the production-shaped score/top-k path
+has exact model-free gates, and the paired MTP-off/on decode matrix has a
+written decision. Prefill may remain a separately tracked follow-up only if the
+decode result is already conclusive; otherwise sparse long prefill is required
+before closure.

package/docs/planning/release-notes-v0.0.12.md

@@ -0,0 +1,80 @@
+# v0.0.12 release notes
+
+mlx-bun 0.0.12 completes the native Colibri/GLM-5.2 program: a Python-free,
+direct-container runtime that can serve the approximately 357 GiB artifact on
+a 32 GiB Apple Silicon Mac by keeping a bounded working set in memory and
+streaming routed experts from fast storage.
+
+## GLM-5.2 direct-container serving
+
+- Loads `mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp` directly. No converted
+  copy and no Python process are involved in the serving path.
+- Preserves checkpoint precision and true top-8 routing. The port is gated
+  against pinned Colibri at tiny, production-component, real-model, MTP, cache,
+  and first-sparse DSA boundaries.
+- Native int8 MTP is enabled by default and uses the serial speculative lane.
+  It produced a measured 1.306x warm end-to-end win over MTP-off on the 32 GiB
+  reference machine while preserving all target tokens.
+- Compressed MLA/DSA/MTP cache persistence, restart restore, compressed-byte
+  admission, ordinary continuous batching, mixed-length join/leave/cancel, and
+  cross-row expert union are included. Per-row batched MTP remains a later
+  feature; MTP requests truthfully report `serial+spec`.
+- Chat completions, text completions, Anthropic Messages, OpenAI Responses,
+  SSE, tools, structured output, stops, sampling, usage, and serial logprobs
+  are supported. Embeddings, vision/audio, LoRA, and training are explicitly
+  reported unsupported for this model instead of being emulated.
+
+## Memory and acquisition contract
+
+- `mlx-bun get` now computes the exact remaining payload before the first
+  payload request, credits verified shared blobs and resumable prefixes, and
+  refuses an acquisition that cannot leave a fixed 1 GiB safety reserve.
+- The GLM header-only preflight runs before resident weights or expert slabs
+  open. With the stock DSA overlay installed, 4,096 context / 128 generation
+  tokens and MTP on produce a 19.89 GiB plan under a 25 GiB process ceiling,
+  leaving 5.11 GiB of process/macOS headroom on a 32 GiB machine.
+- `fit`, `/fit`, `/stats`, and the status page now distinguish the complete
+  artifact on disk from resident weights, main/MTP slabs, compressed KV, and
+  reserves. They also expose main/MTP residency, hit rate, SSD bytes, policy
+  telemetry, and repin events.
+
+## Performance: measured, not predicted
+
+- Warm quality-preserving throughput on the M1 Max 32 GiB reference machine is
+  0.1577 decode tok/s / 0.1487 end-to-end tok/s with MTP on. The same-machine
+  direct-Colibri control is approximately 0.27 tok/s.
+- The 2 tok/s figure remains an aspiration, not a release gate or `fit`
+  prediction. The measured warm end-to-end result is 13.45x below it.
+- Replicated telemetry measured 1,974,949,363,712 logical expert bytes per
+  128-token warm turn (15.429 decimal GB/token) and a 1.66% cache hit rate.
+  Startup auto-pin, live LFRU, PILOT hints, two-step prediction, and coupling
+  did not improve paired end-to-end performance and remain off by default.
+- The checkpoint's 21-full/57-shared DSA schedule is implemented and exact.
+  Paired 2K/8K tests did not establish an MTP-on product-speed win, so this
+  release makes no DSA speed claim. The public Colibri artifact does not ship
+  the stock DSA indexer overlay; without it the runtime follows the artifact's
+  available attention path.
+
+## Distribution and operator experience
+
+- First-run native pack v0.2.0 adds the bounded native expert-I/O helper. The
+  npm/bunx fresh-cache path downloads, verifies, and extracts all five runtime
+  files; the signed/Homebrew bundle includes the same helper beside the binary.
+- The pinned 32 GiB quickstart, volume selection, interrupted-download recovery,
+  model lineage, licenses, resource plan, measured performance, and limitations
+  are documented in README and the reference guides.
+- The interactive Atlas report reproduces Colibri's expert topic-affinity
+  result: 29/30 leave-one-prompt-out classifications (96.7%) and 1,065 strong
+  specialists among 13,236 replicated experts.
+
+## Stability fixes since v0.0.11
+
+- Closed pre-Colibri correctness and lifecycle findings, including expert-slab
+  post-close and lazy-graph reuse hazards, with forced native/MLX churn gates.
+- Fixed selected-logprob readback stalling and revalidated the audio splice
+  boundary against the oracle.
+- Corrected current Hugging Face LFS checksum-field handling and retained the
+  older spelling for compatibility.
+
+Full provenance and exact numbers are in `benchmarks/RESULTS.md`, `STATUS.md`,
+and `docs/design/colibri-glm52-port.md`.

package/docs/reference/cli.md

@@ -26,6 +26,7 @@ mlx-bun serve                              # default model, http://localhost:808
 mlx-bun serve e4b                          # pick a model by substring
 mlx-bun serve gemma --memory-budget 18     # ...with admission control (GB)
 mlx-bun serve e4b --no-open                # don't open the browser chat UI
+mlx-bun serve GLM-5.2 --context-length 4096 # bounded Colibri runtime, native MTP
 ```
 
 Common flags (full list in [server-config.md](server-config.md)):
@@ -51,8 +52,10 @@ Common flags (full list in [server-config.md](server-config.md)):
 | `--draft-kind <kind>` | Override draft-artifact detection: `two-model` \| `assistant` \| `dspark` \| `deepspec` \| `ngram` (default: auto). `ngram` is MODEL-FREE prompt lookup — no `--draft-model` (mounting one alongside it is refused): drafts are copied from the request's own prompt+generation whenever its trailing tokens have occurred earlier in that stream, verified by the target like any other draft, so results stay exact at any temperature. Zero weights, zero memory; best on agentic/RAG/code-edit traffic that re-emits spans already in context (prior art: prompt-lookup decoding / vLLM's `ngram` proposer) |
 | `--num-draft-tokens <n>` | Drafts per verify round (default 3; `ngram` defaults to 10 — drafting is free; a DSpark draft pins to its trained block width) |
 | `--ngram-max <k>` / `--ngram-min <k>` | Prompt-lookup match-window bounds, `--draft-kind ngram` only: the longest/shortest trailing k-gram searched (defaults 3 / 1, the reference values) |
+| `--mtp on\|off` | GLM-5.2 checkpoint-native MTP (default on). Native MTP uses the exact serial verify lane; `off` enables ordinary continuous batching (batched per-row MTP is post-release) |
+| `--context-length <n>` | GLM-5.2 context reserved by the header-only resource equation (default 4096) |
 | `--batch <n>` | Continuous-batched serving cap, mlx-lm B=N parity (default 8; `--batch 1` pins the serial path) |
-| `--temperature` / `--top-p` / `--top-k` / `--max-tokens` | Server-wide sampling defaults (per-request fields still win) |
+| `--temperature` / `--top-p` / `--top-k` / `--max-tokens` | Server-wide sampling defaults (per-request fields still win; GLM-5.2 defaults to the planned 128 generated tokens) |
 | `--l1` / `--l2` | Parity tier alias: bit-exact to mlx-lm / bit-exact to mlx-optiq. **No tier = `--l1`** (the default since 2026-07-05 — output-changing levers are opt-in until they beat the L1 baseline in a paired A/B). Each expands to per-fork flags (`--compiled-decode`, `--compiled-activations`, `--fused-sdpa`, `--kv-quant`); a fork flag overrides one. `--l3` was removed 2026-07-05 and now errors (the Lab replaces it — [unified-engine-frontier-plan.md](../design/unified-engine-frontier-plan.md)). See [server-config.md](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2). |
 | `--allow-private-media` | Let `image_url`/`audio_url` parts fetch from private/loopback/link-local hosts (blocked by default — SSRF guard; the 10 s timeout and 64 MB cap on remote media apply either way) |
 | `--no-open` | Don't auto-open the chat UI |
@@ -96,6 +99,7 @@ decode-path levers mirror `serve` (`--temperature`, `--seed`, `--l1/--l2`,
 ```sh
 mlx-bun generate gemma "a haiku about metal shaders"
 mlx-bun gen e4b --prompt "…" --max-tokens 512 --seed 42
+mlx-bun generate GLM-5.2 --prompt "Explain DSA" --max-tokens 128
 ```
 
 ### `embed` — text embeddings
@@ -118,13 +122,21 @@ Full guide: [models.md](models.md).
 ### `get` — download a model
 
 Resumable, checksum-verified download into the standard Hugging Face cache
-(plain HTTPS, no Xet — no 0% stalls).
+(plain HTTPS, no Xet — no 0% stalls). Before payload transfer, the downloader
+credits complete blobs and partial prefixes, then requires enough free space
+for the exact remainder plus a fixed 1 GiB reserve.
 
 ```sh
 mlx-bun get mlx-community/gemma-4-12B-it-OptiQ-4bit
 mlx-bun get 12B                  # no "/" = registry query; re-gets/refreshes the match
+mlx-bun get mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
+  --revision 3cc8db99b1b13fc79325d987ba3c1c430766b3b8
 ```
 
+The GLM artifact is roughly 357 GiB; select `HF_HUB_CACHE` on a sufficiently
+large fast volume before starting. Rerunning the same command resumes verified
+`.incomplete` prefixes. See the [32 GB GLM quickstart](models.md#glm-52-on-a-32-gb-mac).
+
 ### `scan` — index your cache
 
 Re-index the HF cache into the registry (config + safetensors headers only,
@@ -165,6 +177,20 @@ mlx-bun fit gemma --ctx 32768          # for this machine
 mlx-bun fit gemma --ctx 8192 --skus    # across the Apple Silicon lineup
 ```
 
+The direct Colibri GLM-5.2 artifact is handled by its streamed-runtime
+contract, not by the generic resident-weights estimator. `fit GLM-5.2`
+defaults to the supported 4,096-token / 128-generation-token plan and reports
+the artifact's disk footprint separately from resident weights, the main and
+MTP expert slabs, KV, reserves, and the 25 GiB process ceiling. Its throughput
+line is the measured warm quality-preserving result; the direct Colibri oracle
+and the 2 tok/s aspiration are labeled separately. `--skus` explains that the
+generic resident-model matrix does not apply to this streamed artifact.
+
+```sh
+mlx-bun fit GLM-5.2
+mlx-bun fit GLM-5.2 --ctx 4096
+```
+
 ## Training & model creation
 
 Full reference: [training.md](training.md); guided walkthrough:

package/docs/reference/distribution.md

@@ -14,6 +14,7 @@ tarball (~73 MB compressed, ~228 MB installed):
 |---|---|
 | `mlx-bun` | the CLI/server (`bun build --compile`), ~61 MB |
 | `libmlxc.dylib`, `libmlx.dylib`, `libjaccl.dylib` | MLX native runtime |
+| `libmlx_bun_expert_io.dylib` | bounded native expert-streaming helper required by direct Colibri GLM-5.2 |
 | `mlx.metallib` | Metal kernels (~150 MB — the bulk) |
 | pi assets | `photon_rs_bg.wasm`, `theme/`, `assets/`, `export-html/`, `native/`, `package.json`, `CHANGELOG.md` |
 
@@ -171,7 +172,8 @@ mlx-bun --version
 The npm package is a thin launcher (no postinstall hook): it imports
 `src/cli.ts` under Bun, which fetches the small (~52 MB compressed,
 sha256-verified) native runtime pack — `libmlxc`/`libmlx`/`libjaccl`/
-`mlx.metallib` — into `~/Library/Caches/mlx-bun/` on first invocation if
+`mlx.metallib` plus `libmlx_bun_expert_io.dylib` — into
+`~/Library/Caches/mlx-bun/` on first invocation if
 no copy is already resolvable (env / beside the binary / cache /
 Homebrew). This is the same npm/sidecar mechanism described above, not
 the signed/notarized Homebrew bundle. `bun publish` is already run
@@ -182,3 +184,25 @@ fallback/retry:
 ```sh
 bun publish   # manual retry; package.json version must already be bumped
 ```
+
+### Publishing the first-run native pack
+
+The npm/bunx path has an independent, versioned release asset. Whenever its
+file manifest changes, build and publish it before shipping code that names the
+new `NATIVE_PACK_VERSION`:
+
+```sh
+sh scripts/build-native-pack.sh 0.2.0 dist-native
+gh release create native-v0.2.0 \
+  dist-native/mlx-bun-native-v0.2.0-arm64.tar.gz \
+  dist-native/mlx-bun-native-v0.2.0-arm64.tar.gz.sha256 \
+  --title "mlx-bun native runtime v0.2.0" \
+  --notes "Adds the bounded native expert-I/O helper required by direct Colibri GLM-5.2."
+```
+
+For v0.2.0 the arm64 archive is 52,307,647 bytes with SHA-256
+`9bd3795c5ea8f52b18413501f2d68c32c264a20751302300a08dc04cd67df97c`.
+It contains `libmlxc.dylib`, `libmlx.dylib`, `libjaccl.dylib`,
+`mlx.metallib`, and `libmlx_bun_expert_io.dylib`. These values must match
+`src/native-pack.ts`; a fresh-cache extraction plus real MLX dlopen and native
+expert read is the pre-publish smoke.

package/docs/reference/embedding.md

@@ -7,7 +7,7 @@
 
 Ship local inference inside a Mac app (Tauri, Electron, or anything
 that can spawn a process) with zero user-visible dependencies: one
-executable + three native libraries, dropped into your app's resources.
+executable + four native libraries, dropped into your app's resources.
 
 ## Build the bundle
 
@@ -23,6 +23,7 @@ Produces a relocatable directory:
 | `libmlxc.dylib` | mlx-c, rewritten to load `@loader_path/libmlx.dylib` | ~0.7 MB |
 | `libmlx.dylib` | mlx core (+`@loader_path` rpath added for libjaccl) | ~15 MB |
 | `libjaccl.dylib` | mlx's distributed-comm dependency | ~0.6 MB |
+| `libmlx_bun_expert_io.dylib` | bounded streamed-expert I/O used by direct Colibri GLM-5.2 | ~0.03 MB |
 | `mlx.metallib` | Metal kernels — libmlx loads it from its own directory | ~150 MB |
 | `photon_rs_bg.wasm` | pi image codec — only the web chat's `read`-on-image path; resolved next to the executable | ~1.8 MB |
 
@@ -44,7 +45,9 @@ cache (`~/Library/Caches/mlx-bun/native-v<ver>-<arch>/`, populated on
 first run when no sidecar or homebrew install is found) → homebrew
 (`/opt/homebrew/lib`, `/usr/local/lib`). The whole directory can be
 renamed/moved; nothing references absolute paths after the build
-script's `install_name_tool` fixups.
+script's `install_name_tool` fixups. The expert helper independently resolves
+`MLX_BUN_EXPERT_IO_DYLIB`, then the executable directory, then the same
+native-pack cache.
 
 ## Embedded pi web chat (`/ws/chat`)
 

package/docs/reference/features-matrix.md

@@ -22,7 +22,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | OpenAI Responses shim (`/v1/responses`) | on | both | — | — |
 | Continuous batching (mlx-lm B=N parity) | **on** — cap 8 (default flipped 2026-07-05; a lone request runs the exact serial engine, so the cap engages only under real concurrency; `--batch 1` pins serial) | batch | L1 | `--batch <n>` |
 | Prompt cache (prefix KV reuse) | on, 8 GB | both | — | `--prompt-cache <GB>` (0 = off) |
-| **SSD KV cold tier** (cache survives eviction + restarts) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
+| **SSD KV cold tier** (cache survives eviction + restarts; v3 includes GLM compressed MLA/DSA state without full-K/V reconstruction) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
 | **Runtime isolation** (engine runs as a crash-isolated CHILD process behind a reverse-proxy parent; UI stays instant under GPU load and survives engine crashes — [runtime-isolation.md](../design/runtime-isolation.md)) | off | both | — | `--isolate` |
 | **Model pool** (with `--isolate`: LRU-capped RESIDENT model engines; over the cap the least-recently-used engine demotes its cache to the SSD tier and exits, respawning on the next request) | 1 resident engine | both | — | `--model-pool <n>` (requires `--isolate`) |
 | Mixed-precision KV (`kv_config.json`, optiq's scheme) | **off** — opt-in (default flipped to bf16 2026-07-05; quantized KV trades 5–20% decode for memory headroom) | serial + **batch** (per-layer configs batch on every shipped model — full-attention layers Phase 3.1, rotating layers milestone 2; uniform bits stay serial) | L2 | `--kv-quant config\|off\|4\|8`, `--l2` |
@@ -32,6 +32,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | Fused SDPA (optiq-exact quantized-KV attention) | follows `--kv-quant`: on for `config`, off for uniform/bf16 | serial | L2 | `--fused-sdpa on\|off` |
 | **Paged KV** (vLLM-style block pool: host-side block table + gather before the stock SDPA, no fused kernel in v1 — [paged-kv-cache.md](../design/paged-kv-cache.md)) | off | serial (pins `--batch 1` unless `--batch` given) | gated bit-exact vs the plain `KVCache` (`tests/paged-kv-parity.test.ts`) | `--paged-kv`, `--paged-kv-block-size <n>` (default 256) — v1 scope Gemma4-family bf16; refuses `--batch N>1`, `--kv-quant`, `--draft-model` |
 | **Speculative decoding** (auto-detected drafter: two-model = mlx-lm parity/L1, Gemma `-assistant` = optiq/L2, DSpark local = Lab, DeepSpec released = DeepSpec-reference oracle) | off | serial (forces all-serial) | per-drafter oracle | `--draft-model <path\|query>`, `--draft-kind`, `--num-draft-tokens` |
+| **GLM-5.2 checkpoint-native MTP** (bounded auxiliary expert tier, exact verify) | **on** for GLM-5.2 | serial (`usage.lane: serial+spec`) | Colibri/oracle trajectory + synthetic HTTP gate | `--mtp on\|off` (`off` enables ordinary GLM batching) |
 | Speculative decoding, **model-free prompt lookup** (drafts copied from the request's own context; port of prompt-lookup decoding / vLLM `ngram`; zero weights) | off | serial (forces all-serial) | lossless by verify (gated vs non-spec greedy, tests/spec-ngram.test.ts) | `--draft-kind ngram` (no `--draft-model`), `--ngram-max`, `--ngram-min`, `--num-draft-tokens` (default 10) |
 | Memory admission (refuse what can't fit; never GPU-OOM) | on (RAM × 0.75) | both | — | `--memory-budget <GB>` |
 | Aggregate KV admission for batch rows (queue, don't OOM) | off | batch | — | `--kv-budget <GB>` |
@@ -48,7 +49,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | `guided_grammar` (EBNF) / `guided_regex`¹ / `guided_choice` / `structured_outputs` | on | both | L2 | request fields |
 | Structured output × speculative decoding | on (when both active) | serial | Lab (validity+equivalence) | — |
 | Quantized KV × speculative decoding (any axis: uniform / config / turbo) | KV scheme wins — drafted requests decode serially WITHOUT speculation (spec lane is bf16-KV-only in v1; startup warning) | serial | — | omit `--kv-quant` to speculate |
-| Tool calling (Gemma sentinel / CPM+Qwen XML) + `role:"tool"` loops | on | both | — | request `tools` |
+| Tool calling (Gemma sentinel / CPM+Qwen XML / GLM `arg_key`+`arg_value`) + `role:"tool"` loops | on | both | — | request `tools` |
 | Vision (`image_url` parts; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on (models with a tower); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial | L1/L2 | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
 | **Audio input** (`input_audio`/`audio`/`audio_url` parts; WAV native, mp3/m4a/flac/ogg/aiff via CoreAudio; ≤30 s / 750 tokens per clip; mixes with images) | on (models with `audio_config` + sidecar tower: e4b); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial³ | L2 (greedy stream EXACT vs the optiq internal model) | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
 | LoRA adapters (mount at start / hot-swap at runtime) | off | serial | — | `--adapter <dir>`, `POST /v1/adapters` |
@@ -68,6 +69,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | MiniCPM5 (cpm5) | ✅ L1/L2 | ✅ | the starter model |
 | Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
 | Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
+| GLM-5.2 / Colibri (pre-release port gate) | ✅ chat/text, Messages, Responses, SSE, tools, grammar, logprobs, library/CLI | ✅ compressed MLA/DSA scheduler | native MTP defaults to serial+spec; `--mtp off` enables ordinary batching; artifact-aware CLI/API/status `fit` and live main/MTP expert-tier telemetry; embeddings, vision/audio, adapters, and training are explicitly unsupported; per-row batched MTP is post-release |
 | DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
 | Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family/sliding universal → serial |