mlx-bun 0.0.11 → 0.0.13

package/docs/reference/features-matrix.md

@@ -22,7 +22,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | OpenAI Responses shim (`/v1/responses`) | on | both | — | — |
 | Continuous batching (mlx-lm B=N parity) | **on** — cap 8 (default flipped 2026-07-05; a lone request runs the exact serial engine, so the cap engages only under real concurrency; `--batch 1` pins serial) | batch | L1 | `--batch <n>` |
 | Prompt cache (prefix KV reuse) | on, 8 GB | both | — | `--prompt-cache <GB>` (0 = off) |
-| **SSD KV cold tier** (cache survives eviction + restarts) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
+| **SSD KV cold tier** (cache survives eviction + restarts; v3 includes GLM compressed MLA/DSA state without full-K/V reconstruction) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
 | **Runtime isolation** (engine runs as a crash-isolated CHILD process behind a reverse-proxy parent; UI stays instant under GPU load and survives engine crashes — [runtime-isolation.md](../design/runtime-isolation.md)) | off | both | — | `--isolate` |
 | **Model pool** (with `--isolate`: LRU-capped RESIDENT model engines; over the cap the least-recently-used engine demotes its cache to the SSD tier and exits, respawning on the next request) | 1 resident engine | both | — | `--model-pool <n>` (requires `--isolate`) |
 | Mixed-precision KV (`kv_config.json`, optiq's scheme) | **off** — opt-in (default flipped to bf16 2026-07-05; quantized KV trades 5–20% decode for memory headroom) | serial + **batch** (per-layer configs batch on every shipped model — full-attention layers Phase 3.1, rotating layers milestone 2; uniform bits stay serial) | L2 | `--kv-quant config\|off\|4\|8`, `--l2` |
@@ -32,6 +32,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | Fused SDPA (optiq-exact quantized-KV attention) | follows `--kv-quant`: on for `config`, off for uniform/bf16 | serial | L2 | `--fused-sdpa on\|off` |
 | **Paged KV** (vLLM-style block pool: host-side block table + gather before the stock SDPA, no fused kernel in v1 — [paged-kv-cache.md](../design/paged-kv-cache.md)) | off | serial (pins `--batch 1` unless `--batch` given) | gated bit-exact vs the plain `KVCache` (`tests/paged-kv-parity.test.ts`) | `--paged-kv`, `--paged-kv-block-size <n>` (default 256) — v1 scope Gemma4-family bf16; refuses `--batch N>1`, `--kv-quant`, `--draft-model` |
 | **Speculative decoding** (auto-detected drafter: two-model = mlx-lm parity/L1, Gemma `-assistant` = optiq/L2, DSpark local = Lab, DeepSpec released = DeepSpec-reference oracle) | off | serial (forces all-serial) | per-drafter oracle | `--draft-model <path\|query>`, `--draft-kind`, `--num-draft-tokens` |
+| **GLM-5.2 checkpoint-native MTP** (bounded auxiliary expert tier, exact verify) | **on** for GLM-5.2 | serial (`usage.lane: serial+spec`) | Colibri/oracle trajectory + synthetic HTTP gate | `--mtp on\|off` (`off` enables ordinary GLM batching) |
 | Speculative decoding, **model-free prompt lookup** (drafts copied from the request's own context; port of prompt-lookup decoding / vLLM `ngram`; zero weights) | off | serial (forces all-serial) | lossless by verify (gated vs non-spec greedy, tests/spec-ngram.test.ts) | `--draft-kind ngram` (no `--draft-model`), `--ngram-max`, `--ngram-min`, `--num-draft-tokens` (default 10) |
 | Memory admission (refuse what can't fit; never GPU-OOM) | on (RAM × 0.75) | both | — | `--memory-budget <GB>` |
 | Aggregate KV admission for batch rows (queue, don't OOM) | off | batch | — | `--kv-budget <GB>` |
@@ -48,7 +49,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | `guided_grammar` (EBNF) / `guided_regex`¹ / `guided_choice` / `structured_outputs` | on | both | L2 | request fields |
 | Structured output × speculative decoding | on (when both active) | serial | Lab (validity+equivalence) | — |
 | Quantized KV × speculative decoding (any axis: uniform / config / turbo) | KV scheme wins — drafted requests decode serially WITHOUT speculation (spec lane is bf16-KV-only in v1; startup warning) | serial | — | omit `--kv-quant` to speculate |
-| Tool calling (Gemma sentinel / CPM+Qwen XML) + `role:"tool"` loops | on | both | — | request `tools` |
+| Tool calling (Gemma sentinel / CPM+Qwen XML / GLM `arg_key`+`arg_value`) + `role:"tool"` loops | on | both | — | request `tools` |
 | Vision (`image_url` parts; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on (models with a tower); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial | L1/L2 | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
 | **Audio input** (`input_audio`/`audio`/`audio_url` parts; WAV native, mp3/m4a/flac/ogg/aiff via CoreAudio; ≤30 s / 750 tokens per clip; mixes with images) | on (models with `audio_config` + sidecar tower: e4b); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial³ | L2 (greedy stream EXACT vs the optiq internal model) | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
 | LoRA adapters (mount at start / hot-swap at runtime) | off | serial | — | `--adapter <dir>`, `POST /v1/adapters` |
@@ -68,6 +69,8 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | MiniCPM5 (cpm5) | ✅ L1/L2 | ✅ | the starter model |
 | Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
 | Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
+| Qwen3.8-27B (same qwen3_5 graph) | ✅ L1 (no L2 oracle) | ✅ (SSM path) | thinking depths + preserve_thinking; MTP/vision/video in bring-up |
+| GLM-5.2 / Colibri (pre-release port gate) | ✅ chat/text, Messages, Responses, SSE, tools, grammar, logprobs, library/CLI | ✅ compressed MLA/DSA scheduler | native MTP defaults to serial+spec; `--mtp off` enables ordinary batching; artifact-aware CLI/API/status `fit` and live main/MTP expert-tier telemetry; embeddings, vision/audio, adapters, and training are explicitly unsupported; per-row batched MTP is post-release |
 | DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
 | Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family/sliding universal → serial |
 

package/docs/reference/library-api.md

@@ -16,9 +16,7 @@ surface); direct library callers must do the same.
 ```ts
 import {
   ensureNativeRuntime,
-  loadModelConfig,
-  Weights,
-  createModel,       // dispatches to Gemma4Model / MiniCPM5Model / Qwen35Model
+  openModel,         // artifact-aware dispatch, including bounded GLM-5.2
   loadTokenizer,
   ChatTemplate,
   generate,
@@ -26,13 +24,15 @@ import {
 
 await ensureNativeRuntime();
 const dir = "/path/to/hf-snapshot";          // mlx-bun ls prints these
-const config = await loadModelConfig(dir);
-const model = createModel(await Weights.open(dir), config);   // returns RuntimeModel
+const model = await openModel(dir, {
+  // Used only by the direct Colibri GLM-5.2 artifact; other models ignore it.
+  contextTokens: 4096, maxGenerationTokens: 256, enableMtp: false,
+});
 const tok = await loadTokenizer(dir);
 const template = await ChatTemplate.load(dir);
 // Gemma4Model and MiniCPM5Model can still be imported directly from "mlx-bun"
 // if you need the concrete type. Qwen35Model is NOT exported from the public
-// package — use createModel/RuntimeModel (preferred dispatch), or import from
+// package — use openModel/RuntimeModel (preferred artifact-aware dispatch), or import from
 // "./src/model/qwen3_5" in-repo.
 
 const ids = tok.encode(template.render([{ role: "user", content: "hi" }]));
@@ -45,13 +45,15 @@ console.log(tok.decode(out, true));
 console.log(gen.stats);                          // set once iteration ends
 ```
 
-Weight loading is lazy (mmap + mlx native loader): construction is
-milliseconds; weights materialize on first forward.
+Ordinary safetensors loading is lazy (mmap + mlx native loader). GLM-5.2 first
+runs its header-only exact process equation, then opens the bounded Colibri
+resident/expert tiers; impossible plans fail before either tier is committed.
 
 > **First run:** library consumers bypass the CLI's first-run step, so
 > call `await ensureNativeRuntime()` once before constructing a model on
 > a machine that may not have the MLX native runtime yet — it downloads
-> the sha256-verified native pack to `~/Library/Caches/mlx-bun/` and is
+> the sha256-verified native pack (MLX plus the direct-Colibri expert-I/O
+> helper) to `~/Library/Caches/mlx-bun/` and is
 > a no-op when the runtime is already present (beside the executable,
 > already cached, or installed via homebrew). `nativeRuntimeDir()`
 > returns the resolved directory, or `null` on a fresh machine.
@@ -109,6 +111,24 @@ budget (pre-GPU, mmap-only check). The server exposes OpenAI chat
 completions, Anthropic `/v1/messages`, OpenAI Responses, embeddings,
 adapters, and `/stats` — [server-api.md](./server-api.md).
 
+For GLM-5.2, pass its load-only resource controls under `glm`:
+
+```ts
+const ctx = await loadContext(dir, "glm-5.2", {
+  glm: {
+    contextTokens: 4096,
+    maxGenerationTokens: 128,
+    batchSize: 8,
+    enableMtp: true, // default; native MTP routes requests serial+spec
+  },
+});
+```
+
+`openModel()` and `loadContext()` both select the streamed Colibri runtime;
+callers do not use `Weights.open()` or `createModel()` for this artifact.
+Direct `generate()` supports the full sampling/grammar/logprobs contract but
+does not itself mount a drafter; native MTP is a serving-context feature.
+
 ## Text embeddings
 
 ```ts
@@ -154,16 +174,24 @@ under one adapter must not seed another's prefill).
 ```ts
 import { saveKvCache, loadKvCache } from "./src/kv-store";
 saveKvCache("/tmp/prefix.kv", tokens, caches);       // page-aligned file
-const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model);
-// reload is a zero-copy MAP_PRIVATE mmap straight to the GPU (~1 ms);
-// keep `mmap` referenced as long as the caches live
+const { tokens, caches } = loadKvCache("/tmp/prefix.kv", model, {
+  modelId, configFingerprint, tokenizerHash, verify: true,
+});
+// Restore streams one tensor at a time from a read-only mmap into MLX-owned
+// storage, drops each clean file-page range, and unmaps before returning.
 // Note: loadKvCache accepts anything with makeCache(): Cache[] — it validates
 // the entry count against the cache list makeCache() returns (not
 // model.layers.length, which is wrong for KV-shared models like e4b, whose
 // makeCache() returns donors only). RuntimeModel qualifies directly, no cast.
 ```
 
-Quantized caches are not persistable yet (documented gap).
+Format v3 supports plain, rotating, quantized, TurboQuant, SSM, and GLM-5.2
+checkpoint-native caches. GLM rows use `mla`, `mla-dsa`, and `mtp-mla` kinds and
+store only compressed latent/RoPE/DSA state—never reconstructed full K/V.
+`Glm52NativeMtpSource.clonePersistentCache()` returns an owned MTP snapshot;
+`restorePersistentCache()` adopts a validated restored `mtp-mla` row. Writes
+are page-aligned, hashed, atomic (`.tmp` + fsync + rename), and available in
+both synchronous and event-loop-yielding `saveKvCacheAsync` forms.
 
 ## Registry + fit (model discovery and memory math)
 

package/docs/reference/models.md

@@ -35,6 +35,11 @@ mlx-bun get <substring>          # no "/" = registry query, re-gets the match
 - Plain HTTPS resolve/CDN (no Xet), sequential files, `Range`-resume of
   partial blobs (`<blob>.incomplete`). Auth: `HF_TOKEN` env, then the
   token `hf auth login` writes. 401/403 answers get a "gated repo" hint.
+- Before the first payload request, mlx-bun totals the bytes still absent,
+  credits complete shared blobs and valid `.incomplete` prefixes, and checks
+  the target volume for that remainder plus a fixed 1 GiB safety reserve. An
+  impossible acquisition fails immediately with required/available GiB and a
+  rerun/resume instruction rather than filling the disk halfway through.
 - A substring argument (no `/`) resolves against already-downloaded
   repos — `mlx-bun get 12B` refreshes the 12B to upstream's latest. An
   unknown substring errors with a pointer to `mlx-bun ls <q>`.
@@ -51,14 +56,56 @@ mlx-bun get <substring>          # no "/" = registry query, re-gets the match
 Every blob is checksummed **while streaming** (a resume re-hashes the
 existing prefix in chunks — no whole-file allocation):
 
-- **LFS files** (weights): sha256 must equal the API's `lfs.oid` — which
-  is also the blob's filename.
+- **LFS files** (weights): sha256 must equal the API's current `lfs.sha256`
+  field (the older `lfs.oid` spelling is also accepted) — which is also the
+  blob's filename.
 - **Small files** (configs, tokenizer): git blob identity,
   `sha1("blob <size>\0" + content)`, must equal the API's `blobId`.
 
 A mismatch deletes the partial (never resume corrupt bytes); a short
 read keeps the `.incomplete` for resume.
 
+## GLM-5.2 on a 32 GB Mac
+
+The validated direct-container artifact is
+`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp` at revision
+`3cc8db99b1b13fc79325d987ba3c1c430766b3b8`. It is approximately 384 GB
+decimal / 357 GiB on disk, so choose the Hugging Face cache volume before the
+download. Fast internal NVMe is strongly preferred; an external volume also
+needs to sustain the expert-streaming workload after acquisition.
+
+```sh
+# Optional: choose a large fast volume before downloading.
+export HF_HUB_CACHE=/Volumes/FastSSD/huggingface/hub
+
+mlx-bun get mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
+  --revision 3cc8db99b1b13fc79325d987ba3c1c430766b3b8
+mlx-bun scan
+mlx-bun serve GLM-5.2-colibri \
+  --memory-budget 26.8435456 \
+  --context-length 4096 \
+  --max-tokens 128 \
+  --mtp on
+```
+
+`26.8435456` decimal GB is exactly 25 GiB. The validated plan reserves about
+19.9 GiB for the process and leaves the rest of the 32 GiB machine to macOS.
+Startup runs the exact header-only resource equation before opening resident
+weights or expert slabs. The server reports the resulting plan at
+`GET /stats` under `glm52`.
+
+If acquisition is interrupted, rerun the identical `get` command: mlx-bun
+rehashes each existing prefix in bounded chunks, sends a Range request for the
+remainder, and verifies the complete blob before publishing it. If the disk
+preflight refuses, free space or point `HF_HUB_CACHE` at another volume and
+rerun. No Python environment and no converted copy inside the mlx-bun checkout
+are involved.
+
+The artifact is an external MIT-licensed derivative of
+`zai-org/GLM-5.2-FP8`, converted with Colibri and extended with int8 MTP
+weights. Model weights are not bundled with mlx-bun; review the artifact model
+card before redistribution.
+
 ## scan + the registry
 
 `mlx-bun scan` walks `models--*/snapshots/*` reading only `config.json`

package/docs/reference/server-api.md

@@ -50,14 +50,25 @@ Request body (OpenAI chat schema; unknown fields ignored):
   "tools": [ /* OpenAI function tools */ ],
   "tool_choice": "auto",         // "none" disables tools
   "chat_template_kwargs": {      // forwarded to the chat template
-    "enable_thinking": false     // MiniCPM5 / Qwen3.5: <think> channel on/off
+    "enable_thinking": false,    // MiniCPM5 / Qwen3.5/3.8 / GLM-5.2: <think>
+                                 // channel on/off (Qwen3.8 default: ON)
+    "preserve_thinking": true    // Qwen3.8: keep think blocks from earlier
+                                 // assistant turns in the rendered prompt
+                                 // (template default true — better prompt-
+                                 // cache reuse in agent loops). Ignored by
+                                 // templates that don't read it.
   },
-  "reasoning_effort": "medium",  // "none"|"minimal"|"low"|"medium"|"high"
-                                 // gates enable_thinking on Qwen3.5/MiniCPM5:
-                                 // "none" → off, any other level → on. Only
-                                 // consulted when chat_template_kwargs
-                                 // .enable_thinking is not explicitly set —
-                                 // an explicit enable_thinking always wins.
+  "reasoning_effort": "medium",  // "none"|"minimal"|"low"|"medium"|"high"|"xhigh"
+                                 // gates enable_thinking on Qwen3.5/3.8/
+                                 // MiniCPM5: "none" → off, any other level →
+                                 // on. Only consulted when chat_template_
+                                 // kwargs.enable_thinking is not explicitly
+                                 // set — an explicit enable_thinking always
+                                 // wins. On templates with a reasoning-depth
+                                 // variable (Qwen3.8) the level also maps
+                                 // into the template: minimal/low → low,
+                                 // medium → medium, high/xhigh → xhigh
+                                 // (Qwen3.8's default depth is xhigh).
   "hlg": {                       // HLG tone-curve sampling (per request).
     "enabled": true,             // merged over --hlg-sampling server defaults.
     "width": 4,                  // logit-width of the tone plateau
@@ -78,6 +89,11 @@ Sampling defaults follow the model author's `generation_config.json`
 when a field is omitted (optiq serve's gen_config behavior); explicit
 request values always win. MiniCPM5 defaults to the no-think direct
 answer mode unless `chat_template_kwargs.enable_thinking` is `true`.
+GLM-5.2 also defaults to no-think: its generation primer is
+`<think></think>`. Enabling thinking renders `Reasoning Effort: Max` in the
+system turn and leaves an open `<think>` primer for generation. The same
+precedence applies through Chat Completions, Anthropic Messages, and Responses
+because all three normalize into this one chat request path.
 
 `stop` sequences are matched on **decoded text**, not token ids, so a
 sequence that spans token boundaries still fires. Generation halts at
@@ -303,13 +319,16 @@ with `finish_reason: "stop"`. Works on both lanes — serial and
 `--batch N` (per-row matchers).
 
 Degrade path — a grammar that fails to compile is never rejected (no
-400/500): chat prepends a system message instructing valid JSON output
-(schema included when one was given) and the response carries a
-`Warning` header (`grammar not enforced: …`); `/v1/completions` has no
-chat template to inject into, so it emits the `Warning` header only.
-`MLX_BUN_GRAMMAR=0` disables the feature entirely (the fields are
-ignored: no enforcement, no injection, no Warning). Design and fidelity
-notes: [structured-output.md](../design/structured-output.md).
+400/500): chat prepends a string-valued system instruction for the
+requested JSON schema, raw grammar, regex, or choice constraint, and the
+response carries a `Warning` header (`grammar not enforced: …`);
+`/v1/completions` has no chat template to inject into, so it emits the
+`Warning` header only.
+`MLX_BUN_GRAMMAR=0` disables mask compilation but does not silently
+ignore a requested constraint: it uses this same degrade path (chat
+prompt injection plus `Warning`; raw completions `Warning` only).
+Design and fidelity notes:
+[structured-output.md](../design/structured-output.md).
 
 ### Errors
 
@@ -325,11 +344,12 @@ All errors are `{ "error": { "message": …, ... } }`.
   `top_logprobs must be of type int` / `at least 0` / `at most 11`;
   `top_logprobs: -1` is the accepted "unset" sentinel).
 - `400` with `"type": "memory_admission"`, `"code":
-  "context_over_budget"` — `prompt + max_tokens` exceeds the memory
-  budget's max safe context (only when serving with `--memory-budget`;
-  the GPU OOM this prevents would kill the process, so it is refused
-  up front). Lower `max_tokens` or shorten the prompt; the ceiling is
-  visible at `/stats`.
+  "context_over_budget"` — the request cannot fit the memory budget's max
+  safe context. Generic budgeted serving rejects when `prompt + max_tokens`
+  exceeds the ceiling. GLM-5.2 instead caps a broad `max_tokens` upper bound
+  to the space remaining in its fixed planned context and returns this error
+  only when the prompt itself leaves no generation slot. The ceiling is visible
+  at `/stats`.
 
 ## POST /v1/completions (raw text completion)
 
@@ -486,6 +506,23 @@ The served model is FIRST, with extra capability fields:
                                  //   lazily loadable)
   "audio": false,                // input_audio content parts accepted
                                  //   (tower loaded or lazily loadable)
+  "batch_mode": "batch",       // loaded cache capability; per-request
+                                 //   usage.lane is authoritative
+  "tools": true,
+  "structured_output": true,
+  "embeddings": false,
+  "adapters": false,
+  "training": false,
+  "dsa": true,                  // GLM-5.2 only; false elsewhere
+  "mtp": true,                  // checkpoint-native GLM MTP is mounted
+  "capabilities": {             // explicit discoverable surface
+    "chat_completions": true, "text_completions": true,
+    "anthropic_messages": true, "responses": true,
+    "streaming": true, "tools": true, "structured_output": true,
+    "logprobs": true, "embeddings": false,
+    "vision": false, "audio": false,
+    "adapters": false, "training": false
+  },
   "gen_defaults": {              // model-author sampling defaults
     "temperature": 0.7,          //   (generation_config.json, with any
     "top_p": 0.95,               //   --temperature/--top-p/--top-k server
@@ -518,6 +555,13 @@ matching `mlx_lm.server`.
 `input_audio` content parts (tower loaded or lazily loadable — see
 *Audio input* above for what audio requires), so clients can discover
 capabilities instead of probing for a `400`.
+The served-model row also includes `batch_mode: "off" | "serial" | "batch"`:
+the configured cap and the loaded model's cache capability together. Per-turn
+`usage.lane` remains authoritative for a particular request; for example,
+native-MTP requests route serial even when the served GLM model reports
+`batch_mode: "batch"` for ordinary requests.
+For GLM-5.2, the non-generative fields are deliberately false: embeddings,
+vision/audio, adapters, and training are not emulated by the serving port.
 
 ## GET /health
 
@@ -544,9 +588,40 @@ capabilities instead of probing for a `400`.
     "usable_bytes": 0,
     "weights_bytes": 0
   },
+  // present only for the direct Colibri GLM-5.2 runtime:
+  "glm52": {
+    "preset": "g5-32gb-quality",
+    "planned_process_bytes": 0,
+    "process_limit_bytes": 0,
+    "context_tokens": 4096,
+    "max_generation_tokens": 128,
+    "batch_size": 1,
+    "dsa": true,
+    "mtp": true,
+    "mtp_draft_tokens": 3,
+    "resident_weight_bytes": 0,
+    "main_expert_slab_bytes": 0,
+    "mtp_expert_slab_bytes": 0,
+    "expert_runtime": {
+      "main_residency": { "resident": 0, "pinned": 0, "working": 0,
+                            "hits": 0, "misses": 0, "evictions": 0 },
+      "mtp_residency": { "resident": 0, "pinned": 0, "working": 0,
+                           "hits": 0, "misses": 0, "evictions": 0 },
+      "last_turn": {
+        "main": { "demand": { "hits": 0, "misses": 0,
+                                  "readBytes": 0 } },
+        "mtp": { "demand": { "hits": 0, "misses": 0,
+                                 "readBytes": 0 } },
+        "pilot": null
+      },
+      "last_repin": []
+    }
+  },
   "batch": {
     "configured": 1,                  // the --batch N value
-    "batched": false,                 // batching enabled (N>1)
+    "mode": "off",                   // "off" | "serial" | "batch": configured
+                                           // and model-cache capability together
+    "batched": false,                 // true only when mode == "batch"
     "active_rows": 0,                 // rows currently decoding in the batch
     "pending_rows": 0,                // queued + mid-prefill rows waiting
     "submitted_rows": 0,              // cumulative rows routed to the batch lane since start
@@ -955,6 +1030,15 @@ banner and `mlx-bun fit`. When the eval DB has a real measured decode
 rate for this model snapshot, it is included and takes precedence over
 the prediction.
 
+For the direct Colibri GLM-5.2 runtime this endpoint switches to the exact
+serve-time memory plan. The response separates the full streamed artifact on
+disk from resident weights, main/MTP expert slabs, KV, and other reserves.
+`measured_decode_tps` is the G5 warm quality-preserving measurement;
+`glm52.direct_oracle_warm_decode_tps` is the same-machine direct Colibri
+control, while `glm52.aspirational_decode_tps` is a target, not a release
+gate. The single-row SKU result describes the current machine because the
+generic all-resident SKU estimator is inapplicable.
+
 ```jsonc
 {
   "machine": { "chip": "M4 Pro", "ram_bytes": 0, "bandwidth_gbs": 0.0 },
@@ -973,6 +1057,15 @@ the prediction.
     "max_safe_context": 8192,
     "predicted_decode_tps": 0.0
   },
+  // GLM-5.2 only:
+  "glm52": {
+    "artifact_disk_bytes": 0,       // exact registry size; null if unavailable
+    "main_expert_slab_bytes": 0,
+    "mtp_expert_slab_bytes": 0,
+    "max_generation_tokens": 128,
+    "direct_oracle_warm_decode_tps": 0.27,
+    "aspirational_decode_tps": 2.0
+  },
   "sku_matrix_ctx": 32768,
   "sku_matrix": [{
     "sku": "M4 Pro 24 GB", "ram_gb": 24,

package/docs/reference/server-config.md

@@ -21,7 +21,7 @@ the design rationale is in
 | --- | --- | --- | --- |
 | `--host` | addr | `127.0.0.1` | Interface to bind. Loopback-only by default (mlx_lm.server parity); pass `--host 0.0.0.0` to expose the server on your network. |
 | `--port` | n | `8080` | Listen port. A pre-flight probe refuses to start if the port is already serving. |
-| `--memory-budget` | GB | machine RAM × 0.75 | Admission ceiling. Requests whose `prompt + max_tokens` exceed the budget's max safe context are **rejected with 400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. Also caps the mlx allocator (`mlx_set_memory_limit`) as defense in depth. **Decimal GB (×10⁹).** |
+| `--memory-budget` | GB | machine RAM × 0.75; GLM-5.2: min(25 GiB, physical RAM) | Admission ceiling. Generic budgeted serving rejects requests whose `prompt + max_tokens` exceed the safe context with **400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. GLM-5.2 runs its exact process equation before opening resident weights or expert slabs and caps MLX to the plan's allocator reserve; because its context is fixed up front, a broad client `max_tokens` upper bound is capped to the remaining planned context, while a prompt that leaves no generation slot is still rejected. **Decimal GB (×10⁹).** |
 | `--prompt-cache` | GB | `8` | RAM prefix-KV cache (byte-capped LRU; `0` disables). **PREFIX SHARING (2026-07-05): serves are NON-CONSUMING** — a hit hands out zero-copy clones and leaves the donor entry intact, so N agents (or a brand-new session) sharing a system prompt all reuse ONE prefill without destroying each other's conversation entries (the old consume-and-trim take cannibalized the donor). Extended entries supersede their prefix-ancestors (when trimmable), so a conversation stays one entry. Works on BOTH lanes and tiers over the SSD store (Layer 0). |
 | `--isolate` | (bool) | off | Runtime isolation ([runtime-isolation.md](../design/runtime-isolation.md)): the inference engine runs as a CHILD process on a unix socket; this process stays a pure UI/API reverse proxy — instant under any GPU load, survives engine crashes (auto-respawn, in-flight requests get a clean 502). Measured cost ≈0 (−0.4% tok/s, +2 ms TTFT, per-token SSE granularity preserved). `/ws/chat` is not proxied yet (501). `/engine` reports child pid/restarts. |
 | `--model-pool` | n | `1` | With `--isolate`: max RESIDENT model engines. Requests naming another model (**exact `/v1/models` id** — fuzzy strings keep mlx-lm's ignored-field semantics) spawn/route to that model's own engine child (SPAWN-OVERLAP: the new model loads while the old one keeps serving; nobody's stream is interrupted). Over the cap, the LRU engine drains, **demotes its prompt cache to the SSD tier**, and exits — switching back respawns it with state restored from disk. Measured (M1 Max, cpm5⇄qwen0.8b, pool 1): switch 1.5 s, switch-back 1.2 s with `cached_tokens` 103/104 — the conversation survived the eviction. The pool drives this eviction via `POST /admin/drain` — unix-socket-only (never exposed on the public TCP listener) — which quiesces the gateway and demotes the whole prompt cache to the SSD tier before eviction, responding `{ drained: true, demotions: N }`. |
@@ -38,11 +38,13 @@ the design rationale is in
 | `--draft-kind` | `two-model`\|`assistant`\|`dspark`\|`deepspec`\|`ngram` | auto | Override the draft-artifact kind detection (`dspark.json` → dspark; `Gemma4DSparkModel` architecture stamp → deepspec; `*_assistant` config → assistant; else two-model). **`ngram` is never auto-detected — it has no artifact**: MODEL-FREE prompt lookup (drafts copied from the request's own prompt+generation when its trailing k-gram occurred earlier in the stream; port of prompt-lookup decoding / vLLM's `ngram` proposer). Mount it ALONE — `--draft-kind ngram` with a `--draft-model` is refused, as is any other kind without one. Zero weights/memory, lossless by the same verify (gated: serve-loop output token-identical to non-spec greedy on e4b, tests/spec-ngram.test.ts); a no-match round degrades to one plain target step. Best on agentic/RAG/code-edit traffic that re-emits context spans. |
 | `--num-draft-tokens` | n | `3` (`ngram`: `10`) | Drafts per verify round (mlx_lm.server's default; `mlx_lm.generate`'s is 2). A DSpark draft pins this to its trained block width (`cfg.gamma`); ngram drafting is free so its default is the reference implementation's 10. |
 | `--ngram-max` / `--ngram-min` | k | `3` / `1` | `--draft-kind ngram` only: longest/shortest trailing k-gram searched for a match (the reference values — longest first, first occurrence wins). |
-| `--thinking` | `true`\|`false` | model's own (false for CPM) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM and Qwen3.5 hybrid reasoning). Full precedence: an explicit request `chat_template_kwargs.enable_thinking` always wins; else `reasoning_effort` (`"none"` → off, any other level → on) if the request sent one; else this server default; else the model's own default. |
+| `--mtp` | `on`\|`off` | on for GLM-5.2 | Mount the checkpoint-native GLM MTP row as the server's drafter. It uses the bounded auxiliary expert tier and exact serial verify loop (`usage.lane: "serial+spec"`). `off` removes the draft so ordinary GLM requests can use continuous batching; per-row batched MTP is post-release. Mutually exclusive with an explicit `--draft-model`/`--draft-kind`. |
+| `--context-length` | tokens | `4096` for GLM-5.2 | Context reserved by GLM's header-only resource equation. The same value becomes the request-admission ceiling and is reported in `/stats.glm52`; an impossible plan fails before committing model memory. |
+| `--thinking` | `true`\|`false` | model's own (false for CPM and GLM-5.2) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM, Qwen3.5, and GLM-5.2). Full precedence: an explicit request `chat_template_kwargs.enable_thinking` always wins; else `reasoning_effort` (`"none"` → off, any other level → on) if the request sent one; else this server default; else the model's own default. GLM renders disabled thinking as `<think></think>` and enabled thinking as an open `<think>` generation primer plus `Reasoning Effort: Max`. |
 | `--temperature` | n ∈ [0,5] | `generation_config.json` | Server-wide sampling default. Per-request `temperature` still wins; the browser chat (sends none) inherits this. `--temp` is accepted as an alias (mlx_lm.server compat); explicit `--temperature` wins if both are given. **Migration note:** mlx_lm.server's `--temp` *default* is `0.0` (unset-temperature requests are greedy there); mlx-bun falls back to the model's `generation_config.json`, then `0.7` — pass `--temp 0` for mlx-lm's behavior. |
 | `--top-p` | n ∈ [0,1] | `generation_config.json` | Server-wide top-p default (per-request `top_p` wins). |
 | `--top-k` | n ∈ [0,1e6] | `generation_config.json` | Server-wide top-k default (per-request `top_k` wins). |
-| `--max-tokens` | n | `65536` chat / `512` raw completion | Completion cap when a request omits `max_tokens` (mlx_lm.server flag; its default there is 512 — `--max-tokens 512` reproduces mlx_lm.server exactly). |
+| `--max-tokens` | n | GLM-5.2: `128`; otherwise `65536` chat / `512` raw completion | Completion cap when a request omits `max_tokens` (mlx_lm.server flag; its default there is 512 — `--max-tokens 512` reproduces mlx_lm.server exactly). For GLM this is also reserved by the pre-open resource equation and must fit inside `--context-length`. |
 | `--allow-private-media` | (bool) | off | Let `image_url`/`audio_url` content parts fetch from **private/loopback/link-local** hosts (a NAS, another LAN box). Off by default: a request's URL is attacker-controlled input, so remote media fetches (`src/media-fetch.ts`) refuse non-http(s) schemes and destinations in private/loopback/link-local/CGNAT ranges — including hosts that *resolve* there and every **redirect hop** (each hop re-passes the whole policy) — which keeps the server from being steered at cloud metadata (`169.254.169.254`) or LAN-internal services (SSRF). Independent of the flag, every remote fetch has a **10 s wall-clock timeout** and a **64 MB response cap** (streaming-enforced, not just Content-Length); violations surface as clean `400`s (`prompt build failed: …`). `data:` URLs decode locally and are never policy-checked. |
 | `--no-open` | (bool) | off | Skip the automatic browser open on start. By default an interactive terminal session opens `http://<host>:<port>/#/chat` once the server is ready; pass this flag to suppress it (e.g. headless or non-TTY environments already skip it). |
 | `--hlg-sampling` | `on`\|`off` | off | Piecewise tone-curve (HLG) sampling: rolls off the top-token region, boosts the mids, gentles the tail. The overall gain folds from `--temperature`. See [docs/design/hlg-sampling.md](../design/hlg-sampling.md). |
@@ -112,7 +114,8 @@ for paired A/B harnesses).
 | `MLX_BUN_PAGED_KV` | `--paged-kv` | off (`==="1"`) | Paged KV cache (see the flag row above — v1: serial, Gemma4, bf16; refusals + prompt-cache bypass apply identically). |
 | `MLX_BUN_ALLOW_PRIVATE_MEDIA` | `--allow-private-media` | off (`==="1"`) | Permit `image_url`/`audio_url` fetches to private/loopback/link-local hosts (the SSRF guard's LAN escape hatch — see the flag row above; timeout + size cap still apply). |
 | `MLX_BUN_PREFILL_TAIL_SPLIT` | — | on (`!=="0"`) | Oracle prefill convention (mlx-lm `generate_step` and its server's batched engine): drain the prompt only to len−1, then compute step-0 logits from a separate **L=1 forward of the last prompt token**. Both lanes. **The SPEC lane (`--draft-model`) follows its own oracle's shape under the same flag** (mlx-lm `speculative_generate_step`, re-anchored 2026-07-07): both target AND draft drain to len−1 and there is **no separate step-0 at all** — the un-drained last prompt token heads the first verify window. Gated live: token-for-token vs the oracle venv incl. a knife-edge cell (tests/spec-serve.test.ts "L1 knife-edge"). (The standalone optiq-oracled assistant loop `specGenerate` deliberately keeps FULL-prompt prefill — optiq's own convention.) `=0` restores the pre-2026-07-07 full-final-chunk convention everywhere (A/B lever + kill switch) — that convention is ulp-different in bf16 at step 0 AND in the last prompt token's stored KV, which flips near-tie greedy streams vs mlx-lm (the 2026-07-07 12B completion-probe divergence). |
-| `MLX_BUN_GRAMMAR_JUMP` | — | **off** (`==="1"`) | Jump-forward decoding for structured output (SGLang's technique via xgrammar's `findJumpForwardString`): when the grammar forces a unique continuation (JSON punctuation/keys; longest under `any_whitespace:false` fixed formatting), the SERIAL lane emits its retokenized ids with ONE multi-token forward instead of one masked forward per token. Lossless in STRING space and always grammar-valid, but the token stream (and, conditioning on it, content after a forced span) can legally differ from an unjumped run — no oracle, hence opt-in. Excluded automatically for `logprobs` requests; SentencePiece-family tokenizers whose raw encode can't reproduce a mid-stream span simply never jump (guard degrades to normal decode). The batch lane's per-row grammar (`#stepGrammar`) does not jump yet. Tests: tests/grammar-jump.test.ts (contract + Llama-3.2-1B e2e). |
+| `MLX_BUN_GRAMMAR` | — | on (`!=="0"`) | `=0` disables structured-output mask compilation. Requested constraints still take the documented graceful-degrade route: chat prompt injection plus a `Warning` header; raw completions emit the header only. |
+| `MLX_BUN_GRAMMAR_JUMP` | — | **off** (`==="1"`) | Jump-forward decoding for structured output (SGLang's technique via xgrammar's `findJumpForwardString`): when the grammar forces a unique continuation (JSON punctuation/keys; longest under `any_whitespace:false` fixed formatting), the SERIAL lane emits its retokenized ids with ONE multi-token forward instead of one masked forward per token. Lossless in STRING space and always grammar-valid, but the token stream (and, conditioning on it, content after a forced span) can legally differ from an unjumped run — no oracle, hence opt-in. Excluded automatically when either `logprobs` or `top_logprobs` is requested, because jumped tokens have no sampled distribution row; SentencePiece-family tokenizers whose raw encode can't reproduce a mid-stream span simply never jump (guard degrades to normal decode). The batch lane's per-row grammar (`#stepGrammar`) does not jump yet. Tests: tests/grammar-jump.test.ts (contract + Llama-3.2-1B e2e). |
 | `MLX_BUN_SSD_WRITEBEHIND` | — | on (`!=="0"`) | `--ssd-cache`'s debounced write-behind snapshot (restart survival). `=0` disables it entirely — the paired-A/B lever + kill switch for the 2026-07-07 idle-gating fix (eviction/demotion spills still write). The flush is **idle-gated**: every per-tensor step (blocking GPU sync + `writeSync`) waits for the engine to go idle and pauses when a request arrives mid-flush, so durability work never taxes an active decode (pre-fix: a ~16k entry's flush overlapping cached ctx repeats cost e4b ~9% decode@ctx vs mlx-lm). The snapshot timer also never holds batch admission — while rows are active it re-arms instead of grabbing the exclusive. |
 | `MLX_BUN_SSD_SPILL_QUEUE_GB` | — | 2 | Byte cap on the write-behind queue's PENDING spill clones (they pin the evicted entries' GPU memory until the idle-gated flush runs). Over cap, the oldest queued spill drops — its clones free immediately and the entry becomes a future cache miss (never a wrong result). `=0` keeps only the newest + in-flight clone pinned (the soft cap never drops the item just enqueued). `/stats.ssd_cache` reports `pending_spills` / `pending_spill_bytes` / `dropped_spills`. Note: there is deliberately **no shutdown flush** — exiting under traffic loses queued write-behind entries (best-effort tier; cost = one re-prefill). |
 | `MLX_BUN_BATCH_NO_PIPELINE` | — | off (`==="1"`) | Disables the batch lane's pipelined token readback (the A/B lever for the shipped pipelining, like its peers `MLX_BUN_BATCH_EXTEND` / `MLX_BUN_BATCH_SSM` / `MLX_BUN_BATCH_VEC_SAMPLE`). Numerically equivalent, slower steps. |
@@ -211,7 +214,8 @@ inside it per the table above).
 
 | Option | serial (`--batch 1`) | `--batch N` (N>1) |
 | --- | --- | --- |
-| `--kv-quant config`/`4`/`8` | ✅ applied to all requests | ⚠️ applied, but forces **all** requests to the serial lane (no batching) |
+| `--kv-quant config` | ✅ applied to all requests | ✅ batches where the loaded cache capability supports the per-layer scheme; otherwise routes serial |
+| `--kv-quant 4`/`8` | ✅ applied to all requests | ⚠️ uniform-threshold semantics route requests serial |
 | `--kv-quant turbo[:k<bits>v<bits>]` | ✅ applied to all requests | ⚠️ applied, but forces **all** requests to the serial lane (no batching) |
 | `--kv-quant off` | ✅ bf16 | ✅ bf16 (same as the implicit batch default) |
 | *(kv-quant unset)* | bf16 (the L1 default) | **bf16** (Option B) — incl. serial-lane fallback requests |
@@ -227,6 +231,7 @@ inside it per the table above).
 | `tools` / `stop` | ✅ | ✅ (batches) |
 | structured output (`response_format`/`guided_*`) | ✅ (mask in the decode loop) | ✅ (batches; per-row matchers) |
 | `--draft-model` | ✅ spec decode (grammar composes) | ⚠️ mounts, but routes **every** request serial — spec and batching are different modes |
+| GLM `--mtp on` | ✅ native MTP spec decode | ⚠️ default-on MTP routes every request serial+spec; `--mtp off` exposes ordinary GLM batching |
 | `--compiled-decode` | ✅ (serial decode route) | ✅ at **B=1 only** (Phase 3.2): a lone request's adopted serial-class caches replay the same compiled step, same kill switch; B>1 steps run the plain graph |
 | `--fused-sdpa`/`--force-wire` | ✅ (serial decode route) | **n/a — compat mode, no perf flags by design** |
 
@@ -394,9 +399,9 @@ cells in one run).
   `--expert-offload`. Inherently the serial recipe until batched
   quantized KV lands.
 
-**The two exclusions to remember:** batching ⊕ kv-quant (batch is bf16 by
-contract; the resolver is a Lab-gated follow-up) and spec ⊕ prompt
-cache (v1 bypass; the §7.6 composition is the fix).
+**The two exclusions to remember:** batching excludes uniform/TurboQuant KV
+(per-layer `kv_config.json` does compose with batching), and spec excludes
+prompt-cache reuse (v1 bypass; the §7.6 composition is the fix).
 
 ## Observability — `GET /stats`
 
@@ -405,9 +410,11 @@ lives in [server-api.md](server-api.md#get-stats) (kept in lockstep with
 the code — this section only explains how to read it); highlights:
 `prompt_cache` (8 GB default cap), `kv_quant.mode` (incl. `turbo kXvY`),
 the conditional `ssd_cache` block, and `batch`
-(`active_rows`/`pending_rows`/`submitted_rows`/`kv_bytes`/`kv_budget_bytes`).
-
-`batch.batched` reflects only whether `--batch N` (N>1) is configured;
-with an explicit `--kv-quant` it can read `true` while `active_rows`
-never exceeds 1 (every request routes serial). `active_rows` is the
-honest signal of whether anything is actually batching.
+(`mode`/`active_rows`/`pending_rows`/`submitted_rows`/`kv_bytes`/
+`kv_budget_bytes`).
+
+`batch.mode` is the truthful configured/capability state: `off` for
+`--batch 1`, `serial` when a larger configured cap cannot batch the loaded
+model's cache layout, and `batch` when the model is admitted. `batch.batched`
+is the compatibility boolean for `mode == "batch"`; `active_rows` reports the
+instantaneous live rows.

package/docs/reference/training.md

@@ -177,7 +177,7 @@ are `DEFAULT_TRAIN_CONFIG` (trainer.ts:89).
 | `weight_decay` | float ≥0 | `0.01` | AdamW weight decay (β = `[0.9, 0.999]`, fixed) |
 | `lora_dropout` | float [0,1) | `0.0` | LoRA-input dropout (PEFT-style), training-only regularizer for small sets. Recompute-safe (mask keyed by step+layer, so segmented/checkpointed backward reproduces it) |
 | `rs_lora` | bool | `false` | rsLoRA — scale the update by α/√rank instead of α, so `rank_scaling` changes capacity not step size. Recorded in the adapter config; inference applies the same per-layer scale |
-| `lora_plus_ratio` | float ≥1 | `1.0` | LoRA+ — LR multiplier for the B leaves (A stays at base LR). >1 speeds the B-driven early learning (B is zero-init). 1 = off. Wired in the ORPO loop |
+| `lora_plus_ratio` | float ≥1 | `1.0` | LoRA+ — LR multiplier for the B leaves (A stays at base LR). >1 speeds the B-driven early learning (B is zero-init). 1 = off. The same optimizer policy applies to SFT, DPO, and ORPO |
 | `grad_checkpoint` | bool | `false` | Recompute layer activations in backward (memory↔compute; bit-identical) |
 | `mlp_split` | bool | `false` | Gradient-checkpoint MLP splitting (Gemma4-only, requires `grad_checkpoint`) |
 | `segment_size` | int | `0` (off) | `>0` enables segmented backward — layers per segment (see below) |
@@ -295,6 +295,15 @@ A finished run writes a PEFT-compatible adapter directory:
 - `optiq_lora_config.json` — mlx-bun/optiq adapter metadata (per-layer ranks)
 - `adapter_config.json` — PEFT-compatible config
 
+The save completes only after both config files are written, including
+eval-step checkpoints, so a successful process exit always leaves an
+immediately mountable directory. Serving accepts both mlx-lm tensor layout
+(`[in, rank]` / `[rank, out]`) and standard PEFT layout
+(`[rank, in]` / `[out, rank]`, including `base_model.model.*` names).
+Standard PEFT `use_rslora` metadata is honored as
+`lora_alpha / sqrt(rank)`, and saved rsLoRA adapters emit that metadata so
+their training and serving scales agree.
+
 When `save_checkpoints: true`, each eval step also writes
 `checkpoints/step-<NNNNN>-val<loss>/` and a durable `metrics.json` (config,
 wall seconds, peak GB, final/best train+val loss, full val trajectory).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mlx-bun",
-  "version": "0.0.11",
+  "version": "0.0.13",
   "description": "Native MLX inference for Bun on Apple Silicon — local LLM server (OpenAI/Anthropic-compatible) and TypeScript library. No Python.",
   "type": "module",
   "license": "MIT",