mlx-bun 0.0.11 → 0.0.12

package/docs/reference/library-api.md

@@ -16,9 +16,7 @@ surface); direct library callers must do the same.
 ```ts
 import {
   ensureNativeRuntime,
-  loadModelConfig,
-  Weights,
-  createModel,       // dispatches to Gemma4Model / MiniCPM5Model / Qwen35Model
+  openModel,         // artifact-aware dispatch, including bounded GLM-5.2
   loadTokenizer,
   ChatTemplate,
   generate,
@@ -26,13 +24,15 @@ import {
 
 await ensureNativeRuntime();
 const dir = "/path/to/hf-snapshot";          // mlx-bun ls prints these
-const config = await loadModelConfig(dir);
-const model = createModel(await Weights.open(dir), config);   // returns RuntimeModel
+const model = await openModel(dir, {
+  // Used only by the direct Colibri GLM-5.2 artifact; other models ignore it.
+  contextTokens: 4096, maxGenerationTokens: 256, enableMtp: false,
+});
 const tok = await loadTokenizer(dir);
 const template = await ChatTemplate.load(dir);
 // Gemma4Model and MiniCPM5Model can still be imported directly from "mlx-bun"
 // if you need the concrete type. Qwen35Model is NOT exported from the public
-// package — use createModel/RuntimeModel (preferred dispatch), or import from
+// package — use openModel/RuntimeModel (preferred artifact-aware dispatch), or import from
 // "./src/model/qwen3_5" in-repo.
 
 const ids = tok.encode(template.render([{ role: "user", content: "hi" }]));
@@ -45,13 +45,15 @@ console.log(tok.decode(out, true));
 console.log(gen.stats);                          // set once iteration ends
 ```
 
-Weight loading is lazy (mmap + mlx native loader): construction is
-milliseconds; weights materialize on first forward.
+Ordinary safetensors loading is lazy (mmap + mlx native loader). GLM-5.2 first
+runs its header-only exact process equation, then opens the bounded Colibri
+resident/expert tiers; impossible plans fail before either tier is committed.
 
 > **First run:** library consumers bypass the CLI's first-run step, so
 > call `await ensureNativeRuntime()` once before constructing a model on
 > a machine that may not have the MLX native runtime yet — it downloads
-> the sha256-verified native pack to `~/Library/Caches/mlx-bun/` and is
+> the sha256-verified native pack (MLX plus the direct-Colibri expert-I/O
+> helper) to `~/Library/Caches/mlx-bun/` and is
 > a no-op when the runtime is already present (beside the executable,
 > already cached, or installed via homebrew). `nativeRuntimeDir()`
 > returns the resolved directory, or `null` on a fresh machine.
@@ -109,6 +111,24 @@ budget (pre-GPU, mmap-only check). The server exposes OpenAI chat
 completions, Anthropic `/v1/messages`, OpenAI Responses, embeddings,
 adapters, and `/stats` — [server-api.md](./server-api.md).
 
+For GLM-5.2, pass its load-only resource controls under `glm`:
+
+```ts
+const ctx = await loadContext(dir, "glm-5.2", {
+  glm: {
+    contextTokens: 4096,
+    maxGenerationTokens: 128,
+    batchSize: 8,
+    enableMtp: true, // default; native MTP routes requests serial+spec
+  },
+});
+```
+
+`openModel()` and `loadContext()` both select the streamed Colibri runtime;
+callers do not use `Weights.open()` or `createModel()` for this artifact.
+Direct `generate()` supports the full sampling/grammar/logprobs contract but
+does not itself mount a drafter; native MTP is a serving-context feature.
+
 ## Text embeddings
 
 ```ts
@@ -154,16 +174,24 @@ under one adapter must not seed another's prefill).
 ```ts
 import { saveKvCache, loadKvCache } from "./src/kv-store";
 saveKvCache("/tmp/prefix.kv", tokens, caches);       // page-aligned file
-const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model);
-// reload is a zero-copy MAP_PRIVATE mmap straight to the GPU (~1 ms);
-// keep `mmap` referenced as long as the caches live
+const { tokens, caches } = loadKvCache("/tmp/prefix.kv", model, {
+  modelId, configFingerprint, tokenizerHash, verify: true,
+});
+// Restore streams one tensor at a time from a read-only mmap into MLX-owned
+// storage, drops each clean file-page range, and unmaps before returning.
 // Note: loadKvCache accepts anything with makeCache(): Cache[] — it validates
 // the entry count against the cache list makeCache() returns (not
 // model.layers.length, which is wrong for KV-shared models like e4b, whose
 // makeCache() returns donors only). RuntimeModel qualifies directly, no cast.
 ```
 
-Quantized caches are not persistable yet (documented gap).
+Format v3 supports plain, rotating, quantized, TurboQuant, SSM, and GLM-5.2
+checkpoint-native caches. GLM rows use `mla`, `mla-dsa`, and `mtp-mla` kinds and
+store only compressed latent/RoPE/DSA state—never reconstructed full K/V.
+`Glm52NativeMtpSource.clonePersistentCache()` returns an owned MTP snapshot;
+`restorePersistentCache()` adopts a validated restored `mtp-mla` row. Writes
+are page-aligned, hashed, atomic (`.tmp` + fsync + rename), and available in
+both synchronous and event-loop-yielding `saveKvCacheAsync` forms.
 
 ## Registry + fit (model discovery and memory math)
 

package/docs/reference/models.md

@@ -35,6 +35,11 @@ mlx-bun get <substring>          # no "/" = registry query, re-gets the match
 - Plain HTTPS resolve/CDN (no Xet), sequential files, `Range`-resume of
   partial blobs (`<blob>.incomplete`). Auth: `HF_TOKEN` env, then the
   token `hf auth login` writes. 401/403 answers get a "gated repo" hint.
+- Before the first payload request, mlx-bun totals the bytes still absent,
+  credits complete shared blobs and valid `.incomplete` prefixes, and checks
+  the target volume for that remainder plus a fixed 1 GiB safety reserve. An
+  impossible acquisition fails immediately with required/available GiB and a
+  rerun/resume instruction rather than filling the disk halfway through.
 - A substring argument (no `/`) resolves against already-downloaded
   repos — `mlx-bun get 12B` refreshes the 12B to upstream's latest. An
   unknown substring errors with a pointer to `mlx-bun ls <q>`.
@@ -51,14 +56,56 @@ mlx-bun get <substring>          # no "/" = registry query, re-gets the match
 Every blob is checksummed **while streaming** (a resume re-hashes the
 existing prefix in chunks — no whole-file allocation):
 
-- **LFS files** (weights): sha256 must equal the API's `lfs.oid` — which
-  is also the blob's filename.
+- **LFS files** (weights): sha256 must equal the API's current `lfs.sha256`
+  field (the older `lfs.oid` spelling is also accepted) — which is also the
+  blob's filename.
 - **Small files** (configs, tokenizer): git blob identity,
   `sha1("blob <size>\0" + content)`, must equal the API's `blobId`.
 
 A mismatch deletes the partial (never resume corrupt bytes); a short
 read keeps the `.incomplete` for resume.
 
+## GLM-5.2 on a 32 GB Mac
+
+The validated direct-container artifact is
+`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp` at revision
+`3cc8db99b1b13fc79325d987ba3c1c430766b3b8`. It is approximately 384 GB
+decimal / 357 GiB on disk, so choose the Hugging Face cache volume before the
+download. Fast internal NVMe is strongly preferred; an external volume also
+needs to sustain the expert-streaming workload after acquisition.
+
+```sh
+# Optional: choose a large fast volume before downloading.
+export HF_HUB_CACHE=/Volumes/FastSSD/huggingface/hub
+
+mlx-bun get mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
+  --revision 3cc8db99b1b13fc79325d987ba3c1c430766b3b8
+mlx-bun scan
+mlx-bun serve GLM-5.2-colibri \
+  --memory-budget 26.8435456 \
+  --context-length 4096 \
+  --max-tokens 128 \
+  --mtp on
+```
+
+`26.8435456` decimal GB is exactly 25 GiB. The validated plan reserves about
+19.9 GiB for the process and leaves the rest of the 32 GiB machine to macOS.
+Startup runs the exact header-only resource equation before opening resident
+weights or expert slabs. The server reports the resulting plan at
+`GET /stats` under `glm52`.
+
+If acquisition is interrupted, rerun the identical `get` command: mlx-bun
+rehashes each existing prefix in bounded chunks, sends a Range request for the
+remainder, and verifies the complete blob before publishing it. If the disk
+preflight refuses, free space or point `HF_HUB_CACHE` at another volume and
+rerun. No Python environment and no converted copy inside the mlx-bun checkout
+are involved.
+
+The artifact is an external MIT-licensed derivative of
+`zai-org/GLM-5.2-FP8`, converted with Colibri and extended with int8 MTP
+weights. Model weights are not bundled with mlx-bun; review the artifact model
+card before redistribution.
+
 ## scan + the registry
 
 `mlx-bun scan` walks `models--*/snapshots/*` reading only `config.json`

package/docs/reference/server-api.md

@@ -50,7 +50,7 @@ Request body (OpenAI chat schema; unknown fields ignored):
   "tools": [ /* OpenAI function tools */ ],
   "tool_choice": "auto",         // "none" disables tools
   "chat_template_kwargs": {      // forwarded to the chat template
-    "enable_thinking": false     // MiniCPM5 / Qwen3.5: <think> channel on/off
+    "enable_thinking": false     // MiniCPM5 / Qwen3.5 / GLM-5.2: think channel
   },
   "reasoning_effort": "medium",  // "none"|"minimal"|"low"|"medium"|"high"
                                  // gates enable_thinking on Qwen3.5/MiniCPM5:
@@ -78,6 +78,11 @@ Sampling defaults follow the model author's `generation_config.json`
 when a field is omitted (optiq serve's gen_config behavior); explicit
 request values always win. MiniCPM5 defaults to the no-think direct
 answer mode unless `chat_template_kwargs.enable_thinking` is `true`.
+GLM-5.2 also defaults to no-think: its generation primer is
+`<think></think>`. Enabling thinking renders `Reasoning Effort: Max` in the
+system turn and leaves an open `<think>` primer for generation. The same
+precedence applies through Chat Completions, Anthropic Messages, and Responses
+because all three normalize into this one chat request path.
 
 `stop` sequences are matched on **decoded text**, not token ids, so a
 sequence that spans token boundaries still fires. Generation halts at
@@ -303,13 +308,16 @@ with `finish_reason: "stop"`. Works on both lanes — serial and
 `--batch N` (per-row matchers).
 
 Degrade path — a grammar that fails to compile is never rejected (no
-400/500): chat prepends a system message instructing valid JSON output
-(schema included when one was given) and the response carries a
-`Warning` header (`grammar not enforced: …`); `/v1/completions` has no
-chat template to inject into, so it emits the `Warning` header only.
-`MLX_BUN_GRAMMAR=0` disables the feature entirely (the fields are
-ignored: no enforcement, no injection, no Warning). Design and fidelity
-notes: [structured-output.md](../design/structured-output.md).
+400/500): chat prepends a string-valued system instruction for the
+requested JSON schema, raw grammar, regex, or choice constraint, and the
+response carries a `Warning` header (`grammar not enforced: …`);
+`/v1/completions` has no chat template to inject into, so it emits the
+`Warning` header only.
+`MLX_BUN_GRAMMAR=0` disables mask compilation but does not silently
+ignore a requested constraint: it uses this same degrade path (chat
+prompt injection plus `Warning`; raw completions `Warning` only).
+Design and fidelity notes:
+[structured-output.md](../design/structured-output.md).
 
 ### Errors
 
@@ -486,6 +494,23 @@ The served model is FIRST, with extra capability fields:
                                  //   lazily loadable)
   "audio": false,                // input_audio content parts accepted
                                  //   (tower loaded or lazily loadable)
+  "batch_mode": "batch",       // loaded cache capability; per-request
+                                 //   usage.lane is authoritative
+  "tools": true,
+  "structured_output": true,
+  "embeddings": false,
+  "adapters": false,
+  "training": false,
+  "dsa": true,                  // GLM-5.2 only; false elsewhere
+  "mtp": true,                  // checkpoint-native GLM MTP is mounted
+  "capabilities": {             // explicit discoverable surface
+    "chat_completions": true, "text_completions": true,
+    "anthropic_messages": true, "responses": true,
+    "streaming": true, "tools": true, "structured_output": true,
+    "logprobs": true, "embeddings": false,
+    "vision": false, "audio": false,
+    "adapters": false, "training": false
+  },
   "gen_defaults": {              // model-author sampling defaults
     "temperature": 0.7,          //   (generation_config.json, with any
     "top_p": 0.95,               //   --temperature/--top-p/--top-k server
@@ -518,6 +543,13 @@ matching `mlx_lm.server`.
 `input_audio` content parts (tower loaded or lazily loadable — see
 *Audio input* above for what audio requires), so clients can discover
 capabilities instead of probing for a `400`.
+The served-model row also includes `batch_mode: "off" | "serial" | "batch"`:
+the configured cap and the loaded model's cache capability together. Per-turn
+`usage.lane` remains authoritative for a particular request; for example,
+native-MTP requests route serial even when the served GLM model reports
+`batch_mode: "batch"` for ordinary requests.
+For GLM-5.2, the non-generative fields are deliberately false: embeddings,
+vision/audio, adapters, and training are not emulated by the serving port.
 
 ## GET /health
 
@@ -544,9 +576,40 @@ capabilities instead of probing for a `400`.
     "usable_bytes": 0,
     "weights_bytes": 0
   },
+  // present only for the direct Colibri GLM-5.2 runtime:
+  "glm52": {
+    "preset": "g5-32gb-quality",
+    "planned_process_bytes": 0,
+    "process_limit_bytes": 0,
+    "context_tokens": 4096,
+    "max_generation_tokens": 128,
+    "batch_size": 1,
+    "dsa": true,
+    "mtp": true,
+    "mtp_draft_tokens": 3,
+    "resident_weight_bytes": 0,
+    "main_expert_slab_bytes": 0,
+    "mtp_expert_slab_bytes": 0,
+    "expert_runtime": {
+      "main_residency": { "resident": 0, "pinned": 0, "working": 0,
+                            "hits": 0, "misses": 0, "evictions": 0 },
+      "mtp_residency": { "resident": 0, "pinned": 0, "working": 0,
+                           "hits": 0, "misses": 0, "evictions": 0 },
+      "last_turn": {
+        "main": { "demand": { "hits": 0, "misses": 0,
+                                  "readBytes": 0 } },
+        "mtp": { "demand": { "hits": 0, "misses": 0,
+                                 "readBytes": 0 } },
+        "pilot": null
+      },
+      "last_repin": []
+    }
+  },
   "batch": {
     "configured": 1,                  // the --batch N value
-    "batched": false,                 // batching enabled (N>1)
+    "mode": "off",                   // "off" | "serial" | "batch": configured
+                                           // and model-cache capability together
+    "batched": false,                 // true only when mode == "batch"
     "active_rows": 0,                 // rows currently decoding in the batch
     "pending_rows": 0,                // queued + mid-prefill rows waiting
     "submitted_rows": 0,              // cumulative rows routed to the batch lane since start
@@ -955,6 +1018,15 @@ banner and `mlx-bun fit`. When the eval DB has a real measured decode
 rate for this model snapshot, it is included and takes precedence over
 the prediction.
 
+For the direct Colibri GLM-5.2 runtime this endpoint switches to the exact
+serve-time memory plan. The response separates the full streamed artifact on
+disk from resident weights, main/MTP expert slabs, KV, and other reserves.
+`measured_decode_tps` is the G5 warm quality-preserving measurement;
+`glm52.direct_oracle_warm_decode_tps` is the same-machine direct Colibri
+control, while `glm52.aspirational_decode_tps` is a target, not a release
+gate. The single-row SKU result describes the current machine because the
+generic all-resident SKU estimator is inapplicable.
+
 ```jsonc
 {
   "machine": { "chip": "M4 Pro", "ram_bytes": 0, "bandwidth_gbs": 0.0 },
@@ -973,6 +1045,15 @@ the prediction.
     "max_safe_context": 8192,
     "predicted_decode_tps": 0.0
   },
+  // GLM-5.2 only:
+  "glm52": {
+    "artifact_disk_bytes": 0,       // exact registry size; null if unavailable
+    "main_expert_slab_bytes": 0,
+    "mtp_expert_slab_bytes": 0,
+    "max_generation_tokens": 128,
+    "direct_oracle_warm_decode_tps": 0.27,
+    "aspirational_decode_tps": 2.0
+  },
   "sku_matrix_ctx": 32768,
   "sku_matrix": [{
     "sku": "M4 Pro 24 GB", "ram_gb": 24,

package/docs/reference/server-config.md

@@ -21,7 +21,7 @@ the design rationale is in
 | --- | --- | --- | --- |
 | `--host` | addr | `127.0.0.1` | Interface to bind. Loopback-only by default (mlx_lm.server parity); pass `--host 0.0.0.0` to expose the server on your network. |
 | `--port` | n | `8080` | Listen port. A pre-flight probe refuses to start if the port is already serving. |
-| `--memory-budget` | GB | machine RAM × 0.75 | Admission ceiling. Requests whose `prompt + max_tokens` exceed the budget's max safe context are **rejected with 400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. Also caps the mlx allocator (`mlx_set_memory_limit`) as defense in depth. **Decimal GB (×10⁹).** |
+| `--memory-budget` | GB | machine RAM × 0.75; GLM-5.2: min(25 GiB, physical RAM) | Admission ceiling. Requests whose `prompt + max_tokens` exceed the budget's max safe context are **rejected with 400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. GLM-5.2 runs its header-only exact process equation before opening resident weights or expert slabs, and caps MLX to that plan's allocator reserve. **Decimal GB (×10⁹).** |
 | `--prompt-cache` | GB | `8` | RAM prefix-KV cache (byte-capped LRU; `0` disables). **PREFIX SHARING (2026-07-05): serves are NON-CONSUMING** — a hit hands out zero-copy clones and leaves the donor entry intact, so N agents (or a brand-new session) sharing a system prompt all reuse ONE prefill without destroying each other's conversation entries (the old consume-and-trim take cannibalized the donor). Extended entries supersede their prefix-ancestors (when trimmable), so a conversation stays one entry. Works on BOTH lanes and tiers over the SSD store (Layer 0). |
 | `--isolate` | (bool) | off | Runtime isolation ([runtime-isolation.md](../design/runtime-isolation.md)): the inference engine runs as a CHILD process on a unix socket; this process stays a pure UI/API reverse proxy — instant under any GPU load, survives engine crashes (auto-respawn, in-flight requests get a clean 502). Measured cost ≈0 (−0.4% tok/s, +2 ms TTFT, per-token SSE granularity preserved). `/ws/chat` is not proxied yet (501). `/engine` reports child pid/restarts. |
 | `--model-pool` | n | `1` | With `--isolate`: max RESIDENT model engines. Requests naming another model (**exact `/v1/models` id** — fuzzy strings keep mlx-lm's ignored-field semantics) spawn/route to that model's own engine child (SPAWN-OVERLAP: the new model loads while the old one keeps serving; nobody's stream is interrupted). Over the cap, the LRU engine drains, **demotes its prompt cache to the SSD tier**, and exits — switching back respawns it with state restored from disk. Measured (M1 Max, cpm5⇄qwen0.8b, pool 1): switch 1.5 s, switch-back 1.2 s with `cached_tokens` 103/104 — the conversation survived the eviction. The pool drives this eviction via `POST /admin/drain` — unix-socket-only (never exposed on the public TCP listener) — which quiesces the gateway and demotes the whole prompt cache to the SSD tier before eviction, responding `{ drained: true, demotions: N }`. |
@@ -38,11 +38,13 @@ the design rationale is in
 | `--draft-kind` | `two-model`\|`assistant`\|`dspark`\|`deepspec`\|`ngram` | auto | Override the draft-artifact kind detection (`dspark.json` → dspark; `Gemma4DSparkModel` architecture stamp → deepspec; `*_assistant` config → assistant; else two-model). **`ngram` is never auto-detected — it has no artifact**: MODEL-FREE prompt lookup (drafts copied from the request's own prompt+generation when its trailing k-gram occurred earlier in the stream; port of prompt-lookup decoding / vLLM's `ngram` proposer). Mount it ALONE — `--draft-kind ngram` with a `--draft-model` is refused, as is any other kind without one. Zero weights/memory, lossless by the same verify (gated: serve-loop output token-identical to non-spec greedy on e4b, tests/spec-ngram.test.ts); a no-match round degrades to one plain target step. Best on agentic/RAG/code-edit traffic that re-emits context spans. |
 | `--num-draft-tokens` | n | `3` (`ngram`: `10`) | Drafts per verify round (mlx_lm.server's default; `mlx_lm.generate`'s is 2). A DSpark draft pins this to its trained block width (`cfg.gamma`); ngram drafting is free so its default is the reference implementation's 10. |
 | `--ngram-max` / `--ngram-min` | k | `3` / `1` | `--draft-kind ngram` only: longest/shortest trailing k-gram searched for a match (the reference values — longest first, first occurrence wins). |
-| `--thinking` | `true`\|`false` | model's own (false for CPM) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM and Qwen3.5 hybrid reasoning). Full precedence: an explicit request `chat_template_kwargs.enable_thinking` always wins; else `reasoning_effort` (`"none"` → off, any other level → on) if the request sent one; else this server default; else the model's own default. |
+| `--mtp` | `on`\|`off` | on for GLM-5.2 | Mount the checkpoint-native GLM MTP row as the server's drafter. It uses the bounded auxiliary expert tier and exact serial verify loop (`usage.lane: "serial+spec"`). `off` removes the draft so ordinary GLM requests can use continuous batching; per-row batched MTP is post-release. Mutually exclusive with an explicit `--draft-model`/`--draft-kind`. |
+| `--context-length` | tokens | `4096` for GLM-5.2 | Context reserved by GLM's header-only resource equation. The same value becomes the request-admission ceiling and is reported in `/stats.glm52`; an impossible plan fails before committing model memory. |
+| `--thinking` | `true`\|`false` | model's own (false for CPM and GLM-5.2) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM, Qwen3.5, and GLM-5.2). Full precedence: an explicit request `chat_template_kwargs.enable_thinking` always wins; else `reasoning_effort` (`"none"` → off, any other level → on) if the request sent one; else this server default; else the model's own default. GLM renders disabled thinking as `<think></think>` and enabled thinking as an open `<think>` generation primer plus `Reasoning Effort: Max`. |
 | `--temperature` | n ∈ [0,5] | `generation_config.json` | Server-wide sampling default. Per-request `temperature` still wins; the browser chat (sends none) inherits this. `--temp` is accepted as an alias (mlx_lm.server compat); explicit `--temperature` wins if both are given. **Migration note:** mlx_lm.server's `--temp` *default* is `0.0` (unset-temperature requests are greedy there); mlx-bun falls back to the model's `generation_config.json`, then `0.7` — pass `--temp 0` for mlx-lm's behavior. |
 | `--top-p` | n ∈ [0,1] | `generation_config.json` | Server-wide top-p default (per-request `top_p` wins). |
 | `--top-k` | n ∈ [0,1e6] | `generation_config.json` | Server-wide top-k default (per-request `top_k` wins). |
-| `--max-tokens` | n | `65536` chat / `512` raw completion | Completion cap when a request omits `max_tokens` (mlx_lm.server flag; its default there is 512 — `--max-tokens 512` reproduces mlx_lm.server exactly). |
+| `--max-tokens` | n | GLM-5.2: `128`; otherwise `65536` chat / `512` raw completion | Completion cap when a request omits `max_tokens` (mlx_lm.server flag; its default there is 512 — `--max-tokens 512` reproduces mlx_lm.server exactly). For GLM this is also reserved by the pre-open resource equation and must fit inside `--context-length`. |
 | `--allow-private-media` | (bool) | off | Let `image_url`/`audio_url` content parts fetch from **private/loopback/link-local** hosts (a NAS, another LAN box). Off by default: a request's URL is attacker-controlled input, so remote media fetches (`src/media-fetch.ts`) refuse non-http(s) schemes and destinations in private/loopback/link-local/CGNAT ranges — including hosts that *resolve* there and every **redirect hop** (each hop re-passes the whole policy) — which keeps the server from being steered at cloud metadata (`169.254.169.254`) or LAN-internal services (SSRF). Independent of the flag, every remote fetch has a **10 s wall-clock timeout** and a **64 MB response cap** (streaming-enforced, not just Content-Length); violations surface as clean `400`s (`prompt build failed: …`). `data:` URLs decode locally and are never policy-checked. |
 | `--no-open` | (bool) | off | Skip the automatic browser open on start. By default an interactive terminal session opens `http://<host>:<port>/#/chat` once the server is ready; pass this flag to suppress it (e.g. headless or non-TTY environments already skip it). |
 | `--hlg-sampling` | `on`\|`off` | off | Piecewise tone-curve (HLG) sampling: rolls off the top-token region, boosts the mids, gentles the tail. The overall gain folds from `--temperature`. See [docs/design/hlg-sampling.md](../design/hlg-sampling.md). |
@@ -112,7 +114,8 @@ for paired A/B harnesses).
 | `MLX_BUN_PAGED_KV` | `--paged-kv` | off (`==="1"`) | Paged KV cache (see the flag row above — v1: serial, Gemma4, bf16; refusals + prompt-cache bypass apply identically). |
 | `MLX_BUN_ALLOW_PRIVATE_MEDIA` | `--allow-private-media` | off (`==="1"`) | Permit `image_url`/`audio_url` fetches to private/loopback/link-local hosts (the SSRF guard's LAN escape hatch — see the flag row above; timeout + size cap still apply). |
 | `MLX_BUN_PREFILL_TAIL_SPLIT` | — | on (`!=="0"`) | Oracle prefill convention (mlx-lm `generate_step` and its server's batched engine): drain the prompt only to len−1, then compute step-0 logits from a separate **L=1 forward of the last prompt token**. Both lanes. **The SPEC lane (`--draft-model`) follows its own oracle's shape under the same flag** (mlx-lm `speculative_generate_step`, re-anchored 2026-07-07): both target AND draft drain to len−1 and there is **no separate step-0 at all** — the un-drained last prompt token heads the first verify window. Gated live: token-for-token vs the oracle venv incl. a knife-edge cell (tests/spec-serve.test.ts "L1 knife-edge"). (The standalone optiq-oracled assistant loop `specGenerate` deliberately keeps FULL-prompt prefill — optiq's own convention.) `=0` restores the pre-2026-07-07 full-final-chunk convention everywhere (A/B lever + kill switch) — that convention is ulp-different in bf16 at step 0 AND in the last prompt token's stored KV, which flips near-tie greedy streams vs mlx-lm (the 2026-07-07 12B completion-probe divergence). |
-| `MLX_BUN_GRAMMAR_JUMP` | — | **off** (`==="1"`) | Jump-forward decoding for structured output (SGLang's technique via xgrammar's `findJumpForwardString`): when the grammar forces a unique continuation (JSON punctuation/keys; longest under `any_whitespace:false` fixed formatting), the SERIAL lane emits its retokenized ids with ONE multi-token forward instead of one masked forward per token. Lossless in STRING space and always grammar-valid, but the token stream (and, conditioning on it, content after a forced span) can legally differ from an unjumped run — no oracle, hence opt-in. Excluded automatically for `logprobs` requests; SentencePiece-family tokenizers whose raw encode can't reproduce a mid-stream span simply never jump (guard degrades to normal decode). The batch lane's per-row grammar (`#stepGrammar`) does not jump yet. Tests: tests/grammar-jump.test.ts (contract + Llama-3.2-1B e2e). |
+| `MLX_BUN_GRAMMAR` | — | on (`!=="0"`) | `=0` disables structured-output mask compilation. Requested constraints still take the documented graceful-degrade route: chat prompt injection plus a `Warning` header; raw completions emit the header only. |
+| `MLX_BUN_GRAMMAR_JUMP` | — | **off** (`==="1"`) | Jump-forward decoding for structured output (SGLang's technique via xgrammar's `findJumpForwardString`): when the grammar forces a unique continuation (JSON punctuation/keys; longest under `any_whitespace:false` fixed formatting), the SERIAL lane emits its retokenized ids with ONE multi-token forward instead of one masked forward per token. Lossless in STRING space and always grammar-valid, but the token stream (and, conditioning on it, content after a forced span) can legally differ from an unjumped run — no oracle, hence opt-in. Excluded automatically when either `logprobs` or `top_logprobs` is requested, because jumped tokens have no sampled distribution row; SentencePiece-family tokenizers whose raw encode can't reproduce a mid-stream span simply never jump (guard degrades to normal decode). The batch lane's per-row grammar (`#stepGrammar`) does not jump yet. Tests: tests/grammar-jump.test.ts (contract + Llama-3.2-1B e2e). |
 | `MLX_BUN_SSD_WRITEBEHIND` | — | on (`!=="0"`) | `--ssd-cache`'s debounced write-behind snapshot (restart survival). `=0` disables it entirely — the paired-A/B lever + kill switch for the 2026-07-07 idle-gating fix (eviction/demotion spills still write). The flush is **idle-gated**: every per-tensor step (blocking GPU sync + `writeSync`) waits for the engine to go idle and pauses when a request arrives mid-flush, so durability work never taxes an active decode (pre-fix: a ~16k entry's flush overlapping cached ctx repeats cost e4b ~9% decode@ctx vs mlx-lm). The snapshot timer also never holds batch admission — while rows are active it re-arms instead of grabbing the exclusive. |
 | `MLX_BUN_SSD_SPILL_QUEUE_GB` | — | 2 | Byte cap on the write-behind queue's PENDING spill clones (they pin the evicted entries' GPU memory until the idle-gated flush runs). Over cap, the oldest queued spill drops — its clones free immediately and the entry becomes a future cache miss (never a wrong result). `=0` keeps only the newest + in-flight clone pinned (the soft cap never drops the item just enqueued). `/stats.ssd_cache` reports `pending_spills` / `pending_spill_bytes` / `dropped_spills`. Note: there is deliberately **no shutdown flush** — exiting under traffic loses queued write-behind entries (best-effort tier; cost = one re-prefill). |
 | `MLX_BUN_BATCH_NO_PIPELINE` | — | off (`==="1"`) | Disables the batch lane's pipelined token readback (the A/B lever for the shipped pipelining, like its peers `MLX_BUN_BATCH_EXTEND` / `MLX_BUN_BATCH_SSM` / `MLX_BUN_BATCH_VEC_SAMPLE`). Numerically equivalent, slower steps. |
@@ -211,7 +214,8 @@ inside it per the table above).
 
 | Option | serial (`--batch 1`) | `--batch N` (N>1) |
 | --- | --- | --- |
-| `--kv-quant config`/`4`/`8` | ✅ applied to all requests | ⚠️ applied, but forces **all** requests to the serial lane (no batching) |
+| `--kv-quant config` | ✅ applied to all requests | ✅ batches where the loaded cache capability supports the per-layer scheme; otherwise routes serial |
+| `--kv-quant 4`/`8` | ✅ applied to all requests | ⚠️ uniform-threshold semantics route requests serial |
 | `--kv-quant turbo[:k<bits>v<bits>]` | ✅ applied to all requests | ⚠️ applied, but forces **all** requests to the serial lane (no batching) |
 | `--kv-quant off` | ✅ bf16 | ✅ bf16 (same as the implicit batch default) |
 | *(kv-quant unset)* | bf16 (the L1 default) | **bf16** (Option B) — incl. serial-lane fallback requests |
@@ -227,6 +231,7 @@ inside it per the table above).
 | `tools` / `stop` | ✅ | ✅ (batches) |
 | structured output (`response_format`/`guided_*`) | ✅ (mask in the decode loop) | ✅ (batches; per-row matchers) |
 | `--draft-model` | ✅ spec decode (grammar composes) | ⚠️ mounts, but routes **every** request serial — spec and batching are different modes |
+| GLM `--mtp on` | ✅ native MTP spec decode | ⚠️ default-on MTP routes every request serial+spec; `--mtp off` exposes ordinary GLM batching |
 | `--compiled-decode` | ✅ (serial decode route) | ✅ at **B=1 only** (Phase 3.2): a lone request's adopted serial-class caches replay the same compiled step, same kill switch; B>1 steps run the plain graph |
 | `--fused-sdpa`/`--force-wire` | ✅ (serial decode route) | **n/a — compat mode, no perf flags by design** |
 
@@ -394,9 +399,9 @@ cells in one run).
   `--expert-offload`. Inherently the serial recipe until batched
   quantized KV lands.
 
-**The two exclusions to remember:** batching ⊕ kv-quant (batch is bf16 by
-contract; the resolver is a Lab-gated follow-up) and spec ⊕ prompt
-cache (v1 bypass; the §7.6 composition is the fix).
+**The two exclusions to remember:** batching excludes uniform/TurboQuant KV
+(per-layer `kv_config.json` does compose with batching), and spec excludes
+prompt-cache reuse (v1 bypass; the §7.6 composition is the fix).
 
 ## Observability — `GET /stats`
 
@@ -405,9 +410,11 @@ lives in [server-api.md](server-api.md#get-stats) (kept in lockstep with
 the code — this section only explains how to read it); highlights:
 `prompt_cache` (8 GB default cap), `kv_quant.mode` (incl. `turbo kXvY`),
 the conditional `ssd_cache` block, and `batch`
-(`active_rows`/`pending_rows`/`submitted_rows`/`kv_bytes`/`kv_budget_bytes`).
-
-`batch.batched` reflects only whether `--batch N` (N>1) is configured;
-with an explicit `--kv-quant` it can read `true` while `active_rows`
-never exceeds 1 (every request routes serial). `active_rows` is the
-honest signal of whether anything is actually batching.
+(`mode`/`active_rows`/`pending_rows`/`submitted_rows`/`kv_bytes`/
+`kv_budget_bytes`).
+
+`batch.mode` is the truthful configured/capability state: `off` for
+`--batch 1`, `serial` when a larger configured cap cannot batch the loaded
+model's cache layout, and `batch` when the model is admitted. `batch.batched`
+is the compatibility boolean for `mode == "batch"`; `active_rows` reports the
+instantaneous live rows.

package/docs/reference/training.md

@@ -177,7 +177,7 @@ are `DEFAULT_TRAIN_CONFIG` (trainer.ts:89).
 | `weight_decay` | float ≥0 | `0.01` | AdamW weight decay (β = `[0.9, 0.999]`, fixed) |
 | `lora_dropout` | float [0,1) | `0.0` | LoRA-input dropout (PEFT-style), training-only regularizer for small sets. Recompute-safe (mask keyed by step+layer, so segmented/checkpointed backward reproduces it) |
 | `rs_lora` | bool | `false` | rsLoRA — scale the update by α/√rank instead of α, so `rank_scaling` changes capacity not step size. Recorded in the adapter config; inference applies the same per-layer scale |
-| `lora_plus_ratio` | float ≥1 | `1.0` | LoRA+ — LR multiplier for the B leaves (A stays at base LR). >1 speeds the B-driven early learning (B is zero-init). 1 = off. Wired in the ORPO loop |
+| `lora_plus_ratio` | float ≥1 | `1.0` | LoRA+ — LR multiplier for the B leaves (A stays at base LR). >1 speeds the B-driven early learning (B is zero-init). 1 = off. The same optimizer policy applies to SFT, DPO, and ORPO |
 | `grad_checkpoint` | bool | `false` | Recompute layer activations in backward (memory↔compute; bit-identical) |
 | `mlp_split` | bool | `false` | Gradient-checkpoint MLP splitting (Gemma4-only, requires `grad_checkpoint`) |
 | `segment_size` | int | `0` (off) | `>0` enables segmented backward — layers per segment (see below) |
@@ -295,6 +295,15 @@ A finished run writes a PEFT-compatible adapter directory:
 - `optiq_lora_config.json` — mlx-bun/optiq adapter metadata (per-layer ranks)
 - `adapter_config.json` — PEFT-compatible config
 
+The save completes only after both config files are written, including
+eval-step checkpoints, so a successful process exit always leaves an
+immediately mountable directory. Serving accepts both mlx-lm tensor layout
+(`[in, rank]` / `[rank, out]`) and standard PEFT layout
+(`[rank, in]` / `[out, rank]`, including `base_model.model.*` names).
+Standard PEFT `use_rslora` metadata is honored as
+`lora_alpha / sqrt(rank)`, and saved rsLoRA adapters emit that metadata so
+their training and serving scales agree.
+
 When `save_checkpoints: true`, each eval step also writes
 `checkpoints/step-<NNNNN>-val<loss>/` and a durable `metrics.json` (config,
 wall seconds, peak GB, final/best train+val loss, full val trajectory).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mlx-bun",
-  "version": "0.0.11",
+  "version": "0.0.12",
   "description": "Native MLX inference for Bun on Apple Silicon — local LLM server (OpenAI/Anthropic-compatible) and TypeScript library. No Python.",
   "type": "module",
   "license": "MIT",

package/src/anthropic.ts

@@ -469,58 +469,68 @@ export function translateOpenAiSse(
   const dec = new TextDecoder();
   let buf = "";
   let finalized = false;
+  let reader: ReadableStreamDefaultReader<Uint8Array> | null = null;
+  let cancelled = false;
   return new ReadableStream<Uint8Array>({
-    async start(controller) {
+    start(controller) {
       const emit = (frames: string[]) => {
         for (const f of frames) controller.enqueue(enc.encode(f));
       };
-      const reader = upstream.getReader();
-      try {
-        while (true) {
-          const { done, value } = await reader.read();
-          if (done) break;
-          buf += dec.decode(value, { stream: true });
-          let sep: number;
-          while ((sep = buf.indexOf("\n\n")) !== -1) {
-            const frame = buf.slice(0, sep);
-            buf = buf.slice(sep + 2);
-            const data = frame
-              .split("\n")
-              .find((l) => l.startsWith("data: "))
-              ?.slice(6);
-            if (data == null) continue;
-            if (data === "[DONE]" || data === '"[DONE]"') {
-              if (!finalized) {
-                finalized = true;
-                emit(translator.finalize());
+      reader = upstream.getReader();
+      void (async () => {
+        try {
+          while (!cancelled) {
+            const { done, value } = await reader!.read();
+            if (done) break;
+            buf += dec.decode(value, { stream: true });
+            let sep: number;
+            while ((sep = buf.indexOf("\n\n")) !== -1) {
+              const frame = buf.slice(0, sep);
+              buf = buf.slice(sep + 2);
+              const data = frame
+                .split("\n")
+                .find((l) => l.startsWith("data: "))
+                ?.slice(6);
+              if (data == null) continue;
+              if (data === "[DONE]" || data === '"[DONE]"') {
+                if (!finalized) {
+                  finalized = true;
+                  emit(translator.finalize());
+                }
+                continue;
               }
-              continue;
-            }
-            const parsed = JSON.parse(data);
-            if (parsed?.error) {
-              emit([
-                sse("error", {
-                  type: "error",
-                  error: { type: "api_error", message: parsed.error.message ?? "generation failed" },
-                }),
-              ]);
-              continue;
+              const parsed = JSON.parse(data);
+              if (parsed?.error) {
+                emit([
+                  sse("error", {
+                    type: "error",
+                    error: { type: "api_error", message: parsed.error.message ?? "generation failed" },
+                  }),
+                ]);
+                continue;
+              }
+              emit(translator.addChunk(parsed));
             }
-            emit(translator.addChunk(parsed));
           }
+          if (!cancelled && !finalized) emit(translator.finalize());
+        } catch (e) {
+          if (!cancelled)
+            emit([
+              sse("error", {
+                type: "error",
+                error: { type: "api_error", message: (e as Error).message },
+              }),
+            ]);
+        } finally {
+          reader?.releaseLock();
+          reader = null;
+          if (!cancelled) controller.close();
         }
-        if (!finalized) emit(translator.finalize());
-      } catch (e) {
-        emit([
-          sse("error", {
-            type: "error",
-            error: { type: "api_error", message: (e as Error).message },
-          }),
-        ]);
-      } finally {
-        reader.releaseLock();
-        controller.close();
-      }
+      })();
+    },
+    async cancel(reason) {
+      cancelled = true;
+      await reader?.cancel(reason);
     },
   });
 }