mlx-bun 0.0.12 → 0.0.13

package/README.md

@@ -225,7 +225,7 @@ On the site: [The lab](https://mlx-bun.dev/about/lab/).
 
 Scope is deliberate: a few model families held to explicit oracle contracts,
 rather than dozens held to none. Currently MiniCPM5, the Gemma-4 OptiQ quants,
-Qwen3.5-4B, and the direct-container GLM-5.2 path:
+the Qwen 3.x family, and the direct-container GLM-5.2 path:
 
 | Model | Download | Fits on | Vision | Notes |
 |---|---|---|---|---|
@@ -234,14 +234,13 @@ Qwen3.5-4B, and the direct-container GLM-5.2 path:
 | [`mlx-community/gemma-4-e4b-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-e4b-it-OptiQ-4bit) | 7.0 GB | 16 GB | ✓ | **Recommended starter** (16 GB+); ~56 tok/s |
 | [`mlx-community/gemma-4-12B-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-12B-it-OptiQ-4bit) | 8.4 GB | 16 GB | ✓ | Vision + tool calling, both verified end-to-end |
 | [`mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit) | 18 GB | 24 GB | — | MoE (top-8 of 128 experts); ~54 tok/s |
+| [`mlx-community/Qwen3.8-27B-OptiQ-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-OptiQ-4bit) | 20.4 GB | 32 GB (24 GB with `--memory-budget`) | text-only for now | Current-gen Qwen; bf16-KV bit-exact parity (vs mlx-lm); thinking with `reasoning_effort` (xhigh/medium/low) + `preserve_thinking` + tool calling; MTP speculation, vision + video in bring-up |
 | [`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`](https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp) | ~384 GB / 357 GiB | 32 GB + fast disk | — | Direct streamed Colibri container; true top-8, native MTP, bounded 25 GiB process plan; warm single-stream performance is disk-bound and below the aspirational 2 tok/s target |
 
 Not sure what fits your machine? `bun src/cli.ts fit <model> --ctx 8192`
-gives a deterministic answer (see below). The larger
-`Qwen3.6-27B-OptiQ-4bit` is still in bring-up — parity and serving polish
-remain (see [PLAN.md](./PLAN.md)). Downloading, cache layout, and reclaiming
-disk: [docs/reference/models.md](./docs/reference/models.md); on the site:
-[Choosing a model](https://mlx-bun.dev/getting-started/models/).
+gives a deterministic answer (see below). Downloading, cache layout, and
+reclaiming disk: [docs/reference/models.md](./docs/reference/models.md);
+on the site: [Choosing a model](https://mlx-bun.dev/getting-started/models/).
 
 ### GLM-5.2: 32 GB quickstart
 
@@ -342,8 +341,10 @@ agent CLIs like pi/OpenClaw via their provider config.
   `temperature`, `top_p`, `top_k`, `max_tokens`, `seed`,
   `repetition_penalty`, `stop` (string or array, matched on decoded
   text with streaming hold-back), `reasoning_effort` (thinking on/off for
-  Qwen3.5 / MiniCPM5 — `none` disables, any level enables), and an `hlg`
-  tone-curve sampling override;
+  Qwen3.5/3.8 / MiniCPM5 — `none` disables, any level enables; on Qwen3.8
+  the level also sets reasoning depth: `low`/`medium`/`xhigh`), a Qwen3.8
+  `chat_template_kwargs.preserve_thinking` toggle (keep think blocks from
+  history; default on), and an `hlg` tone-curve sampling override;
   omitted sampling fields default to the model's own
   `generation_config.json` recipe; usage includes `cached_tokens`.
   Full schemas in
@@ -415,9 +416,12 @@ agent CLIs like pi/OpenClaw via their provider config.
   targets, not small fast models. Serial lane only — with `--batch N` a
   mounted draft routes every request serial, like mlx_lm.server.
 - **Memory admission control** — `--memory-budget <GB>` refuses to load
-  a model that can't serve within the budget and rejects requests whose
-  `prompt + max_tokens` exceed the budget's max safe context with a 400
-  (`type: "memory_admission"`) *before* generating. The GPU
+  a model that can't serve within the budget. Generic budgeted serving rejects
+  requests whose `prompt + max_tokens` exceed the safe context with a 400
+  (`type: "memory_admission"`) *before* generating. GLM-5.2 has a fixed,
+  pre-planned context instead: broad client `max_tokens` values are capped to
+  the remaining context, and only a prompt that leaves no generation slot is
+  rejected. The GPU
   out-of-memory crash it prevents is uncatchable by design — rejection
   up front is the only defense. Ceiling visible at `GET /stats`.
 - **Anthropic Messages API** (`POST /v1/messages`, on by default) —
@@ -571,7 +575,7 @@ Pre-alpha, moving fast. See [PLAN.md](./PLAN.md) for phases, exit
 criteria, measured numbers, and the findings log.
 
 **Complete** — load path; bit-exact model port (MiniCPM5, Qwen3.5-4B,
-e4b per-layer-input, 12B dense, 26B MoE); sampling + serving (tools, vision,
+Qwen3.8-27B, e4b per-layer-input, 12B dense, 26B MoE); sampling + serving (tools, vision,
 prompt cache); registry / fit / KV persistence; quantized +
 mixed-precision KV serving (rotating-cache KV-quant, Phase 9) with fused
 quantized prefill (Phase 10); LoRA hot-swap with per-request selection;
@@ -586,9 +590,11 @@ head-to-head benchmark harness; the decode-gap root-cause fix
 (commit 4625fe5); the embeddable single-binary build (signed +
 notarized — Homebrew, direct-download, and npm/bunx).
 
-**In progress** — `Qwen3.6-27B` bring-up (Phase 14f): same architecture
-as the verified 4B (untied + larger geometry); parity and serving polish
-remain. MTP speculation and Qwen3-VL vision deferred.
+**In progress** — `Qwen3.8-27B` full support (Phase 14 retarget): text
+parity is bit-exact and thinking controls (`reasoning_effort` depths,
+`preserve_thinking`) + tool calling are wired; native MTP speculation
+(the Qwen-trained head as a `--draft-model`), vision, and video input
+are in bring-up.
 
 **Experimental** — opt-in, default-off, still being hardened: transparent
 expert offload for MoE models (`serve --expert-offload`, Phase 20:

package/docs/design/colibri-glm52-port.md

@@ -1425,6 +1425,13 @@ advertised false, not emulated.
     `native-v0.2.0` GitHub release. Both assets are uploaded; anonymous HTTP
     resolution and an empty-cache default-URL download/checksum/extraction
     passed with all five required files. G8 is closed.
+  - [x] Published mlx-bun v0.0.12 itself: the signed/notarized GitHub release
+    and stable asset, npm package, and Homebrew tap are live and independently
+    verified. Both 80,420,638-byte binary assets carry SHA-256
+    `64c4d697faba65789c2af7c1344ee39024f8a03bd6839d2c8df4ec7dce872a74`;
+    npm reports shasum `0dcc5c200fa81dbea1f8be854e21a0efbbdebbfa`.
+    The final full gate passed 1,936 tests / 28,378 assertions with 71 skips
+    and zero failures.
 - Update README plus `docs/reference/{models,memory,cli,server-config,
   server-api,library-api,features-matrix}.md` in the same feature changes that
   add the corresponding flags, fields, routes, defaults, and limitations.

package/docs/planning/release-notes-v0.0.13.md

@@ -0,0 +1,77 @@
+# v0.0.13 release notes
+
+Everything since v0.0.12. One theme: **Qwen 3.8**. The current-generation
+Qwen (released August 2026) is brought up on the published OptiQ
+artifact at the repo's standard bar — bit-exact logit parity with
+mlx-lm — with its thinking controls served faithfully. Vision, video,
+and native Qwen MTP remain separately gated follow-ups.
+
+## Qwen3.8-27B (text, bit-exact)
+
+- `mlx-community/Qwen3.8-27B-OptiQ-4bit` serves on the existing
+  qwen3_5 hybrid graph (gated DeltaNet + gated attention) — the
+  architecture audit found byte-identical geometry to the verified
+  port, and parity confirmed it: **every logit at every prompt
+  position plus 32 greedy decode steps, bit-exact vs stock mlx-lm**
+  (the parity harness grew a full prefill-grid golden and
+  manifest-driven step counts in the process).
+- The config's new `output_gate_type: "swish"` field is inert in every
+  implementation (transformers ground truth included) — a load-time
+  guard now fails loudly if a future checkpoint carries a value we
+  haven't verified.
+- Fit reality: the artifact is 20.35 GB of shards. 32 GB machines
+  serve it comfortably; 24 GB machines need `--memory-budget` and
+  should prefer the uniform 4-bit conversion until the planned
+  ~4 bpw TurboQuant×weights artifact lands (PLAN 14z).
+
+## Thinking controls (Qwen 3.8's serving surface)
+
+- Thinking is on by default and streams as OpenAI `reasoning` deltas;
+  `chat_template_kwargs.enable_thinking: false` selects instruct mode.
+- `reasoning_effort` is now a real depth control on templates that read
+  it: OpenAI levels map onto Qwen's `low`/`medium`/`xhigh` (with
+  `xhigh` accepted as a first-class request value), and the variable is
+  only passed to templates that declare it — the Qwen 3.8 template
+  raises on unknown level names, so the server never forwards unmapped
+  strings.
+- `preserve_thinking` (new in Qwen 3.8, default on) is forwarded:
+  history keeps its think blocks, which also improves prompt-cache
+  reuse in agent loops.
+- Tool calling: Qwen 3.8's XML call format
+  (`<tool_call><function=name><parameter=…>`) parses through the
+  existing Qwen-style parser into OpenAI `tool_calls`, multiline
+  parameters included.
+
+## Qwen native MTP is deliberately not shipped yet
+
+- Qwen 3.8 ships a multi-step MTP head trained alongside the model;
+  mlx-community publishes it split out
+  (`mlx-community/Qwen3.8-27B-MTP-bf16`). Review found that the target's
+  48 recurrent DeltaNet caches cannot roll back rejected verify rows,
+  while the shared speculative loop requires exactly that operation.
+  The initial implementation therefore never reached a draft round.
+- MTP artifacts are detected and refused at startup with the exact reason,
+  before the 20 GB target is opened. Shipping a switch that silently disables
+  itself—or corrupts recurrent state when forced—would be worse than a clear
+  deferral. The follow-up needs recurrent-state snapshot/restore or a verify
+  schedule that never advances state beyond the accepted prefix.
+
+## GLM-5.2 client compatibility
+
+- Fixed-context GLM serving now treats request `max_tokens` as an upper bound:
+  when a client sends a family-wide value larger than the planned remaining
+  context, the server caps it instead of rejecting an otherwise valid first
+  prompt. The reported 2,788-prompt + 8,192-cap failure now admits a 1,308-token
+  completion ceiling within the 4,096-token plan. Prompts that consume the
+  whole context remain rejected before generation.
+
+## Also in this release
+
+- The parity harness's step-0 check now covers the full prefill logit
+  grid for all Qwen targets — masking/position bugs can no longer hide
+  behind a last-position comparison.
+- A GPU command-buffer failure mode on memory-saturated machines was
+  root-caused (Metal completion-thread exception, uncatchable by
+  design) and documented in PLAN 14g, along with the finding that some
+  mlx-c paths surface C++ errors as process panics instead of
+  catchable errors (hardening item).

package/docs/reference/features-matrix.md

@@ -69,6 +69,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | MiniCPM5 (cpm5) | ✅ L1/L2 | ✅ | the starter model |
 | Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
 | Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
+| Qwen3.8-27B (same qwen3_5 graph) | ✅ L1 (no L2 oracle) | ✅ (SSM path) | thinking depths + preserve_thinking; MTP/vision/video in bring-up |
 | GLM-5.2 / Colibri (pre-release port gate) | ✅ chat/text, Messages, Responses, SSE, tools, grammar, logprobs, library/CLI | ✅ compressed MLA/DSA scheduler | native MTP defaults to serial+spec; `--mtp off` enables ordinary batching; artifact-aware CLI/API/status `fit` and live main/MTP expert-tier telemetry; embeddings, vision/audio, adapters, and training are explicitly unsupported; per-row batched MTP is post-release |
 | DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
 | Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family/sliding universal → serial |

package/docs/reference/server-api.md

@@ -50,14 +50,25 @@ Request body (OpenAI chat schema; unknown fields ignored):
   "tools": [ /* OpenAI function tools */ ],
   "tool_choice": "auto",         // "none" disables tools
   "chat_template_kwargs": {      // forwarded to the chat template
-    "enable_thinking": false     // MiniCPM5 / Qwen3.5 / GLM-5.2: think channel
+    "enable_thinking": false,    // MiniCPM5 / Qwen3.5/3.8 / GLM-5.2: <think>
+                                 // channel on/off (Qwen3.8 default: ON)
+    "preserve_thinking": true    // Qwen3.8: keep think blocks from earlier
+                                 // assistant turns in the rendered prompt
+                                 // (template default true — better prompt-
+                                 // cache reuse in agent loops). Ignored by
+                                 // templates that don't read it.
   },
-  "reasoning_effort": "medium",  // "none"|"minimal"|"low"|"medium"|"high"
-                                 // gates enable_thinking on Qwen3.5/MiniCPM5:
-                                 // "none" → off, any other level → on. Only
-                                 // consulted when chat_template_kwargs
-                                 // .enable_thinking is not explicitly set —
-                                 // an explicit enable_thinking always wins.
+  "reasoning_effort": "medium",  // "none"|"minimal"|"low"|"medium"|"high"|"xhigh"
+                                 // gates enable_thinking on Qwen3.5/3.8/
+                                 // MiniCPM5: "none" → off, any other level →
+                                 // on. Only consulted when chat_template_
+                                 // kwargs.enable_thinking is not explicitly
+                                 // set — an explicit enable_thinking always
+                                 // wins. On templates with a reasoning-depth
+                                 // variable (Qwen3.8) the level also maps
+                                 // into the template: minimal/low → low,
+                                 // medium → medium, high/xhigh → xhigh
+                                 // (Qwen3.8's default depth is xhigh).
   "hlg": {                       // HLG tone-curve sampling (per request).
     "enabled": true,             // merged over --hlg-sampling server defaults.
     "width": 4,                  // logit-width of the tone plateau
@@ -333,11 +344,12 @@ All errors are `{ "error": { "message": …, ... } }`.
   `top_logprobs must be of type int` / `at least 0` / `at most 11`;
   `top_logprobs: -1` is the accepted "unset" sentinel).
 - `400` with `"type": "memory_admission"`, `"code":
-  "context_over_budget"` — `prompt + max_tokens` exceeds the memory
-  budget's max safe context (only when serving with `--memory-budget`;
-  the GPU OOM this prevents would kill the process, so it is refused
-  up front). Lower `max_tokens` or shorten the prompt; the ceiling is
-  visible at `/stats`.
+  "context_over_budget"` — the request cannot fit the memory budget's max
+  safe context. Generic budgeted serving rejects when `prompt + max_tokens`
+  exceeds the ceiling. GLM-5.2 instead caps a broad `max_tokens` upper bound
+  to the space remaining in its fixed planned context and returns this error
+  only when the prompt itself leaves no generation slot. The ceiling is visible
+  at `/stats`.
 
 ## POST /v1/completions (raw text completion)
 

package/docs/reference/server-config.md

@@ -21,7 +21,7 @@ the design rationale is in
 | --- | --- | --- | --- |
 | `--host` | addr | `127.0.0.1` | Interface to bind. Loopback-only by default (mlx_lm.server parity); pass `--host 0.0.0.0` to expose the server on your network. |
 | `--port` | n | `8080` | Listen port. A pre-flight probe refuses to start if the port is already serving. |
-| `--memory-budget` | GB | machine RAM × 0.75; GLM-5.2: min(25 GiB, physical RAM) | Admission ceiling. Requests whose `prompt + max_tokens` exceed the budget's max safe context are **rejected with 400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. GLM-5.2 runs its header-only exact process equation before opening resident weights or expert slabs, and caps MLX to that plan's allocator reserve. **Decimal GB (×10⁹).** |
+| `--memory-budget` | GB | machine RAM × 0.75; GLM-5.2: min(25 GiB, physical RAM) | Admission ceiling. Generic budgeted serving rejects requests whose `prompt + max_tokens` exceed the safe context with **400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. GLM-5.2 runs its exact process equation before opening resident weights or expert slabs and caps MLX to the plan's allocator reserve; because its context is fixed up front, a broad client `max_tokens` upper bound is capped to the remaining planned context, while a prompt that leaves no generation slot is still rejected. **Decimal GB (×10⁹).** |
 | `--prompt-cache` | GB | `8` | RAM prefix-KV cache (byte-capped LRU; `0` disables). **PREFIX SHARING (2026-07-05): serves are NON-CONSUMING** — a hit hands out zero-copy clones and leaves the donor entry intact, so N agents (or a brand-new session) sharing a system prompt all reuse ONE prefill without destroying each other's conversation entries (the old consume-and-trim take cannibalized the donor). Extended entries supersede their prefix-ancestors (when trimmable), so a conversation stays one entry. Works on BOTH lanes and tiers over the SSD store (Layer 0). |
 | `--isolate` | (bool) | off | Runtime isolation ([runtime-isolation.md](../design/runtime-isolation.md)): the inference engine runs as a CHILD process on a unix socket; this process stays a pure UI/API reverse proxy — instant under any GPU load, survives engine crashes (auto-respawn, in-flight requests get a clean 502). Measured cost ≈0 (−0.4% tok/s, +2 ms TTFT, per-token SSE granularity preserved). `/ws/chat` is not proxied yet (501). `/engine` reports child pid/restarts. |
 | `--model-pool` | n | `1` | With `--isolate`: max RESIDENT model engines. Requests naming another model (**exact `/v1/models` id** — fuzzy strings keep mlx-lm's ignored-field semantics) spawn/route to that model's own engine child (SPAWN-OVERLAP: the new model loads while the old one keeps serving; nobody's stream is interrupted). Over the cap, the LRU engine drains, **demotes its prompt cache to the SSD tier**, and exits — switching back respawns it with state restored from disk. Measured (M1 Max, cpm5⇄qwen0.8b, pool 1): switch 1.5 s, switch-back 1.2 s with `cached_tokens` 103/104 — the conversation survived the eviction. The pool drives this eviction via `POST /admin/drain` — unix-socket-only (never exposed on the public TCP listener) — which quiesces the gateway and demotes the whole prompt cache to the SSD tier before eviction, responding `{ drained: true, demotions: N }`. |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mlx-bun",
-  "version": "0.0.12",
+  "version": "0.0.13",
   "description": "Native MLX inference for Bun on Apple Silicon — local LLM server (OpenAI/Anthropic-compatible) and TypeScript library. No Python.",
   "type": "module",
   "license": "MIT",

package/src/chat-template.ts

@@ -40,6 +40,14 @@ export interface RenderOptions {
   addGenerationPrompt?: boolean;
   tools?: ToolDefinition[] | null;
   enableThinking?: boolean;
+  /** Template-level reasoning depth (Qwen3.8: xhigh|medium|low — the template
+   *  RAISES on other values, so callers must map OpenAI levels first and only
+   *  set this for templates that read it; see ChatTemplate.readsReasoningEffort). */
+  reasoningEffort?: "xhigh" | "medium" | "low";
+  /** Keep think blocks from historical assistant turns in the rendered prompt
+   *  (Qwen3.8 `preserve_thinking`, template default true — better prompt-cache
+   *  reuse and agent-trace continuity). Only passed when the template reads it. */
+  preserveThinking?: boolean;
 }
 
 type ChatRenderer = (
@@ -240,6 +248,14 @@ export class ChatTemplate {
   readonly #renderer: ChatRenderer | null;
   readonly #bosToken: string | null;
   readonly #eosToken: string | null;
+  /** True when the template reads `reasoning_effort` (Qwen3.8's xhigh/medium/
+   *  low depth control). Callers must not pass the variable otherwise — and
+   *  must never pass unmapped OpenAI level names; this template family raises
+   *  on values outside its supported set. */
+  readonly readsReasoningEffort: boolean;
+  /** True when the template reads `preserve_thinking` (Qwen3.8: keep think
+   *  blocks from historical turns; template default true). */
+  readonly readsPreserveThinking: boolean;
   /** Which reasoning format the template's `enable_thinking` channel uses, or
    *  null if the model has no switchable reasoning:
    *   - "think-tag": `<think>…</think>` text markers (Qwen3.5, MiniCPM5) — split
@@ -267,6 +283,8 @@ export class ChatTemplate {
     this.#bosToken = bosToken;
     this.#eosToken = eosToken;
     const gatesThinking = source?.includes("enable_thinking") ?? false;
+    this.readsReasoningEffort = source?.includes("reasoning_effort") ?? false;
+    this.readsPreserveThinking = source?.includes("preserve_thinking") ?? false;
     // `forceNoThinking` suppresses the switchable channel even when the template
     // carries the gemma markers. DiffusionGemma ships the shared gemma-family
     // template (with the `<|channel>thought…<channel|>` reasoning channel), but
@@ -334,7 +352,13 @@ export class ChatTemplate {
 
   render(messages: ChatMessage[], options: RenderOptions = {}): string {
     if (this.#renderer) return this.#renderer(messages, options);
-    const { addGenerationPrompt = true, tools = null, enableThinking } = options;
+    const {
+      addGenerationPrompt = true,
+      tools = null,
+      enableThinking,
+      reasoningEffort,
+      preserveThinking,
+    } = options;
     return this.#template!.render({
       messages,
       add_generation_prompt: addGenerationPrompt,
@@ -342,6 +366,15 @@ export class ChatTemplate {
       // `value['type'] | upper` (Gemma) never see UndefinedValue.
       tools: normalizeToolSchemas(tools),
       ...(enableThinking !== undefined ? { enable_thinking: enableThinking } : {}),
+      // Only set for templates that read them: reasoning_effort values are
+      // template-validated (Qwen3.8 raises on unknown levels), and an unread
+      // variable in other templates is dead context at best.
+      ...(reasoningEffort !== undefined && this.readsReasoningEffort
+        ? { reasoning_effort: reasoningEffort }
+        : {}),
+      ...(preserveThinking !== undefined && this.readsPreserveThinking
+        ? { preserve_thinking: preserveThinking }
+        : {}),
       bos_token: this.#bosToken,
       eos_token: this.#eosToken,
     });

package/src/config.ts

@@ -253,6 +253,16 @@ export async function loadModelConfig(modelDir: string): Promise<ModelConfig> {
   // partial_rotary_factor}), not the gemma per-attention-type map — and
   // type "default" means plain partial nn.RoPE (mrope_section ignored for text).
   const qwenRope = (t.rope_parameters ?? {}) as Record<string, any>;
+  // Qwen3.8 adds output_gate_type ("swish"), which every implementation
+  // (transformers ground truth, mlx-lm pinned + main) currently ignores —
+  // the attention output gate is hardcoded o_proj(out·σ(gate)). Only accept
+  // values verified to mean that; an unknown value must fail at load, not
+  // silently mis-gate a future checkpoint where the field starts mattering.
+  if (isQwen35 && t.output_gate_type != null && t.output_gate_type !== "swish")
+    throw new Error(
+      `qwen3_5: unverified output_gate_type "${t.output_gate_type}" (only "swish" ` +
+        `is verified to mean the standard sigmoid output gate)`,
+    );
 
   const text: TextConfig = {
     hiddenSize: glm52?.hiddenSize ?? t.hidden_size,

package/src/server.ts

@@ -293,7 +293,7 @@ export interface GenSamplingDefaults {
   repetitionPenalty?: number;
 }
 
-export type DraftKind = "dspark" | "deepspec" | "assistant" | "two-model" | "ngram";
+export type DraftKind = "dspark" | "deepspec" | "assistant" | "two-model" | "ngram" | "mtp";
 
 /** Detect the draft artifact's kind so the right provider is loaded. All
  *  providers share ONE serve loop (src/spec/serve-loop.ts). Exported for the
@@ -311,12 +311,22 @@ export async function detectDraftKind(dir: string): Promise<DraftKind> {
     // dspark.json, plain HF config stamped Gemma4DSparkModel.
     if (cfg.architectures?.[0] === "Gemma4DSparkModel") return "deepspec";
     if (String(cfg.model_type ?? "").includes("assistant")) return "assistant";
+    // Detect Qwen's companion head so loadContext can refuse it precisely.
+    // The target's 48 recurrent DeltaNet caches cannot yet roll back rejected
+    // verification rows, so treating this as a generic two-model draft would
+    // fail later and obscure the actual unsupported composition.
+    if (String(cfg.model_type ?? "").endsWith("_mtp")) return "mtp";
   } catch {
     // no/unreadable config → fall through to a full second model
   }
   return "two-model";
 }
 
+export const QWEN_MTP_UNSUPPORTED_REASON =
+  "Qwen native MTP is not supported yet: its target has recurrent DeltaNet " +
+  "caches that cannot roll back rejected verification tokens. Serve Qwen3.8 " +
+  "without --draft-model; MTP remains a gated follow-up.";
+
 async function loadGenSamplingDefaults(modelDir: string): Promise<GenSamplingDefaults> {
   const file = Bun.file(`${modelDir}/generation_config.json`);
   if (!(await file.exists())) return {};
@@ -341,6 +351,15 @@ export async function loadContext(
   const config = await loadModelConfig(modelDir);
   const glm = isGlm52Config(config);
   const externalDraft = opts.draftModelDir !== undefined || opts.draftKind !== undefined;
+  // Refuse the known-impossible composition before opening the 20 GB target
+  // or the companion head. The shared speculative loop requires target-cache
+  // rollback after a partial reject; Qwen's DeltaNet state is recurrent and
+  // deliberately non-trimmable. Silently degrading would make --draft-model
+  // a lie, while forcing it would corrupt the next target step.
+  const resolvedDraftKind = opts.draftModelDir
+    ? opts.draftKind ?? await detectDraftKind(opts.draftModelDir)
+    : opts.draftKind;
+  if (resolvedDraftKind === "mtp") throw new Error(QWEN_MTP_UNSUPPORTED_REASON);
   if (glm && externalDraft && opts.glm?.enableMtp === true)
     throw new Error("native GLM MTP and --draft-model/--draft-kind are mutually exclusive");
   let weights: Weights | null = null;
@@ -412,7 +431,7 @@ export async function loadContext(
     };
   } else if (opts.draftModelDir) {
     const dir = opts.draftModelDir;
-    const kind = opts.draftKind ?? (await detectDraftKind(dir));
+    const kind = resolvedDraftKind!;
     let provider: import("./spec/source").DraftProvider;
     let numDraftTokens = Math.max(1, opts.numDraftTokens ?? 3);
     if (kind === "dspark") {
@@ -663,15 +682,21 @@ interface ChatRequest {
   tools?: ToolDefinition[];
   tool_choice?: "auto" | "none" | { type: string; function?: { name: string } };
   /** Forwarded to HF chat templates, matching optiq serve. MiniCPM5 uses
-   *  enable_thinking to select direct answers vs the <think> channel. */
+   *  enable_thinking to select direct answers vs the <think> channel; Qwen3.8
+   *  adds preserve_thinking (keep think blocks from history; template default
+   *  true — better prompt-cache reuse in agent loops). */
   chat_template_kwargs?: {
     enable_thinking?: boolean;
+    preserve_thinking?: boolean;
     [key: string]: unknown;
   };
   /** OpenAI reasoning control. For models with a switchable <think> channel
    *  (Qwen3.5/MiniCPM5) it gates enable_thinking: "none" → off, any level → on.
-   *  This is what Pi sends when the provider advertises reasoning. */
-  reasoning_effort?: "none" | "minimal" | "low" | "medium" | "high";
+   *  This is what Pi sends when the provider advertises reasoning. For
+   *  templates that read a reasoning-depth variable (Qwen3.8: xhigh/medium/
+   *  low), the level ALSO maps into the template via qwenReasoningEffort —
+   *  "xhigh" accepted as a first-class value (Qwen3.8's default/deepest). */
+  reasoning_effort?: "none" | "minimal" | "low" | "medium" | "high" | "xhigh";
   /** Mounted LoRA adapter selection: "id", "a+b" (stacked), or "none". */
   adapter?: string;
   /** HLG tone-curve sampling override (per request). Snake_case wire fields,
@@ -708,6 +733,36 @@ type TextCompletionRequest = Omit<ChatRequest, "messages" | "tools" | "tool_choi
   prompt?: unknown;
 };
 
+export interface ContextAdmissionDecision {
+  /** Effective completion ceiling after admission. */
+  maxTokens: number;
+  /** True when a fixed-context runtime reduced the client's upper bound. */
+  clamped: boolean;
+}
+
+/** Resolve a request's completion upper bound against the admitted context.
+ *
+ * Generic memory-budget serving preserves the established fail-fast contract:
+ * an oversized reservation is rejected. A fixed-context runtime such as GLM
+ * may instead clamp the client's upper bound to the remaining planned context.
+ * `max_tokens` is a ceiling, not a promise that every token will be emitted;
+ * clamping keeps clients that send a broad family-wide default usable without
+ * weakening the prompt/context OOM guard. A prompt that leaves no generation
+ * slot is always rejected. */
+export function admitRequestContext(
+  promptTokens: number,
+  requestedMaxTokens: number,
+  maxSafeContext: number,
+  clampToFit: boolean,
+): ContextAdmissionDecision | null {
+  const available = maxSafeContext - promptTokens;
+  if (available < 1) return null;
+  if (requestedMaxTokens <= available)
+    return { maxTokens: requestedMaxTokens, clamped: false };
+  if (!clampToFit) return null;
+  return { maxTokens: available, clamped: true };
+}
+
 /** Per-field default HLG knobs when enabling without specifying them. */
 const HLG_DEFAULTS = { width: 4, shoulder: 4, toe: 6, pivotOffset: 6 } as const;
 
@@ -2162,10 +2217,36 @@ export function createServer(
     return { controller: r.controller, degradeHint: r.degradeHint };
   };
 
+  // Map OpenAI reasoning_effort levels onto the Qwen3.8 template's supported
+  // set (xhigh|medium|low — the template raises on anything else). "none"
+  // means thinking off (handled by resolveEnableThinking), so no depth is
+  // passed. Only consumed for templates with readsReasoningEffort.
+  const qwenReasoningEffort = (
+    effort: ChatRequest["reasoning_effort"],
+  ): "xhigh" | "medium" | "low" | undefined => {
+    switch (effort) {
+      case "minimal":
+      case "low":
+        return "low";
+      case "medium":
+        return "medium";
+      case "high":
+      case "xhigh":
+        return "xhigh";
+      default:
+        return undefined; // "none" / unset → template default (xhigh)
+    }
+  };
+
   const templateOptionsFor = (req: ChatRequest, tools: ToolDefinition[] | null) => {
     // enableThinking resolution (and its precedence) lives in
     // resolveEnableThinking so template rendering and sampling stay in sync.
-    return { tools, enableThinking: resolveEnableThinking(req) };
+    return {
+      tools,
+      enableThinking: resolveEnableThinking(req),
+      reasoningEffort: qwenReasoningEffort(req.reasoning_effort),
+      preserveThinking: req.chat_template_kwargs?.preserve_thinking,
+    };
   };
 
   const promptIdsFor = (
@@ -2200,7 +2281,7 @@ export function createServer(
     trimmed: number[],
   ): number => {
     const opts = templateOptionsFor(req, tools);
-    const mode = `${opts.enableThinking}|${!!tools?.length}`;
+    const mode = `${opts.enableThinking}|${!!tools?.length}|${opts.reasoningEffort ?? ""}|${opts.preserveThinking ?? ""}`;
     const primer = primerLenByMode.get(mode);
     if (primer !== undefined) return Math.max(0, trimmed.length - primer);
     let stableLen = trimmed.length;
@@ -3277,15 +3358,22 @@ export function createServer(
         // Admission: reject what cannot finish within the memory budget
         // (the GPU OOM it would otherwise hit is uncatchable and kills
         // the process — Phase 6 finding).
-        const requiredCtx = promptIds.length + (options.maxTokens ?? 1024);
-        if (requiredCtx > admission.maxSafeContext) {
+        const requestedMaxTokens = options.maxTokens ?? 1024;
+        const requiredCtx = promptIds.length + requestedMaxTokens;
+        const contextDecision = admitRequestContext(
+          promptIds.length,
+          requestedMaxTokens,
+          admission.maxSafeContext,
+          ctx.glmMemoryPlan !== null,
+        );
+        if (!contextDecision) {
           disposeRejected();
           return Response.json(
             {
               error: {
                 message:
                   `request needs ${requiredCtx} tokens of context ` +
-                  `(prompt ${promptIds.length} + max_tokens ${options.maxTokens}) but the ` +
+                  `(prompt ${promptIds.length} + max_tokens ${requestedMaxTokens}) but the ` +
                   `memory budget caps safe context at ${admission.maxSafeContext} — ` +
                   `shorten the prompt or lower max_tokens`,
                 type: "memory_admission",
@@ -3295,6 +3383,7 @@ export function createServer(
             { status: 400 },
           );
         }
+        options.maxTokens = contextDecision.maxTokens;
         try {
           // A request's explicit `adapter` (incl. "none") wins over the
           // startup default from `serve --adapter <dir>`.
@@ -3632,8 +3721,15 @@ export function createServer(
         // completion: with no template an EOS may never come.
         options.maxTokens = body.max_completion_tokens ?? body.max_tokens ??
           defaultGeneratedTokens ?? 512;
-        const requiredCtx = promptIds.length + options.maxTokens;
-        if (requiredCtx > admission.maxSafeContext) {
+        const requestedMaxTokens = options.maxTokens;
+        const requiredCtx = promptIds.length + requestedMaxTokens;
+        const contextDecision = admitRequestContext(
+          promptIds.length,
+          requestedMaxTokens,
+          admission.maxSafeContext,
+          ctx.glmMemoryPlan !== null,
+        );
+        if (!contextDecision) {
           // Pre-run reject: the gateway only takes grammar disposal ownership
           // once run — free the WASM matcher here (chat lane: disposeRejected).
           textGrammarCtrl?.dispose();
@@ -3652,6 +3748,7 @@ export function createServer(
             { status: 400 },
           );
         }
+        options.maxTokens = contextDecision.maxTokens;
         try {
           const adapterIds = ctx.adapters.resolveSpec(body.adapter ?? serverOptions.defaultAdapter);
           if (adapterIds.length) options.adapters = adapterIds;