mlx-bun 0.0.11 → 0.0.13

package/README.md

@@ -90,8 +90,9 @@ nothing to install:
 bunx mlx-bun
 ```
 
-First run fetches the MLX native runtime (~52 MB) and the model into your
-caches, then serves. (Bun only — `npx mlx-bun` under Node exits with a
+First run fetches the MLX native runtime plus the streamed-expert helper
+(~52 MB) and the model into your caches, then serves. (Bun only — `npx
+mlx-bun` under Node exits with a
 "requires Bun" notice by design.)
 
 ### From source
@@ -116,7 +117,8 @@ mlx-bun
 Bare `mlx-bun` is an alias for `mlx-bun serve`. On its first run, with no
 model named, it does everything for you:
 
-1. pulls the MLX native runtime (~52 MB) into `~/Library/Caches/mlx-bun/`
+1. pulls the MLX native runtime plus streamed-expert helper (~52 MB) into
+   `~/Library/Caches/mlx-bun/`
    (skipped on the Homebrew install — the runtime ships in the bottle);
 2. downloads the sub-GB `MiniCPM5-1B` starter and serves it, so you're
    chatting in well under a minute;
@@ -221,9 +223,9 @@ On the site: [The lab](https://mlx-bun.dev/about/lab/).
 
 ## Supported models
 
-Scope is deliberate: a few model families held to **bit-exact** logit
-parity with the Python reference, rather than dozens held to none.
-Currently MiniCPM5, the Gemma-4 OptiQ quants, and Qwen3.5-4B:
+Scope is deliberate: a few model families held to explicit oracle contracts,
+rather than dozens held to none. Currently MiniCPM5, the Gemma-4 OptiQ quants,
+the Qwen 3.x family, and the direct-container GLM-5.2 path:
 
 | Model | Download | Fits on | Vision | Notes |
 |---|---|---|---|---|
@@ -232,13 +234,33 @@ Currently MiniCPM5, the Gemma-4 OptiQ quants, and Qwen3.5-4B:
 | [`mlx-community/gemma-4-e4b-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-e4b-it-OptiQ-4bit) | 7.0 GB | 16 GB | ✓ | **Recommended starter** (16 GB+); ~56 tok/s |
 | [`mlx-community/gemma-4-12B-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-12B-it-OptiQ-4bit) | 8.4 GB | 16 GB | ✓ | Vision + tool calling, both verified end-to-end |
 | [`mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit) | 18 GB | 24 GB | — | MoE (top-8 of 128 experts); ~54 tok/s |
+| [`mlx-community/Qwen3.8-27B-OptiQ-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-OptiQ-4bit) | 20.4 GB | 32 GB (24 GB with `--memory-budget`) | text-only for now | Current-gen Qwen; bf16-KV bit-exact parity (vs mlx-lm); thinking with `reasoning_effort` (xhigh/medium/low) + `preserve_thinking` + tool calling; MTP speculation, vision + video in bring-up |
+| [`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`](https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp) | ~384 GB / 357 GiB | 32 GB + fast disk | — | Direct streamed Colibri container; true top-8, native MTP, bounded 25 GiB process plan; warm single-stream performance is disk-bound and below the aspirational 2 tok/s target |
 
 Not sure what fits your machine? `bun src/cli.ts fit <model> --ctx 8192`
-gives a deterministic answer (see below). The larger
-`Qwen3.6-27B-OptiQ-4bit` is still in bring-up — parity and serving polish
-remain (see [PLAN.md](./PLAN.md)). Downloading, cache layout, and reclaiming
-disk: [docs/reference/models.md](./docs/reference/models.md); on the site:
-[Choosing a model](https://mlx-bun.dev/getting-started/models/).
+gives a deterministic answer (see below). Downloading, cache layout, and
+reclaiming disk: [docs/reference/models.md](./docs/reference/models.md);
+on the site: [Choosing a model](https://mlx-bun.dev/getting-started/models/).
+
+### GLM-5.2: 32 GB quickstart
+
+Choose a volume with at least the artifact remainder plus 1 GiB free before
+starting; mlx-bun checks this before the first payload and credits resumable
+partials. The validated artifact revision and exact 25 GiB serving command are
+documented in the [model-management guide](./docs/reference/models.md#glm-52-on-a-32-gb-mac):
+
+```sh
+export HF_HUB_CACHE=/Volumes/FastSSD/huggingface/hub  # optional large fast volume
+mlx-bun get mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
+  --revision 3cc8db99b1b13fc79325d987ba3c1c430766b3b8
+mlx-bun scan
+mlx-bun serve GLM-5.2-colibri --memory-budget 26.8435456 \
+  --context-length 4096 --max-tokens 128 --mtp on
+```
+
+Rerun `get` after interruption; partial files resume and every completed blob
+is checksum-verified. The model remains in the standard Hugging Face cache—no
+Python environment and no second converted copy inside this repository.
 
 ## Why
 
@@ -278,6 +300,7 @@ bun src/cli.ts ls                   # list models (size, params, quant, capabili
 bun src/cli.ts ls --vision --max-size 10GB
 bun src/cli.ts fit gemma --ctx 32768          # memory contract: fits? max context? predicted tok/s
 bun src/cli.ts fit gemma --ctx 8192 --skus    # ...same, across the Apple Silicon lineup
+bun src/cli.ts fit GLM-5.2                    # streamed plan: disk vs resident slabs + measured speed
 bun src/cli.ts serve gemma --port 8080        # OpenAI-compatible server
 bun src/cli.ts serve gemma --memory-budget 18 # ...with admission control (GB)
 bun src/cli.ts generate gemma "a haiku about metal shaders"   # raw one-shot generation
@@ -299,7 +322,10 @@ listing the candidates — just make it more specific.
 headers, KV bytes/token from the config (sliding-window layers capped,
 MoE active-expert bytes for decode), calibrated prefill transient, and
 the machine's wired-memory ceiling. Predictions are recorded next to
-measured peaks in the eval DB.
+measured peaks in the eval DB. The direct Colibri GLM-5.2 path uses its exact
+streamed plan instead: full artifact-on-disk size is separated from bounded
+resident weights, main/MTP expert slabs, KV, and reserves, and measured warm
+decode is separated from the direct oracle and the 2 tok/s aspiration.
 
 Every verb, with flags: [docs/reference/cli.md](./docs/reference/cli.md);
 on the site: [CLI reference](https://mlx-bun.dev/reference/cli/).
@@ -315,8 +341,10 @@ agent CLIs like pi/OpenClaw via their provider config.
   `temperature`, `top_p`, `top_k`, `max_tokens`, `seed`,
   `repetition_penalty`, `stop` (string or array, matched on decoded
   text with streaming hold-back), `reasoning_effort` (thinking on/off for
-  Qwen3.5 / MiniCPM5 — `none` disables, any level enables), and an `hlg`
-  tone-curve sampling override;
+  Qwen3.5/3.8 / MiniCPM5 — `none` disables, any level enables; on Qwen3.8
+  the level also sets reasoning depth: `low`/`medium`/`xhigh`), a Qwen3.8
+  `chat_template_kwargs.preserve_thinking` toggle (keep think blocks from
+  history; default on), and an `hlg` tone-curve sampling override;
   omitted sampling fields default to the model's own
   `generation_config.json` recipe; usage includes `cached_tokens`.
   Full schemas in
@@ -328,13 +356,17 @@ agent CLIs like pi/OpenClaw via their provider config.
   (Gemma 4 `<|tool_call>` sentinel tokens; MiniCPM5 / Qwen3.5
   `<function name=…>` XML with schema-aware argument decoding);
   `role: "tool"` round-trips, including multi-turn agent loops.
+  The built-in `web_fetch` tool accepts public HTTP(S) destinations only,
+  revalidates DNS and every redirect, and streams into a 1 MiB response cap.
 - **Structured output** — `response_format` (JSON mode / JSON schema),
   plus the vLLM/oMLX aliases `guided_grammar`, `guided_regex`,
   `guided_choice`, and `structured_outputs`, enforced by
   grammar-constrained decoding (xgrammar) on chat and raw completions,
   serial and batched lanes alike; a grammar that fails to compile
-  degrades to prompt injection (oMLX parity) instead of an error
-  (`MLX_BUN_GRAMMAR=0` disables grammar-constrained decoding server-wide).
+  degrades to a string-valued constraint-specific prompt injection
+  (oMLX parity) instead of an error
+  (`MLX_BUN_GRAMMAR=0` disables mask compilation server-wide and routes
+  requested constraints through that same warning-bearing degrade path).
 - **Vision** — `image_url` content parts (data: URLs or http/s), on
   models with the vision sidecar. PNG, JPEG, HEIC, AVIF, WebP, TIFF,
   GIF, BMP via native OS codecs. Remote fetches are SSRF-guarded by
@@ -384,9 +416,12 @@ agent CLIs like pi/OpenClaw via their provider config.
   targets, not small fast models. Serial lane only — with `--batch N` a
   mounted draft routes every request serial, like mlx_lm.server.
 - **Memory admission control** — `--memory-budget <GB>` refuses to load
-  a model that can't serve within the budget and rejects requests whose
-  `prompt + max_tokens` exceed the budget's max safe context with a 400
-  (`type: "memory_admission"`) *before* generating. The GPU
+  a model that can't serve within the budget. Generic budgeted serving rejects
+  requests whose `prompt + max_tokens` exceed the safe context with a 400
+  (`type: "memory_admission"`) *before* generating. GLM-5.2 has a fixed,
+  pre-planned context instead: broad client `max_tokens` values are capped to
+  the remaining context, and only a prompt that leaves no generation slot is
+  rejected. The GPU
   out-of-memory crash it prevents is uncatchable by design — rejection
   up front is the only defense. Ceiling visible at `GET /stats`.
 - **Anthropic Messages API** (`POST /v1/messages`, on by default) —
@@ -424,13 +459,12 @@ in [docs/reference/embedding.md](./docs/reference/embedding.md).
 
 ```ts
 import {
-  createModel,        // dispatches to Gemma4 / MiniCPM5 / Qwen3.5
-  Weights, loadModelConfig, loadTokenizer, ChatTemplate, generate,
+  openModel,          // artifact-aware, incl. bounded streamed GLM-5.2
+  loadTokenizer, ChatTemplate, generate,
 } from "mlx-bun";     // or "./src/index" in a clone
 
 const dir = "/path/to/hf/snapshot";
-const config = await loadModelConfig(dir);
-const model = createModel(await Weights.open(dir), config);  // RuntimeModel
+const model = await openModel(dir);  // RuntimeModel
 const tok = await loadTokenizer(dir);
 const template = await ChatTemplate.load(dir);
 
@@ -541,7 +575,7 @@ Pre-alpha, moving fast. See [PLAN.md](./PLAN.md) for phases, exit
 criteria, measured numbers, and the findings log.
 
 **Complete** — load path; bit-exact model port (MiniCPM5, Qwen3.5-4B,
-e4b per-layer-input, 12B dense, 26B MoE); sampling + serving (tools, vision,
+Qwen3.8-27B, e4b per-layer-input, 12B dense, 26B MoE); sampling + serving (tools, vision,
 prompt cache); registry / fit / KV persistence; quantized +
 mixed-precision KV serving (rotating-cache KV-quant, Phase 9) with fused
 quantized prefill (Phase 10); LoRA hot-swap with per-request selection;
@@ -556,9 +590,11 @@ head-to-head benchmark harness; the decode-gap root-cause fix
 (commit 4625fe5); the embeddable single-binary build (signed +
 notarized — Homebrew, direct-download, and npm/bunx).
 
-**In progress** — `Qwen3.6-27B` bring-up (Phase 14f): same architecture
-as the verified 4B (untied + larger geometry); parity and serving polish
-remain. MTP speculation and Qwen3-VL vision deferred.
+**In progress** — `Qwen3.8-27B` full support (Phase 14 retarget): text
+parity is bit-exact and thinking controls (`reasoning_effort` depths,
+`preserve_thinking`) + tool calling are wired; native MTP speculation
+(the Qwen-trained head as a `--draft-model`), vision, and video input
+are in bring-up.
 
 **Experimental** — opt-in, default-off, still being hardened: transparent
 expert offload for MoE models (`serve --expert-offload`, Phase 20:

package/THIRD_PARTY_LICENSES.md

@@ -21,6 +21,7 @@ the upstream algorithm and is a derivative work of the original.
 | [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) (BSD-3-Clause) — Gemma-4 unified `VisionEmbedder`, `MultimodalEmbedder`, image preprocessing (as vendored in mlx-optiq) | BSD-3-Clause | `src/vision/embedder.ts`, `src/vision/preprocess.ts` |
 | mlx-optiq (MIT) — vision sidecar wiring, fused quantized SDPA orchestration | MIT | `src/vision/`, quantized-KV paths in `src/model/gemma4.ts` |
 | [Pillow](https://github.com/python-pillow/Pillow) (MIT-CMU / HPND) — `ImagingResample` bicubic convolution algorithm | HPND | `resizeBicubic` in `src/vision/preprocess.ts` (algorithm port) |
+| [Colibri](https://github.com/JustVugg/colibri) (pinned development oracle `44e489b196c9b7876b3d37a0570ebf1c6f90f54c`) — direct-container layout, expert-residency/I/O policy, GLM-5.2 execution and Metal-kernel reference | Apache-2.0 | Native GLM-5.2 implementation under `src/model/glm52-*`, `src/expert-*`, and `src/native/expert_io.c`; rewritten for Bun + MLX, with direct Colibri retained only as an external oracle |
 
 ## Runtime npm dependencies
 
@@ -40,3 +41,11 @@ Model weights (e.g. `mlx-community/gemma-4-12B-it-OptiQ-4bit`) are not
 part of this repository and are governed by their own licenses and use
 policies (e.g. Google's Gemma Terms of Use). Check the model card on
 Hugging Face before redistribution.
+
+The validated GLM-5.2 direct-container artifact,
+[`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`](https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp),
+declares MIT and identifies itself as a Colibri conversion/derivative of
+[`zai-org/GLM-5.2-FP8`](https://huggingface.co/zai-org/GLM-5.2-FP8), with
+lineage through `jlnsrk/GLM-5.2-colibri-int4` and added int8 MTP weights. The
+artifact is downloaded separately into the Hugging Face cache and is not part
+of the mlx-bun distribution.