mlx-bun 0.0.12 → 0.2.0

package/docs/planning/video-series.md

@@ -0,0 +1,421 @@
+# mlx-bun video series
+
+## Episode 1: local AI on a Mac without the setup ritual
+
+### Production brief
+
+- **Working title:** Running a 27B Model Locally on a Mac, Without Python
+- **Thumbnail text:** 27B. LOCAL. NO PYTHON.
+- **Target length:** 8 minutes
+- **Audience:** developers who know what local models are but have not used MLX
+- **Machine:** whichever machine is used for the final recorded benchmark; show
+  its chip and memory alongside the results
+- **Demo model:** `mlx-community/Qwen3.8-27B-OptiQ-4bit`, already downloaded
+- **Presentation:** calm, direct, mostly terminal and browser capture with short
+  pieces to camera
+- **Question the episode answers:** Can local AI on a Mac feel like installed
+  software instead of an ML engineering project?
+
+This episode proves the product before explaining the implementation. It does
+not tour every feature. Memory, training, model ports, Bun FFI, and the failed
+experiments belong in later episodes.
+
+### Recording setup
+
+Prepare two terminal profiles:
+
+1. A clean profile for the installation shot. Do not begin the model download
+   during the take unless the download duration is known and useful on camera.
+2. The normal development profile with Qwen3.8-27B already downloaded. Use
+   this for the live server and API demonstrations.
+
+Before recording, confirm the exact model selector accepted on the machine:
+
+```sh
+mlx-bun ls
+mlx-bun fit Qwen3.8-27B --ctx 8192
+```
+
+Start the demo server shortly before the live take if model loading or browser
+placement would interrupt the opening. The visible command should still be the
+real command:
+
+```sh
+mlx-bun serve Qwen3.8-27B
+```
+
+Use a fresh chat and disable notifications. Set the terminal to at least 18 pt.
+Keep the browser and terminal at the same zoom level throughout.
+
+### Shot-by-shot script
+
+#### 0:00 to 0:12, cold open
+
+**Picture**
+
+Close shot of an empty terminal. Type `mlx-bun serve Qwen3.8-27B`. Cut directly
+to the browser as the chat opens. Ask a short question whose answer starts
+quickly.
+
+Suggested prompt:
+
+> Why can a 27-billion-parameter model fit on this 32 GB Mac? Answer in three
+> sentences and show the memory arithmetic.
+
+Do not wait for the full answer. Once several words appear, cut to camera.
+
+**Voiceover**
+
+> That is a 27-billion-parameter Qwen model running locally on a Mac. There is no
+> Python server behind it, no API key, and no request leaving the machine. One
+> command started the model, the chat interface, and an API that existing tools
+> can use. This is mlx-bun.
+
+**On-screen text**
+
+```text
+<machine> · Qwen3.8-27B · 20.4 GB weights
+```
+
+#### 0:12 to 0:35, state the problem
+
+**Picture**
+
+Piece to camera. Brief cuts to a deliberately restrained list of familiar setup
+artifacts: a Python environment, package installation, a server command, and a
+model configuration file. Do not use fake error messages.
+
+**Dialogue**
+
+> Local AI already works on a Mac. The rough part is everything around the
+> model. You choose a model that might fit, assemble a Python environment,
+> start a server, and then configure the application that was supposed to use
+> it.
+>
+> I wanted the default path to feel like installing software. Run one command,
+> get a useful local assistant, and keep the controls available when you
+> actually need them.
+
+#### 0:35 to 1:10, show the real first-run path
+
+**Picture**
+
+Show the installation command in the clean terminal. Use a pre-recorded,
+time-compressed capture of the first run beside it. Label the time compression.
+
+```sh
+curl -fsSL https://mlx-bun.dev/install.sh | sh
+mlx-bun
+```
+
+Show the starter model becoming ready. Then cut to the explicit Qwen download
+command used before the main recording:
+
+```sh
+mlx-bun get mlx-community/Qwen3.8-27B-OptiQ-4bit
+```
+
+**Voiceover**
+
+> The installer downloads a signed, notarized executable. On a fresh machine,
+> bare `mlx-bun` downloads a sub-gigabyte MiniCPM5 starter so the first chat can
+> begin quickly.
+>
+> This episode is about a much larger model, Qwen3.8-27B. Its quantized weights
+> are 20.4 gigabytes, so I downloaded them before recording. I do not think
+> watching a twenty-gigabyte progress bar would improve the video.
+
+**Verification note**
+
+Capture this section again if first-run behavior changes. Do not narrate the
+device-aware model selection described in the product roadmap until that lookup
+is implemented.
+
+#### 1:10 to 1:55, one process with three ways in
+
+**Picture**
+
+Use a simple screen composition with the running `mlx-bun` process in the
+center. Reveal the chat UI, an HTTP request, and a TypeScript file around it.
+
+```text
+Chat UI
+   \
+    mlx-bun + MLX + Metal
+   /                     \
+OpenAI-compatible API    TypeScript library
+```
+
+**Dialogue**
+
+> The visible chat is only one way into the runtime. The same process exposes an
+> OpenAI-compatible API, so applications that already speak that protocol can
+> point at localhost. A Bun application can also call the generation library
+> directly without running a sidecar server.
+>
+> Those are different entry points into the same native runtime. The model
+> weights stay on the Mac, and MLX runs the computation through Metal on the
+> Apple Silicon GPU.
+
+#### 1:55 to 2:35, prove the API claim
+
+**Picture**
+
+Keep the server log visible on the left. Run this request on the right:
+
+```sh
+curl http://localhost:8080/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "messages": [{
+      "role": "user",
+      "content": "Name two reasons unified memory helps local inference."
+    }],
+    "max_tokens": 96,
+    "temperature": 0
+  }'
+```
+
+Format or crop the JSON response so the answer and usage fields remain legible.
+
+**Voiceover**
+
+> This is a normal chat-completions request. Change an OpenAI client base URL to
+> `localhost:8080`, and the request goes to the model loaded by mlx-bun. There
+> is no translation service on the internet and no account involved.
+
+Do not say every OpenAI feature is implemented. The claim is protocol
+compatibility for the documented API, not the entirety of OpenAI's platform.
+
+#### 2:35 to 3:25, explain why Bun matters
+
+**Picture**
+
+Piece to camera, followed by a clean runtime diagram. Avoid scrolling through
+large source files here.
+
+```text
+TypeScript application
+        ↓
+       Bun
+        ↓
+  mlx-bun native bindings
+        ↓
+       MLX
+        ↓
+ Apple Silicon GPU
+```
+
+**Dialogue**
+
+> Most MLX applications are written in Python. mlx-bun calls MLX's native C API
+> from Bun instead.
+>
+> It loads safetensors weights, builds the model graph, manages the attention
+> cache, samples tokens, and serves requests from TypeScript. It does not start
+> Python in the background.
+>
+> That does not make GPU matrix multiplication faster because it passed through
+> JavaScript. The GPU still sets that limit. It does make the application easier
+> to distribute, and it gives the server less work to do around each model step.
+
+#### 3:25 to 4:15, show the finite-matrix idea
+
+**Picture**
+
+Run the fit command. Hold on the result long enough to read memory use, safe
+context, and predicted speed. Use the exact live output rather than rebuilding
+it for the edit.
+
+```sh
+mlx-bun fit Qwen3.8-27B --ctx 8192
+```
+
+**Dialogue**
+
+> The project makes one opinionated bet. Local AI on a Mac is a finite problem.
+> Apple ships a known set of chips and memory sizes. mlx-bun supports a curated
+> set of model families. We can measure those combinations instead of asking
+> every user to become a model-serving specialist.
+>
+> The `fit` command estimates whether a model and context length are safe before
+> loading the weights. The default path can be simple because the measurements
+> and the failure checks live underneath it.
+
+Do not claim that automatic device-specific recommendation is complete. Today,
+the fit contract is the concrete proof of this idea.
+
+#### 4:15 to 5:20, correctness before speed
+
+**Picture**
+
+Show a two-column comparison titled `Python oracle` and `mlx-bun`. Animate two
+small arrays of identical hexadecimal or decimal values. Follow with a quick
+shot of a parity test name from the repository.
+
+```text
+same model + same input
+
+mlx-lm logits  =  mlx-bun logits
+               bit-exact
+```
+
+**Dialogue**
+
+> Reimplementing a model runtime creates a nasty testing problem. Wrong models
+> still write convincing sentences.
+>
+> So I do not use "the answer looked reasonable" as the correctness test. For
+> the compatibility path, the Python MLX implementation is the oracle. The same
+> weights and inputs go through both runtimes, and the tests compare the logits
+> before a token is selected.
+>
+> On the validated paths, those logits match bit for bit. That catches errors in
+> attention masks, normalization, position encoding, cache updates, and model
+> loading that a chat demo can easily hide.
+>
+> Compatibility work has a strict order here. Match the reference first. Then
+> measure any optimization against that baseline.
+
+#### 5:20 to 6:20, measured performance
+
+**Picture**
+
+Display one benchmark card. Do not show the entire results table.
+
+```text
+Qwen3.8-27B · <machine> · warm HTTP serving
+Preflight-gated run · 2026-08-22 · Bun 1.4.0
+
+                 short decode   decode at 15.8k   warm TTFT   aggregate ×4
+mlx-bun          18.6 tok/s      17.0 tok/s        116 ms      19.5 tok/s
+mlx-lm           16.5 tok/s      14.8 tok/s        284 ms      18.6 tok/s
+```
+
+Add a small source line: `benchmarks/RESULTS.md`.
+
+**Dialogue**
+
+> Here is a measured comparison on this Mac. Same Qwen model snapshot, same
+> machine state, warm HTTP serving.
+>
+> On the short request, mlx-bun generated 18.6 tokens per second. mlx-lm
+> generated 16.5. At about 15,800 tokens of context, the result was 17.0 versus
+> 14.8. Warm time to first token was 116 milliseconds for mlx-bun and 284 for
+> mlx-lm.
+>
+> This is the kind of comparison I care about. It uses the server path an
+> application actually hits, the same weights, and a machine that passed the
+> benchmark preflight. The GPU sets the speed limit, but the runtime still has
+> to reach that limit without wasting memory or time around each token.
+
+**Editorial note**
+
+These are controlled same-machine results. Put the run date and Bun version on screen.
+If a fresh cleared-machine benchmark is available before publication, replace
+the full card and narration together. Never mix live numbers with this table.
+
+#### 6:20 to 7:05, show the larger project without losing focus
+
+**Picture**
+
+Use four quick clips, about three seconds each:
+
+1. An image input in chat.
+2. The Markdown memory directory.
+3. A training progress display.
+4. A benchmark or parity result from the lab.
+
+Return to camera before naming more features.
+
+**Dialogue**
+
+> Serving is the base of the project, not its boundary. The same binary also has
+> image and video input on supported Qwen models, a local Markdown memory, LoRA
+> training, and a lab for decoding and quantization experiments.
+>
+> I am not going to explain those in a thirty-second feature list. Each one has
+> an implementation story, measurements, and a few failures worth discussing.
+> Those will be separate videos.
+
+#### 7:05 to 7:50, close and set up episode 2
+
+**Picture**
+
+Return to the original chat. Show the local URL, server process, and activity
+monitor GPU history in quick succession. End on the repository and site URLs.
+
+**Dialogue**
+
+> mlx-bun is my attempt to make local AI on Apple Silicon behave like software
+> you can install, use, and build on.
+>
+> The short version is one command. Underneath it is a TypeScript model runtime,
+> native bindings, model-family ports, memory accounting, and a lot of parity
+> tests.
+>
+> In the next video, I am going to trace one generated token through that whole
+> path, from the TypeScript call to MLX and the GPU, then back to the response.
+> That is where the project gets strange.
+
+**On-screen text**
+
+```text
+mlx-bun.dev
+github.com/joshuarossi/mlx-bun
+```
+
+### Required captures
+
+- Clean installer command.
+- Time-compressed first run with MiniCPM5 becoming ready.
+- The explicit Qwen3.8-27B download command or a short time-compressed capture.
+- `mlx-bun serve Qwen3.8-27B` startup and automatic browser open.
+- A short streamed chat response.
+- A successful `/v1/chat/completions` curl request.
+- Live `mlx-bun fit Qwen3.8-27B --ctx 8192` output.
+- One real parity test or stored oracle comparison.
+- Four brief clips for vision, memory, training, and benchmark evidence.
+- Repository and website closing shot.
+
+### Graphics to prepare
+
+Only three graphics are needed:
+
+1. The three-entry-point runtime diagram.
+2. The bit-exact oracle comparison.
+3. The dated Qwen3.8-27B benchmark card.
+
+Keep them monochrome except for one mlx-bun accent color. Use the same terminal
+font in the diagrams and screen recordings.
+
+### Recording-day checks
+
+- Confirm the current release installation command.
+- Confirm the Qwen3.8-27B selector and exact displayed model name.
+- Confirm port 8080 is unused before the take.
+- Confirm no private paths, tokens, conversations, or model-cache usernames are
+  visible.
+- Confirm first-run behavior still matches the narration.
+- Confirm the API request and response fields against the current reference
+  docs.
+- Confirm the benchmark card still matches `benchmarks/RESULTS.md`.
+- Record room tone and five seconds of clean terminal footage for edits.
+- Capture every important terminal action twice.
+
+### Episode 2 handoff
+
+The promised follow-up is: **What happens when mlx-bun generates one token?**
+
+Trace this path without turning the episode into a source-tree tour:
+
+```text
+request
+  → tokenizer
+  → model forward pass
+  → MLX lazy graph evaluation
+  → Metal execution
+  → GPU sampling
+  → one token returned to Bun
+  → streamed protocol event
+```

package/docs/reference/cli.md

@@ -35,7 +35,7 @@ Common flags (full list in [server-config.md](server-config.md)):
 |---|---|
 | `--host <addr>` | Interface to bind (default `127.0.0.1`, loopback only; `0.0.0.0` to expose) |
 | `--port <n>` | Listen port (default 8080) |
-| `--memory-budget <GB>` | Reject loads/requests that can't fit the budget |
+| `--memory-budget <GB>` | Admission ceiling for requests (bills the active `--kv-quant` scheme; clamps `max_tokens`, 400s only prompts that leave no room — never refuses to start) |
 | `--kv-budget <GB>` | Aggregate KV budget across concurrent batch rows: over-budget joiners queue until rows finish; a request over the budget alone is rejected (off unless set) |
 | `--prompt-cache <GB>` | RAM prefix-KV cache, byte-capped LRU (default 8; `0` disables). Non-consuming prefix sharing: a hit clones instead of consuming the donor entry |
 | `--isolate` | Run the inference engine as a crash-isolated child process on a unix socket; this process becomes a pure reverse-proxy (instant under GPU load, auto-respawns on engine crash) — default off ([runtime-isolation.md](../design/runtime-isolation.md)) |
@@ -49,12 +49,12 @@ Common flags (full list in [server-config.md](server-config.md)):
 | `--paged-kv-block-size <n>` | Tokens per KV block (default 256 = the plain cache's growth step; `--paged-kv` only) |
 | `--adapter <dir>` | Mount a LoRA adapter at startup (`--adapter-path` accepted as the mlx_lm.server alias) |
 | `--draft-model <path\|query>` | Speculative decoding: a drafter proposes, the main model verifies — exact results, faster decode when drafts land. The artifact's kind is auto-detected: a full same-tokenizer model (mlx_lm.server parity), a Gemma `-assistant` KV-borrowing drafter, a DSpark checkpoint trained here, or one of DeepSeek's released DSpark drafters (DeepSpec `Gemma4DSparkModel`, e.g. `dspark_gemma4_12b_block7`). Serial lane only: with `--batch N` a mounted draft routes every request serial |
-| `--draft-kind <kind>` | Override draft-artifact detection: `two-model` \| `assistant` \| `dspark` \| `deepspec` \| `ngram` (default: auto). `ngram` is MODEL-FREE prompt lookup — no `--draft-model` (mounting one alongside it is refused): drafts are copied from the request's own prompt+generation whenever its trailing tokens have occurred earlier in that stream, verified by the target like any other draft, so results stay exact at any temperature. Zero weights, zero memory; best on agentic/RAG/code-edit traffic that re-emits spans already in context (prior art: prompt-lookup decoding / vLLM's `ngram` proposer) |
+| `--draft-kind <kind>` | Override draft-artifact detection: `two-model` \| `assistant` \| `dspark` \| `deepspec` \| `mtp` \| `ngram` (default: auto). `mtp` is a NATIVE multi-token-prediction head split from the target's own release (e.g. `mlx-community/Qwen3.8-27B-MTP-bf16`; auto-detected from a `*_mtp` `model_type`) — it binds the target's embeddings/lm-head and taps its pre-final-norm hiddens, and the serve loop rolls the target's recurrent DeltaNet caches back across rejected drafts via bit-exact snapshot/replay. `--draft-kind mtp` WITHOUT `--draft-model` resolves the companion from the artifact's own `mtp/` subfolder (single-repo bundles, e.g. the TQ releases) and refuses with a clear error when the model bundles none. `ngram` is MODEL-FREE prompt lookup — no `--draft-model` (mounting one alongside it is refused): drafts are copied from the request's own prompt+generation whenever its trailing tokens have occurred earlier in that stream, verified by the target like any other draft, so results stay exact at any temperature. Zero weights, zero memory; best on agentic/RAG/code-edit traffic that re-emits spans already in context (prior art: prompt-lookup decoding / vLLM's `ngram` proposer) |
 | `--num-draft-tokens <n>` | Drafts per verify round (default 3; `ngram` defaults to 10 — drafting is free; a DSpark draft pins to its trained block width) |
 | `--ngram-max <k>` / `--ngram-min <k>` | Prompt-lookup match-window bounds, `--draft-kind ngram` only: the longest/shortest trailing k-gram searched (defaults 3 / 1, the reference values) |
 | `--mtp on\|off` | GLM-5.2 checkpoint-native MTP (default on). Native MTP uses the exact serial verify lane; `off` enables ordinary continuous batching (batched per-row MTP is post-release) |
 | `--context-length <n>` | GLM-5.2 context reserved by the header-only resource equation (default 4096) |
-| `--batch <n>` | Continuous-batched serving cap, mlx-lm B=N parity (default 8; `--batch 1` pins the serial path) |
+| `--batch <n>` | Continuous-scheduler concurrency cap, mlx-lm parity at the same B (default 8; a lone admitted request uses the gated B=1 fast path, while `--batch 1` pins the preserved strict serial executor) |
 | `--temperature` / `--top-p` / `--top-k` / `--max-tokens` | Server-wide sampling defaults (per-request fields still win; GLM-5.2 defaults to the planned 128 generated tokens) |
 | `--l1` / `--l2` | Parity tier alias: bit-exact to mlx-lm / bit-exact to mlx-optiq. **No tier = `--l1`** (the default since 2026-07-05 — output-changing levers are opt-in until they beat the L1 baseline in a paired A/B). Each expands to per-fork flags (`--compiled-decode`, `--compiled-activations`, `--fused-sdpa`, `--kv-quant`); a fork flag overrides one. `--l3` was removed 2026-07-05 and now errors (the Lab replaces it — [unified-engine-frontier-plan.md](../design/unified-engine-frontier-plan.md)). See [server-config.md](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2). |
 | `--allow-private-media` | Let `image_url`/`audio_url` parts fetch from private/loopback/link-local hosts (blocked by default — SSRF guard; the 10 s timeout and 64 MB cap on remote media apply either way) |
@@ -170,11 +170,15 @@ mlx-bun gc --yes    # delete
 
 Deterministic memory assessment: does it fit, what's the max context, predicted
 tok/s. Weights bytes from safetensors headers, KV bytes/token from the config,
-calibrated prefill transient, wired-memory ceiling.
+calibrated prefill transient, wired-memory ceiling. `--kv-quant` mirrors serve's
+flag: it bills the quantized cache at its true bytes/element, so the reported
+window matches what serving with the same flag admits (a 4-bit cache holds
+~3.5× the context of bf16 in the same budget).
 
 ```sh
 mlx-bun fit gemma --ctx 32768          # for this machine
 mlx-bun fit gemma --ctx 8192 --skus    # across the Apple Silicon lineup
+mlx-bun fit qwen --kv-quant 4          # window with a 4-bit-quantized KV cache
 ```
 
 The direct Colibri GLM-5.2 artifact is handled by its streamed-runtime
@@ -235,8 +239,15 @@ MLX snapshot — uniform 4/8-bit, or mixed-precision via `--target-bpw`
 ```sh
 mlx-bun convert Qwen/Qwen3-4B -q                      # uniform 4-bit
 mlx-bun convert Qwen/Qwen3-4B --target-bpw 4.5        # mixed precision
+mlx-bun convert ./qwen3.5-bf16 -q --rotate-weights    # fold then quantize
 ```
 
+`--rotate-weights` makes the offline rotation part of the same conversion
+pipeline and records its recipe in `optiq_metadata.json`. The adapter is
+selected from the model/tensor schema (Llama, Qwen3.5 trunk, or Qwen MTP);
+`--rotation-seed` defaults to `42`. Folding requires full-precision source
+weights and an MTP companion must use the same seed as its trunk.
+
 ### `upload` — push to the Hugging Face Hub
 
 Native push-to-hub, no Python: creates the repo if needed, uploads via the

package/docs/reference/distribution.md

@@ -15,6 +15,7 @@ tarball (~73 MB compressed, ~228 MB installed):
 | `mlx-bun` | the CLI/server (`bun build --compile`), ~61 MB |
 | `libmlxc.dylib`, `libmlx.dylib`, `libjaccl.dylib` | MLX native runtime |
 | `libmlx_bun_expert_io.dylib` | bounded native expert-streaming helper required by direct Colibri GLM-5.2 |
+| `mlx-bun-frame-extract` | AVFoundation frame-extraction sidecar for video input (darwin-native codecs, the afconvert doctrine) |
 | `mlx.metallib` | Metal kernels (~150 MB — the bulk) |
 | pi assets | `photon_rs_bg.wasm`, `theme/`, `assets/`, `export-html/`, `native/`, `package.json`, `CHANGELOG.md` |
 

package/docs/reference/features-matrix.md

@@ -20,7 +20,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | Raw text completions (`/v1/completions`) | on | both | — | — |
 | Anthropic Messages (`/v1/messages`) | on | both | — | — |
 | OpenAI Responses shim (`/v1/responses`) | on | both | — | — |
-| Continuous batching (mlx-lm B=N parity) | **on** — cap 8 (default flipped 2026-07-05; a lone request runs the exact serial engine, so the cap engages only under real concurrency; `--batch 1` pins serial) | batch | L1 | `--batch <n>` |
+| Continuous batching (mlx-lm parity at the same B) | **on** — cap 8 (default flipped 2026-07-05; a lone admitted request uses the gated B=1 fast path and concurrent requests can form B=N; `--batch 1` pins the strict serial executor) | batch | L1 | `--batch <n>` |
 | Prompt cache (prefix KV reuse) | on, 8 GB | both | — | `--prompt-cache <GB>` (0 = off) |
 | **SSD KV cold tier** (cache survives eviction + restarts; v3 includes GLM compressed MLA/DSA state without full-K/V reconstruction) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
 | **Runtime isolation** (engine runs as a crash-isolated CHILD process behind a reverse-proxy parent; UI stays instant under GPU load and survives engine crashes — [runtime-isolation.md](../design/runtime-isolation.md)) | off | both | — | `--isolate` |
@@ -51,6 +51,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | Quantized KV × speculative decoding (any axis: uniform / config / turbo) | KV scheme wins — drafted requests decode serially WITHOUT speculation (spec lane is bf16-KV-only in v1; startup warning) | serial | — | omit `--kv-quant` to speculate |
 | Tool calling (Gemma sentinel / CPM+Qwen XML / GLM `arg_key`+`arg_value`) + `role:"tool"` loops | on | both | — | request `tools` |
 | Vision (`image_url` parts; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on (models with a tower); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial | L1/L2 | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
+| **Video input** (`video_url`/`video` parts; mp4/mov H.264/HEVC/ProRes/… via the AVFoundation sidecar `mlx-bun-frame-extract`; 2 fps sampling, ≤768 frames, 256 MB body cap; never combines with audio) | on (Qwen3.5-family models); same SSRF guard | serial | mlx-vlm oracle (frames pipeline bit-exact; tests/qwen38-vision.test.ts) | `--allow-private-media` for LAN hosts |
 | **Audio input** (`input_audio`/`audio`/`audio_url` parts; WAV native, mp3/m4a/flac/ogg/aiff via CoreAudio; ≤30 s / 750 tokens per clip; mixes with images) | on (models with `audio_config` + sidecar tower: e4b); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial³ | L2 (greedy stream EXACT vs the optiq internal model) | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
 | LoRA adapters (mount at start / hot-swap at runtime) | off | serial | — | `--adapter <dir>`, `POST /v1/adapters` |
 | Sampling: temperature / top-p / top-k / min-p / XTC / logit_bias / presence+frequency+repetition penalties | per-request | both | L1 (mlx-lm-faithful) | request fields / server defaults |
@@ -69,6 +70,7 @@ parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
 | MiniCPM5 (cpm5) | ✅ L1/L2 | ✅ | the starter model |
 | Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
 | Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
+| Qwen3.8-27B (same qwen3_5 graph) | ✅ L1 (no L2 oracle) | ✅ (SSM path) | thinking depths + preserve_thinking; native MTP head mounts via `--draft-model` (DeltaNet spec rollback = snapshot/replay, lossless-gated; quiet-box A/B: 0.82× — correct but slower, opt-in); **vision (images) AND video SERVE** — bf16 tower sidecar + mRoPE + embed splice, oracle = mlx-vlm on pinned mlx (preprocessor/tower/positions bit-exact, e2e token-exact on 2/3 fixtures, serve smokes green; tests/qwen38-vision.test.ts); video files decode via the shipped AVFoundation sidecar (`mlx-bun-frame-extract`, src/native/frame_extract.swift — 2 fps, ≤768 frames, temporal pairs, per-frame-group attention) |
 | GLM-5.2 / Colibri (pre-release port gate) | ✅ chat/text, Messages, Responses, SSE, tools, grammar, logprobs, library/CLI | ✅ compressed MLA/DSA scheduler | native MTP defaults to serial+spec; `--mtp off` enables ordinary batching; artifact-aware CLI/API/status `fit` and live main/MTP expert-tier telemetry; embeddings, vision/audio, adapters, and training are explicitly unsupported; per-row batched MTP is post-release |
 | DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
 | Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family/sliding universal → serial |

package/docs/reference/library-api.md

@@ -8,7 +8,7 @@ admission — is importable directly into a Bun process. Published as
 
 One rule everywhere: **one generation at a time per process** (one GPU).
 The server serializes through a queue (or, under `--batch N`, its
-continuous-batching scheduler — a server-lane feature, not a library
+continuous scheduler — a server scheduling mechanism, not a library
 surface); direct library callers must do the same.
 
 ## Quick start
@@ -49,6 +49,46 @@ Ordinary safetensors loading is lazy (mmap + mlx native loader). GLM-5.2 first
 runs its header-only exact process equation, then opens the bounded Colibri
 resident/expert tiers; impossible plans fail before either tier is committed.
 
+## Model profiles
+
+`openModel()` and `loadContext()` resolve a declared model profile before they
+open weights. `loadContext()` exposes that decision as `ctx.profile`, including
+the exact external fingerprint when the path is an immutable Hugging Face
+snapshot:
+
+```ts
+const ctx = await loadContext(dir);
+console.log(ctx.profile.profile.id);
+console.log(ctx.profile.artifact.fingerprint); // hf:<org>/<repo>@<revision> or null
+console.log(ctx.profile.profile.fidelity);
+console.log(ctx.profile.profile.execution);    // loader + graph + loop + specialization
+```
+
+Direct library construction can resolve and pass the same immutable value:
+
+```ts
+import { loadModelConfig, Weights, resolveModelProfile, createModel } from "mlx-bun";
+
+const config = await loadModelConfig(dir);
+const profile = resolveModelProfile(config);
+const model = createModel(await Weights.open(dir), config, profile);
+```
+
+For a local artifact that is not stored under an immutable HF snapshot path, an
+application may supply its own provenance fingerprint and exact declaration via
+`resolveModelProfile(config, { artifactFingerprint, artifactProfiles })`.
+Declarations use the exported `ArtifactModelProfile`, `FidelityTarget`,
+`EngineCapability`, and `ModelExecutionComposition` types. The resolver checks
+the declared config fingerprint (compute it with exported
+`configFingerprint(config)`) and required engine capabilities. An exact match
+that fails either check refuses instead of taking the family or generic
+fallback.
+
+Profile selection is limited to model construction. Generation options remain
+authoritative: passing mixed `kvConfig`, TurboQuant KV, adapters, grammar,
+sampling, or an explicitly configured draft method is not rewritten by the
+profile.
+
 > **First run:** library consumers bypass the CLI's first-run step, so
 > call `await ensureNativeRuntime()` once before constructing a model on
 > a machine that may not have the MLX native runtime yet — it downloads

package/docs/reference/models.md

@@ -106,6 +106,39 @@ The artifact is an external MIT-licensed derivative of
 weights. Model weights are not bundled with mlx-bun; review the artifact model
 card before redistribution.
 
+## Declared model profiles
+
+Model loading resolves one immutable profile before weights are opened. A
+profile names four things: the external artifact fingerprint when one is
+known, its fidelity target, the engine capabilities it requires, and the
+composed loader/model-graph/generation-loop path.
+
+Exact artifact profiles outrank family profiles. The current exact declarations
+are:
+
+| profile | external artifact | fidelity | execution |
+| --- | --- | --- | --- |
+| `qwen3.8-27b-optiq-4bit` | `mlx-community/Qwen3.8-27B-OptiQ-4bit` revision `b04599de95d7a9bfbd7f208d347c0f10d9432a42` | L1: bit-exact mlx-lm | safetensors + dedicated Qwen3.5/3.8 graph + autoregressive loop |
+| `glm5.2-colibri-int4-int8-mtp` | `mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp` revision `3cc8db99b1b13fc79325d987ba3c1c430766b3b8` | L3: measured, no matching tier oracle | Colibri container + dedicated GLM-5.2 graph + autoregressive loop |
+
+The fingerprint is the Hugging Face repo plus immutable snapshot revision, so
+moving the cache does not change it. A mutable alias such as `snapshots/staged`
+or an arbitrary local directory is not treated as exact. Those artifacts use
+the dedicated model-family profile when one matches, then the universal dense
+profile when that architecture is supported. Unsupported architectures still
+refuse with the existing targeted/generic support list.
+
+Profiles select construction only. They do not enable, disable, or replace MTP,
+mixed-precision KV from `kv_config.json`, TurboQuant KV, adapters, grammar, or
+sampling. Those methods remain explicit flags/request fields or documented
+defaults. If an exact profile requires a capability the running engine does not
+have, loading refuses; it never downgrades to another model path.
+
+Fidelity labels mean L1 = bit-exact mlx-lm, L2 = bit-exact mlx-optiq, and L3 =
+measured evidence without a matching oracle. Model profiles do not imply that a
+quantized artifact retains the base model's benchmark score; quality evidence
+belongs to that exact artifact.
+
 ## scan + the registry
 
 `mlx-bun scan` walks `models--*/snapshots/*` reading only `config.json`