mlx-bun 0.0.13 → 0.3.0

package/README.md

@@ -1,616 +1,109 @@
 # mlx-bun
 
-**One command, chatting in under a minute** — a fully local AI on your
-Apple Silicon Mac's GPU, no Python, no API key:
-
-```sh
-curl -fsSL https://mlx-bun.dev/install.sh | sh
-mlx-bun
-```
+A Bun/TypeScript native inference engine and OpenAI-compatible HTTP server
+for MLX on Apple Silicon — no Python, no sidecar process. `mlx-bun serve` is
+a drop-in for `mlx_lm.server` (same port, endpoints, request fields, and
+flags); the correctness contract is logit parity with mlx-lm as the oracle,
+verified bit-exact by the test suite. Ships as one signed, notarized binary,
+plus a library-first TypeScript API for embedding generation directly in
+Bun, Tauri, or Electron apps.
 
 Docs: **[mlx-bun.dev](https://mlx-bun.dev)**
 
-Native MLX inference for Bun, and the beginning of a one-command local AI
-product for Apple Silicon Macs. `mlx-bun` detects the machine, starts a useful
-local assistant, exposes OpenAI/Anthropic-compatible APIs, and gives
-TypeScript apps direct access to MLX without Python or a sidecar server.
-
-Six goals, in one binary:
-
-1. **A drop-in replacement for `mlx_lm.server`** — same port, endpoints,
-   request fields, and flags; stop the Python server, start `mlx-bun serve`,
-   the same curl works ([site guide](https://mlx-bun.dev/guides/drop-in-mlx-lm/)).
-2. **A TypeScript library** — embed MLX generation in Bun/Tauri/Electron apps,
-   or ship it as a signed single-binary sidecar
-   ([library](./docs/reference/library-api.md), [embedding](./docs/reference/embedding.md)).
-3. **Bit-exact, fast, and memory-honest** — logit parity with the Python
-   reference as the correctness oracle, measured speed wins, and a
-   deterministic `fit` memory contract ([benchmarks](#benchmarks),
-   [correctness](#correctness)).
-4. **One command, under a minute** — the appliance path above; the default
-   experience needs zero configuration ([getting started](#getting-started)).
-5. **A personal memory** — a local, git-tracked wiki your assistant reads and
-   (via local synthesis) writes ([memory](#personal-memory)).
-6. **An AI lab** — training, speculative decoding, and sampling research kept
-   next to the runtime that serves them ([the lab](#the-lab)).
-
-The product bet is that local AI on Mac is a finite optimization problem, not
-an infinite configuration maze: Apple ships a known set of chip/RAM/bandwidth
-SKUs, the supported model list is curated, and the task mode is explicit. That
-means defaults can be chosen from measured device × model × mode profiles
-instead of asking every user to become an ML systems engineer.
+## Scope
 
-Measured head-to-head on an M4 Pro (24 GB), same models, same machine
-state: logits **bit-exact** against the Python reference; served over
-HTTP the **fastest stack on every model tested** — fastest decode, plus
-TTFT **34–85 ms** vs python's 218–326 ms at **~0% server overhead**
-(mlx-lm pays −5–6%); server start to ready in **0.17–0.47 s**, 2–5×
-faster than the Python servers. Full table: [benchmarks](#benchmarks).
+Apple Silicon only — MLX is a Metal framework, so this runs on macOS on M-series
+chips by design, nothing else. Serving is single-user: one process serves one
+loaded model to one caller at a time (`--isolate --model-pool N` pools
+multiple single-model children behind a router; it isn't a multi-tenant
+scheduler). Model support is a curated, oracle-backed list, not a generic
+"any HF repo" loader — see the roster for what's validated.
 
-## Getting started
+## Install
 
-You need an Apple Silicon Mac (MLX is Metal-only, so macOS only by
-design). Four ways in.
-
-### Direct download (recommended)
-
-No Homebrew, Bun, or git — just curl the install script. It downloads the
-signed, notarized self-contained binary from the latest release and puts
-`mlx-bun` on your PATH:
+Four ways in; all need an Apple Silicon Mac running macOS.
 
 ```sh
+# Direct download — signed, notarized, no toolchain
 curl -fsSL https://mlx-bun.dev/install.sh | sh
-mlx-bun
-```
-
-Self-contained (binary + MLX runtime) and notarized, so it runs without a
-Gatekeeper prompt. Installs to `~/.mlx-bun` by default — override with
-`MLX_BUN_INSTALL_DIR`, or pin a release with `MLX_BUN_VERSION=<tag>`
-(see [releases](https://github.com/joshuarossi/mlx-bun/releases)).
-
-### Homebrew
-
-A self-contained, signed + notarized binary — no toolchain to install by
-hand:
 
-```sh
+# Homebrew
 brew install joshuarossi/tap/mlx-bun
-mlx-bun
-```
-
-That's the whole thing. The bottle already bundles the MLX native
-runtime, so the only thing the first run fetches is the model.
 
-### bunx (no install)
-
-Already have [Bun](https://bun.sh) ≥ 1.3.14? Run it straight from npm,
-nothing to install:
-
-```sh
+# bunx — no install, needs Bun >= 1.4.0
 bunx mlx-bun
-```
-
-First run fetches the MLX native runtime plus the streamed-expert helper
-(~52 MB) and the model into your caches, then serves. (Bun only — `npx
-mlx-bun` under Node exits with a
-"requires Bun" notice by design.)
-
-### From source
-
-Needs [Bun](https://bun.sh) ≥ 1.3.14 — no Python, no native library to
-install by hand:
-
-```sh
-# 1. Install Bun
-curl -fsSL https://bun.sh/install | bash
-exec $SHELL -l
 
-# 2. Clone, install, link the CLI
+# From source
 git clone https://github.com/joshuarossi/mlx-bun.git && cd mlx-bun
-bun install
-bun run link-cli                 # adds the `mlx-bun` command — or run `bun src/cli.ts <verb>`
-
-# 3. Run it — that's the whole thing
-mlx-bun
-```
-
-Bare `mlx-bun` is an alias for `mlx-bun serve`. On its first run, with no
-model named, it does everything for you:
-
-1. pulls the MLX native runtime plus streamed-expert helper (~52 MB) into
-   `~/Library/Caches/mlx-bun/`
-   (skipped on the Homebrew install — the runtime ships in the bottle);
-2. downloads the sub-GB `MiniCPM5-1B` starter and serves it, so you're
-   chatting in well under a minute;
-3. starts downloading `gemma-4-e4b` (the stronger 4B model) in the
-   background — it becomes the default on your next `mlx-bun serve`;
-4. opens the chat UI in your browser at
-   [`http://localhost:8080/#/chat`](http://localhost:8080/#/chat)
-   (pass `--no-open` to skip).
-
-Want a specific model instead? Name it — `mlx-bun serve e4b` (substring
-match against your downloaded models), grabbing it first with
-`mlx-bun get <repo-id>` if you don't have it. See
-[Supported models](#supported-models) for the full list and what fits
-your machine.
-
-**Prefer the API?** It's OpenAI-compatible — any OpenAI client works:
-
-```sh
-curl http://localhost:8080/v1/chat/completions \
-  -H 'Content-Type: application/json' \
-  -d '{
-    "messages": [{"role": "user", "content": "Hello!"}],
-    "max_tokens": 128, "temperature": 0.7
-  }'
+bun install && bun run link-cli
 ```
 
-That's it. The server serves the one model it was started with (the
-request's `model` field is ignored and the loaded model id is echoed
-back in responses). Run `serve --isolate --model-pool N` instead and
-the parent proxy pools per-model engine children, routing each request
-by its `model` field and spawning on demand — see
-[docs/reference/server-config.md](./docs/reference/server-config.md)
-and [docs/design/runtime-isolation.md](./docs/design/runtime-isolation.md).
-
-Longer walkthrough on the site:
-[Installation](https://mlx-bun.dev/getting-started/installation/) and
-[Quickstart](https://mlx-bun.dev/getting-started/quickstart/).
-
-## What mlx-bun is
-
-`mlx-bun` has several faces, all sharing one native runtime:
-
-- **The local AI product** — run `mlx-bun` or `bunx mlx-bun` and get a
-  working local chat UI plus a local server. The default path should feel like
-  an appliance: download the starter model, start chatting quickly, then move
-  to the stronger recommended model as it becomes available.
-- **The app developer library** — embed local MLX inference inside Bun,
-  Tauri, Electron, and other TypeScript applications without running Python in
-  the background.
-- **The background agent runtime** — leave a local OpenAI-compatible endpoint
-  available for scripts, recurring jobs, app integrations, and per-request
-  adapter routing.
-- **The local memory layer** and **the AI lab** — each gets its own section
-  below.
-
-The default experience is intentionally opinionated. Power users can override
-models, budgets, KV modes, adapters, and flags; the average user should not
-need to know those knobs exist.
-
-## Personal memory
-
-`mlx-bun memory init` creates a git-tracked Markdown wiki at
-`~/.mlx-bun/wiki`; `Reference/` is seeded with read-only symlinks to
-mlx-bun's own docs, and the built-in pi agents read both quietly through two
-separate tool families — `reference_search`/`reference_read` for the seeded
-Reference docs, `memory_search`/`memory_section`/`memory_read` for your own
-articles — and can open it in Obsidian for you (`mlx-bun memory open
-[article]`). Chat-time memory is
-read-only; synthesis is the separate writer path: `mlx-bun memory synthesize`
-runs the built-in local pipeline (segment → extract → route → create/patch →
-reconcile → link → wikify) that turns your conversations into cross-linked
-subject articles, using the same local models with a fine-tuned chunking LoRA
-adapter.
-
-It's your data in your format: plain Markdown, git history, editable in any
-tool, never leaves the machine. Reference:
-[docs/reference/memory.md](./docs/reference/memory.md); synthesis design:
-[docs/design/memory-synthesis.md](./docs/design/memory-synthesis.md); on the
-site: [Personal memory](https://mlx-bun.dev/guides/memory/).
-
-## The lab
-
-Parity, evals, kernels, training, and sampling research live next to the
-runtime that serves them:
-
-- **LoRA fine-tuning on your Mac** (SFT / DPO / ORPO) — `mlx-bun train
-  <model> --data <dir>`, with a flash-CCE Metal head, segmented backward, and
-  prefix-sharing so long-context preference training fits laptop memory;
-  watch runs live with `mlx-bun train-watch`
-  ([training](./docs/reference/training.md),
-  [quickstart](./docs/reference/orpo-quickstart.md)).
-- **The curve designer** — HLG tone-curve sampling with an interactive
-  designer in the server UI at `/curves`
-  ([design](./docs/design/hlg-sampling.md)).
-- **Speculative decoding research (DSpark)** — a KV-injection drafter
-  ([design](./docs/design/dspark-speculative-decoding.md)).
-- The full research trail — including dead ends — is tracked in
-  [docs/design/](./docs/design/) and
-  [docs/investigations/](./docs/investigations/).
-
-On the site: [The lab](https://mlx-bun.dev/about/lab/).
+Every channel builds from the same bundle. Details, signing, and the native
+runtime resolution order: [docs/reference/distribution.md](./docs/reference/distribution.md).
 
-## Supported models
+## Quickstart
 
-Scope is deliberate: a few model families held to explicit oracle contracts,
-rather than dozens held to none. Currently MiniCPM5, the Gemma-4 OptiQ quants,
-the Qwen 3.x family, and the direct-container GLM-5.2 path:
-
-| Model | Download | Fits on | Vision | Notes |
-|---|---|---|---|---|
-| [`mlx-community/MiniCPM5-1B-OptiQ-4bit`](https://huggingface.co/mlx-community/MiniCPM5-1B-OptiQ-4bit) | 0.92 GB | 8 GB | — | Sub-GB option for 8 GB machines; bit-exact 100-step oracle parity (bf16 + mixed KV), tool calling + agent loop verified |
-| [`mlx-community/Qwen3.5-4B-OptiQ-4bit`](https://huggingface.co/mlx-community/Qwen3.5-4B-OptiQ-4bit) | 3.1 GB | 8 GB | — | bf16-KV bit-exact parity (vs mlx-lm); thinking + tool calling; ~74 tok/s predicted |
-| [`mlx-community/gemma-4-e4b-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-e4b-it-OptiQ-4bit) | 7.0 GB | 16 GB | ✓ | **Recommended starter** (16 GB+); ~56 tok/s |
-| [`mlx-community/gemma-4-12B-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-12B-it-OptiQ-4bit) | 8.4 GB | 16 GB | ✓ | Vision + tool calling, both verified end-to-end |
-| [`mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit`](https://huggingface.co/mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit) | 18 GB | 24 GB | — | MoE (top-8 of 128 experts); ~54 tok/s |
-| [`mlx-community/Qwen3.8-27B-OptiQ-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-OptiQ-4bit) | 20.4 GB | 32 GB (24 GB with `--memory-budget`) | text-only for now | Current-gen Qwen; bf16-KV bit-exact parity (vs mlx-lm); thinking with `reasoning_effort` (xhigh/medium/low) + `preserve_thinking` + tool calling; MTP speculation, vision + video in bring-up |
-| [`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`](https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp) | ~384 GB / 357 GiB | 32 GB + fast disk | — | Direct streamed Colibri container; true top-8, native MTP, bounded 25 GiB process plan; warm single-stream performance is disk-bound and below the aspirational 2 tok/s target |
-
-Not sure what fits your machine? `bun src/cli.ts fit <model> --ctx 8192`
-gives a deterministic answer (see below). Downloading, cache layout, and
-reclaiming disk: [docs/reference/models.md](./docs/reference/models.md);
-on the site: [Choosing a model](https://mlx-bun.dev/getting-started/models/).
-
-### GLM-5.2: 32 GB quickstart
-
-Choose a volume with at least the artifact remainder plus 1 GiB free before
-starting; mlx-bun checks this before the first payload and credits resumable
-partials. The validated artifact revision and exact 25 GiB serving command are
-documented in the [model-management guide](./docs/reference/models.md#glm-52-on-a-32-gb-mac):
+Bare `mlx-bun` aliases to `mlx-bun serve`; with no model named it downloads a
+sub-GB starter and opens a chat UI. To pick a model explicitly:
 
 ```sh
-export HF_HUB_CACHE=/Volumes/FastSSD/huggingface/hub  # optional large fast volume
-mlx-bun get mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
-  --revision 3cc8db99b1b13fc79325d987ba3c1c430766b3b8
-mlx-bun scan
-mlx-bun serve GLM-5.2-colibri --memory-budget 26.8435456 \
-  --context-length 4096 --max-tokens 128 --mtp on
+mlx-bun serve e4b --port 8080
 ```
 
-Rerun `get` after interruption; partial files resume and every completed blob
-is checksum-verified. The model remains in the standard Hugging Face cache—no
-Python environment and no second converted copy inside this repository.
-
-## Why
-
-MLX is Apple's ML framework: hand-tuned Metal kernels for Apple Silicon,
-with official bindings for Python, C++, Swift, and C — but no JavaScript
-story. Today, a JS/TS app that wants local MLX inference must shell out to
-a Python server (mlx-lm, optiq) and accept that stack's fragility: venv
-setup, brittle download tooling, segfaults on exit, monkey-patched HTTP
-layers.
-
-The performance-critical work — every matmul, every attention pass — lives
-in MLX's C++/Metal core and is exposed through `mlx-c`. The Python layer on
-top is pure orchestration: model loading, tokenization, the sampling loop,
-serving. That layer is performance-neutral (the GPU dominates), so it can
-be rewritten in any runtime without losing speed — and Bun is the right
-one:
-
-- **`bun:ffi`** — lowest-overhead FFI of any JS runtime; binds `mlx-c`
-  directly, no node-gyp, no binding compilation.
-- **Lazy native weight loading** — mlx's loader materializes tensors on
-  first use; opening an 8.4 GB model takes milliseconds, and warm
-  restarts are near-instant on cached pages.
-- **`Bun.Image`** — native OS image codecs (HEIC, AVIF, WebP, JPEG, …)
-  for the vision path, EXIF auto-orientation included.
-- **`bun:sqlite`** — built-in storage for the model registry and eval DB.
-- **One binary, one toolchain** — runtime, package manager, test runner.
-
-## CLI
-
-Commands are shown as `bun src/cli.ts <verb>` (runs straight from a
-clone). Once installed/linked, `mlx-bun <verb>` is the identical command.
-
 ```sh
-bun src/cli.ts get mlx-community/gemma-4-12B-it-OptiQ-4bit   # resumable, verified download
-bun src/cli.ts scan                 # index your HF cache into the registry
-bun src/cli.ts ls                   # list models (size, params, quant, capabilities)
-bun src/cli.ts ls --vision --max-size 10GB
-bun src/cli.ts fit gemma --ctx 32768          # memory contract: fits? max context? predicted tok/s
-bun src/cli.ts fit gemma --ctx 8192 --skus    # ...same, across the Apple Silicon lineup
-bun src/cli.ts fit GLM-5.2                    # streamed plan: disk vs resident slabs + measured speed
-bun src/cli.ts serve gemma --port 8080        # OpenAI-compatible server
-bun src/cli.ts serve gemma --memory-budget 18 # ...with admission control (GB)
-bun src/cli.ts generate gemma "a haiku about metal shaders"   # raw one-shot generation
-bun src/cli.ts train e4b --data ./pref-data --method orpo     # LoRA fine-tune (SFT/DPO/ORPO)
-bun src/cli.ts memory init          # git-tracked Markdown wiki at ~/.mlx-bun/wiki
-bun src/cli.ts memory synthesize    # conversations -> cross-linked wiki articles, all local
-bun src/cli.ts pi                   # built-in agentic coding CLI (pi's TUI, in-process)
-bun src/cli.ts evals                # recorded benchmark runs
-bun src/cli.ts harness pi           # connect your own pi install to the local server
-./benchmark.sh                      # head-to-head matrix vs mlx-lm/optiq (reboot first;
-                                    #   preflight-gated, resumable, writes benchmarks-h2h-<date>-<host>.md)
+curl http://localhost:8080/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{"messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 128}'
 ```
 
-Model arguments are substring queries against the registry (`e4b`,
-`26B`, `12B-it`); a query matching more than one model errors out
-listing the candidates — just make it more specific.
-
-`fit` is deterministic, not vibes: weights bytes from safetensors
-headers, KV bytes/token from the config (sliding-window layers capped,
-MoE active-expert bytes for decode), calibrated prefill transient, and
-the machine's wired-memory ceiling. Predictions are recorded next to
-measured peaks in the eval DB. The direct Colibri GLM-5.2 path uses its exact
-streamed plan instead: full artifact-on-disk size is separated from bounded
-resident weights, main/MTP expert slabs, KV, and reserves, and measured warm
-decode is separated from the direct oracle and the 2 tok/s aspiration.
-
-Every verb, with flags: [docs/reference/cli.md](./docs/reference/cli.md);
-on the site: [CLI reference](https://mlx-bun.dev/reference/cli/).
-
-## HTTP API
-
-`serve` exposes an OpenAI-compatible API. Anything that speaks the
-OpenAI protocol can point at it — the OpenAI SDK
-(`baseURL: "http://localhost:8080/v1"`, any non-empty `apiKey`), or
-agent CLIs like pi/OpenClaw via their provider config.
-
-- **`POST /v1/chat/completions`** — streaming (SSE) and non-streaming;
-  `temperature`, `top_p`, `top_k`, `max_tokens`, `seed`,
-  `repetition_penalty`, `stop` (string or array, matched on decoded
-  text with streaming hold-back), `reasoning_effort` (thinking on/off for
-  Qwen3.5/3.8 / MiniCPM5 — `none` disables, any level enables; on Qwen3.8
-  the level also sets reasoning depth: `low`/`medium`/`xhigh`), a Qwen3.8
-  `chat_template_kwargs.preserve_thinking` toggle (keep think blocks from
-  history; default on), and an `hlg` tone-curve sampling override;
-  omitted sampling fields default to the model's own
-  `generation_config.json` recipe; usage includes `cached_tokens`.
-  Full schemas in
-  [docs/reference/server-api.md](./docs/reference/server-api.md); start
-  flags and the `--batch N` compatibility matrix in
-  [docs/reference/server-config.md](./docs/reference/server-config.md).
-- **Tool calling** — pass OpenAI `tools`; each family's native format
-  is parsed into `tool_calls` JSON with `finish_reason: "tool_calls"`
-  (Gemma 4 `<|tool_call>` sentinel tokens; MiniCPM5 / Qwen3.5
-  `<function name=…>` XML with schema-aware argument decoding);
-  `role: "tool"` round-trips, including multi-turn agent loops.
-  The built-in `web_fetch` tool accepts public HTTP(S) destinations only,
-  revalidates DNS and every redirect, and streams into a 1 MiB response cap.
-- **Structured output** — `response_format` (JSON mode / JSON schema),
-  plus the vLLM/oMLX aliases `guided_grammar`, `guided_regex`,
-  `guided_choice`, and `structured_outputs`, enforced by
-  grammar-constrained decoding (xgrammar) on chat and raw completions,
-  serial and batched lanes alike; a grammar that fails to compile
-  degrades to a string-valued constraint-specific prompt injection
-  (oMLX parity) instead of an error
-  (`MLX_BUN_GRAMMAR=0` disables mask compilation server-wide and routes
-  requested constraints through that same warning-bearing degrade path).
-- **Vision** — `image_url` content parts (data: URLs or http/s), on
-  models with the vision sidecar. PNG, JPEG, HEIC, AVIF, WebP, TIFF,
-  GIF, BMP via native OS codecs. Remote fetches are SSRF-guarded by
-  default (private/loopback/LAN destinations refused); opt out with
-  `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1`.
-- **Audio input** — `input_audio` content parts (base64 WAV, plus
-  `audio` / `audio_url` aliases), on models whose sidecar ships the
-  audio tower (gemma-4 e4b). WAV decoded natively; mp3, m4a, FLAC, ogg,
-  AIFF via CoreAudio. Up to 30 s per clip; images and audio mix in one
-  message — a capability neither mlx-lm nor optiq serves. Same SSRF
-  guard and opt-out as `image_url` applies to `audio_url`.
-- **Prompt caching** — a byte-capped LRU KV cache reuses the longest
-  common token prefix across requests (multi-turn conversations
-  re-prefill only the new turn) — automatic, no client changes.
-  Entries are adapter-specific when LoRA adapters are in play.
-- **LoRA hot-swap** — mount adapters at runtime (`POST /v1/adapters`)
-  and discover on-disk ones (`GET /v1/adapters/available`), select per
-  request with the `adapter` body field (`"id"`, stacked `"a+b"`, or
-  `"none"`). The base model never reloads; an unselected adapter costs
-  nothing.
-- **Mixed-precision KV** — opt in with `--kv-quant config` when the model
-  repo ships `kv_config.json` (every Gemma-4 OptiQ repo does): per-layer
-  KV quantization with bits set per layer for both full-attention and
-  sliding-window (rotating) layers (Phase 9). It trades decode speed
-  (measured 5–20% at ≤16k, on mlx-lm's uniform scheme too) for KV bytes
-  ÷4 on the quantized layers — reach for it when context, not speed, is
-  the constraint. The default KV cache is bf16 (the mlx-lm-parity L1
-  route); `--kv-quant 4|8` selects uniform bits instead, and
-  `--kv-quant turbo[:kXvY]` opts into TurboQuant rotation-based KV
-  (affine keys + FWHT/Lloyd-Max values, full-attention layers, serial
-  lane — [design](./docs/design/turboquant-kv.md)). Long prefills
-  over quantized caches run a fused FlashAttention-2 tiling that never
-  materializes the full scores matrix (bounded transient;
-  `MLX_BUN_NO_FUSED_SDPA=1` to disable).
-- **Speculative decoding** — `serve --draft-model <path|query>`
-  (mlx_lm.server parity; `--num-draft-tokens`): the drafter KIND is
-  auto-detected from the artifact (`--draft-kind` overrides) — a full
-  smaller same-tokenizer model (mlx-lm parity), a Gemma `-assistant`
-  KV-borrowing drafter (optiq parity), a DeepSpec/DSpark
-  hidden-tapping drafter (e.g. DeepSeek's released
-  `dspark_gemma4_12b_block7`), or **model-free prompt-lookup drafting**
-  (`--draft-kind ngram`, no draft artifact at all — drafts are copied
-  from the request's own prompt/generation; free wins on echo-heavy
-  workloads like extraction and code edits, lossless at any
-  temperature). The main model verifies every draft —
-  exact results, faster decode when drafts land; pays on 12B-class
-  targets, not small fast models. Serial lane only — with `--batch N` a
-  mounted draft routes every request serial, like mlx_lm.server.
-- **Memory admission control** — `--memory-budget <GB>` refuses to load
-  a model that can't serve within the budget. Generic budgeted serving rejects
-  requests whose `prompt + max_tokens` exceed the safe context with a 400
-  (`type: "memory_admission"`) *before* generating. GLM-5.2 has a fixed,
-  pre-planned context instead: broad client `max_tokens` values are capped to
-  the remaining context, and only a prompt that leaves no generation slot is
-  rejected. The GPU
-  out-of-memory crash it prevents is uncatchable by design — rejection
-  up front is the only defense. Ceiling visible at `GET /stats`.
-- **Anthropic Messages API** (`POST /v1/messages`, on by default) —
-  point any Anthropic-SDK tool at the server
-  (`ANTHROPIC_BASE_URL=http://localhost:8080`); Claude Code runs
-  against it as a fully local backend. Streaming event grammar, native
-  tool_use/tool_result mapping, image blocks via the vision path.
-- **OpenAI Responses API** (`POST /v1/responses`) — the protocol
-  Codex/Cursor/Continue speak; includes `previous_response_id`
-  resumption backed by a TTL + byte-capped store (visible in /stats).
-- **OpenAI Embeddings API** (`POST /v1/embeddings`) — when the served
-  model is an embedding model (`mlx-bun serve Qwen3-Embedding-4B-4bit-DWQ`):
-  last-token pooled, L2-normalized vectors, bit-exact vs mlx-lm. Also a
-  one-shot CLI: `mlx-bun embed <model> --text "…"`.
-- **`GET /v1/models`**, **`GET /stats`** (cache hit rates, bytes,
-  active KV scheme, response store), **`GET/POST/DELETE /v1/adapters`**,
-  plus **`GET /library`**, **`GET /fit`**, **`GET /downloads`**, and
-  **`GET /v1`** (a self-describing index of every endpoint).
-
-Guide with examples on the site:
-[The HTTP API](https://mlx-bun.dev/guides/http-api/); coming from
-`mlx_lm.server`:
-[Drop-in for mlx-lm](https://mlx-bun.dev/guides/drop-in-mlx-lm/).
-
-## Library
-
-The server is one consumer of a library-first API. Published to npm as
-`mlx-bun` — import from the package (`from "mlx-bun"`)
-or `./src/index` in a clone; `bunx mlx-bun` runs the CLI without
-installing. Full reference:
-[docs/reference/library-api.md](./docs/reference/library-api.md). For shipping inside a
-Mac app (Tauri/Electron sidecar), `./scripts/build-binary.sh` produces
-a relocatable single-binary bundle — recipe incl. signing/notarization
-in [docs/reference/embedding.md](./docs/reference/embedding.md).
-
 ```ts
-import {
-  openModel,          // artifact-aware, incl. bounded streamed GLM-5.2
-  loadTokenizer, ChatTemplate, generate,
-} from "mlx-bun";     // or "./src/index" in a clone
-
-const dir = "/path/to/hf/snapshot";
-const model = await openModel(dir);  // RuntimeModel
-const tok = await loadTokenizer(dir);
-const template = await ChatTemplate.load(dir);
-
-const ids = tok.encode(template.render([{ role: "user", content: "Hi!" }]));
-const gen = generate(model, ids, {
-  maxTokens: 256,
-  temperature: 0.7, topP: 0.95, seed: 42,   // reproducible sampling
-  kvBits: 8, quantizedKvStart: 4096,        // optional: quantized KV past 4k
+import OpenAI from "openai";
+const client = new OpenAI({ baseURL: "http://localhost:8080/v1", apiKey: "local" });
+const res = await client.chat.completions.create({
+  model: "local",
+  messages: [{ role: "user", content: "Hello!" }],
 });
-for await (const t of gen) process.stdout.write(tok.decode([t.token], true));
-console.log(gen.stats); // prompt/decode tok/s, cached tokens, ...
 ```
 
-KV caches can be persisted to disk (page-aligned files that reload as
-zero-copy GPU-safe mmaps — see `src/kv-store.ts`) so a standard agent
-preamble prefills once, ever.
-
-On the site: [Using the library](https://mlx-bun.dev/guides/library/) and
-[Embedding in a Mac app](https://mlx-bun.dev/guides/embedding/).
-
-## Benchmarks
-
-Head-to-head against the Python stacks (mlx-lm 0.31.3, mlx-optiq 0.2.1),
-same machine (M4 Pro 24 GB), same HF snapshots, the 2026-06-14
-cleared-machine run, preflight-gated, median-of-N with warmups
-discarded. Full curated table (parity / performance / quality) with
-per-row provenance: [benchmarks/RESULTS.md](./benchmarks/RESULTS.md).
-
-**Served (warm)** — how agents actually use a local model.
-decode tok/s · TTFT ms · start s:
-
-| model | mlx-bun | mlx-lm | optiq |
-|---|---|---|---|
-| MiniCPM5-1B | **252.9** · 34 · 0.17 | — | 223.6 · 64 · 0.84 |
-| gemma-4-e4b | **55.7** · 44 · 0.36 | 53.5 · 218 · 0.98 | 53.4 · 221 · 0.78 |
-| gemma-4-12B | **25.9** · 85 · 0.38 | — | 25.5 · 326 · 1.24 |
-| gemma-4-26B | **54.2** · 45 · 0.47 | 52.3 · 228 · 0.77 | — |
-
-Across every served model mlx-bun has the fastest decode and the fastest
-TTFT/startup (2–5×), at **~0% server tax** vs its own direct engine —
-mlx-lm pays −5–6%, optiq −1 to −10%.
-
-**Direct (engine only).** Decode is at parity-to-slightly-behind mlx-lm
-(e4b +1.1%, 12B +0.4%, 26B −1.1%, MiniCPM5 −0.9% — within the per-step
-host-overhead residual), and prefill leads on the larger models (12B
-1.2×, 26B 1.1×, MiniCPM5 2.3×; e4b trails at 0.8×). The earlier decode
-deficit was root-caused and fixed (2026-06-11) — a prefill→decode
-allocator-reclaim stall mlx-lm clears with `mx.clear_cache` and bills to
-prompt time; see PLAN.md "Decode gap RESOLVED".
-
-**Long context (12B).** At 16k and 64k mlx-bun holds decode parity with
-mlx-lm on bf16 (16k 23.9 = 23.9; 64k 20.9 = 20.9) while optiq drops to
-21.6 then collapses to 12.3 at 64k; mixed-KV trades ~2 tok/s for ~5 GB
-lower peak.
-
-Two cross-stack served cells are absent: no mlx-lm 12B row, and optiq
-produced no output on the 26B (the Metal-OOM class from python's
-non-lazy load transient — reproduced in isolation; mlx-bun and mlx-lm
-both served it from the same machine state). One further optiq cell
-(12B/kv=config) is blocked on an upstream `quantized_matmul` bug. Both
-are documented in the results file.
-
-On the site (with the honest negatives spelled out):
-[Benchmarks](https://mlx-bun.dev/about/benchmarks/).
-
-## Correctness
-
-Logit parity with mlx-lm (same weights, Python reference) is the
-project's oracle. The test suite holds the forward pass **bit-exact**
-against it — including every quantized-KV configuration (kv8, kv4, and
-the 26B's mixed per-layer scheme) and the fused quantized-attention
-prefill, which is bit-exact against optiq's reference implementation.
-Every ported helper follows the reference implementation's exact op
-composition, down to constants built at load time (the one latent
-divergence ever found — rope frequencies computed host-side instead of
-on-device — was root-caused and fixed in Phase 10; see PLAN.md
-findings). Golden files are regenerated only by explicit scripts
-(`scripts/regen-*.ts`) running the Python oracle.
-
-```sh
-bun test    # fast tier runs everywhere; model-loaded tests run only
-            # when the reference snapshot is in your HF cache
-```
-
-On the site: [Correctness](https://mlx-bun.dev/about/correctness/) and
-[How it compares](https://mlx-bun.dev/about/comparison/).
-
-## Troubleshooting
-
-- **`dlopen` / `libmlxc.dylib` not found** — the MLX runtime
-  auto-downloads on first `serve` into `~/Library/Caches/mlx-bun/`; if a
-  download was interrupted, just rerun `serve` (it resumes). To point at
-  your own copy instead, set `MLX_BUN_LIBMLXC=/path/to/libmlxc.dylib`.
-- **`no models match`** — run `bun src/cli.ts scan` after downloading;
-  models must be in the standard HF cache (`~/.cache/huggingface/hub`).
-- **HF download stalls at 0%** — use `bun src/cli.ts get <org/repo>`
-  (plain HTTPS, no Xet, resumes where it left off); for the Python CLI,
-  `HF_HUB_DISABLE_XET=1` before the download command.
-- **Slow decode on a model near your RAM ceiling** — close memory-heavy
-  apps; a model that doesn't fit the wired-memory budget pages weights
-  every token (check with `fit`).
-- **Bun version** — requires ≥ 1.3.14 (`bun upgrade`); pinned for
-  `Bun.Image` and verified FFI behavior.
-
-## Status
-
-Pre-alpha, moving fast. See [PLAN.md](./PLAN.md) for phases, exit
-criteria, measured numbers, and the findings log.
-
-**Complete** — load path; bit-exact model port (MiniCPM5, Qwen3.5-4B,
-Qwen3.8-27B, e4b per-layer-input, 12B dense, 26B MoE); sampling + serving (tools, vision,
-prompt cache); registry / fit / KV persistence; quantized +
-mixed-precision KV serving (rotating-cache KV-quant, Phase 9) with fused
-quantized prefill (Phase 10); LoRA hot-swap with per-request selection;
-segmented-backward LoRA training (SFT / DPO / ORPO, with an `[M,vocab]`-free
-flash-CCE head + prefix-sharing for long-context preference data — `mlx-bun
-train <model> --data <dir>`, see [training guide](docs/reference/training.md));
-resumable
-verified downloads (`mlx-bun get`); memory admission control; the
-head-to-head benchmark harness; the decode-gap root-cause fix
-(2026-06-11); Anthropic Messages + Responses API (`/v1/messages` and
-`/v1/responses`, Phase 11); SigLIP vision sidecar for e4b
-(commit 4625fe5); the embeddable single-binary build (signed +
-notarized — Homebrew, direct-download, and npm/bunx).
+Longer walkthroughs on the site:
+[Installation](https://mlx-bun.dev/getting-started/installation/) and
+[Quickstart](https://mlx-bun.dev/getting-started/quickstart/).
 
-**In progress** — `Qwen3.8-27B` full support (Phase 14 retarget): text
-parity is bit-exact and thinking controls (`reasoning_effort` depths,
-`preserve_thinking`) + tool calling are wired; native MTP speculation
-(the Qwen-trained head as a `--draft-model`), vision, and video input
-are in bring-up.
+## Reference docs
+
+Each fact about the project has exactly one home:
+
+| Topic | Doc |
+|---|---|
+| Current state, what's next | [STATUS.md](./STATUS.md) |
+| Benchmark numbers (parity / performance / quality) | [docs/reference/benchmarks.md](./docs/reference/benchmarks.md) |
+| Supported models roster | [docs/reference/models.md](./docs/reference/models.md) |
+| Server start flags, `MLX_BUN_*` env, defaults | [docs/reference/server-config.md](./docs/reference/server-config.md) |
+| HTTP API routes and request/response schemas | [docs/reference/server-api.md](./docs/reference/server-api.md) |
+| CLI verbs | [docs/reference/cli.md](./docs/reference/cli.md) |
+| TypeScript library API | [docs/reference/library-api.md](./docs/reference/library-api.md) |
+| LoRA fine-tuning (SFT / DPO / ORPO) | [docs/reference/training.md](./docs/reference/training.md) |
+| Build, sign, notarize, publish | [docs/reference/distribution.md](./docs/reference/distribution.md) |
+| Troubleshooting | [docs/reference/troubleshooting.md](./docs/reference/troubleshooting.md) |
+| Personal memory (local wiki) | [docs/reference/memory.md](./docs/reference/memory.md) |
+| Reference environment / oracle setup | [docs/reference/environment.md](./docs/reference/environment.md) |
+| Active engineering design docs | [docs/design/](./docs/README.md) |
+| Contributing / repo rules | [CONTRIBUTING.md](./CONTRIBUTING.md) |
+
+Correctness is verified bit-exact against the Python mlx-lm reference as the
+project's oracle: [benchmarks.md § Parity](./docs/reference/benchmarks.md#1-parity-porting-correctness--bit-exact-vs-the-oracle).
 
-**Experimental** — opt-in, default-off, still being hardened: transparent
-expert offload for MoE models (`serve --expert-offload`, Phase 20:
-page-aligned mmap-backed experts, bit-exact — 26B-A4B 17.1→4.2 GB
-resident, decode unregressed); batched serving (`--batch N`, default 8
-since 2026-07-05 — a lone request runs the exact serial engine, so the cap
-engages only under real concurrency, e.g. coding sub-agents; `--batch 1`
-pins strict serial; Phase 18: continuous-batching bf16 decode at mlx-lm B=N parity,
-B=2 bit-parity verified for MiniCPM5/12B/e4b/26B; sampler extras,
-repetition penalty, and grammar-constrained requests batch; Qwen3.5's
-SSM caches batch, as do plain full-attention Tier-0 archs (e.g. Llama) —
-gemma2-family / sliding-window universal archs and DiffusionGemma route
-serial).
+## Why
 
-**Open** — e4b's ~5% per-step host-overhead decode residual (Phase 7);
-SigLIP vision for 26B.
+MLX is Apple's ML framework — hand-tuned Metal kernels with official
+bindings for Python, C++, Swift, and C, but no JavaScript story. The
+performance-critical work (every matmul, every attention pass) lives in
+MLX's C++/Metal core, exposed through `mlx-c`; the layer on top — model
+loading, tokenization, sampling, serving — is pure orchestration and
+performance-neutral, so it can move to a better runtime without losing
+speed. Bun fits: `bun:ffi` binds `mlx-c` directly with no node-gyp step,
+`Bun.Image` gives native OS image codecs for vision input, and the result
+ships as one binary with no Python venv.
 
 ## License