mlx-bun 0.0.4 → 0.0.6

package/docs/reference/distribution.md

@@ -54,7 +54,7 @@ and notarized as one artifact.
 This runs [build-binary.sh](../../scripts/build-binary.sh), then:
 - signs every nested Mach-O (dylibs, any `.node`) with the hardened
   runtime, then the executable with the JIT
-  [entitlements](../../scripts/entitlements.plist) (JavaScriptCore JITs —
+  [entitlements](../../packaging/entitlements.plist) (JavaScriptCore JITs —
   without `allow-jit` the binary is killed on launch);
 - notarizes the zipped bundle via `AC_PROFILE` and waits;
 - emits `dist-release/mlx-bun-v<ver>-arm64.tar.gz` (+ `.sha256`) and
@@ -75,19 +75,20 @@ gh release create v0.0.4 dist-release/mlx-bun-v0.0.4-arm64.tar.gz \
 
 ## The tap
 
-Homebrew taps must be a repo named `homebrew-<name>`:
+Homebrew taps must be a repo named `homebrew-<name>`. The canonical tap is
+**`joshuarossi/homebrew-tap`** (Homebrew shorthand: `joshuarossi/tap`).
 
-1. Create a public repo **`joshuarossi/homebrew-mlx-bun`**.
+1. Create a public repo **`joshuarossi/homebrew-tap`** (one-time).
 2. Copy [packaging/homebrew/mlx-bun.rb](../../packaging/homebrew/mlx-bun.rb)
    to `Formula/mlx-bun.rb` in it, updating `version`/`url`/`sha256` from
    the release-script output. Commit and push.
 3. Install:
    ```sh
-   brew install joshuarossi/mlx-bun/mlx-bun
+   brew install joshuarossi/tap/mlx-bun
    mlx-bun --version
    ```
 
-`brew audit --strict --new joshuarossi/mlx-bun/mlx-bun` before pushing
+`brew audit --strict --new joshuarossi/tap/mlx-bun` before pushing
 catches most formula issues; `brew install --build-from-source` +
 `brew test` runs the formula's `test do` locally.
 
@@ -119,3 +120,58 @@ Users then upgrade with:
 ```sh
 brew upgrade joshuarossi/tap/mlx-bun
 ```
+
+## Direct download (stable URL)
+
+`publish-release.sh` uploads two assets per release: the versioned tarball
+(`mlx-bun-v<ver>-arm64.tar.gz`) **and** a versionless stable copy
+(`mlx-bun-arm64.tar.gz`) pinned to `releases/latest`. This lets the
+install script target a URL that never changes:
+
+```
+https://github.com/joshuarossi/mlx-bun/releases/latest/download/mlx-bun-arm64.tar.gz
+```
+
+The user-facing install path is the script at
+[`website/public/install.sh`](../../website/public/install.sh) (served at
+`mlx-bun.dev/install.sh`), which resolves that URL, extracts the bundle, and
+links `mlx-bun` onto `PATH`:
+```sh
+curl -fsSL https://mlx-bun.dev/install.sh | sh
+```
+
+Without the script, the same stable URL streams straight to `tar`:
+```sh
+curl -fsSL https://github.com/joshuarossi/mlx-bun/releases/latest/download/mlx-bun-arm64.tar.gz \
+  | tar -xz -C /usr/local/bin mlx-bun
+```
+
+Note: **browser downloads and `curl`-to-file** (e.g. `curl -O ...`) are
+quarantined by Gatekeeper — macOS sets the `com.apple.quarantine` xattr on
+the saved file. The piped one-liner above streams directly to `tar` and
+never writes a quarantined file, so it runs without any Gatekeeper prompt.
+For the browser/curl-to-file path, the notarization ticket resolves the
+Gatekeeper check online; `xattr -d com.apple.quarantine mlx-bun` is the
+manual escape hatch if offline.
+
+## npm / bunx
+
+mlx-bun is published to npm as `mlx-bun` (current: 0.0.4). Install or
+run without a permanent install:
+
+```sh
+# one-off (no install)
+bunx mlx-bun
+
+# permanent global install
+bun install -g mlx-bun
+mlx-bun --version
+```
+
+The npm package wraps the same signed binary via a postinstall download
+step (it does not bundle the ~228 MB dylibs/metallib in the npm tarball
+itself). Publish to npm as part of a release:
+
+```sh
+bun publish   # from repo root; package.json version must already be bumped
+```

package/docs/reference/library-api.md

@@ -2,8 +2,9 @@
 
 The HTTP server is one consumer of a library-first API. Everything it
 does — generation, sampling, KV schemes, LoRA, vision, caching, memory
-admission — is importable directly into a Bun process. Not yet
-published to npm; import from `src/`.
+admission — is importable directly into a Bun process. Published as
+`mlx-bun` on npm (0.0.4); `bunx mlx-bun` works out of the box, or
+`import … from "mlx-bun"` for programmatic use.
 
 One rule everywhere: **one generation at a time per process** (one GPU,
 batch=1). The server serializes through a queue; do the same.
@@ -11,17 +12,24 @@ batch=1). The server serializes through a queue; do the same.
 ## Quick start
 
 ```ts
-import { loadModelConfig } from "./src/config";
-import { Weights } from "./src/weights";
-import { Gemma4Model } from "./src/model/gemma4";
-import { loadTokenizer } from "./src/tokenizer";
-import { ChatTemplate } from "./src/chat-template";
-import { generate } from "./src/generate";
+import {
+  loadModelConfig,
+  Weights,
+  createModel,       // dispatches to Gemma4Model / MiniCPM5Model / Qwen35Model
+  loadTokenizer,
+  ChatTemplate,
+  generate,
+} from "mlx-bun";   // or "./src/index" in-repo
 
 const dir = "/path/to/hf-snapshot";          // mlx-bun ls prints these
-const model = new Gemma4Model(await Weights.open(dir), await loadModelConfig(dir));
+const config = await loadModelConfig(dir);
+const model = createModel(await Weights.open(dir), config);   // returns RuntimeModel
 const tok = await loadTokenizer(dir);
 const template = await ChatTemplate.load(dir);
+// Gemma4Model and MiniCPM5Model can still be imported directly from "mlx-bun"
+// if you need the concrete type. Qwen35Model is NOT exported from the public
+// package — use createModel/RuntimeModel (preferred dispatch), or import from
+// "./src/model/qwen35" in-repo.
 
 const ids = tok.encode(template.render([{ role: "user", content: "hi" }]));
 const promptIds = ids[0] === ids[1] && ids[0] === tok.bosTokenId ? ids.slice(1) : ids;
@@ -67,7 +75,7 @@ prompt time), the decode clock from there.
 ## Serving pieces
 
 ```ts
-import { loadContext, createServer } from "./src/server";
+import { loadContext, createServer } from "mlx-bun";
 
 const ctx = await loadContext(dir, "my-model", { memoryBudgetBytes: 12e9 });
 const server = createServer(ctx, 8090, {
@@ -84,6 +92,12 @@ completions, Anthropic `/v1/messages`, OpenAI Responses, adapters, and
 
 ## Prompt cache (in-process prefix reuse)
 
+> **Note:** `PromptCache`, `AdapterManager`, `saveKvCache`, and
+> `loadKvCache` are **not** re-exported from the public `"mlx-bun"`
+> entry point (`src/index.ts`). Import them from their internal paths
+> as shown below; this is intentional — they are implementation-level
+> pieces rather than stable public API surface.
+
 ```ts
 import { PromptCache } from "./src/prompt-cache";
 const cache = new PromptCache(2e9);                  // BYTE-capped LRU
@@ -102,9 +116,11 @@ under one adapter must not seed another's prefill).
 ```ts
 import { saveKvCache, loadKvCache } from "./src/kv-store";
 saveKvCache("/tmp/prefix.kv", tokens, caches);       // page-aligned file
-const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model);
+const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model as Gemma4Model);
 // reload is a zero-copy MAP_PRIVATE mmap straight to the GPU (~1 ms);
 // keep `mmap` referenced as long as the caches live
+// Note: loadKvCache is typed to accept Gemma4Model, but only uses model.layers.length
+// at runtime — a cast from RuntimeModel is safe for any model that has a layers array.
 ```
 
 Quantized caches are not persistable yet (documented gap).
@@ -112,8 +128,7 @@ Quantized caches are not persistable yet (documented gap).
 ## Registry + fit (model discovery and memory math)
 
 ```ts
-import { Registry } from "./src/registry";
-import { fit, skuMatrix } from "./src/fit";
+import { Registry, fit, skuMatrix } from "mlx-bun";
 
 const reg = new Registry();           // bun:sqlite, scans the HF cache
 await reg.scan();
@@ -150,4 +165,4 @@ any amount of switching.
   handler → process death). Use `fit`/admission before generating, not
   try/catch around it.
 - Never read a typed array that native code wrote in a hot path; use
-  the `read.*` helpers (bun#32054 — see PLAN.md Phase 4 findings).
+  the `read.*` helpers (bun#32054 — see PLAN-archive.md Phase 4 findings).

package/docs/reference/memory.md

@@ -0,0 +1,77 @@
+# Memory
+
+mlx-bun memory is a local Markdown wiki that the built-in pi agents can read as durable user context.
+
+Core rule:
+
+> Chat agents read memory articles. `mlx-bun memory synthesize` is the only writer.
+
+The current release includes the vault, read path, Obsidian open flow, and scheduler. Synthesis is scaffolded but still a no-op M1 stub. The planned synthesis engine uses the same local models you chat with, switched into an editorial pipeline with stage-specific LoRA adapters.
+
+## Storage
+
+- Vault: `~/.mlx-bun/wiki/`
+- User articles: `~/.mlx-bun/wiki/articles/*.md`
+- Read-only reference docs: `~/.mlx-bun/wiki/Reference/*.md`
+- Meta pages: `~/.mlx-bun/wiki/Meta/*.md`
+- Talk pages: `~/.mlx-bun/wiki/Talk/`
+- Override: `MLX_BUN_WIKI=/path/to/wiki`
+- Derived synthesis DB: `~/.cache/mlx-bun/memory.sqlite`
+
+The vault is plain Markdown, git-tracked, and opens directly in Obsidian. `Reference/` contains read-only symlinks to mlx-bun's own docs so memory has useful built-in residents on day one; synthesis must never write them.
+
+## CLI
+
+```bash
+mlx-bun memory init              # create the wiki; optional import + schedule
+mlx-bun memory status            # path, article count, git, schedule, recent articles
+mlx-bun memory open              # open wiki in Obsidian/Finder
+mlx-bun memory open <article>    # open a specific article
+mlx-bun memory list              # list article stems + Reference/* docs
+mlx-bun memory search <query>    # search articles
+mlx-bun memory toc <article>     # headings + anchors
+mlx-bun memory section <article> <anchor>
+mlx-bun memory links <article>   # outbound/inbound wikilinks
+mlx-bun memory read <article>    # print article body
+mlx-bun memory synthesize --dry-run
+mlx-bun memory schedule --at 03:00
+mlx-bun memory unschedule
+```
+
+`memory synthesize` is currently safe and side-effect-free: it reports that M1 stage bodies are not implemented and writes nothing. In the full flow, `memory init` will be able to offer the HF-hosted memory adapter pack download so synthesis can run locally.
+
+## Agent tools
+
+Both `mlx-bun pi` and the web chat expose these read-only tools:
+
+- `memory_search`
+- `memory_read`
+- `memory_toc`
+- `memory_section`
+- `memory_links`
+- `memory_list` — user articles plus read-only `Reference/*` docs
+- `memory_status` — vault path, setup state, article count, git state, last-synthesis placeholder, schedule state, recent changed articles
+
+The tools are always registered. If no vault exists, they return a setup message instead of disappearing. Search/read results prefixed with `Reference/` are mlx-bun docs, not user memory articles.
+
+## Obsidian
+
+```bash
+mlx-bun memory open
+mlx-bun memory open Archie_Project
+```
+
+The first command opens the vault. The second opens one article. Obsidian is preferred; Finder/default Markdown app is the fallback.
+
+## Synthesis roadmap
+
+M1/M2 will turn saved pi sessions into articles using the local model plus four specialized LoRA adapters trained by Josh from existing Lucien-scale pipeline data and distributed from Hugging Face:
+
+1. deterministic ingest of saved web/terminal pi session JSONL;
+2. **memory-chunk LoRA** — conversation → topic chunks (already trained as a WIP artifact);
+3. **memory-bucket LoRA** — chunk → existing bucket or new article proposal;
+4. **memory-synthesis LoRA** — chunks + existing article → conservative article update candidate;
+5. **memory-editor LoRA** — candidate article → polished wiki Markdown;
+6. deterministic footnote/wikilink normalization, quality gates, and git commit.
+
+The adapters propose; the pipeline validates and writes. The pipeline writes only `articles/`; `Reference/` is immutable context. Until synthesis lands, memory works with imported/hand-written user articles plus the built-in read-only mlx-bun reference docs.

package/docs/reference/orpo-quickstart.md

@@ -0,0 +1,130 @@
+# ORPO training quickstart — the full stack, preconfigured
+
+One launcher runs the complete new system — **flash-CCE Metal head** (steel fwd+bwd,
+`[M,V]`-free) + **segmented backward** (gradient-checkpointed layer activations) +
+**prefix-sharing** (one forward over `[prompt; chosen; rejected]`) + LoRA. Each feature
+**falls back and logs** if a precondition isn't met (e.g. a row whose chosen/rejected
+prompts aren't identical → two-forward for that row).
+
+## Command — `mlx-bun train` (recommended)
+
+```bash
+mlx-bun train <model> --data <dir>
+```
+- `<model>` is a query or snapshot path (e.g. `e4b`, `MiniCPM5`); omit it to auto-pick.
+- `--data <dir>` is a directory with `train.jsonl` (+ optional `valid.jsonl`); rows are
+  `{"prompt": "...", "chosen": "...", "rejected": "..."}`.
+- The command **auto-detects e4b/Gemma** and sets its required training env flags
+  (`MLX_BUN_PERF_KERNEL=0`, `MLX_BUN_FUSED_GELU=0`) before loading the model — you don't
+  set them yourself. The full stack (flash head + prefix-share + segmented) is on by default.
+- It prints a plan box (dataset counts, format, the resolved knobs), streams per-step loss
+  + peak memory, and saves a **mountable** adapter (`mlx-bun serve <model> --adapter <dir>`).
+- `--dry-run` inspects the dataset and prints the plan without training. `mlx-bun help train`
+  lists every flag.
+
+```bash
+# e4b ORPO on your preference data, checkpointing every 200 steps; run it detached
+nohup mlx-bun train e4b --data ./prefs --iters 1000 --save-every 200 > train.log 2>&1 &
+# resume from a checkpoint (warm-start the LoRA weights)
+mlx-bun train e4b --data ./prefs --resume ~/.cache/mlx-bun/mlx-bun-finetunes/orpo-e4b/checkpoints/step-00200
+```
+
+| flag | default | meaning |
+|---|---|---|
+| `--method` | `orpo` | `sft` · `dpo` · `orpo` |
+| `--iters` | 100 | iterations |
+| `--lr` | orpo 1e-5 · dpo 5e-5 · sft 2e-4 | learning rate (cosine + short warmup) |
+| `--rank` / `--scale` | orpo 16 / 2.0 | LoRA rank (by_bits scaled) / scale |
+| `--seq` | e4b 8192 · cpm 4096 | max sequence length |
+| `--lambda` | 0.1 | ORPO odds-ratio weight |
+| `--seg` | 2 | layers/segment (segmented backward); `--no-segment` to disable |
+| `--save-every` | off | mountable checkpoint every N steps |
+| `--resume <dir>` | — | warm-start LoRA weights from a checkpoint/adapter |
+| `--no-flash` / `--no-prefix` | (on) | drop the flash Metal head / prefix-sharing |
+| `--adapter <dir>` | `~/.cache/mlx-bun/mlx-bun-finetunes/<method>-<model>` | output dir |
+
+> **Long runs:** launch detached from your own shell (`nohup … &`). Foreground is fine for
+> short runs; the command blocks and streams until done.
+
+## The launcher (env-var alternative)
+
+`scripts/train-orpo.ts` is the same stack driven by env vars — handy for scripting:
+
+```bash
+MODEL=/path/to/snapshot DATA=/path/to/datadir bun scripts/train-orpo.ts
+```
+- `DATA` is a directory with `train.jsonl` (+ optional `valid.jsonl`).
+- Auto-detects e4b/Gemma and sets the training env flags, same as the CLI.
+
+### Preconfigured defaults (override via env)
+| env | default | meaning |
+|---|---|---|
+| `SEQ` | e4b 8192 · cpm 4096 | max sequence length |
+| `SEG` | 2 | segment size (layers/segment); the validated e4b@8192 sweet spot |
+| `ITERS` | 100 | iterations |
+| `LR` | 1e-5 | learning rate (cosine + short warmup) |
+| `RANK` / `SCALE` | 16 / 2.0 | LoRA rank (by_bits scaled) / scale |
+| `LAMBDA` | 0.1 | ORPO odds-ratio weight |
+| `FLASH=0` | (on) | use the MLX fused head instead of the flash Metal head |
+| `PREFIX=0` | (on) | disable prefix-sharing (plain two-forward) |
+| `SEGOFF=1` | (off) | disable the segmented backward (more memory) |
+| `ADAPTER` | `~/.cache/mlx-bun/mlx-bun-finetunes/orpo-<model>` | output dir (the finetunes cache, not the repo) |
+
+Example — e4b overnight at 8192:
+```bash
+MODEL=~/.cache/huggingface/hub/models--mlx-community--gemma-4-e4b-it-OptiQ-4bit/snapshots/<snap> \
+DATA=./data/mydpo ITERS=200 bun scripts/train-orpo.ts
+```
+
+## What to expect — performance (measured, M1 Max 32 GB)
+
+### e4b (Gemma4, H=2560, V=262144) — full stack
+| seq | config | peak GB | s/step |
+|---|---|---|---|
+| 8192 | flash + segmented(2) + **prefix-share** (prompt-dominant data) | **13.3** | **~70** |
+| 8192 | flash + segmented(2), no prefix | 16.1 | ~175 |
+| 4096 | flash + segmented(2), no prefix | 11.7 | ~81 |
+| 2048 | flash + segmented(2), no prefix | 10.2 | ~40 |
+| 1024 | flash + segmented(2), no prefix | 8.7 | ~21 |
+
+- **Fits comfortably in 32 GB at 8192** (the historical "e4b OOMs ≥2048" ceiling is gone).
+- **Prefix-sharing is a real win when the prompt dominates** — at 8192 with a long shared
+  prompt it cut memory 16→13 GB *and* step time 175→70 s (prompt encoded once, not twice).
+  On response-dominant data the saving shrinks toward zero (and it falls back per-row when
+  prompts differ); it never hurts correctness.
+- Memory is **linear in sequence length** (~+1.5 GB / 1024 tokens) — segmenting + flash
+  attention keep it linear, not quadratic. The head itself is **flat 0.93 GB** regardless
+  of vocab/seq.
+- An ~8 h run at 8192 ≈ 100–400 steps depending on the prompt/response split.
+
+### MiniCPM5 (1B, H=1536, V=130560) — full stack
+- Fits easily (~1–2 GB at short seq). ~0.4–1 s/step at seq 256; scales with seq.
+- Everything composes (flash / segmented / prefix-share, in any combination).
+
+### The flash-CCE head in isolation (the piece that made large-vocab affordable)
+- Forward 180 ms, backward **754 ms** (e4b) / 224 ms (cpm) — exact (dh 0.40% / 0.28%
+  bf16-class), `[M,V]`-free, peak **0.93 GB flat at M=8192**. The backward is **5× the old
+  hand-rolled kernel** (3687 ms) via the verbatim MLX steel GEMM + fused dequant.
+
+## Fallback behavior (logged)
+- **Prefix-share, per row**: if `chosen`/`rejected` prompts aren't byte-identical →
+  two-forward for that row. The first occurrence logs `prefix-share: row prompt mismatch
+  → two-forward fallback`; the saving is lost only on mismatched rows.
+- **`FLASH=0`** → the MLX fused linear-CE head (`quantizedMatmul` both ways; exact,
+  `[chunk,V]` transient) instead of the Metal kernel.
+- **`SEGOFF=1`** → all layer activations resident (only viable at short seq on e4b).
+- Models other than MiniCPM5 / Gemma4 with `orpoPrefixShared` error out (not silently).
+
+## Optional / experimental
+- The Apple-CCE backward skips (`MLX_BUN_CCE_BWD_BLOCK_EPS=1e-5`, the lossless vocab-block
+  skip; `MLX_BUN_CCE_BWD_FILTER_EPS=1e-5`, the coeff filter) are **off by default** — on
+  the now-fast steel kernel they give little and the coeff filter costs gradient accuracy.
+  The block skip *can* help on long genuinely-peaked real text; A/B it on your data and
+  watch that the loss doesn't move.
+
+## Caveats (honest)
+- The e4b numbers above are from **short probe runs** (2–3 steps); they confirm the
+  pipeline runs, fits memory, and produces finite/decreasing loss — **not** training
+  quality (that's your data + hyperparameters).
+- e4b gradients are **bf16-class** (~1–3% vs an exact reference) — expected for the fused
+  bf16 path; MiniCPM5 is tighter (forward bit-exact, grads ~1%).

package/docs/reference/server-api.md

@@ -30,7 +30,18 @@ Request body (OpenAI chat schema; unknown fields ignored):
   "tools": [ /* OpenAI function tools */ ],
   "tool_choice": "auto",         // "none" disables tools
   "chat_template_kwargs": {      // forwarded to the chat template
-    "enable_thinking": false     // MiniCPM5: <think> channel on/off
+    "enable_thinking": false     // MiniCPM5 / Qwen3.5: <think> channel on/off
+  },
+  "reasoning_effort": "medium",  // "none"|"minimal"|"low"|"medium"|"high"
+                                 // gates enable_thinking on Qwen3.5/MiniCPM5:
+                                 // "none" → off, any other level → on.
+                                 // Overrides chat_template_kwargs.enable_thinking.
+  "hlg": {                       // HLG tone-curve sampling (per request).
+    "enabled": true,             // merged over --hlg-sampling server defaults.
+    "width": 4,                  // logit-width of the tone plateau
+    "shoulder": 4,               // top shoulder width
+    "toe": 6,                    // bottom toe width
+    "pivot_offset": 6            // pivot point offset from top
   },
   "adapter": "id"                // LoRA: "id", stacked "a+b", or "none"
 }
@@ -92,12 +103,15 @@ messages; multi-turn prompt prefixes reuse the KV prompt cache
 automatically.
 
 Tool-call parsing is per model family. Gemma 4 uses its native
-`<|tool_call>`…`<tool_call|>` sentinel tokens. MiniCPM5 emits XML in
-decoded text (`<function name="…"><param name="…">…`, CDATA-wrapped
-values supported); content before the tool markup still streams live,
-only the markup is withheld and converted to `tool_calls`. Argument
-values are decoded against the tool's JSON schema (string-typed params
-stay strings); markup that fails to parse falls back to plain content.
+`<|tool_call>`…`<tool_call|>` sentinel tokens. Qwen3.5 emits decoded
+text wrapped in `<tool_call><function=name><parameter=key>value` XML
+equals-style blocks. MiniCPM5 emits XML in decoded text
+(`<function name="…"><param name="…">…`, CDATA-wrapped values
+supported). For both Qwen3.5 and MiniCPM5 content before the tool
+markup still streams live, only the markup is withheld and converted to
+`tool_calls`. Argument values are decoded against the tool's JSON schema
+(string-typed params stay strings); markup that fails to parse falls
+back to plain content.
 
 ### Errors
 
@@ -191,6 +205,90 @@ OpenAI SDK speak this now). Oracle: optiq responses shim.
 }
 ```
 
+## GET /library
+
+Returns all models found in the local HuggingFace hub cache (via the
+registry scan), each annotated with a fit assessment for this machine.
+Response is cached for 30 seconds (registry scan + config reads; no
+tensor bytes are read).
+
+```jsonc
+{
+  "models": [{
+    "repo_id": "…",
+    "model_type": "gemma3" | "minicpm5" | "qwen3" | …,
+    "size_bytes": 0,
+    "quant_bits": 4,
+    "vision": false,             // has vision sidecar
+    "supported": true,           // recognized model family
+    "serving": false,            // currently loaded in this server
+    "assessment": {              // null if config unreadable
+      "fits": true,
+      "max_safe_context": 8192,
+      "predicted_decode_tps": 0.0
+    }
+  }]
+}
+```
+
+## GET /downloads
+
+Snapshot of the last 5 model downloads (active, completed, or errored)
+initiated via `mlx-bun download` or the web library panel.
+
+```jsonc
+{
+  "downloads": [{
+    "repo_id": "…",
+    "state": "active" | "done" | "error",
+    "current_file": "model.safetensors" | null,
+    "received_bytes": 0,
+    "total_bytes": 0,
+    "files_done": 0,
+    "files_total": 0,
+    "bytes_per_sec": 0,          // rolling ~5 s window
+    "started_at": 1760000000,
+    "finished_at": null,
+    "error": "…"                 // present on state "error"
+  }]
+}
+```
+
+## GET /fit
+
+Fit assessment for the loaded model on this machine, plus a capability
+matrix across Apple SKUs. Used by the status page. Experts bytes come
+from the registry so MoE active-parameter predictions match the serve
+banner and `mlx-bun fit`. When the eval DB has a real measured decode
+rate for this model snapshot, it is included and takes precedence over
+the prediction.
+
+```jsonc
+{
+  "machine": { "chip": "M4 Pro", "ram_bytes": 0, "bandwidth_gbs": 0.0 },
+  "context_tokens": 8192,          // current admission ceiling
+  "typical_context_tokens": 8192,  // min(8192, context_tokens)
+  "typical_decode_tps": 0.0,       // predicted at typical_context_tokens
+  "measured_decode_tps": null,     // real number from eval DB, or null
+  "measured_at": null,             // unix ms of measurement, or null
+  "report": {
+    "fits": true,
+    "weights_bytes": 0,
+    "kv_bytes": 0,
+    "transient_bytes": 0,
+    "total_bytes": 0,
+    "usable_bytes": 0,
+    "max_safe_context": 8192,
+    "predicted_decode_tps": 0.0
+  },
+  "sku_matrix_ctx": 32768,
+  "sku_matrix": [{
+    "sku": "M4 Pro 24 GB", "ram_gb": 24,
+    "fits": true, "max_context": 32768, "decode_tps": 0.0
+  }]
+}
+```
+
 ## Adapters (LoRA hot-swap)
 
 - `GET /v1/adapters` — `{ adapters: [{ id, path, rank, scale, size_bytes, mounted_layers }] }`

package/docs/reference/server-config.md

@@ -25,11 +25,18 @@ the design rationale is in
 | `--prompt-cache` | GB | `2` (≈2e9 bytes) | Byte cap for the prompt (prefix-reuse KV) cache. **Binary GiB (×2³⁰)** on input. `0`/unset ⇒ default; there is no flag to disable the cache outright. |
 | `--batch` | n | `1` (serial) | Max concurrent requests batched through the mlx-lm-parity engine. `>1` switches the **whole server** into bf16 continuous batching — a *mode*, not a load fallback. See [Execution modes](#execution-modes-serial-vs---batch-n). Alias: `--decode-concurrency` (mlx_lm.server drop-in). |
 | `--kv-quant` | `config`\|`off`\|`4`\|`8` | `config` serial / `off`(bf16) under `--batch N` | KV-cache quantization. `config` = per-layer `kv_config.json` (optiq parity); `off` = bf16; `4`/`8` = uniform bits (group 64, start 0). Under `--batch N`, an explicit value routes those requests to the **serial** lane (batched is bf16-only). |
-| `--thinking` | `true`\|`false` | model's own (false for CPM) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM hybrid reasoning). A request's `chat_template_kwargs.enable_thinking` overrides it. |
+| `--thinking` | `true`\|`false` | model's own (false for CPM) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM and Qwen3.5 hybrid reasoning). A request's `chat_template_kwargs.enable_thinking` overrides it. |
 | `--temperature` | n ∈ [0,5] | `generation_config.json` | Server-wide sampling default. Per-request `temperature` still wins; the browser chat (sends none) inherits this. |
 | `--top-p` | n ∈ [0,1] | `generation_config.json` | Server-wide top-p default (per-request `top_p` wins). |
 | `--top-k` | n ∈ [0,1e6] | `generation_config.json` | Server-wide top-k default (per-request `top_k` wins). |
-| `--compiled-decode` | on\|off | on | Replay the per-step decode graph in C++ (`MLX_BUN_COMPILED_DECODE`). Bit-exact A/B lever. **Serial lane only** (see note below). |
+| `--no-open` | (bool) | off | Skip the automatic browser open on start. By default an interactive terminal session opens `http://<host>:<port>/#/chat` once the server is ready; pass this flag to suppress it (e.g. headless or non-TTY environments already skip it). |
+| `--hlg-sampling` | `on`\|`off` | off | Piecewise tone-curve (HLG) sampling: rolls off the top-token region, boosts the mids, gentles the tail. The overall gain folds from `--temperature`. See [docs/design/hlg-sampling.md](../design/hlg-sampling.md). |
+| `--hlg-width` | nats | `4` | HLG mid-region half-width (nats). Only meaningful with `--hlg-sampling on`. |
+| `--hlg-shoulder` | nats | `4` | HLG highlight rolloff scale (nats). Only meaningful with `--hlg-sampling on`. |
+| `--hlg-toe` | nats | `6` | HLG shadow rolloff scale (nats). Only meaningful with `--hlg-sampling on`. |
+| `--hlg-pivot-offset` | nats | `6` | HLG pivot point: nats below the top token. Only meaningful with `--hlg-sampling on`. |
+| `--expert-offload` | (bool) | off | **MoE models only.** Serve experts from a page-aligned file mmap (built on first use). Keeps the model out of memory pressure — physical footprint ≈ active params. Ignored with a warning on dense models. Bit-exact with the resident path. |
+| `--compiled-decode` | on\|off | on | Replay the per-step decode graph in C++ (`MLX_BUN_COMPILED_DECODE`). Bit-exact A/B lever. **Serial lane only** (see note below). **Gemma4-dense only** — LoRA, MoE, and non-Gemma4 models (MiniCPM5 / Qwen3.5) run eager; an unsupported step falls back to eager for the rest of that generation. |
 | `--perf-kernel` | on\|off | **on** | Fused quantized-KV decode-SDPA Metal kernel (`MLX_BUN_PERF_KERNEL`), the perf side of the compat A/B. Engages on quantized caches at decode. **Serial lane only.** |
 | `--fused-decode` | on\|off | off | Experimental: tile the quantized decode SDPA (`MLX_BUN_FUSED_DECODE`). **Serial lane only.** |
 | `--fused-sdpa` | on\|off | on | Fused SDPA path for quantized prefill/continuation (inverted env `MLX_BUN_NO_FUSED_SDPA`). **Serial lane only.** |
@@ -40,7 +47,7 @@ winners; flip them to compare. They are set as `MLX_BUN_*` env vars
 before the model loads, so they apply to `mlx-bun pi` too. They affect
 the **serial** decode path (`generate()`); the batched scheduler calls
 `model.forwardHidden` directly and so is unaffected by all of them — see
-[Levers that don't reach the batched lane](#levers-that-dont-reach-the-batched-lane).
+[Levers that don't reach the batched lane](#--batch-n-is-compat-mode--perf-flags-dont-apply-by-design).
 
 > **Note — `--perf-kernel` default.** The code defaults it **on**
 > (`perfKernelEnabled()` returns true unless `MLX_BUN_PERF_KERNEL=0`, and
@@ -129,7 +136,7 @@ concurrently with each other.
 | `--thinking` / `enable_thinking` | ✅ batches (template-render concern, lane-independent) |
 | multi-turn / long prompt | ✅ batches, but **no prompt-cache reuse** (`cached_tokens=0`) |
 
-Both full-attention (CPM) and sliding-window (Gemma) models batch; the
+All three model families — full-attention (CPM), sliding-window (Gemma), and hybrid gated-DeltaNet (Qwen3.5) — batch; the
 scheduler assembles each layer's cache by attention type.
 
 ## Compatibility matrix
@@ -216,7 +223,7 @@ The live config and batch state:
 
 ```jsonc
 {
-  "server":  { "owner": "serve", "model": "...", "started_at": 0 },
+  "server":  { "owner": "serve" | "pi-session" | "embedded", "model": "...", "started_at": 0 },
   "prompt_cache":  { "entries": 0, "bytes": 0, "max_bytes": 2000000000, "hits": 0, "misses": 0 },
   "response_store": { "entries": 0, "bytes": 0, "max_bytes": 33554432, "ttl_ms": 3600000 },
   "kv_quant": {