mlx-bun 0.0.7 → 0.0.9

package/docs/reference/models.md

@@ -0,0 +1,137 @@
+# Model management: get / scan / ls / gc
+
+How mlx-bun downloads, indexes, lists, and reclaims models. The store is
+the standard Hugging Face hub cache — nothing proprietary; `hf` and
+mlx-lm read/write the same tree.
+
+## The cache layout (and why duplicates happen)
+
+Default root: `~/.cache/huggingface/hub` (override with `HF_HUB_CACHE`,
+or `HF_HOME` — the root becomes `$HF_HOME/hub`). Per repo:
+
+```
+models--<org>--<name>/
+  blobs/<digest>                     # verified content, one file per blob
+  snapshots/<commit>/<file> → ../../blobs/<digest>   # symlink farm
+  refs/<revision>                    # e.g. refs/main = the commit sha
+```
+
+**Snapshot-per-commit semantics:** every revision you download gets its
+own `snapshots/<commit>` directory, and nothing ever deletes old ones.
+When upstream pushes a new commit and you `get` again, you gain a new
+snapshot; the previous one — and any blobs only it references, i.e. the
+old weights — stays on disk forever. That is why one repo can appear
+several times in a raw snapshot listing, and how tens of GB of dead
+blobs accumulate. `refs/main` always names the current (canonical)
+snapshot; the registry and every query verb resolve through it.
+
+## get — download (resumable, verified)
+
+```
+mlx-bun get <org/repo> [--revision main]
+mlx-bun get <substring>          # no "/" = registry query, re-gets the match
+```
+
+- Plain HTTPS resolve/CDN (no Xet), sequential files, `Range`-resume of
+  partial blobs (`<blob>.incomplete`). Auth: `HF_TOKEN` env, then the
+  token `hf auth login` writes. 401/403 answers get a "gated repo" hint.
+- A substring argument (no `/`) resolves against already-downloaded
+  repos — `mlx-bun get 12B` refreshes the 12B to upstream's latest. An
+  unknown substring errors with a pointer to `mlx-bun ls <q>`.
+- Concurrent-writer safety: an `O_EXCL` lockfile (`<blob>.lock`, pid +
+  timestamp) serializes two processes downloading the same blob (e.g. a
+  foreground `get` racing the server's background auto-download). Locks
+  from dead pids, or older than ~1 h, are stolen; a live lock fails fast
+  with "another download … is in progress".
+- If the download created a new snapshot (upstream pushed), `get` says
+  so and estimates what `mlx-bun gc` would reclaim.
+
+### Verification story
+
+Every blob is checksummed **while streaming** (a resume re-hashes the
+existing prefix in chunks — no whole-file allocation):
+
+- **LFS files** (weights): sha256 must equal the API's `lfs.oid` — which
+  is also the blob's filename.
+- **Small files** (configs, tokenizer): git blob identity,
+  `sha1("blob <size>\0" + content)`, must equal the API's `blobId`.
+
+A mismatch deletes the partial (never resume corrupt bytes); a short
+read keeps the `.incomplete` for resume.
+
+## scan + the registry
+
+`mlx-bun scan` walks `models--*/snapshots/*` reading only `config.json`
++ safetensors **headers** (never tensor bytes) into
+`~/.cache/mlx-bun/registry.sqlite`. The registry is a derived cache: on
+schema drift it rebuilds; rows whose snapshot dir vanished are reaped.
+
+Schema (table `models`, one row **per snapshot**, keyed on `path`):
+
+| column | meaning |
+| --- | --- |
+| `path` (PK) | `<hub>/models--…/snapshots/<commit>` |
+| `repo_id`, `model_type` | identity |
+| `param_count`, `num_layers`, `hidden_size`, `vocab_size` | shape |
+| `size_bytes` | language-model weights (sidecar excluded) |
+| `sidecar_bytes` | `optiq_vision.safetensors` (bf16 SigLIP sidecar) |
+| `experts_bytes` | `.experts.` tensor bytes (MoE fit math) |
+| `quant_bits`, `quant_group_size`, `quant_mode` | quantization |
+| `has_vision_sidecar` | sidecar file present |
+| `vision_config_type` | `config.json`'s `vision_config.model_type` when it names a vision tower (`*_vision`) |
+| `has_kv_config`, `has_tool_template`, `license` | capabilities/terms |
+| `scanned_at` | freshness (canonical tie-break) |
+
+**Vision capability** is `has_vision_sidecar OR vision_config_type =
+*unified_vision`: the SigLIP models (e2b/e4b/26B/31B,
+`gemma4_vision`) need the bf16 sidecar; the unified 12B
+(`gemma4_unified_vision`) declares its encoder-free tower in config.
+Presence of a `vision_config` key alone is **not** a signal — Qwen3.5
+nests a copy of its own text config there.
+
+## Query resolution rules (per verb)
+
+Every verb resolves fuzzy queries the same way, with revision collapsing
+so a stale snapshot never makes a repo "ambiguous":
+
+- **`ls [query]`** — substring match on repo id / model type; shows **one
+  row per repo** (the canonical revision). `--all-revisions` shows the
+  per-snapshot truth with the canonical marked `*`. Capabilities column
+  labels the support tier: `supported (targeted)` (dedicated forward,
+  L2/L3 paths) vs `supported (generic)` (Tier-0 universal module).
+- **`fit` / `serve` / `train` / … (single-model verbs)** — `resolve()`:
+  the query must match exactly one **repo**; multiple cached revisions of
+  that repo collapse to the canonical snapshot (refs/main, else most
+  recently scanned). Drafter companions (`*_assistant`) never count.
+- **`get <substring>`** — same `resolve()`, then re-downloads that repo id.
+
+Canonical = the snapshot `refs/main` points at; if no `refs/main`, the
+most recently scanned snapshot (stable path tie-break).
+
+## gc — reclaim superseded snapshots + dead blobs
+
+```
+mlx-bun gc              # plan + per-repo reclaim summary, deletes nothing
+mlx-bun gc --yes        # actually delete (destructive)
+mlx-bun gc --dry-run    # never delete, even with --yes
+mlx-bun gc --yes --force  # also prune warned snapshots (see below)
+```
+
+Per repo: keep every snapshot a `refs/*` file points at, delete the
+rest, then delete blobs no surviving snapshot symlinks to. Prints
+keep/prune/skip counts and reclaimable bytes per repo.
+
+Safety rails:
+
+- **Deletion requires `--yes`.** The default run is a report.
+- **File-loss tripwire:** a superseded snapshot containing files the
+  kept snapshots *lack* is skipped with a warning naming those files —
+  deleting it would delete the machine's only copy (live example: a
+  stale gemma-4-12B snapshot holds an `optiq_vision.safetensors` that
+  the canonical revision dropped). `--force` overrides, deliberately.
+- Repos with no usable refs (no `refs/`, or refs naming a missing
+  snapshot) are left untouched.
+- `.incomplete` / `.lock` resume artifacts in `blobs/` are never touched.
+
+After a real deletion, gc re-scans so the registry drops the reaped
+rows immediately.

package/docs/reference/orpo-quickstart.md

@@ -37,6 +37,11 @@ mlx-bun train e4b --data ./prefs --resume ~/.cache/mlx-bun/mlx-bun-finetunes/orp
 | `--rank` / `--scale` | orpo 16 / 2.0 | LoRA rank (by_bits scaled) / scale |
 | `--seq` | e4b 8192 · cpm 4096 | max sequence length |
 | `--lambda` | 0.1 | ORPO odds-ratio weight |
+| `--sft-scope` | `full` | ORPO chosen-NLL scope (`sft_scope`): `full` = paper/TRL-faithful prompt+response CE · `response` = response-only, bit-exact to pre-2026-07 runs |
+| `--grad-accum` | 1 | gradient accumulation steps (effective batch = batch × grad-accum at batch-size-1 memory) |
+| `--grad-clip` | 1.0 | gradient-norm clip (0 = off) |
+| `--seed` | 0 | data-shuffle / init seed |
+| `--val-size` | 256 | max validation examples per eval |
 | `--seg` | 2 | layers/segment (segmented backward); `--no-segment` to disable |
 | `--save-every` | off | mountable checkpoint every N steps |
 | `--resume <dir>` | — | warm-start LoRA weights from a checkpoint/adapter |
@@ -65,6 +70,7 @@ MODEL=/path/to/snapshot DATA=/path/to/datadir bun scripts/train-orpo.ts
 | `LR` | 1e-5 | learning rate (cosine + short warmup) |
 | `RANK` / `SCALE` | 16 / 2.0 | LoRA rank (by_bits scaled) / scale |
 | `LAMBDA` | 0.1 | ORPO odds-ratio weight |
+| `SFT_SCOPE` | `full` | chosen-NLL scope: `full` (paper/TRL-faithful, prompt+response) · `response` (pre-2026-07 response-only, reproduces old runs) |
 | `FLASH=0` | (on) | use the MLX fused head instead of the flash Metal head |
 | `PREFIX=0` | (on) | disable prefix-sharing (plain two-forward) |
 | `SEGOFF=1` | (off) | disable the segmented backward (more memory) |

package/docs/reference/server-api.md

@@ -11,7 +11,9 @@ For the full set of start flags (`--port`, `--memory-budget`,
 defaults, and the perf levers) and a compatibility matrix of which
 combinations compose, see [server-config.md](./server-config.md). This
 doc covers the request/response wire format. LoRA adapters are mounted at
-runtime via `POST /v1/adapters` (below), not a start flag.
+runtime via `POST /v1/adapters` (below), or at startup with
+`serve --adapter <dir>` (which also makes that adapter the default for
+requests that don't select one).
 
 ## POST /v1/chat/completions
 
@@ -25,7 +27,25 @@ Request body (OpenAI chat schema; unknown fields ignored):
   "temperature": 0.7,            // 0 = greedy
   "top_p": 0, "top_k": 0,        // 0 = off
   "seed": 1234,                  // omit for time-derived
+  "min_p": 0.05,                 // min-p sampling (0 = off)
+  "xtc_probability": 0.5,        // XTC sampling: chance per step of dropping
+  "xtc_threshold": 0.1,          //   every token above threshold except the
+                                 //   least likely (EOS + newline exempt,
+                                 //   matching mlx_lm.server)
+  "logit_bias": { "42": -5.0 },  // additive per-token-id logit bias
   "repetition_penalty": 1.1,     // optional
+  "repetition_context_size": 20, // recent-token window (0 = whole history)
+  "presence_penalty": 0.0,       // subtracted once if the token occurred
+  "presence_context_size": 20,   //   in the window (mlx-lm extension)
+  "frequency_penalty": 0.0,      // subtracted per occurrence in the window
+  "frequency_context_size": 20,  //   (mlx-lm extension)
+  "logprobs": false,             // BOOL (mlx_lm.server's type, even on
+                                 //   /v1/completions): return each emitted
+                                 //   token's logprob
+  "top_logprobs": -1,            // int in [0, 11] or -1 = unset (mlx-lm's
+                                 //   validation — its cap is 11, not
+                                 //   OpenAI's 20); k > 0 returns the top-k
+                                 //   (token, logprob) pairs per position
   "stop": "\n\n",                // or ["###", "\n\n"] (spec: up to 4)
   "tools": [ /* OpenAI function tools */ ],
   "tool_choice": "auto",         // "none" disables tools
@@ -88,6 +108,43 @@ Non-streaming response:
 }
 ```
 
+### logprobs / top_logprobs
+
+With `"logprobs": true` and/or `"top_logprobs": k` (1–11), the
+non-streaming response's `choices[0].logprobs` carries
+**`mlx_lm.server`'s block, not OpenAI's** (server.py
+`generate_response`): entries are keyed by token *id*, token strings
+are the raw vocab pieces (`convert_ids_to_tokens` — e.g. `"▁Hello"`),
+and there is no `bytes` field. The same block appears on
+`/v1/completions` (mlx-lm shares the response builder; there is no
+legacy `{tokens, token_logprobs}` text-completion shape).
+
+```jsonc
+// top_logprobs: k > 0 (wins over logprobs when both are set — mlx-lm's if/elif)
+"logprobs": { "content": [
+  { "id": 9906, "token": "▁Hello", "logprob": -0.02,   // = top-1, merged from
+    "top_logprobs": [                                   //   dict(top[0], top_logprobs=top)
+      { "id": 9906, "token": "▁Hello", "logprob": -0.02 },
+      { "id": 13347, "token": "▁Hi", "logprob": -4.1 }   // … k entries, sorted desc
+    ] },
+  // … one entry per generated token (reasoning/tool tokens included)
+] }
+
+// logprobs: true only
+"logprobs": { "content": [ { "id": 9906, "logprob": -0.02 }, … ] }
+```
+
+The distribution matches mlx-lm `generate_step` exactly: full-vocab
+log-softmax of the logits **after** logits processors (logit_bias,
+penalties), **before** the sampler's temperature/top-p/top-k/min-p/XTC
+(mlx_lm/generate.py L409–422). mlx-lm's top-k order is unspecified
+(argpartition); ours is sorted descending — the same set, so the entry
+is deterministically the argmax. Stream chunks never carry logprobs
+(mlx-lm's streaming responses don't either). Requests with logprobs
+run on the serial lane under `--batch N` (like the other mlx-lm
+sampler extensions). Invalid values are rejected with mlx-lm's exact
+messages (see Errors).
+
 Streaming (`"stream": true`) is SSE: `data: <chunk>\n\n` per event,
 terminated by `data: "[DONE]"`. Chunks are `chat.completion.chunk`
 objects whose `choices[0].delta` carries `{role}`, then `{content}`
@@ -118,7 +175,12 @@ back to plain content.
 All errors are `{ "error": { "message": …, ... } }`.
 
 - `400` — malformed JSON, empty `messages`, unknown adapter id, vision
-  request on a model without a sidecar, prompt build failures.
+  request on a model without a sidecar, prompt build failures,
+  non-numeric `logit_bias` keys/values (`logit_bias must be a dict of
+  int to float`, mlx-lm's coercion error), invalid logprobs params
+  (mlx-lm's exact validation: `logprobs must be of type bool`,
+  `top_logprobs must be of type int` / `at least 0` / `at most 11`;
+  `top_logprobs: -1` is the accepted "unset" sentinel).
 - `400` with `"type": "memory_admission"`, `"code":
   "context_over_budget"` — `prompt + max_tokens` exceeds the memory
   budget's max safe context (only when serving with `--memory-budget`;
@@ -126,11 +188,56 @@ All errors are `{ "error": { "message": …, ... } }`.
   up front). Lower `max_tokens` or shorten the prompt; the ceiling is
   visible at `/stats`.
 
+## POST /v1/completions (raw text completion)
+
+`mlx_lm.server`'s text-completion endpoint: **no chat template** — the
+`prompt` string is tokenized directly (the tokenizer's own BOS handling,
+exactly mlx-lm's `tokenizer.encode(request.prompt)`) and the model
+continues it. Same generation gateway, admission control, prompt cache,
+and adapter selection as chat.
+
+```jsonc
+{
+  "prompt": "Once upon a time",  // REQUIRED, string only (token arrays
+                                 // rejected, matching mlx_lm.server)
+  "max_tokens": 512,             // default 512 (mlx_lm.server's default) —
+                                 // NOT the chat lane's generous default
+  "stream": false,
+  "stop": "\n\n",                // string or array, decoded-text matching
+  // plus every sampling/penalty field from /v1/chat/completions:
+  // temperature, top_p, top_k, seed, min_p, xtc_probability,
+  // xtc_threshold, logit_bias, repetition_penalty,
+  // repetition_context_size, presence_penalty, presence_context_size,
+  // frequency_penalty, frequency_context_size, adapter,
+  // logprobs (BOOL, mlx-lm's type — not OpenAI's legacy int),
+  // top_logprobs (0-11 or -1) — same response block as chat, see
+  // "logprobs / top_logprobs" above (mlx-lm shares the builder)
+}
+```
+
+Response is the OpenAI text-completion object:
+
+```jsonc
+{
+  "id": "cmpl-…", "object": "text_completion", "created": 1760000000,
+  "model": "<loaded model id>",
+  "choices": [{ "index": 0, "text": "…", "finish_reason": "stop" | "length" }],
+  "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0,
+             "prompt_tokens_details": { "cached_tokens": 0 } }
+}
+```
+
+Streaming (`"stream": true`) is SSE: `text_completion` chunks whose
+`choices[0].text` carries the delta; the final chunk carries
+`finish_reason` and `usage` (always attached — mlx-lm gates usage behind
+`stream_options.include_usage`; ours is an additive superset), then
+`data: [DONE]`. There is no `echo` parameter (mlx_lm.server has none).
+
 ## POST /v1/messages (Anthropic Messages API)
 
 Anthropic-protocol surface over the same engine — on by default, like
 `optiq serve`. Point any Anthropic-SDK tool at the server
-(`ANTHROPIC_BASE_URL=http://localhost:8090`, any `x-api-key`) — Claude
+(`ANTHROPIC_BASE_URL=http://localhost:8080`, any `x-api-key`) — Claude
 Code works as a client this way.
 
 - `system` (string or text blocks), `messages` with string or
@@ -141,7 +248,11 @@ Code works as a client this way.
 - `tools` (`{name, description, input_schema}`) map to function tools;
   server-tool types (web_search, …) are dropped silently.
 - `max_tokens`, `temperature`, `top_p`, `top_k`, `stop_sequences`,
-  `stream` as in the Anthropic spec.
+  `stream` as in the Anthropic spec. The mlx-lm sampler/penalty
+  extensions (`min_p`, `xtc_probability`, `xtc_threshold`,
+  `repetition_penalty`, `presence_penalty`, `frequency_penalty` + their
+  `*_context_size` windows) pass through as extras; `logit_bias` does
+  not exist in this protocol and is not accepted here.
 - Response: `{id: "msg_…", type: "message", content: [{type: "text"} |
   {type: "tool_use"}…], stop_reason, usage: {input_tokens,
   output_tokens, cache_read_input_tokens}}` —
@@ -162,7 +273,11 @@ OpenAI SDK speak this now). Oracle: optiq responses shim.
   `function_call_output`), `instructions` (merged with any
   system/developer items into one leading system message),
   `max_output_tokens`, `temperature`, `top_p`, `top_k`, flat
-  `tools`/`tool_choice` (built-in tool types dropped), `stream`.
+  `tools`/`tool_choice` (built-in tool types dropped), `stream`. The
+  mlx-lm sampler/penalty extensions (`min_p`, `xtc_probability`,
+  `xtc_threshold`, `logit_bias`, `repetition_penalty`,
+  `presence_penalty`, `frequency_penalty` + `*_context_size`) pass
+  through as extras.
 - **`previous_response_id` resumption**: pass a prior response id
   instead of resending the conversation; the server splices the stored
   input + output back in (instructions carry forward when omitted).
@@ -177,9 +292,46 @@ OpenAI SDK speak this now). Oracle: optiq responses shim.
   response.completed` (+ `response.function_call_arguments.delta/.done`
   for tool calls).
 
+## POST /v1/embeddings (OpenAI Embeddings API)
+
+Text embeddings, available when the **served model is an embedding model**
+(plain Qwen3 / Qwen3-Embedding) — consistent with the single-model server:
+`mlx-bun serve mlx-community/Qwen3-Embedding-4B-4bit-DWQ`, then POST here. On
+any other served model the route returns `400 invalid_request_error`.
+
+- `input` (string or array of strings), `model` (ignored — the served model
+  is used), and a **non-standard** `instruction` (optional): applies
+  Qwen3-Embedding's query format `Instruct: {instruction}\nQuery:{text}`,
+  which steers *which* similarity axis the geometry reflects. Omit it for
+  plain document embeddings.
+- Pooling is last-token + L2-normalization (the vectors are unit-length);
+  similarity is the dot product. Bit-exact vs the mlx-lm `qwen3` reference
+  (`tests/qwen3-embed-parity.test.ts`).
+- Response: `{ "object": "list", "data": [{ "object": "embedding", "index":
+  0, "embedding": [float…] }, …], "model": "<served id>", "usage": {
+  "prompt_tokens": N, "total_tokens": N } }`. Embedding is a single forward
+  (no decode loop), so it runs inline — not through the generation gateway.
+
+```sh
+curl localhost:8080/v1/embeddings -H 'content-type: application/json' \
+  -d '{"input": ["the cat sat on the mat", "a kitten by the window"]}'
+```
+
 ## GET /v1/models
 
-`{ "object": "list", "data": [{ "id": "<model id>", "object": "model", … }] }`
+`{ "object": "list", "data": [{ "id": "<model id>", "object": "model",
+"created": <unix s>, … }] }`
+
+The served model is FIRST (with extra capability fields:
+`context_window`, `reasoning`, `vision`, `owned_by`), followed by every other
+servable model the local registry knows (mlx-lm scans the HF cache
+here; the registry is that scan, filtered to supported architectures).
+`GET /v1/models/<id>` filters the list to that id — same list shape,
+matching `mlx_lm.server`.
+
+## GET /health
+
+`{"status": "ok"}` — byte-for-byte what `mlx_lm.server` returns.
 
 ## GET /stats
 
@@ -301,15 +453,37 @@ Select per request with the `adapter` body field. Prompt-cache entries
 are namespaced per adapter spec, so switching adapters never reuses
 another adapter's KV.
 
+`serve --adapter <dir>` (alias `--adapter-path`, mlx_lm.server's
+spelling) mounts an adapter at startup through this same machinery and
+makes it the default for requests that send no `adapter` field; an
+explicit `adapter` (including `"none"`) always wins, and hot-swap via
+these endpoints is unchanged.
+
 ## Client setup: pi
 
-`~/.pi/agent/models.json`:
+The supported one-command path:
+
+```bash
+mlx-bun harness pi            # install; then: pi --provider mlx-bun
+mlx-bun harness pi --remove   # undo
+```
+
+It writes a self-contained extension to
+`~/.pi/agent/extensions/mlx-bun-provider.ts` that registers the `mlx-bun`
+provider with live discovery: at pi startup it fetches `/v1/models` from the
+running server, so the stable `mlx-bun/local` model handle always resolves to
+whatever the server is actually serving (context window, reasoning, and
+vision capability included), and never goes stale across model swaps. The
+model list baked at install time is only the fallback for when the server is
+down.
+
+Manual fallback — `~/.pi/agent/models.json`:
 
 ```jsonc
 {
   "providers": {
     "mlx-bun": {
-      "baseUrl": "http://localhost:8090/v1",
+      "baseUrl": "http://127.0.0.1:8080/v1",
       "api": "openai-completions",
       "apiKey": "sk-anything-nonempty",
       "models": [{ "id": "<model id from /v1/models>" }]
@@ -318,5 +492,9 @@ another adapter's KV.
 }
 ```
 
-Any OpenAI SDK works the same way: `baseURL: "http://localhost:8090/v1"`,
+Note the manual route bakes a concrete model id, which goes stale when you
+serve a different model — the staleness `harness pi`'s `local` id exists to
+solve.
+
+Any OpenAI SDK works the same way: `baseURL: "http://127.0.0.1:8080/v1"`,
 any non-empty `apiKey`.

package/docs/reference/server-config.md

@@ -19,16 +19,18 @@ the design rationale is in
 
 | Flag | Arg | Default | What it does |
 | --- | --- | --- | --- |
-| `--host` | addr | all interfaces | Interface to bind. `127.0.0.1` = loopback only. |
-| `--port` | n | `8090` | Listen port. A pre-flight probe refuses to start if the port is already serving. |
+| `--host` | addr | `127.0.0.1` | Interface to bind. Loopback-only by default (mlx_lm.server parity); pass `--host 0.0.0.0` to expose the server on your network. |
+| `--port` | n | `8080` | Listen port. A pre-flight probe refuses to start if the port is already serving. |
 | `--memory-budget` | GB | machine RAM × 0.75 | Admission ceiling. Requests whose `prompt + max_tokens` exceed the budget's max safe context are **rejected with 400** (`type: memory_admission`) instead of risking an uncatchable GPU OOM. Also caps the mlx allocator (`mlx_set_memory_limit`) as defense in depth. **Decimal GB (×10⁹).** |
-| `--prompt-cache` | GB | `2` (≈2e9 bytes) | Byte cap for the prompt (prefix-reuse KV) cache. **Binary GiB (×2³⁰)** on input. `0`/unset ⇒ default; there is no flag to disable the cache outright. |
-| `--batch` | n | `1` (serial) | Max concurrent requests batched through the mlx-lm-parity engine. `>1` switches the **whole server** into bf16 continuous batching — a *mode*, not a load fallback. See [Execution modes](#execution-modes-serial-vs---batch-n). Alias: `--decode-concurrency` (mlx_lm.server drop-in). |
+| `--prompt-cache` | GB | `2` (≈2e9 bytes) | Byte cap for the prompt (prefix-reuse KV) cache. **Binary GiB (×2³⁰)** on input. Unset ⇒ the 2 GB default; an explicit `--prompt-cache 0` **disables** the cache (maxBytes 0 evicts every entry). |
+| `--batch` | n | `1` (serial) | Max concurrent requests batched through the mlx-lm-parity engine. `>1` switches the **whole server** into bf16 continuous batching — a *mode*, not a load fallback. See [Execution modes](#execution-modes-serial-vs---batch-n). `--decode-concurrency` is accepted for drop-in compatibility, but the semantics differ: in mlx_lm.server it caps per-BatchGenerator decode parallelism (default 32); in mlx-bun it enables continuous batching with this cap (mlx-bun's default is the optimized serial path). |
 | `--kv-quant` | `config`\|`off`\|`4`\|`8` | `config` serial / `off`(bf16) under `--batch N` | KV-cache quantization. `config` = per-layer `kv_config.json` (optiq parity); `off` = bf16; `4`/`8` = uniform bits (group 64, start 0). Under `--batch N`, an explicit value routes those requests to the **serial** lane (batched is bf16-only). |
+| `--adapter` | dir | none | Mount a LoRA adapter at startup (same machinery as `POST /v1/adapters`; the adapter id is the directory's basename) and make it the **default** for requests that send no `adapter` field. A request's explicit `adapter` — including `"none"` — always wins, and hot-swap via `/v1/adapters` is unchanged. `--adapter-path` is accepted as the mlx_lm.server-named alias. A bad adapter fails startup loudly rather than silently serving the base model. This is the flag `mlx-bun train`'s completion message points at. |
 | `--thinking` | `true`\|`false` | model's own (false for CPM) | Server-wide default for the chat template's `enable_thinking` (MiniCPM5/CPM and Qwen3.5 hybrid reasoning). A request's `chat_template_kwargs.enable_thinking` overrides it. |
-| `--temperature` | n ∈ [0,5] | `generation_config.json` | Server-wide sampling default. Per-request `temperature` still wins; the browser chat (sends none) inherits this. |
+| `--temperature` | n ∈ [0,5] | `generation_config.json` | Server-wide sampling default. Per-request `temperature` still wins; the browser chat (sends none) inherits this. `--temp` is accepted as an alias (mlx_lm.server compat); explicit `--temperature` wins if both are given. **Migration note:** mlx_lm.server's `--temp` *default* is `0.0` (unset-temperature requests are greedy there); mlx-bun falls back to the model's `generation_config.json`, then `0.7` — pass `--temp 0` for mlx-lm's behavior. |
 | `--top-p` | n ∈ [0,1] | `generation_config.json` | Server-wide top-p default (per-request `top_p` wins). |
 | `--top-k` | n ∈ [0,1e6] | `generation_config.json` | Server-wide top-k default (per-request `top_k` wins). |
+| `--max-tokens` | n | `65536` chat / `512` raw completion | Completion cap when a request omits `max_tokens` (mlx_lm.server flag; its default there is 512 — `--max-tokens 512` reproduces mlx_lm.server exactly). |
 | `--no-open` | (bool) | off | Skip the automatic browser open on start. By default an interactive terminal session opens `http://<host>:<port>/#/chat` once the server is ready; pass this flag to suppress it (e.g. headless or non-TTY environments already skip it). |
 | `--hlg-sampling` | `on`\|`off` | off | Piecewise tone-curve (HLG) sampling: rolls off the top-token region, boosts the mids, gentles the tail. The overall gain folds from `--temperature`. See [docs/design/hlg-sampling.md](../design/hlg-sampling.md). |
 | `--hlg-width` | nats | `4` | HLG mid-region half-width (nats). Only meaningful with `--hlg-sampling on`. |
@@ -36,12 +38,18 @@ the design rationale is in
 | `--hlg-toe` | nats | `6` | HLG shadow rolloff scale (nats). Only meaningful with `--hlg-sampling on`. |
 | `--hlg-pivot-offset` | nats | `6` | HLG pivot point: nats below the top token. Only meaningful with `--hlg-sampling on`. |
 | `--expert-offload` | (bool) | off | **MoE models only.** Serve experts from a page-aligned file mmap (built on first use). Keeps the model out of memory pressure — physical footprint ≈ active params. Ignored with a warning on dense models. Bit-exact with the resident path. |
+| `--l1` | (bool) | — | **Parity tier preset:** bit-for-bit IDENTICAL to mlx-lm (bf16 KV, unfused SDPA, no perf kernel). Sets the whole decode route; any explicit per-fork flag (`--kv-quant`/`--perf-kernel`/…) overrides the preset. See [docs/design/parity-tier-dag.md](../design/parity-tier-dag.md). |
+| `--l2` | (bool) | — | **Parity tier preset:** bit-for-bit IDENTICAL to mlx-optiq (quantized KV per `kv_config.json` + fused N-tiled prefill SDPA + **stock unfused decode** — the composition the optiq goldens track, `scripts/regen-kvq-goldens.ts`). The perf kernel stays **off**: it is envelope-gated, not bit-exact (see `--l3`), and opting into it (`--l2 --perf-kernel on`) is an explicit choice that leaves the bare-tier guarantee. |
+| `--l3` | (bool) | — | **Parity tier preset:** best performance, no bit-exact oracle (KL + test gated). On the decode path L3 = L2 + the envelope-gated perf kernel; L3 also owns the no-oracle features (HLG sampler, expert offload). No tier given ⇒ the per-flag defaults below, which equal `--l3`. |
 | `--compiled-decode` | on\|off | on | Replay the per-step decode graph in C++ (`MLX_BUN_COMPILED_DECODE`). Bit-exact A/B lever. **Serial lane only** (see note below). **Gemma4-dense only** — LoRA, MoE, and non-Gemma4 models (MiniCPM5 / Qwen3.5) run eager; an unsupported step falls back to eager for the rest of that generation. |
-| `--perf-kernel` | on\|off | **on** | Fused quantized-KV decode-SDPA Metal kernel (`MLX_BUN_PERF_KERNEL`), the perf side of the compat A/B. Engages on quantized caches at decode. **Serial lane only.** |
+| `--perf-kernel` | on\|off | **on** | Fused quantized-KV decode-SDPA Metal kernel (`MLX_BUN_PERF_KERNEL`), the perf side of the compat A/B. **Not bit-exact** — envelope-gated (≥56/64 teacher-forced argmax vs the frozen compat trajectory, `tests/perf-kernel-oracle.test.ts`), so it is an **L3 lever**: on in `--l3`/no-tier default, **off in bare `--l1`/`--l2`**. Engages on quantized caches at decode. **Serial lane only.** |
 | `--fused-decode` | on\|off | off | Experimental: tile the quantized decode SDPA (`MLX_BUN_FUSED_DECODE`). **Serial lane only.** |
 | `--fused-sdpa` | on\|off | on | Fused SDPA path for quantized prefill/continuation (inverted env `MLX_BUN_NO_FUSED_SDPA`). **Serial lane only.** |
 | `--force-wire` | (bool) | off | Wire weights into memory for the whole generation (`MLX_BUN_FORCE_WIRE`). Near-ceiling models (e.g. 26B) need it. **Serial lane only.** |
 
+The default host/port (`127.0.0.1:8080`) match `mlx_lm.server`, so running
+mlx-bun alongside the Python reference server needs an explicit `--port`.
+
 The five performance levers are A/B knobs whose defaults are the measured
 winners; flip them to compare. They are set as `MLX_BUN_*` env vars
 before the model loads, so they apply to `mlx-bun pi` too. They affect
@@ -62,7 +70,9 @@ Most quality knobs can be set per request in the chat body and override
 the server-wide default. Precedence, highest first:
 
 1. explicit request field (`temperature`, `top_p`, `top_k`, `seed`,
-   `repetition_penalty`, `max_tokens`/`max_completion_tokens`, `stop`,
+   `min_p`, `xtc_probability`/`xtc_threshold`, `logit_bias`,
+   `repetition_penalty`, `presence_penalty`/`frequency_penalty` (+ their
+   `*_context_size` windows), `max_tokens`/`max_completion_tokens`, `stop`,
    `chat_template_kwargs.enable_thinking`, `adapter`, `tools`)
 2. the matching `--temperature`/`--top-p`/`--top-k`/`--thinking` server default
 3. the model's `generation_config.json`
@@ -128,16 +138,36 @@ concurrently with each other.
 | vision (image parts) | ❌ serial — needs offset-0 single-seq prefill + bidirectional image mask |
 | LoRA `adapter` (resolves to ≥1) | ❌ serial — `loraState.active` is one per-generation field; per-row adapters unsupported |
 | `repetition_penalty` | ❌ serial — per-row logits processors are a later refinement |
+| `min_p` / `xtc_*` / `logit_bias` / `presence_penalty` / `frequency_penalty` | ❌ serial — one gate for the whole sampler/processor family (safe v1; min_p/XTC are per-row samplers and could batch later) |
 | explicit `seed` | ❌ serial — reproducibility ⇒ solo (matches mlx-lm) |
 | KV quant active (explicit `--kv-quant`) | ❌ serial — batched is bf16-only in v1 |
+| hybrid-cache model (Qwen3.5 gated-DeltaNet) | ❌ serial — model-level cache-capability gate (see below) |
 | `temperature` / `top_p` / `top_k` | ✅ batches (each row samples with its own seed) |
 | `stop` sequences | ✅ batches (per-row `StopMatcher` in the onToken closure) |
 | `tools` / `tool_choice` | ✅ batches (per-row tool router; decode-layer parse) |
 | `--thinking` / `enable_thinking` | ✅ batches (template-render concern, lane-independent) |
 | multi-turn / long prompt | ✅ batches, but **no prompt-cache reuse** (`cached_tokens=0`) |
 
-All three model families — full-attention (CPM), sliding-window (Gemma), and hybrid gated-DeltaNet (Qwen3.5) — batch; the
-scheduler assembles each layer's cache by attention type.
+Full-attention (CPM) and sliding-window (Gemma) models batch — the
+scheduler assembles each layer's cache by attention type. Hybrid
+gated-DeltaNet models (Qwen3.5) **route serial**: their `SSMCache` has no
+dynamic-B merge/filter, so the gateway's cache-capability gate (the
+mirror of mlx-lm server.py's all-caches-have-`merge` check) sends every
+request down the serial lane with the standard drain behavior. A batched
+SSM path is planned (mlx-lm's `ArraysCache` is the oracle) — see
+[batching-v2-plan.md](../design/batching-v2-plan.md) item (h).
+
+A non-batchable request **drains** the batch: while it waits, the
+scheduler stops admitting new rows, finishes the running ones, and
+releases the GPU so the serial request runs (mlx-lm's `drain_batch`);
+admission then resumes. So a steady stream of batchable traffic cannot
+starve a serial-lane request.
+
+`--batch N` is a **mode switch, not a load-dependent fallback**:
+auto-batching "when >1 request arrives" was considered and rejected —
+an idle vs. loaded server would produce different numerics for the same
+request, breaking determinism and the drop-in-for-`mlx_lm.server`
+promise.
 
 ## Compatibility matrix
 
@@ -157,6 +187,7 @@ inside it per the table above).
 | vision request | ✅ | ✅ via serial lane (in bf16 under Option B) |
 | LoRA `adapter` | ✅ | ✅ via serial lane |
 | `repetition_penalty` | ✅ | ✅ via serial lane |
+| `min_p` / `xtc_*` / `logit_bias` / presence+frequency penalties | ✅ | ✅ via serial lane |
 | `seed` | ✅ | ✅ via serial lane |
 | `tools` / `stop` | ✅ | ✅ (batches) |
 | `--compiled-decode`/`--perf-kernel`/`--fused-*`/`--force-wire` | ✅ (serial perf tree) | **n/a — compat mode, no perf flags by design** |

package/docs/reference/training.md

@@ -22,7 +22,7 @@ Training is reachable several ways — all of them drive the **same** runner
 
 | Path | How | Use when |
 |---|---|---|
-| **CLI** | `mlx-bun train <model> --data <dir> [options]` — foreground, full ORPO stack on by default; auto-detects e4b/Gemma + sets its env; `--save-every`, `--resume`, `--dry-run`; streams loss to the terminal. `mlx-bun help train` for all flags. Run long jobs detached: `nohup mlx-bun train … &` | **The default for CLI users** |
+| **CLI** | `mlx-bun train <model> --data <dir> [options]` — foreground, full ORPO stack on by default; auto-detects e4b/Gemma + sets its env; `--save-every`, `--resume`, `--dry-run`, `--sft-scope full\|response`, `--grad-accum`, `--grad-clip`, `--seed`, `--val-size`; streams loss to the terminal. `mlx-bun help train` for all flags. Run long jobs detached: `nohup mlx-bun train … &` | **The default for CLI users** |
 | **Web UI** | `mlx-bun serve`, open `/finetune` — pick model → dataset → hyperparameters → train; watch live train/val loss; merge/export the adapter | Interactive, in the browser |
 | **HTTP API** | `POST /api/finetune/submit` (job id + SSE events); `POST /api/finetune/inspect-dataset` to probe a file; `POST /api/finetune/merge` to fold an adapter into base weights | Scripted / remote |
 | **ORPO launcher** | [`scripts/train-orpo.ts`](../../scripts/train-orpo.ts) — the same stack driven by env vars (`MODEL=`, `DATA=`, `RANK=`, `RESUME=`, …). See [orpo-quickstart](./orpo-quickstart.md) | Env-var scripting |
@@ -36,7 +36,7 @@ isolation + a single GPU lease across concurrent server requests).
 ### Quick start (HTTP API)
 
 ```bash
-curl -s localhost:8090/api/finetune/submit -X POST -H 'content-type: application/json' -d '{
+curl -s localhost:8080/api/finetune/submit -X POST -H 'content-type: application/json' -d '{
   "model_dir": "/path/to/snapshot",
   "data_dir":  "/path/to/dataset",        // dir with train.jsonl (+ optional valid.jsonl)
   "adapter_path": "/path/to/output-adapter",
@@ -81,7 +81,7 @@ differ from the trainer/API defaults in the table below:
 
 The script hard-codes `method=sft`, `batch_size=1`, `steps_per_report=1`, and
 uses the default `ops.sdpa` training attention (set `MLX_BUN_TRAIN_ATTN=flash`
-to override — but flash crashes e4b at multi-K; see Methodology).
+to override — experimental, much slower; see Methodology).
 
 ## Data formats
 
@@ -112,12 +112,19 @@ Probe a file before submitting: `POST /api/finetune/inspect-dataset` with
   and a preference term, `L = L_NLL(chosen) + λ·L_OR`, with
   `L_OR = -log σ(log_odds)` and
   `log_odds = (ℓ_w − ℓ_r) − (log1mexp(ℓ_w) − log1mexp(ℓ_r))`, where `ℓ` is the
-  **length-normalized** (mean over response tokens) log-prob. Uses the same
+  **length-normalized** (mean over response tokens) log-prob. The SFT term's
+  scope is `sft_scope`: **`full`** (default; paper/TRL-faithful) computes
+  `L_NLL` as the token-mean cross-entropy over the **full prompt+response** of
+  the chosen sequence (only padding excluded — TRL's `chosen_nll_loss`),
+  from the same chosen forward (no extra forward); **`response`** uses
+  `L_NLL = −ℓ_w` (response-only — the pre-2026-07 mlx-bun behavior, kept for
+  reproducing old runs). The odds-ratio `ℓ_w/ℓ_r` stay response-only in **both**
+  modes (that also matches TRL). Uses the same
   `{prompt, chosen, rejected}` preference data as DPO but needs **no reference
   model** (2 forwards/step vs DPO's 4). Default LR `1e-5` (lower than DPO — the
   loss carries a full SFT NLL term a high LR destabilizes). Tune with
-  `orpo_lambda`, `orpo_warmup_iters`, `orpo_lr_schedule`. Design + optimization
-  roadmap: [orpo-training](../design/orpo-training.md).
+  `orpo_lambda`, `orpo_warmup_iters`, `orpo_lr_schedule`, `sft_scope`. Design +
+  optimization roadmap: [orpo-training](../design/orpo-training.md).
 
 The ORPO head + long-context machinery compose into one stack (all B=1), each
 piece independently toggled and each falling back cleanly:
@@ -179,6 +186,7 @@ are `DEFAULT_TRAIN_CONFIG` (trainer.ts:89).
 | `orpo_fused_ce` | bool | `false` | Fused linear-CE head: one CustomVjp with an analytic softmax−onehot backward (MLX `quantizedMatmul` both ways) — no autograd through the head, no retained `[M,vocab]`. Exact; `[chunk,vocab]` transient (orpo only) |
 | `orpo_flash_ce` | bool | `false` | Route the fused head through the **flash-CCE Metal kernel** (verbatim MLX steel GEMM + ORPO epilogue, fwd **and** bwd): neither `[M,vocab]` nor a dequantized head touches HBM → `[M,vocab]`-free, fastest on large vocab (e4b bwd 754 ms, 0.93 GB flat @ M=8192). Implies the fused head. The Apple-CCE coeff filter / blockMax skip are opt-in via `MLX_BUN_CCE_BWD_FILTER_EPS` / `_BLOCK_EPS` (default off — exact). B=1 (orpo only) |
 | `orpo_prefix_shared` | bool | `false` | Shared prompt-prefix: one forward over `[prompt; chosen; rejected]` with a block-sparse mask + block-wise RoPE, so the shared prompt is encoded **once** (a big win when the prompt dominates). Falls back to two-forward per row on prompt mismatch. Composes with the flash head and the segmented backward (MiniCPM5 + e4b). B=1 (orpo only) |
+| `sft_scope` | `full` \| `response` | `full` | Scope of ORPO's chosen-NLL term. `full` (paper/TRL-faithful): token-mean CE over the **full prompt+response** (only padding excluded), from the same chosen forward. `response`: `L_NLL = −ℓ_w` — reproduces pre-2026-07 runs bit-exactly. The odds-ratio `ℓ_w/ℓ_r` are response-only in both modes. Applies to every ORPO path (naive/chunked/fused/flash/prefix/segmented) (orpo only). CLI spelling: `mlx-bun train … --sft-scope full\|response` |
 
 ### Environment variables (training)
 
@@ -186,7 +194,7 @@ are `DEFAULT_TRAIN_CONFIG` (trainer.ts:89).
 |---|---|---|---|
 | `MLX_BUN_FUSED_GELU` | **`0` (required for Gemma)** | on | The fused GeGLU is a **CustomKernel with no gradient (vjp)** ([`fused-geglu-kernel.ts`](../../src/model/fused-geglu-kernel.ts)); the Gemma forward uses it ([`gemma4.ts:277`](../../src/model/gemma4.ts)), so a Gemma (e4b/12B/26B) backward fails unless it's off. MiniCPM5 (Llama-arch SwiGLU) never hits it, so the `.ts` script's MiniCPM5 default doesn't need it. |
 | `MLX_BUN_PERF_KERNEL` | **`0` for training** | on | The fused quantized-decode kernel likewise has no vjp. It only fires at decode L=1 (rare in the L>1 training forward), but the e4b recipe sets `0` to be safe. |
-| `MLX_BUN_TRAIN_ATTN` | **leave unset** | unset → `ops.sdpa` | Default `ops.sdpa` **is** mlx's fused flash-attention kernel — the correct, working path. `flash` selects a *different* hand-rolled custom kernel that crashes e4b at multi-K; do not set it. |
+| `MLX_BUN_TRAIN_ATTN` | **leave unset** | unset → `ops.sdpa` | Default `ops.sdpa` **is** mlx's fused flash-attention kernel — the correct, working path. `flash` selects a *different* hand-rolled O(L)-memory kernel: its two port bugs (dK buffer-transpose, dQ divergent causal barrier) are fixed and FD-validated (`tests/flash-attention.test.ts`, T≤256), but it is ~30× slower and the historical e4b ≥2K crash has **not been re-validated at that scale** since the fix — leave unset unless memory-bound and re-verified. |
 | `MLX_BUN_MEM_LOG` | `1` to profile | off | Print per-step peak/active/cache memory |
 
 > Important: the trainer itself ([`trainer.ts`](../../src/train/trainer.ts))
@@ -221,7 +229,7 @@ What the recipe pins (and why it differs from the bare defaults):
 | `RANK` / `SCALE` / `LR` | `16` / `20` / `1e-5` | task-tuned |
 | `ITERS` | `2` (probe) / `900` (train) | ~2 epochs over 450 examples |
 | `MLX_BUN_PERF_KERNEL` / `MLX_BUN_FUSED_GELU` | `0` / `0` | **required** — the fused kernels have no vjp (see env table) |
-| attention | default `ops.sdpa` | mlx's fused flash kernel; **not** `MLX_BUN_TRAIN_ATTN=flash` (that one crashes e4b) |
+| attention | default `ops.sdpa` | mlx's fused flash kernel; **not** `MLX_BUN_TRAIN_ATTN=flash` (~30× slower; e4b ≥2K unvalidated since the kernel fix) |
 
 The two non-negotiables for e4b: **segmented backward** (`SEG>0`, so the
 long-context activations fit) and the **fused kernels off** (so the backward
@@ -265,9 +273,15 @@ At long `max_seq_length`, activation memory dominates. Two levers:
 ### Training attention kernel (`MLX_BUN_TRAIN_ATTN`)
 - **default `ops.sdpa`** — mlx's fused SDPA; correct (0.00% vs autograd),
   O(L²) backward memory. Use this.
-- **`flash`** — opt-in O(L) memory path, but the hand-rolled dK kernel is
-  slow and **crashes e4b at multi-K (≥2K)**; do not use it for e4b LoRA
-  training. Detail in segmented-backward-training §6.
+- **`flash`** — opt-in O(L)-memory path. Its two port bugs (dK
+  buffer-transpose, dQ divergent causal barrier) were **fixed** and
+  FD-validated (`scripts/experiments/flash-fd-check.ts`; regression test
+  `tests/flash-attention.test.ts`, T≤256), but it remains ~30× slower than
+  `ops.sdpa`, and the historical e4b multi-K (≥2K) SIGTRAP has **not been
+  re-validated at that scale since the fix**
+  (`scripts/experiments/segmented-grad-test-e4b.ts` is the repro harness).
+  Prefer the default; treat `flash` as experimental on e4b long context.
+  Detail in segmented-backward-training §6.
 
 ## Outputs