mlx-bun 0.2.0 → 0.4.0

package/docs/reference/embedding.md

@@ -1,197 +0,0 @@
-# Embedding mlx-bun (single-binary sidecar)
-
-> **Not text embeddings.** This page is about *embedding the mlx-bun binary*
-> inside an app. For text-vector embeddings (Qwen3-Embedding, `mlx-bun embed`,
-> `POST /v1/embeddings`), see [server-api.md](./server-api.md#post-v1embeddings-openai-embeddings-api)
-> and [library-api.md](./library-api.md#text-embeddings).
-
-Ship local inference inside a Mac app (Tauri, Electron, or anything
-that can spawn a process) with zero user-visible dependencies: one
-executable + four native libraries, dropped into your app's resources.
-
-## Build the bundle
-
-```sh
-./scripts/build-binary.sh dist
-```
-
-Produces a relocatable directory:
-
-| file | what | size (arm64) |
-|---|---|---|
-| `mlx-bun` | the CLI/server, compiled with `bun build --compile` | ~70-75 MB |
-| `libmlxc.dylib` | mlx-c, rewritten to load `@loader_path/libmlx.dylib` | ~0.7 MB |
-| `libmlx.dylib` | mlx core (+`@loader_path` rpath added for libjaccl) | ~15 MB |
-| `libjaccl.dylib` | mlx's distributed-comm dependency | ~0.6 MB |
-| `libmlx_bun_expert_io.dylib` | bounded streamed-expert I/O used by direct Colibri GLM-5.2 | ~0.03 MB |
-| `mlx.metallib` | Metal kernels — libmlx loads it from its own directory | ~150 MB |
-| `photon_rs_bg.wasm` | pi image codec — only the web chat's `read`-on-image path; resolved next to the executable | ~1.8 MB |
-
-`build-binary.sh` also sidecars a second set of assets for the embedded
-pi **terminal** (`mlx-bun pi`, `src/pi-terminal.ts` → `InteractiveMode`),
-none of which the headless web chat touches:
-
-| file | what |
-|---|---|
-| `theme/*.json` | TUI color themes (built-in dark/light fallback exists in code; shipped so custom theme reads and the schema resolve) |
-| `assets/*.png` | TUI startup art |
-| `export-html/template.html`, `export-html/vendor/*.js` | the `/export` command's HTML template + vendor JS |
-| `package.json`, `CHANGELOG.md` | version banner + startup changelog |
-| `native/darwin/prebuilds/<arch>/darwin-modifiers.node` | pi-tui's native modifier-key detection (degrades gracefully if absent) |
-
-Library resolution order (src/mlx/ffi.ts): `MLX_BUN_LIBMLXC` env var →
-`libmlxc.dylib` next to the executable → the downloaded native-pack
-cache (`~/Library/Caches/mlx-bun/native-v<ver>-<arch>/`, populated on
-first run when no sidecar or homebrew install is found) → homebrew
-(`/opt/homebrew/lib`, `/usr/local/lib`). The whole directory can be
-renamed/moved; nothing references absolute paths after the build
-script's `install_name_tool` fixups. The expert helper independently resolves
-`MLX_BUN_EXPERT_IO_DYLIB`, then the executable directory, then the same
-native-pack cache.
-
-## Embedded pi web chat (`/ws/chat`)
-
-The compiled binary includes the full embedded pi AgentSession
-(`src/pi-web.ts`, dep `@earendil-works/pi-coding-agent`) that drives the
-browser chat. The web chat runs pi **headless** — no TUI — so it needs
-far less than pi's own `bun build --compile` ships.
-
-**Bundled automatically.** All of pi's JavaScript (the SDK,
-`createAgentSession`, the seven tools, the `openai-completions` provider,
-the resource loader) is pulled into the executable by `bun build
---compile`. Nothing extra to do for the JS.
-
-**Sidecar'd by `build-binary.sh`.** The only runtime asset pi resolves
-*by path* on the headless web-chat path is `photon_rs_bg.wasm` (pi's
-image codec). pi resolves it relative to `process.execPath`, so the build
-script copies it next to `mlx-bun`. It is reached only when the `read`
-tool is asked to read an image file, and pi degrades gracefully when it's
-absent (the tool returns a text "[Image omitted]" note instead of
-crashing), so it is best-effort, not load-bearing.
-
-> Known limitation: `@silvia-odwyer/photon-node`'s wasm-bindgen glue
-> currently fails to decode under Bun (both `bun` and `bun build
-> --compile`) with *"Unreachable code should not be executed"* — a Bun
-> wasm-bindgen gap, not an asset-placement issue (the bytes are valid and
-> found). Net effect today: web-chat image *reads* aren't resized inline.
-> The web chat itself is unaffected (its provider is text-only), and the
-> wasm is shipped so it will work once Bun's support lands.
-
-## Embedded pi terminal (`mlx-bun pi`)
-
-The compiled binary also embeds pi's full interactive TUI
-(`src/pi-terminal.ts` → `InteractiveMode`), reachable via `mlx-bun pi`.
-Unlike the headless web chat, the terminal *does* resolve theme JSON,
-startup art, the `/export` template, the version banner, and the
-pi-tui native modifier-key helper by path — the second table above is
-sidecar'd for exactly this mode. `src/pi-web.ts`'s session (the web
-chat) still builds with `noThemes`/`noSkills`/`noExtensions`/
-`noPromptTemplates`/`noContextFiles`, so none of that table is reached
-from `/ws/chat` — only `mlx-bun pi` uses it.
-
-**Omitted** (upstream pi ships these beside its own binary; neither
-mlx-bun mode reaches them):
-
-| omitted asset | what it's for |
-|---|---|
-| `docs/`, `examples/`, `README.md` | upstream's own help/doc browsing, not exposed by either mlx-bun entry point |
-
-`createAgentSession`'s import graph (`core/sdk.js`) pulls in no loaders
-for these; both `src/pi-web.ts` and `src/pi-terminal.ts` build sessions
-that never reference them.
-
-**Verifying it.** `build-binary.sh` runs `scripts/verify-binary-pi.ts` as
-a sibling compiled binary inside the bundle (so `process.execPath` points
-at the bundle dir, matching what the real binary sees). It builds the
-exact headless session `src/pi-web.ts` builds — same provider/registry/
-resource-loader config — against an unreachable provider, far enough to
-prove the bundled SDK + assets resolve without a missing-asset crash. It
-needs **no model and no server**; a provider/model error is the success
-signal. The one thing this can't cover offline is a live token-streaming
-turn over `/ws/chat` — that still requires `mlx-bun serve` running a real
-model (run it yourself; the smoke covers everything up to the model call).
-
-## Sidecar pattern
-
-Spawn the server on app start, point your in-app client at it:
-
-```ts
-// Electron main / Tauri sidecar config
-const child = spawn(resourcePath("mlx-bun"), [
-  "serve", "gemma-4-12B", "--port", "8080", "--memory-budget", "12",
-]);
-// readiness: poll GET /v1/models (start→ready is ~0.4 s warm)
-```
-
-- One generation runs at a time (single queue, one GPU) — serialize or
-  queue client-side too if you show progress.
-- `--memory-budget <GB>` is the OOM defense: requests that can't finish
-  within the budget are rejected with HTTP 400 (`memory_admission`)
-  *before* generating. The Metal OOM it prevents is uncatchable.
-- Protocols: OpenAI chat completions, Anthropic `/v1/messages`, OpenAI
-  Responses (`previous_response_id` works) — see
-  [server-api.md](./server-api.md). Use whichever your client SDK
-  already speaks.
-- Kill the child freely (SIGKILL is fine): restart→ready is ~0.4 s and
-  prompt-cache misses are the only loss.
-
-## First-run weights
-
-The binary includes the resumable verified downloader:
-
-```sh
-mlx-bun get mlx-community/gemma-4-12B-it-OptiQ-4bit   # → standard HF cache
-mlx-bun scan                                          # index it
-mlx-bun serve gemma-4-12B --port 8080
-```
-
-Downloads resume across interruption, every blob is checksum-verified
-(sha256/sha1), and the cache layout is exactly huggingface_hub's — a
-user's existing HF cache is picked up as-is. For a custom weights
-location inside your app's container, set `HF_HOME` in the child's
-environment.
-
-## Signing & notarization
-
-The build script ad-hoc signs the dylibs (required on arm64 after
-`install_name_tool`). For distribution, re-sign everything with your
-Developer ID and notarize:
-
-```sh
-IDENTITY="Developer ID Application: Your Name (TEAMID)"
-
-# 1. entitlements — Bun executables REQUIRE allow-jit under the
-#    hardened runtime (JavaScriptCore JIT); without it the binary is
-#    killed on launch.
-cat > entitlements.plist <<'EOF'
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
-  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
-<plist version="1.0"><dict>
-  <key>com.apple.security.cs.allow-jit</key><true/>
-  <key>com.apple.security.cs.allow-unsigned-executable-memory</key><true/>
-</dict></plist>
-EOF
-
-# 2. sign (dylibs first, then the executable with entitlements)
-codesign -f --options runtime -s "$IDENTITY" \
-  dist/libjaccl.dylib dist/libmlx.dylib dist/libmlxc.dylib
-codesign -f --options runtime --entitlements entitlements.plist \
-  -s "$IDENTITY" dist/mlx-bun
-
-# 3. notarize (zip the bundle dir; metallib needs no signature)
-ditto -c -k --keepParent dist mlx-bun.zip
-xcrun notarytool submit mlx-bun.zip --keychain-profile "AC_PROFILE" --wait
-```
-
-Inside an `.app`, put the bundle under
-`Contents/Resources/` (Electron: `extraResources`; Tauri:
-`bundle.resources` + `externalBin` for the executable) and let the
-app's normal signing pass cover it — the same entitlements apply to
-the sidecar binary.
-
-## Library API
-
-The server is one consumer of the library — embed generation directly
-in a Bun process instead of spawning, via `loadContext`/`generate`:
-see [library-api.md](./library-api.md).

package/docs/reference/features-matrix.md

@@ -1,101 +0,0 @@
-# Features matrix — everything mlx-bun serves, in one table
-
-The complete option inventory: every serving feature, its default, which
-execution lane it runs on, which fidelity tier verifies it, and its knob.
-This is the *what-exists* view; per-flag detail lives in
-[server-config.md](server-config.md), wire formats in
-[server-api.md](server-api.md), CLI verbs in [cli.md](cli.md).
-
-**Lanes:** `serial` = the default single-queue path; `batch` = the
-`--batch N` continuous-batching engine (bf16 compat mode, mlx-lm B=N
-parity). **Tiers:** L1 = bit-exact vs mlx-lm · L2 = bit-exact vs mlx-optiq
-· Lab = no external oracle; KL/eval-gated experiments (see
-[server-config.md → Fidelity tiers](server-config.md#fidelity-tiers-and-the-decode-route---l1----l2)).
-
-## Serving & performance
-
-| Feature | Default | Lane | Tier | Knob |
-| --- | --- | --- | --- | --- |
-| OpenAI chat completions (`/v1/chat/completions`, SSE) | on | both | — | — |
-| Raw text completions (`/v1/completions`) | on | both | — | — |
-| Anthropic Messages (`/v1/messages`) | on | both | — | — |
-| OpenAI Responses shim (`/v1/responses`) | on | both | — | — |
-| Continuous batching (mlx-lm parity at the same B) | **on** — cap 8 (default flipped 2026-07-05; a lone admitted request uses the gated B=1 fast path and concurrent requests can form B=N; `--batch 1` pins the strict serial executor) | batch | L1 | `--batch <n>` |
-| Prompt cache (prefix KV reuse) | on, 8 GB | both | — | `--prompt-cache <GB>` (0 = off) |
-| **SSD KV cold tier** (cache survives eviction + restarts; v3 includes GLM compressed MLA/DSA state without full-K/V reconstruction) | off | both | — | `--ssd-cache <dir>` (+ `-max`, `-verify`, `-demote-idle <sec>` — default 300s once `--ssd-cache` is on, `0` disables) |
-| **Runtime isolation** (engine runs as a crash-isolated CHILD process behind a reverse-proxy parent; UI stays instant under GPU load and survives engine crashes — [runtime-isolation.md](../design/runtime-isolation.md)) | off | both | — | `--isolate` |
-| **Model pool** (with `--isolate`: LRU-capped RESIDENT model engines; over the cap the least-recently-used engine demotes its cache to the SSD tier and exits, respawning on the next request) | 1 resident engine | both | — | `--model-pool <n>` (requires `--isolate`) |
-| Mixed-precision KV (`kv_config.json`, optiq's scheme) | **off** — opt-in (default flipped to bf16 2026-07-05; quantized KV trades 5–20% decode for memory headroom) | serial + **batch** (per-layer configs batch on every shipped model — full-attention layers Phase 3.1, rotating layers milestone 2; uniform bits stay serial) | L2 | `--kv-quant config\|off\|4\|8`, `--l2` |
-| **TurboQuant KV** (rotation-based: affine keys + FWHT/Lloyd-Max values) | off — opt-in ([turboquant-kv.md](../design/turboquant-kv.md); v1 dequantize-on-fetch, full-attention layers only, no speed claim) | serial only (solo-only, unconditionally) | Lab (codec is oracle-backed vs vllm-metal; the cache class is unvalidated) | `--kv-quant turbo[:k<bits>v<bits>]` (default `k8v3`) |
-| Compiled decode (bit-exact graph replay) | on, every tier | serial | L1/L2 | `--compiled-decode on\|off` |
-| Compiled activations (faithful geglu/swiglu — mlx-lm's `@mx.compile`) | **on**, every tier | both | L1 | `--compiled-activations on\|off` |
-| Fused SDPA (optiq-exact quantized-KV attention) | follows `--kv-quant`: on for `config`, off for uniform/bf16 | serial | L2 | `--fused-sdpa on\|off` |
-| **Paged KV** (vLLM-style block pool: host-side block table + gather before the stock SDPA, no fused kernel in v1 — [paged-kv-cache.md](../design/paged-kv-cache.md)) | off | serial (pins `--batch 1` unless `--batch` given) | gated bit-exact vs the plain `KVCache` (`tests/paged-kv-parity.test.ts`) | `--paged-kv`, `--paged-kv-block-size <n>` (default 256) — v1 scope Gemma4-family bf16; refuses `--batch N>1`, `--kv-quant`, `--draft-model` |
-| **Speculative decoding** (auto-detected drafter: two-model = mlx-lm parity/L1, Gemma `-assistant` = optiq/L2, DSpark local = Lab, DeepSpec released = DeepSpec-reference oracle) | off | serial (forces all-serial) | per-drafter oracle | `--draft-model <path\|query>`, `--draft-kind`, `--num-draft-tokens` |
-| **GLM-5.2 checkpoint-native MTP** (bounded auxiliary expert tier, exact verify) | **on** for GLM-5.2 | serial (`usage.lane: serial+spec`) | Colibri/oracle trajectory + synthetic HTTP gate | `--mtp on\|off` (`off` enables ordinary GLM batching) |
-| Speculative decoding, **model-free prompt lookup** (drafts copied from the request's own context; port of prompt-lookup decoding / vLLM `ngram`; zero weights) | off | serial (forces all-serial) | lossless by verify (gated vs non-spec greedy, tests/spec-ngram.test.ts) | `--draft-kind ngram` (no `--draft-model`), `--ngram-max`, `--ngram-min`, `--num-draft-tokens` (default 10) |
-| Memory admission (refuse what can't fit; never GPU-OOM) | on (RAM × 0.75) | both | — | `--memory-budget <GB>` |
-| Aggregate KV admission for batch rows (queue, don't OOM) | off | batch | — | `--kv-budget <GB>` |
-| Expert offload (MoE cold experts on mmap) | off | serial | Lab | `--expert-offload` |
-| Extend-join (O(1) batch admission) | on | batch | L1 (own oracle) | `MLX_BUN_BATCH_EXTEND=0` |
-| Vectorized greedy batch sampling | on | batch | bit-equal A/B | `MLX_BUN_BATCH_VEC_SAMPLE=0` |
-
-## Request features
-
-| Feature | Default | Lane | Tier | Knob |
-| --- | --- | --- | --- | --- |
-| **Structured output** (`response_format` json_object/json_schema) | on | both | L2 (oMLX) | request field; `MLX_BUN_GRAMMAR=0` kills |
-| Structured output **jump-forward** (grammar-forced spans emitted without per-token forwards; SGLang technique via xgrammar) | **off** | serial only | Lab (string-lossless + validity gates, tests/grammar-jump.test.ts; token stream may differ — no oracle) | `MLX_BUN_GRAMMAR_JUMP=1` |
-| `guided_grammar` (EBNF) / `guided_regex`¹ / `guided_choice` / `structured_outputs` | on | both | L2 | request fields |
-| Structured output × speculative decoding | on (when both active) | serial | Lab (validity+equivalence) | — |
-| Quantized KV × speculative decoding (any axis: uniform / config / turbo) | KV scheme wins — drafted requests decode serially WITHOUT speculation (spec lane is bf16-KV-only in v1; startup warning) | serial | — | omit `--kv-quant` to speculate |
-| Tool calling (Gemma sentinel / CPM+Qwen XML / GLM `arg_key`+`arg_value`) + `role:"tool"` loops | on | both | — | request `tools` |
-| Vision (`image_url` parts; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on (models with a tower); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial | L1/L2 | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
-| **Video input** (`video_url`/`video` parts; mp4/mov H.264/HEVC/ProRes/… via the AVFoundation sidecar `mlx-bun-frame-extract`; 2 fps sampling, ≤768 frames, 256 MB body cap; never combines with audio) | on (Qwen3.5-family models); same SSRF guard | serial | mlx-vlm oracle (frames pipeline bit-exact; tests/qwen38-vision.test.ts) | `--allow-private-media` for LAN hosts |
-| **Audio input** (`input_audio`/`audio`/`audio_url` parts; WAV native, mp3/m4a/flac/ogg/aiff via CoreAudio; ≤30 s / 750 tokens per clip; mixes with images) | on (models with `audio_config` + sidecar tower: e4b); remote fetches refuse private/loopback/link-local hosts by default (SSRF guard) | serial³ | L2 (greedy stream EXACT vs the optiq internal model) | `--allow-private-media` / `MLX_BUN_ALLOW_PRIVATE_MEDIA=1` re-permits LAN hosts |
-| LoRA adapters (mount at start / hot-swap at runtime) | off | serial | — | `--adapter <dir>`, `POST /v1/adapters` |
-| Sampling: temperature / top-p / top-k / min-p / XTC / logit_bias / presence+frequency+repetition penalties | per-request | both | L1 (mlx-lm-faithful) | request fields / server defaults |
-| `logprobs` / `top_logprobs` (mlx-lm semantics) | off | serial | L1 | request fields |
-| Fixed `seed` reproducibility | off | serial (routes solo) | — | request field |
-| Thinking-mode control (hybrid-reasoning models) | model default | both | — | `--thinking`, `chat_template_kwargs` |
-| Stop sequences / streaming / usage accounting | on | both | — | request fields |
-| HLG tone-curve sampling | off | serial | Lab | `--hlg-sampling on` |
-| Spec-decode telemetry (`usage.speculation`) | on with a draft | serial | — | — |
-| Per-turn lane telemetry (`usage.lane`: serial / serial+spec / batched) | on | both | — | — |
-
-## Model coverage (per-model validated cells — no generic path is trusted untested)
-
-| Family | Serial | Batch | Notes |
-| --- | --- | --- | --- |
-| MiniCPM5 (cpm5) | ✅ L1/L2 | ✅ | the starter model |
-| Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
-| Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
-| Qwen3.8-27B (same qwen3_5 graph) | ✅ L1 (no L2 oracle) | ✅ (SSM path) | thinking depths + preserve_thinking; native MTP head mounts via `--draft-model` (DeltaNet spec rollback = snapshot/replay, lossless-gated; quiet-box A/B: 0.82× — correct but slower, opt-in); **vision (images) AND video SERVE** — bf16 tower sidecar + mRoPE + embed splice, oracle = mlx-vlm on pinned mlx (preprocessor/tower/positions bit-exact, e2e token-exact on 2/3 fixtures, serve smokes green; tests/qwen38-vision.test.ts); video files decode via the shipped AVFoundation sidecar (`mlx-bun-frame-extract`, src/native/frame_extract.swift — 2 fps, ≤768 frames, temporal pairs, per-frame-group attention) |
-| GLM-5.2 / Colibri (pre-release port gate) | ✅ chat/text, Messages, Responses, SSE, tools, grammar, logprobs, library/CLI | ✅ compressed MLA/DSA scheduler | native MTP defaults to serial+spec; `--mtp off` enables ordinary batching; artifact-aware CLI/API/status `fit` and live main/MTP expert-tier telemetry; embeddings, vision/audio, adapters, and training are explicitly unsupported; per-row batched MTP is post-release |
-| DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
-| Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family/sliding universal → serial |
-
-## Beyond serving (the same binary)
-
-Training (`mlx-bun train`: ORPO/DPO/SFT, flash-CCE, segmented backward) ·
-quantization (`convert`, `fuse`, `--target-bpw`) · embeddings · local
-memory/wiki (`mlx-bun memory`) · pi agent integration (`mlx-bun pi`) ·
-model registry + `fit` planner + `gc` · HF `upload` · web chat Model Hub
-(`/api/hub/local` + `/search` + `/download`, per-row `/fit` verdict;
-`/api/hub/serve` is `restart_required`-only today — no in-process model-swap
-seam reaches `/ws/chat`, see [server-api.md](server-api.md#model-hub-apihub))
-· **PWA installability** (manifest + shell-only service worker —
-installable, instant static-shell paint; explicitly not offline chat, see
-[server-api.md](server-api.md#web-app-static-routes--pwa-installability)).
-See [cli.md](cli.md) and the per-topic reference docs.
-
----
-¹ `guided_regex` accepts the regex∩EBNF subset today (no `\d`/anchors — those degrade to prompt injection); real regex support is tracked.
-² Gated token-exact vs mlx-lm B=2 (static + dynamic join/leave) on Llama-3.2-3B.
-³ Audio is a capability neither ancestor SERVES: mlx-lm strips the audio tower at load and 400s non-text parts; optiq ships the full machinery but never wires it into its serve frontend — mlx-bun exposes it. Serial lane by design (embeddings prefill; batched audio has no oracle → novel-territory gates would apply), prompt cache skipped.
-
-Composition rules and exclusions (what combines with what):
-[server-config.md → Compatibility matrix](server-config.md#compatibility-matrix).
-Performance per mode: `scripts/bench-modes.ts` (TTFT / prefill / decode /
-peak-memory across every cell above); feature-composition perf:
-`scripts/bench-feature-matrix.ts`.

package/docs/reference/orpo-quickstart.md

@@ -1,135 +0,0 @@
-# ORPO training quickstart — the full stack, preconfigured
-
-One launcher runs the complete new system — **flash-CCE Metal head** (steel fwd+bwd,
-`[M,V]`-free) + **segmented backward** (gradient-checkpointed layer activations) +
-**prefix-sharing** (one forward over `[prompt; chosen; rejected]`) + LoRA. Each feature
-**falls back and logs** if a precondition isn't met (e.g. a row whose chosen/rejected
-prompts aren't identical → two-forward for that row).
-
-## Command — `mlx-bun train` (recommended)
-
-```bash
-mlx-bun train <model> --data <dir>
-```
-- `<model>` is a query or snapshot path (e.g. `e4b`, `MiniCPM5`); omit it to auto-pick.
-- `--data <dir>` is a directory with `train.jsonl` (+ optional `valid.jsonl`); rows are
-  `{"prompt": "...", "chosen": "...", "rejected": "..."}`.
-- The full stack (flash head + prefix-share + segmented) is on by default. No env-flag
-  setup is needed for any model (the old Gemma training-flag requirement is gone —
-  the no-vjp kernels were deleted 2026-07-05).
-- It prints a plan box (dataset counts, format, the resolved knobs), streams per-step loss
-  + peak memory, and saves a **mountable** adapter (`mlx-bun serve <model> --adapter <dir>`).
-- `--dry-run` inspects the dataset and prints the plan without training. `mlx-bun help train`
-  lists every flag.
-
-```bash
-# e4b ORPO on your preference data, checkpointing every 200 steps; run it detached
-nohup mlx-bun train e4b --data ./prefs --iters 1000 --save-every 200 > train.log 2>&1 &
-# resume from a checkpoint (warm-start the LoRA weights)
-mlx-bun train e4b --data ./prefs --resume ~/.cache/mlx-bun/mlx-bun-finetunes/orpo-e4b/checkpoints/step-00200
-```
-
-| flag | default | meaning |
-|---|---|---|
-| `--method` | `orpo` | `sft` · `dpo` · `orpo` |
-| `--iters` | 100 | iterations |
-| `--lr` | orpo 1e-5 · dpo 5e-5 · sft 2e-4 | learning rate (cosine + short warmup) |
-| `--rank` / `--scale` | orpo 16 / 2.0 | LoRA rank (by_bits scaled) / scale |
-| `--seq` | e4b 8192 · cpm 4096 | max sequence length |
-| `--lambda` | 0.1 | ORPO odds-ratio weight |
-| `--sft-scope` | `full` | ORPO chosen-NLL scope (`sft_scope`): `full` = paper/TRL-faithful prompt+response CE · `response` = response-only, bit-exact to pre-2026-07 runs |
-| `--grad-accum` | 1 | gradient accumulation steps (effective batch = batch × grad-accum at batch-size-1 memory) |
-| `--grad-clip` | 1.0 | gradient-norm clip (0 = off) |
-| `--seed` | 0 | data-shuffle / init seed |
-| `--val-size` | 256 | max validation examples per eval |
-| `--seg` | 2 | layers/segment (segmented backward); `--no-segment` to disable |
-| `--save-every` | off | mountable checkpoint every N steps |
-| `--resume <dir>` | — | warm-start LoRA weights from a checkpoint/adapter |
-| `--no-flash` / `--no-prefix` | (on) | drop the flash Metal head / prefix-sharing |
-| `--adapter <dir>` | `~/.cache/mlx-bun/mlx-bun-finetunes/<method>-<model>` | output dir |
-
-> **Long runs:** launch detached from your own shell (`nohup … &`). Foreground is fine for
-> short runs; the command blocks and streams until done.
-
-## The launcher (env-var alternative)
-
-`scripts/train-orpo.ts` is the same stack driven by env vars — handy for scripting:
-
-```bash
-MODEL=/path/to/snapshot DATA=/path/to/datadir bun scripts/train-orpo.ts
-```
-- `DATA` is a directory with `train.jsonl` (+ optional `valid.jsonl`).
-
-### Preconfigured defaults (override via env)
-| env | default | meaning |
-|---|---|---|
-| `SEQ` | e4b 8192 · cpm 4096 | max sequence length |
-| `SEG` | 2 | segment size (layers/segment); the validated e4b@8192 sweet spot |
-| `ITERS` | 100 | iterations |
-| `LR` | 1e-5 | learning rate (cosine + short warmup) |
-| `RANK` / `SCALE` | 16 / 2.0 | LoRA rank (by_bits scaled) / scale |
-| `LAMBDA` | 0.1 | ORPO odds-ratio weight |
-| `SFT_SCOPE` | `full` | chosen-NLL scope: `full` (paper/TRL-faithful, prompt+response) · `response` (pre-2026-07 response-only, reproduces old runs) |
-| `FLASH=0` | (on) | use the MLX fused head instead of the flash Metal head |
-| `PREFIX=0` | (on) | disable prefix-sharing (plain two-forward) |
-| `SEGOFF=1` | (off) | disable the segmented backward (more memory) |
-| `ADAPTER` | `~/.cache/mlx-bun/mlx-bun-finetunes/orpo-<model>` | output dir (the finetunes cache, not the repo) |
-
-Example — e4b overnight at 8192:
-```bash
-MODEL=~/.cache/huggingface/hub/models--mlx-community--gemma-4-e4b-it-OptiQ-4bit/snapshots/<snap> \
-DATA=./data/mydpo ITERS=200 bun scripts/train-orpo.ts
-```
-
-## What to expect — performance (measured, M1 Max 32 GB)
-
-### e4b (Gemma4, H=2560, V=262144) — full stack
-| seq | config | peak GB | s/step |
-|---|---|---|---|
-| 8192 | flash + segmented(2) + **prefix-share** (prompt-dominant data) | **13.3** | **~70** |
-| 8192 | flash + segmented(2), no prefix | 16.1 | ~175 |
-| 4096 | flash + segmented(2), no prefix | 11.7 | ~81 |
-| 2048 | flash + segmented(2), no prefix | 10.2 | ~40 |
-| 1024 | flash + segmented(2), no prefix | 8.7 | ~21 |
-
-- **Fits comfortably in 32 GB at 8192** (the historical "e4b OOMs ≥2048" ceiling is gone).
-- **Prefix-sharing is a real win when the prompt dominates** — at 8192 with a long shared
-  prompt it cut memory 16→13 GB *and* step time 175→70 s (prompt encoded once, not twice).
-  On response-dominant data the saving shrinks toward zero (and it falls back per-row when
-  prompts differ); it never hurts correctness.
-- Memory is **linear in sequence length** (~+1.5 GB / 1024 tokens) — segmenting + flash
-  attention keep it linear, not quadratic. The head itself is **flat 0.93 GB** regardless
-  of vocab/seq.
-- An ~8 h run at 8192 ≈ 100–400 steps depending on the prompt/response split.
-
-### MiniCPM5 (1B, H=1536, V=130560) — full stack
-- Fits easily (~1–2 GB at short seq). ~0.4–1 s/step at seq 256; scales with seq.
-- Everything composes (flash / segmented / prefix-share, in any combination).
-
-### The flash-CCE head in isolation (the piece that made large-vocab affordable)
-- Forward 180 ms, backward **754 ms** (e4b) / 224 ms (cpm) — exact (dh 0.40% / 0.28%
-  bf16-class), `[M,V]`-free, peak **0.93 GB flat at M=8192**. The backward is **5× the old
-  hand-rolled kernel** (3687 ms) via the verbatim MLX steel GEMM + fused dequant.
-
-## Fallback behavior (logged)
-- **Prefix-share, per row**: if `chosen`/`rejected` prompts aren't byte-identical →
-  two-forward for that row. The first occurrence logs `prefix-share: row prompt mismatch
-  → two-forward fallback`; the saving is lost only on mismatched rows.
-- **`FLASH=0`** → the MLX fused linear-CE head (`quantizedMatmul` both ways; exact,
-  `[chunk,V]` transient) instead of the Metal kernel.
-- **`SEGOFF=1`** → all layer activations resident (only viable at short seq on e4b).
-- Models other than MiniCPM5 / Gemma4 with `orpoPrefixShared` error out (not silently).
-
-## Optional / experimental
-- The Apple-CCE backward skips (`MLX_BUN_CCE_BWD_BLOCK_EPS=1e-5`, the lossless vocab-block
-  skip; `MLX_BUN_CCE_BWD_FILTER_EPS=1e-5`, the coeff filter) are **off by default** — on
-  the now-fast steel kernel they give little and the coeff filter costs gradient accuracy.
-  The block skip *can* help on long genuinely-peaked real text; A/B it on your data and
-  watch that the loss doesn't move.
-
-## Caveats (honest)
-- The e4b numbers above are from **short probe runs** (2–3 steps); they confirm the
-  pipeline runs, fits memory, and produces finite/decreasing loss — **not** training
-  quality (that's your data + hyperparameters).
-- e4b gradients are **bf16-class** (~1–3% vs an exact reference) — expected for the fused
-  bf16 path; MiniCPM5 is tighter (forward bit-exact, grads ~1%).

package/src/flags.ts

@@ -1,11 +0,0 @@
-import { runtimeFlag, type RuntimeKey } from "./runtime-config";
-
-// Generic on/off resolver for per-fork decode flags. Process environment is
-// captured once by runtime-config; CLI tier aliases install an explicit
-// immutable override before model modules are loaded.
-
-/** Resolve an on/off env flag. An explicit "1"/"0" always wins; otherwise the
- *  flag takes `defaultOn`. */
-export function flagOn(name: RuntimeKey, defaultOn: boolean): boolean {
-  return runtimeFlag(name, defaultOn);
-}