mlx-bun 0.4.0 → 0.5.0

package/docs/reference/server-config.md

@@ -23,6 +23,39 @@ Sections: [Start flags](#start-flags) · [`--isolate` semantics](#--isolate-sema
 · [Fidelity tiers](#fidelity-tiers-and-the-decode-route---l1----l2) · [Feature matrix](#feature-matrix)
 · [Performance & recipes](#performance-characteristics--recipes) · [`GET /stats`](#observability--get-stats).
 
+## Configuration ownership
+
+CLI presets and explicit flags resolve before model loading. Request preparation
+combines server defaults with request overrides once; explicit request values
+win. A bound execution keeps its immutable runtime snapshot across awaits.
+The tables below own the individual defaults and effective-value rules; this
+map identifies who consumes them.
+
+| Settings | Owning interface / implementation | Binding and use |
+|---|---|---|
+| Model selection, host/port, isolation, native paths, shutdown | CLI and server host | Startup and process lifecycle; no model-forward policy |
+| Batch capacity, queue/KV budget, prefill cohort budget, `MIXED_PREFILL`, `MIXED_TOKEN_BUDGET` | Scheduling policy and execution group | Choose admission and work size; never choose a codec or sample |
+| Draft kind/model/depth, `DSPARK_MINCONF`, `GRAMMAR_DRAFT_TOKENS` | Draft provider and inference method | Bind provider policy; propose tokens through the shared verifier |
+| `GRAMMAR_BATCH`, `BATCH_SSM`, `QWEN_SPEC_KV4`, compiled/grammar/fill eligibility | Model gateway and execution planner | Resolve supported composition before placement |
+| Temperature, seed, penalties, probability filters, logprobs, `BATCH_VEC_SAMPLE` | Sampling session | Request-owned policy and RNG; selected-token output remains device-backed where supported |
+| Thinking/template defaults, grammar schema, `GRAMMAR`, `TOKEN_MASK` | Chat renderer and grammar controller | Render and compile per request; no scheduling decisions |
+| KV scheme/start/group size, `PAGED_KV`, `PAGED_ATTN`, `TURBOQUANT_FUSED_DECODE` | KV scheme, request-state policy and codec | Capture layout/kernel choice; preserve it through deferred construction and RAM copies |
+| Prompt/SSD budgets, demotion, `SSD_*`, `CACHE_RETENTION`, `SESSION_CACHE`, `MTP_PROMPT_CACHE` | Prefix cache and persistence service | Cache owns residency and session lookup; persistence owns queued storage work |
+| `COMPILED_DECODE`, `COMPILED_GEGLU`, `COMPILED_SWIGLU`, `NO_FUSED_SDPA`, `TRELLIS*`, `MIXED_PACKED_MLP` | Model backend and numerical kernels | Read bound execution policy; select implementations by supported tensor geometry |
+| `RD_PREFILL_CHUNK`, `PREFILL_TAIL_SPLIT`, `EARLY_FIRST_TOKEN`, `BATCH_NO_PIPELINE`, `BATCH_EXTEND` | Prefill/step execution and row state | Preserve method boundaries, output ordering and state ownership; the batch-prefixed names remain compatibility controls |
+| `FILL*`, `GRAMMAR_JUMP` | Fill session or grammar proposal provider | Capture request policy; supported methods consume proposals/appends independently of scheduling |
+| Context limits, checkpoints, media access, trace/debug settings | Request preparation, checkpoint service, media adapter and diagnostics | Each service applies its own options; evaluation-only settings stay in the evaluation adapter |
+
+Environment names in this map omit the `MLX_BUN_` prefix. Historical A/B switches
+remain available while their control paths are supported. They are not alternate
+configuration systems. Internal boolean reads use the same `runtimeFlag` port;
+the old `flagOn` wrapper has been removed. CLI aliases remain compatible.
+
+Paged request state captures both its cache namespace and its direct-attention
+choice. Changing host settings after binding cannot change later prefetch, lookup,
+publication or cache construction for that request. The same rule applies to
+TurboQuant's fused codec across copies and delayed precision conversion.
+
 ## Start flags
 
 How flags are parsed (`src/cli.ts` `opt`/`flag`/`positional`):
@@ -76,26 +109,27 @@ When batch rows finish, full-attention caches drop padding shared by every
 surviving row, matching MLX-LM while preserving each row's absolute position.
 
 Qwen prefills longer than four tokens materialize each layer and its cache
-state before continuing, bounding deferred buffer lifetimes. Execution has
-no reservation checks between layers or during decode. Serial requests
-finish outstanding SSD snapshots before generation. The RAM prefix-cache
-cap controls retained entries; it does not promise that every workload fits.
+state before continuing, bounding deferred buffer lifetimes. Cache maintenance
+demotes least recently used snapshots under allocator pressure without
+refusing the request. With SSD enabled, an unwritten victim stays in RAM until
+its queued write completes. The RAM target can be exceeded while writes are
+pending or failing;
+it does not promise that every workload fits.
 Actual allocation failures use the ordinary error path when recoverable;
 a native Metal allocation failure can terminate the process.
-With `--ssd-cache`, an exact recurrent-cache hit persists its boundary before
-transferring the live state to the request. This keeps that boundary reusable
-after the recurrent state advances. Already-durable boundaries need no write;
-a failed required store retains the cache entry and refuses the transfer.
+Recurrent-cache hits lend independently owned views before any donor is
+reclaimed. The request can advance those views while the original boundary
+remains in RAM or in a completed SSD snapshot. Persistence stays asynchronous.
 
 | Flag | Arg | Default | Lane/tier | What it does |
 | --- | --- | --- | --- | --- |
 | `--memory-budget` | GB (decimal, ×10⁹) | unset; memory estimates are advisory | both (per request) | Explicit opt-in budget. Sets the allocator limit and enforces the fit estimate: a prompt leaving no generation slot gets **400** (`type: memory_admission`); a broader completion cap is reduced to the remaining context. Estimates account for active KV quantization and prefill temporaries. Not an aggregate batch cap — see `--kv-budget`. GLM-5.2 also uses this value in its resource plan before loading. |
 | `--kv-budget` | GB (decimal) | off | batch | Aggregate KV budget across concurrently admitted batch rows: a joiner whose projected KV (prompt + `max_tokens`, window-capped) would exceed it **queues** until rows finish; a request over the budget alone is rejected. Without it, N large-context rows can collectively exceed memory. Reported in `/stats.batch.kv_budget_bytes`. |
-| `--prompt-cache` | GB (binary GiB) | `8` | both | RAM prefix-KV cache (byte-capped LRU). `--prompt-cache 0` disables it. RAM hits lend zero-copy views and leave the donor intact, including recurrent Qwen and method state. SSD persistence runs in the background; RAM reuse never waits for a write. Explicit cache flush and graceful shutdown wait for durability. Batch-lane joiners `take()` the longest usable prefix at admission; a row that finishes without ever merging `put()`s its caches back (merged rows' entries age out). |
-| `--ssd-cache` | dir | off | both | SSD cold tier under the prompt cache ([docs/design/kv-cache.md](../design/kv-cache.md)): prefix KV spills to disk on RAM eviction and idle demotion, is snapshotted after requests settle (debounced, each tensor step owns the generation lock so it cannot overlap inference; `MLX_BUN_SSD_WRITEBEHIND=0` disables), and **survives restarts**. Entries are keyed by model fingerprint + effective KV scheme + backend numerical identity + tokenizer hash + adapter namespace. MLX includes its runtime version and GPU architecture; older numerical identities use separate directories. `SIGINT`/`SIGTERM` drains active requests and flushes dirty snapshots before exit (`MLX_BUN_SHUTDOWN_TIMEOUT_MS`); `POST /admin/cache/flush` is the explicit boundary. Requires the RAM cache. |
+| `--prompt-cache` | GB (binary GiB) | `8` | both | RAM prefix-KV residency target, managed by LRU. `--prompt-cache 0` disables it. RAM hits lend zero-copy views, including recurrent Qwen and method state. With SSD enabled, eviction persists an unwritten donor before removing it from RAM; pending writes can temporarily exceed the target. A successful write alone does not evict RAM. The target does not reserve that much memory. SSD persistence runs in the background; RAM reuse never waits for a write. Explicit cache flush and graceful shutdown report durability, including snapshots lost before persistence. Batch-lane joiners `take()` the longest usable prefix at admission; completed decode publishes reusable processed-token state through the same interface. |
+| `--ssd-cache` | dir | off | both | SSD cold tier under the prompt cache ([docs/design/kv-cache.md](../design/kv-cache.md)): prefix KV spills to disk on RAM eviction and idle demotion, is also persisted after publication by a dedicated CPU worker while inference continues (`MLX_BUN_SSD_WRITEBEHIND=0` disables proactive writes), and **survives restarts**. Entries are keyed by model fingerprint + effective KV scheme + backend numerical identity + tokenizer hash + adapter namespace. MLX includes its runtime version and GPU architecture; older numerical identities use separate directories. `SIGINT`/`SIGTERM` drains active requests and flushes dirty snapshots before exit (`MLX_BUN_SHUTDOWN_TIMEOUT_MS`); `POST /admin/cache/flush` is the explicit boundary. Requires the RAM cache. |
 | `--ssd-cache-max` | GB (binary GiB; `0` = unlimited) | unlimited | both | Optional SSD tier byte cap; oldest-mtime entries are evicted only when a positive cap is configured. Warns and is ignored without `--ssd-cache`. |
 | `--ssd-cache-verify` | (bool) | off | both | Verify every tensor hash on restore (reads all bytes eagerly, defeating lazy fault-in) — integrity paranoia only; the header hash is always verified. Warns and is ignored without `--ssd-cache`. |
-| `--ssd-demote-idle` | seconds | `300` with `--ssd-cache`, else off | both | Prompt-cache entries unused this long spill to the SSD tier and free their GPU memory; the next hit restores them. Swept only when the engine is fully idle. `0` disables. Warns and is ignored without `--ssd-cache`. |
+| `--ssd-demote-idle` | seconds | `300` with `--ssd-cache`, else off | both | Prompt-cache entries unused this long spill to the SSD tier and free their GPU memory; the next hit restores them. The cache sweeps by entry age independently of scheduler activity; RAM remains resident until SSD coverage exists. `0` disables. Warns and is ignored without `--ssd-cache`. |
 | `--generation-checkpoint` | output tokens | off | qualified requests | Queues an owned in-flight generation snapshot every N emitted tokens. Repeating the identical request after a restart replays the saved assistant prefix and continues from its already-sampled next token. Requires `--ssd-cache`. Shared execution qualifies ordinary requests with supported row-cache layouts, including Qwen, Llama, MiniCPM and Gemma4, with or without adapters, excluding media, grammar, fill, paging, logprobs or speculative methods; existing serial qualification remains available. Explicit cache flush drains queued writes. Completed generations remove their checkpoint, while client cancellation preserves the latest checkpoint. |
 
 Generation checkpoint identity includes stop strings, sampling/KV policy, the
@@ -107,9 +141,13 @@ identity. MLX runtime or GPU architecture changes select a fresh cache
 directory and leave the previous directory intact. A different backend codec
 identity refuses incompatible files. Adapter namespaces include the mounted weight contents and
 scale, so a changed adapter starts with a fresh prefix and checkpoint.
-Native media and grammar preparation waits for the generation execution lease
-and retains a reservation through completion. At most one media preparation and
-`--batch` grammar preparations can be retained. Generation and preparation
+Native media and grammar preparation uses the backend execution lease between
+decode iterations when the backend supports queued preparation. Exclusive model
+changes retain drain priority. Media downloads and container transcoding run
+outside that lease; a slow remote URL does not hold up decode. Preparations
+retain a reservation through request
+completion. Gemma4 and Qwen shared serving can retain up to `--batch` media preparations;
+other media bindings retain one. Grammar preparation retains up to `--batch`. Generation and preparation
 queues each allow 64 waiting requests; overflow returns `429` before a response
 stream opens, or a terminal stream error after it opens. Disconnects release
 queued reservations and completed preparation resources.
@@ -129,11 +167,29 @@ Disconnected serial requests leave the admission queue immediately. Active AR
 requests check cancellation between prefill chunks and decode steps; a native
 operation already running completes before that boundary.
 
+
+Qwen image/video and Gemma image/audio encoder features use the existing
+prompt-cache RAM budget and, when configured, its SSD persistence layer.
+Exact encoder-weight, computation-setting and input hashes identify reusable
+features; each request still renders its own tokens, timestamps, masks and
+positions. Qwen hashes prepared pixels and grids; Gemma hashes image/WAV bytes
+within each encoder's versioned schema. No separate media-cache capacity is
+allocated. `--prompt-cache 0` disables retention. CPU preprocessing and media
+extraction still run before feature lookup, outside the native execution queue.
+Encoder reuse is independent of batch/method selection. Experimental Gemma
+media KV reuse is controlled by `MLX_BUN_MEDIA_PREFIX_CACHE` below. Cache counters
+are documented in [server-api.md](server-api.md).
+
+
 ### Runtime isolation
 
 | Flag | Arg | Default | Lane/tier | What it does |
 | --- | --- | --- | --- | --- |
 | `--isolate` | (bool) | off | both | Run the inference engine as a **child process** on a unix socket while this process owns CPU application routes and proxies inference — instant under any GPU load, survives engine crashes (auto-respawn). Full semantics in [`--isolate` semantics](#--isolate-semantics). |
+| `--whisper-model` | path \| query | first downloaded `whisper` checkpoint, resolved on the first `/v1/audio/*` request | both | Whisper checkpoint served beside the chat model for [audio transcription](server-api.md#post-v1audiotranscriptions--post-v1audiotranslations-speech-to-text). Serving a Whisper checkpoint as the MAIN model (`serve mlx-community/whisper-large-v3-turbo`) starts a **transcription-only server** (audio routes, `/v1/models`, `/health`, `/stats`; no chat model, no prompt cache). |
+| `--whisper-idle-unload` | seconds | `0` | both | How long the Whisper weights (and MLX's allocator cache) stay after a take. Default `0`: released right after every take, so the chat model has the memory back before it prefills the dictated prompt; the process drops by ≈1.4 GB and the next take pages them back in from the OS file cache (a cold 11 s take is ≈520 ms vs 365 warm on the M1 Max, `TranscriptionService`). Reported per response in `mlx_bun.timings.load_ms`. |
+| `--whisper-resident` | flag | off | both | Never dispose the Whisper weights (sotto's always-loaded behaviour). |
+| `--preload` | flag | off | both | Transcription-only server: load the Whisper weights before listening instead of on the first request. |
 | `--model-pool` | n (≥1) | `1` | both | With `--isolate`: max **resident** model engines. A request whose `model` field is an **exact** `/v1/models` id spawns/routes to that model's own engine child (spawn-overlap: the new model loads while the old keeps serving). Over the cap the least-recently-used engine is drained (`POST /admin/drain` over its socket), demotes its prompt cache to the SSD tier, and exits; switching back respawns it with state restored from disk. Without `--isolate` the flag warns and is ignored. |
 
 ### Scheduling
@@ -147,22 +203,30 @@ operation already running completes before that boundary.
 | Flag | Arg | Default | Lane/tier | What it does |
 | --- | --- | --- | --- | --- |
 | `--kv-quant` | `config` \| `off` \| `4` \| `8` \| `turbo[:k<bits>v<bits>]` | `off` (bf16) | see row | KV-cache quantization. `off` = bf16 (the L1 default: quantized KV measured 5–20% slower decode than bf16 at ≤16k on both stacks, so it is an opt-in that buys **memory headroom**). `4`/`8` = **uniform** bits (group 64, start 0 — mlx-lm's `--kv-bits` scheme); with fused-sdpa off (the default for uniform) our unfused quantized SDPA is op-for-op mlx-lm's `quantized_scaled_dot_product_attention`, so uniform is **bit-exact L1**; uniform requests use shared B=1/B>1 execution where the loaded cache layout supports affine conversion. `config` = **per-layer mixed precision** from the model's `kv_config.json` — optiq-only → **L2**; per-layer configs **batch** on every shipped model (full-attention and rotating layers; gated bit-exact per row vs the serial composition). `turbo[:k<bits>v<bits>]` = **TurboQuant** ([docs/design/turboquant.md](../design/turboquant.md)): rotation-based KV quantization, default `k8v3`; `kBits` ∈ {2,4,5,8}, `vBits` ∈ {2,3,4,5,8}; a separate axis (mutually exclusive with `config`/`4`/`8`), uses the existing decoded or deferred rotated-value attention path, **full-attention layers only** (sliding-window layers stay bf16 with a one-time warning), refuses head dims outside `TURBOQUANT_HEAD_DIMS`, and supports shared ordinary and grouped drafting execution at B=1/B>1 including positive library conversion thresholds when the cache layout qualifies. Any explicit `--kv-quant` overrides the tier preset. |
-| `--paged-kv` | (bool) | off | shared / gated bit-exact | **Optional vLLM-style paged KV cache** ([docs/design/kv-cache.md](../design/kv-cache.md)): full-attention layers store K/V in fixed-size block pools (host-side block table, gather back to contiguous before the stock SDPA; no new attention math). Env mirror `MLX_BUN_PAGED_KV=1`. Paging retains the configured batch size; request state owns independent block pools through the shared cache-layout interface. `createServer` **refuses** any `--kv-quant` (incl. turbo), `--draft-model`, and non-Gemma4 models. Media and adapter requests run the plain cache path per request. Paged requests bypass the prompt cache and run uncompiled decode; pool exhaustion is a typed error, never truncation. Gated bit-exact vs the plain path (`tests/parity/paged-kv-parity.test.ts`). |
+| `--paged-kv` | (bool) | off | shared / optional | Full-attention block pools through the common cache-layout and attention interfaces. Supports bf16 and uniform affine `--kv-quant 4|8`; affine pages encode from their first token. Default attention gathers into stock SDPA; `MLX_BUN_PAGED_ATTN=1` selects the experimental direct Metal reader. RAM/SSD reuse uses a separate numerical namespace. Gemma4-family only; draft models, per-layer KV schemes and TurboQuant pages remain unsupported. Media/adapter requests retain their existing non-paged behavior. Env mirror `MLX_BUN_PAGED_KV=1`. |
 | `--paged-kv-block-size` | n | `256` | shared | Tokens per KV block (`--paged-kv` only); 256 = the plain cache's growth step. |
 
 ### Adapters and speculative decoding
 
-In the current unreleased source, Qwen MTP publishes aligned target/draft
+In v0.4.0, Qwen MTP publishes aligned target/draft
 prefill snapshots through the shared prompt cache by default. The RAM budget
 includes the draft KV and pending hidden row; configured SSD persistence
 queues those tensors with the target state and supports reuse after restart.
 RAM hits retain immutable snapshots for later requests. Target, draft-weight,
 adapter-revision and KV-policy identities separate incompatible state.
 `usage.cached_tokens` reports the reused target prefix.
-`MLX_BUN_MTP_PROMPT_CACHE=0` disables this reuse. Generated-output snapshots
-remain unfinished.
-
-In the current unreleased source, Qwen uniform affine KV4 with
+`MLX_BUN_MTP_PROMPT_CACHE=0` disables this reuse. Completed decode also
+publishes checkpoints for processed tokens through the same cache.
+Long-conversation performance acceptance remains open.
+
+Uniform KV4 attention automatically groups three query heads for the key
+multiplication on M4 Pro when B≤2, L=3, GQA6/D256, N≥8192 and bf16/f32
+match the qualified shape. Masking, softmax and value multiplication retain
+their existing geometry. This internal selection needs no additional flag;
+other shapes and machines keep the existing implementation. Exactness and
+timing evidence are in [benchmarks.md](benchmarks.md#three-query-affine-attention-head-grouping).
+
+In v0.4.0, Qwen uniform affine KV4 with
 `quantizedKvStart=0` supports configured speculative execution by default.
 `MLX_BUN_QWEN_SPEC_KV4=0` restores the ordinary-decode compatibility control. Recurrent state and draft KV keep their original precision. Shared Qwen MTP
 also supports bf16, uniform KV8 and TurboQuant at start zero; per-layer mixed
@@ -170,14 +234,25 @@ KV remains incompatible. Strict legacy serial speculation retains bf16/KV4.
 The KV4 control does not disable bf16, KV8 or TQ shared speculation. Six paired M4 Pro combined suites and two completed Kanban tasks support
 this default. Configuring a drafter and selecting KV4 remain explicit choices.
 
-For controlled experiments, `MLX_BUN_RD_PREFILL_CHUNK` sets the prefill chunk,
-default 2048. Shared ordinary and speculative methods use the group's captured
-default; an explicit group option overrides the environment and a library
-request's `prefillChunkSize` overrides that default for its request.
+The shared prefill policy selects a chunk once per request. The usual size is
+2,048 tokens. For recurrent-attention models with materialized SDPA scores,
+it halves that size until the estimated per-layer score workspace is at most
+1 GiB (minimum eight queries, where native fused decode attention applies).
+Qwen3.8-27B therefore uses 2,048 at 10,398 prompt tokens and 256 at 78,678.
+This changes the work size; it does not reject or shorten the request.
+Ordinary and speculative execution consume the same bound policy.
+
+`MLX_BUN_RD_PREFILL_CHUNK` selects an explicit fixed chunk. A group option
+supersedes the environment, and a library request's `prefillChunkSize`
+supersedes either. Explicit values bypass the automatic choice.
 `MLX_BUN_RD_CONTEXT_LIMIT` sets an explicit request context cap
-without enlarging it. Both require positive integers; the context cap is unset
-by default. These are benchmark controls, not changes to the published model
-profile or sampling policy.
+without enlarging it. Both controls require positive integers; the context
+cap is unset by default. Sampling and the published model context remain
+independent of prefill chunk selection. To reproduce a seeded response, keep
+the prefill chunk policy fixed along with the model, runtime, machine and
+sampling settings. Changing chunk boundaries can change numerical results
+and the sampled continuation. The [Kanban comparison](benchmarks.md#full-kanban-after-decode-integration)
+records this effect on the same M4 Pro with identical initial request JSON.
 
 | Flag | Arg | Default | Lane/tier | What it does |
 | --- | --- | --- | --- | --- |
@@ -204,11 +279,18 @@ profile or sampling policy.
 | `--hlg-toe` | nats ∈ [0, 100] | `6` | serial | HLG shadow rolloff scale. Only with `--hlg-sampling on`. |
 | `--hlg-pivot-offset` | nats ∈ [0, 100] | `6` | serial | HLG pivot: nats below the top token. Only with `--hlg-sampling on`. |
 
+Greedy sampling without requested logprobs uses a shared normalized-selection
+kernel after applying penalties and grammar masks. It preserves dtype-rounded
+scores and lowest-ID ties while avoiding a full normalized vocabulary output.
+Independent speculative verification uses the same operation. Requests needing
+logprobs, stochastic sampling or a custom sampler keep their existing operations;
+no new flag is required. Measurements are in [benchmarks](benchmarks.md#fused-normalized-greedy-selection).
+
 ### Media and UX
 
 | Flag | Arg | Default | Lane/tier | What it does |
 | --- | --- | --- | --- | --- |
-| `--allow-private-media` | (bool) | off | serial (media requests) | Let `image_url`/`audio_url`/`video_url` parts fetch from **private/loopback/link-local** hosts. Off by default: a request's URL is attacker-controlled, so `src/media-fetch.ts` refuses non-http(s) schemes and private/loopback/link-local/CGNAT destinations — including hosts that *resolve* there and every redirect hop (SSRF guard). Independent of the flag, every remote fetch has a 10 s timeout and a 64 MB streaming-enforced cap; violations are clean `400`s. `data:` URLs are never policy-checked. Env mirror `MLX_BUN_ALLOW_PRIVATE_MEDIA=1`. |
+| `--allow-private-media` | (bool) | off | both (model-qualified media) | Let `image_url`/`audio_url`/`video_url` parts fetch from **private/loopback/link-local** hosts. Off by default: a request's URL is attacker-controlled, so `src/media-fetch.ts` refuses non-http(s) schemes and private/loopback/link-local/CGNAT destinations — including hosts that *resolve* there and every redirect hop (SSRF guard). Independent of the flag, every remote fetch has a 10 s timeout and a 64 MB streaming-enforced cap; violations are clean `400`s. `data:` URLs are never policy-checked. Env mirror `MLX_BUN_ALLOW_PRIVATE_MEDIA=1`. |
 | `--no-open` | (bool) | off | both | Skip the automatic browser open. By default an interactive TTY opens `http://<host>:<port>/#/chat` once the server is ready; non-TTY runs never open. Parent-only under `--isolate`. |
 
 ### Parity tier and kill switches
@@ -367,7 +449,7 @@ chat stage (`src/serve/chat-stage.ts`). Full field list: [server-api.md](./serve
 All `MLX_BUN_*` variables are captured once at process start into an
 immutable snapshot (`src/runtime-config.ts`); CLI flags install overrides
 into the same snapshot before the model loads. Boolean levers read the
-literal strings `"1"`/`"0"` — `flagOn` treats any other value as unset.
+literal strings `"1"`/`"0"` — `runtimeFlag` treats any other value as unset.
 Under `--isolate` the whole environment is inherited by the engine child.
 
 ### Serving levers (flag-backed and lane kill switches)
@@ -375,8 +457,12 @@ Under `--isolate` the whole environment is inherited by the engine child.
 | Env var | CLI flag | Default | Effect |
 | --- | --- | --- | --- |
 | `MLX_BUN_COMPILED_DECODE` | `--compiled-decode` | on (`"0"` disables) | Compiled decode graph replay (serial lane; batch lane at B=1). |
-| `MLX_BUN_EARLY_FIRST_TOKEN` | — | off (`=1`) | Serial/native generation yields token zero before constructing the next decode step. This can reduce first visible output latency when token zero contains visible text. Later decode remains pipelined. Serial fill, grammar, checkpoint resume and single-token budgets retain their existing order. When a native consumer stops at the first yield and retains caller-owned caches, a non-aborted return completes that token’s M=1 forward before returning the cache. This preserves the ordinary pipeline’s boundary for later prefix reuse. Aborted requests do not start another forward; caches owned and disposed by the generation need no alignment. Native M1 packed-Qwen/MiniCPM/Gemma continuation gates pass; M4 packed/affine and serving acceptance remain. The continuous scheduler also yields after preparation creates its first active row when no other request is queued, allowing prepared output to flush before decode. It then rechecks cancellation, admission and shutdown; queued short admissions still group together. The setting is captured by generation and the batch runtime. Experimental pending broader cached/pressure and quiet-machine acceptance. |
-| `MLX_BUN_TURBOQUANT_FUSED_DECODE` | — | off (`=1`) | Experimental packed K/V decode fusion for an existing `--kv-quant turbo:...` cache, captured when each cache is created. A shared Metal operation unpacks keys and values, applies the existing key zero/scale and Lloyd-Max value scale, and preserves the codec's eager or deferred inverse rotation. It accepts supported bit widths, head dimensions 64/128/256/512 and 32/64-element groups with fp16/bf16/f32 metadata. Eager k8v3 with B1/H4, head dimension 256, fp16 metadata, group32 and at least 8192 cached tokens also fuses inverse rotation; other shapes keep the existing rotation path. Unsupported inputs, CPU streams and shapeless traces retain ordinary operations. Quantization, stored cache format and serving eligibility are unchanged. Joint-decoder Qwen native/serial and repeated long-context HTTP gates pass. The inverse operation passes both integrated Qwen model gates and six HTTP pairs per quant. MiniCPM/Gemma deferred-consumer serving gates also pass. The unreleased shared codec/layout supports ordinary and supported drafting groups without changing this kernel selection. Gemma now owns pre-write row positions across cache appends; the fix passes native checks on both Macs. Combined settings, pressure and strict M4 Pro acceptance remain. |
+| `MLX_BUN_MEDIA_PREFIX_CACHE` | (env only) | on (`=0` disables) | Reuse prepared Gemma image/audio or Qwen image/video conversation KV in the shared executor. Media contents, encoder identity and preceding tokens scope the ordinary RAM/SSD cache. Generated follow-ups reuse state; rewinds before a delayed-precision boundary recompute. Qwen recurrent layers require an exact retained prefix. Explicit serial retains encoder-only reuse. |
+| `MLX_BUN_MIXED_PREFILL` | — | off (`=1`) | Lab mixed prefill/decode execution for Gemma 4 and Qwen3.5/3.8 text groups, including grouped speculative methods. The scheduler reserves running tokens before prompt work; the model packs feed-forward operations across real tokens while attention and KV state retain their original group geometry. Methods supply candidate-token demand and per-group hidden taps; speculative verification preserves its existing matmul geometry. Packed matmuls can select different kernels from solo decode; this does not claim solo-logit identity or a measured serving win. |
+| `MLX_BUN_MIXED_PACKED_MLP` | — | on (`=0` disables) | Lab control inside mixed model work: disable feed-forward packing while keeping the same scheduling budget and attention geometry, to measure packing separately from chunk scheduling. No effect when mixed execution is off. |
+| `MLX_BUN_MIXED_TOKEN_BUDGET` | — | `256` | Total real tokens in an enabled mixed iteration, including running decode rows. At least one token per participating row progresses. A lone preparation retains its existing chunk setting. Precision transitions and checkpoint endpoints remain method-owned. |
+| `MLX_BUN_EARLY_FIRST_TOKEN` | — | on (`=0` disables) | Eligible ordinary generation yields token zero before constructing the next decode step; later decode remains pipelined. The continuous scheduler yields after preparation creates its first active row when no other request is queued, allowing prepared output to flush before decode. It then rechecks cancellation, admission and shutdown; queued short admissions still group together. Serial fill, grammar, checkpoint resume and single-token budgets retain their existing order. If a native consumer stops at the first yield and retains caller-owned caches, a non-aborted return completes that token’s M=1 forward before returning the cache, preserving the ordinary pipeline’s boundary for prefix reuse. Aborted requests do not start another forward; generation-owned caches need no alignment. The setting is captured by generation and the batch runtime. Native M1/M4 state and continuation, HTTP cancellation/overlap and cached/SSD gates pass. Monitored paired chat measurements support earlier visible output; they do not establish higher decode throughput, and an invisible token zero need not improve client latency. See benchmarks.md. |
+| `MLX_BUN_TURBOQUANT_FUSED_DECODE` | — | on (`=0` disables) | Packed K/V decode fusion for an existing `--kv-quant turbo:...` cache, captured when each cache is created. A shared Metal operation unpacks keys and values, applies the existing key zero/scale and Lloyd-Max value scale, and preserves the codec's eager or deferred inverse rotation. It accepts supported bit widths, head dimensions 64/128/256/512 and 32/64-element groups with fp16/bf16/f32 metadata. Eager k8v3 with B1/H4, head dimension 256, fp16 metadata, group32 and at least 8192 cached tokens also fuses inverse rotation; other shapes keep the existing rotation path. Unsupported inputs, CPU streams and shapeless traces retain ordinary operations. Quantization, stored cache format and serving eligibility are unchanged. Joint-decoder Qwen native/serial and repeated long-context HTTP gates pass. The inverse operation passes both integrated Qwen model gates and six HTTP pairs per quant. MiniCPM/Gemma deferred-consumer serving gates also pass. The shared codec/layout supports ordinary and supported drafting groups without changing this kernel selection. Gemma now owns pre-write row positions across cache appends; the fix passes native checks on both Macs. The combined fused-k8v3/MTP3 pressure fixture passes on M4 and M1 with exact responses/usage and durable SSD writes. Codec policy is resolved once for ordinary, batched and delayed caches; `=0` retains the unfused control. This does not enable TurboQuant quantization itself: `--kv-quant` remains a separate choice. |
 | `MLX_BUN_NO_FUSED_SDPA` | `--fused-sdpa` (inverted) | follows `--kv-quant` | `=1` forces the stock unfused SDPA everywhere. |
 | `MLX_BUN_COMPILED_GEGLU` | `--compiled-activations` | on (`"0"` disables) | Gemma geglu via mlx-lm's `@mx.compile` closure. `=0` → uncompiled composition (same parity, slower). |
 | `MLX_BUN_COMPILED_SWIGLU` | `--compiled-activations` | on (`!== "0"`) | Compiled SwiGLU on MiniCPM5 decode and prefill, matching mlx-lm. An enclosing compiled graph owns its own fusion. qwen3/qwen3.5/universal compile unconditionally. |
@@ -384,14 +470,15 @@ Under `--isolate` the whole environment is inherited by the engine child.
 | `MLX_BUN_TRELLIS` | — | `kernel` (`=expand`) | Packed trellis-coded weights (`mode: "trellis"` modules, Q2b — design: `docs/design/turboquant.md`). `kernel` serves them through the Metal decode kernels (M≤4 matvec; larger M expands one tensor to bf16 and runs a stock matmul). `=expand` decodes every trellis tensor at LOAD into 8-bit g64 affine (+~4 GiB at 27B, the eval-carrier numerics) and serves it through the stock quantized path — the fallback for a machine where the kernels lose. |
 | `MLX_BUN_TRELLIS_VARIANT` | — | `13` | Trellis kernel variant (see [Q2b experiments](../design/turboquant.md)): `6` = code computed inline × reciprocal, weight served as f32 code×scale; `1` adds a residual step and bf16 rounding to reproduce the fake-quant artifact's weights. `0`/`2`/`3`/`4`/`5` are bench-only decoder variants. Variants `7`–`13` retain variant-6 decode values; `13` is the measured default. These variants tune work assignment; `11` tiles eligible short axis-1 prefills, `12` adds split-K axis-0 prefill at M=5..8, and `13` also vectorizes remaining bf16 expansion for k2/k3/k4, T=256, L=12. On MLX 0.32.2/M3+ the axis-1 M5..15 path instead uses direct packed decoding with native wide-matvec arithmetic when Qwen's RMSNorm establishes aligned row-contiguous inputs; callers without that layout proof use native expansion/matmul. Qwen27B additionally selects an integer codebook for its interleaved k3 down projection at M=3/4 with bf16 activations; other calls retain the computed decoder. Prefill rounds weights to the activation dtype. |
 | `MLX_BUN_TRELLIS_ASYNC_EXPAND` | — | off (`=1`) | Experimental variant-13 expansion scheduling. Submit an expanded projection asynchronously while MLX active allocation is below 75% of the device's recommended working set; retain blocking evaluation above it. Other variants are unchanged. Uses the execution's runtime-policy snapshot and preserves the caller's layer barriers. The threshold is not a total-memory cap. Qwen27B integrated native, repeated serial/continuous HTTP and both saved-agent pressure gates pass on M4 Pro 24 GB. Broader-model, combined-optimization and quiet M4 Pro acceptance remain. |
-| `MLX_BUN_PAGED_KV` | `--paged-kv` | off (`=1`) | Paged KV cache; the same refusals and prompt-cache bypass as the flag. |
+| `MLX_BUN_PAGED_KV` | `--paged-kv` | off (`=1`) | Paged KV cache; the same eligibility and separate RAM/SSD namespace as the flag. |
 | `MLX_BUN_ALLOW_PRIVATE_MEDIA` | `--allow-private-media` | off (`=1`) | Permit media fetches to private/loopback/link-local hosts (timeout + size cap still apply). |
 | `MLX_BUN_EXPERT_OFFLOAD` | `--expert-offload` | off | `=<dir>` — the path of a built expert-offload file, activated at module load (`src/expert-offload.ts`) for scripts and library runs that never parse serve flags. The CLI flag builds the file and activates it itself. |
 | `MLX_BUN_PREFILL_TAIL_SPLIT` | — | on (`"0"` disables) | Oracle prefill convention: drain the prompt to len−1, then compute step-0 logits from a separate L=1 forward of the last prompt token (mlx-lm `generate_step` and its batched engine). Both lanes. The spec lane follows its own oracle's shape under the same flag (mlx-lm `speculative_generate_step`: target and draft drain to len−1, no separate step 0). `=0` restores the full-final-chunk convention everywhere — ulp-different at step 0, flips near-tie greedy streams vs mlx-lm. |
 | `MLX_BUN_GRAMMAR` | — | on (`"0"` disables) | `=0` disables structured-output mask compilation; requested constraints take the graceful-degrade route (chat prompt injection + a `Warning` header; raw completions header only). |
 | `MLX_BUN_TOKEN_MASK` | `host` \| `metal` | `host` | Experimental grammar-mask implementation. `metal` uploads packed bits and applies the same additive mask in a compiled shared Metal kernel. Captured when the grammar controller is created; serving performance gates remain open. |
-| `MLX_BUN_GRAMMAR_JUMP` | — | **off** (`=1`) | Jump-forward decoding for structured output (SGLang's technique via xgrammar's `findJumpForwardString`): when the grammar forces a unique continuation, the **serial** lane emits its retokenized ids with one multi-token forward. Lossless in string space and always grammar-valid, but the token stream can legally differ from an unjumped run — no oracle, hence opt-in. Skipped when `logprobs`/`top_logprobs` is requested; SentencePiece-family tokenizers that can't reproduce a mid-stream span never jump. The batch lane does not jump. |
-| `MLX_BUN_FILL` | — | **off** (`=strict` \| `=echo`) | **Token fast-forwarding** — lookup, not speculation. Spans of the assistant turn that the request's `tools` + the chat template already DETERMINE (tool-call open scaffold, the rest of a tool name after its first disambiguating token, a sole-required-key skeleton) are appended as committed context in bounded model forwards, and sampling resumes after them. No draft, no verify, no rollback — an injected token is context, indistinguishable to the model from one it sampled. Rows are compiled per request by diffing probe renderings of the model's OWN template, always sliced from a rendering carrying the REAL tool names and schema keys (so a span can never split a merged token such as Qwen3.5's `=get`), so a template that does not render `tool_calls` yields no rows and no fill. Key and call-close assertions require a closed object schema with exactly one required property and no pattern properties; optional/additional arguments stay available. If no property is required, the compiler preserves the empty-argument choice. Template-derived rows require a fresh request-local parser context. Tool-only output after closed reasoning qualifies; quoted/fenced examples, incomplete delimiters and ambiguous literals decline. Requests with more than 32 tools decline template-derived fills. Held-out identity coverage remains open. **Serial lane only** (it never forces a request off the batch lane) and refused for: fixed `seed`, `logprobs`/`top_logprobs`, structured output (grammar owns forced tokens), image/audio/video prompts, a mounted draft model, quantized/TurboQuant KV, and sliding-window models (one warning). Injection bypasses the sampler — a behavior-policy deviation at `temperature > 0` — hence opt-in. `=echo` additionally arms the **echo index** (Lab tier): spans copied from earlier in the same session, held under policy `verify` — the engine appends them in the same single forward, reads the argmax already in that forward's logits at every span position (free, no extra pass), keeps the prefix the model agrees with, and rewinds the rest through the same cache contract the spec lane's rounds use. A wrong echo costs a rewound forward, never a wrong token. Strict rows stay policy `assert` (no readback, no checkpoint). Telemetry: `usage.fill`. |
+| `MLX_BUN_GRAMMAR_JUMP` | — | **off** (`=1`) | Grammar-supplied continuations. Eligible shared requests without a configured drafter propose xgrammar's forced string to the existing target verifier; the sampler validates every emitted token, including requested logprobs. Candidates leave the matcher unchanged until sampling. Affine/TurboQuant layouts compose through their speculative capabilities; paged requests keep ordinary grammar sampling. Explicit serial execution retains direct jump-forward: it emits retokenized forced spans with one multi-token forward, skips jumping with logprobs, and can choose different tokens than ordinary sampling. A configured drafter retains its own proposal policy. This remains opt-in; tokenization/verification width can change numerical trajectories. |
+| `MLX_BUN_GRAMMAR_DRAFT_TOKENS` | — | `3` | Maximum grammar candidates per shared verification round when `MLX_BUN_GRAMMAR_JUMP=1`. Changes proposal depth, independently of scheduler capacity. |
+| `MLX_BUN_FILL` | — | **off** (`=strict` \| `=echo`) | **Token fast-forwarding.** Strict mode appends tool-call spans determined by the request's schema and chat template, then resumes sampling. Request-local parsing excludes ambiguous literals and incomplete delimiters; closed, single-required-property schemas qualify for key/call-close assertions. Echo mode additionally proposes text copied from the session and verifies it with the request's sampler. Shared execution supports seeded requests and the qualified bf16/affine/TurboQuant row layouts. Homogeneous echo cohorts use the existing grouped target verifier; mixed proposed/unproposed rows retain one-position sampling. Rejected suffixes are rolled back before reusable state is published. Wider verification can change floating-point rounding relative to sequential decode, so sampled text need not be identical. Explicit serial fill still excludes fixed seeds and uses its existing model/cache eligibility. Fill is disabled for logprobs, structured output, media, a mounted draft model and shared paging. Strict appends bypass sampling and therefore change behavior policy at nonzero temperature. Both modes remain opt-in; see the shared-fill section below and `usage.fill`. |
 | `MLX_BUN_FILL_MAX_SPAN` | — | `32` (floor 2) | Hard cap on one injected span. |
 | `MLX_BUN_FILL_APPEND_CHUNK_SIZE` | — | `0` (model limit) | Execution chunk cap for assert-policy fill, captured by each FillSession. Zero selects the model's qualified limit; a positive value can lower that limit. The Qwen 27B text path on `applegpu_g16s`, validated on the M4 Pro, supports up to four positions with supported affine projections and optional Trellis MLPs. It rechecks the limit after each chunk and splits at MLX's attention arithmetic boundaries to preserve one-token numerics. Other configurations use one position per forward. The engine commits the complete span before emission and resumes sampling after it, with no intermediate vocabulary heads or verification. Verify-policy proposals keep a single forward for recurrent rollback. The injected-token cap is unchanged. Library callers can set `FillSession`'s `appendChunkSize` option. Held-out, broader-model and quiet M4 Pro acceptance remain. |
 | `MLX_BUN_FILL_K` | — | `8` (floor 2) | Echo anchor length: the k-gram that must match before a copied span is proposed (the corpus study's token-level threshold). |
@@ -402,9 +489,15 @@ Under `--isolate` the whole environment is inherited by the engine child.
 | `MLX_BUN_BATCH_SSM` | — | on (`"0"` forces serial) | `=0` excludes SSMCache (Qwen3.5 gated-DeltaNet hybrids) from the batch capability gate → those models route serial. |
 | `MLX_BUN_BATCH_EXTEND` | — | on (`"0"` reverts) | Joining rows append to the running batch's KV in one pad+concat (mlx-lm `BatchKVCache.extend`). `=0` reverts to whole-batch re-merge (numerically equivalent, O(B·S)). |
 | `MLX_BUN_BATCH_VEC_SAMPLE` | — | on (`"0"` reverts) | Vectorized greedy batch sampling; `=0` falls back to per-row sampling (bit-equal A/B). |
-| `MLX_BUN_BATCH_NO_PIPELINE` | — | off (`=1`) | Read each batch step's tokens synchronously instead of pipelined (A/B lever; numerically equivalent, slower). Read once at module load. |
-| `MLX_BUN_SSD_WRITEBEHIND` | — | on (`"0"` disables) | `--ssd-cache`'s debounced write-behind snapshot (restart survival). Each tensor write waits for an idle engine, then owns the generation lock for that step. Pending writes do not drain active batches or delay request admission for a full flush. `=0` disables it; eviction and demotion spills still write. |
-| `MLX_BUN_SSD_SPILL_QUEUE_GB` | — | `2` (GiB; any finite value ≥ 0) | Byte cap on pending write-behind spill clones (they pin evicted entries' GPU memory until the generation-locked flush runs). Over cap the oldest queued spill drops and frees immediately; the durability coordinator keeps its snapshot dirty so an explicit or shutdown flush can retry. `=0` keeps only the newest + in-flight clone pinned. Counters in `/stats.ssd_cache`. |
+| `MLX_BUN_BATCH_NO_PIPELINE` | — | off (`=1`) | Read each batch step's tokens synchronously instead of pipelined (A/B lever; numerically equivalent, slower). Captured when the execution group is constructed. |
+| `MLX_BUN_SSD_WRITEBEHIND` | — | on (`"0"` disables) | Proactive persistence after cache publication. A CPU worker packs, hashes and writes immutable state without a generation lock. Writing does not evict RAM. `=0` keeps eviction-triggered persistence; RAM victims remain resident until their SSD copy commits. |
+| `MLX_BUN_SSD_LAYOUT` | — | `whole` | `blocks` writes immutable content-addressed blocks with atomic checkpoint manifests (format 5). Existing whole files remain readable. Shared blocks count once toward an explicit SSD cap and survive deletion of other referencing checkpoints. Experimental space/time tradeoff; see benchmarks. |
+| `MLX_BUN_SSD_SEGMENTED` | — | on (`"0"` disables) | With block storage, read contiguous spans directly and pack arbitrary strides in at most 1 MiB scratch. `=0` retains whole-tensor CPU packing for paired measurements. |
+| `MLX_BUN_SSD_PREFETCH` | — | on (`"0"` disables) | Prepare cold prefixes before execution, using a separate CPU read queue and owned page-aligned allocations. Concurrent identical reads coalesce. Cancellation releases request interest; the cache chooses residency. `=0` retains synchronous restore for comparison. |
+| `MLX_BUN_SESSION_CACHE` | — | on (`"0"` disables) | Honor application session affinity for checkpoint lookup and soft RAM retention. `=0` ignores session metadata for paired HTTP measurements. Full input is still required; see server-api.md for body/header fields and session closure. |
+| `MLX_BUN_CACHE_RETENTION` | — | `lru` | `cost-size` selects GreedyDual-style reuse/size retention, with reused prefix tokens as the processing-cost proxy. This changes RAM residency only; SSD durability remains independent. |
+| `MLX_BUN_PAGED_ATTN` | — | off | With paged KV, `1` selects direct Metal reads for short causal queries (≤8 tokens), bf16 or affine KV4/KV8. Longer prefill and explicit masks use the established SDPA shape dispatch. Lab numerical contract: tolerance-tested against the same stored values, not an L1 bit-exact claim. |
+| `MLX_BUN_SSD_SPILL_QUEUE_GB` | — | `2` (GiB; any finite value ≥ 0) | Pending snapshot cap for optional interrupted-generation checkpoints. Older queued intervals may be superseded; `=0` retains the newest plus any in-flight interval. Normal prompt-history persistence does not use this drop policy; its cache owns RAM residency and SSD demotion. Counters in `/stats.ssd_cache`. |
 | `MLX_BUN_SHUTDOWN_TIMEOUT_MS` | — | `120000` (any finite value > 0) | Maximum time `serve` gives active requests plus the SSD durability flush after `SIGINT`/`SIGTERM`; on timeout it logs the remaining snapshot/spill counters and exits. |
 | `MLX_BUN_DSPARK_MINCONF` | — | checkpoint-driven | Overrides the DSpark/DeepSpec draft scheduler's minimum-confidence threshold (draft-length pruning). Spec lane only ([docs/design/speculative-decoding.md](../design/speculative-decoding.md)). |
 | `MLX_BUN_MEMORY_BATCH` | — | `1` (integer ≥ 1) | Row width for `mlx-bun memory` model calls. 1 = serial (batching measured 1.7–1.9× slower on the heterogeneous extract/chunk workload — [docs/design/dreaming-nightly-pipeline.md](../design/dreaming-nightly-pipeline.md)). |
@@ -427,16 +520,27 @@ Under `--isolate` the whole environment is inherited by the engine child.
 
 | Env var | Value | Effect |
 | --- | --- | --- |
-| `MLX_BUN_P2R_TRACE` | `=1` | Per-request prompt→response phase trace (admission wait, prefill, token-zero, …) for `/v1/chat/completions` and `/v1/completions`; records print to stderr as JSON lines. The trace id is `x-mlx-bun-trace-id` when the request sends it. |
-| `MLX_BUN_P2R_SYNC` | `=1` (with `MLX_BUN_P2R_TRACE`) | Attribution mode: synchronizes the GPU at phase boundaries so each phase is charged its own work (MLX is lazy — without it, unsubmitted work lands in the next phase). Slows the traced request. |
+| `MLX_BUN_P2R_TRACE` | `=1` | Per-request prompt→response phase trace (request-slot wait, execution admission wait, cache prefetch/restore, prefill forward/evaluation/maintenance/checkpoints, mixed-forward token counts, token-zero, bounded initial token routing, response writes) for `/v1/chat/completions` and `/v1/completions`; records print to stderr as JSON lines. The trace id is `x-mlx-bun-trace-id` when the request sends it. |
+| `MLX_BUN_P2R_SYNC` | `=1` (with `MLX_BUN_P2R_TRACE`) | Token-zero attribution mode: synchronizes hidden/cache state, projection and sampling at the instrumented boundaries. This changes overlap and slows the traced request. Ordinary prefill traces add no synchronization: forward includes any backend evaluation, while evaluate measures the remaining state wait. |
+| `MLX_BUN_SPEC_TRACE` | `=1` | Explicit serial speculative-loop round diagnostics, captured once at run setup. |
 | `MLX_BUN_LANE_DEBUG` | `=1` | Logs each request's scheduling placement (`mechanism` + shape) to stderr. |
-| `MLX_BUN_BATCH_STEP_TRACE` | `=1` | Per-step phase timing in the batch scheduler (build / read / emit / gap), read once at module load; summarized by `stepTraceReport()`. |
+| `MLX_BUN_BATCH_STEP_TRACE` | `=1` | Per-step phase timing in the batch scheduler (build / read / emit / gap), captured when the execution group is constructed; summarized by `stepTraceReport()`. |
 | `MLX_BUN_GRAMMAR_DEBUG` | `=1` | Logs per-step grammar row state in the batch scheduler. |
 | `MLX_BUN_PREFILL_MEM_LOG` | `=1` | Logs active/peak memory after each serial prefill chunk. |
 | `MLX_BUN_EXPERT_TRACE` | `=<path>` | Records every MoE router decision as JSONL to that path (adds a per-call GPU→host sync — a measurement tool, not a serving path). |
 | `MLX_BUN_PI_DEBUG` | any non-empty | Extra `[pi-web]` logging (prompt fingerprint, tool/memory surface) for the web-chat pi session. |
 | `MLX_BUN_EVAL_DEBUG` | `=1` | HumanEval: pipe the sandbox's stderr and print failures. |
 
+P2R records include a process-local `startedAtMs` origin to align concurrent
+requests. Prefill `workId` attributes identify shared spans recorded on several
+rows; these spans and their nested children must not be summed as separate
+GPU work. The first eight token-routing spans stop at the first semantic event
+and distinguish hidden channel markers from visible output. With tracing
+enabled, `bench-serve.ts all` retains complete records by child PID in
+`promptResponseTraces`, including records emitted during shutdown. Summarize a
+saved report with `bun scripts/bench/prefill-trace.ts report.md.json --out
+breakdown.json`. Clocks from different server processes are independent.
+
 ### Eval and training
 
 | Env var | Default | Effect |
@@ -506,7 +610,7 @@ GPU (one `AsyncMutex`).
 
 | Request property | Continuous scheduler? |
 | --- | --- |
-| vision / audio / video parts | ❌ serial — offset-0 single-sequence prefill + media masks |
+| vision / audio / video parts | Gemma4 image/audio and Qwen image/video use atomic media prefill followed by shared decode; diffusion retains its separate serial method |
 | LoRA `adapter` (resolves to ≥1) | ✅ identical ordered adapter sets share a group; different sets wait for the next group |
 | `logprobs` / `top_logprobs` | ✅ ordinary and qualified MTP/lookup/standalone-draft groups capture per-request probabilities; other exclusions still apply |
 | explicit `seed` | ✅ request-local random stream; reproducibility also depends on model arithmetic and batch composition |
@@ -517,7 +621,7 @@ GPU (one `AsyncMutex`).
 | structured output (`response_format` / `guided_*`) | ✅ batches — per-row grammar matchers (`MLX_BUN_GRAMMAR_BATCH=0` forces serial) |
 | `temperature` / `top_p` / `top_k` | ✅ batches (each row samples with its own seed) |
 | `stop` sequences | ✅ batches (per-row `StopMatcher`) |
-| `tools` / `tool_choice` | ✅ batches (per-row tool router) |
+| `tools` / `tool_choice` | ✅ batches (per-row tool router); opt-in `MLX_BUN_FILL=strict` uses the shared known-token method |
 | `--thinking` / `enable_thinking` | ✅ batches (template-render concern) |
 | multi-turn / long prompt | ✅ batches with prompt-cache reuse: a joiner restores the longest usable cached prefix and prefills only the suffix (`cached_tokens` reported) |
 
@@ -544,69 +648,85 @@ declared composition may still require the serial mechanism as shown above.
 | Option | serial (`--batch 1`) | `--batch N` (N>1) |
 | --- | --- | --- |
 | `--kv-quant config` | ✅ applied to all requests | ✅ batches where the loaded cache capability supports the per-layer scheme; otherwise routes serial |
-| `--kv-quant 4`/`8` | ✅ applied to all requests | ✅ server start=0 converts during prefill and batches on supported cache layouts; delayed library thresholds still use serial |
+| `--kv-quant 4`/`8` | ✅ applied to all requests | ✅ server start=0 converts during prefill and batches on supported cache layouts; delayed library thresholds use qualified shared affine row layouts |
 | `--kv-quant turbo[:k<bits>v<bits>]` | ✅ ordinary decode; strict serial speculation remains excluded | ✅ ordinary TQ, including delayed library conversion; supported grouped drafting also supports delayed library conversion |
 | `--kv-quant off` / unset | ✅ bf16 (the L1 default) | ✅ bf16 |
-| `--paged-kv` | ✅ | ✅ Gemma4 bf16; prompt-cache bypass |
+| `--paged-kv` | ✅ | ✅ Gemma4 bf16/affine4/8 with separate RAM/SSD namespace; experimental direct attention is opt-in |
 | `--memory-budget` | ✅ per-request admission | ✅ per-request admission — not aggregate (use `--kv-budget`) |
 | `--kv-budget` | n/a | ✅ aggregate queue/reject across rows |
-| `--prompt-cache` / `--ssd-cache` | ✅ prefix reuse + SSD restore | ✅ on both lanes: joiners `take()` at admission; never-merged rows `put()` back |
+| `--prompt-cache` / `--ssd-cache` | ✅ prefix/output reuse + SSD restore | ✅ immutable prefill/output checkpoints, session lookup and queued SSD persistence |
 | `--temperature`/`--top-p`/`--top-k` | ✅ | ✅ (per-row) |
 | `--thinking` | ✅ | ✅ |
-| vision / audio / video request | ✅ | ✅ via serial lane |
+| vision / audio / video request | ✅ | Gemma4 image/audio and Qwen image/video shared; diffusion via its serial method |
 | LoRA `adapter` | ✅ | ✅ compatible groups on adapter-capable backends |
 | `repetition_penalty` / `min_p` / `xtc_*` / `logit_bias` / presence+frequency | ✅ | ✅ (batches — per-row processors) |
 | `seed` | ✅ | ✅ request-local sampling |
 | `tools` / `stop` | ✅ | ✅ (batches) |
 | structured output (`response_format`/`guided_*`) | ✅ (mask in the decode loop) | ✅ (batches; per-row matchers) |
-| `--draft-model` / `--draft-kind` | ✅ eligible spec decode | ✅ qualified MTP/lookup/standalone-draft groups, including grammar/logprobs; other providers remain serial |
-| GLM `--mtp on` | ✅ native MTP spec decode | ⚠️ default-on MTP routes every request serial+spec; `--mtp off` exposes ordinary GLM batching |
+| `--draft-model` / `--draft-kind` | ✅ eligible spec decode | ✅ qualified MTP, lookup, standalone, Gemma assistant/DeepSpec/DSpark groups, including grammar/logprobs |
+| GLM `--mtp on` | ✅ native MTP spec decode | ✅ native MTP uses grouped state/verification; tiny-model and oracle tests pass, final Colibri artifact testing is deferred |
 | `--compiled-decode` | ✅ | ✅ at **B=1 only**: a lone request's adopted serial-class caches replay the same compiled step; B>1 steps run the plain graph |
-| `--fused-sdpa` / `--force-wire` | ✅ (serial decode route) | n/a — compat mode, no perf flags by design |
-
-### `--batch N` is compat mode — perf flags don't apply by design
-
-The bit-parity guarantee (mlx-lm B=N) is the *whole point* of `--batch N`,
-and it requires running the plain forward path. The scheduler
-([batch-scheduler.ts](../../src/serve/batch-scheduler.ts)) drives the
-model through `forwardHidden`/`logitsFromHidden` directly (not
-`generate()`), so:
-
-- **`--compiled-decode`** engages at **B=1 only** (adopt-don't-copy: the
-  lone request's caches stay serial-class, so the scheduler replays the
-  serial engine's compiled step — same kill switch). A second row ⇒ the
-  plain batched graph.
-- **`--fused-sdpa`** never engages in the batched lane (an L2 serial-lane
-  composition).
-- **`--force-wire`** doesn't wire (the scheduler bypasses `generate()`'s
-  wired scope).
-- **Always-on bit-exact kernels still run.** The compiled activations are
-  bit-exact with the spelled-out MLP, so they stay on in both lanes —
-  "compat mode" means *no parity-breaking optionality*, not -O0.
-
-## Known limitations under `--batch N`
-
-Deliberate v1 scope, not bugs:
-
-1. **Other draft providers lack shared state.** Qwen MTP, prompt lookup and
-   standalone drafting reuse aligned prefill and completed-output state through
-   the common RAM/SSD cache. Supported full-attention targets also retain
-   lookup and standalone-draft state. Gemma assistant drafting retains its target hidden
-   and donor KV through the same cache. Other providers start fresh. Long-conversation timing and pressure
-   acceptance remain open.
-2. **Aggregate admission is opt-in** via `--kv-budget`; without it N
-   large-context rows can collectively exceed memory.
-3. **Coverage depends on composition.** Same-B Gemma oracle tests cover
-   sliding-window wrap and late joins. Broader media, resume, paged and method
-   combinations remain open in Phase 18.
-4. **bf16 by contract; mixed-KV batching beyond it.** mlx-lm's batched
-   path *is* bf16. Per-layer `config` batching is a beyond-mlx-lm
-   composition verified per row against the optiq oracle. Batched
-   uniform/TurboQuant now has shared layouts; additional same-B external
-   oracle and feature-composition checks remain.
-5. **`extend` join** appends a joining request to the running batch's
-   full-attention KV in one pad+concat (`MLX_BUN_BATCH_EXTEND=0` reverts
-   to whole-batch re-merge); sliding-window layers still re-merge on join.
+| `--fused-sdpa` | ✅ eligible affine attention | ✅ kernel eligibility depends on dtype, quantization and mask geometry |
+| `--force-wire` | ✅ serial wired scope | not applied by the shared executor |
+
+### Shared execution and kernel settings
+
+The scheduler selects request work. The inference method advances it through
+model-owned numerical operations. Specialized kernels and sampling policies
+remain available wherever their backend capabilities qualify; batching does
+not disable performance settings as a category.
+
+- `--compiled-decode` uses the qualified compiled graph for one active,
+  unpadded request. Multi-row steps use the batched graph.
+- `--fused-sdpa` follows affine kernel eligibility. Padded/array masks can
+  require the ordinary attention operation; this is an attention decision.
+- `--force-wire` controls the explicit serial wired scope. The shared executor
+  does not enter that scope.
+- Trellis, compiled activations, affine row reuse and TurboQuant kernel choices
+  belong to the numerical backend and remain independent of admission policy.
+
+`MLX_BUN_FILL=strict` uses a grouped method for eligible tool requests. Each
+row owns its fill session and sampler history. Known positions bypass sampling;
+when all next positions are known, the method omits the vocabulary head.
+Shared fill uses the model-qualified multi-position append when all active
+rows have known spans and the model permits that geometry. Qualified Qwen B1
+uses its specialized append; wider rows currently advance one position per step. The shared method supports
+bf16, affine KV4/KV8 and supported TurboQuant layouts, including explicit seeds.
+It does not use compiled replay, interrupted-generation checkpoints, paging,
+probability metadata, media, grammar or a mounted draft provider.
+`MLX_BUN_FILL=echo` uses the same shared method with sampled verification and
+row-specific rollback. Ordinary rows can join with empty proposals; eligible
+plain-greedy requests reuse independent verification sampling. With a Qwen MTP
+companion, echo proposals use the speculative method and the provider consumes
+accepted tokens before resuming learned drafting. Asserted template spans remain
+sampled in this combination. Other draft providers are not yet qualified. Fill stays off by default;
+per-model performance acceptance remains separate from numerical replay.
+
+Executors capture an immutable runtime configuration. Later host configuration
+changes do not change an active request's kernel choices. TurboQuant storage
+carries its decode policy through RAM copies, row extraction and delayed
+conversion; restored SSD state is opened under the receiving binding's policy.
+
+## Known limitations under shared execution
+
+1. Gemma4 image/audio uses shared decode after an indivisible embeddings prefill.
+   Qwen image/video positions stay with each request through shared decode; diffusion retains its separate serial method. Strict tool-call fill
+   and echo verification have shared methods. Qwen MTP supports verified echo;
+   asserted fill with a drafter, other providers and shared paging remain unsupported. Grammar has shared masking and opt-in verified
+   proposals; direct serial jump-forward is a different algorithm.
+2. Aggregate admission is opt-in through `--kv-budget`. Without it, concurrent
+   contexts can exceed available memory. Default fit estimates remain advisory.
+3. Native GLM MTP has tiny-model/same-B oracle coverage; final Colibri artifact
+   testing is deferred. DSpark/DFlash execution has fixture coverage, with no
+   trained checkpoint available for a performance claim.
+4. Paged Gemma storage supports bf16/affine4/8. TurboQuant pages, per-layer
+   paged KV and paged speculative methods remain unsupported. Qualified
+   ordinary resume and adapter resume are implemented; exact combinations are
+   tracked in [batching](../design/batching.md).
+5. Stock bf16, affine mixed-KV and TurboQuant retain separate correctness
+   contracts. A passing storage test does not qualify an untested model or
+   numerical geometry. Existing accepted matrices remain in the design and
+   benchmark references.
 
 ## Fidelity tiers and the decode route (`--l1` / `--l2`)
 
@@ -678,14 +798,14 @@ Everything mlx-bun serves, with its default, lane, fidelity tier, and knob.
 | Feature | Default | Lane | Tier | Knob |
 | --- | --- | --- | --- | --- |
 | Structured output (`response_format` json_object/json_schema) | on | both | L2 (oMLX) | request field; `MLX_BUN_GRAMMAR=0` kills |
-| Structured-output jump-forward (grammar-forced spans in one forward) | off | serial only | Lab | `MLX_BUN_GRAMMAR_JUMP=1` |
-| Token fast-forwarding for tool calls (template-determined spans in one forward) | off | serial only | opt-in; identity tested on covered fixtures, parser/held-out gates remain (`tests/parity/fill-strict.test.ts`) | `MLX_BUN_FILL=strict` |
-| Echo injection (session self-copy spans, verified against the same forward's logits) | off | serial only | Lab (paired A/B on task success + wall clock before any default) | `MLX_BUN_FILL=echo`, `MLX_BUN_FILL_K`, `MLX_BUN_FILL_CANDIDATES`, `MLX_BUN_FILL_INDEX_MAX` |
+| Structured-output continuations (shared verified proposals / serial direct jump) | off | shared and serial | Lab | `MLX_BUN_GRAMMAR_JUMP=1` |
+| Token fast-forwarding for tool calls (template-determined spans) | off | serial + shared strict method | opt-in; identity tested on covered fixtures, parser/held-out gates remain (`tests/parity/fill-strict.test.ts`) | `MLX_BUN_FILL=strict` |
+| Echo injection (session self-copy spans, verified against the same forward's logits) | off | shared bf16/affine/TurboQuant; explicit serial eligibility remains separate | Lab (paired A/B on task success + wall clock before any default) | `MLX_BUN_FILL=echo`, `MLX_BUN_FILL_K`, `MLX_BUN_FILL_CANDIDATES`, `MLX_BUN_FILL_INDEX_MAX` |
 | `guided_grammar` (EBNF) / `guided_regex`¹ / `guided_choice` / `structured_outputs` | on | both | L2 | request fields |
 | Structured output × speculative decoding | on when both active | Qwen grouped MTP, lookup and standalone drafting / eligible serial methods | Lab | — |
 | Quantized KV × speculative decoding | Shared Qwen supports uniform KV4/KV8 and TQ with zero or positive conversion thresholds; strict serial supports qualified KV4; per-layer mixed KV remains excluded | Qwen grouped MTP, lookup and standalone drafting / eligible serial methods | scheme and method gates | `MLX_BUN_QWEN_SPEC_KV4=0` disables the KV4 composition |
 | Tool calling (Gemma sentinel / CPM+Qwen XML / GLM `arg_key`+`arg_value`) + `role:"tool"` loops | on | both | — | request `tools` |
-| Vision (`image_url`; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on for models with a tower; SSRF guard on remote URLs | serial | L1/L2 | `--allow-private-media` |
+| Vision (`image_url`; PNG/JPEG/HEIC/AVIF/WebP/TIFF/GIF/BMP) | on for models with a tower; SSRF guard on remote URLs | Gemma4 shared; other media bindings serial | L1/L2 | `--allow-private-media` |
 | Video input (`video_url`/`video`; AVFoundation sidecar, 2 fps, ≤768 frames, 256 MB body cap; never with audio) | on for Qwen3.5-family | serial | mlx-vlm oracle | `--allow-private-media`, `MLX_BUN_FRAME_EXTRACT` |
 | Audio input (`input_audio`/`audio`/`audio_url`; WAV native, mp3/m4a/flac/ogg/aiff via CoreAudio; ≤30 s per clip; mixes with images) | on for models with `audio_config` + sidecar tower (e4b) | serial³ | L2 (greedy stream exact vs optiq's internal model) | `--allow-private-media` |
 | LoRA adapters (mount at start / hot-swap) | off | compatible groups | — | `--adapter <dir>`, `POST /v1/adapters` |
@@ -696,7 +816,7 @@ Everything mlx-bun serves, with its default, lane, fidelity tier, and knob.
 | Stop sequences / streaming / usage accounting | on | both | — | request fields |
 | HLG tone-curve sampling | off | serial | Lab | `--hlg-sampling on` |
 | Spec-decode telemetry (`usage.speculation`) | on with a draft | Qwen shared MTP, lookup and standalone drafting / eligible serial methods | — | — |
-| Token-fast-forwarding telemetry (`usage.fill`) | on with `MLX_BUN_FILL=strict` | serial | — | — |
+| Token-fast-forwarding telemetry (`usage.fill`) | on with `MLX_BUN_FILL=strict` | serial + shared | — | — |
 | Per-turn lane telemetry (`usage.lane`: serial / serial+spec / batched) | on | both | — | — |
 
 ### Model coverage (per-model validated cells)
@@ -707,7 +827,7 @@ Everything mlx-bun serves, with its default, lane, fidelity tier, and knob.
 | Gemma 4 (1B/e4b/12B/26B, + vision e4b/12B, + audio e4b) | ✅ L1/L2 | ✅ | sliding+full interleaved; MoE 26B |
 | Qwen3.5 (gated-DeltaNet hybrid) | ✅ L1/L2 | ✅ (SSM path) | `MLX_BUN_BATCH_SSM=0` reverts |
 | Qwen3.8-27B (same qwen3_5 graph) | ✅ L1 | ✅ (SSM path) | native MTP head via `--draft-model`/`--draft-kind mtp` (lossless-gated; slower on a quiet box — opt-in); images and video serve (mlx-vlm oracle) |
-| GLM-5.2 / Colibri | ✅ chat/text, Messages, Responses, SSE, tools, grammar, logprobs | ✅ compressed MLA/DSA scheduler | native MTP defaults to serial+spec; `--mtp off` enables batching; embeddings, vision/audio, adapters, training unsupported |
+| GLM-5.2 / Colibri | ✅ chat/text, Messages, Responses, SSE, tools, grammar, logprobs | ✅ compressed MLA/DSA scheduler | native MTP shares the executor; final Colibri artifact testing is deferred; embeddings, vision/audio, adapters, training unsupported |
 | DiffusionGemma-26B (non-autoregressive) | ✅ (own engine) | — serial always | first bit-exact non-AR port |
 | Tier-0 universal (llama/qwen2/qwen3/olmo2/…, 11 archs) | ✅ L1 | ✅ plain full-attention archs² | gemma2-family / sliding-window universal → serial |
 
@@ -753,8 +873,8 @@ composition cells in one run).
   On 12B+ add `--draft-model <small-same-tokenizer>`.
 - *Several clients at once (throughput):*
   `mlx-bun serve <model> --batch 4 --ssd-cache <dir> --kv-budget <GB>` —
-  supports qualified start-zero TurboQuant and a grouped Qwen MTP companion.
-  Other draft providers still use serial speculation.
+  supports the qualified grouped methods and cache layouts in the compatibility
+  matrix. Use the default cap eight unless a workload calls for another cap.
 - *UI must never lag / survive engine crashes:* add `--isolate`
   (`--model-pool 2` to keep two models resident).
 - *Reproducibility:* bare / `--l1` (≡ mlx-lm), `--l1 --batch 1` (strict
@@ -763,16 +883,23 @@ composition cells in one run).
   `--memory-budget <GB>` + `--ssd-cache <dir>`; MoE adds
   `--expert-offload`. Start-zero TurboQuant supports shared execution.
 
-**Remaining exclusions:** media, paged KV, generation resume, fill and draft
-providers without grouped implementations still need shared execution work.
+**Remaining exclusions:** fill combined with media, speculation or paging, and unsupported provider/layout
+combinations. Qualified ordinary resume, paged storage and grouped draft
+providers are implemented; see the compatibility matrix above.
 Qwen ordinary, shared MTP and prompt lookup support positive library thresholds for uniform
 affine KV4/KV8 and TurboQuant. Ordinary Gemma supports positive affine KV4/KV8
 thresholds, including sliding-window layers. Other-model delayed affine remains
 open. Qwen MTP
 prefill and completed decode state use the common RAM/SSD cache. Generated
 checkpoints contain only processed tokens; persistence uses the existing
-queue. Strict serial deletion and complete feature parity
-remain acceptance work in Phase 6/18.
+queue. Chat preparation preserves generated token IDs when their decoded text
+exactly prefixes the next rendered request, including provenance recovered from
+SSD headers. This avoids a full prefill when re-encoding identical generated
+text would choose a different BPE segmentation. Edited text retains ordinary
+tokenization. Input token counts can differ from encoding the full prompt again;
+cache state is always matched by exact IDs and execution namespace.
+Strict serial deletion is deferred. Remaining feature and performance work
+is tracked in Phase 6/18.
 
 ## Observability — `GET /stats`