mlx-bun 0.3.0 → 0.5.0

package/docs/reference/server-api.md

@@ -30,6 +30,10 @@ every other path is unique. Unmatched paths return
 | POST | `/v1/messages` | [Anthropic Messages](#post-v1messages-anthropic-messages-api) |
 | POST | `/v1/responses` | [OpenAI Responses](#post-v1responses-openai-responses-api) |
 | POST | `/v1/embeddings` | [Embeddings](#post-v1embeddings-openai-embeddings-api) |
+| POST | `/v1/audio/transcriptions`, `/v1/audio/translations` | [Audio transcription](#post-v1audiotranscriptions--post-v1audiotranslations-speech-to-text) |
+| POST | `/v1/audio/sessions`, `/v1/audio/sessions/<id>/audio`, `/v1/audio/sessions/<id>/finish` | [Streaming sessions](#streaming-transcription-sessions-v1audiosessions) |
+| DELETE | `/v1/audio/sessions/<id>` | [Streaming sessions](#streaming-transcription-sessions-v1audiosessions) |
+| POST | `/admin/transcription/unload` | [Audio transcription](#post-v1audiotranscriptions--post-v1audiotranslations-speech-to-text) |
 | GET | `/v1` | [API index](#get-v1) |
 | GET | `/v1/models`, `/v1/models/<id>` | [Models](#get-v1models) |
 | GET / POST | `/v1/adapters` | [Adapters](#adapters-lora-hot-swap) |
@@ -40,6 +44,7 @@ every other path is unique. Unmatched paths return
 | GET | `/fit` | [Fit](#get-fit) |
 | GET | `/library` | [Library](#get-library) |
 | GET | `/downloads` | [Downloads](#get-downloads) |
+| POST | `/admin/cache/session/close` | [Close cache session](#post-admincachesessionclose) |
 | POST | `/admin/cache/flush` | [SSD cache flush](#post-admincacheflush) |
 | GET | `/api/jobs`, `/api/jobs/<id>`, `/api/jobs/<id>/stream` | [Jobs](#jobs-apijobs) |
 | POST | `/api/finetune/submit`, `/inspect-dataset`, `/merge`, `/export` | [Fine-tune](#fine-tune-apifinetune) |
@@ -117,8 +122,8 @@ Request body (OpenAI chat schema; unknown fields ignored):
   "stream": false,
   "max_tokens": 1024,            // or max_completion_tokens (wins); unset =
                                  // no cap — generate until EOS or the
-                                 // admitted context is exhausted (admission
-                                 // clamps; --max-tokens sets a server default)
+                                 // an explicit operator limit is reached;
+                                 // --max-tokens sets a server default
   "temperature": 0.7,            // 0 = greedy
   "top_p": 0, "top_k": 0,        // 0 = off
   "seed": 1234,                  // omit for time-derived
@@ -369,15 +374,34 @@ Speculative decoding is a **server-level mode** (`serve --draft-model`,
 or the model-free `serve --draft-kind ngram`, which mounts no draft
 model); there is no per-request draft field. The `speculation` usage
 extension appears on chat and text completions alike, non-streaming and on
-the final stream chunk. Spec-eligible requests are text-only on base weights
-(no adapter or logprobs capture, bf16 KV by default); experimental Qwen
-uniform KV4 start=0 can qualify through `MLX_BUN_QWEN_SPEC_KV4=1`.
-Ineligible requests decode normally and omit the field. The spec path bypasses
-the ordinary prompt cache. Qwen MTP with `MLX_BUN_MTP_PROMPT_CACHE=1` can reuse
-a paired target/draft RAM prefix, reported in `cached_tokens`; the default is
-zero. These prefixes do not persist across restarts. While a draft is mounted every request routes
-through the serial lane (mlx_lm.server parity: `is_batchable = draft is
-None`) — speculation and `--batch N` are different modes.
+the final stream chunk. Spec-eligible requests are text-only on base weights.
+In the shared executor, Qwen grouped MTP, prompt lookup and standalone drafting support logprobs and
+bf16, uniform KV4/KV8 or per-layer affine KV, or TurboQuant. Qwen MTP and prompt lookup also support
+positive library conversion thresholds for uniform KV4/KV8. Per-layer mixed KV and
+adapters remain incompatible with speculation. Strict legacy serial execution
+retains its narrower bf16/KV4 policy. The KV4 compatibility control is
+`MLX_BUN_QWEN_SPEC_KV4=0`.
+Ineligible requests decode normally and omit the field. Qwen MTP reuses paired
+target/draft prefill and completed decode state through the shared RAM/SSD
+cache by default in v0.4.0, reported in `cached_tokens`. Configured SSD persistence
+supports reuse after restart; `MLX_BUN_MTP_PROMPT_CACHE=0` disables reuse.
+Generated checkpoints cover processed tokens and use queued SSD persistence.
+Long-conversation performance acceptance remains open. Qwen prompt lookup also
+retains prefill and completed-output state in the common RAM/SSD cache, with
+its committed token history stored as a companion attachment. Standalone drafting
+on Qwen targets stores the draft model’s attention/recurrent state and pending
+last draft token as companion state, including restoration after restart. Qualified Qwen
+MTP, prompt lookup and standalone drafting use shared execution at one or several active rows and
+report `usage.lane: "batched"`. Supported full-attention and rotating targets, including
+Llama, MiniCPM and Gemma, use shared bf16, uniform KV4/KV8 or per-layer affine KV or TurboQuant lookup and standalone drafting, with
+logprobs and generated RAM/SSD checkpoints. Delayed and per-layer affine rotating speculation uses the same row lifecycle; positive conversion thresholds are a library setting, while served schemes default to zero. Gemma assistant drafting also shares execution with bf16, uniform KV4/KV8 or per-layer affine KV or TurboQuant target storage and stores its
+last true target hidden alongside generated RAM/SSD prefixes. Donor attention
+handles row validity independently of the draft graph. DeepSpec and DSpark also use these shared target layouts and the common RAM/SSD cache
+for projected context, with independent confidence-based proposal lengths.
+Other provider/target combinations still
+use the serial executor.
+Prompt-lookup concurrent performance acceptance remains open. Ineligible requests can use continuous ordinary execution
+when their features support it, even while a draft is mounted.
 
 `usage.fill` reports **token fast-forwarding** (`MLX_BUN_FILL=strict|echo`).
 Strict mode appends structural spans from the request's tools and chat
@@ -387,19 +411,45 @@ using its qualified append chunk size, then resumes sampling
 (design: [speculative-decoding.md](../design/speculative-decoding.md) §"Token
 fast-forwarding"). Injected tokens are billed in `completion_tokens` like any
 other generated token; `decodeSteps` counts ordinary sampling steps, so
-`injected / (injected + decodeSteps)` is the share appended without sampling.
+`injected / (injected + decodeSteps)` is the share emitted through fill.
+Strict spans bypass sampling; echo spans still undergo sampled verification.
 Injected positions still run through the model layers. The field is absent when the feature is off or the
-request's shape refuses it (fixed `seed`, `logprobs`, structured output,
-media, a mounted draft model, quantized KV).
+request's shape excludes it, including `logprobs`, structured output or media.
+Strict shared fill supports bf16, affine KV and
+TurboQuant layouts and retains absolute sampling positions for explicit seeds.
+Its `usage.fill` statistics use the same fields as explicit serial execution.
+Known tokens still advance model state. All-known steps skip intermediate
+vocabulary heads and use the model-qualified append geometry, including the
+specialized Qwen B1 append. Wider Qwen rows retain one-position arithmetic. Compiled
+replay, paging and interruption checkpoints remain outside this shared method.
+Explicit serial fill still excludes user-fixed seeds.
 
 Under `MLX_BUN_FILL=echo` a second, weaker source joins: spans copied from
 earlier in the same session, carried under policy `verify`. Those ride the same
-single forward, are checked against the argmax already in its logits, and have
+single forward, are checked by the request's sampler on its logits, and have
 their rejected tail rewound — so `verifyAccepted + verifyRejected` is what the
 echo index proposed and `injected` is what survived. `wastedSamples` counts
 only ASSERT fills (a verify fill consumes the in-flight sample as its first
 position's check instead of discarding it).
 
+Qwen MTP can alternate learned draft rounds with verified echo rounds. The
+provider consumes accepted copied tokens and target context before drafting
+again. Its asserted template spans remain ordinary sampled output. Learned
+draft counts, acceptance histograms and `speculation.rounds` exclude echo;
+`targetCalls`, `tokensPerForward` and `forwardsSaved` cover both round types.
+Echo activity remains in `usage.fill`, including matching prefixes established
+by ordinary MTP samples, so it is not additive with learned-draft counts.
+A request fully served by echoes can
+have zero learned drafts. Other mounted providers do not yet support this
+combination.
+
+An explicit `seed` controls a request-local random stream and can use
+continuous execution. Repeating a request reproduces sampling at the same
+model arithmetic and batch composition. Different batch shapes can change
+floating-point results even with identical random draws; `--batch 1` retains
+the existing fixed-shape control. This differs from mlx-lm's policy of routing
+explicit seeds to its single-request generator.
+
 ### logprobs / top_logprobs
 
 With `"logprobs": true` and/or `"top_logprobs": k` (1–11), the
@@ -432,8 +482,8 @@ penalties), **before** the sampler's temperature/top-p/top-k/min-p/XTC.
 mlx-lm's top-k order is unspecified (argpartition); ours is sorted
 descending — the same set, so the entry is deterministically the argmax.
 Stream chunks never carry logprobs (mlx-lm's streaming responses don't
-either). Requests with logprobs run on the serial lane under `--batch N`
-(like the other mlx-lm sampler extensions). Invalid values are rejected
+either). Requests with logprobs can use ordinary continuous execution under
+`--batch N`; other feature exclusions still apply. Invalid values are rejected
 with mlx-lm's exact messages (see Errors).
 
 ### Streaming
@@ -544,11 +594,10 @@ Design and fidelity notes: [docs/reference/server-api.md](./server-api.md).
 
 ### Errors
 
-Runtime memory pressure can fail a request even after prompt-length admission.
-The serial memory guard first reclaims older prompt-cache entries, persisting
-them to the SSD tier when configured. If insufficient GPU headroom remains,
-the normal request/stream error path reports the failure. Saved in-flight
-generation checkpoints remain available for an identical retry.
+Memory estimates do not reject or shorten requests by default. Actual runtime
+failures use the request/stream error path when recoverable; native Metal
+allocation failure can terminate the process. Saved generation checkpoints
+remain available for an identical retry.
 
 All errors are `{ "error": { "message": …, ... } }`.
 
@@ -567,10 +616,12 @@ All errors are `{ "error": { "message": …, ... } }`.
   'low', 'medium', 'high', 'xhigh'`).
 - `400` with `"type": "memory_admission"`, `"code":
   "context_over_budget"` — the PROMPT itself leaves no generation slot
-  within the max safe context (`src/serve/request-plan.ts`). A prompt that
+  within an explicitly configured context/budget limit (`src/serve/request-plan.ts`). A prompt that
   fits is never rejected for a broad `max_tokens`: the upper bound is
   capped to the remaining room (`max_tokens` is a ceiling, not a promise)
-  and generation proceeds. The ceiling is visible at `/stats`.
+  and generation proceeds. This is opt-in via an operator budget/context cap
+  (or the fixed GLM-5.2 layout), never enforced from default fit estimates.
+  `/stats.admission.enforced_context_tokens` is null when no cap applies.
 - `429` with `"type": "resource_admission"`, `"code": "queue_full"` when
   a preparation or generation queue has 64 waiting requests. If SSE headers
   are already sent, the stream emits its terminal error instead.
@@ -590,7 +641,7 @@ and adapter selection as chat.
   "prompt": "Once upon a time",  // REQUIRED, non-empty string only (token
                                  // arrays rejected, matching mlx_lm.server)
   "max_tokens": 512,             // unset = no cap: generate until EOS or the
-                                 // admitted context (DEVIATION: mlx_lm.server
+                                 // an explicit operator limit (DEVIATION: mlx_lm.server
                                  // defaults 512 — pass --max-tokens 512 to
                                  // reproduce); max_completion_tokens wins
   "stream": false,
@@ -737,6 +788,97 @@ curl localhost:8080/v1/embeddings -H 'content-type: application/json' \
   -d '{"input": ["the cat sat on the mat", "a kitten by the window"]}'
 ```
 
+## POST /v1/audio/transcriptions · POST /v1/audio/translations (speech-to-text)
+
+OpenAI-compatible Whisper transcription (`translations` = the same route with
+task `translate`, English output). Served by every `mlx-bun serve` beside the
+chat model — the Whisper checkpoint (`--whisper-model`, default: the first
+downloaded `whisper` model) loads for each take and is released right after
+it unless `--whisper-idle-unload`/`--whisper-resident` say otherwise — and by the transcription-only server that
+`serve <whisper checkpoint>` starts ([server-config.md](server-config.md)).
+Model support and the parity contract: [models.md](models.md#supported-models);
+CLI equivalent: [`mlx-bun transcribe`](cli.md#transcribe--speech-to-text).
+
+**Request** — `multipart/form-data` (OpenAI's shape) or JSON:
+
+| Field | Meaning |
+| --- | --- |
+| `file` | audio upload (multipart) or base64 / `data:` URL string (JSON; `input_audio.data` and `audio` are accepted aliases). WAV decodes through the exact PCM parser; any other CoreAudio-readable container (mp3/m4a/aac/flac/ogg/aiff/caf, mp4/mov audio tracks…) decodes in-process through AudioToolbox (`src/audio/audiotoolbox.ts`, 3.7 ms for an 11 s mp3). Undecodable bytes are a `400`. |
+| `model` | ignored — the configured Whisper checkpoint is used |
+| `language` | ISO code or name; omitted / `auto` detects from the first 30 s |
+| `prompt` | initial prompt (vocabulary, style); the last 223 tokens are kept |
+| `response_format` | `json` (default) · `verbose_json` · `text` · `srt` · `vtt` |
+| `temperature` | one temperature in [0, 1]; omitted = mlx-whisper's `(0, 0.2, …, 1.0)` fallback ladder |
+| `stream` | `true` → server-sent events (below) |
+| `timestamp_granularities[]` | `segment` (default); `word` adds `words` (`word, start, end, probability`) to `verbose_json` segments via cross-attention DTW on the checkpoint's alignment heads — one extra decoder pass per window |
+| `beam_size` | **non-standard**: beam search width (default greedy); requires temperature 0 |
+| `vocabulary` | **non-standard**: array (JSON) or comma-separated (multipart) recognition hints, fitted whole-term into the 223-token prompt budget after `prompt`; usage is reported |
+| `condition_on_previous_text` | **non-standard**, default `true` |
+| `no_speech_threshold`, `without_timestamps` | **non-standard** mlx-whisper decode options |
+| `vad`, `vad_threshold`, `vad_min_speech_ms`, `vad_trim` | **non-standard**: Silero VAD gate (v6.2, `ggml-org/whisper-vad` weights in the HF cache). No detected speech → `{"text": ""}` with `mlx_bun.vad.speech = false` and Whisper never runs (≈25 ms for 11 s). `vad_trim` crops to the speech span ± 0.5 s first; timestamps stay clip-relative. Segments are reported in `mlx_bun.vad.segments` (seconds). Defaults: threshold 0.5, min speech 250 ms |
+| `faithful` | **non-standard**: `true` runs the oracle-parity graph instead of the fast path (fused attention, compiled decoder step); same tokens up to documented near-ties, slower |
+| `audio_ctx` | **non-standard, Lab**: encode only the first n of 1500 encoder positions (whisper.cpp `-ac`). Cheaper for short clips; measured hallucinated repeats below ~1024 on the fixture clips — off unless set |
+
+**Response** — `json`: `{ "text": "…", "mlx_bun": { "model", "language",
+"duration", "timings": { "load_ms", "transcribe_ms", "total_ms" },
+"vocabulary"?: { "included", "omitted", "token_count", "token_budget" } } }`.
+`verbose_json` adds `task`, `language`, `duration` and openai-whisper's
+`segments` (`id, seek, start, end, text, tokens, temperature, avg_logprob,
+compression_ratio, no_speech_prob`). `text`/`srt`/`vtt` return the bare body
+(`text/plain`, `application/x-subrip`, `text/vtt`) with `x-mlx-bun-language`
+and `x-mlx-bun-timings` headers. `load_ms` is non-zero exactly when the
+request paged the weights in.
+
+**Streaming** (`stream=true`): `text/event-stream` with `transcript.text.delta`
+`{ "delta": "<segment text>", "segment": { "id", "start", "end" } }` per
+finalized segment, `transcript.progress` `{ "done", "total" }` (mel frames),
+and a final `transcript.text.done` carrying the non-streaming body.
+
+**Errors**: `400 invalid_request_error` (fields, undecodable audio, clips
+shorter than 0.1 s), `415` (content type), `499` on client cancel, and
+`503 model_unavailable` when no Whisper checkpoint is on disk (the message
+carries the `mlx-bun get` command).
+
+`POST /admin/transcription/unload` pages the weights out now and returns
+`{ "unloaded": bool, "resident", "loads", "unloads", "requests",
+"last_load_ms", "idle_unload_sec" }` — the same block `/health` and
+`/stats` report as `transcription` on the transcription-only server.
+Transcriptions run one at a time and never overlap chat generation on the
+GPU (the generation gateway's exclusive lock).
+
+```sh
+curl localhost:8080/v1/audio/transcriptions -F file=@meeting.m4a -F language=en -F response_format=srt
+curl localhost:8080/v1/audio/transcriptions -H 'content-type: application/json' \
+  -d '{"file":"<base64 wav>","beam_size":5,"vocabulary":["Sotto","SwiftUI","Metal"]}'
+```
+
+## Streaming transcription sessions (`/v1/audio/sessions`)
+
+For audio that is being recorded while it is sent (dictation). Every
+completed 30 s window is transcribed as soon as it exists, under the same
+exclusive lock as everything else, so `finish` costs one window regardless
+of take length (measured: a 62 s take that needs 1165 ms in batch returns
+327 ms after its last chunk, identical text — M1 Max). With `vad: true` the
+Silero gate runs on the stream: nothing is transcribed until speech has
+been seen, and a take with no speech finishes as `{"text": ""}` without
+ever running Whisper. The one-shot parameters (`language`, `prompt`,
+`vocabulary`, `beam_size`, `temperature`, `vad*`, `timestamp_granularities`,
+`faithful`) apply; `response_format` is `json` or `verbose_json`.
+
+| Route | Body / response |
+| --- | --- |
+| `POST /v1/audio/sessions` | JSON params → `{ "id", "model", "vocabulary" }` (the weights load here if they were paged out) |
+| `POST /v1/audio/sessions/<id>/audio` | one chunk: `content-type: audio/pcm;rate=16000` raw little-endian float32 mono (sotto's format, no decode), or any CoreAudio container. → `{ "samples", "duration", "speech", "segments": [...] }` with every segment finalized so far |
+| `POST /v1/audio/sessions/<id>/finish` | → the one-shot response body (`json`/`verbose_json` with `mlx_bun`), closes the session |
+| `DELETE /v1/audio/sessions/<id>` | discard, `204` |
+
+Chunk order is the caller's responsibility (send sequentially and await
+each receipt). Unknown ids are `404`; a finished session is `409`; at most
+64 sessions are open at once (`429`). Windows fed early are normalized
+against the audio received so far (the log-mel clamp is relative to the
+clip maximum), which can differ from a batch transcription in near-silent
+mel bins — the fixture takes decode identically.
+
 ## GET /v1
 
 API index for discovery tooling:
@@ -744,7 +886,8 @@ API index for discovery tooling:
 ```jsonc
 { "name": "mlx-bun", "version": "<package version>", "model": "<served model id>",
   "endpoints": ["POST /v1/chat/completions", "POST /v1/completions", "POST /v1/messages",
-                "POST /v1/responses", "POST /v1/embeddings", "GET /v1/models",
+                "POST /v1/responses", "POST /v1/embeddings", "POST /v1/audio/transcriptions",
+                "POST /v1/audio/translations", "GET /v1/models",
                 "GET/POST/DELETE /v1/adapters", "GET /health", "GET /stats", "GET /fit",
                 "GET /library", "GET /downloads"] }
 ```
@@ -836,11 +979,11 @@ vision/audio, adapters, and training are not emulated by the serving port.
 ```jsonc
 {
   "server": { "owner": "serve" | "pi-session" | "embedded", "model": "...", "started_at": 0 },
-  "prompt_cache": { "entries": 0, "bytes": 0, "max_bytes": 0, "hits": 0, "misses": 0 },
+  "prompt_cache": { "entries": 0, "bytes": 0, "max_bytes": 0, "hits": 0, "misses": 0, "object_hits": 0, "object_misses": 0, "object_restores": 0 },
   "response_store": { "entries": 0, "bytes": 0, "max_bytes": 33554432, "ttl_ms": 3600000 },
   "kv_quant": { "mode": "bf16" | "uniform-kv8" | "turbo k8v3" | "mixed (kv_config.json)",
                  "layers": { "kv4": 8, "bf16": 40 },     // turbo: { "turbo-k8v3": 8, "bf16": 40 }
-                 "attention": { "global": 10, "sliding_window": 38 } },
+                 "attention": { "global": 10, "sliding_window": 38 }, "recurrent_layers": 0 },
   // present only when --ssd-cache is on:
   "ssd_cache": { "dir": "...", "entries": 0, "bytes": 0, "max_bytes": 0,
                  "restores": 0, "spills": 0, "restore_ms_last": 0, "demotions": 0,
@@ -849,8 +992,9 @@ vision/audio, adapters, and training are not emulated by the serving port.
                  "pending_spill_bytes": 0, "dropped_spills": 0,
                  "failed_spills": 0, "longest_durable_prefix_tokens": 0 },
   "admission": {
-    "max_safe_context": 0,            // tokens; requests above this 400
-    "memory_budget_bytes": null,      // explicit budget, or null (machine default)
+    "max_safe_context": 0,            // advisory fit estimate, tokens
+    "enforced_context_tokens": null,  // explicit cap or fixed layout; null = none
+    "memory_budget_bytes": null,      // explicit budget, or null
     "usable_bytes": 0,
     "weights_bytes": 0
   },
@@ -898,6 +1042,12 @@ appears only when `--kv-quant config` was passed explicitly and the
 checkpoint ships a per-layer `kv_config.json` — per-layer KV quantization
 is never selected automatically.
 
+`kv_quant.layers` counts configured attention KV codecs; `attention` separates
+full and sliding attention. `recurrent_layers` counts linear/recurrent layers
+whose convolution and recurrent state retain their own precision. They do not
+count as bf16 or quantized attention KV. These are model/policy counts, not
+live allocation measurements or a per-request conversion-progress report.
+
 `pending_snapshots` counts prompt-cache entries scheduled for persistence but
 not yet confirmed by the atomic SSD store. `longest_durable_prefix_tokens` is
 the longest prefix already present in the SSD index, not merely queued.
@@ -924,7 +1074,7 @@ generic all-resident SKU estimator is inapplicable
 ```jsonc
 {
   "machine": { "chip": "M4 Pro", "ram_bytes": 0, "bandwidth_gbs": 0.0 },
-  "context_tokens": 8192,          // current admission ceiling
+  "context_tokens": 8192,          // estimated context capacity
   "typical_context_tokens": 8192,  // min(8192, context_tokens)
   "typical_decode_tps": 0.0,       // predicted at typical_context_tokens
   "measured_decode_tps": null,     // real number from eval DB, or null
@@ -959,7 +1109,11 @@ generic all-resident SKU estimator is inapplicable
 ## Adapters (LoRA hot-swap)
 
 Routes in `src/serve/model-admin-routes.ts`; the manager is
-`AdapterManager` in `src/lora.ts`.
+`AdapterManager` in `src/lora.ts`. Requests selecting the same ordered adapter
+set can share a continuous group. Different sets wait for the current group
+to finish and then use the same executor. Prefix-cache identity is resolved
+under the execution lease, so replacing an adapter while a request is queued
+cannot reuse state from its previous weights.
 
 - `GET /v1/adapters` — `{ adapters: [{ id, path, rank, scale, size_bytes,
   mounted_layers, ram_bytes }] }` — currently-mounted adapters only.
@@ -998,6 +1152,50 @@ makes it the default for requests that send no `adapter` field; an
 explicit `adapter` (including `"none"`) always wins, and hot-swap via
 these endpoints is unchanged.
 
+## Cache session affinity
+
+Completion requests may supply `session_id` or `prompt_cache_key` in the JSON
+body, or `x-session-affinity` / `session_id` in a header. Precedence is body
+`session_id`, body `prompt_cache_key`, `x-session-affinity`, then `session_id`.
+This applies to chat completions, text completions, Responses and Messages.
+The shared mlx-bun Pi provider sends session-affinity headers automatically.
+
+A session indexes its latest published checkpoint within each numerical cache
+namespace. Compatible continuations use that checkpoint directly; edited,
+branched or missing histories use ordinary prefix matching. The full intended
+input is still required: a session label does not append omitted messages,
+isolate access, change model settings or bypass token compatibility.
+
+The cache prefers session-referenced checkpoints when choosing RAM residents,
+but can evict them when required. SSD persistence remains independent. Session
+indexes are process-local; after restart, the first request uses ordinary SSD
+lookup and re-establishes the association. Entries remain shareable across
+sessions and requests without session metadata. `MLX_BUN_SESSION_CACHE` provides
+the A/B control described in server-config.md.
+
+`GET /stats` adds `prompt_cache.session_hits`, `session_misses` and
+`prefix_scans`. A session hit counts direct RAM checkpoint selection, including
+state prepared from SSD before execution. A session miss counts execution
+lookups with session metadata that need ordinary candidate search. Prefix scans
+count ordinary execution searches, not tokenization or preparation probes.
+
+
+Qwen image/video encoder reuse reports `prompt_cache.object_hits`,
+`object_misses` and `object_restores`. A hit borrows an exact RAM tensor object;
+a miss may restore from SSD or recompute. Object entries and bytes are included
+in the existing cache totals and share its RAM budget and persistence queue.
+These counters do not count language-model KV prefix hits. Media prompt KV
+reuse remains disabled because token IDs alone do not identify media contents.
+
+
+## POST /admin/cache/session/close
+
+Send `{ "session_id": "agent-42" }` to remove that session's cache association
+and retention preference. Returns `{ "closed": true }` when an ID is supplied,
+or `{ "closed": false }` otherwise. Closing does not delete checkpoints from
+RAM or SSD. Applications can close idle or completed sessions; a later request
+with the same ID establishes a new association.
+
 ## POST /admin/cache/flush
 
 Forces every pending SSD prompt-cache snapshot through the serial write queue
@@ -1621,7 +1819,11 @@ repetition_context_size?, presence_penalty?, frequency_penalty?, seed?}`
 `delete_session {path}`, `regenerate`, `edit_resend {text}`,
 `switch_sibling {entryId}`, `context {context}` (app-aware assistant, below).
 
-Server → client: `ready {model, vision, audio, thinking, genDefaults}`,
+Server → client: `ready {model, vision, audio, thinking, genDefaults, transcription}`
+(`transcription` is true when a Whisper checkpoint serves [`/v1/audio/sessions`](#streaming-transcription-sessions-v1audiosessions);
+the web composer then shows a hold-to-talk mic that streams 16 kHz float32
+into a session and inserts the transcript into the message box — voice
+input for any chat model),
 `turn_start`, `text_delta {delta}`, `thinking_delta {delta}`, `tool_start
 {callId, tool, args}`, `tool_approval_request {callId, tool, args}`,
 `tool_update {callId, chunk}`, `tool_end {callId, ok, result}`, `turn_end
@@ -1719,3 +1921,25 @@ Under `--isolate`, the parent owns Responses continuation history and exposes it
 bounded store counters in `GET /engine` as `response_store`. Model-worker restart
 or eviction preserves that history; restarting the parent clears it. Direct
 serving retains the same process-local TTL and byte limits.
+
+
+Supported grouped TurboQuant drafting retains `usage.speculation`, seeded
+sampling, logprobs, structured output and generated RAM/SSD continuations.
+The cache codec does not select the scheduler or sampler implementation.
+Composition coverage: [batching design](../design/batching.md).
+
+With paged KV enabled, supported Gemma4 bf16 text requests report the batched lane when submitted through continuous scheduling. Seeded sampling, logprobs and grammar use the same interfaces. These requests bypass prompt-cache lookup/publication; paging does not provide SSD persistence.
+
+Ordinary adapter checkpoints are isolated by ordered adapter IDs and mounted
+content revisions. Base-model requests cannot restore or remove an adapter checkpoint.
+With generation checkpointing enabled, repeating an identical eligible request
+replays its saved completion prefix and resumes from the saved pending token.
+Cancellation retains queued snapshots; successful completion removes them.
+`POST /admin/cache/flush` drains both prompt spills and generation checkpoints,
+and its elapsed time and durability statistics include both queues. A hard
+process exit can lose snapshots that have not drained.
+
+For resumed generation checkpoints, `usage.prompt_tokens_details.cached_tokens`
+counts only the reused original prompt, bounded by `usage.prompt_tokens`. Saved
+completion tokens are replayed as completion output and do not increase that
+prompt-cache usage counter.