mlx-bun 0.4.0 → 0.5.0

package/docs/reference/server-api.md

@@ -30,6 +30,10 @@ every other path is unique. Unmatched paths return
 | POST | `/v1/messages` | [Anthropic Messages](#post-v1messages-anthropic-messages-api) |
 | POST | `/v1/responses` | [OpenAI Responses](#post-v1responses-openai-responses-api) |
 | POST | `/v1/embeddings` | [Embeddings](#post-v1embeddings-openai-embeddings-api) |
+| POST | `/v1/audio/transcriptions`, `/v1/audio/translations` | [Audio transcription](#post-v1audiotranscriptions--post-v1audiotranslations-speech-to-text) |
+| POST | `/v1/audio/sessions`, `/v1/audio/sessions/<id>/audio`, `/v1/audio/sessions/<id>/finish` | [Streaming sessions](#streaming-transcription-sessions-v1audiosessions) |
+| DELETE | `/v1/audio/sessions/<id>` | [Streaming sessions](#streaming-transcription-sessions-v1audiosessions) |
+| POST | `/admin/transcription/unload` | [Audio transcription](#post-v1audiotranscriptions--post-v1audiotranslations-speech-to-text) |
 | GET | `/v1` | [API index](#get-v1) |
 | GET | `/v1/models`, `/v1/models/<id>` | [Models](#get-v1models) |
 | GET / POST | `/v1/adapters` | [Adapters](#adapters-lora-hot-swap) |
@@ -40,6 +44,7 @@ every other path is unique. Unmatched paths return
 | GET | `/fit` | [Fit](#get-fit) |
 | GET | `/library` | [Library](#get-library) |
 | GET | `/downloads` | [Downloads](#get-downloads) |
+| POST | `/admin/cache/session/close` | [Close cache session](#post-admincachesessionclose) |
 | POST | `/admin/cache/flush` | [SSD cache flush](#post-admincacheflush) |
 | GET | `/api/jobs`, `/api/jobs/<id>`, `/api/jobs/<id>/stream` | [Jobs](#jobs-apijobs) |
 | POST | `/api/finetune/submit`, `/inspect-dataset`, `/merge`, `/export` | [Fine-tune](#fine-tune-apifinetune) |
@@ -370,15 +375,15 @@ or the model-free `serve --draft-kind ngram`, which mounts no draft
 model); there is no per-request draft field. The `speculation` usage
 extension appears on chat and text completions alike, non-streaming and on
 the final stream chunk. Spec-eligible requests are text-only on base weights.
-In the unreleased shared executor, Qwen grouped MTP, prompt lookup and standalone drafting support logprobs and
+In the shared executor, Qwen grouped MTP, prompt lookup and standalone drafting support logprobs and
 bf16, uniform KV4/KV8 or per-layer affine KV, or TurboQuant. Qwen MTP and prompt lookup also support
 positive library conversion thresholds for uniform KV4/KV8. Per-layer mixed KV and
 adapters remain incompatible with speculation. Strict legacy serial execution
 retains its narrower bf16/KV4 policy. The KV4 compatibility control is
 `MLX_BUN_QWEN_SPEC_KV4=0`.
 Ineligible requests decode normally and omit the field. Qwen MTP reuses paired
-target/draft prefill and completed decode state through the shared RAM/SSD cache by default in the
-unreleased source, reported in `cached_tokens`. Configured SSD persistence
+target/draft prefill and completed decode state through the shared RAM/SSD
+cache by default in v0.4.0, reported in `cached_tokens`. Configured SSD persistence
 supports reuse after restart; `MLX_BUN_MTP_PROMPT_CACHE=0` disables reuse.
 Generated checkpoints cover processed tokens and use queued SSD persistence.
 Long-conversation performance acceptance remains open. Qwen prompt lookup also
@@ -406,19 +411,38 @@ using its qualified append chunk size, then resumes sampling
 (design: [speculative-decoding.md](../design/speculative-decoding.md) §"Token
 fast-forwarding"). Injected tokens are billed in `completion_tokens` like any
 other generated token; `decodeSteps` counts ordinary sampling steps, so
-`injected / (injected + decodeSteps)` is the share appended without sampling.
+`injected / (injected + decodeSteps)` is the share emitted through fill.
+Strict spans bypass sampling; echo spans still undergo sampled verification.
 Injected positions still run through the model layers. The field is absent when the feature is off or the
-request's shape refuses it (fixed `seed`, `logprobs`, structured output,
-media, a mounted draft model, quantized KV).
+request's shape excludes it, including `logprobs`, structured output or media.
+Strict shared fill supports bf16, affine KV and
+TurboQuant layouts and retains absolute sampling positions for explicit seeds.
+Its `usage.fill` statistics use the same fields as explicit serial execution.
+Known tokens still advance model state. All-known steps skip intermediate
+vocabulary heads and use the model-qualified append geometry, including the
+specialized Qwen B1 append. Wider Qwen rows retain one-position arithmetic. Compiled
+replay, paging and interruption checkpoints remain outside this shared method.
+Explicit serial fill still excludes user-fixed seeds.
 
 Under `MLX_BUN_FILL=echo` a second, weaker source joins: spans copied from
 earlier in the same session, carried under policy `verify`. Those ride the same
-single forward, are checked against the argmax already in its logits, and have
+single forward, are checked by the request's sampler on its logits, and have
 their rejected tail rewound — so `verifyAccepted + verifyRejected` is what the
 echo index proposed and `injected` is what survived. `wastedSamples` counts
 only ASSERT fills (a verify fill consumes the in-flight sample as its first
 position's check instead of discarding it).
 
+Qwen MTP can alternate learned draft rounds with verified echo rounds. The
+provider consumes accepted copied tokens and target context before drafting
+again. Its asserted template spans remain ordinary sampled output. Learned
+draft counts, acceptance histograms and `speculation.rounds` exclude echo;
+`targetCalls`, `tokensPerForward` and `forwardsSaved` cover both round types.
+Echo activity remains in `usage.fill`, including matching prefixes established
+by ordinary MTP samples, so it is not additive with learned-draft counts.
+A request fully served by echoes can
+have zero learned drafts. Other mounted providers do not yet support this
+combination.
+
 An explicit `seed` controls a request-local random stream and can use
 continuous execution. Repeating a request reproduces sampling at the same
 model arithmetic and batch composition. Different batch shapes can change
@@ -764,6 +788,97 @@ curl localhost:8080/v1/embeddings -H 'content-type: application/json' \
   -d '{"input": ["the cat sat on the mat", "a kitten by the window"]}'
 ```
 
+## POST /v1/audio/transcriptions · POST /v1/audio/translations (speech-to-text)
+
+OpenAI-compatible Whisper transcription (`translations` = the same route with
+task `translate`, English output). Served by every `mlx-bun serve` beside the
+chat model — the Whisper checkpoint (`--whisper-model`, default: the first
+downloaded `whisper` model) loads for each take and is released right after
+it unless `--whisper-idle-unload`/`--whisper-resident` say otherwise — and by the transcription-only server that
+`serve <whisper checkpoint>` starts ([server-config.md](server-config.md)).
+Model support and the parity contract: [models.md](models.md#supported-models);
+CLI equivalent: [`mlx-bun transcribe`](cli.md#transcribe--speech-to-text).
+
+**Request** — `multipart/form-data` (OpenAI's shape) or JSON:
+
+| Field | Meaning |
+| --- | --- |
+| `file` | audio upload (multipart) or base64 / `data:` URL string (JSON; `input_audio.data` and `audio` are accepted aliases). WAV decodes through the exact PCM parser; any other CoreAudio-readable container (mp3/m4a/aac/flac/ogg/aiff/caf, mp4/mov audio tracks…) decodes in-process through AudioToolbox (`src/audio/audiotoolbox.ts`, 3.7 ms for an 11 s mp3). Undecodable bytes are a `400`. |
+| `model` | ignored — the configured Whisper checkpoint is used |
+| `language` | ISO code or name; omitted / `auto` detects from the first 30 s |
+| `prompt` | initial prompt (vocabulary, style); the last 223 tokens are kept |
+| `response_format` | `json` (default) · `verbose_json` · `text` · `srt` · `vtt` |
+| `temperature` | one temperature in [0, 1]; omitted = mlx-whisper's `(0, 0.2, …, 1.0)` fallback ladder |
+| `stream` | `true` → server-sent events (below) |
+| `timestamp_granularities[]` | `segment` (default); `word` adds `words` (`word, start, end, probability`) to `verbose_json` segments via cross-attention DTW on the checkpoint's alignment heads — one extra decoder pass per window |
+| `beam_size` | **non-standard**: beam search width (default greedy); requires temperature 0 |
+| `vocabulary` | **non-standard**: array (JSON) or comma-separated (multipart) recognition hints, fitted whole-term into the 223-token prompt budget after `prompt`; usage is reported |
+| `condition_on_previous_text` | **non-standard**, default `true` |
+| `no_speech_threshold`, `without_timestamps` | **non-standard** mlx-whisper decode options |
+| `vad`, `vad_threshold`, `vad_min_speech_ms`, `vad_trim` | **non-standard**: Silero VAD gate (v6.2, `ggml-org/whisper-vad` weights in the HF cache). No detected speech → `{"text": ""}` with `mlx_bun.vad.speech = false` and Whisper never runs (≈25 ms for 11 s). `vad_trim` crops to the speech span ± 0.5 s first; timestamps stay clip-relative. Segments are reported in `mlx_bun.vad.segments` (seconds). Defaults: threshold 0.5, min speech 250 ms |
+| `faithful` | **non-standard**: `true` runs the oracle-parity graph instead of the fast path (fused attention, compiled decoder step); same tokens up to documented near-ties, slower |
+| `audio_ctx` | **non-standard, Lab**: encode only the first n of 1500 encoder positions (whisper.cpp `-ac`). Cheaper for short clips; measured hallucinated repeats below ~1024 on the fixture clips — off unless set |
+
+**Response** — `json`: `{ "text": "…", "mlx_bun": { "model", "language",
+"duration", "timings": { "load_ms", "transcribe_ms", "total_ms" },
+"vocabulary"?: { "included", "omitted", "token_count", "token_budget" } } }`.
+`verbose_json` adds `task`, `language`, `duration` and openai-whisper's
+`segments` (`id, seek, start, end, text, tokens, temperature, avg_logprob,
+compression_ratio, no_speech_prob`). `text`/`srt`/`vtt` return the bare body
+(`text/plain`, `application/x-subrip`, `text/vtt`) with `x-mlx-bun-language`
+and `x-mlx-bun-timings` headers. `load_ms` is non-zero exactly when the
+request paged the weights in.
+
+**Streaming** (`stream=true`): `text/event-stream` with `transcript.text.delta`
+`{ "delta": "<segment text>", "segment": { "id", "start", "end" } }` per
+finalized segment, `transcript.progress` `{ "done", "total" }` (mel frames),
+and a final `transcript.text.done` carrying the non-streaming body.
+
+**Errors**: `400 invalid_request_error` (fields, undecodable audio, clips
+shorter than 0.1 s), `415` (content type), `499` on client cancel, and
+`503 model_unavailable` when no Whisper checkpoint is on disk (the message
+carries the `mlx-bun get` command).
+
+`POST /admin/transcription/unload` pages the weights out now and returns
+`{ "unloaded": bool, "resident", "loads", "unloads", "requests",
+"last_load_ms", "idle_unload_sec" }` — the same block `/health` and
+`/stats` report as `transcription` on the transcription-only server.
+Transcriptions run one at a time and never overlap chat generation on the
+GPU (the generation gateway's exclusive lock).
+
+```sh
+curl localhost:8080/v1/audio/transcriptions -F file=@meeting.m4a -F language=en -F response_format=srt
+curl localhost:8080/v1/audio/transcriptions -H 'content-type: application/json' \
+  -d '{"file":"<base64 wav>","beam_size":5,"vocabulary":["Sotto","SwiftUI","Metal"]}'
+```
+
+## Streaming transcription sessions (`/v1/audio/sessions`)
+
+For audio that is being recorded while it is sent (dictation). Every
+completed 30 s window is transcribed as soon as it exists, under the same
+exclusive lock as everything else, so `finish` costs one window regardless
+of take length (measured: a 62 s take that needs 1165 ms in batch returns
+327 ms after its last chunk, identical text — M1 Max). With `vad: true` the
+Silero gate runs on the stream: nothing is transcribed until speech has
+been seen, and a take with no speech finishes as `{"text": ""}` without
+ever running Whisper. The one-shot parameters (`language`, `prompt`,
+`vocabulary`, `beam_size`, `temperature`, `vad*`, `timestamp_granularities`,
+`faithful`) apply; `response_format` is `json` or `verbose_json`.
+
+| Route | Body / response |
+| --- | --- |
+| `POST /v1/audio/sessions` | JSON params → `{ "id", "model", "vocabulary" }` (the weights load here if they were paged out) |
+| `POST /v1/audio/sessions/<id>/audio` | one chunk: `content-type: audio/pcm;rate=16000` raw little-endian float32 mono (sotto's format, no decode), or any CoreAudio container. → `{ "samples", "duration", "speech", "segments": [...] }` with every segment finalized so far |
+| `POST /v1/audio/sessions/<id>/finish` | → the one-shot response body (`json`/`verbose_json` with `mlx_bun`), closes the session |
+| `DELETE /v1/audio/sessions/<id>` | discard, `204` |
+
+Chunk order is the caller's responsibility (send sequentially and await
+each receipt). Unknown ids are `404`; a finished session is `409`; at most
+64 sessions are open at once (`429`). Windows fed early are normalized
+against the audio received so far (the log-mel clamp is relative to the
+clip maximum), which can differ from a batch transcription in near-silent
+mel bins — the fixture takes decode identically.
+
 ## GET /v1
 
 API index for discovery tooling:
@@ -771,7 +886,8 @@ API index for discovery tooling:
 ```jsonc
 { "name": "mlx-bun", "version": "<package version>", "model": "<served model id>",
   "endpoints": ["POST /v1/chat/completions", "POST /v1/completions", "POST /v1/messages",
-                "POST /v1/responses", "POST /v1/embeddings", "GET /v1/models",
+                "POST /v1/responses", "POST /v1/embeddings", "POST /v1/audio/transcriptions",
+                "POST /v1/audio/translations", "GET /v1/models",
                 "GET/POST/DELETE /v1/adapters", "GET /health", "GET /stats", "GET /fit",
                 "GET /library", "GET /downloads"] }
 ```
@@ -863,11 +979,11 @@ vision/audio, adapters, and training are not emulated by the serving port.
 ```jsonc
 {
   "server": { "owner": "serve" | "pi-session" | "embedded", "model": "...", "started_at": 0 },
-  "prompt_cache": { "entries": 0, "bytes": 0, "max_bytes": 0, "hits": 0, "misses": 0 },
+  "prompt_cache": { "entries": 0, "bytes": 0, "max_bytes": 0, "hits": 0, "misses": 0, "object_hits": 0, "object_misses": 0, "object_restores": 0 },
   "response_store": { "entries": 0, "bytes": 0, "max_bytes": 33554432, "ttl_ms": 3600000 },
   "kv_quant": { "mode": "bf16" | "uniform-kv8" | "turbo k8v3" | "mixed (kv_config.json)",
                  "layers": { "kv4": 8, "bf16": 40 },     // turbo: { "turbo-k8v3": 8, "bf16": 40 }
-                 "attention": { "global": 10, "sliding_window": 38 } },
+                 "attention": { "global": 10, "sliding_window": 38 }, "recurrent_layers": 0 },
   // present only when --ssd-cache is on:
   "ssd_cache": { "dir": "...", "entries": 0, "bytes": 0, "max_bytes": 0,
                  "restores": 0, "spills": 0, "restore_ms_last": 0, "demotions": 0,
@@ -926,6 +1042,12 @@ appears only when `--kv-quant config` was passed explicitly and the
 checkpoint ships a per-layer `kv_config.json` — per-layer KV quantization
 is never selected automatically.
 
+`kv_quant.layers` counts configured attention KV codecs; `attention` separates
+full and sliding attention. `recurrent_layers` counts linear/recurrent layers
+whose convolution and recurrent state retain their own precision. They do not
+count as bf16 or quantized attention KV. These are model/policy counts, not
+live allocation measurements or a per-request conversion-progress report.
+
 `pending_snapshots` counts prompt-cache entries scheduled for persistence but
 not yet confirmed by the atomic SSD store. `longest_durable_prefix_tokens` is
 the longest prefix already present in the SSD index, not merely queued.
@@ -1030,6 +1152,50 @@ makes it the default for requests that send no `adapter` field; an
 explicit `adapter` (including `"none"`) always wins, and hot-swap via
 these endpoints is unchanged.
 
+## Cache session affinity
+
+Completion requests may supply `session_id` or `prompt_cache_key` in the JSON
+body, or `x-session-affinity` / `session_id` in a header. Precedence is body
+`session_id`, body `prompt_cache_key`, `x-session-affinity`, then `session_id`.
+This applies to chat completions, text completions, Responses and Messages.
+The shared mlx-bun Pi provider sends session-affinity headers automatically.
+
+A session indexes its latest published checkpoint within each numerical cache
+namespace. Compatible continuations use that checkpoint directly; edited,
+branched or missing histories use ordinary prefix matching. The full intended
+input is still required: a session label does not append omitted messages,
+isolate access, change model settings or bypass token compatibility.
+
+The cache prefers session-referenced checkpoints when choosing RAM residents,
+but can evict them when required. SSD persistence remains independent. Session
+indexes are process-local; after restart, the first request uses ordinary SSD
+lookup and re-establishes the association. Entries remain shareable across
+sessions and requests without session metadata. `MLX_BUN_SESSION_CACHE` provides
+the A/B control described in server-config.md.
+
+`GET /stats` adds `prompt_cache.session_hits`, `session_misses` and
+`prefix_scans`. A session hit counts direct RAM checkpoint selection, including
+state prepared from SSD before execution. A session miss counts execution
+lookups with session metadata that need ordinary candidate search. Prefix scans
+count ordinary execution searches, not tokenization or preparation probes.
+
+
+Qwen image/video encoder reuse reports `prompt_cache.object_hits`,
+`object_misses` and `object_restores`. A hit borrows an exact RAM tensor object;
+a miss may restore from SSD or recompute. Object entries and bytes are included
+in the existing cache totals and share its RAM budget and persistence queue.
+These counters do not count language-model KV prefix hits. Media prompt KV
+reuse remains disabled because token IDs alone do not identify media contents.
+
+
+## POST /admin/cache/session/close
+
+Send `{ "session_id": "agent-42" }` to remove that session's cache association
+and retention preference. Returns `{ "closed": true }` when an ID is supplied,
+or `{ "closed": false }` otherwise. Closing does not delete checkpoints from
+RAM or SSD. Applications can close idle or completed sessions; a later request
+with the same ID establishes a new association.
+
 ## POST /admin/cache/flush
 
 Forces every pending SSD prompt-cache snapshot through the serial write queue
@@ -1653,7 +1819,11 @@ repetition_context_size?, presence_penalty?, frequency_penalty?, seed?}`
 `delete_session {path}`, `regenerate`, `edit_resend {text}`,
 `switch_sibling {entryId}`, `context {context}` (app-aware assistant, below).
 
-Server → client: `ready {model, vision, audio, thinking, genDefaults}`,
+Server → client: `ready {model, vision, audio, thinking, genDefaults, transcription}`
+(`transcription` is true when a Whisper checkpoint serves [`/v1/audio/sessions`](#streaming-transcription-sessions-v1audiosessions);
+the web composer then shows a hold-to-talk mic that streams 16 kHz float32
+into a session and inserts the transcript into the message box — voice
+input for any chat model),
 `turn_start`, `text_delta {delta}`, `thinking_delta {delta}`, `tool_start
 {callId, tool, args}`, `tool_approval_request {callId, tool, args}`,
 `tool_update {callId, chunk}`, `tool_end {callId, ok, result}`, `turn_end