mlx-bun 0.0.12 → 0.2.0

package/docs/design/mlx-lm-serving-execution-seam.md

@@ -0,0 +1,81 @@
+# mlx-lm serving execution seam
+
+## Question
+
+What should mlx-bun borrow from the pinned mlx-lm oracle when deepening Completion execution around this flow?
+
+```text
+prepared prompt -> semantic events -> terminal summary
+```
+
+This note reads the local mlx-lm 0.31.3 source installed in the reference venv. The comparison uses the serving term defined in `CONTEXT.md`: a **Completion execution** includes output, resource lifetime, usage, and scheduling placement.
+
+mlx-lm is the correctness oracle only for behavior and numeric schemes it implements. It is not mlx-bun's architecture oracle or capability ceiling. mlx-bun also implements explicitly selected features such as mixed-precision KV, TurboQuant, and native draft methods. The execution path must preserve those resolved settings exactly.
+
+## Implementation status
+
+The first consolidation slice landed on 2026-08-21. `prepareCompletion()` performs admission before a streaming response starts and returns an opaque, single-use `PreparedCompletion`. `CompletionExecutor.execute()` then owns placement reporting, `CompletionSink`, logprobs, semantic event delivery, finish reason, usage, and pre-generation cleanup. [`completion-executor.ts:75`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:75) [`completion-executor.ts:219`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:219)
+
+Chat and raw-text adapters still declare their semantic components and format their own wire responses. Their streaming and non-streaming branches now call the same executor. `GenerationGateway` still owns placement and model execution. The change does not add or remove flags, alter defaults, or substitute one inference method for another.
+
+S3 completed the scheduling declaration at the same seam. A
+`GenerationPlacement` now names `serial` or `continuous`; it no longer calls a
+lone scheduler request "batched." The continuous scheduler chooses its B=1
+fast path or B=N step from active rows. The strict serial executor remains for
+`--batch 1` and for declared compositions the continuous scheduler does not yet
+implement. Existing parity evidence covers both the default scheduler's B=1
+path and strict serial. The recorded scheduler/serial decode ratios are
+0.992–0.996, and a fresh main-vs-branch serial gate after the sampler
+consolidation measured a 1.000 median-best ratio (about 265.0 tok/s on both
+sides, MiniCPM5, two interleaved 256-token repeats).
+
+Final acceptance after the GPQA pause exercised the seam through the complete
+two-shard suite and a real two-turn browser conversation. The suite passed
+2,064 tests with zero failures. The browser run verified streaming, tool
+events, context carryover, metrics, and turn recovery. A mainline-only queued
+DOM render race found by that run was fixed at the UI lifecycle boundary; no
+decode-loop work was added. The architecture changes likewise keep accounting
+and capability decisions outside numerical kernels. The fresh serial A/B plus
+the established scheduler B=1 parity and latency evidence are the performance
+gate.
+
+## What the oracle actually does
+
+The HTTP adapter creates a small `CompletionRequest` that still contains protocol-shaped chat messages or a raw text prompt. It separately builds `GenerationArguments`. Prompt preparation has not happened yet. [`server.py:182`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:182) [`server.py:198`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:198) [`server.py:1368`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1368)
+
+`ResponseGenerator.generate()` submits that pair to one worker thread through a queue. The worker applies the chat template, tokenizes, finds cache segments, chooses the initial reasoning state, constructs stop/tool/reasoning transitions, selects a lane, owns cache work, and emits per-token `Response` records. [`server.py:516`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:516) [`server.py:626`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:626) [`server.py:688`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:688) [`server.py:1026`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1026)
+
+That organization can be summarized as:
+
+```text
+HTTP request
+  -> CompletionRequest + GenerationArguments
+  -> ResponseGenerator queue
+  -> tokenize + state machine + lane + cache
+  -> token Response { text, token, state, match, logprob, finish_reason }
+  -> HTTP handler accumulates semantics and formats the protocol response
+```
+
+The model loader is separate. `utils.load()` returns the model and a tokenizer wrapper, while tokenizer loading chooses the streaming detokenizer and tool parser. This keeps model construction out of per-request execution. [`utils.py:453`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/utils.py:453) [`tokenizer_utils.py:578`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/tokenizer_utils.py:578)
+
+## Comparison by concern
+
+| Concern | Oracle organization | Consequence for the mlx-bun seam |
+| --- | --- | --- |
+| Prepared prompt | The worker owns chat templating, tokenization, initial reasoning state, and cache-oriented prompt segmentation. The HTTP adapter does not know those details. [`server.py:530`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:530) [`server.py:565`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:565) | A prepared-prompt value is a sound seam, but it should be richer than token ids. It must carry the generation inputs and owned media or grammar state needed by either lane. Protocol adapters should not assemble `RequestShape` or transfer resources piecemeal. |
+| Semantic events | The state machine classifies tokens as `normal`, `reasoning`, or `tool`. `_process_control_tokens` withholds matched control sequences. The HTTP handler then accumulates reasoning, tool text, visible text, and parses completed tool calls. [`server.py:236`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:236) [`server.py:654`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:654) [`server.py:1442`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1442) | Do not copy this split. mlx-bun's `CompletionSink` already emits content, reasoning, and tool-call events through one interface. [`completion-sink.ts:7`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-sink.ts:7) [`completion-sink.ts:90`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-sink.ts:90) The deepened module should own that semantic work so all protocol adapters consume the same events. |
+| Terminal summary | `generate.stream_generate()` puts timing, token counts, memory, and finish reason on every low-level response, but the server discards most of it. The HTTP handler counts emitted tokens, reads prompt and cache counts from `GenerationContext`, derives tool-call finish, and builds usage. [`generate.py:269`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/generate.py:269) [`generate.py:728`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/generate.py:728) [`server.py:1495`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1495) [`server.py:1517`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1517) | The oracle has no single terminal summary at the server seam. mlx-bun should add one. It should be the sole source of finish reason, prompt/generated/cached counts, final lane, speculation facts, and collected result. |
+| Streaming and non-streaming | Both modes consume the same response iterator and semantic accumulation loop. Streaming flushes deltas and optionally emits a final usage packet; non-streaming sends one accumulated response. [`server.py:1454`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1454) [`server.py:1507`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1507) [`server.py:1529`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1529) | Keep one Completion execution interface. Streaming and non-streaming are protocol adapters over the same semantic events and terminal summary, not separate execution paths. |
+| Scheduling placement | Model loading marks a model batchable only when no draft model exists and every cache supports merge. Per-request placement also excludes a fixed seed. The worker joins the active batch, creates a batch, runs serially, or drains an incompatible batch. [`server.py:370`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:370) [`server.py:685`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:685) [`server.py:731`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:731) [`server.py:801`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:801) [`server.py:831`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:831) | `RequestShape` describes settings already resolved from flags, request fields, model metadata, and defaults. `place()` asks only whether the continuous scheduler implements that exact composition; it never replaces MTP, a KV scheme, or another requested method. The frozen declaration names the `serial` or `continuous` mechanism, and both reporting and `run()` consume it. Active-row count selects B=1 or B=N inside the continuous mechanism. |
+| Cancellation and flow | Cancellation is cooperative. The handler's `finally` sets a flag; serial execution checks it after tokens, while batched execution checks it during prompt and generation work and removes the row. [`server.py:1008`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1008) [`server.py:861`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:861) [`server.py:911`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:911) [`server.py:1551`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1551) The response queue is unbounded, and socket writes block only the HTTP thread, so there is no end-to-end demand signal from the client to generation. [`server.py:444`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:444) [`server.py:1032`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1032) [`server.py:1489`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1489) | Do not regress to the oracle here. mlx-bun's token callback accepts a promise or `false`, and `GenerationGateway.run()` accepts an `AbortSignal`. [`generation-gateway.ts:94`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/generation-gateway.ts:94) [`generation-gateway.ts:354`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/generation-gateway.ts:354) Cancellation and async flow control are part of the interface, not HTTP details. |
+| Cache and resource lifetime | The worker fetches or creates the serial cache, generates, then inserts it. Batched execution fetches before admission, snapshots segment caches, inserts the final cache, and removes cancelled rows. [`server.py:963`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:963) [`server.py:1018`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1018) [`server.py:752`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:752) [`server.py:864`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:864) [`server.py:901`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:901) `LRUPromptCache` hides prefix matching, copying, trimming, and count/byte eviction behind a small interface. [`cache.py:1623`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/models/cache.py:1623) [`cache.py:1674`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/models/cache.py:1674) [`cache.py:1696`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/models/cache.py:1696) | `prepareCompletion()` now disposes owned media and grammar state on admission rejection. `execute()` disposes the same plan on placement or pre-generation failure, then transfers ownership exactly once before `GenerationGateway.run()`. Cache and in-generation disposal remain in the gateway and generation path. [`completion-executor.ts:75`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:75) [`completion-executor.ts:219`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:219) |
+| Errors | Errors before the response iterator exists become a JSON error, although the oracle uses status 404 for that whole class. Errors delivered later through the worker queue are raised by the iterator. The handler guarantees `ctx.stop()` but has no matching mid-stream error packet in this loop. [`server.py:1417`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1417) [`server.py:1424`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1424) [`server.py:1035`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1035) [`server.py:1454`](/Users/joshrossi/Code/mlx-lm/.venv/lib/python3.14/site-packages/mlx_lm/server.py:1454) | Preserve a typed distinction between rejection before execution and failure after events begin. Both must run the same ownership cleanup. Protocol adapters decide how to encode the failure after headers have been sent. |
+
+## Strongest conclusions
+
+1. **Keep resolved inference settings authoritative.** Serial and continuous execution are scheduling mechanisms for the already-selected composition. Consolidating placement may shrink the HTTP adapter, but it must not turn explicit features or opinionated defaults into inferred choices.
+2. **Do not borrow the oracle's semantic split.** Its worker emits state-tagged token records, then the HTTP handler performs semantic accumulation and tool parsing. mlx-bun already has a better starting point in `CompletionSink`. Move that module behind the Completion execution seam so protocol adapters see semantic events only.
+3. **Keep the terminal summary as the only completion result.** It now settles finish reason, usage, cached-token count, final lane, speculation, logprobs, and collected output once. The four HTTP branches only format that summary. [`completion-executor.ts:103`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/serve/completion-executor.ts:103) [`server.ts:3104`](/Users/joshrossi/.codex/worktrees/e408/mlx-bun/src/server.ts:3104)
+4. **Treat cancellation, async flow control, and ownership as interface requirements.** The oracle's queue makes cancellation and socket pressure indirect. mlx-bun has stronger mechanisms already. The deepening should hide their implementation without weakening their behavior.
+
+The deletion test is positive. If this Completion execution module disappeared, scheduling policy, cache participation, semantic parsing, finish rules, usage, and cleanup would spread back across four protocol branches. Concentrating those rules gives callers leverage and gives maintainers locality. Feature selection and fidelity policy remain outside this seam as resolved inputs.

package/docs/design/orpo-dynamic-lambda.md

@@ -158,7 +158,7 @@ than the absolute max-bearable (the edge is fragile).
 **Synergy with the diffusion frontier:** all of this is noise-robustness machinery for
 controlling off a noisy process variable — so it transfers *especially* well to a
 diffusion LM, where the likelihood is a stochastic ELBO (even noisier). See
-[orpo-base-uf-experiment-and-directions.md](../investigations/orpo-base-uf-experiment-and-directions.md) §6.7.
+[orpo-base-uf-experiment-and-directions.md](../archive/investigations/orpo-base-uf-experiment-and-directions.md) §6.7.
 
 ## Wiring
 

package/docs/design/orpo-future-enhancements.md

@@ -11,7 +11,7 @@ comfortable.
 > trainer)** shipped as `orpo_prefix_shared`, composed with the flash head AND the
 > segmented backward for MiniCPM5 **and** e4b. See [training.md](../reference/training.md),
 > [orpo-quickstart.md](../reference/orpo-quickstart.md), and the live record
-> [steel-flash-cce-handoff.md](../investigations/steel-flash-cce-handoff.md). The
+> [steel-flash-cce-handoff.md](../archive/investigations/steel-flash-cce-handoff.md). The
 > remaining sections (§2, §4–§9, the objective variants) are still genuinely future.
 
 ## Deferred levers — queued, do NOT start mid-run (2026-06-19)

package/docs/design/orpo-training.md

@@ -19,7 +19,7 @@ runs half the forwards per step.
 > The ✅/🔬 entries + optimization log below are the **design/build history**; for the
 > current usage, flags, and numbers see [training.md](../reference/training.md) +
 > [orpo-quickstart.md](../reference/orpo-quickstart.md), and the live engineering record
-> [steel-flash-cce-handoff.md](../investigations/steel-flash-cce-handoff.md).
+> [steel-flash-cce-handoff.md](../archive/investigations/steel-flash-cce-handoff.md).
 
 > Oracle note: neither `mlx_lm` nor `optiq` ships ORPO, so — unlike DPO
 > ([`loss.ts`](../../src/train/loss.ts) ported `optiq/lora/dpo.py`) — ORPO is
@@ -38,7 +38,7 @@ runs half the forwards per step.
 > hiddens before judging** (it's plausibly a free speedup to enable). Standing Lab
 > gates (filter-on-real-data, teacher-forced grad fidelity, end-to-end quality)
 > + the host-buffer pin-leak post-mortem:
-> [orpo-flash-cce-pin-leak.md](../investigations/orpo-flash-cce-pin-leak.md).
+> [orpo-flash-cce-pin-leak.md](../archive/investigations/orpo-flash-cce-pin-leak.md).
 
 ## Implementation status
 

package/docs/design/parity-tier-dag.md

@@ -26,7 +26,7 @@ exists AND it matches bit-exactly; otherwise it bubbles up. Note the subtlety th
 map already shows: an *optimization* that still matches the oracle stays low —
 `compiled-decode` is ours but is L1 because it replays the same ops bit-for-bit.
 Only nodes with **no** oracle (flash-CCE) are *forced* into the Lab. (Background +
-post-mortem: [orpo-flash-cce-pin-leak.md](../investigations/orpo-flash-cce-pin-leak.md).)
+post-mortem: [orpo-flash-cce-pin-leak.md](../archive/investigations/orpo-flash-cce-pin-leak.md).)
 
 ## Two axes: parity gates performance
 

package/docs/design/repo-cleanup-plan.md

@@ -45,7 +45,7 @@ check). Untracked ≠ deleted ≠ unrunnable.
   + the two docs that reference `repro/` (grep first: `grep -rn "repro/"
   docs/ *.md`).
   - Also updated every path reference (CLAUDE.md, PLAN.md, PLAN-archive.md,
-    docs/investigations/lab-build-journal.md, docs/design/docs-reorg-plan.md,
+    docs/archive/investigations/lab-build-journal.md, docs/design/docs-reorg-plan.md,
     and the `repro/`-pointing comments in src/mlx/{ffi,array,autograd,ops}.ts
     + tests/{ffi-jit,train-autograd}.test.ts + the repro's own ISSUE.md).
 - `[x]` **A2. root strays** (all untracked; no git surgery): delete
@@ -168,12 +168,12 @@ suspenders until both machines are re-cloned.
     threads). The closed `oMLX` phase (2026-07-02) and Phase 19 are
     *newer* than 17, so they stay in PLAN.md as recent history per the
     criterion. Revisit D1 when a <17 phase is confirmed closed.
-- `[x]` **D2.** `docs/investigations/curve-runs/` + two untracked HTML
+- `[x]` **D2.** `docs/archive/investigations/curve-runs/` + two untracked HTML
   strays in docs/investigations: gitignore or move under `reports/`.
-  - Already satisfied: `docs/investigations/curve-runs/` is gitignored
+  - Already satisfied: `docs/archive/investigations/curve-runs/` is gitignored
     (`.gitignore`), and the four HTML strays (`curve-terrain.html`,
     `hlg-ab.html`, `hlg-report.html`, `hlg-explorer.html`) are gitignored
-    via `docs/investigations/*.html`. No action needed.
+    via `docs/archive/investigations/*.html`. No action needed.
 - `[x]` **D3.** The CLAUDE.md doc-map lists are hand-maintained; add a
   `scripts/check-hygiene.ts` assertion that every `docs/**/*.md` appears in
   the map (same guardrail spirit as B1).
@@ -187,3 +187,45 @@ A and D anytime (safe, agent-executable solo) — DONE 2026-07-02. B gate
 DONE 2026-07-02 (B2 adapter-untrack deferred to Josh — see B2 finding).
 C executed 2026-07-02 on Josh's explicit go — only remaining tail is the
 M4 Pro reset + deleting the backup tarball once both boxes are green.
+
+## Phase D — work-artifact hygiene + rules (2026-08-18) — DONE
+
+Trigger: Josh — "this is a software project; we should not be adding all
+these random work things into it." Compared against ml-explore/mlx-lm's
+root (9 files, 4 dirs, zero work artifacts).
+
+- **D1 root sweep**: six tracked `benchmarks-serve-2026-07-0*.md` dumps
+  git-rm'd (they leaked past a gitignore pattern that only covered
+  `benchmarks-h2h-*`); seven untracked root bench artifacts moved to
+  `reports/bench-archive/`; `tmp/` deleted. `.gitignore` now blocks
+  `/benchmarks-*`, `/bench-*`, `/tmp/`, `/*.log` at root.
+- **D2 guardrail**: `scripts/check-hygiene.ts` gained a third gate —
+  tracked ROOT files must be on an explicit allowlist (16 meta files).
+- **D3 CONTRIBUTING.md**: the rules (what-goes-where table; no dated
+  artifacts; models → `~/models`; docs lifecycle; PLAN/STATUS discipline;
+  external oracle venv is not a repo).
+- **D4 docs archive**: all 20 `docs/investigations/` → 
+  `docs/archive/investigations/` (investigations document finished work);
+  5 release-notes + journal + 2 chat-ui visions + 2 pass-plans →
+  `docs/archive/planning/`; `hlg-sampling.md` (self-declared superseded) →
+  `docs/archive/`. All inbound links rewritten; docs-map gate green.
+- **D5 journal trim (rule 4)**: STATUS.md 1648 → 106 lines (Colibri
+  gate-history + all "where we were" + the stale July ranked backlog →
+  PLAN-archive under "STATUS archive 2026-08-18"); PLAN.md 3815 → 2679
+  (8 closed `[x]` phase blocks → PLAN-archive with one-line pointers).
+- Model artifacts moved out of the tree entirely (`~/models/`) earlier
+  the same day.
+
+Remaining (curated, not mechanical — needs judgment per file):
+- `docs/design/` still holds 46 docs; many describe LANDED work and
+  should archive under rule 3 (candidates: optimization_plan,
+  faithful-base superseded pair, diffusion/audio/omlx/dspark docs whose
+  phases closed). Do this per-doc with link fixes, not as a bulk move.
+- `docs/planning/` leftovers (chunk-task-roadmap, curve-sampler-research-
+  plan, memory-docs-and-dag-plan, pi-integration-review) — keep-or-archive
+  per Josh.
+- `scripts/experiments/` = 199 files; prune ones whose findings are fully
+  recorded in archived docs (keep regen/oracle scripts).
+- `runs/` is 32 GB machine-local (Colibri evidence + quant lab arms) —
+  delete lab arms once the TQ/GPTQ program closes; Colibri evidence dirs
+  are cited from the archive and stay.

package/docs/design/segmented-backward-training.md

@@ -35,7 +35,7 @@ All committed-worthy, validated this session:
   Baseline 11.89/100 → fine-tuned **91.70/100** on lucien's 25-case chunk holdout
   (valid-JSON rate 12% → 100%). Full mlx-bun loop: train → save adapter (hot-swap
   format) → `AdapterManager.mount` → generate. Writeup:
-  `docs/investigations/chunk-finetune-experiment.md`. Scripts: `scripts/chunk-eval.ts`,
+  `docs/archive/investigations/chunk-finetune-experiment.md`. Scripts: `scripts/chunk-eval.ts`,
   `scripts/chunk-finetune.ts`, `scripts/chunk-filter.ts`. Adapter:
   `~/.cache/mlx-bun-finetunes/minicpm5-chunk-final`.
 
@@ -69,7 +69,7 @@ active-after 8.66 + cache-after 6.61 = 15.27 > peak 14.90, impossible if peak we
 total). MLX holds freed buffers in a **cache** (not returned to OS) so RSS overstates
 live memory; use `activeMemory()` / `cacheMemory()` / `clearCache()` to separate
 them (`src/mlx/ffi.ts`). Caveat from the MoE expert-offload work
-(`docs/investigations/expert-offload-single-user-moe.md`): `clearCache()` does NOT
+(`docs/archive/investigations/expert-offload-single-user-moe.md`): `clearCache()` does NOT
 reliably return device-buffer RAM to the OS — only the mmap clean-page path does.
 
 Facts established:

package/docs/design/ssd-kv-cold-tier.md

@@ -286,3 +286,130 @@ SIGINT/SIGTERM during (or within the debounce after) traffic loses queued
 write-behind entries; restart survival degrades to whatever already
 flushed. This is a best-effort cache tier; the cost of a lost entry is one
 re-prefill. `SpillQueue.drain()` exists if a shutdown hook is ever wanted.
+
+This acceptance was superseded by the 2026-08-22 durability fix below.
+
+## Bug brief 2026-08-22: restart can outrun SSD write-behind
+
+### User-visible failure
+
+A completed long-context request can report a full RAM-cache hit on repeat,
+then lose that prefix when the server restarts a few seconds later. The next
+request pays a full prefill even though `--ssd-cache` was enabled.
+
+The 2026-08-22 M1 Max serve matrix reproduced this with the standard restart
+leg, which waits 2.5 seconds after the long request before sending SIGTERM:
+
+| model and mode | cached tokens after restart | expected |
+|---|---:|---:|
+| MiniCPM5, unified bf16 | 15,817 | about 15.8k |
+| e4b, unified bf16 | 4 | about 15.9k |
+| e4b, serial bf16 | 15,940 | about 15.9k |
+| 12B, unified bf16 | 4 | about 15.9k |
+| 12B, serial bf16 | 15,866 | about 15.9k |
+| Qwen3.8, unified bf16 | 0 | about 15.1k |
+| Qwen3.8, serial bf16 | 0 | about 15.1k |
+| Qwen3.8, unified mixed KV | 15,112 | about 15.1k |
+
+The Bun 1.4 rerun reproduced the lane split before this brief was written:
+e4b unified restored 2 of 15,942 tokens, while e4b serial restored 15,939.
+The completed run also restored only 4 tokens for unified 12B versus 15,794
+for serial 12B, and 5 for mixed-KV 12B. Qwen bf16 restored zero in both
+lanes, while mixed-KV Qwen restored all 15,114 tokens. This is not a Bun
+1.3-only result.
+
+### Expected contract
+
+After a request completes and the server has reported the cache durable, a
+normal SIGTERM and restart must restore the longest compatible prefix. For
+the benchmark's exact-repeat request, `cached_tokens` should be at least
+`prompt_tokens - 1`. The server must never imply durability while a matching
+snapshot is still waiting on its debounce timer or the spill queue.
+
+### Evidence and current diagnosis
+
+The read and reconstruction path works when the file reaches disk. MiniCPM,
+serial e4b and 12B, and mixed-KV Qwen all restore a full prefix through the
+same `PromptCache.take()` and `SsdCacheStore.restore()` path. That makes a
+model-format or prefix-matching defect unlikely as the primary cause.
+
+The likely failure is the durability boundary:
+
+1. `PromptCache.onPut` schedules a snapshot with a one-second debounce.
+2. A busy gateway re-arms that timer for five seconds.
+3. The snapshot then joins a serial `SpillQueue`; large bf16 entries can still
+   be queued or in flight when the benchmark's 2.5-second wait expires.
+4. SIGTERM stops the process without forcing pending timers to fire or awaiting
+   `SpillQueue.drain()`. The queue's own comment calls drain a future shutdown
+   hook.
+
+Entry size explains the broad pattern. Small MiniCPM and smaller mixed-KV Qwen
+files finish in time; large Qwen bf16 files do not. The remaining unified versus
+serial split needs telemetry before assigning a second root cause. The unified
+lane may enqueue more snapshots, take longer to extract row caches, or hit the
+queue cap. The current report does not capture that state before termination.
+
+### Fix plan
+
+1. Expose `pending_snapshots`, `pending_spills`, `pending_spill_bytes`,
+   `dropped_spills`, and the longest durable prefix in `/stats.ssd_cache`.
+   `pending_snapshots` must include debounce and busy-rearm timers, not only
+   entries already placed in `SpillQueue`.
+2. Add one async durability operation that cancels debounce timers, snapshots
+   matching RAM entries under the gateway lock, and awaits
+   `SpillQueue.drain()`. Return success only after the atomic rename completes.
+3. Route graceful SIGINT/SIGTERM through that operation with a bounded timeout.
+   Stop new admission first. A timeout may drop the cache, but must log the
+   number of pending and dropped snapshots.
+4. Change the benchmark restart leg to request or poll durability instead of
+   sleeping for 2.5 seconds. A fixed sleep measures SSD timing luck, not the
+   restart contract.
+5. Capture pre-kill stats and the recovered entry count in every restart row so
+   a write miss cannot masquerade as a restore miss again.
+
+### Implemented 2026-08-22
+
+- `SsdDurabilityCoordinator` owns debounce, busy retry, queue completion, and
+  explicit flush as one state machine. A snapshot stays dirty until the atomic
+  store succeeds. Queue drops and write failures remain retryable.
+- Dirty keys include the exact token sequence, so two conversations with the
+  same namespace and token count cannot cancel each other.
+- `POST /admin/cache/flush` returns `200` only at the durability boundary and
+  exposes pending, dropped, failed, entry-count, and longest-prefix evidence.
+- `mlx-bun serve` handles `SIGINT` and `SIGTERM` by stopping admission,
+  draining active requests, and awaiting the same flush, bounded by
+  `MLX_BUN_SHUTDOWN_TIMEOUT_MS`.
+- The standard restart benchmark calls the endpoint instead of sleeping and
+  records its evidence in the Markdown report and eval database notes.
+- Model-free regression coverage holds an atomic store open and proves flush
+  waits, forces the busy-rearm path, retries a failed write, preserves
+  equal-length conversations, and reports a missing RAM boundary honestly.
+
+Focused real-server smoke on unified e4b passed the repaired path: a
+4,026-token prompt flushed in 408 ms with `durable=true`, zero pending,
+dropped, or failed work, then restored 4,025 cached tokens after SIGTERM and
+respawn. The machine failed the clean-performance preflight, so this is cache
+correctness evidence only, not a performance result. The full clean 16k
+e4b/12B/Qwen matrix remains the acceptance run.
+
+### Acceptance tests
+
+- A deterministic coordinator test holds `storeAsync`, requests a flush,
+  releases the store, and proves the durability boundary waits for the atomic
+  file. The focused real-server smoke exercises the SIGTERM path through that
+  same operation before restart.
+- A busy-gateway test forces the five-second re-arm path, then requests a flush
+  and proves the timer becomes an immediate queued snapshot without blocking an
+  active generation.
+- Batch 8 and `--batch 1` both restore at least `prompt_tokens - 1` for e4b,
+  12B, and Qwen3.8 at roughly 16k context.
+- The mixed-KV and SSM cache kinds retain their existing continuation parity.
+- Steady-state decode remains unchanged because normal writes still pause while
+  the engine is busy. Queued clone memory stays within the spill cap.
+
+Focused reproduction:
+
+```sh
+./benchmark.sh --models e4b,12B,qwen27b \
+  --arms mlx-bun,mlx-bun-serial,mlx-bun-mixed
+```