mlx-bun 0.4.0 → 0.5.0

package/docs/design/kv-cache.md

@@ -3,7 +3,7 @@ status: landed
 axis: ON
 canonical-for: kv-cache
 plan-anchor: "Phase 18 — Concurrent / batched serving (slots) + parallel load benchmark `[~]` (2026-06-13)"
-last-verified: 2026-09-09
+last-verified: 2026-09-12
 ---
 
 # KV cache — residency and layout
@@ -24,27 +24,31 @@ and open items.
 Code: `src/model/gemma4-base.ts` (Cache interface + the K/V cache family),
 `src/model/batched-*.ts`, `src/lab/paged-kv/paged-kv.ts`, `src/prompt-cache.ts`,
 `src/kv-store.ts`, `src/ssd-cache.ts`, `src/ssd-durability.ts`,
+`src/tiered-prompt-cache.ts`, `src/storage/kv-writer.ts`,
 `src/kv-scheme.ts`, `src/generate.ts` (`maybeQuantizeKv` / `maybePageKv`),
 `src/server.ts` (wiring, `/stats`, `POST /admin/cache/flush`),
 `src/serve/generation-gateway.ts`, `src/serve/batch-scheduler.ts`.
 
 ## 1. Layering
 
-Residency is a three-tier stack, each tier keyed by the exact token prefix
-its KV encodes:
+Active generation owns mutable cache containers. Published checkpoints belong
+to one logical cache, with RAM and SSD storage. On Apple Silicon the GPU and
+CPU access unified memory; publication does not require a GPU-to-CPU copy.
+Each checkpoint is keyed by the exact token prefix its state encodes:
 
 | tier | owner | unit | key |
 |---|---|---|---|
 | live caches | one generation (serial lane or a batch row) | `Cache[]`, one per layer | — |
-| RAM prompt cache | `PromptCache` (`src/prompt-cache.ts`) | whole prefix entry (tokens + `Cache[]`) | tokens + adapter ns |
+| RAM prompt cache | `TieredPromptCache` over `PromptCache` | whole prefix entry (tokens + `Cache[]`) | tokens + adapter ns |
 | SSD cold tier | `SsdCacheStore` (`src/ssd-cache.ts`) over `src/kv-store.ts` files | one `.mlxkv` file per entry | tokens + ns + model/scheme/tokenizer fingerprint |
 
 The prompt cache tiers over the SSD store *inside* `take()` (structural
 `ColdTier` interface), so every consumer (serial lane, batch scheduler
 joiners, `/admin/drain`) gets both tiers through the same `take()`/`put()`
-calls. Requests and evictions enqueue persistence work. The writer performs
-it when the engine is idle; RAM reuse does not enqueue an extra write or wait
-for the queue. Explicit flushes await the queued work. The opt-in `--generation-checkpoint N` path transfers owned snapshots at coarse decode boundaries and queues atomic
+calls. The cache queues persistence on publication or eviction. A dedicated
+CPU worker writes evaluated immutable buffers while inference continues.
+Persistence does not evict RAM; only cache residency policy does. RAM reuse
+does not enqueue an extra write or wait for the queue. Explicit flushes await the queued work. The opt-in `--generation-checkpoint N` path transfers owned snapshots at coarse decode boundaries and queues atomic
 persistence without awaiting disk in decode (section 5.7). Paged KV
 (section 6) is a *layout* choice for the live tier and, in v1, bypasses the
 other two.
@@ -217,9 +221,11 @@ changing the retained checkpoint.
 
 SSD is an eventually consistent persistence layer. RAM reuse never writes
 to SSD or waits for durability. Background writes retain immutable snapshot
-views and yield between tensor writes; explicit flush and graceful shutdown
-wait for persistence. A pending or failed write does not prevent RAM reuse.
-The existing byte cap controls retained donors and LRU eviction.
+views while a CPU worker packs, hashes and writes them; explicit flush and
+graceful shutdown wait for persistence. A pending or failed write does not prevent RAM reuse.
+The byte target controls LRU eviction. With SSD enabled, an unwritten victim
+stays resident until storage commits; the target is soft while writes are
+pending or failing.
 
 The native sharing gate checks logical checkpoint bytes after repeated decode
 and after SSD restore. It passes on both development Macs; the M1 long-prefix
@@ -232,18 +238,23 @@ and explicit flush remains pending until the write completes.
 longer usable prefix; then `restore(handle)` materializes it and the
 divergent tail is trimmed. A cold entry that would need a trim but is
 untrimmable (wrapped ring, SSM) is disposed loudly and the RAM candidate
-(or a fresh prefill) serves. `hits`/`misses` count RAM candidacy only.
+(or a fresh prefill) serves. Tiered restore immediately publishes shared views
+in RAM so another request can reuse them before the borrower finishes.
+`hits`/`misses` count RAM candidacy only.
 
 **`put(tokens, caches, ns, retain)` — supersession + eviction.** Exact
 duplicates (same tokens) are replaced regardless of trimmability. Strict
 prefix-ancestors in the same ns are superseded without spill — but only
 when the *new* entry is fully trimmable; an untrimmable new entry (wrapped
 ring) can only serve exact-length matches, so shorter ancestors (the
-prompt-boundary snapshot) must survive. Then LRU-evict until under cap,
-spilling each evictee to the cold tier first. An entry larger than the cap
-is spilled and disposed, never stored. `onPut` fires after every
-successful put — the server hangs the write-behind scheduler there so both
-lanes' entries persist.
+prompt-boundary snapshot) must survive. `TieredPromptCache` selects the
+least recently used RAM entry when over budget. If it lacks durable SSD
+coverage, the cache queues a write and retains it until that write succeeds.
+It does not evict a hotter entry merely because that entry is already on SSD.
+An oversized checkpoint follows the same persist-before-demotion rule.
+Without SSD, the RAM-only cache discards evicted or oversized entries.
+`TieredPromptCache.put` also owns proactive persistence; `onPut` remains an
+independent observer for generated-token history.
 
 **Prompt-boundary snapshot.** Serial lane (`server.ts`) and batch
 scheduler (`snapshotAt`) both clone-and-put a trim-free *prompt-only*
@@ -257,12 +268,11 @@ always an exact prefix of the next rendering.
 **Namespaces.** `ns` = the adapter spec (`adapters.join("+")`, `""` = base).
 KV computed under one adapter never seeds another's prefill.
 
-**Bypasses.** Media (vision/audio) requests skip the cache (soft tokens are
-identical placeholder ids → false hits), as do paged-KV requests (v1) and
-the speculative lane (v1). Merged batch rows are not re-put on finish;
-never-merged lone rows put their adopted caches back zero-copy, and
-merged rows are extracted per row (`#extractAndPut`) before `filter()`
-mutates the batched inners.
+**Bypasses.** Media and optional paged-KV requests retain their documented
+cache qualification rules. Shared speculative execution publishes target and
+method state together where supported; method attachments preserve the exact
+boundary needed for reuse. Merged batch rows are extracted before retirement
+changes the batched state. The cache owns the extracted checkpoint.
 
 **Idle demotion (`demoteIdle`).** Entries unused past a threshold spill to
 the cold tier and free their GPU arrays; the prefix stays reachable via
@@ -287,17 +297,14 @@ full prefill (GEMV-vs-GEMM reduction order). Both tiers share it.
 
 ### 5.1 Decisions
 
-- **D1 — Whole-prefix-entry spill, not paged blocks.** Single-user serving
-  with a prefix-entry cache that grows per conversation. Per-block hashing
-  and save-queue bookkeeping during decode is where oMLX's measured ~20%
-  steady-state tax came from; ours does zero work on the token loop
-  (prefix tokens are the key). Partial-prefix reuse works without paging:
-  restore + `trim()`. The economics that justify the tier are
-  SSD-vs-recompute, not SSD-vs-RAM: at 12B prefill rates a 30k agent
-  context is minutes of GPU compute against a ~1 s NVMe restore.
-- **D2 — One store, two tiers.** The cold tier is bound *into*
-  `PromptCache` (section 4); restart persistence comes from a debounced
-  write-behind snapshot after each put, not only from spill-on-evict.
+- **D1 — Whole-prefix-entry storage.** A checkpoint stores a complete
+  conversation boundary. This reuses the existing cache codecs and avoids
+  per-token storage bookkeeping, but duplicates shared attention history
+  between files. Block deduplication is a separate layout optimization to
+  measure against this baseline; recurrent state still needs exact boundaries.
+- **D2 — One store, two tiers.** `TieredPromptCache` owns RAM residency and
+  the SSD backend. Proactive persistence after publication provides restart
+  survival without waiting for an eviction.
 - **D3 — Own page-aligned format, not safetensors.** `src/kv-store.ts`
   writes `MLXBUNKV2\n`-magic files with a streaming writer: ordinary KV
   retains format v3; checkpoints with method companion state use v4.
@@ -377,9 +384,9 @@ ordering is unchanged (the server's `coldTier.restore` returns
 `() => {}`). Comments in `prompt-cache.ts` that still say "zero-copy
 mmap" describe the superseded mechanism.
 
-The write side is the mirror: hashing and writing from a zero-copy view
-of the contiguous mlx buffer (`rawBytesView`), no JS-heap copy at any
-point.
+The asynchronous writer reads evaluated native storage directly. Contiguous
+tensors need no packing copy. Strided tensors are packed on the CPU, one
+tensor at a time, to omit capacity padding without a GPU `contiguous` call.
 
 ### 5.3 `find()` returns partial matches, gated on trimmability
 
@@ -394,56 +401,52 @@ Store-side supersession mirrors `PromptCache.put`: exact duplicates
 replaced; prefix-ancestors superseded only when the new entry is
 trimmable.
 
-### 5.4 Write-behind scheduling contract: own each tensor step
-
-Every per-tensor flush step is real GPU-stream and JS-thread work
-(`ops.contiguous` on the decode stream, a synchronous eval for
-`rawBytesView`, a synchronous multi-MB `writeSync`). Interleaving those
-between decode tokens taxed decode at long context, so:
-
-- `saveKvCacheAsync` / `SsdCacheStore.storeAsync` accept a per-step runner.
-  The server passes `gateway.runWhenIdle`, so the idle decision and the
-  blocking MLX readback are one atomic engine turn. Background work waits
-  without registering a serial waiter or draining active batches. The earlier
-  `gateway.onIdle()` check had a check/use race: a request could begin after
-  the check and overlap the next tensor sync. A request arriving mid-flush
-  can run between tensor writes. Neither execution lane drains pending
-  durability work before generation.
-- `GenerationGateway.busy` covers both lanes (serial mutex held/awaited or
-  batch rows active/pending).
-- `MLX_BUN_SSD_WRITEBEHIND=0` disables write-behind snapshots entirely
-  (kill switch + paired-A/B lever; restart survival then degrades to
-  spill-on-evict).
-
-Explicit flush and shutdown drain outstanding writes outside the generation
-lock because the writer needs that same lock. Ordinary requests do not.
-
-The RAM cache's configured byte cap is only an upper bound. Serial serving
-checks MLX's live allocation count against the smaller of admission's usable
-memory and Metal's recommended working set, reserving prefill workspace and
-the prompt's KV footprint for growth/copy-on-write overlap. Pressure relief
-persists LRU entries synchronously before disposing them, avoiding deferred
-spill clones that would keep the memory resident. The selected request cache
-remains owned by the request; older entries remain reusable from SSD. Qwen
-checks between prefill layers, and serial decode checks every 256 tokens.
-If no reclaimable entries remain, the guard raises a request error instead
-of intentionally proceeding over that budget. In-flight checkpoints survive.
-
-### 5.5 Bounded retention: `SpillQueue` (`kv-store.ts`)
-
-All three producers — eviction spills, idle demotions, write-behind
-snapshots — go through one serial, byte-capped queue. Pending clones pin
-their entries' GPU buffers until the generation-locked flush gets a turn; without
-a cap, sustained traffic (gate starved, evictions ongoing) made resident
-memory = prompt-cache cap + every queued clone. Policy: cap default 2 GiB
-(`MLX_BUN_SSD_SPILL_QUEUE_GB`; `0` keeps only the newest + in-flight
-clone); over cap the *oldest* not-in-flight item drops and its clones are
-disposed immediately; the item just enqueued is never its own victim
-(soft cap). A dropped spill is a future cache miss, never a wrong result.
-Clones are disposed on every settle path (fulfil, reject, drop). The
-snapshot timer never holds batch admission: while rows are active it
-re-arms instead of grabbing the exclusive (entries are immutable; a late
-snapshot is equally valid).
+### 5.4 Background persistence and RAM residency
+
+`TieredPromptCache` owns RAM residency, SSD writes and durable-coverage checks.
+`put` materializes the checkpoint before publication. `saveKvCacheAsync`
+uses the existing codecs to describe the file and obtains host-visible native
+pointers, shapes and strides on the MLX owner thread. Its optional preparation
+runner does not cover disk I/O. No MLX handles cross into the writer.
+
+`KvWriter` embeds a dedicated Bun CPU worker in source and compiled builds.
+The worker packs strided layouts, hashes tensor bytes, writes a temporary file,
+fsyncs it and renames it atomically. It imports no MLX or scheduler code and
+needs no generation lock. Contiguous tensors use direct buffer views; packing
+scratch is reused across strided tensors. The cache owns the retained state,
+and the queued write holds shared views through its completion. Those views
+share immutable storage rather than copying the entire checkpoint.
+
+The owner thread still prepares native views and updates the SSD metadata
+index after completion. A background worker removes blocking payload I/O from
+that thread; it does not make shared memory bandwidth or snapshot preparation
+free. Measure those costs with concurrent decode before claiming a speedup.
+
+Writing a copy does not remove RAM residency. Byte pressure chooses the actual
+LRU entry. If its SSD copy is missing, it queues persistence and waits for the
+completion callback to retry eviction. Failed writes leave the RAM entry
+usable and dirty. Age-based eviction follows the same rule and runs without
+acquiring the scheduler's generation lock. Neither policy refuses requests or
+changes context, batch size or sampling settings.
+
+`MLX_BUN_SSD_WRITEBEHIND=0` disables proactive persistence after publication;
+eviction still queues persistence before removing RAM. Explicit flush and
+shutdown await outstanding writes. A hard exit can lose uncommitted writes.
+
+### 5.5 Persistence queue and capacity
+
+Normal prefix persistence uses one serial `SpillQueue` with no independent
+byte-drop policy. RAM ownership and pending write views share the checkpoint;
+queue completion releases only the writer's reference. RAM cache capacity
+and age policy decide demotion. They do not cancel an unwritten snapshot to
+reduce the queue's accounting. While the oldest victim is writing, RAM can
+exceed its target. A storage failure keeps that victim resident and retryable.
+
+The optional interrupted-generation coordinator still uses the generic
+bounded queue for supersedable intervals. `MLX_BUN_SSD_SPILL_QUEUE_GB`
+(default 2 GiB) now controls that queue only. It does not limit retained prompt
+history. SSD capacity is independent: no capacity eviction without an explicit
+positive `--ssd-cache-max`.
 
 ### 5.6 Durability boundary (`src/ssd-durability.ts`)
 
@@ -454,17 +457,16 @@ the write — the server implied durability it did not have.
 
 - `schedule(tokens, ns)` records a *dirty* key that includes the exact
   token sequence (two same-ns, same-length conversations cannot cancel
-  each other) and arms the debounce.
-- An attempt checks SSD coverage under `gateway.runWhenIdle` before cloning.
+  each other) and arms a next-tick attempt.
+- An attempt checks committed SSD coverage before cloning.
   Otherwise it snapshots the RAM entry (`findExact` + `cloneKvCaches`) and
-  enqueues it; a busy gateway re-arms;
-  a dropped or failed store leaves the key dirty so it is retryable.
+  enqueues it without consulting scheduler activity. A failed store leaves
+  the key dirty so it is retryable.
   A snapshot that vanished from RAM is "stored" only if the SSD index
   already covers the prefix (`hasDurablePrefix`), else "missing" and still
   dirty. A second flush cannot clear that failure without durable coverage.
 - `flush()` cancels timers, awaits in-flight attempts and `drain()`, then
-  forces each dirty record version once, one at a time (so the queue cap
-  cannot drop a boundary snapshot while a large final snapshot is in flight).
+  forces each dirty record version once, one at a time.
   A replacement scheduled during a write receives its own attempt. After
   all writes settle, the coordinator checks missing ancestors against the
   committed SSD index again: a later trimmable descendant may now cover a
@@ -513,7 +515,8 @@ recover sampler seed, history and sample index. The server binding qualifies
 ordinary requests through supported row-cache layouts, including recurrent Qwen
 and full-attention Llama/MiniCPM as well as Gemma4. It transfers owned row
 snapshots to a bounded persistence coordinator without awaiting disk in decode.
-Tensor serialization and file maintenance use the existing idle gate. Completion
+Snapshot preparation runs on the owner thread; payload serialization and
+writes use the CPU worker. Completion
 invalidates reuse immediately and queues cleanup after any in-flight write;
 new attempts cannot be erased by older cleanup. Cancellation retains eventual
 persistence, and explicit flush reports failed or pending durability. A hard
@@ -525,8 +528,8 @@ compare interrupted/restored runs at the same batch geometry; arbitrary
 cohort changes are not an exact numerical identity guarantee. The HTTP fixture
 checks serial B1, shared B1 and shared B4 with identical seeded requests and
 held arrivals, and inspects the generated checkpoint before restart.
-When checkpointing is enabled, the existing total spill queue budget is split
-between prompt spills and continuation snapshots. Flush duration and pending/failed
+The configured spill queue budget applies to continuation snapshots; normal
+prefix persistence follows the RAM cache residency policy. Flush duration and pending/failed
 statistics cover both queues.
 
 Only the newest successful checkpoint for a request is retained. The old file
@@ -559,7 +562,7 @@ Method and drafter compatibility must be part of the namespace before a
 producer publishes these records. Storage does not interpret draft state as
 extra target-model layers.
 
-RAM accounting, eviction, idle demotion and the bounded SSD spill queue include
+RAM accounting, eviction, idle demotion and SSD persistence include
 companion tensors. Checkpoints containing them require an exact prefix
 boundary and preserve shorter fallback entries. Write-behind captures companion
 views together with target-cache views under the existing exclusive boundary.
@@ -598,13 +601,45 @@ through repeated RAM reuse and SSD cache/provider restart. SSD allocation
 padding is excluded from the live-state comparison. Generated-output acceptance
 is described below; long-conversation timing remains R17 work.
 
-The unreleased shared method now publishes completed row state before
+The shared method publishes completed row state before
 retirement through the same `PrefixCache.put` port. It records only inputs
 retained by committed target/draft rounds; an unprocessed correction or bonus
 token is absent from the key. The target row layout and draft provider supply
 their snapshots independently. Failed and cancelled consumers publish no
 generated checkpoint. Persistence continues through the existing queue.
 
+Generated BPE IDs are not necessarily the canonical encoding of their decoded
+text. `GeneratedTokenHistory` retains bounded text/ID provenance from published
+prefixes and recovered SSD headers. Request preparation uses the original IDs
+when the rendered conversation contains that exact text prefix, encodes only
+the remaining suffix without another BOS, and checks that the combined IDs
+decode to the complete rendered request. Edited history or an unsafe decoder
+boundary keeps canonical tokenization. KV lookup still requires exact IDs and
+the existing execution namespace; no state is relabeled under different IDs.
+The template's primer length is measured canonically and applied to the actual
+history-preserving token sequence. This can change input token counts, and
+therefore continuation logits, relative to re-encoding all generated text.
+Logit parity remains a contract for the same input IDs and execution shape.
+The stable-boundary probe includes both text and tool replies followed by a
+new user turn. Qwen can omit an empty thinking primer when rendering a prior
+tool turn; a text-only probe retained three extra tokens in the small-model
+HTTP fixture and prevented recurrent-cache reuse. The corrected probe keeps
+the last shared prefix. It never rewrites existing KV under the changed text.
+Republishing an identical token sequence refreshes its existing provenance
+entry without decoding the whole conversation again. Equality covers every ID;
+equal text from different IDs still replaces provenance. This avoids repeated
+text conversion during request completion and preserves the retention limit.
+
+Cache pressure maintenance uses 85% of the smaller of the recommended device
+working set and an explicit allocator limit as a residency target. It includes
+known weight bytes when file-backed weights have not appeared in MLX active
+allocation. `PrefixCache.reclaim` asks the cache to demote LRU donors. With SSD,
+only durable donors can leave RAM; pending writes remain queued. A selected
+request retains its own views. Shared buffer ownership is not counted twice as
+RAM plus queued data. Lookup, prefill boundaries and periodic decode maintenance
+call this interface; decode checks every 256 steps. It does not constrain
+request admission, context, sampling or batch size.
+
 M1 and M4 native bf16, affine KV4 and fused TurboQuant K8/V3 tests cover four active
 rows, early retirement, exact sampled-ID coverage, unchanged retained bytes
 after sibling and follow-up work, shorter-prefix selection after an edit, and
@@ -662,120 +697,254 @@ the model's prototype cache before opening the tensor mmap.
   ever lands.
 - Block-granular dedup for multi-user shared prefixes stays out of scope
   (D1); revisit only with the paged follow-ups in 6.4.
-- Stale comments: `prompt-cache.ts` still describes restore as zero-copy
-  mmap (5.2).
-
-## 6. Optional paged KV (`src/lab/paged-kv/paged-kv.ts`, `--paged-kv`)
-
-Default off; with the flag unset no paged code executes and the plain
-`KVCache`/`RotatingKVCache` path is byte-identical.
-
-### 6.1 What it buys here, honestly
-
-vLLM's PagedAttention solves contiguous worst-case per-sequence
-reservation and allocator fragmentation for a multi-tenant server.
-mlx-bun is one process on unified memory with byte-budget admission
-already shipped, so what transfers is: (a) padded-batch waste removal —
-today's batched buffer width tracks the longest live row
-(`mergeKVRows`/`extendKVRows`), so a short row cohabiting with a long one
-pays the long one's KV; and (b) block-level copy-on-write prefix sharing.
-Persistent pools now stay separate per request under the shared executor.
-Attention still gathers and pads a temporary batch tensor; block-level prefix
-sharing and direct paged attention remain open. At batch one, the gather adds
-a copy. Paging remains optional and does not imply a speed improvement.
-
-This is neither of the two paging rejections already on record:
-paged KV as a *prompt-cache substitute* (rejected 2026-07-07 — `take()`'s
-zero-copy clones already share physically) and paged *blocks* as the SSD
-spill granularity (D1). It is the rung-3 allocation abstraction from
-docs/design/batching.md.
-
-No external oracle exists (mlx-lm's `cache.py` has no paged cache); the
-gate is mlx-bun's own plain `KVCache`, valid because the claim is storage
-equivalence (same bytes, different arrangement), so the bar is bit-exact
-(tol 0): `tests/paged-kv.test.ts` (model-free: block-boundary crossing,
-free-list reuse after trim, typed exhaustion, gathered fetch vs a plain
-reference, dispose) and `tests/paged-kv-parity.test.ts` (weights-gated
-greedy trajectory paged-on vs paged-off).
-
-### 6.2 Mechanism
-
-- **`BlockPool`** — per-layer arena: K and V pool tensors
-  `[numBlocks, H_kv, blockSize, headDim]` (V may differ in head dim),
-  `ops.zeros`-allocated so they are mlx-owned end to end (no host-pointer
-  alignment/dtor hazards). LIFO free list; `alloc()` throws the typed
-  `PagedPoolExhausted`; writes are `ops.sliceUpdate` at
-  `[block, 0, within, 0]` — the incoming `[1,H,l,D]` piece matches the
-  destination slice directly.
-- **`PagedKVCache implements Cache`** — standalone (not a `KVCache`
-  subclass). The cache-layout binding supplies `PagedKvRows` for dynamic
-  membership, masks and per-row positions. `updateAndFetch` splits the
-  incoming `L` along block boundaries, allocates tail blocks as reached,
-  then gathers occupied blocks with `ops.takeAxis(pool, blockTable, 0)` →
-  transpose `[1,0,2,3]` → reshape `[1,H,nb·bs,D]` → slice to `offset`. Only
-  existing bound ops — no `mlx_gather` FFI binding was needed. `makeMask`
-  is `KVCache.makeMask`'s logic; `trim(n)` rewinds `offset` and frees
-  now-unoccupied tail blocks (stale bytes past `offset` are never read —
-  the `KVCache` padding invariant, block-shaped). Block table is a
-  host-side `number[]`; all intermediates are bound and disposed.
-- Pool lazily allocates on the first write (head count/dims/dtype from
-  the first k/v pair) and is sized once from `capacityTokens` =
-  prompt + `max_tokens`, so exhaustion is unreachable absent an
-  accounting bug — it exists as a tripwire, never silent truncation.
-- **Wiring** — `maybePageKv(cache, options, capacityTokens)`
-  (`backends/mlx/request-state-policy.ts`, re-exported by `generate.ts`)
-  mirrors `maybeQuantizeKv`'s post-construction in-place
-  swap but runs *once before prefill*: paging changes layout, not
-  arithmetic, so there is no "convert when populated" trigger. Only fresh
-  (`offset === 0`) plain `KVCache` entries are replaced; rotating layers
-  keep their scheme — mixed paged-full + rotating-sliding is the supported
-  shape. Default block size 256 = `KVCache.STEP`, so v1's growth
-  granularity is a permutation of today's into reusable slots, not a new
-  tuning axis.
-
-### 6.3 Gates (explicit refusals, never silent downgrades)
-
-- Startup (`server.ts`): `--paged-kv` with any `--kv-quant`
-  (affine or TurboQuant), `--draft-model`, or a non-`gemma4*` model exits
-  with a clear message; `--paged-kv-block-size` must be a positive
-  integer; `--ssd-cache` alongside it warns that the tier sees nothing.
-  Paging preserves the selected batch size. Env `MLX_BUN_PAGED_KV=1`
-  is equivalent to the flag.
-- Request scope: media (vision/audio) and LoRA-adapter requests strip the
-  flag and run the plain cache path (v1 non-goal cells, never a 400). One
-  effective `pagedKv` value per request keeps the strip and the
-  prompt-cache bypass coherent.
-- Both execution modes: paged requests skip `take`/`put` and the boundary
-  snapshot; caches are disposed on completion. Spec eligibility also
-  excludes `options.pagedKv` (belt on top of the startup refusal).
-- Compiled decode: `CompiledDecode.supports()` excludes `PagedKVCache`
-  automatically (not one of the four supported classes) — a
-  data-dependent block-list length is the shape shapeless replay already
-  broke on.
-- Shared execution: `PagedKvRows` owns independent block pools and immutable
-  row snapshots. A request-state policy creates the storage and supplies no
-  reusable-prefix store. Its opaque compatibility key keeps incompatible
-  layouts in separate cohorts without making scheduling inspect paging flags.
-
-### 6.4 Follow-ups (dependency order) and open items
-
-1. Direct paged attention and block sharing — row integration now uses the
-   shared layout interface with independent per-request pools. Attention still
-   gathers padded contiguous inputs. Measure block-level shared arenas and
-   direct reads before replacing this compatibility implementation. Tracked under the plan anchor (Phase 18
-   S3+).
-2. Block-level CoW prefix sharing — refcounted block table, fork on
-   divergent write; extends `PromptCache`'s entry-level ref-counting down
-   to blocks.
-3. Quantized paged blocks — dtype-parametric block descriptor sharing one
-   block-index space.
-4. Fused paged-attention Metal kernel (optiq's `sdpa_2pass_paged` is the
-   port source) — kills the per-step gather copy.
-- Perf disclosure: paged-on vs paged-off decode/prefill on the quiet
-  reference machine has not been recorded in `docs/reference/benchmarks.md`
-  (expected: a small decode regression at batch=1, to be reported plainly).
-- `PagedKVCache` has no `signature()` (2.4) and no kv-store codec; both
-  are prerequisites for any prompt-cache or SSD integration.
+
+### 5.10 Comparison with other serving caches
+
+Reviewed 2026-09-12 against published designs, not private provider internals.
+
+| System | Algorithm and ownership | Application here |
+| --- | --- | --- |
+| [Mooncake / Kimi](https://github.com/kvcache-ai/Mooncake), [SSD offload](https://kvcache-ai.github.io/Mooncake/design/store/ssd-offload.html) | A cache service owns RAM/SSD placement. Background offload records a disk replica on successful write; the same lookup can restore from SSD. | One cache API with independent storage work. No application-managed demotion. |
+| [SGLang HiCache](https://docs.sglang.io/docs/advanced_features/hicache_design) | A radix tree shares token spans and tracks memory placement. Supports proactive, selective and eviction-triggered writes, plus storage prefetch. Decode output can populate storage. | Keep output-produced checkpoints. Evaluate block sharing and prefetch after the persistence correction. |
+| [LMCache MP](https://docs.lmcache.ai/mp/) | Independent store, eviction and prefetch controllers. Watermark-triggered RAM eviction uses LRU; asynchronous storage adapters handle lower tiers. | Separate write completion from residency policy. Keep the storage interface independent of inference. |
+| [vLLM prefix caching](https://docs.vllm.ai/en/latest/design/prefix_caching/) | Token blocks use hashes including their preceding prefix. Unused blocks are evicted by recency. | Shared prefix blocks could reduce whole-entry duplication. |
+| [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) | Public API documents exact-prefix reuse and expiration. | The same avoided-prefill benefit; this documentation does not establish its internal storage policy. |
+
+The baseline storage unit is a complete checkpoint. Repeated conversation
+snapshots duplicate attention history on SSD. Block deduplication in §5.11
+preserves the existing state-kind contract: Qwen recurrent state and method
+attachments cannot be trimmed to arbitrary earlier tokens. They need valid
+checkpoint boundaries alongside any shared attention blocks. RAM LRU is the
+initial policy; §5.11 records the measured frequency-weighting, prefetch and
+block-layout alternatives.
+
+### 5.11 Cache expansion program
+
+Authorized scope: implement and measure all five opportunities from §5.10.
+Baseline is `c953f2f`, with an independent CPU writer and one RAM/SSD cache.
+Preserve existing cache files and explicit serial support. Storage format,
+residency policy, transfer work and attention kernels remain independently
+selectable for paired comparisons; select defaults from measured results.
+
+| Step | Implementation | Measurement and completion |
+| --- | --- | --- |
+| C1 | Immutable content-addressed blocks shared by checkpoint manifests. Attention token spans share storage; recurrent and method state preserve exact boundaries. | Same restored bytes/continuations, atomic publication and shared-block lifetime; bytes written and retained across growing/divergent conversations. |
+| C2 | Cache-owned asynchronous reads and preparation-time prefetch with promotion into the existing RAM tier. | Same selected prefix and continuation; restore/TTFT, event-loop delay and concurrent decode against synchronous restore. |
+| C3 | Pluggable retention policy, with LRU baseline and an alternative using reuse frequency, state size and saved processing cost. | Replay identical access traces; compare misses, avoided prefill, eviction/reload traffic and peak residency. |
+| C4 | Segmented transfers and bounded packing scratch, preserving tensor layout contracts. | Copied bytes, temporary memory, write/read time and inference overlap on the same payload. |
+| C5 | Direct paged-attention Metal execution through cache/attention interfaces, including shared row layouts and quantized representations. | Same-B numerical controls, B1/B>1 prefill/decode, gather/padding allocations and complete-request timing. |
+
+Implementation is in `storage/kv-writer.worker.js` (CPU transfers), `kv-store`
+(codec reconstruction), `PromptCache.prefetch` (request preparation),
+`storage/retention-policy` (RAM choices), and the paged attention/layout modules.
+Format 5 manifests reference SHA-256 content blocks. Block boundaries reset per
+leading row so growing token axes preserve earlier head blocks. Blocks are
+committed before their manifest; garbage collection is serialized with writes
+and removes only content no remaining manifest references. Explicit SSD caps
+account unique committed blocks rather than charging every checkpoint for a
+shared copy. Existing format 3/4 files remain readable.
+
+Restore reads use a separate worker queue and page-aligned host allocations.
+After reads finish, the owner thread transfers allocation ownership to MLX
+through a native destructor; no JavaScript callback runs on GPU completion.
+Cache codecs are shared with synchronous restore. Plain KV still uses its
+established capacity-growth operation; the async transport does not change
+cached values or recurrent/checkpoint boundaries.
+
+Use the saved Kanban token histories and seeded synthetic branches to separate
+cache-policy effects from model-generated changes in task length. Existing
+native oracles remain authoritative. Each implementation records paired M4
+measurements and applicable M1 correctness; a slower arm stays explicit rather
+than being promoted on architectural grounds alone. Raw reports stay under
+`reports/`; published measurements belong in benchmarks.md.
+
+The bounded C1–C5 implementation and measurements are complete. Shared blocks
+trade write/read latency for space; segmentation removes packing copies for
+contiguous spans. Asynchronous transport overlaps ongoing decode, while a lone
+restart request still pays restore completion. Cost-size retention loses on the
+saved Kanban boundary replay; direct paged attention has shape-dependent wins
+and regressions. The measured selection keeps whole files and LRU, enables
+asynchronous preparation, and leaves direct attention and block storage explicit.
+[Measurements and reproduction](../reference/benchmarks.md#cache-expansion-c1c5-storage-restore-retention-and-paged-attention)
+include the negative results and limits. Full long-task retention acceptance
+and the paging extensions below remain separate work.
+
+### 5.12 Session checkpoint index
+
+The cache owns a second index: application session ID plus numerical namespace
+maps to its latest immutable checkpoint. Numerical/content identity remains
+independent, so session and anonymous requests can share the same state.
+The request carries a session hint through the existing cache port's lookup,
+preparation and publication calls. Ordinary, speculative and paged methods
+forward the hint; scheduling does not own the index or residency decisions.
+
+A matching resident checkpoint bypasses candidate scans. For an evicted known
+checkpoint, the SSD exact index selects its manifest before asynchronous
+preparation adopts state through the existing codecs. Hashes accelerate SSD
+selection; the selected token sequence still establishes identity. A missing
+or incompatible session checkpoint uses ordinary prefix matching, allowing
+history edits, branches and retries without rejecting the request.
+
+New publications advance the session's reference. Explicit sessions also retain
+ordinary generated checkpoints below the anonymous publication threshold.
+RAM eviction first considers
+entries with no session references, then applies the configured policy to
+referenced entries when needed. This is a preference, not an unlimited memory
+pin. Request-scoped preparation interests retain their existing short-lived
+ownership. Closing a session removes its references without deleting cached
+state; clearing the cache removes the session index. Session metadata is
+process-local, and restart rebuilds associations through ordinary SSD hits.
+
+The current API still accepts full histories. This change removes candidate
+search and supplies retention information; it does not claim delta-only input,
+zero tokenization, or automatic reconstruction from a session label. The Pi
+provider passes the session identity its application already owns.
+The index, native retention and fixed-input M4 HTTP comparisons are complete.
+Lookup and reload work decrease; warmed model throughput is effectively
+unchanged. The full Kanban task now completes with all follow-up session hits
+and a durable final flush; independent app acceptance retains two defects.
+[Measurements](../reference/benchmarks.md#session-checkpoint-index-c6) preserve
+both the benefits and the limits. Existing idle-demotion settings remain in
+effect; session preference changes budget/pressure victim selection.
+
+
+### 5.13 Exact tensor objects in the shared cache
+
+`ObjectCache<Value>` exposes exact-key lookup and ownership transfer separately
+from `PrefixCache` matching. `PromptCache.objects` uses the same resident records,
+byte accounting, retention policy, pressure callback and SSD durability queue.
+An object has an empty conversation history, no target KV layers and a private
+namespace containing its full key. Its versioned tensor attachment owns the
+payload. It never supplies a language-model prefix. No synthetic language token
+IDs or second RAM budget are introduced.
+
+RAM reads lend immutable native views. Evicting the resident entry cannot
+invalidate a request's borrowed tensors. A cold read uses the existing async
+serializer with an empty target-layer schema, outside the execution queue;
+publication joins the same RAM policy. Cache clear invalidates pending restore
+publication. A newer resident publication wins over an older in-flight restore.
+The durability coordinator includes attachment-only objects; RAM eviction waits
+for the shared writer to finish as it does for conversation checkpoints.
+
+The Qwen encoder adapter hashes exact preprocessed pixel bytes, grid dimensions
+and its schema revision. A SHA-256 of the actual loaded encoder tensors supplies weight identity,
+computed once under native preparation. The containing SSD store also supplies
+configuration, tokenizer and native numerical identity. Different image bytes or grids miss;
+repeated media can reuse the tensor across request histories. Prompt rendering,
+video timestamps, mRoPE positions, sampling and scheduling remain independent.
+The producer retains its own result while storage takes a separate native view.
+The object lookup finishes before the prompt builder enters the native queue.
+Preprocessing and video extraction are not cached by this change.
+
+Gemma's encoder-free and SigLIP vision towers and its audio tower use the same
+`EncoderCache` adapter. Each tower hashes its immutable weight names, shapes,
+dtypes, bytes and computation settings once. Image/WAV content hashes and a
+producer revision identify individual objects. Vision retains its existing
+pre-divided features; audio retains raw float32 features, then each prompt casts
+them to its embedding dtype before dividing by the embedding scale. That order
+is unchanged. CPU image preprocessing and audio mel extraction precede the
+native execution lease, as do RAM/SSD lookups. The producer never caches chat
+tokens, bidirectional masks or positions. Custom vision encoders without an
+immutable `cacheIdentity` continue to encode normally.
+
+[vLLM's encoder cache](https://docs.vllm.ai/en/v0.18.0/api/vllm/v1/core/encoder_cache_manager/)
+also identifies individual media items by hash and retains encoder outputs
+across requests. Here the existing cache owns both RAM residency and queued
+SSD persistence on unified memory. The bounded Qwen screen and integrated
+acceptance pass on M4; both-machine native/HTTP checks and fresh-process SSD
+reuse preserve their recorded outputs. [Measurements](../reference/benchmarks.md#qwen-encoder-reuse-through-shared-ram-and-ssd-storage-2026-09-13)
+include the one-time weight-fingerprint cost. Gemma prepared inputs and mixed
+RAM/SSD HTTP reuse pass both-machine identity checks. Its M4 ABBA preserves all
+168 paired responses/usage and improves every measured workload; first-use
+cost and changed image overlap are recorded in the
+[Gemma measurements](../reference/benchmarks.md#gemma-encoder-reuse-through-shared-ram-and-ssd-storage-2026-09-13).
+Prepared media conversation state uses the same cache, as described below.
+
+
+### 5.14 Prepared media prefix identity
+
+The Gemma producer can supply a versioned `prefixIdentity` containing every
+media item's exact input/encoder identity, its causal policy, and the rendered
+token prefix through the last media span. The ordinary cache namespace combines
+this identity with the adapter namespace. Consequently, matching histories
+include identical media and preceding tokens; a changed media set or a change
+from image-only bidirectional attention to mixed causal input cannot collide.
+Appending later text leaves the identity stable.
+
+By default, the shared Gemma binding uses ordinary
+RAM/SSD lookup, generated snapshots, prefetch and persistence. The prepared-input
+interface receives the cache's starting position. An uncached input performs
+its atomic media prefill; a restored prefix resumes text after all media through
+the model's ordinary forward method. Scheduling and sampling are unchanged.
+Qwen supplies the same opaque identity with prepared image/video hashes,
+rendered timestamps and all mRoPE positions through its final media span. Its
+resumed text forward uses the current request positions and preserves the
+recurrent state. Recurrent layers still require an exact retained prefix;
+explicit serial prefix reuse remains unqualified. `MLX_BUN_MEDIA_PREFIX_CACHE=0`
+selects encoder-only reuse for comparisons.
+
+The existing minimum reusable offset still identifies a precision transition.
+If delayed quantization occurs after an atomic media prefill, its cached state
+can serve a later conversation turn but cannot rewind before that conversion.
+A shorter request recomputes normally. Start-zero affine/TurboQuant and bf16
+retain their applicable prefix-trimming behavior. This is cache selection,
+not request admission or a memory refusal.
+
+
+## 6. Optional paged KV
+
+`PagedKVCache` stores full-attention planes in per-request block pools.
+`PagedKvRows` composes those pools through the existing row-state interface.
+Scheduling sees request state and attention ports, not block tables or codecs.
+Flags, defaults and supported combinations are documented in
+[server-config.md](../reference/server-config.md).
+
+### 6.1 Storage and ownership
+
+Pools hold bf16 planes or affine packed/scales/biases planes. New tokens alone
+are quantized and written into their destination blocks. Logical block tables
+map sequence positions to physical slots. Trimming returns unused tail blocks;
+restored prefixes can grow beyond their original request's capacity.
+
+Published attention views, cache borrowers and persistence jobs own independent
+MLX handles. A subsequent functional append cannot overwrite their buffers.
+The backend's `paged-cache-codec` preserves block tables, encoded planes and
+layout metadata through the common RAM/SSD cache. `namespacedCache` separates
+paged numerical settings from ordinary prefixes. Neither persistence nor a
+successful write removes a RAM donor.
+
+The current pools are separate per request. Their snapshots share immutable
+MLX storage, but divergent writes can detach a whole pool. Shared physical
+arenas with per-block copy-on-write remain a further optimization.
+
+### 6.2 Attention
+
+The established attention arm gathers blocks and uses stock SDPA. Its bf16
+storage-layout contract remains bit-exact against plain KVCache.
+
+The optional direct Metal arm reads block tables for short causal queries.
+Each SIMD group processes one query/head/sequence partition, retaining a
+softmax maximum, denominator and partial value vector. A second kernel merges
+those partials. Affine variants unpack KV4/KV8 values in the read operation;
+no full-history dequantization or contiguous KV gather is materialized.
+Shared rows retain their own lengths, so direct attention does not pad KV to
+the longest row. Long prefill and explicit masks use the established SDPA
+shape dispatch through the same attention view.
+
+This changes reduction order. Direct attention is Lab, with numerical tolerance
+checks against the same encoded values; it is not a bit-exact mlx-lm claim.
+Tests cover unequal K/V dimensions, grouped heads, partial blocks, multiple
+queries, row retirement, immutable views and restored growth. Performance
+measurements retain both short-context losses and long-context wins in
+[benchmarks.md](../reference/benchmarks.md).
+
+### 6.3 Remaining extensions
+
+TurboQuant page encoding, paged speculative transactions, media/adapter paging
+and shared physical arenas are outside the implemented bf16/affine page layout.
+Existing non-paged TurboQuant and speculative RAM/SSD state remain supported
+through their own codecs and attention interfaces. Extending a page codec or
+kernel does not require a second scheduler.
 
 ## History