mlx-bun 0.3.0 → 0.5.0

package/docs/design/kv-cache.md

@@ -3,7 +3,7 @@ status: landed
 axis: ON
 canonical-for: kv-cache
 plan-anchor: "Phase 18 — Concurrent / batched serving (slots) + parallel load benchmark `[~]` (2026-06-13)"
-last-verified: 2026-08-23
+last-verified: 2026-09-12
 ---
 
 # KV cache — residency and layout
@@ -24,29 +24,32 @@ and open items.
 Code: `src/model/gemma4-base.ts` (Cache interface + the K/V cache family),
 `src/model/batched-*.ts`, `src/lab/paged-kv/paged-kv.ts`, `src/prompt-cache.ts`,
 `src/kv-store.ts`, `src/ssd-cache.ts`, `src/ssd-durability.ts`,
+`src/tiered-prompt-cache.ts`, `src/storage/kv-writer.ts`,
 `src/kv-scheme.ts`, `src/generate.ts` (`maybeQuantizeKv` / `maybePageKv`),
 `src/server.ts` (wiring, `/stats`, `POST /admin/cache/flush`),
 `src/serve/generation-gateway.ts`, `src/serve/batch-scheduler.ts`.
 
 ## 1. Layering
 
-Residency is a three-tier stack, each tier keyed by the exact token prefix
-its KV encodes:
+Active generation owns mutable cache containers. Published checkpoints belong
+to one logical cache, with RAM and SSD storage. On Apple Silicon the GPU and
+CPU access unified memory; publication does not require a GPU-to-CPU copy.
+Each checkpoint is keyed by the exact token prefix its state encodes:
 
 | tier | owner | unit | key |
 |---|---|---|---|
 | live caches | one generation (serial lane or a batch row) | `Cache[]`, one per layer | — |
-| RAM prompt cache | `PromptCache` (`src/prompt-cache.ts`) | whole prefix entry (tokens + `Cache[]`) | tokens + adapter ns |
+| RAM prompt cache | `TieredPromptCache` over `PromptCache` | whole prefix entry (tokens + `Cache[]`) | tokens + adapter ns |
 | SSD cold tier | `SsdCacheStore` (`src/ssd-cache.ts`) over `src/kv-store.ts` files | one `.mlxkv` file per entry | tokens + ns + model/scheme/tokenizer fingerprint |
 
 The prompt cache tiers over the SSD store *inside* `take()` (structural
 `ColdTier` interface), so every consumer (serial lane, batch scheduler
 joiners, `/admin/drain`) gets both tiers through the same `take()`/`put()`
-calls. The default path does no persistence work on the token loop:
-spill/persist happen at request boundaries, evictions, idle sweeps, and
-explicit flushes. The opt-in `--generation-checkpoint N` path deliberately
-pauses at coarse, safe decode boundaries to atomically persist an in-flight
-generation (section 5.7). Paged KV
+calls. The cache queues persistence on publication or eviction. A dedicated
+CPU worker writes evaluated immutable buffers while inference continues.
+Persistence does not evict RAM; only cache residency policy does. RAM reuse
+does not enqueue an extra write or wait for the queue. Explicit flushes await the queued work. The opt-in `--generation-checkpoint N` path transfers owned snapshots at coarse decode boundaries and queues atomic
+persistence without awaiting disk in decode (section 5.7). Paged KV
 (section 6) is a *layout* choice for the live tier and, in v1, bypasses the
 other two.
 
@@ -114,10 +117,10 @@ Two independent checks decide whether a model/scheme can batch
   `KVCache`, `RotatingKVCache`, `BatchableCache`, or (unless
   `MLX_BUN_BATCH_SSM=0`) `SSMCache`. Novel classes (TurboQuant, paged) fail
   this by construction.
-- `#kvBatchable()` — `KvScheme.batchable(config, canConvert)`: only the
-  per-layer `affine-config` scheme batches, and only when every configured
-  layer is a plain or rotating cache that can convert. Uniform `kvBits`
-  and TurboQuant stay serial; a scheme-less gateway must never silently
+- `#kvBatchable()` — `KvScheme.batchable(config, canConvert)`: per-layer
+  `affine-config` and uniform affine KV4/KV8 batch when their cache-owning
+  layers can convert. TurboQuant still requires merged-row layout support;
+  a scheme-less gateway must never silently
   drop quantization (`kv-scheme.ts` makes the capability probe mandatory
   for quantized schemes).
 
@@ -210,34 +213,48 @@ upstream is byte-capped now.
 **`take(prompt, ns)` — prefix sharing.** Finds the entry with the longest
 usable common prefix (usable = common prefix capped at `prompt.length - 1`,
 so at least one token is forwarded; an entry longer than the match must be
-trimmable). Trimmable entries are cloned zero-copy (`cloneKvCaches`: slice
-views of live arrays), trimmed, and returned with a ref-counted `retain`
-share; the donor stays in place, so N agents can share one prefill. An exact
-hit on a Qwen hybrid entry containing recurrent SSM state transfers the donor
-itself to bound resident snapshots; the request puts its extended state back
-on completion. With an SSD tier, `take` persists that exact boundary before
-transferring ownership. An already-durable prefix needs only an index check.
-The extended recurrent state cannot trim back to the consumed boundary, so
-keeping its SSD copy preserves later exact prompt repeats. A failed required
-store leaves the donor and backing owned by the cache and refuses the transfer.
+trimmable). Every RAM hit lends zero-copy views (`cloneKvCaches` and
+`cloneAttachments`) with a ref-counted backing share; the donor remains in
+place. This includes exact-boundary recurrent and method state. Only
+trimmable state may be shortened. Decode extends its own views without
+changing the retained checkpoint.
+
+SSD is an eventually consistent persistence layer. RAM reuse never writes
+to SSD or waits for durability. Background writes retain immutable snapshot
+views while a CPU worker packs, hashes and writes them; explicit flush and
+graceful shutdown wait for persistence. A pending or failed write does not prevent RAM reuse.
+The byte target controls LRU eviction. With SSD enabled, an unwritten victim
+stays resident until storage commits; the target is soft while writes are
+pending or failing.
+
+The native sharing gate checks logical checkpoint bytes after repeated decode
+and after SSD restore. It passes on both development Macs; the M1 long-prefix
+case also preserves three independent 512-token continuations. The queued
+writer test retains snapshot state after both donor and borrower disposal,
+and explicit flush remains pending until the write completes.
 
 **Tier order inside `take()`.** The cold tier is consulted with
 `find(prompt, ns)` (index-only, no I/O) and wins only with a *strictly*
 longer usable prefix; then `restore(handle)` materializes it and the
 divergent tail is trimmed. A cold entry that would need a trim but is
 untrimmable (wrapped ring, SSM) is disposed loudly and the RAM candidate
-(or a fresh prefill) serves. `hits`/`misses` count RAM candidacy only.
+(or a fresh prefill) serves. Tiered restore immediately publishes shared views
+in RAM so another request can reuse them before the borrower finishes.
+`hits`/`misses` count RAM candidacy only.
 
 **`put(tokens, caches, ns, retain)` — supersession + eviction.** Exact
 duplicates (same tokens) are replaced regardless of trimmability. Strict
 prefix-ancestors in the same ns are superseded without spill — but only
 when the *new* entry is fully trimmable; an untrimmable new entry (wrapped
 ring) can only serve exact-length matches, so shorter ancestors (the
-prompt-boundary snapshot) must survive. Then LRU-evict until under cap,
-spilling each evictee to the cold tier first. An entry larger than the cap
-is spilled and disposed, never stored. `onPut` fires after every
-successful put — the server hangs the write-behind scheduler there so both
-lanes' entries persist.
+prompt-boundary snapshot) must survive. `TieredPromptCache` selects the
+least recently used RAM entry when over budget. If it lacks durable SSD
+coverage, the cache queues a write and retains it until that write succeeds.
+It does not evict a hotter entry merely because that entry is already on SSD.
+An oversized checkpoint follows the same persist-before-demotion rule.
+Without SSD, the RAM-only cache discards evicted or oversized entries.
+`TieredPromptCache.put` also owns proactive persistence; `onPut` remains an
+independent observer for generated-token history.
 
 **Prompt-boundary snapshot.** Serial lane (`server.ts`) and batch
 scheduler (`snapshotAt`) both clone-and-put a trim-free *prompt-only*
@@ -251,12 +268,11 @@ always an exact prefix of the next rendering.
 **Namespaces.** `ns` = the adapter spec (`adapters.join("+")`, `""` = base).
 KV computed under one adapter never seeds another's prefill.
 
-**Bypasses.** Media (vision/audio) requests skip the cache (soft tokens are
-identical placeholder ids → false hits), as do paged-KV requests (v1) and
-the speculative lane (v1). Merged batch rows are not re-put on finish;
-never-merged lone rows put their adopted caches back zero-copy, and
-merged rows are extracted per row (`#extractAndPut`) before `filter()`
-mutates the batched inners.
+**Bypasses.** Media and optional paged-KV requests retain their documented
+cache qualification rules. Shared speculative execution publishes target and
+method state together where supported; method attachments preserve the exact
+boundary needed for reuse. Merged batch rows are extracted before retirement
+changes the batched state. The cache owns the extracted checkpoint.
 
 **Idle demotion (`demoteIdle`).** Entries unused past a threshold spill to
 the cold tier and free their GPU arrays; the prefix stays reachable via
@@ -281,20 +297,18 @@ full prefill (GEMV-vs-GEMM reduction order). Both tiers share it.
 
 ### 5.1 Decisions
 
-- **D1 — Whole-prefix-entry spill, not paged blocks.** Single-user serving
-  with a prefix-entry cache that grows per conversation. Per-block hashing
-  and save-queue bookkeeping during decode is where oMLX's measured ~20%
-  steady-state tax came from; ours does zero work on the token loop
-  (prefix tokens are the key). Partial-prefix reuse works without paging:
-  restore + `trim()`. The economics that justify the tier are
-  SSD-vs-recompute, not SSD-vs-RAM: at 12B prefill rates a 30k agent
-  context is minutes of GPU compute against a ~1 s NVMe restore.
-- **D2 — One store, two tiers.** The cold tier is bound *into*
-  `PromptCache` (section 4); restart persistence comes from a debounced
-  write-behind snapshot after each put, not only from spill-on-evict.
+- **D1 — Whole-prefix-entry storage.** A checkpoint stores a complete
+  conversation boundary. This reuses the existing cache codecs and avoids
+  per-token storage bookkeeping, but duplicates shared attention history
+  between files. Block deduplication is a separate layout optimization to
+  measure against this baseline; recurrent state still needs exact boundaries.
+- **D2 — One store, two tiers.** `TieredPromptCache` owns RAM residency and
+  the SSD backend. Proactive persistence after publication provides restart
+  survival without waiting for an eviction.
 - **D3 — Own page-aligned format, not safetensors.** `src/kv-store.ts`
-  writes `MLXBUNKV2\n`-magic files whose current layout is the **v3
-  streaming writer**: header sized up front with fixed-width tensor
+  writes `MLXBUNKV2\n`-magic files with a streaming writer: ordinary KV
+  retains format v3; checkpoints with method companion state use v4.
+  The header is sized up front with fixed-width tensor
   hashes, tensors materialized/hashed/written one at a time at 16 KiB
   aligned offsets, header patched last (peak host transient = one
   tensor). Header carries `formatVersion`, `modelId`,
@@ -344,10 +358,13 @@ full prefill (GEMV-vs-GEMM reduction order). Both tiers share it.
   `--memory-budget` is unchanged (admission already assumes worst-case
   bf16). `--paged-kv` warns that the tier sees nothing (section 6).
 - **D6 — In-flight resume is explicit.** `--generation-checkpoint <tokens>`
-  requires `--ssd-cache` and `--batch 1`. It is off by default because each
-  checkpoint synchronizes and writes the live KV, trading decode speed and
-  SSD writes for bounded lost work. An identical request resumes; a changed
-  prompt or sampling policy starts normally.
+  requires `--ssd-cache` and supports eligible shared ordinary execution. It
+  is off by default. Captured states queue for SSD persistence; an explicit
+  flush establishes durability of queued work (section 5.7). An identical request resumes; a changed
+  prompt or sampling policy starts normally. Ordinary adapter requests use the
+  same continuation hooks under their existing execution context. The checkpoint
+  identity includes the ordered adapter IDs and content-revision namespace; a
+  base-model request or replacement adapter cannot restore that checkpoint.
 
 ### 5.2 Restore is a streamed copy (supersedes the original zero-copy plan)
 
@@ -367,9 +384,9 @@ ordering is unchanged (the server's `coldTier.restore` returns
 `() => {}`). Comments in `prompt-cache.ts` that still say "zero-copy
 mmap" describe the superseded mechanism.
 
-The write side is the mirror: hashing and writing from a zero-copy view
-of the contiguous mlx buffer (`rawBytesView`), no JS-heap copy at any
-point.
+The asynchronous writer reads evaluated native storage directly. Contiguous
+tensors need no packing copy. Strided tensors are packed on the CPU, one
+tensor at a time, to omit capacity padding without a GPU `contiguous` call.
 
 ### 5.3 `find()` returns partial matches, gated on trimmability
 
@@ -384,57 +401,52 @@ Store-side supersession mirrors `PromptCache.put`: exact duplicates
 replaced; prefix-ancestors superseded only when the new entry is
 trimmable.
 
-### 5.4 Write-behind scheduling contract: own each tensor step
-
-Every per-tensor flush step is real GPU-stream and JS-thread work
-(`ops.contiguous` on the decode stream, a synchronous eval for
-`rawBytesView`, a synchronous multi-MB `writeSync`). Interleaving those
-between decode tokens taxed decode at long context, so:
-
-- `saveKvCacheAsync` / `SsdCacheStore.storeAsync` accept a per-step runner.
-  The server passes `gateway.runExclusive`, so the idle decision and the
-  blocking MLX readback are one atomic engine turn. The earlier
-  `gateway.onIdle()` check had a check/use race: a request could begin after
-  the check and overlap the next tensor sync. A request arriving mid-flush
-  previously could run between tensor writes. Serial generation now first
-  drains the durability coordinator outside the engine lock, so pending
-  snapshot clones release their buffers before the next prefill.
-- `GenerationGateway.busy` covers both lanes (serial mutex held/awaited or
-  batch rows active/pending).
-- `MLX_BUN_SSD_WRITEBEHIND=0` disables write-behind snapshots entirely
-  (kill switch + paired-A/B lever; restart survival then degrades to
-  spill-on-evict).
-
-Serial requests may wait for outstanding snapshots to finish. The drain
-runs before acquiring the generation lock because the writer needs that
-same lock. Continuous batching retains per-tensor scheduling.
-
-The RAM cache's configured byte cap is only an upper bound. Serial serving
-checks MLX's live allocation count against the smaller of admission's usable
-memory and Metal's recommended working set, reserving prefill workspace and
-the prompt's KV footprint for growth/copy-on-write overlap. Pressure relief
-persists LRU entries synchronously before disposing them, avoiding deferred
-spill clones that would keep the memory resident. The selected request cache
-remains owned by the request; older entries remain reusable from SSD. Qwen
-checks between prefill layers, and serial decode checks every 256 tokens.
-If no reclaimable entries remain, the guard raises a request error instead
-of intentionally proceeding over that budget. In-flight checkpoints survive.
-
-### 5.5 Bounded retention: `SpillQueue` (`kv-store.ts`)
-
-All three producers — eviction spills, idle demotions, write-behind
-snapshots — go through one serial, byte-capped queue. Pending clones pin
-their entries' GPU buffers until the generation-locked flush gets a turn; without
-a cap, sustained traffic (gate starved, evictions ongoing) made resident
-memory = prompt-cache cap + every queued clone. Policy: cap default 2 GiB
-(`MLX_BUN_SSD_SPILL_QUEUE_GB`; `0` keeps only the newest + in-flight
-clone); over cap the *oldest* not-in-flight item drops and its clones are
-disposed immediately; the item just enqueued is never its own victim
-(soft cap). A dropped spill is a future cache miss, never a wrong result.
-Clones are disposed on every settle path (fulfil, reject, drop). The
-snapshot timer never holds batch admission: while rows are active it
-re-arms instead of grabbing the exclusive (entries are immutable; a late
-snapshot is equally valid).
+### 5.4 Background persistence and RAM residency
+
+`TieredPromptCache` owns RAM residency, SSD writes and durable-coverage checks.
+`put` materializes the checkpoint before publication. `saveKvCacheAsync`
+uses the existing codecs to describe the file and obtains host-visible native
+pointers, shapes and strides on the MLX owner thread. Its optional preparation
+runner does not cover disk I/O. No MLX handles cross into the writer.
+
+`KvWriter` embeds a dedicated Bun CPU worker in source and compiled builds.
+The worker packs strided layouts, hashes tensor bytes, writes a temporary file,
+fsyncs it and renames it atomically. It imports no MLX or scheduler code and
+needs no generation lock. Contiguous tensors use direct buffer views; packing
+scratch is reused across strided tensors. The cache owns the retained state,
+and the queued write holds shared views through its completion. Those views
+share immutable storage rather than copying the entire checkpoint.
+
+The owner thread still prepares native views and updates the SSD metadata
+index after completion. A background worker removes blocking payload I/O from
+that thread; it does not make shared memory bandwidth or snapshot preparation
+free. Measure those costs with concurrent decode before claiming a speedup.
+
+Writing a copy does not remove RAM residency. Byte pressure chooses the actual
+LRU entry. If its SSD copy is missing, it queues persistence and waits for the
+completion callback to retry eviction. Failed writes leave the RAM entry
+usable and dirty. Age-based eviction follows the same rule and runs without
+acquiring the scheduler's generation lock. Neither policy refuses requests or
+changes context, batch size or sampling settings.
+
+`MLX_BUN_SSD_WRITEBEHIND=0` disables proactive persistence after publication;
+eviction still queues persistence before removing RAM. Explicit flush and
+shutdown await outstanding writes. A hard exit can lose uncommitted writes.
+
+### 5.5 Persistence queue and capacity
+
+Normal prefix persistence uses one serial `SpillQueue` with no independent
+byte-drop policy. RAM ownership and pending write views share the checkpoint;
+queue completion releases only the writer's reference. RAM cache capacity
+and age policy decide demotion. They do not cancel an unwritten snapshot to
+reduce the queue's accounting. While the oldest victim is writing, RAM can
+exceed its target. A storage failure keeps that victim resident and retryable.
+
+The optional interrupted-generation coordinator still uses the generic
+bounded queue for supersedable intervals. `MLX_BUN_SSD_SPILL_QUEUE_GB`
+(default 2 GiB) now controls that queue only. It does not limit retained prompt
+history. SSD capacity is independent: no capacity eviction without an explicit
+positive `--ssd-cache-max`.
 
 ### 5.6 Durability boundary (`src/ssd-durability.ts`)
 
@@ -445,17 +457,16 @@ the write — the server implied durability it did not have.
 
 - `schedule(tokens, ns)` records a *dirty* key that includes the exact
   token sequence (two same-ns, same-length conversations cannot cancel
-  each other) and arms the debounce.
-- An attempt checks SSD coverage under `gateway.runExclusive` before cloning.
+  each other) and arms a next-tick attempt.
+- An attempt checks committed SSD coverage before cloning.
   Otherwise it snapshots the RAM entry (`findExact` + `cloneKvCaches`) and
-  enqueues it; a busy gateway re-arms;
-  a dropped or failed store leaves the key dirty so it is retryable.
+  enqueues it without consulting scheduler activity. A failed store leaves
+  the key dirty so it is retryable.
   A snapshot that vanished from RAM is "stored" only if the SSD index
   already covers the prefix (`hasDurablePrefix`), else "missing" and still
   dirty. A second flush cannot clear that failure without durable coverage.
 - `flush()` cancels timers, awaits in-flight attempts and `drain()`, then
-  forces each dirty record version once, one at a time (so the queue cap
-  cannot drop a boundary snapshot while a large final snapshot is in flight).
+  forces each dirty record version once, one at a time.
   A replacement scheduled during a write receives its own attempt. After
   all writes settle, the coordinator checks missing ancestors against the
   committed SSD index again: a later trimmable descendant may now cover a
@@ -495,13 +506,161 @@ IDs and every sampling/KV policy field. Grammar, token-fill, media, paged KV,
 and logprob requests are not checkpointed in v1 because they carry additional
 state or response data that cannot yet be reconstructed by token replay.
 
+Ordinary continuation adds an optional request port to
+that same ordinary driver. Checkpoint lookup, keys and persistence are shared
+with the compatibility executor; the numerical driver supplies owned row
+snapshots at an emitted-prefix/pending-token boundary. Restored rows complete
+preparation without a target forward, before merging cold prefill rows, and
+recover sampler seed, history and sample index. The server binding qualifies
+ordinary requests through supported row-cache layouts, including recurrent Qwen
+and full-attention Llama/MiniCPM as well as Gemma4. It transfers owned row
+snapshots to a bounded persistence coordinator without awaiting disk in decode.
+Snapshot preparation runs on the owner thread; payload serialization and
+writes use the CPU worker. Completion
+invalidates reuse immediately and queues cleanup after any in-flight write;
+new attempts cannot be erased by older cleanup. Cancellation retains eventual
+persistence, and explicit flush reports failed or pending durability. A hard
+process exit may lose queued intervals. Plain greedy retains the ordinary
+vectorized sampler when no history-dependent processors are active. Supported KV layouts retain their existing affine, per-layer and full-attention
+TurboQuant conversion policy. Requests remain without
+media, grammar, fill, paging, logprobs or speculative methods. Native tests
+compare interrupted/restored runs at the same batch geometry; arbitrary
+cohort changes are not an exact numerical identity guarantee. The HTTP fixture
+checks serial B1, shared B1 and shared B4 with identical seeded requests and
+held arrivals, and inspects the generated checkpoint before restart.
+The configured spill queue budget applies to continuation snapshots; normal
+prefix persistence follows the RAM cache residency policy. Flush duration and pending/failed
+statistics cover both queues.
+
 Only the newest successful checkpoint for a request is retained. The old file
-remains valid until the new file's fsync+rename completes; a crash mid-write
-therefore loses at most one interval. Normal completion removes the checkpoint.
+remains valid until the new file's fsync+rename completes. Queue backpressure
+can leave more than one interval undurable; a crash loses work since the latest
+successful write. Normal completion immediately invalidates the checkpoint and
+queues its disk removal.
 Client cancellation throws through the serial token callback and preserves the
 last checkpoint, so terminating a long-lived client does not erase its restart
 point.
 
+Fresh-model compiled continuation preserves the device sampler's uint32 pending
+register. The Gemma model materializes its computed FP32 RoPE frequencies once
+before compiled graph capture: a restored request may skip the prefill that
+normally evaluates those constants. Capturing their unevaluated construction
+graph can change arithmetic on M1. The forced-token continuation regression
+compares a warmed model and a fresh model with cloned state, including every
+active cache byte and full logits across delayed KV conversion. Live delayed
+row layouts publish compact state through their extraction interface before
+persistence; they do not need storage codecs for the live wrapper itself.
+
+### Method companion state
+
+`PrefixCache<State, Attachment>` is the shared consumer contract for ordinary
+and batched execution. It has no scheduling or storage-tier operations. The
+MLX binding supplies target cache layers plus optional `CheckpointAttachment`
+records containing a method schema, scalar metadata and tensor snapshots.
+The method owns their meaning and alignment; the shared cache owns retention.
+Method and drafter compatibility must be part of the namespace before a
+producer publishes these records. Storage does not interpret draft state as
+extra target-model layers.
+
+RAM accounting, eviction, idle demotion and SSD persistence include
+companion tensors. Checkpoints containing them require an exact prefix
+boundary and preserve shorter fallback entries. Write-behind captures companion
+views together with target-cache views under the existing exclusive boundary.
+The writer persists them in the same atomic file and the same per-tensor write
+loop. Restore copies and verifies them through the existing tensor reader.
+Ordinary files retain v3 compatibility; older readers reject v4 files rather
+than restore an incomplete checkpoint.
+
+The storage implementation is in the working tree. Synthetic gates cover RAM
+ownership, divergent-prefix fallback, SSD eviction, fresh-process restore,
+corruption, queue drops/failures and durability on both development Macs.
+Qwen MTP now uses this interface for prefill snapshots; its private store and
+separate budget option are removed. `DraftSource.checkpoint` supplies the draft
+KV and pending true hidden row. The namespace includes a SHA-256 of draft
+configuration and shard bytes, resolved at provider load, plus the mounted
+adapter revision and KV policy. Model loading resolves that digest through
+`artifactIdentity`, independently of inference and scheduling. A memo key includes
+the configuration seed, sorted shard names, resolved paths, device/inode, size,
+and nanosecond modification/change times. Unchanged revisions reuse the exact
+existing SHA-256; a miss reads every shard byte. Same-size writes with restored
+mtime and changed symlink targets invalidate the memo. Rechecking revisions after
+hashing prevents persisting a digest under a revision that changed during reading.
+
+`ArtifactIdentityStore` owns memo retention and persistence. Its file adapter uses
+`~/.cache/mlx-bun/artifact-identities/`, keeps RAM hits and queues atomic file writes
+without awaiting them in model loading. Missing, malformed or unwritable memo
+storage does not prevent computing the identity. It stores only digests, not
+weights, prompts or KV. Qwen MTP, standalone, assistant, DeepSpec and DSpark use
+the same identity service; existing namespaces and SSD entries remain compatible.
+Tests cover byte identity, no weight reads after a fresh-store memo hit, file
+mutation, configuration changes, symlink replacement and write failure.
+
+Native bf16/KV4 gates on M1 Max and M4 Pro
+preserve live target/companion state hashes, emitted IDs and acceptance decisions
+through repeated RAM reuse and SSD cache/provider restart. SSD allocation
+padding is excluded from the live-state comparison. Generated-output acceptance
+is described below; long-conversation timing remains R17 work.
+
+The shared method publishes completed row state before
+retirement through the same `PrefixCache.put` port. It records only inputs
+retained by committed target/draft rounds; an unprocessed correction or bonus
+token is absent from the key. The target row layout and draft provider supply
+their snapshots independently. Failed and cancelled consumers publish no
+generated checkpoint. Persistence continues through the existing queue.
+
+Generated BPE IDs are not necessarily the canonical encoding of their decoded
+text. `GeneratedTokenHistory` retains bounded text/ID provenance from published
+prefixes and recovered SSD headers. Request preparation uses the original IDs
+when the rendered conversation contains that exact text prefix, encodes only
+the remaining suffix without another BOS, and checks that the combined IDs
+decode to the complete rendered request. Edited history or an unsafe decoder
+boundary keeps canonical tokenization. KV lookup still requires exact IDs and
+the existing execution namespace; no state is relabeled under different IDs.
+The template's primer length is measured canonically and applied to the actual
+history-preserving token sequence. This can change input token counts, and
+therefore continuation logits, relative to re-encoding all generated text.
+Logit parity remains a contract for the same input IDs and execution shape.
+The stable-boundary probe includes both text and tool replies followed by a
+new user turn. Qwen can omit an empty thinking primer when rendering a prior
+tool turn; a text-only probe retained three extra tokens in the small-model
+HTTP fixture and prevented recurrent-cache reuse. The corrected probe keeps
+the last shared prefix. It never rewrites existing KV under the changed text.
+Republishing an identical token sequence refreshes its existing provenance
+entry without decoding the whole conversation again. Equality covers every ID;
+equal text from different IDs still replaces provenance. This avoids repeated
+text conversion during request completion and preserves the retention limit.
+
+Cache pressure maintenance uses 85% of the smaller of the recommended device
+working set and an explicit allocator limit as a residency target. It includes
+known weight bytes when file-backed weights have not appeared in MLX active
+allocation. `PrefixCache.reclaim` asks the cache to demote LRU donors. With SSD,
+only durable donors can leave RAM; pending writes remain queued. A selected
+request retains its own views. Shared buffer ownership is not counted twice as
+RAM plus queued data. Lookup, prefill boundaries and periodic decode maintenance
+call this interface; decode checks every 256 steps. It does not constrain
+request admission, context, sampling or batch size.
+
+M1 and M4 native bf16, affine KV4 and fused TurboQuant K8/V3 tests cover four active
+rows, early retirement, exact sampled-ID coverage, unchanged retained bytes
+after sibling and follow-up work, shorter-prefix selection after an edit, and
+matching RAM/SSD continuation and acceptance. Existing paired-prefill tests
+also pass. A second native gate records actual sampled IDs for an echo tool
+turn with thinking off/on, passes them through the production output parsers,
+and renders the subsequent tool-result request with the same chat template.
+Both published checkpoints are exact token prefixes of that next request.
+A production HTTP gate now passes on both Macs for bf16, KV4 and fused K8/V3, with
+thinking off/on. The next tool-result request reuses generated content in RAM.
+After a durable flush, a separate server process restores the first turn's
+SSD state and reproduces the same response, logprobs and acceptance. The test
+copies only the first turn's flushed files before running the RAM continuation,
+so no later prefill checkpoint can explain the restart hit. All six cases
+pass, including the actual wire `reasoning` alias and template options.
+The five-file change is adopted alongside the captured prefill-policy fix.
+Native acceptance of that final composition, long saved Kanban replay,
+pressure and complete next-turn timing remain open.
+It does not resolve the older serial prototype's bf16 comparison with fresh
+full prefill, whose operation shapes differ from a generated history.
+
 ### 5.8 Invalidation
 
 Compatibility key = `configFingerprint(config)` + `KvScheme.cacheKey` +
@@ -538,117 +697,254 @@ the model's prototype cache before opening the tensor mmap.
   ever lands.
 - Block-granular dedup for multi-user shared prefixes stays out of scope
   (D1); revisit only with the paged follow-ups in 6.4.
-- Stale comments: `prompt-cache.ts` still describes restore as zero-copy
-  mmap (5.2).
-
-## 6. Optional paged KV (`src/lab/paged-kv/paged-kv.ts`, `--paged-kv`)
-
-Default off; with the flag unset no paged code executes and the plain
-`KVCache`/`RotatingKVCache` path is byte-identical.
-
-### 6.1 What it buys here, honestly
-
-vLLM's PagedAttention solves contiguous worst-case per-sequence
-reservation and allocator fragmentation for a multi-tenant server.
-mlx-bun is one process on unified memory with byte-budget admission
-already shipped, so what transfers is: (a) padded-batch waste removal —
-today's batched buffer width tracks the longest live row
-(`mergeKVRows`/`extendKVRows`), so a short row cohabiting with a long one
-pays the long one's KV; and (b) block-level copy-on-write prefix sharing.
-Neither is in v1. **At batch=1 paging buys nothing** — gather-before-attend
-is bandwidth tax on a bandwidth-bound decode. v1 ships the
-correctness-proven block-manager abstraction at serial scope as
-scaffolding for the follow-ups, not as a speed win.
-
-This is neither of the two paging rejections already on record:
-paged KV as a *prompt-cache substitute* (rejected 2026-07-07 — `take()`'s
-zero-copy clones already share physically) and paged *blocks* as the SSD
-spill granularity (D1). It is the rung-3 allocation abstraction from
-docs/design/batching.md.
-
-No external oracle exists (mlx-lm's `cache.py` has no paged cache); the
-gate is mlx-bun's own plain `KVCache`, valid because the claim is storage
-equivalence (same bytes, different arrangement), so the bar is bit-exact
-(tol 0): `tests/paged-kv.test.ts` (model-free: block-boundary crossing,
-free-list reuse after trim, typed exhaustion, gathered fetch vs a plain
-reference, dispose) and `tests/paged-kv-parity.test.ts` (weights-gated
-greedy trajectory paged-on vs paged-off).
-
-### 6.2 Mechanism
-
-- **`BlockPool`** — per-layer arena: K and V pool tensors
-  `[numBlocks, H_kv, blockSize, headDim]` (V may differ in head dim),
-  `ops.zeros`-allocated so they are mlx-owned end to end (no host-pointer
-  alignment/dtor hazards). LIFO free list; `alloc()` throws the typed
-  `PagedPoolExhausted`; writes are `ops.sliceUpdate` at
-  `[block, 0, within, 0]` — the incoming `[1,H,l,D]` piece matches the
-  destination slice directly.
-- **`PagedKVCache implements Cache`** — standalone (not a `KVCache`
-  subclass, the TurboQuant reasoning: every `instanceof` gate excludes it
-  and routes to the monolith forward). `updateAndFetch` splits the
-  incoming `L` along block boundaries, allocates tail blocks as reached,
-  then gathers occupied blocks with `ops.takeAxis(pool, blockTable, 0)` →
-  transpose `[1,0,2,3]` → reshape `[1,H,nb·bs,D]` → slice to `offset`. Only
-  existing bound ops — no `mlx_gather` FFI binding was needed. `makeMask`
-  is `KVCache.makeMask`'s logic; `trim(n)` rewinds `offset` and frees
-  now-unoccupied tail blocks (stale bytes past `offset` are never read —
-  the `KVCache` padding invariant, block-shaped). Block table is a
-  host-side `number[]`; all intermediates are bound and disposed.
-- Pool lazily allocates on the first write (head count/dims/dtype from
-  the first k/v pair) and is sized once from `capacityTokens` =
-  prompt + `max_tokens`, so exhaustion is unreachable absent an
-  accounting bug — it exists as a tripwire, never silent truncation.
-- **Wiring** — `maybePageKv(cache, options, capacityTokens)`
-  (`generate.ts`) mirrors `maybeQuantizeKv`'s post-construction in-place
-  swap but runs *once before prefill*: paging changes layout, not
-  arithmetic, so there is no "convert when populated" trigger. Only fresh
-  (`offset === 0`) plain `KVCache` entries are replaced; rotating layers
-  keep their scheme — mixed paged-full + rotating-sliding is the supported
-  shape. Default block size 256 = `KVCache.STEP`, so v1's growth
-  granularity is a permutation of today's into reusable slots, not a new
-  tuning axis.
-
-### 6.3 Gates (explicit refusals, never silent downgrades)
-
-- Startup (`server.ts`): `--paged-kv` with `--batch N>1`, any `--kv-quant`
-  (affine or TurboQuant), `--draft-model`, or a non-`gemma4*` model exits
-  with a clear message; `--paged-kv-block-size` must be a positive
-  integer; `--ssd-cache` alongside it warns that the tier sees nothing.
-  With no explicit `--batch`, the CLI pins `--batch 1` (default is 8) and
-  says so. Env `MLX_BUN_PAGED_KV=1` is equivalent to the flag.
-- Request scope: media (vision/audio) and LoRA-adapter requests strip the
-  flag and run the plain cache path (v1 non-goal cells, never a 400). One
-  effective `pagedKv` value per request keeps the strip and the
-  prompt-cache bypass coherent.
-- Serial lane: paged requests skip `take`/`put` and the boundary
-  snapshot; caches are disposed on completion. Spec eligibility also
-  excludes `options.pagedKv` (belt on top of the startup refusal).
-- Compiled decode: `CompiledDecode.supports()` excludes `PagedKVCache`
-  automatically (not one of the four supported classes) — a
-  data-dependent block-list length is the shape shapeless replay already
-  broke on.
-- Batch lane: unreachable by construction (startup pins serial); the
-  `RequestShape` gate to add when batched paging lands is noted here.
-
-### 6.4 Follow-ups (dependency order) and open items
-
-1. Batched integration — `PagedKVCache` into the scheduler's inner union;
-   block-count admission replacing byte projection; join/filter/extract
-   allocating and freeing blocks instead of pad+concat. This is where the
-   padded-batch-waste win lives. Tracked under the plan anchor (Phase 18
-   S3+).
-2. Block-level CoW prefix sharing — refcounted block table, fork on
-   divergent write; extends `PromptCache`'s entry-level ref-counting down
-   to blocks.
-3. Quantized paged blocks — dtype-parametric block descriptor sharing one
-   block-index space.
-4. Fused paged-attention Metal kernel (optiq's `sdpa_2pass_paged` is the
-   port source) — kills the per-step gather copy.
-- Perf disclosure: paged-on vs paged-off decode/prefill on the quiet
-  reference machine has not been recorded in `docs/reference/benchmarks.md`
-  (expected: a small decode regression at batch=1, to be reported plainly).
-- `PagedKVCache` has no `signature()` (2.4) and no kv-store codec; both
-  are prerequisites for any prompt-cache or SSD integration.
+
+### 5.10 Comparison with other serving caches
+
+Reviewed 2026-09-12 against published designs, not private provider internals.
+
+| System | Algorithm and ownership | Application here |
+| --- | --- | --- |
+| [Mooncake / Kimi](https://github.com/kvcache-ai/Mooncake), [SSD offload](https://kvcache-ai.github.io/Mooncake/design/store/ssd-offload.html) | A cache service owns RAM/SSD placement. Background offload records a disk replica on successful write; the same lookup can restore from SSD. | One cache API with independent storage work. No application-managed demotion. |
+| [SGLang HiCache](https://docs.sglang.io/docs/advanced_features/hicache_design) | A radix tree shares token spans and tracks memory placement. Supports proactive, selective and eviction-triggered writes, plus storage prefetch. Decode output can populate storage. | Keep output-produced checkpoints. Evaluate block sharing and prefetch after the persistence correction. |
+| [LMCache MP](https://docs.lmcache.ai/mp/) | Independent store, eviction and prefetch controllers. Watermark-triggered RAM eviction uses LRU; asynchronous storage adapters handle lower tiers. | Separate write completion from residency policy. Keep the storage interface independent of inference. |
+| [vLLM prefix caching](https://docs.vllm.ai/en/latest/design/prefix_caching/) | Token blocks use hashes including their preceding prefix. Unused blocks are evicted by recency. | Shared prefix blocks could reduce whole-entry duplication. |
+| [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) | Public API documents exact-prefix reuse and expiration. | The same avoided-prefill benefit; this documentation does not establish its internal storage policy. |
+
+The baseline storage unit is a complete checkpoint. Repeated conversation
+snapshots duplicate attention history on SSD. Block deduplication in §5.11
+preserves the existing state-kind contract: Qwen recurrent state and method
+attachments cannot be trimmed to arbitrary earlier tokens. They need valid
+checkpoint boundaries alongside any shared attention blocks. RAM LRU is the
+initial policy; §5.11 records the measured frequency-weighting, prefetch and
+block-layout alternatives.
+
+### 5.11 Cache expansion program
+
+Authorized scope: implement and measure all five opportunities from §5.10.
+Baseline is `c953f2f`, with an independent CPU writer and one RAM/SSD cache.
+Preserve existing cache files and explicit serial support. Storage format,
+residency policy, transfer work and attention kernels remain independently
+selectable for paired comparisons; select defaults from measured results.
+
+| Step | Implementation | Measurement and completion |
+| --- | --- | --- |
+| C1 | Immutable content-addressed blocks shared by checkpoint manifests. Attention token spans share storage; recurrent and method state preserve exact boundaries. | Same restored bytes/continuations, atomic publication and shared-block lifetime; bytes written and retained across growing/divergent conversations. |
+| C2 | Cache-owned asynchronous reads and preparation-time prefetch with promotion into the existing RAM tier. | Same selected prefix and continuation; restore/TTFT, event-loop delay and concurrent decode against synchronous restore. |
+| C3 | Pluggable retention policy, with LRU baseline and an alternative using reuse frequency, state size and saved processing cost. | Replay identical access traces; compare misses, avoided prefill, eviction/reload traffic and peak residency. |
+| C4 | Segmented transfers and bounded packing scratch, preserving tensor layout contracts. | Copied bytes, temporary memory, write/read time and inference overlap on the same payload. |
+| C5 | Direct paged-attention Metal execution through cache/attention interfaces, including shared row layouts and quantized representations. | Same-B numerical controls, B1/B>1 prefill/decode, gather/padding allocations and complete-request timing. |
+
+Implementation is in `storage/kv-writer.worker.js` (CPU transfers), `kv-store`
+(codec reconstruction), `PromptCache.prefetch` (request preparation),
+`storage/retention-policy` (RAM choices), and the paged attention/layout modules.
+Format 5 manifests reference SHA-256 content blocks. Block boundaries reset per
+leading row so growing token axes preserve earlier head blocks. Blocks are
+committed before their manifest; garbage collection is serialized with writes
+and removes only content no remaining manifest references. Explicit SSD caps
+account unique committed blocks rather than charging every checkpoint for a
+shared copy. Existing format 3/4 files remain readable.
+
+Restore reads use a separate worker queue and page-aligned host allocations.
+After reads finish, the owner thread transfers allocation ownership to MLX
+through a native destructor; no JavaScript callback runs on GPU completion.
+Cache codecs are shared with synchronous restore. Plain KV still uses its
+established capacity-growth operation; the async transport does not change
+cached values or recurrent/checkpoint boundaries.
+
+Use the saved Kanban token histories and seeded synthetic branches to separate
+cache-policy effects from model-generated changes in task length. Existing
+native oracles remain authoritative. Each implementation records paired M4
+measurements and applicable M1 correctness; a slower arm stays explicit rather
+than being promoted on architectural grounds alone. Raw reports stay under
+`reports/`; published measurements belong in benchmarks.md.
+
+The bounded C1–C5 implementation and measurements are complete. Shared blocks
+trade write/read latency for space; segmentation removes packing copies for
+contiguous spans. Asynchronous transport overlaps ongoing decode, while a lone
+restart request still pays restore completion. Cost-size retention loses on the
+saved Kanban boundary replay; direct paged attention has shape-dependent wins
+and regressions. The measured selection keeps whole files and LRU, enables
+asynchronous preparation, and leaves direct attention and block storage explicit.
+[Measurements and reproduction](../reference/benchmarks.md#cache-expansion-c1c5-storage-restore-retention-and-paged-attention)
+include the negative results and limits. Full long-task retention acceptance
+and the paging extensions below remain separate work.
+
+### 5.12 Session checkpoint index
+
+The cache owns a second index: application session ID plus numerical namespace
+maps to its latest immutable checkpoint. Numerical/content identity remains
+independent, so session and anonymous requests can share the same state.
+The request carries a session hint through the existing cache port's lookup,
+preparation and publication calls. Ordinary, speculative and paged methods
+forward the hint; scheduling does not own the index or residency decisions.
+
+A matching resident checkpoint bypasses candidate scans. For an evicted known
+checkpoint, the SSD exact index selects its manifest before asynchronous
+preparation adopts state through the existing codecs. Hashes accelerate SSD
+selection; the selected token sequence still establishes identity. A missing
+or incompatible session checkpoint uses ordinary prefix matching, allowing
+history edits, branches and retries without rejecting the request.
+
+New publications advance the session's reference. Explicit sessions also retain
+ordinary generated checkpoints below the anonymous publication threshold.
+RAM eviction first considers
+entries with no session references, then applies the configured policy to
+referenced entries when needed. This is a preference, not an unlimited memory
+pin. Request-scoped preparation interests retain their existing short-lived
+ownership. Closing a session removes its references without deleting cached
+state; clearing the cache removes the session index. Session metadata is
+process-local, and restart rebuilds associations through ordinary SSD hits.
+
+The current API still accepts full histories. This change removes candidate
+search and supplies retention information; it does not claim delta-only input,
+zero tokenization, or automatic reconstruction from a session label. The Pi
+provider passes the session identity its application already owns.
+The index, native retention and fixed-input M4 HTTP comparisons are complete.
+Lookup and reload work decrease; warmed model throughput is effectively
+unchanged. The full Kanban task now completes with all follow-up session hits
+and a durable final flush; independent app acceptance retains two defects.
+[Measurements](../reference/benchmarks.md#session-checkpoint-index-c6) preserve
+both the benefits and the limits. Existing idle-demotion settings remain in
+effect; session preference changes budget/pressure victim selection.
+
+
+### 5.13 Exact tensor objects in the shared cache
+
+`ObjectCache<Value>` exposes exact-key lookup and ownership transfer separately
+from `PrefixCache` matching. `PromptCache.objects` uses the same resident records,
+byte accounting, retention policy, pressure callback and SSD durability queue.
+An object has an empty conversation history, no target KV layers and a private
+namespace containing its full key. Its versioned tensor attachment owns the
+payload. It never supplies a language-model prefix. No synthetic language token
+IDs or second RAM budget are introduced.
+
+RAM reads lend immutable native views. Evicting the resident entry cannot
+invalidate a request's borrowed tensors. A cold read uses the existing async
+serializer with an empty target-layer schema, outside the execution queue;
+publication joins the same RAM policy. Cache clear invalidates pending restore
+publication. A newer resident publication wins over an older in-flight restore.
+The durability coordinator includes attachment-only objects; RAM eviction waits
+for the shared writer to finish as it does for conversation checkpoints.
+
+The Qwen encoder adapter hashes exact preprocessed pixel bytes, grid dimensions
+and its schema revision. A SHA-256 of the actual loaded encoder tensors supplies weight identity,
+computed once under native preparation. The containing SSD store also supplies
+configuration, tokenizer and native numerical identity. Different image bytes or grids miss;
+repeated media can reuse the tensor across request histories. Prompt rendering,
+video timestamps, mRoPE positions, sampling and scheduling remain independent.
+The producer retains its own result while storage takes a separate native view.
+The object lookup finishes before the prompt builder enters the native queue.
+Preprocessing and video extraction are not cached by this change.
+
+Gemma's encoder-free and SigLIP vision towers and its audio tower use the same
+`EncoderCache` adapter. Each tower hashes its immutable weight names, shapes,
+dtypes, bytes and computation settings once. Image/WAV content hashes and a
+producer revision identify individual objects. Vision retains its existing
+pre-divided features; audio retains raw float32 features, then each prompt casts
+them to its embedding dtype before dividing by the embedding scale. That order
+is unchanged. CPU image preprocessing and audio mel extraction precede the
+native execution lease, as do RAM/SSD lookups. The producer never caches chat
+tokens, bidirectional masks or positions. Custom vision encoders without an
+immutable `cacheIdentity` continue to encode normally.
+
+[vLLM's encoder cache](https://docs.vllm.ai/en/v0.18.0/api/vllm/v1/core/encoder_cache_manager/)
+also identifies individual media items by hash and retains encoder outputs
+across requests. Here the existing cache owns both RAM residency and queued
+SSD persistence on unified memory. The bounded Qwen screen and integrated
+acceptance pass on M4; both-machine native/HTTP checks and fresh-process SSD
+reuse preserve their recorded outputs. [Measurements](../reference/benchmarks.md#qwen-encoder-reuse-through-shared-ram-and-ssd-storage-2026-09-13)
+include the one-time weight-fingerprint cost. Gemma prepared inputs and mixed
+RAM/SSD HTTP reuse pass both-machine identity checks. Its M4 ABBA preserves all
+168 paired responses/usage and improves every measured workload; first-use
+cost and changed image overlap are recorded in the
+[Gemma measurements](../reference/benchmarks.md#gemma-encoder-reuse-through-shared-ram-and-ssd-storage-2026-09-13).
+Prepared media conversation state uses the same cache, as described below.
+
+
+### 5.14 Prepared media prefix identity
+
+The Gemma producer can supply a versioned `prefixIdentity` containing every
+media item's exact input/encoder identity, its causal policy, and the rendered
+token prefix through the last media span. The ordinary cache namespace combines
+this identity with the adapter namespace. Consequently, matching histories
+include identical media and preceding tokens; a changed media set or a change
+from image-only bidirectional attention to mixed causal input cannot collide.
+Appending later text leaves the identity stable.
+
+By default, the shared Gemma binding uses ordinary
+RAM/SSD lookup, generated snapshots, prefetch and persistence. The prepared-input
+interface receives the cache's starting position. An uncached input performs
+its atomic media prefill; a restored prefix resumes text after all media through
+the model's ordinary forward method. Scheduling and sampling are unchanged.
+Qwen supplies the same opaque identity with prepared image/video hashes,
+rendered timestamps and all mRoPE positions through its final media span. Its
+resumed text forward uses the current request positions and preserves the
+recurrent state. Recurrent layers still require an exact retained prefix;
+explicit serial prefix reuse remains unqualified. `MLX_BUN_MEDIA_PREFIX_CACHE=0`
+selects encoder-only reuse for comparisons.
+
+The existing minimum reusable offset still identifies a precision transition.
+If delayed quantization occurs after an atomic media prefill, its cached state
+can serve a later conversation turn but cannot rewind before that conversion.
+A shorter request recomputes normally. Start-zero affine/TurboQuant and bf16
+retain their applicable prefix-trimming behavior. This is cache selection,
+not request admission or a memory refusal.
+
+
+## 6. Optional paged KV
+
+`PagedKVCache` stores full-attention planes in per-request block pools.
+`PagedKvRows` composes those pools through the existing row-state interface.
+Scheduling sees request state and attention ports, not block tables or codecs.
+Flags, defaults and supported combinations are documented in
+[server-config.md](../reference/server-config.md).
+
+### 6.1 Storage and ownership
+
+Pools hold bf16 planes or affine packed/scales/biases planes. New tokens alone
+are quantized and written into their destination blocks. Logical block tables
+map sequence positions to physical slots. Trimming returns unused tail blocks;
+restored prefixes can grow beyond their original request's capacity.
+
+Published attention views, cache borrowers and persistence jobs own independent
+MLX handles. A subsequent functional append cannot overwrite their buffers.
+The backend's `paged-cache-codec` preserves block tables, encoded planes and
+layout metadata through the common RAM/SSD cache. `namespacedCache` separates
+paged numerical settings from ordinary prefixes. Neither persistence nor a
+successful write removes a RAM donor.
+
+The current pools are separate per request. Their snapshots share immutable
+MLX storage, but divergent writes can detach a whole pool. Shared physical
+arenas with per-block copy-on-write remain a further optimization.
+
+### 6.2 Attention
+
+The established attention arm gathers blocks and uses stock SDPA. Its bf16
+storage-layout contract remains bit-exact against plain KVCache.
+
+The optional direct Metal arm reads block tables for short causal queries.
+Each SIMD group processes one query/head/sequence partition, retaining a
+softmax maximum, denominator and partial value vector. A second kernel merges
+those partials. Affine variants unpack KV4/KV8 values in the read operation;
+no full-history dequantization or contiguous KV gather is materialized.
+Shared rows retain their own lengths, so direct attention does not pad KV to
+the longest row. Long prefill and explicit masks use the established SDPA
+shape dispatch through the same attention view.
+
+This changes reduction order. Direct attention is Lab, with numerical tolerance
+checks against the same encoded values; it is not a bit-exact mlx-lm claim.
+Tests cover unequal K/V dimensions, grouped heads, partial blocks, multiple
+queries, row retirement, immutable views and restored growth. Performance
+measurements retain both short-context losses and long-context wins in
+[benchmarks.md](../reference/benchmarks.md).
+
+### 6.3 Remaining extensions
+
+TurboQuant page encoding, paged speculative transactions, media/adapter paging
+and shared physical arenas are outside the implemented bf16/affine page layout.
+Existing non-paged TurboQuant and speculative RAM/SSD state remain supported
+through their own codecs and attention interfaces. Extending a page codec or
+kernel does not require a second scheduler.
 
 ## History
 
@@ -678,3 +974,20 @@ greedy trajectory paged-on vs paged-off).
 - Paged KV v1 (serial, Gemma4, bf16, gather-then-SDPA) landed with
   bit-exact gates; flag rows in cli.md / server-config.md (landing date
   not recorded in the source doc, which still reads "in progress").
+
+## Minimum reusable offsets
+
+Physical trimmability does not prove that an older prefix has a valid
+numerical representation. Caches may expose `minimumReusableOffset` after
+an irreversible transition, such as delayed TurboQuant conversion. Prefix
+selection requires the matched offset to meet every layer's bound. A
+trimmable descendant supersedes an ancestor only when it can serve that
+ancestor's offset; otherwise the earlier donor remains.
+
+Cloning and per-row extraction preserve this state metadata. New SSD
+headers record it, and indexing, durable-prefix checks and restore use the
+same rule as RAM. Legacy TurboQuant headers without a recorded boundary
+are conservatively reusable from their full stored offset only. Other
+legacy cache kinds retain their existing interpretation. This changes
+reuse eligibility, not scheduling, quantization arithmetic or memory
+admission policy.