@qvac/harness 0.2.4 → 0.2.6

package/docs/kvcache-reference.md

@@ -0,0 +1,655 @@
+# KV cache — reference
+
+How the QVAC SDK's prompt cache actually works, and the rules for using it from
+core-v2. Mechanics only.
+
+> **"core-v2" is this stack**: `@qvac/harness` (completion, tools, the model
+> pool — this repo) plus `@qvac/core` (chat, storage, compaction). The name
+> distinguishes it from the retired v1 assistant that still sits in
+> `tetherto/qvac-app`. Paths below are repo-relative when they are harness
+> files, and named by package when they are not. A handful of
+> `qvac-assistant/*` specifiers are the desktop app's own aliases and live in
+> `tetherto/qvac-app`.
+
+The sliding window and dynamic tools this doc once covered are retired on every
+layer of the stack; what remains of them is one section,
+[Retired knobs](#retired-knobs). The remaining work items this implies are the
+"KV cache follow-ups" checklist in `assistant/MIGRATION.md`, which is still in
+`tetherto/qvac-app`.
+
+## Versions
+
+Behaviour here is version-specific and has changed more than once. Check first:
+
+```bash
+grep '"@qvac/sdk"' package.json
+grep '"version"' node_modules/@qvac/inference/package.json node_modules/@qvac/llm-llamacpp/package.json
+```
+
+core-v2 ships **`@qvac/sdk` 0.19.0** → **`@qvac/inference` 0.19.0** →
+**`@qvac/llm-llamacpp` 0.49.1** (`@qvac/harness` declares `^0.19.0` for
+both packages; inference pulls `^0.49.1`). If the lockfile resolves more than
+one copy, read the one `@qvac/harness` gets, not whichever is hoisted.
+
+Since 0.19 `@qvac/sdk` is a thin re-export over `@qvac/inference`, which owns
+the engine, the plugins and the cache session. The harness runs in a Bare
+sidecar with the engine in-process, so its `#sdk` import seam resolves to
+`@qvac/inference` there (`@qvac/sdk` only under Node) — the cache code that
+executes is `@qvac/inference/dist/plugins/builtin/llamacpp-completion/ops/`.
+
+What 0.19 changed for everything below ([qvac#4163](https://github.com/tetherto/qvac/pull/4163)):
+`n_discarded` is gone from the `loadModel` schema and rejected at validation;
+the `toolBlockEvictable` gate that re-sent the tool block under sliding is
+deleted; a thrown `ContextOverflowError` no longer rolls the cache back; and the
+error carries `requiredTokens`/`cachedTokens`/`ctxSize`, parsed from every
+wording the addon emits. Addon 0.48 removed sliding outright (and
+`contextSlides`), stops a generation that fills the window with
+`stopReason=contextOverflow`, and rewrites reasoning compaction as rewind and
+re-decode. Earlier revisions of this doc described 0.18.2 (rollback on overflow,
+sliding still possible) and 0.17.x (tool block in the prime); both are history.
+Read source in this order:
+
+1. `node_modules/@qvac/inference/dist/**` — compiled but readable, and exactly
+   what runs. `@qvac/sdk/dist` holds only the re-export and the RPC client.
+2. A checkout of the `qvac` monorepo (`packages/inference`,
+   `packages/llm-llamacpp`) — **fetch first and read `tether/main`.** A stale
+   checkout produced two wrong conclusions in this doc's history. The addon C++
+   is `packages/llm-llamacpp/addon/src/model-interface/`.
+3. Never a standalone `qvac-lib-infer-llamacpp-llm` checkout — the addon source
+   of record is the monorepo; standalone clones drift stale.
+
+The live C++ class is **`MtmdLlmContext`**, not `TextLlmContext`: chat models load
+with a projection model, so the multimodal context is what executes. The two are
+parallel implementations.
+
+## The three layers
+
+Three state sets, keyed differently, never reconciled with each other:
+
+| Layer   | Owns                                               | Keyed by        |
+| ------- | -------------------------------------------------- | --------------- |
+| core-v2 | the message **list**                               | chat            |
+| SDK     | a message **count** (`savedCount`)                 | cache file path |
+| addon   | token **positions** (`nPast`) + the on-disk `.bin` | cache key       |
+
+`savedCount` means _"already sent"_, never _"still resident"_: nothing below the
+SDK reports what it evicted. That is what made both retired knobs fatal, and it
+is the test any new eviction idea has to pass.
+
+## The knobs
+
+| Knob                                                                          | Where          | core-v2 call site                                                                                                                                             |
+| ----------------------------------------------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `history`                                                                     | `completion()` | full history, no pre-trim                                                                                                                                     |
+| `kvCache: 'chat-<id>'` (a repair generation nests under it: `chat-<id>/r<n>`) | `completion()` | [assistant.ts](https://github.com/tetherto/qvac-core/blob/main/lib/assistant.ts) `_execute` → [completion.ts](../lib/capabilities/completion.ts) `completion` |
+
+Nothing else. The model loads with `tools: true` and the cache-relevant config
+(`ctx_size`, `cache-type-k/v`); every other cache behaviour is the SDK's own.
+
+## Retired knobs
+
+core-v2 stopped passing both on 2026-08-20
+([2026-08-20-kvcache-static-placement](./decisions/2026-08-20-kvcache-static-placement.md)),
+and upstream has since deleted them. They stay here as bug shapes, not options.
+
+- **`n_discarded` (sliding window).** Was `floor(ctxSize / 10)` at `loadModel()`.
+  SDK 0.19 dropped it from the config schema, so `loadModel` rejects the key;
+  addon 0.48 removed sliding entirely
+  ([qvac#3938](https://github.com/tetherto/qvac/pull/3938)). Two present-day
+  defects retired it before upstream did. A slide was **invisible to the SDK**:
+  the session recorded a turn's full message count on success and next sent
+  `history.slice(savedCount)`, so a span the addon discarded in C++ was gone from
+  the live cache and the saved `.bin` while the app still counted it as sent —
+  the model silently lost the middle of the chat, with no error and no TTFT
+  tell. And while sliding was possible the SDK set
+  `toolBlockEvictable = n_discarded > 0`, which made the whole tool block travel
+  on every turn (~331 tokens for a 4-tool agent) instead of being cached once.
+  Prefill also slid exactly once and gave up, so long prompts overflowed anyway.
+  A full window is now a normal end state: overflow at prefill, `stopReason:
+length` mid-generation, recovered by the harness ladder and by the fold.
+- **`toolsMode: 'dynamic'` (dynamic tools).** SDK 0.18 deleted the key
+  ([qvac#3380](https://github.com/tetherto/qvac/pull/3380); the parser rejects
+  it, so passing it fails every load) and the addon dropped `tools_compact`
+  after 0.39.4 ([qvac#3373](https://github.com/tetherto/qvac/pull/3373)). It
+  was a silent no-op on every model we ship: the addon gated `tools_compact` on
+  `general.architecture == "qwen3"` exactly, our catalog is `qwen35`/`qwen36`,
+  so it logged `tools_compact is not supported … ignoring` and ran without
+  anchoring or trimming — while the SDK still sliced as-if-dynamic from the
+  requested flag, sending `[assistant, user]` plus the tool block and ignoring
+  `savedCount`. The previous answer was duplicated into the KV every turn, the
+  block accumulated, and on a cold cache only the last exchange was sent, so the
+  model confabulated instead of overflowing. Where the addon half did run it
+  trimmed the tool block, every tool result and the assistant's answer after a
+  turn, which is what the SDK's re-send assumed. Static placement is the only
+  mode now: `history.slice(savedCount)`, full history on a cold cache at TTFT
+  cost only — and that is what makes the overflow ladder effective, since
+  dropping oldest turns changes what the slice sends.
+
+Each removal changed `modelConfig`, so each re-keyed `modelId` and cold-primed
+every chat once. Static tool calling is unaffected by either removal.
+
+## Cache identity
+
+```
+{kvCacheDir}/{cacheKey}/{modelId}/{configHash}.bin        # ~/.qvac/kv-cache/...
+```
+
+- `cacheKey` — ours: `chat-<chatId>`.
+- `modelId` — see below.
+- `configHash` — `sha256(canonical({ systemPrompt, tools }))[:16]`, where
+  `tools` is the **complete tool definitions** the call carries (names,
+  descriptions, parameter schemas; object keys sorted, array order kept), or
+  `[]` when the call has none. **A changing system prompt changes the file**,
+  and so does any edit to any tool — one cold prime per change. An `@skill`
+  mention changes it for that turn only.
+
+Two `.bin` files under one chat key means `configHash` moved.
+
+### What `modelId` actually is
+
+The SDK's `modelId` is **not** a model name — it is identity-by-config, computed
+at `loadModel` (`@qvac/inference`, `dist/handlers/load-model/handler.js`,
+`canonicalConfigString` + `generateShortHash`):
+
+```
+modelId = sha256("{modelType}:{modelSrc}:{canonicalConfigString(modelConfig)}").slice(0, 16)
+```
+
+`canonicalConfigString` recursively sorts keys, so key _order_ never matters —
+but **every key-value pair in `modelConfig` does**. Two loads of the same weights
+with any differing config field are two different `modelId`s: separate pool
+entries in memory, separate `{modelId}/` cache directories on disk.
+
+Properties that follow:
+
+- **Deterministic.** Same type + src + config → same id, across restarts. That is
+  what lets `chat-<id>/{modelId}/` cache files survive an app restart, and it is
+  why `loadModel` on an already-loaded id is a no-op ("already loaded").
+- **The SDK cannot re-configure a loaded model.** Changing ctx or KV quant means
+  loading a _different_ `modelId`; core-v2's model pool keys on exactly this.
+
+### What happens when you change …
+
+| You change                                                                                               | `modelId`   | `configHash` | Cache effect                                                                             |
+| -------------------------------------------------------------------------------------------------------- | ----------- | ------------ | ---------------------------------------------------------------------------------------- |
+| system prompt (in history)                                                                               | same        | **changes**  | new `.bin` in the same dir; cold prime; old file orphaned                                |
+| tool set — add, drop, or edit any tool's name, description or parameters                                 | same        | **changes**  | new `.bin`; cold prime                                                                   |
+| **any** `modelConfig` field (ctx_size, `cache-type-k/v`, device, gpu_layers, stop sequences, verbosity…) | **changes** | same         | new `{modelId}/` directory — **every chat on this model cold-primes**; old dirs orphaned |
+| model weights (`modelSrc`)                                                                               | **changes** | —            | same as above                                                                            |
+| chat                                                                                                     | —           | —            | different `{cacheKey}/` directory (by design)                                            |
+
+Two traps this table encodes:
+
+- **Observing the system changes it.** core-v2 puts `verbosity: llmVerbosity()`
+  into `modelConfig` ([model-pool.ts:330](../lib/models/model-pool.ts)),
+  read from `QVAC_LLM_VERBOSITY`. Turning addon debug logging on flips a config
+  field → new `modelId` → every chat cold-primes under a fresh directory. Debug
+  the cache with harness logs (`QVAC_HARNESS_LOG`) or `xxd` on the `.bin`, not by
+  raising addon verbosity, or the cold behaviour you observe is one you caused.
+  Any env-dependent config field (`openclCacheDir`, `main-gpu`) has the same
+  property.
+- **Orphaned files accumulate — and only the app can delete them.** Nothing
+  cleans custom-key cache directories (the retention sweep is auto-key only), so
+  every config or system-prompt change strands the previous `.bin`. The SDK
+  cannot safely auto-delete: identity-by-config cuts both ways, so a "stale"
+  file **revives** the moment the same `(key, modelId, configHash)` recurs —
+  `beginTurn` probes the disk path and reuses it (flip ctx back, revert the
+  prompt, unset `QVAC_LLM_VERBOSITY`, and the old cache is live again). Whether
+  that rare revival is worth the disk is a product call, which is why retention
+  for custom keys is deliberately the caller's job (`deleteCache({ kvCacheKey,
+modelId? })`). core-v2 implements **delete on re-key** (2026-08-20,
+  `@qvac/core`'s `lib/kv-cache-reclaim.ts`): every device with a harness watches
+  agents and deletes the whole `chat-<id>` key for an agent's chats when a
+  re-keying field changes, deletes a chat's key when the chat disappears, and
+  deletes it again after a fold (below) — four moments in all, each of which
+  reclaims any repair generations with it, because they are children of that key —
+  the cold prime is already inevitable, so it costs nothing beyond the
+  revert-reuse case. Residual: changes replicated while the assistant was
+  closed reclaim nothing (no diff to observe). Scale that made this
+  non-optional: observed 21–26 MB per file at ctx 2048, and the state file
+  scales with the _allocated_ context — at the 51,200 desktop default,
+  plausibly hundreds of MB per chat (unmeasured).
+- **Apps can delete caches too, device-locally.** The engine API exposes
+  `clearKvCache()` (every cache on this device) and `deleteChatKvCache({ id })`
+  (2026-09-10; the chat's key with its repair children). Both act only on the
+  device whose Core is called — nothing is replicated, and the chat's warm
+  cache may sit on the provider device. The desktop app's
+  `qvac-assistant/kv-cache` wraps both behind the `supportsKvCacheDeletion` gate.
+
+One more footgun: the SDK also accepts `system_prompt` **inside `modelConfig`**
+as a fallback for the prime when the history carries no system message. Putting
+it there would drag the system prompt into `modelId` — a prompt edit would then
+cold-prime _every chat_ instead of one. core-v2 correctly keeps it in history;
+keep it that way.
+
+## Building the history
+
+The `history` argument is an ordered array of `{ role, content }` (the field is
+`content`, not `message`; user turns may also carry `attachments`). Canonical
+shape, mid tool-chain:
+
+```jsonc
+[
+  { "role": "system", "content": "<harness buildSystemPrompt output>" },
+  { "role": "user", "content": "question" },
+  { "role": "assistant", "content": "answer" },
+  { "role": "user", "content": "next question" },
+  { "role": "assistant", "content": "<tool_call>…web_search…</tool_call>" },
+  { "role": "tool", "content": "<tool result>" },
+  { "role": "assistant", "content": "answer grounded in the result" }
+]
+```
+
+Structural rules:
+
+1. **Exactly one `system` message, first.** The SDK primes from it and filters it
+   out of cold sends; a second one mid-history is never valid.
+2. **A `tool` message directly follows the assistant message that called it.**
+   The chat template assumes it.
+3. **The last message is `user`** (fresh turn) **or `tool`** (chain
+   continuation).
+4. **Tool definitions are never history.** They ride the separate `tools`
+   argument.
+5. **Pass the whole history, every turn.** The SDK computes the delta. Never
+   pre-trim — the only sanctioned shrinking is the overflow ladder, which edits
+   the _outgoing candidate_, not the stored history.
+
+Note the two shapes core-v2 actually has: **within a run**, the round loop
+appends `assistant(tool-call markup) → tool(result)` pairs per round. **Across
+turns**, history is rebuilt from storage and collapses each run to
+`[user, final assistant answer]` — thinking chunks never join assistant content,
+and tool results are not replayed (the one exception is a `skill` load, replayed
+so the next turn keeps the granted tools). `sanitizeTurn` then strips control
+tokens from every turn and residual tool-call markup from assistant turns.
+
+### Editing history: what resets the cache, and what silently doesn't
+
+**Editing history content never resets the cache.** For our custom `chat-<id>`
+key the cache identity is `(cacheKey, modelId, configHash)` and `configHash`
+hashes only the system prompt and the tool definitions — history is not part
+of it. The only edits that force a fresh cache file are a **system prompt
+change**, a tool-set change, or any model-load-config change
+(new `modelId`). (Auto-key mode hashes prior history, so there any edit is a
+cache miss — core-v2 does not use it.)
+
+That is the trap, not a safety: warm turns only ever send the tail, so an edit
+to anything **before** the tail — removing a turn, stripping a thinking block or
+tool output from an old assistant message, regenerating an earlier answer —
+
+- is **silently ignored while the cache is warm**: the model keeps conditioning
+  on the original text in the KV — which is why core-v2 deletes the chat cache
+  before executing a superseding (retry/regenerate) run (2026-08-20,
+  `@qvac/core`'s `lib/kv-cache-reclaim.ts`); any new divergence writer must do the
+  same;
+- **suddenly takes effect on the next cold prime** (restart, cancel-wipe, other
+  device), so the model's effective context changes across a restart;
+- and **additionally corrupts the slice boundary**:
+  `savedCount` is a message _count_, so inserting or removing anything before
+  the tail shifts which messages the count refers to. The SDK detects only the
+  empty-slice case.
+
+Edits that are safe, because they touch only what is about to be sent:
+
+- the **last user message** — `annotateLastUserTime` does exactly this, and its
+  own comment states the rule: _"earlier turns must stay byte-identical so the
+  KV-cache prefix reuses"_;
+- **tool results in the current chain** — they are the outgoing slice; the
+  overflow ladder's compaction relies on this.
+
+**Compaction is the second divergence writer, and it is implemented.** A fold
+rewrites history before the tail on purpose: `historyFromChunks` replaces every
+turn through the boundary with one summary turn. So it does exactly what this
+section warns about, and pays for it the same way — `compactChat` deletes the
+chat key once the `compaction` chunk lands. The chunk is replicated, so a fold
+written on another device reaches this one as data with no local delete behind
+it; `_reconcileCompaction` closes that at the start of the next run, once per
+device per fold. The folded chunks stay in the store and keep rendering: only the
+prompt shrinks.
+
+The general rule: **normalize deterministically, from turn 1, identically every
+turn.** A transform applied late or conditionally (stripping thinking only
+sometimes, truncating tool outputs after the fact) creates app-vs-KV drift that
+is invisible warm and surfaces cold. core-v2's storage shape already conforms —
+thinking and tool traffic never enter cross-turn history, so there is nothing to
+strip later. If a true edit feature (edit/delete a past message) needs the model
+to actually honour the change, it must invalidate the chat cache the way retry
+does (`deleteKvCache('chat-<id>')` on the executing device before the next
+completion) and pay the cold-start cost — it is not a content tweak.
+
+## What actually gets sent
+
+Every turn the SDK either finds the cache or **primes** it — a separate
+`model.run(..., { prefill: true })` carrying the **system prompt alone** (since
+0.17.1, qvac `598f1551b`, QVAC-23251; 0.17.0 primed the tool block too). Tools
+stay out of the prime because a message list with no user turn is not a
+renderable conversation for every chat template — Qwen3.5 raises
+`No user query found in messages.` — and a template failure would degrade the
+whole prefix rather than just the block.
+
+Then it sends a _slice_, not the history (`prepareMessagesForCache`,
+`decideCachedHistorySlice`):
+
+| Condition                                                         | Sent                                                                          |
+| ----------------------------------------------------------------- | ----------------------------------------------------------------------------- |
+| `savedCount` in `(0, history.length]`, slice non-empty            | `history.slice(savedCount)`                                                   |
+| otherwise — cold cache, or a stale count left by a cancelled turn | full history minus system; the stale count is dropped (`dropStaleSavedCount`) |
+
+The **tool block** rides in front of the slice, never inside history. It is
+written into the cache by the first turn that actually renders it — the SDK
+requires a user message in that payload before believing it landed, because a
+Qwen template anchors the block on the last user query and the addon answers a
+missing one by re-rendering the turn with tools stripped — and skipped on every
+later turn once `toolBlockCached` is recorded at commit. A stale-count fallback
+re-sends it with the whole conversation. The block is never trimmed back out of
+the cache; that is why `configHash` includes the tool definitions, so a changed
+set lands on a fresh file rather than a warm prefix holding the old block.
+
+Until 0.19 a second gate sat in front of the skip: `toolBlockEvictable =
+n_discarded > 0`, because the sliding window's discard region opened exactly
+where the block sits. While core-v2 passed `n_discarded` (until 2026-08-20) the
+block travelled on every turn, ~331 tokens for a 4-tool agent; qvac#4163 deleted
+the gate with the knob, and a config that still carried the key would not force
+a resend.
+
+`tools: []` (or none) means no block and a different `configHash` — a cold
+prime, not a strategy switch.
+
+## Addon semantics
+
+Nothing evicts. Since addon 0.48 the KV holds exactly what was decoded into it
+(minus compacted reasoning, below); a full window is terminal for the turn and
+the core-v2 **compact** flow (see _Compaction_) is what replaces eviction.
+
+**A prompt that does not fit is refused at prefill.** Two guards, both `>=`:
+the slice alone against `ctx_size`, and the cached total plus the slice against
+`ctx_size`. Either raises `ContextOverflow` with the cache untouched — there is
+no discard step in between. The two guards, and the two context classes, format
+their numbers differently; `parseContextOverflowMessage` normalizes every form
+into the error's sizes (see _Reading the numbers_).
+
+**A thrown overflow keeps the cache** (0.19, qvac#4163). The completion handler
+marks a prefill overflow and every addon pre-mutation refusal (scheduler cap,
+bad `json_schema`, missing media) as non-destructive: the deferred unwind runs
+`releaseTurn`, which leaves the committed `.bin` and its `savedCount` alone, and
+only rolls back a cache this same turn had freshly primed. Before 0.19 the unwind
+was always `rollback`, so a legitimate tool-result overflow unlinked the chat's
+whole file — the defect this doc used to carry as its top upstream report.
+
+**A generation that reaches the ceiling stops, with what it produced** (addon
+0.48: `stopReason=contextOverflow`, where earlier builds reported the
+per-sequence limit). The SDK maps it to the public `stopReason: 'length'` and
+does **not** commit the turn — `shouldCommitCachedTurn` refuses a
+context-boundary stop the way it refuses a cancel or a zero-token reply — so the
+deferred `rollback` unlinks the `.bin` and the next turn cold-primes from
+history, the truncated answer included. Nothing throws, which is why
+compaction's end-of-run trigger exists at all.
+
+**Reasoning is compacted out of the KV by the addon.**
+`generationParams.remove_thinking_from_context` defaults to `true` (addon
+0.33.0); a closed thinking block is removed after the turn — since 0.48 by
+rewinding to the block boundary and re-decoding what follows, with no `seq_add`
+shifting. `cacheTokens` after a turn therefore excludes thinking, and a
+compaction failure surfaces as an error rather than a cache that still holds the
+trace. This is why core-v2 never has to strip thinking from history itself.
+
+**Cache file.** Four ints of metadata, written on every committed turn:
+`nPast`, `cacheTokens`, and two retired slots that carried the sliding window's
+first-message counters. 0.48+ fills those by mirroring the live values —
+`{nPast, nPast, cacheTokens, cacheTokens}` — so a still-sliding build that reads
+the file refuses to slide instead of evicting from position 0.
+
+**Reset happens only on key CHANGE.** Same key → no reload, no reset. Switching
+keys saves the old cache first. Since addon 0.35.1 a same-key turn does check
+whether the `.bin` still exists and discards its in-memory state if not.
+
+**Disk growth is unbounded for our keys.** SDK 0.17.0 added a retention sweep
+(size + idle budgets, mobile/desktop split, plus empty-directory pruning), but it
+only considers entries with an `.auto-cache-<16hex>` marker — i.e. **auto keys
+only**. A `chat-<chatId>` cache is never swept or size-capped, and every turn
+rewrites the whole `.bin`. The upside: the sweep cannot evict a chat cache, so it
+is not a cold-prime trigger.
+
+## Compaction
+
+The app-side answer to a full window, and the only sanctioned one:
+`@qvac/core`'s `lib/compaction.ts` plus `compactChat` in its `lib/assistant.ts`.
+It folds a chat's older turns into a single summary turn, persisted as a
+replicated `compaction` chunk carrying `summary` and `throughId`, so every device
+rebuilds the same prompt from chat data rather than from cache state. Decision
+record: [2026-08-21-compaction-is-chat-data](https://github.com/tetherto/qvac-app/blob/main/docs/decisions/2026-08-21-compaction-is-chat-data.md),
+still in `tetherto/qvac-app`.
+
+**What it evicts is the prompt, not the KV.** The folded chunks stay in the store
+and keep rendering; the fold then deletes the chat key so the next completion
+primes from the compacted history.
+
+**Only the last turn stays verbatim** (`COMPACTION_KEEP_TURNS`, 1), and it has to:
+fold the last user message away and the model has nothing to answer. Keeping two
+was measured as the reason a fold could free nothing — the second kept turn drags
+its assistant answer along, which is usually the largest thing in the chat. At ctx
+4096 that left a fold at ~3,890 of 4,096 and the next answer truncated at 90
+generated tokens.
+
+**A fold that would grow the prompt is refused** (`summary-not-smaller`). The
+summary turn carries a 132-character prefix, so summarizing a couple of one-word
+turns is always a loss; `foldReducesPrompt` compares it against the turns it
+replaces before anything is appended or any cache is dropped. A comparison on
+characters, never a token budget.
+
+**A summarizer that overflows folds a narrower range.** The fold's own prompt is
+the range it folds, so a history that no longer fits overflows the fold meant to
+rescue it — and the ladder cannot trim it, because it is one user turn.
+`compactChat` halves the range and retries instead of giving up, which keeps a
+chat opened on a smaller window (a desktop chat at 51k on a phone at 12k) from
+failing every message. A boundary no later than the previous fold's is refused
+(`already-compacted`), so narrowing can never un-collapse an earlier fold.
+
+**The boundary never splits a kept turn from its uploads.** An upload chunk is
+appended before the user message it belongs to, so `compactionBoundary` walks
+back over the uploads bound to the first kept message before it picks the point.
+
+**A fold announces only once it is going to summarize.** The `compacting` phase is
+written from inside `compactChat`, past the cheap refusals, so a run that gets
+`already-compacted` never flashes the label.
+
+**A fold summarizes in the run's own window.** The harness folds `ctxSize` into
+the model key, so the summarizer gets the agent row with the app-level context
+size applied — otherwise a fold evicts the resident model, reloads it at the
+device default, and the run reloads it again.
+
+**A fold survives a superseding run.** The reactive path folds and re-asks the
+question with `parentId`, which opens a dead range over the sequence numbers
+between the two run-requests — and that is exactly where the fold writes its
+chunk. A compaction chunk is therefore exempt from the branch filters in
+`_execute`: dropping it would un-fold the chat and the retry would re-send
+everything.
+
+**Floor.** `COMPACTION_MIN_CTX_SIZE` is 4,096, exported renderer-safe from the app's
+`qvac-assistant/constants`. Below it the summary cannot pay for the ~420-token
+prefix (system prompt + tool block, measured on a 4-tool agent) it has to share
+the window with, so those chats stay on the overflow ladder. The context picker
+still offers 1k and 2k, so a client must read the constant before telling a user
+their conversation will be summarized.
+
+**Triggers**, all on the SDK's own counts, never an estimate:
+
+| When                | Reads                                                                                             |
+| ------------------- | ------------------------------------------------------------------------------------------------- |
+| before a run        | the previous round's `cacheTokens` vs its `ctxSize`                                               |
+| at the end of a run | this run's own numbers, because a turn can fill the window by itself                              |
+| after an overflow   | the harness ladder is exhausted; the fold lands and the question is re-asked as a superseding run |
+
+`ctxSize` is per-device — mobile loads every model at 12k, desktop at its
+per-model default — so only a turn _this_ device ran is a reading for it. A chat
+that moved here from a phone carries a window this host does not have.
+
+**Why the margin is 95%, and 90% at or below 4,096.** Holding back a fixed share
+costs most where the window is largest, so the margin stays thin. What it has to
+buy is a turn that lands _between_ the trigger and the ceiling — that is the turn
+whose successor folds before generating, visibly and with a real post-fold
+reading. At 4,096 a 5% margin is 204 tokens, narrower than an ordinary turn:
+measured on a 4k chat, turns arrived at 58, 107, 147, 325, 416 and 1,267 tokens,
+so the chat kept crossing the whole band inside one turn, truncating that answer
+against the ceiling (`stopReason: length`) and leaving only a silent fold behind
+it. A tenth of 4,096 is 409 tokens, which most of those turns fit inside.
+
+**A fold leaves the chat with no occupancy reading for one turn**, and that is
+deliberate. The number the folding run measured describes history the fold has
+just replaced, so publishing it renders the chat as still full — that is what
+locked the composer before this landed — and `_lastRunMetrics` would read it back
+and fold again on the next run, since a new user turn moves the boundary and
+`already-compacted` no longer refuses. The post-fold number does not exist until
+the next prime: occupancy is whatever the SDK reports after it builds the prompt,
+and deriving it here would be the chars-per-token estimate the rules forbid. So
+`withoutOccupancy` strips `cacheTokens`/`promptTokens` from the terminal status of
+a run that folded, and the next completed turn restores a real reading. The client
+currently renders that absence as 0.
+
+**A turn that fills the window ends cleanly.** The SDK reports `stopReason:
+length` and the run completes — nothing throws — which is why the end-of-run
+check exists at all: the reactive trigger never sees that turn. The SDK does not
+commit that turn either (its `.bin` is unlinked, see _Addon semantics_), so the
+fold's key delete costs nothing extra there: the next completion was going to
+cold-prime regardless.
+
+**Every fold announces itself.** The run status carries a `compacting` phase,
+app-runtime passes it through to the live status, and desktop and mobile name it
+"Compacting conversation...". That took one narrowing on the client: a
+non-terminal run-status normally means the run restarted, so a client discards
+whatever streamed — which is why announcing the end-of-run fold used to erase the
+answer on screen. `chunk-to-turn` now exempts a status carrying that phase, since
+a fold is not a new attempt. What stays conditional is _clearing_ the phase: the
+clear rewrites the chunk into a plain non-terminal status, so it only runs for the
+before-run fold, whose run carries on. Where a terminal status follows — the end
+of a turn, a failed overflow — that status closes the phase.
+
+**So the fold is shown from its chunk, not from the phase.** The `compaction`
+chunk is already replicated and already parsed, so no new event was needed — only
+rendering. `chunk-to-turn` keeps it as a turn of its own kind (that is what puts
+the marker in the right place), the message list excludes it, and
+`anchorCompactionMarkers` puts it on `thread.compactionMarkers` in the reducer,
+anchored to the user message it precedes — never to a reply, which attempt
+grouping collapses to its selected attempt — so neither client re-derives the
+position and a retry cannot orphan the row. Desktop and mobile render it as a quiet centered row —
+"Compacted session" — in the same shape as the day divider. Unlike the phase, it
+survives: it is there when the user scrolls back.
+
+## Reading the numbers
+
+| Field                                 | Means                                                                                                                                    |
+| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
+| `stats.promptTokens`                  | tokens in **this turn's slice**                                                                                                          |
+| `stats.generatedTokens`               | tokens produced                                                                                                                          |
+| `stats.cacheTokens`                   | **physical KV-cell usage**                                                                                                               |
+| `ContextOverflowError.requiredTokens` | what the failing guard needed, in `ctxSize` units — the slice alone, or cached + slice; can **equal** `ctxSize`, the guards fire on `>=` |
+| `ContextOverflowError.cachedTokens`   | the resident total on the warm-cache guard; unset on the slice-alone guard                                                               |
+| `ContextOverflowError.promptTokens`   | the slice, **only** when the guard counted tokens (TextLlm); unset when it reported KV cells/positions (Mtmd)                            |
+| `ContextOverflowError.ctxSize`        | the window — or the per-sequence cap, on a scheduler refusal                                                                             |
+
+All four are parsed out of the addon's message (`parseContextOverflowMessage`,
+0.19). Since qvac#4163 every wording the addon emits parses; before that Mtmd's
+forms matched nothing and `promptTokens` alternately meant the slice or a cached
+total. Mtmd is the class production executes — chat models load with a
+projection model — so expect `requiredTokens` and `ctxSize`, `cachedTokens` on a
+warm overflow, and **no `promptTokens`**. Where a guard reports both positions
+and KV cells the larger is taken. core-v2 reads none of them:
+`isContextOverflowError` matches the error's `name` (`CONTEXT_OVERFLOW`) and the
+ladder retries on shape alone.
+
+Two traps:
+
+- `cacheTokens` is a _different quantity_ from `nPast` (logical positional span).
+  They are separate metadata fields because M-RoPE media can occupy more cells
+  than its positional span. **Do not do arithmetic mixing them** — in particular,
+  `cacheTokens - promptTokens` is not the primed prefix size.
+- `promptTokens` on the error is usually absent and, when present, counts one
+  guard's view of the slice; on the stats it is always the slice the addon
+  processed. Do not compare the two.
+
+## Rules for using it
+
+1. **Pass the full history.** The SDK computes the delta.
+2. **No client-side token math.** No chars-per-token estimates, no reserved
+   response margin, no pre-emptive trimming, no computed budgets. core-v2 has none
+   and must keep none.
+3. **React, never predict.** The only supported "will this fit" answer is to send
+   it and handle the typed `ContextOverflowError`. Compaction's proactive trigger is
+   not an exception: it acts on the SDK's own `cacheTokens` from a round that already
+   completed, never on an estimate of the outgoing prompt.
+4. **Use the SDK's real counts** for diagnosis, never estimates.
+5. **Keep the system prompt stable** for a chat — every change costs a cold prime.
+6. **Keep the tool set stable** — `tools` joins `configHash`, so adding,
+   dropping, or editing a tool costs a cold prime.
+7. **Only two legitimate overflows** in steady state: the current prompt alone
+   exceeds ctx, or a tool result alone exceeds ctx. A third legitimate case: a
+   device that never ran this chat receives the whole history (the cache is
+   device-local). A fourth: a window below the 4,096 fold floor, which never
+   compacts and degrades through the ladder instead. Anything else is a bug.
+
+## Debugging
+
+Enable harness logs — `QVAC_HARNESS_LOG=1`, or `--log-verbose`. What ships today:
+
+```
+round start   {"round":0,"turns":16,"tools":1}
+round stats   {"round":0,"wallMs":2188,"timeToFirstToken":450,"cacheTokens":835,
+               "promptTokens":458,"generatedTokens":84,"backendDevice":"gpu"}
+final         {"stopReason":...,"toolCalls":0,"truncated":false}
+add_text:     <|im_start|>system ...   ← addon output, NOT a harness line (below)
+```
+
+- `round start` — `turns` is the history length, `tools` the count passed as the
+  separate `tools` argument. Tool definitions never appear in history.
+- `round stats` carries the SDK's real token counts. `cacheTokens` climbing turn
+  over turn = warm reuse; dropping back toward the system-prompt size = a re-prime.
+- **Two `add_text` blocks = a cold prime** (system prompt, then the slice); one
+  block = warm. But `add_text` is **addon-level** output (it comes from below
+  the addon, in llama.cpp), visible only at elevated `QVAC_LLM_VERBOSITY` — the
+  addon maps `0→ERROR, 1→WARNING, 2→INFO, 3→DEBUG` (`LoggingMacros.cpp`), so
+  use `3` to be sure — and that env feeds `modelConfig.verbosity`, so enabling
+  it **re-keys `modelId` and cold-primes every chat** (the trap above). Use it only when verbosity is already on;
+  otherwise the safe cold-prime signals are `round stats` (`cacheTokens`
+  dropping back toward prompt size) and `xxd` on the `.bin`.
+- A failed turn logs `completion failed {...}`
+  ([index.ts:1281](../index.ts)) and **no**
+  `round stats`, because
+  the stats only exist once the SDK settles the run. An absent `cacheTokens` line
+  means the turn failed, not that the cache was bypassed.
+- No addon line reports eviction any more: `discarded N tokens after the first
+message`, `[ContextSlider] context is full but cannot discard tokens` and
+  `runtimeStats().contextSlides` all left with sliding in addon 0.48. A log
+  that shows them is an old build. `normalizeCompletionStats` forwards
+  TTFT/TPS/ppTPS/cacheTokens/promptTokens/generatedTokens/avgConcurrentSeq/
+  backendDevice, nothing about the KV's shape.
+
+Nothing currently reports which context-recovery rung ran. If you need that,
+log per candidate from
+`runRecoveredRoundWith` in `capabilities/completion.ts`: the rung index,
+`droppedTurns` (stage 1), a compaction counter (stage 2), and a roles-only shape
+of the candidate history plus the tool count. Roles only — never message content.
+
+**Read the cache file directly** — the header carries the metadata, no
+instrumentation needed:
+
+```bash
+xxd -l 32 ~/.qvac/kv-cache/chat-<chatId>/*/*.bin
+# after an accepted diagram repair the live file is one level deeper:
+xxd -l 32 ~/.qvac/kv-cache/chat-<chatId>/r*/*/*.bin
+```
+
+```
+7173 6767  0200 0000  0400 0000  7c00 0000  7c00 0000 ...
+^ 'ggsq'   ^ ver 2    ^ 4 fields ^ nPast    ^ nPast again (retired slot)
+```
+
+Little-endian `uint32` after the count: `nPast`, `nPast`, `cacheTokens`,
+`cacheTokens` on addon 0.48+ — slots 1 and 3 are the retired first-message
+counters, mirrored from the live values (a file from an older build carries
+`firstMsgTokens` / `firstMsgCacheTokens` there instead). `nPast` climbing turn
+over turn is warm reuse; back to prompt size is a re-prime. An empty
+`chat-<id>/<modelId>/` directory means a rollback unlinked the file — after a
+cancelled turn, a zero-token reply, or a generation that hit the ceiling.
+
+To A/B the cache, drop `cacheKey` where `_execute` builds the completion
+request in [assistant.ts](https://github.com/tetherto/qvac-core/blob/main/lib/assistant.ts).
+There is no env kill-switch in core-v2.