pi-context-guard-jev 0.1.0 → 0.2.0

package/README.md

@@ -7,9 +7,11 @@ This Pi extension keeps the lines that mattered.**
 
 When a run finishes, the extension asks the Jev classifier (by TypeSafe) which parts of each large tool
 output the work still needs: a broad `rg`, a full-file `read`, a failing test log. It then uses Pi's
-append-only `context_edit` entries to replace each output with only those parts, copied verbatim. Later
-turns carry the answer and its evidence instead of everything that was searched. The raw output stays in
-the session file, and the model can fetch it again with the `recall` tool.
+append-only `context_edit` entries to replace each output with only those parts, copied verbatim. Small
+outputs the work no longer needs become one-line notes, and earlier exchanges that are unrelated to the
+current work are omitted down to a one-line note. Later turns carry the answer and its evidence instead of
+everything that was searched. The raw entries stay in the session file, and the model can fetch them again
+with the `recall` tool.
 
 ```bash
 pi install npm:pi-context-guard-jev     # needs an OpenRouter key: OPENROUTER_API_KEY or /login
@@ -17,45 +19,53 @@ pi install npm:pi-context-guard-jev     # needs an OpenRouter key: OPENROUTER_AP
 
 ## Why use it
 
-- **Most of a session's context is tool output that stopped mattering.** Pi keeps every output for the
-  rest of the session, so every later request re-sends it, the context window fills, and compaction (a
-  lossy summary of everything) comes sooner. In the [measured examples](#examples-measured), the next
-  request carried **9.2k instead of 35.3k tokens** after reading a 164 KB file, **11.5k instead of 34.3k**
-  after a code search, and **5.7k instead of 7.5k** after a failing test run. Every follow-up was still
-  answered correctly.
+- **Long sessions fill up with things that stopped mattering.** Old search results, file reads and test
+  logs, and whole earlier exchanges about other tasks, stay in the context for the rest of the session.
+  Every later request re-sends them, the context window fills, and compaction (a lossy summary of
+  everything) comes sooner. In the [measured examples](#examples-measured), the next request carried
+  **9.2k instead of 35.3k tokens** after reading a 164 KB file, **11.5k instead of 34.3k** after a code
+  search, and **5.7k instead of 7.5k** after a failing test run. In an autonomous 30-request bug-fixing
+  run, the context peaked at **45k instead of 68–88k tokens** ([Long autonomous runs](#long-autonomous-runs)).
+  Every follow-up and every task still succeeded.
 - **Verbatim, never rewritten.** Jev is a classifier: it answers yes/no and multiple-choice questions and
   does not write text. Kept lines are copied byte for byte, and each removed stretch becomes one line that
-  says what was there (`[… 79 lines omitted: matches in dist/core/tools/grep.js, …]`). Unlike a summary,
-  nothing the model sees afterwards is paraphrased or invented.
+  says what was there (`[… 79 lines omitted: matches in dist/core/tools/grep.js, …]`). Small outputs and
+  earlier exchanges that are no longer needed are removed whole and leave a one-line note. Unlike a
+  summary, nothing the model sees afterwards is paraphrased or invented.
 - **Nothing is lost.** The original stays in the session file. The model can `recall` it (whole, by
-  regex, or by line range), and `/tree` back to before the edit restores it.
+  regex, or by line range), including a whole omitted exchange, and `/tree` back to before the edit
+  restores it.
 - **Judged against the work, not by size.** Truncation and size limits can't tell a needed line from
-  noise. Jev sees the question, the final answer and the earlier conversation, and decides chunk by chunk,
-  so lines the ongoing task needs survive even when this run's question was about something else. A
-  failed command the agent already got past is removed outright.
-- **Cheap and quick.** $0.0004–0.003 and 0.6–1 s per qualifying run (measured). It runs once, after the
-  run has finished, never while the agent works.
-- **Claude and OpenAI, cache-aware.** Tested with Claude Sonnet 5.5 and GPT-6 Luna. On Anthropic it adds a
-  cache breakpoint at the previous question, so the request after an edit still reads the conversation
-  up to there from cache.
+  noise. Jev sees the task, the agent's progress or final answer, and the earlier conversation, and decides
+  per chunk, per small output and per earlier exchange, so what the ongoing task needs survives. A failed
+  command the agent already got past, or a file read made stale by a later edit, goes.
+- **Works during long autonomous runs and in subagents.** Besides the end of each run, it checkpoints
+  during long runs, once old output has piled up, so a 2-hour run or a long worker doesn't carry
+  everything until it ends.
+- **Cheap, and careful with the prompt cache.** Jev costs $0.0003–0.007 and 0.6–2 s per pass (measured).
+  Edits are batched and made only when the smaller context pays for the prompt-cache rewrite they cause.
+  On Anthropic, cache breakpoints keep everything before the first edit readable from cache. Tested with
+  Claude Sonnet 5.5 and GPT-6 Luna.
 - **You can see what it does.** The footer shows `🛡 −10.1k · 1`. `/guard` lists every decision with Jev's
   probabilities, and `/guard off` turns it off for good.
 
 **Trade-offs:**
-- **It sends data to a third party.** Tool output goes to OpenRouter and TypeSafe ([Privacy](#privacy)).
-- **It can cost a little more in short sessions.** The first request after an edit rewrites the prompt
-  cache, so the first gain is context room; the bill shrinks only in longer sessions
+- **It sends data to a third party.** Tool output and conversation excerpts go to OpenRouter and TypeSafe
+  ([Privacy](#privacy)).
+- **It can cost a little more in short sessions.** The first request after an edit rewrites part of the
+  prompt cache, so the first gain is context room; the bill shrinks only in longer sessions
   ([Examples](#examples-measured)).
-- **Follow-ups may need `recall`.** A question that goes somewhere new can need something that was
-  removed.
+- **Follow-ups may need `recall`.** A question that goes back to something removed makes the model recall
+  it or search again.
 
 ## How it works
 
 1. **A run finishes.** When Pi is about to settle a completed run (`agent_before_settle`), the extension
-   looks at that run only: everything after the last user message.
-2. **It picks candidates.** These are text-only tool results of at least 4,000 characters, including the
-   output of failed commands. The run is
-   skipped if the candidates add up to less than 8,000 characters.
+   looks at the tool outputs of that run (everything after the last user message) and at earlier
+   exchanges (steps 6 and 7). Long runs also get checkpoints before they end
+   ([Long autonomous runs](#long-autonomous-runs)).
+2. **It picks large outputs.** These are text-only tool results of at least 4,000 characters, including
+   the output of failed commands. They are skipped if they add up to less than 8,000 characters.
 3. **It chunks each output.** grep-style output is grouped by file. Other output is cut into windows of
    roughly 0.8–4k characters, with breaks at blank lines where possible.
 4. **It sends one Jev request per segment** (up to 32,000 characters and 40 chunks). Each request asks three
@@ -69,10 +79,97 @@ pi install npm:pi-context-guard-jev     # needs an OpenRouter key: OPENROUTER_AP
    removed chunks with an omission line, and starts with a header that tells the model how to
    `recall` the full output. If the kept part would still be more than 60% of the original, the result is
    left as it is.
+6. **Small outputs** (400 characters up to the 4,000-character limit) are judged as whole items, many per
+   Jev request: "will the agent still need this output?" An output that is not needed becomes
+   `[context-guard] Omitted the output of bash `…` (1.2k chars): judged no longer needed. Full output: recall(…).`
+7. **Old exchanges** (a user prompt and everything up to the next prompt: replies, tool calls and results,
+   custom messages) from earlier, finished runs are judged against the current work, several per request:
+   "is this earlier exchange still relevant?" The last 2 exchanges before the current prompt are never
+   touched. An unrelated exchange keeps a one-line note on its prompt
+   (`[context-guard] Omitted an earlier exchange judged unrelated to the current work: "…" (N messages, ~Mk tokens). Full exchange: recall(…).`)
+   and every other entry of it is omitted (`replacement: null`), so tool calls and their results always go
+   together. Exchanges with images, with entries another extension changed, or under 2,000 characters
+   are not judged. An omission early in the session makes the next request rewrite the prompt cache
+   from there, so at run end it happens only when it pays off over as many later requests as the session
+   has had so far and saves at least 8,000 characters (`exchangeBreakEven`); otherwise the exchanges
+   wait, and are judged again at later run ends until their total pays off. All three kinds are judged
+   in parallel, at most `concurrency` (6) Jev requests at a time.
+
+During long runs, **mid-run checkpoints** do the same for older items before the run ends (next
+section).
 
 Errors, timeouts and unclear answers always mean *keep*. For the full algorithm, see
 [docs/DESIGN.md](docs/DESIGN.md).
 
+## Long autonomous runs
+
+Run-end distillation helps the *next* prompt. A 30–120 minute autonomous run (the main agent or a long
+worker) would still carry every tool output until it ends, and Pi compacts only near the window
+minus 16k tokens. So, while a run is going, the extension also checkpoints:
+
+1. **At a turn boundary** (`turn_end`; Pi applies the edit to the very next request, before its compaction
+   check), large and small outputs of the current run that are at least `midRunMinAgeTurns` (4) turns old
+   and not judged before are eligible, and so are old exchanges not judged during this run. The final turn
+   of a run is left to run-end distillation.
+2. **In batches:** a checkpoint runs only when the eligible items add up to `midRunBatchChars` (60,000)
+   characters, so edits, and the cache rewrites they cause, are rare.
+3. **Only when it pays:** an edit makes the next request rewrite the prompt cache once (OpenAI Codex: the
+   whole context; Anthropic: everything after the latest cache entry it can still read, at worst after the
+   question). The break-even rule
+   (`midRunBreakEven`) skips a checkpoint unless the expected saving over the rest of the run covers that.
+   Old exchanges come early in the context, so they move the rewrite back to the start; if the batch with
+   them does not pay off, the checkpoint runs on the tool outputs alone, and the exchanges wait for run end.
+4. **Jev gets a checkpoint question:** no final answer exists yet, so it sees the task, the agent's latest
+   notes, the tool calls made since this output, and whether the output was superseded (the file was
+   edited afterwards, or the same command ran again). It answers whether the agent will still need each
+   chunk to finish. Each output is judged mid-run at most once; run-end distillation judges the rest,
+   including outputs kept at a checkpoint, with the final answer.
+5. **On Anthropic** the previous question stays pinned, and the first request after a checkpoint also gets
+   a breakpoint at the latest cache entry this process wrote before the first edited output, so the
+   unchanged part of the run is still read from cache.
+
+**Measured** (Pi 1.0.3, e2e scenario `long`: "fix all failing tests" in a generated project with 10 bugs,
+the full suite after every fix; single runs, not a benchmark; costs are what Pi reports at list price):
+
+| Main model | Configuration | Requests | Peak context | Main-model cost | Checkpoints (Jev) | Suite green |
+|---|---|---|---|---|---|---|
+| GPT-6 Luna | guard off | 32 / 37 | 68.4k / 87.7k | $0.020 / $0.030 | – | yes / yes |
+| GPT-6 Luna | `midRun: false` (run end only) | 26 | 86.3k | $0.022 | – (run end $0.006) | yes |
+| GPT-6 Luna | default | 29 | **45.0k** | $0.021 | 2 at requests 16 and 24, 0.8 s each, removed 62k and 60k chars ($0.006 with run end) | yes |
+| GPT-6 Luna | `smallResultMinChars: 0` (large outputs only) | 31 | 72.4k | $0.024 | 1 at request 19, 1.0 s, removed 154k chars ($0.007 with run end) | yes |
+| Claude Sonnet 5.5 | guard off | 14 | 89.8k | $0.41 | – | yes |
+| Claude Sonnet 5.5 | `smallResultMinChars: 0` | 8 | 65.7k | $0.24 | none: too short to pay off (run end $0.003) | yes |
+| Claude Sonnet 5.5 | forced: `midRunBreakEven: false`, `midRunMinAgeTurns: 2`, `midRunBatchChars: 20000`, `smallResultMinChars: 0` | 12 | 43.3k | $0.34 | 3 ($0.0032 with run end) | yes |
+
+Context per request (every 5th request), GPT-6 Luna:
+
+| Request | 1 | 6 | 11 | 16 | 21 | 26 | 31 | 36 |
+|---|---|---|---|---|---|---|---|---|
+| guard off (37 requests) | 3.0k | 14.6k | 29.3k | 41.4k | 52.1k | 62.5k | 73.3k | 83.4k |
+| default (29 requests) | 3.2k | 17.0k | 35.3k | 41.0k | 39.3k | 28.9k | – | – |
+
+At the default run's first checkpoint, Jev distilled four superseded test logs (for example 30,164 → 8,163)
+and replaced 4 of 7 small reads of files the agent had already edited with one-line notes (for example
+3,386 → 145); the next request carried 20.1k instead of 41.0k tokens. No run needed `recall`.
+
+**Trade-offs:**
+- **OpenAI Codex reads nothing from cache after any edit** (2,560 or 0 tokens on the next request, after
+  every checkpoint measured, wherever the edit was). Each checkpoint re-sends the context uncached once and
+  pays back after about 8–10 later requests. In these 26–37-request runs the main-model cost came out
+  about the same with and without checkpoints, plus about $0.005 of Jev per run; the gain is a smaller
+  context.
+- **On Anthropic a checkpoint rewrites the cache after the latest entry the next request can read** (at worst after the question; writing costs 12.5× the read price).
+  In the forced run, the requests after the three checkpoints read 5.8k / 16.7k / 16.7k tokens and wrote
+  10.9k / 22.8k / 20.0k. After the first checkpoint of a run only the system prompt and question can be
+  read, because the run's first output is usually the first one edited. Claude solved the task in 8–14
+  requests, too few for a checkpoint to pay off, so with the default break-even rule it never
+  checkpointed; forced checkpoints in such short runs cost more than they saved. Checkpoints are meant
+  for runs that go on for dozens of requests.
+- **Latency mid-run:** a checkpoint pauses the run for the Jev requests (0.5–2.2 s measured, at most
+  `timeoutMs`).
+- Details: [docs/CACHE.md](docs/CACHE.md#mid-run-checkpoints), [docs/DESIGN.md](docs/DESIGN.md#mid-run-checkpoints),
+  wording probe in [docs/JEV.md](docs/JEV.md#checkpoint-wording).
+
 ## Install
 
 ```bash
@@ -103,12 +200,19 @@ each Jev request sends the following to OpenRouter, which forwards it to TypeSaf
 - excerpts of the earlier conversation in this session: the latest compaction or branch summary (up to
   2,000 characters), the first prompt (up to 1,000), and the last 3 earlier exchanges, each one prompt (up to
   800) and the last assistant text after it (up to 1,200). Tool calls and tool outputs of earlier runs are not
-  included. Set `historyExchanges` to `0` to send none of it,
+  included in this part. Set `historyExchanges` to `0` to send none of it,
 - the tool name and arguments,
-- the full text of the large tool output being judged. This can include source code, logs or anything else
-  the tools returned.
-
-Small results, edit/write results and runs below the size limits are never sent.
+- the full text of the large tool output being judged, and of small outputs (from 400 characters; up to
+  4,000 characters each). This can include source code, logs or anything else the tools returned,
+- excerpts of old exchanges being judged: each prompt (up to 600 characters), the last assistant text (up to
+  800), the tool calls as short labels (tool name and the command or path, up to 80 characters each) and the
+  size. Set `pruneExchanges` to `false` to send none of it.
+
+Never sent: tool outputs below `smallResultMinChars` (400 characters), results of excluded tools (`edit`,
+`write`), outputs with images, old exchanges with images or with entries another extension changed,
+exchanges below 2,000 characters, and the last `keepRecentExchanges` (2) exchanges before the current
+prompt. Large outputs are sent only when the run's large outputs add up to `minRunChars` (8,000); small
+outputs and old exchanges are sent at every run end and every mid-run checkpoint that has any.
 
 `/guard off` turns the extension off and saves that choice in the settings file, so it stays off in later
 sessions. See OpenRouter's data policy:
@@ -119,22 +223,22 @@ sessions. See OpenRouter's data policy:
 | Text | Meaning |
 |---|---|
 | `🛡 0 saved` | On and ready; nothing is distilled on this branch yet |
-| `🛡 distilling…` | Jev is judging the outputs of the run that just finished |
-| `🛡 −4.2k · 1` | ≈ tokens kept out of context · distilled results: about 4.2k tokens are currently kept out of context, by 1 distilled tool result |
+| `🛡 distilling…` | Jev is judging the outputs of the run that just finished, or of a mid-run checkpoint |
+| `🛡 −4.2k · 1` | ≈ tokens kept out of context · items: about 4.2k tokens are currently kept out of context, by 1 distilled or omitted tool output or old exchange |
 | `🛡 guard off` | Turned off with `/guard off` or `"enabled": false` |
 | `🛡 no openrouter key` | No credential for the classifier's provider (the provider name is filled in) |
 | `🛡 Jev model not found` | The configured `model` is not in Pi's classifier catalog |
 
 The token figure is Pi's usual estimate, characters ÷ 4. It counts what is kept out of the model's context
-**on the current branch right now**: original length minus replacement length, for every distilled result
-that is still in the context. It goes down after `/tree` to a point before an edit, and after compaction
+**on the current branch right now**: original length minus replacement length, for every distilled output
+and omitted exchange that is still in the context. It goes down after `/tree` to a point before an edit, and after compaction
 removes distilled results. It is not a measured token count and not a cumulative total.
 
 ## `/guard` command
 
 | Command | Effect |
 |---|---|
-| `/guard` or `/guard status` | Shows whether the guard is on, the model, the savings on this branch, Jev runs, requests and cost, and the last run's result for each tool output, including Jev traces such as `kw.15 focus=chunk_4:.52 keep 9/29` |
+| `/guard` or `/guard status` | Shows whether the guard is on, the model, the savings on this branch (distilled outputs and omitted exchanges), Jev runs and mid-run checkpoints, requests and cost, whether checkpoints and exchange pruning are on, and the last run's or checkpoint's result for each item (large, small, exchange), including Jev traces such as `kw.15 focus=chunk_4:.52 keep 9/29` or `p.06` |
 | `/guard on` / `/guard off` | Turns the guard on or off and saves `enabled` to the settings file |
 | `/guard reload` | Re-reads the settings file and shows any warnings again |
 
@@ -142,8 +246,9 @@ Settings are read at session start and on `/guard reload`.
 
 ## `recall` tool
 
-Every distilled result starts with `[context-guard]` and ends its header with
-`Full output: recall({"entryId":"…"}).` The model can call:
+Every distilled output and omitted exchange starts with `[context-guard]` and ends with
+`recall({"entryId":"…"})`. For an omitted exchange, recall returns the whole exchange as a transcript
+(prompt, replies, tool calls and results, custom messages). The model can call:
 
 | Parameter | Meaning |
 |---|---|
@@ -164,8 +269,8 @@ fall back to the default.
 |---|---|---|
 | `enabled` | `true` | Master switch (`/guard on` and `/guard off` write it) |
 | `model` | `"openrouter/~typesafe/jev-latest"` | Classifier as `provider/id`; the first `/` separates the provider |
-| `minResultChars` | `4000` | Ignore tool results shorter than this many characters |
-| `minRunChars` | `8000` | Skip the run if its candidates add up to less than this |
+| `minResultChars` | `4000` | Tool results from this size are chunked; shorter ones are judged whole (see `smallResultMinChars`) |
+| `minRunChars` | `8000` | Distill a run's large outputs only if they add up to at least this (small outputs and exchanges are judged regardless) |
 | `keepWholeThreshold` | `0.7` | Keep a segment untouched if P(`keep_whole`) is at least this |
 | `focusWholeThreshold` | `0.6` | Keep a segment untouched if `focus` picks "whole" with at least this probability |
 | `chunkKeepThreshold` | `0.6` | Keep a chunk if P(chunk needed) is at least this |
@@ -179,7 +284,20 @@ fall back to the default.
 | `excludeTools` | `["edit", "write"]` | Tools whose results are never distilled |
 | `distillErrors` | `true` | Also distill error results, such as the log of a failing test or build; Jev is told the call failed, and that its details are usually no longer needed once the agent got past it |
 | `historyExchanges` | `3` | Earlier exchanges (prompt + last assistant text) sent to Jev so it judges relevance against the ongoing work; `0` sends no earlier conversation at all (integer 0–10) |
-| `pinAnthropicCache` | `true` | Add an Anthropic cache breakpoint at the previous user question ([docs/CACHE.md](docs/CACHE.md)) |
+| `pinAnthropicCache` | `true` | Anthropic cache breakpoints: at the previous user question on every request, and on the first request after context edits a read point (at a cache entry this process wrote) and a write anchor ([docs/CACHE.md](docs/CACHE.md)) |
+| `midRun` | `true` | Mid-run checkpoints during long runs ([Long autonomous runs](#long-autonomous-runs)) |
+| `midRunMinAgeTurns` | `4` | Judge an output mid-run only once it is at least this many turns old (integer 1–100) |
+| `midRunBatchChars` | `60000` | Run a checkpoint only when the not-yet-judged eligible outputs add up to this many characters |
+| `midRunBatchCharsOpenAI` | `60000` | The same, for `openai-codex` models |
+| `midRunChunkKeepThreshold` | `0.6` | Keep a chunk at a checkpoint if P(still needed) is at least this (probe in docs/JEV.md) |
+| `midRunBreakEven` | `true` | Skip a checkpoint unless its one-time prompt-cache rewrite is likely to pay off; `false` checkpoints purely for context room |
+| `smallResultMinChars` | `400` | Tool results from this size up to `minResultChars` are judged as whole items; `0` turns this off |
+| `smallKeepThreshold` | `0.45` | Keep a small output if P(still needed) is at least this (probe in docs/JEV.md) |
+| `pruneExchanges` | `true` | Judge old exchanges and omit the unrelated ones |
+| `keepRecentExchanges` | `2` | The last this many exchanges before the current prompt are never omitted |
+| `exchangeOmitThreshold` | `0.2` | Omit an exchange only if P(still relevant) is below this (probe in docs/JEV.md) |
+| `exchangeBreakEven` | `true` | At run end, omit unrelated exchanges only when the cache rewrite they cause pays off; deferred ones are judged again at later run ends and omitted once their total pays off. `false` always omits them |
+| `exchangeMinSavingChars` | `8000` | A run end omits exchanges only if they save at least this many characters together (when they cost a cache rewrite) |
 
 Example:
 
@@ -189,23 +307,28 @@ Example:
 
 ## What it does not touch
 
-- user and assistant messages, custom messages
+- messages of the current run other than tool results, and the last `keepRecentExchanges` exchanges before
+  the current prompt
+- compaction and branch summaries, exchanges that did not finish (the last reply has tool calls), and
+  messages Pi cannot edit (system messages, `!` shell executions), which stay in place inside an omitted
+  exchange
 - tool results that contain images
 - error results, if `distillErrors` is `false`
 - results of excluded tools (`edit`, `write` by default)
-- results shorter than `minResultChars`, and runs below `minRunChars`
+- results shorter than `smallResultMinChars` (400), and large results of runs below `minRunChars`
 - aborted or failed runs
 - results that are already distilled, or that another extension already edited
 - single lines too long for one Jev request (kept as they are)
-- earlier runs: only the run that just finished is edited
+- tool outputs of earlier runs one by one: earlier runs are only touched as whole exchanges
 
 ## Limitations
 
 - **Lossy by design.** Relevance is judged against the run's question and answer and the earlier
   conversation (the ongoing task), so a follow-up that goes somewhere new may need something that was
-  removed. The header and omission lines say what is missing, and `recall` returns it.
+  removed. The header and omission lines say what is missing, and `recall` returns it. An earlier task
+  that comes back after several unrelated ones may have been omitted as a whole exchange.
 - **Latency.** Settling waits for Jev. This adds about 0.5–1.2 s after each qualifying run (measured), and
-  never more than `timeoutMs` (8 s).
+  never more than `timeoutMs` (8 s). A mid-run checkpoint pauses the run the same way.
 - **Cost.** Each qualifying run costs a few Jev requests, measured at under $0.001 per run. The first prompt
   after an edit also has a one-time partial prompt-cache miss ([docs/CACHE.md](docs/CACHE.md)).
 - **Steering.** A message sent during a run starts a new span. Tool results from before it are never
@@ -216,6 +339,8 @@ Example:
 Real runs on public code with Pi 1.0.3: each example ran once with the guard on and once with it off
 (`--no-guard`), from the same prompts. These are single runs, not a benchmark: the main model takes a
 different path each time. Reproduce with `node test/e2e/examples.mjs` ([docs/TESTING.md](docs/TESTING.md)).
+Examples A–C judge large outputs only (their numbers match `smallResultMinChars: 0`); example D shows the
+default configuration and `exchangeBreakEven: false`.
 
 **A. Searching a real codebase.** In a copy of the published Pi package (`dist/` without source maps and
 the minified bundle, plus `docs/`), the agent was asked where Pi decides to auto-compact, starting with
@@ -241,8 +366,7 @@ carried **9.2k tokens instead of 35.3k**.
 the unit tests (a 15.8k-character log, 3 failures among 116 tests) and reported the cause. Jev kept the
 3 failures with their assertion messages and the failing-test summary, and dropped the passing tests:
 15.8k → 8.6k chars. Turn 2 ("now fix it") carried **5.7k tokens instead of 7.5k**, and both runs fixed
-the bug. (In the first run of this example, failing commands were never distilled; `distillErrors` was
-added because of it.)
+the bug. Failing commands are distilled because `distillErrors` is on by default.
 
 What the model sees after distilling (example A, Claude; the first lines of the real replacement):
 
@@ -264,6 +388,24 @@ dist/core/compaction/compaction.d.ts:59:export declare function shouldCompact(co
 | A. code search | GPT-6 Luna | 56.6k → 14.8k chars (2 outputs; 2 more, 15k, left whole) | 5 requests, 0.6 s, $0.0015 | 11.5k / 34.3k tokens | $0.0012 / $0.0005 | correct, no `recall` |
 | C. 164 KB file | GPT-6 Luna | 163k → 26k chars (7 reads) | 7 requests, 0.9 s, $0.0027 | 9.2k / 35.3k tokens | $0.0007 / $0.0005 | correct, no `recall` |
 | B. failing test log | GPT-6 Luna | 15.8k → 8.6k chars | 1 request, 0.8 s, $0.0004 | 5.7k / 7.5k tokens | $0.0022 / $0.0016 | fixed in both runs |
+| D. five topics | Claude Sonnet 5.5, `exchangeBreakEven: false` | 1 old exchange omitted (8.8k chars) | 2 requests, $0.0003 | 8.0k / 12.8k tokens (prompt 5) | $0.017 / $0.006 (prompt 5's first request) | correct, searched again |
+
+**D. Several topics in one session.** In the generated test repo, five prompts in a row: which API
+clients retry on 503; a README paragraph for `StatusBar`; a count of HTTP 500 lines in a log; where
+`StatusBadge` is used; and then "back to the clients that retry on 503: which of them also retry on 502?"
+(`node test/e2e/run-e2e.mjs --scenario topics --turns 5`). The README exchange is under 2,000 characters
+and is never judged; the last 2 exchanges before each prompt are never judged either.
+
+| Model, configuration | First exchange at the end of prompt 4 | At the end of prompt 5 | Prompt 5 | Context of prompt 5's first request | Main-model cost | Jev |
+|---|---|---|---|---|---|---|
+| GPT-6 Luna, default | judged unrelated (P = 0.07), **deferred**: too small to pay for the cache rewrite in a 4-prompt session | relevant (P = 0.95), kept | correct, no `recall` | 7.5k, 6,656 read from cache | $0.0018 | $0.0005 |
+| Claude Sonnet 5.5, default | unrelated (P = 0.07), deferred | relevant (P = 0.97), kept | correct, no `recall` | 11.5k, 11,049 read from cache | $0.073 | $0.0003 |
+| Claude Sonnet 5.5, `exchangeBreakEven: false` | unrelated (P = 0.08), **omitted** (8.8k characters, 5 entries) | – | correct, searched again | 8.0k: 1,760 read, 6,280 written | $0.087 | $0.0003 |
+| Claude Sonnet 5.5, guard off | – | – | – | 12.8k | $0.083 | – |
+
+In this short session the gate kept the first exchange, which the follow-up then needed. With the gate off,
+the omission made the next request rewrite the cache from the omitted exchange on (it comes before the
+question pin) and the model searched again; the smaller context pays off only in longer sessions.
 
 **When it doesn't help.**
 
@@ -274,13 +416,14 @@ dist/core/compaction/compaction.d.ts:59:export declare function shouldCompact(co
   tokens instead of 13,824: a partial miss).
   At the per-token prices in these runs, the smaller context earns that back after about 7 more requests
   (Claude, A), about 3 (GPT, A) or about 1 (GPT, C). Before that, the benefit is a smaller context window,
-  not a lower bill. Anthropic requests also get a cache breakpoint at the previous question, which raised
-  the first request's cache read from 1,693 to 5,810 tokens in a long run ([docs/CACHE.md](docs/CACHE.md)).
+  not a lower bill. Anthropic requests also get a cache breakpoint at the previous question; in a run with
+  11 sequential tool calls it raised the first request's cache read from 1,693 to 5,810 tokens compared
+  with `pinAnthropicCache: false` ([docs/CACHE.md](docs/CACHE.md)).
 - **Small or fully needed outputs are left whole.** In A, GPT's 9k-character read of `agent-session.js` was
   `not-worth` (too much of it was needed) and a 6k narrowed search was judged needed whole.
 
-Earlier numbers on the generated StatusBadge test repo (96% removed, `recall` of a removed line) are in
-[docs/TESTING.md](docs/TESTING.md#results-so-far-pi-103).
+Numbers on the generated StatusBadge test repo (96% removed, `recall` of a removed line) are in
+[docs/TESTING.md](docs/TESTING.md#results-pi-103).
 
 ## Documentation
 

package/docs/CACHE.md

@@ -21,7 +21,8 @@ runs, not averages.
   Consecutive `tool_use` blocks count as one position, and so do consecutive `tool_result` blocks.
 - Entries with a longer TTL must come before entries with a shorter TTL.
 - Reading an entry refreshes its TTL. The docs do **not** say that reading a longer prefix refreshes a
-  shorter entry that was written separately.
+  shorter entry that was written separately; measured below, it does
+  ([Mid-run checkpoints](#mid-run-checkpoints)).
 - Pricing relative to base input: 5-minute write 1.25×, 1-hour write 2×, read 0.1× (0.05× on Opus 5.5).
 - The minimum cacheable prompt is 512 tokens on Sonnet/Opus 5.5.
 
@@ -97,8 +98,10 @@ had more than 4 breakpoints.
 
 Over WebSocket, Pi's `openai-codex` transport sends only the new input items with `previous_response_id`, as
 long as the input only grows. After an edit, the input is no longer an extension of the previous one, so
-Pi sends the full input. The server-side continuation is lost for that request, and the prompt cache is reused
-only up to the first edited item. This happens once, on the first request after the edit.
+Pi sends the full input. The server-side continuation is lost for that request, and in every measurement the
+cache read was at most the 2,560-token stable prefix, **wherever the first edited item was** (also when it
+was 18 turns into a run; see [Mid-run checkpoints](#mid-run-checkpoints)). This happens once, on the first
+request after the edit.
 
 ### Measured
 
@@ -112,6 +115,135 @@ GPT-6 Luna (openai-codex), badge scenario, turn 2 first request:
 
 The pin does not apply to OpenAI.
 
+## Mid-run checkpoints
+
+A checkpoint edits outputs from earlier in the *current* run, so the next request cannot read the run from
+cache beyond the first edited output. Two questions decide the design: does a cache entry at the
+unchanged prefix still exist when the checkpoint comes, possibly many minutes after it was written; and
+how much does the one-time rewrite cost against the per-request saving.
+
+### Do cache entries on the read path stay alive? (measured)
+
+`node test/e2e/ttl-probe.mjs` (Claude Sonnet 5.5, `pi-claude-auth`). The model `cat`s a ~31k-token file
+(entry A ends at that result, written once as Pi's rolling breakpoint), then makes further requests; at the
+end, the request after A is changed so that only A can be read.
+
+| Arm | Between writing A and the test request | Test request |
+|---|---|---|
+| refresh | 8 requests in 7.6 min, each marking A (5m) and reading the longer prefix | cacheRead **31,001** (A alive) |
+| control | the same 8 requests, A **not** marked again | cacheRead **30,999** (A alive) |
+| ttl1h | as refresh, A at `ttl: "1h"` | cacheRead 31,001; but switching the earlier breakpoints to 1h was itself a full miss (cacheRead 0 → 31,001 written at the 1h price) |
+| idle (5m) | **no request for 362 s** (Pi's cache warming stopped) | cacheRead **1,431**, cacheWrite 29,633 (A expired) |
+| idle1h | no request for 365 s, A at 1h | cacheRead 30,977 (A alive) |
+
+So the 5-minute TTL is real (idle), but **every request that reads a longer prefix keeps the shorter entries
+on its path alive** (control). During an active run (requests seconds apart, and Pi's own cache warming,
+setting `cacheWarming: "streaming"` by default, during long tool calls), every turn boundary that was once
+the rolling breakpoint stays readable. A 1-hour TTL is not needed and would cost a full rewrite at 2× once.
+The extension therefore keeps Pi's TTL (5m by default). Pi's warming requests go through
+`before_provider_request` (seen in the capture), so they carry the extension's breakpoints too.
+
+What the lookback limit still breaks: the request after an edit has its rolling breakpoint at the end; the
+first edited output is usually many more than 20 positions back. So every Anthropic request keeps the
+previous-question pin (the floor: system prompt and question; with `pi-claude-auth`, the relocated system
+prompt sits in that same user message), and the first request after any edits (checkpoint, run end, or
+another extension; the `context_edit` entries after the last answered assistant message) gets two more
+breakpoints (`cacheAnchors` in `src/checkpoint.ts`):
+
+- a **read point**, only at a cache entry this process saw itself write: a `tool_result` block that carried
+  a breakpoint in a request it sent, to the same model, still on the branch, and with no `context_edit` at
+  or before it and no compaction or branch summary since, written or read through within the last 5
+  minutes. A response refreshes an entry when its `cacheRead` shows the request read clearly past it: at
+  least the run's baseline (the full input of the run's first request) plus the characters from the
+  question to the entry divided by 1.5, which overestimates the tokens (code ≈ 2.3 characters per token).
+  This follows the control arm above. The latest trusted entry before the first edited entry is used. The
+  log lives in memory and is empty after a reload, so a read point is never inferred;
+- a **write anchor** at the end of the first edited tool-result batch, so the edited prefix gets an entry
+  that the next checkpoint can read. It enters the log only if it was really in the request.
+
+Budget: Pi uses 3 (API key: tools, system, rolling) or 4 breakpoints (OAuth: plus the identity
+`system[0]`; `pi-claude-auth`: tools, `system[1]`, question, rolling). To make room, the identity breakpoint
+(covered by `system[1]`) and then the tools breakpoint (covered by the system breakpoint) are dropped. If
+that is not enough, the write anchor goes first, then the read point; the question stays. Marks use the
+rolling breakpoint's TTL and are skipped if they would break the TTL order. Only breakpoints actually left in
+the payload are logged, so an anchor dropped for the budget is never treated as written.
+
+### The first request after a checkpoint (measured)
+
+Claude Sonnet 5.5, long scenario with settings that force checkpoints into a short run
+(`midRunBreakEven: false`, `midRunMinAgeTurns: 2`, `midRunBatchChars: 20000`), with `pi-claude-auth` and
+`test/e2e/plain-test-runs.ts`:
+
+| Configuration | Checkpoint | First edited output | Next request |
+|---|---|---|---|
+| default anchors | 1st (after request 3) | the run's first output | cacheRead **5,832** (system prompt + question, via the question pin), cacheWrite 10,867 |
+| default anchors | 2nd (after request 8) | 3 turns into the run | cacheRead **16,699** (read at the entry written after the 1st checkpoint), cacheWrite 22,778 |
+| default anchors | 3rd (after request 10) | 5 turns into the run | cacheRead **16,699**, cacheWrite 20,008 |
+| `pinAnthropicCache: false` | 1st | the run's first output | cacheRead 5,832, cacheWrite 29,570 |
+| `pinAnthropicCache: false` | 2nd, 3rd, 4th | 2–6 turns into the run | cacheRead **5,832** each time, cacheWrite 41,162 / 32,183 / 33,872 |
+
+Without the guard's breakpoints, every request after a checkpoint read only the system-and-question entry
+and rewrote the whole run. With them, the later checkpoints read up to the entry written by the first
+request after the first checkpoint. The first checkpoint of a run usually edits the run's first output, so
+only the system prompt and question can be read then. When a read point is placed, the write anchor often
+does not fit the budget (question, read point, `system[1]` and rolling take all four slots).
+
+GPT-6 Luna (openai-codex), long scenario, 8 checkpoints in 5 runs (default batch and age settings), at
+requests 13–30: the first request after every checkpoint read **2,560 or 0** tokens from cache (17k–29k
+uncached), against 35k–72k read on the request before it. That is the transport, not the position of the
+edit.
+
+### Economics and the break-even rule
+
+At the prices in these runs (Sonnet 5.5: input $2/M, 5m write $2.5/M, read $0.2/M; GPT-6 Luna: input
+$0.10/M, cached $0.01/M), a checkpoint costs once, and saves on every later request:
+
+- OpenAI Codex: the whole context is sent uncached once: extra ≈ 0.9 × context × input price.
+- Anthropic: everything after the read point is written again: extra ≈ (1.25 − 0.1) × rewritten tokens.
+- Saving per later request: the removed tokens × the cache-read price. Jev removed 60–90% of checkpoint
+  outputs in the runs.
+
+Measured (GPT, a run with two checkpoints): checkpoint 1 removed 57k characters; the next request was 17,144 tokens uncached
+(≈ $0.0015 extra) and every later request carried about 19k tokens less (≈ $0.0002 saved each): break-even
+after about 8 requests, and the run went on for 12 more. Checkpoint 2 needed about 9 requests and the run
+ended 2 requests later. On Anthropic the ratio of write to read price is larger (12.5:1 vs 10:1).
+
+So checkpoints pay for themselves only when the run goes on for roughly 8–30 more requests. The
+**break-even rule** (`midRunBreakEven`, on) skips a checkpoint unless
+`pending chars × turns so far ≥ factor × rewrite chars` (OpenAI: factor 13, rewrite = whole context;
+Anthropic: factor 16, rewrite = context after the read point the next request would get, or after the
+question when no logged entry is trusted, or the whole context with pinning off; other providers: factor
+16, rewrite = context from the first eligible output), assuming the run goes on
+about as long as it has run so far. In the measured Claude runs (8–14 requests) it never allowed a
+checkpoint; in the GPT runs (26–37 requests) it allowed one or two. Turn it off to checkpoint purely for
+context room.
+
+## Old exchanges
+
+An omitted exchange comes before the current question, so the first request after it cannot read the
+question's cache entry: that prefix contains the edit. `cacheAnchors` handles this like any other edit: the
+read point is the latest trusted logged entry before the earliest edited entry, and the question pin is
+still placed (it writes a new entry for the requests that follow). Without such an entry, only the tools and
+system prompt are read. The break-even estimate (`anthropicRewriteChars`) follows the same rule: with an edit
+at or before the question and no trusted read point, the rewrite is the whole context.
+
+Measured (`topics` scenario, Claude Sonnet 5.5, first request of prompt 5 after the first exchange was
+omitted at the end of prompt 4 with `exchangeBreakEven: false`): cacheRead 1,760 (tools and system),
+cacheWrite 6,280, against cacheRead 12,376 and cacheWrite 438 with the guard off. With the default gate the
+omission was deferred and prompt 5 read 11,049 tokens from cache. On OpenAI Codex any edit makes the next
+request read at most the 2,560-token stable prefix (see above).
+
+So each omission costs one rewrite of everything after the omitted exchange on the next request (Pi's
+default `cacheWarming: "streaming"` keeps the cache warm between prompts that come within the TTL) and saves
+the exchange on every request after it. At run end, omissions therefore go through a break-even gate
+(`exchangeBreakEven`): their marginal rewrite (beyond what the pass's other edits rewrite anyway) must pay
+off over as many later requests as the session has had so far, and they must save at least
+`exchangeMinSavingChars` (8,000 characters); otherwise they wait and accumulate. On OpenAI Codex a pass that
+edits anything else already costs the whole context, so exchange omissions in that pass are free. Mid-run, exchanges join the checkpoint batch only when the batch with them pays off; their early
+position makes the rewrite most of the context, so with factor 16 the batch must reach about
+1/16 of the context per turn so far (for example, old exchanges that are half of the context pay off from
+about turn 32).
+
 ## Cost model
 
 All units are input-token equivalents, with a 5-minute TTL (write 1.25×, read 0.1×).