pi-context-guard-jev 0.1.0 → 0.3.1

package/README.md

@@ -1,294 +1,139 @@
 # pi-context-guard-jev
 
-**Your agent ran a 50 KB search to answer one question. Every later request still carries all 50 KB.
-This Pi extension keeps the lines that mattered.**
+A [Pi](https://github.com/earendil-works/pi) extension that removes tool output your agent no longer needs
+from its context. The Jev classifier decides which lines to keep. Kept lines are copied verbatim, and the
+model can `recall` anything that was removed.
 
-![A broad 496-line search is distilled after the run; the footer shows about 10.1k tokens kept out of context](https://raw.githubusercontent.com/alandarev/pi-context-guard-jev/master/assets/screenshot.png)
-
-When a run finishes, the extension asks the Jev classifier (by TypeSafe) which parts of each large tool
-output the work still needs: a broad `rg`, a full-file `read`, a failing test log. It then uses Pi's
-append-only `context_edit` entries to replace each output with only those parts, copied verbatim. Later
-turns carry the answer and its evidence instead of everything that was searched. The raw output stays in
-the session file, and the model can fetch it again with the `recall` tool.
-
-```bash
-pi install npm:pi-context-guard-jev     # needs an OpenRouter key: OPENROUTER_API_KEY or /login
-```
-
-## Why use it
-
-- **Most of a session's context is tool output that stopped mattering.** Pi keeps every output for the
-  rest of the session, so every later request re-sends it, the context window fills, and compaction (a
-  lossy summary of everything) comes sooner. In the [measured examples](#examples-measured), the next
-  request carried **9.2k instead of 35.3k tokens** after reading a 164 KB file, **11.5k instead of 34.3k**
-  after a code search, and **5.7k instead of 7.5k** after a failing test run. Every follow-up was still
-  answered correctly.
-- **Verbatim, never rewritten.** Jev is a classifier: it answers yes/no and multiple-choice questions and
-  does not write text. Kept lines are copied byte for byte, and each removed stretch becomes one line that
-  says what was there (`[… 79 lines omitted: matches in dist/core/tools/grep.js, …]`). Unlike a summary,
-  nothing the model sees afterwards is paraphrased or invented.
-- **Nothing is lost.** The original stays in the session file. The model can `recall` it (whole, by
-  regex, or by line range), and `/tree` back to before the edit restores it.
-- **Judged against the work, not by size.** Truncation and size limits can't tell a needed line from
-  noise. Jev sees the question, the final answer and the earlier conversation, and decides chunk by chunk,
-  so lines the ongoing task needs survive even when this run's question was about something else. A
-  failed command the agent already got past is removed outright.
-- **Cheap and quick.** $0.0004–0.003 and 0.6–1 s per qualifying run (measured). It runs once, after the
-  run has finished, never while the agent works.
-- **Claude and OpenAI, cache-aware.** Tested with Claude Sonnet 5.5 and GPT-6 Luna. On Anthropic it adds a
-  cache breakpoint at the previous question, so the request after an edit still reads the conversation
-  up to there from cache.
-- **You can see what it does.** The footer shows `🛡 −10.1k · 1`. `/guard` lists every decision with Jev's
-  probabilities, and `/guard off` turns it off for good.
-
-**Trade-offs:**
-- **It sends data to a third party.** Tool output goes to OpenRouter and TypeSafe ([Privacy](#privacy)).
-- **It can cost a little more in short sessions.** The first request after an edit rewrites the prompt
-  cache, so the first gain is context room; the bill shrinks only in longer sessions
-  ([Examples](#examples-measured)).
-- **Follow-ups may need `recall`.** A question that goes somewhere new can need something that was
-  removed.
-
-## How it works
-
-1. **A run finishes.** When Pi is about to settle a completed run (`agent_before_settle`), the extension
-   looks at that run only: everything after the last user message.
-2. **It picks candidates.** These are text-only tool results of at least 4,000 characters, including the
-   output of failed commands. The run is
-   skipped if the candidates add up to less than 8,000 characters.
-3. **It chunks each output.** grep-style output is grouped by file. Other output is cut into windows of
-   roughly 0.8–4k characters, with breaks at blank lines where possible.
-4. **It sends one Jev request per segment** (up to 32,000 characters and 40 chunks). Each request asks three
-   kinds of question about the user's question and the final answer, and, when there is one, the earlier
-   conversation of the session (so output that the ongoing work still needs is kept even if this run's
-   question was about something else):
-   - `keep_whole`: is all of this output still needed?
-   - `focus`: does the answer use most of it, none of it, or a few chunks (and if so, which one matters most)?
-   - one yes/no question per chunk: is this chunk needed?
-5. **It writes a `context_edit` per result.** The edit keeps the needed chunks verbatim, replaces each run of
-   removed chunks with an omission line, and starts with a header that tells the model how to
-   `recall` the full output. If the kept part would still be more than 60% of the original, the result is
-   left as it is.
-
-Errors, timeouts and unclear answers always mean *keep*. For the full algorithm, see
-[docs/DESIGN.md](docs/DESIGN.md).
+![A 496-line search is distilled after the run; the footer shows about 10.1k tokens kept out of context](https://raw.githubusercontent.com/alandarev/pi-context-guard-jev/master/assets/screenshot.png)
 
 ## Install
 
 ```bash
 pi install npm:pi-context-guard-jev
-# or from GitHub
-pi install git:github.com/alandarev/pi-context-guard-jev
-# or, for development, load it for one session from a checkout
-pi -e ./src/index.ts
 ```
 
-### Requirements
+Requires Pi ≥ 1.0.3 and an OpenRouter credential (`OPENROUTER_API_KEY` or `/login`). Once it is ready, the
+footer shows `🛡 0 saved`.
 
-- Pi ≥ 1.0.3
-- An OpenRouter credential (`OPENROUTER_API_KEY` or `/login`) for the default classifier,
-  `openrouter/~typesafe/jev-latest`. Any classifier model in Pi's catalog can be set with the `model`
-  setting, but only the default has been tested.
+## What it removes
 
-If the model or credential is missing, the extension does nothing and the status bar says so.
+| Item | Rule | The model then sees |
+|---|---|---|
+| Large tool output (≥ 4k chars) | Jev keeps the chunks the work still needs | The kept lines, verbatim, and one line per removed stretch |
+| Small tool output (400–4k chars) | Jev: is it still needed? | A one-line note |
+| Earlier exchange about another task | Jev: is it still relevant? | A one-line note on its prompt |
+| Image in a tool result | After 3 turns (no Jev call) | The result's text and a one-line note |
 
-## Privacy
+The extension runs when a run finishes. In long runs it also runs mid-run, once enough old output has piled
+up. An edit makes the next request rewrite part of the prompt cache, so mid-run edits and edits to earlier
+exchanges wait until the smaller context pays for that. It leaves alone images you send, results of `edit`
+and `write`, and the last two exchanges before your current prompt.
 
-**This extension sends data to a third party, and it is on by default.** After every run that qualifies,
-each Jev request sends the following to OpenRouter, which forwards it to TypeSafe:
+## Example
 
-- the user's question (up to 4,000 characters),
-- the final answer (up to 6,000 characters),
-- the assistant's notes written during the run (up to 2,000 characters),
-- excerpts of the earlier conversation in this session: the latest compaction or branch summary (up to
-  2,000 characters), the first prompt (up to 1,000), and the last 3 earlier exchanges, each one prompt (up to
-  800) and the last assistant text after it (up to 1,200). Tool calls and tool outputs of earlier runs are not
-  included. Set `historyExchanges` to `0` to send none of it,
-- the tool name and arguments,
-- the full text of the large tool output being judged. This can include source code, logs or anything else
-  the tools returned.
+The agent ran `rg -n -i compact dist docs` (50 KB) to find where Pi decides to auto-compact. After the
+run, the output that later requests carry starts like this:
 
-Small results, edit/write results and runs below the size limits are never sent.
+```text
+[context-guard] Distilled the output of bash `rg -n -i compact dist docs`: kept 141 of 431 lines (15.1k of 50.4k chars); the rest was judged not needed for the answer. Full output: recall({"entryId":"1704787f"}).
+[… 79 lines omitted: matches in dist/core/tools/grep.js, dist/core/session-manager.js, dist/core/export-html/template.js, dist/core/export-html/template.css, dist/core/sdk.js and 5 more …]
+dist/core/compaction/compaction.d.ts:25:    /** Extension-specific data (e.g., ArtifactIndex, version markers for structured compaction) */
+dist/core/compaction/compaction.d.ts:28:export interface CompactionSettings {
+dist/core/compaction/compaction.d.ts:33:export declare const DEFAULT_COMPACTION_SETTINGS: CompactionSettings;
+dist/core/compaction/compaction.d.ts:54:/** Estimate projected context without trusting usage captured before a later edit or compaction. */
+dist/core/compaction/compaction.d.ts:57: * Check if compaction should trigger based on context usage.
+dist/core/compaction/compaction.d.ts:59:export declare function shouldCompact(contextTokens: number, contextWindow: number, settings: CompactionSettings): boolean;
+[… 44 lines omitted: matches in dist/core/compaction/compaction.d.ts, dist/core/compaction/branch-summarization.d.ts, dist/core/compaction/index.d.ts, dist/core/session-manager.d.ts, dist/core/cache-stats.js and 2 more …]
+…
+```
 
-`/guard off` turns the extension off and saves that choice in the settings file, so it stays off in later
-sessions. See OpenRouter's data policy:
-<https://openrouter.ai/docs/guides/privacy/data-collection>.
+With the matches it kept, the model then answered which compaction settings a user can change and their
+defaults correctly, without `recall`.
 
-## Status bar
+Small outputs, old exchanges and images leave one line each:
 
-| Text | Meaning |
-|---|---|
-| `🛡 0 saved` | On and ready; nothing is distilled on this branch yet |
-| `🛡 distilling…` | Jev is judging the outputs of the run that just finished |
-| `🛡 −4.2k · 1` | ≈ tokens kept out of context · distilled results: about 4.2k tokens are currently kept out of context, by 1 distilled tool result |
-| `🛡 guard off` | Turned off with `/guard off` or `"enabled": false` |
-| `🛡 no openrouter key` | No credential for the classifier's provider (the provider name is filled in) |
-| `🛡 Jev model not found` | The configured `model` is not in Pi's classifier catalog |
+```text
+[context-guard] Omitted the output of bash `npm test` (1.2k chars): judged no longer needed. Full output: recall({"entryId":"…"}).
+[context-guard] Omitted an earlier exchange judged unrelated to the current work: "which API clients retry on 503?" (6 messages, ~2.2k tokens). Full exchange: recall({"entryId":"…"}).
+[context-guard] Removed 1 image from this output (1280×960; ~1.6k tokens), 4 turns old. recall({"entryId":"…"}) shows it again.
+```
 
-The token figure is Pi's usual estimate, characters ÷ 4. It counts what is kept out of the model's context
-**on the current branch right now**: original length minus replacement length, for every distilled result
-that is still in the context. It goes down after `/tree` to a point before an edit, and after compaction
-removes distilled results. It is not a measured token count and not a cumulative total.
+## Results
 
-## `/guard` command
+Context size of the next request, with the guard on and off:
 
-| Command | Effect |
-|---|---|
-| `/guard` or `/guard status` | Shows whether the guard is on, the model, the savings on this branch, Jev runs, requests and cost, and the last run's result for each tool output, including Jev traces such as `kw.15 focus=chunk_4:.52 keep 9/29` |
-| `/guard on` / `/guard off` | Turns the guard on or off and saves `enabled` to the settings file |
-| `/guard reload` | Re-reads the settings file and shows any warnings again |
+| Task | Model | On | Off |
+|---|---|---|---|
+| Code search (50 KB `rg`), then a follow-up question | Claude Sonnet 5.5 | 15.5k | 28.2k |
+| The same | GPT-6 Luna | 11.5k | 34.3k |
+| Reading a 164 KB file, then a question about a detail | GPT-6 Luna | 9.2k | 35.3k |
+| Failing test log, then "now fix it" | GPT-6 Luna | 5.7k | 7.5k |
+| Autonomous run fixing 10 bugs (peak over the run) | GPT-6 Luna | 45k | 68–88k |
+| Six screenshots, then a question about one of them | Claude Sonnet 5.5 | 9.4k | ~14.1k |
 
-Settings are read at session start and on `/guard reload`.
+Every follow-up was answered correctly, and every task was finished. The screenshot row's "off" value is
+the request before the images were removed. Jev costs $0.0003–0.007 and takes 0.6–2 s per pass. These are
+single runs, not a benchmark. Details: [docs/RESULTS.md](docs/RESULTS.md).
 
-## `recall` tool
+## Use
 
-Every distilled result starts with `[context-guard]` and ends its header with
-`Full output: recall({"entryId":"…"}).` The model can call:
+It works on its own; there is nothing to call.
 
-| Parameter | Meaning |
-|---|---|
-| `entryId` (required) | The entry id from the header |
-| `pattern` | JavaScript regex (≤ 500 chars, 1 s limit); returns only matching lines, prefixed with their line numbers |
-| `offset` | First line to return, 1-based. When `pattern` is set, this counts matching lines. |
-| `limit` | Most lines to return |
+- **Footer:** `🛡 −4.2k · 3` means about 4.2k tokens are kept out of context right now, by 3 edits. After a
+  compaction, `🛡 0 · Σ−79k` shows the total saved in the session.
+- **`/guard`:** shows every decision of the last pass, with Jev's probabilities. `/guard off` and
+  `/guard on` are saved for later sessions. `/guard reload` re-reads the settings.
+- **`recall`:** the model calls it when it needs something removed: the whole original, lines that match a
+  regex, or a line range. You can also ask for it ("recall the full test log").
+- **Undo:** use `/tree` to go back to before an edit; the original is still in the session file.
 
-Like Pi's `read` tool, the output is capped at 2,000 lines and 50 KB, and a note says how to continue.
+## Configure
 
-## Configuration
+Optional. Put the settings in `~/.pi/agent/context-guard.json`:
 
-The settings file is `~/.pi/agent/context-guard.json` (inside Pi's agent directory), or the path in
-`PI_CONTEXT_GUARD_CONFIG`. Every key is optional. Values that are missing, the wrong type or out of range
-fall back to the default.
+```json
+{
+  "excludeTools": ["edit", "write", "bash"],
+  "imageKeepTurns": 5,
+  "historyExchanges": 0
+}
+```
 
 | Key | Default | Meaning |
 |---|---|---|
 | `enabled` | `true` | Master switch (`/guard on` and `/guard off` write it) |
-| `model` | `"openrouter/~typesafe/jev-latest"` | Classifier as `provider/id`; the first `/` separates the provider |
-| `minResultChars` | `4000` | Ignore tool results shorter than this many characters |
-| `minRunChars` | `8000` | Skip the run if its candidates add up to less than this |
-| `keepWholeThreshold` | `0.7` | Keep a segment untouched if P(`keep_whole`) is at least this |
-| `focusWholeThreshold` | `0.6` | Keep a segment untouched if `focus` picks "whole" with at least this probability |
-| `chunkKeepThreshold` | `0.6` | Keep a chunk if P(chunk needed) is at least this |
-| `noneThreshold` | `0.5` | Remove a segment entirely if `focus` picks "none" with at least this probability and no chunk passes |
-| `maxKeepRatio` | `0.6` | Skip the edit if the kept chunks are more than this fraction of the original |
-| `keepCitedFiles` | `false` | Also keep grep chunks for files that the final answer names (see DESIGN.md for why it is off) |
-| `timeoutMs` | `8000` | Total time budget for all Jev requests of one run; unfinished parts are kept |
-| `concurrency` | `6` | Jev requests in parallel |
-| `maxSegmentChars` | `32000` | Most characters of one output per Jev request (keeps the worst-case request under Jev's 32k-token context, see docs/JEV.md) |
-| `maxChunksPerSegment` | `40` | Most chunks per Jev request |
-| `excludeTools` | `["edit", "write"]` | Tools whose results are never distilled |
-| `distillErrors` | `true` | Also distill error results, such as the log of a failing test or build; Jev is told the call failed, and that its details are usually no longer needed once the agent got past it |
-| `historyExchanges` | `3` | Earlier exchanges (prompt + last assistant text) sent to Jev so it judges relevance against the ongoing work; `0` sends no earlier conversation at all (integer 0–10) |
-| `pinAnthropicCache` | `true` | Add an Anthropic cache breakpoint at the previous user question ([docs/CACHE.md](docs/CACHE.md)) |
-
-Example:
-
-```json
-{ "minResultChars": 6000, "excludeTools": ["edit", "write", "bash"] }
-```
+| `excludeTools` | `["edit", "write"]` | Tools whose output is never touched |
+| `imageKeepTurns` | `3` | Remove images from tool results after this many turns; `0` turns this off |
+| `pruneExchanges` | `true` | Omit earlier exchanges about other tasks |
+| `midRun` | `true` | Also run during long runs, not only at the end |
+| `historyExchanges` | `3` | Earlier exchanges sent to Jev as context; `0` sends none |
+| `model` | `openrouter/~typesafe/jev-latest` | Classifier model |
 
-## What it does not touch
-
-- user and assistant messages, custom messages
-- tool results that contain images
-- error results, if `distillErrors` is `false`
-- results of excluded tools (`edit`, `write` by default)
-- results shorter than `minResultChars`, and runs below `minRunChars`
-- aborted or failed runs
-- results that are already distilled, or that another extension already edited
-- single lines too long for one Jev request (kept as they are)
-- earlier runs: only the run that just finished is edited
-
-## Limitations
-
-- **Lossy by design.** Relevance is judged against the run's question and answer and the earlier
-  conversation (the ongoing task), so a follow-up that goes somewhere new may need something that was
-  removed. The header and omission lines say what is missing, and `recall` returns it.
-- **Latency.** Settling waits for Jev. This adds about 0.5–1.2 s after each qualifying run (measured), and
-  never more than `timeoutMs` (8 s).
-- **Cost.** Each qualifying run costs a few Jev requests, measured at under $0.001 per run. The first prompt
-  after an edit also has a one-time partial prompt-cache miss ([docs/CACHE.md](docs/CACHE.md)).
-- **Steering.** A message sent during a run starts a new span. Tool results from before it are never
-  distilled.
-
-## Examples (measured)
-
-Real runs on public code with Pi 1.0.3: each example ran once with the guard on and once with it off
-(`--no-guard`), from the same prompts. These are single runs, not a benchmark: the main model takes a
-different path each time. Reproduce with `node test/e2e/examples.mjs` ([docs/TESTING.md](docs/TESTING.md)).
-
-**A. Searching a real codebase.** In a copy of the published Pi package (`dist/` without source maps and
-the minified bundle, plus `docs/`), the agent was asked where Pi decides to auto-compact, starting with
-`rg -n -i compact dist docs`. Pi's bash tool returned the last 50 KB of that search. Jev kept the matches
-in the compaction module, the settings and the decision points in `agent-session.js`, and dropped matches
-in the grep tool, HTML export templates, the SDK and so on. Then the agent was asked "which of those
-settings can a user change, and what are their defaults?".
-- Claude Sonnet 5.5: 50,413 → 16,210 chars (141 of 431 lines kept). The first turn-2 request carried
-  **15.5k tokens instead of 28.2k**. The answer was correct (`enabled` true, `reserveTokens` 16384,
-  `keepRecentTokens` 20000), without `recall`.
-- GPT-6 Luna: 51,216 → 13,023 chars (107 of 385 lines), and a 5.4k read of `compaction.js` → 1.8k; two
-  smaller outputs were left whole. Turn 2 carried **11.5k tokens instead of 34.3k** and was correct, without
-  `recall`.
-
-**C. Reading a 164 KB source file.** The agent read `dist/core/agent-session.js` (3,535 lines) in
-seven 500-line pieces and explained how a prompt reaches the model. Jev kept 45–164 lines of each of the first
-four pieces and removed the last three completely (each became a one-line stub). The follow-up asked for a
-detail the explanation never mentioned: the exact error thrown when a prompt arrives during compaction.
-That line was among the kept ones, so the agent quoted it exactly without `recall`. The turn-2 request
-carried **9.2k tokens instead of 35.3k**.
-
-**B. A failing test run.** In a copy of this repository with one injected off-by-one bug, the agent ran
-the unit tests (a 15.8k-character log, 3 failures among 116 tests) and reported the cause. Jev kept the
-3 failures with their assertion messages and the failing-test summary, and dropped the passing tests:
-15.8k → 8.6k chars. Turn 2 ("now fix it") carried **5.7k tokens instead of 7.5k**, and both runs fixed
-the bug. (In the first run of this example, failing commands were never distilled; `distillErrors` was
-added because of it.)
-
-What the model sees after distilling (example A, Claude; the first lines of the real replacement):
+All settings: [docs/REFERENCE.md](docs/REFERENCE.md#configuration).
 
-```text
-[context-guard] Distilled the output of bash `rg -n -i compact dist docs`: kept 141 of 431 lines (15.1k of 50.4k chars); the rest was judged not needed for the answer. Full output: recall({"entryId":"1704787f"}).
-[… 79 lines omitted: matches in dist/core/tools/grep.js, dist/core/session-manager.js, dist/core/export-html/template.js, dist/core/export-html/template.css, dist/core/sdk.js and 5 more …]
-dist/core/compaction/compaction.d.ts:25:    /** Extension-specific data (e.g., ArtifactIndex, version markers for structured compaction) */
-dist/core/compaction/compaction.d.ts:28:export interface CompactionSettings {
-dist/core/compaction/compaction.d.ts:33:export declare const DEFAULT_COMPACTION_SETTINGS: CompactionSettings;
-dist/core/compaction/compaction.d.ts:54:/** Estimate projected context without trusting usage captured before a later edit or compaction. */
-dist/core/compaction/compaction.d.ts:57: * Check if compaction should trigger based on context usage.
-dist/core/compaction/compaction.d.ts:59:export declare function shouldCompact(contextTokens: number, contextWindow: number, settings: CompactionSettings): boolean;
-[… 44 lines omitted: matches in dist/core/compaction/compaction.d.ts, dist/core/compaction/branch-summarization.d.ts, dist/core/compaction/index.d.ts, dist/core/session-manager.d.ts, dist/core/cache-stats.js and 2 more …]
-```
-
-| Example | Model | Tool output before → after | Jev | Next request's context, on / off | Main-model cost of turn 2, on / off | Follow-up |
-|---|---|---|---|---|---|---|
-| A. code search | Claude Sonnet 5.5 | 50.4k → 16.2k chars | 3 requests, 0.8 s, $0.0012 | 15.5k / 28.2k tokens | $0.033 / $0.016 | correct, no `recall` |
-| A. code search | GPT-6 Luna | 56.6k → 14.8k chars (2 outputs; 2 more, 15k, left whole) | 5 requests, 0.6 s, $0.0015 | 11.5k / 34.3k tokens | $0.0012 / $0.0005 | correct, no `recall` |
-| C. 164 KB file | GPT-6 Luna | 163k → 26k chars (7 reads) | 7 requests, 0.9 s, $0.0027 | 9.2k / 35.3k tokens | $0.0007 / $0.0005 | correct, no `recall` |
-| B. failing test log | GPT-6 Luna | 15.8k → 8.6k chars | 1 request, 0.8 s, $0.0004 | 5.7k / 7.5k tokens | $0.0022 / $0.0016 | fixed in both runs |
-
-**When it doesn't help.**
+## Privacy
 
-- **The first request after an edit costs more, not less.** The prompt cache only matches up to the
-  first changed message, so the next request writes the shorter history again instead of reading the long
-  one from cache. In A that made Claude's turn-2 request cost $0.033 instead of $0.016, and GPT's $0.0012
-  instead of $0.0005; OpenAI read nothing from cache on that request (in the fixture runs it read 2,560
-  tokens instead of 13,824: a partial miss).
-  At the per-token prices in these runs, the smaller context earns that back after about 7 more requests
-  (Claude, A), about 3 (GPT, A) or about 1 (GPT, C). Before that, the benefit is a smaller context window,
-  not a lower bill. Anthropic requests also get a cache breakpoint at the previous question, which raised
-  the first request's cache read from 1,693 to 5,810 tokens in a long run ([docs/CACHE.md](docs/CACHE.md)).
-- **Small or fully needed outputs are left whole.** In A, GPT's 9k-character read of `agent-session.js` was
-  `not-worth` (too much of it was needed) and a 6k narrowed search was judged needed whole.
+Jev runs on OpenRouter and TypeSafe. Each pass sends them your prompt, the agent's answer or progress
+notes, short excerpts of earlier messages, and the text of the tool outputs being judged. That text can
+include source code and logs. Images, `edit` and `write` results, and outputs under 400 chars are never
+sent. Turn the extension off with `/guard off`. The full list of what is sent:
+[docs/REFERENCE.md](docs/REFERENCE.md#data-sent-to-jev).
 
-Earlier numbers on the generated StatusBadge test repo (96% removed, `recall` of a removed line) are in
-[docs/TESTING.md](docs/TESTING.md#results-so-far-pi-103).
+## Trade-offs
 
-## Documentation
+- **The first request after an edit costs more.** The prompt cache has to be rewritten from the edit
+  onward. In short sessions you gain context room, not a lower bill.
+- **Lossy by design.** A follow-up that goes somewhere new may need `recall` (the model sees what was
+  removed and how to get it back).
+- **It adds 0.5–2 s** after a run, or mid-run while a checkpoint runs (at most 8 s).
 
-- [docs/DESIGN.md](docs/DESIGN.md): architecture and algorithm
-- [docs/CACHE.md](docs/CACHE.md): prompt-cache impact (Anthropic, OpenAI) and the cost model
-- [docs/JEV.md](docs/JEV.md): the Jev questions, probe results and threshold choices
-- [docs/TESTING.md](docs/TESTING.md): unit tests, live end-to-end tests, manual TUI check
+## More
 
-## License
+- [docs/REFERENCE.md](docs/REFERENCE.md): every setting, status, command and `recall` parameter
+- [docs/RESULTS.md](docs/RESULTS.md): measured runs, including where it does not help
+- [docs/DESIGN.md](docs/DESIGN.md): how it works
+- [docs/CACHE.md](docs/CACHE.md): prompt-cache costs on Anthropic and OpenAI
+- [docs/JEV.md](docs/JEV.md): the Jev questions and threshold choices
+- [docs/TESTING.md](docs/TESTING.md): unit and live tests
 
-MIT
+MIT license.

package/docs/CACHE.md

@@ -21,7 +21,8 @@ runs, not averages.
   Consecutive `tool_use` blocks count as one position, and so do consecutive `tool_result` blocks.
 - Entries with a longer TTL must come before entries with a shorter TTL.
 - Reading an entry refreshes its TTL. The docs do **not** say that reading a longer prefix refreshes a
-  shorter entry that was written separately.
+  shorter entry that was written separately; measured below, it does
+  ([Mid-run checkpoints](#mid-run-checkpoints)).
 - Pricing relative to base input: 5-minute write 1.25×, 1-hour write 2×, read 0.1× (0.05× on Opus 5.5).
 - The minimum cacheable prompt is 512 tokens on Sonnet/Opus 5.5.
 
@@ -97,8 +98,10 @@ had more than 4 breakpoints.
 
 Over WebSocket, Pi's `openai-codex` transport sends only the new input items with `previous_response_id`, as
 long as the input only grows. After an edit, the input is no longer an extension of the previous one, so
-Pi sends the full input. The server-side continuation is lost for that request, and the prompt cache is reused
-only up to the first edited item. This happens once, on the first request after the edit.
+Pi sends the full input. The server-side continuation is lost for that request, and in every measurement the
+cache read was at most the 2,560-token stable prefix, **wherever the first edited item was** (also when it
+was 18 turns into a run; see [Mid-run checkpoints](#mid-run-checkpoints)). This happens once, on the first
+request after the edit.
 
 ### Measured
 
@@ -112,6 +115,139 @@ GPT-6 Luna (openai-codex), badge scenario, turn 2 first request:
 
 The pin does not apply to OpenAI.
 
+## Mid-run checkpoints
+
+A checkpoint edits outputs from earlier in the *current* run, so the next request cannot read the run from
+cache beyond the first edited output. Two questions decide the design: does a cache entry at the
+unchanged prefix still exist when the checkpoint comes, possibly many minutes after it was written; and
+how much does the one-time rewrite cost against the per-request saving.
+
+### Do cache entries on the read path stay alive? (measured)
+
+`node test/e2e/ttl-probe.mjs` (Claude Sonnet 5.5, `pi-claude-auth`). The model `cat`s a ~31k-token file
+(entry A ends at that result, written once as Pi's rolling breakpoint), then makes further requests; at the
+end, the request after A is changed so that only A can be read.
+
+| Arm | Between writing A and the test request | Test request |
+|---|---|---|
+| refresh | 8 requests in 7.6 min, each marking A (5m) and reading the longer prefix | cacheRead **31,001** (A alive) |
+| control | the same 8 requests, A **not** marked again | cacheRead **30,999** (A alive) |
+| ttl1h | as refresh, A at `ttl: "1h"` | cacheRead 31,001; but switching the earlier breakpoints to 1h was itself a full miss (cacheRead 0 → 31,001 written at the 1h price) |
+| idle (5m) | **no request for 362 s** (Pi's cache warming stopped) | cacheRead **1,431**, cacheWrite 29,633 (A expired) |
+| idle1h | no request for 365 s, A at 1h | cacheRead 30,977 (A alive) |
+
+So the 5-minute TTL is real (idle), but **every request that reads a longer prefix keeps the shorter entries
+on its path alive** (control). During an active run (requests seconds apart, and Pi's own cache warming,
+setting `cacheWarming: "streaming"` by default, during long tool calls), every turn boundary that was once
+the rolling breakpoint stays readable. A 1-hour TTL is not needed and would cost a full rewrite at 2× once.
+The extension therefore keeps Pi's TTL (5m by default). Pi's warming requests go through
+`before_provider_request` (seen in the capture), so they carry the extension's breakpoints too.
+
+What the lookback limit still breaks: the request after an edit has its rolling breakpoint at the end; the
+first edited output is usually many more than 20 positions back. So every Anthropic request keeps the
+previous-question pin (the floor: system prompt and question; with `pi-claude-auth`, the relocated system
+prompt sits in that same user message), and the first request after any edits (checkpoint, run end, or
+another extension; the `context_edit` entries after the last answered assistant message) gets two more
+breakpoints (`cacheAnchors` in `src/checkpoint.ts`):
+
+- a **read point**, only at a cache entry this process saw itself write: a `tool_result` block that carried
+  a breakpoint in a request it sent, to the same model, still on the branch, and with no `context_edit` at
+  or before it and no compaction or branch summary since, written or read through within the last 5
+  minutes. A response refreshes an entry when its `cacheRead` shows the request read clearly past it: at
+  least the run's baseline (the full input of the run's first request) plus the characters from the
+  question to the entry divided by 1.5, which overestimates the tokens (code ≈ 2.3 characters per token).
+  This follows the control arm above. The latest trusted entry before the first edited entry is used. The
+  log lives in memory and is empty after a reload, so a read point is never inferred;
+- a **write anchor** at the end of the first edited tool-result batch, so the edited prefix gets an entry
+  that the next checkpoint can read. It enters the log only if it was really in the request.
+
+Budget: Pi uses 3 (API key: tools, system, rolling) or 4 breakpoints (OAuth: plus the identity
+`system[0]`; `pi-claude-auth`: tools, `system[1]`, question, rolling). To make room, the identity breakpoint
+(covered by `system[1]`) and then the tools breakpoint (covered by the system breakpoint) are dropped. If
+that is not enough, the write anchor goes first, then the read point; the question stays. Marks use the
+rolling breakpoint's TTL and are skipped if they would break the TTL order. Only breakpoints actually left in
+the payload are logged, so an anchor dropped for the budget is never treated as written.
+
+### The first request after a checkpoint (measured)
+
+Claude Sonnet 5.5, long scenario with settings that force checkpoints into a short run
+(`midRunBreakEven: false`, `midRunMinAgeTurns: 2`, `midRunBatchChars: 20000`), with `pi-claude-auth` and
+`test/e2e/plain-test-runs.ts`:
+
+| Configuration | Checkpoint | First edited output | Next request |
+|---|---|---|---|
+| default anchors | 1st (after request 3) | the run's first output | cacheRead **5,832** (system prompt + question, via the question pin), cacheWrite 10,867 |
+| default anchors | 2nd (after request 8) | 3 turns into the run | cacheRead **16,699** (read at the entry written after the 1st checkpoint), cacheWrite 22,778 |
+| default anchors | 3rd (after request 10) | 5 turns into the run | cacheRead **16,699**, cacheWrite 20,008 |
+| `pinAnthropicCache: false` | 1st | the run's first output | cacheRead 5,832, cacheWrite 29,570 |
+| `pinAnthropicCache: false` | 2nd, 3rd, 4th | 2–6 turns into the run | cacheRead **5,832** each time, cacheWrite 41,162 / 32,183 / 33,872 |
+
+Without the guard's breakpoints, every request after a checkpoint read only the system-and-question entry
+and rewrote the whole run. With them, the later checkpoints read up to the entry written by the first
+request after the first checkpoint. The first checkpoint of a run usually edits the run's first output, so
+only the system prompt and question can be read then. When a read point is placed, the write anchor often
+does not fit the budget (question, read point, `system[1]` and rolling take all four slots).
+
+GPT-6 Luna (openai-codex), long scenario, 8 checkpoints in 5 runs (default batch and age settings), at
+requests 13–30: the first request after every checkpoint read **2,560 or 0** tokens from cache (17k–29k
+uncached), against 35k–72k read on the request before it. That is the transport, not the position of the
+edit.
+
+### Economics and the break-even rule
+
+At the prices in these runs (Sonnet 5.5: input $2/M, 5m write $2.5/M, read $0.2/M; GPT-6 Luna: input
+$0.10/M, cached $0.01/M), a checkpoint costs once, and saves on every later request:
+
+- OpenAI Codex: the whole context is sent uncached once: extra ≈ 0.9 × context × input price.
+- Anthropic: everything after the read point is written again: extra ≈ (1.25 − 0.1) × rewritten tokens.
+- Saving per later request: the removed tokens × the cache-read price. Jev removed 60–90% of checkpoint
+  outputs in the runs.
+
+Measured (GPT, a run with two checkpoints): checkpoint 1 removed 57k characters; the next request was 17,144 tokens uncached
+(≈ $0.0015 extra) and every later request carried about 19k tokens less (≈ $0.0002 saved each): break-even
+after about 8 requests, and the run went on for 12 more. Checkpoint 2 needed about 9 requests and the run
+ended 2 requests later. On Anthropic the ratio of write to read price is larger (12.5:1 vs 10:1).
+
+So checkpoints pay for themselves only when the run goes on for roughly 8–30 more requests. The
+**break-even rule** (`midRunBreakEven`, on) skips a checkpoint unless
+`pending chars × turns so far ≥ factor × rewrite chars` (OpenAI: factor 13, rewrite = whole context;
+Anthropic: factor 16, rewrite = context after the read point the next request would get, or after the
+question when no logged entry is trusted, or the whole context with pinning off; other providers: factor
+16, rewrite = context from the first eligible output), assuming the run goes on
+about as long as it has run so far. Context size is measured in text characters (`src/size.ts`): an image
+counts as 4 characters per token (Pi's chars ÷ 4; Anthropic: width × height / 750 tokens), not its base64 length, and a
+thinking signature counts at half its length. Counting the base64 once blocked every checkpoint in a long
+Claude run with screenshots (rewrite estimated at 2–22M characters for a 150–370k-token context).
+In the measured Claude runs (8–14 requests) it never allowed a
+checkpoint; in the GPT runs (26–37 requests) it allowed one or two. Turn it off to checkpoint purely for
+context room.
+
+## Old exchanges
+
+An omitted exchange comes before the current question, so the first request after it cannot read the
+question's cache entry: that prefix contains the edit. `cacheAnchors` handles this like any other edit: the
+read point is the latest trusted logged entry before the earliest edited entry, and the question pin is
+still placed (it writes a new entry for the requests that follow). Without such an entry, only the tools and
+system prompt are read. The break-even estimate (`anthropicRewriteChars`) follows the same rule: with an edit
+at or before the question and no trusted read point, the rewrite is the whole context.
+
+Measured (`topics` scenario, Claude Sonnet 5.5, first request of prompt 5 after the first exchange was
+omitted at the end of prompt 4 with `exchangeBreakEven: false`): cacheRead 1,760 (tools and system),
+cacheWrite 6,280, against cacheRead 12,376 and cacheWrite 438 with the guard off. With the default gate the
+omission was deferred and prompt 5 read 11,049 tokens from cache. On OpenAI Codex any edit makes the next
+request read at most the 2,560-token stable prefix (see above).
+
+So each omission costs one rewrite of everything after the omitted exchange on the next request (Pi's
+default `cacheWarming: "streaming"` keeps the cache warm between prompts that come within the TTL) and saves
+the exchange on every request after it. At run end, omissions therefore go through a break-even gate
+(`exchangeBreakEven`): their marginal rewrite (beyond what the pass's other edits rewrite anyway) must pay
+off over as many later requests as the session has had so far, and they must save at least
+`exchangeMinSavingChars` (8,000 characters); otherwise they wait and accumulate. On OpenAI Codex a pass that
+edits anything else already costs the whole context, so exchange omissions in that pass are free. Mid-run, exchanges join the checkpoint batch only when the batch with them pays off; their early
+position makes the rewrite most of the context, so with factor 16 the batch must reach about
+1/16 of the context per turn so far (for example, old exchanges that are half of the context pay off from
+about turn 32).
+
 ## Cost model
 
 All units are input-token equivalents, with a 5-minute TTL (write 1.25×, read 0.1×).