@hachej/boring-agent 0.4.5 → 0.4.7

package/README.md

@@ -91,7 +91,7 @@ The host supplies byte access and optional OCR; this package performs no file/ne
 
 Results use native `output` for the model and the same metadata in `details`. As in the reference, typed refusals are ordinary results carrying `error`, not `isError: true`. Document content is untrusted evidence, never instructions or consent. The readers set Pi's native output byte limit to 8 MiB to retain valid JSON at the extraction caps; the JSON occupies one native output line. These are returned-extraction limits, not bounds on input bytes or parser memory/CPU. Enforce input and resource budgets in the host.
 
-The regular dependencies are pinned `unpdf@1.9.1` (MIT) and the official SheetJS `xlsx@0.20.3` HTTPS tarball (Apache-2.0), not npm's old `xlsx@0.18.5`. Both are imported lazily inside their reader only. Their notices ship in `THIRD_PARTY_NOTICES.md`. `test/packages/agent-documents.test.mjs` drives public registrations through a real Harness with generated fictional files; no live OCR provider is qualified.
+PDF reading uses the pinned regular dependency `unpdf@1.9.1` (MIT). Spreadsheet reading needs the optional peer `xlsx` (SheetJS, Apache-2.0); install the official distribution in the host with `npm i https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz`, not npm's old `xlsx@0.18.5`. Without it, the spreadsheet tool returns a clear refusal; PDF reading still works. Both packages are imported lazily inside their reader only. Their notices ship in `THIRD_PARTY_NOTICES.md`. `test/packages/agent-documents.test.mjs` drives public registrations through a real Harness with generated fictional files, and `npm run test:question-consumer` exercises the packed agent without SheetJS; no live OCR provider is qualified.
 
 ## Approval gate
 
@@ -114,13 +114,13 @@ await conversation.configure({ extensions: { add: [memory.extension] } }, contex
 await conversation.configure({ extensions: { remove: [memory.extension] } }, context); // off again
 ```
 
-Pi applies an extension's hooks only in conversations that select it, so memory is opt-in per conversation and there is no separate mode. A host that selects extensions by default selects it everywhere; give conversations an explicit `extensions` list to keep it opt-in. `createOptChatMemory(options)` returns `{ extension, nap(id), stats(id), zoom(id, leafId, n), date(id, leafId), forgetMessage(id, messageIndex), forget(id), dispose() }`. Options: `harness`, `context` (long-lived, for background work), `summarizer` (required: `{ model }`, where `model` may be a function `(conversationId) => ModelRef`, sync or async, for example that conversation's own model, and `models` defaults to the Harness's; or `{ summarize(job, signal) }` returning one line), `nodeBytes` (512), `viewBytes` (128000), `compactBytes` (32000, the compactor's separate view), `notes` (seed strings, stored once per conversation before the transcript), `workOf(message)` (host classification of subagent or job reports), `jobs` (64, how many summary tasks of a conversation the host keeps live at once), `tries` (5), `cap` (30000), `retryMs` (10000 initially, doubling after each failure to at most 60000), `settleTimeoutMs` (60000), `agentName`, `name`, `onError`. `dispose()` only stops this process's waits: the summary tasks belong to Pi, which keeps and recovers them, so it is harmless to skip and safe to call before `harness.close`.
+Pi applies an extension's hooks only in conversations that select it, so memory is opt-in per conversation and there is no separate mode. A host that selects extensions by default selects it everywhere; give conversations an explicit `extensions` list to keep it opt-in. `createOptChatMemory(options)` returns `{ extension, nap(id), stats(id), zoom(id, leafId, n), date(id, leafId), forgetMessage(id, messageIndex), forget(id), dispose() }`. Options: `harness`, `context` (long-lived, for background work), `summarizer` (required: `{ model }`, where `model` may be a function `(conversationId) => ModelRef`, sync or async, for example that conversation's own model, and `models` defaults to the Harness's; or `{ summarize(job, signal) }` returning one line), `nodeBytes` (512), `viewBytes` (128000), `compactBytes` (32000, the cold-start/callback compactor view), `foldHighWater` (1.25) and `foldLowWater` (0.75), fractions of both view budgets, `notes` (seed strings, stored once per conversation before the transcript), `workOf(message)` (host classification of subagent or job reports), `jobs` (64, how many summary tasks of a conversation the host keeps live at once), `tries` (5), `cap` (30000), `retryMs` (10000 initially, doubling after each failure to at most 60000), `settleTimeoutMs` (60000), `agentName`, `name`, `onError`. `dispose()` only stops this process's waits: the summary tasks belong to Pi, which keeps and recovers them, so it is harmless to skip and safe to call before `harness.close`.
 
 | Reference | Here |
 | --- | --- |
 | Append-only log of user, talk, tool and echo messages | The Pi transcript, never rewritten. Leaves are derived from the model context: a user message is `user`, assistant text `talk`, each tool call `tool`, each tool result `echo`; thinking is not logged. A leaf over `cap` characters keeps head and tail. Seeded `note` leaves precede them; messages marked by `workOf` become one `work` leaf |
 | Summary tree: node (l, i) covers messages [i·2^l, (i+1)·2^l), at most `nodeBytes` | The same, purely binary. A short message, or two children that fit together, are their own text and cost no call |
-| The view tiles the whole chat, one `id+n\|text` line per node, fit incrementally by merging the most due pair (age / 2^(level+2)), never splitting | The same fold (`advanceFold`), kept per conversation and advanced as the log grows, so the start of the view stays the same between requests. `renderView` produces the `<chat>` block |
+| The view tiles the whole chat, one `id+n\|text` line per node, fit incrementally by merging the most due pair (age / 2^(level+2)), never splitting | The checkpoint fold (`advanceFold`): append between checkpoints; above 125% of the budget, merge deterministically down to 75%. Both fractions are configurable. The public four-argument core helper retains strict fitting. `renderView` produces the `<chat>` block |
 | Each turn calls the model with [system][view][new message] | `GenerationTask.beforeRequest` replaces the messages before the current turn with the view, as four-line text blocks at the start of the current turn's first message. The turn in progress starts at the latest user or work message. Work reports and sibling results whose tool calls fell outside the turn become ordinary user-role input, with their full text preserved. Every positional system message is kept in order. The explanation of the view is a prompt section, so the system prompt and tools never change between calls |
 | Compactor: context first, no ids, scale line, resubmit with the cut at the limit, `JOBS` in parallel, patient retry | The same protocol (`askModel`), run by native pi-durable tasks: each node `(level, index, fingerprint)` is one `boring.memory.optchat.node` task (shipped in the extension's `tasks`, input `{ key, level, index, of, upTo }`, one `summarize` phase) that calls the summarizer through `runtime.models` (or your `summarize`) and, in one commit, writes the node and the summarizer's usage into the extension's state. Pi schedules, runs and recovers them. See below |
 | A turn waits until every line of the view is a summary and never shows cut text | The request waits up to `settleTimeoutMs`; after that the missing lines show `(not summarized yet: zoom it)`, never a clipped message, and `zoom(id, 1)` returns the message whole |
@@ -134,14 +134,15 @@ What is native, what is host-side:
 - **Waiting.** A request waits with `waitForTask` on the live node tasks, up to `settleTimeoutMs`, then shows the placeholder. `nap` waits until all nodes are built, including retries. These waits are in-process; if the process dies the tasks are not lost, only the cascade to parents waits for the next trigger.
 - **Retry.** A failed node is a failed native task (`outcome.failed`). Its failure time and streak length live in `boring.memory.optchat.state` with call count and spend. The host's idle admission loop waits from `retryMs`, doubling to at most 60 seconds, then creates another native task without a new turn. `dispose()` aborts those waits, not Pi's tasks. After a process restart, `nap` or another trigger restarts admission. Requests retain their bounded wait and can show placeholders during an outage. `onError` reports only the first failure of a streak.
 - **Usage.** In the commit that stores a node (or records its failure) the task adds the summarizer's usage, all attempts of the cut-and-resubmit loop included, to the extension's own state document (`boring.memory.optchat.state`), reported by `stats().compactor.usage`. It is not written into Pi's `pi.usage`: Pi has no public way for a task to record model usage (`recordUsage` is internal), so per BORING-PI-4 that guarantee is unavailable here rather than worked around; a public usage-recording seam is an upstream request. Hosts that budget model spend add `stats().compactor.usage` to `harness.usage()`. Task and document kinds are all in `boring.memory.optchat.*`; this package writes no `pi.*` document.
-- **Still host-side.** The trigger and admission logic, the in-memory fold and log per process (a cache; fingerprints keep it honest), and the summarizer prompt. A custom `summarize` function has no model usage to record. A model call interrupted by a crash is rerun from the start of the task, and its spend is not recorded.
+- **Still host-side.** The trigger and admission logic, the log cache and the summarizer prompt. Context refreshes reuse a prior leaf only after exact kind, capped-text and timestamp comparison, avoiding repeated byte counting and hashing without trusting native message identity. The emitted turn and compactor tilings, open flags and content fingerprints are retained in `boring.memory.optchat.fold` (no duplicated message or summary text), so cache eviction and restart restore the same prefix with unchanged options and history. A sent interior placeholder stays frozen until a checkpoint (the tail may settle); later summaries remain accessible through zoom. History changes, forgetting or changed fold options rebuild the view. A custom `summarize` function has no model usage to record. A model call interrupted by a crash is rerun from the start of the task, and its spend is not recorded.
 
 Deviations, honestly:
 
 - The log is Pi's model context, not an OptChat file, so ids are positions in that context. A native compaction, reset or edit that changes earlier messages invalidates the summaries it touches (fingerprints), and the view is refolded. `beforeCompact` declines while the extension is selected: the view replaces compaction, and a native summary would cut the transcript the view is built from. That is the only effect on Pi's lifecycle.
 - The current turn is verbatim (including its tool steps) instead of being summarized to one line each as soon as it is logged; it enters the view when the next user message arrives. A user message sent mid-run starts a new "current turn" for the next step.
 - A request is rewritten once per attempt from the full context. The summaries are a cache rebuilt from the log; the summarizer's calls are not Pi generations: they run as Pi tasks and their usage is in `stats().compactor.usage`, not `pi.usage` (see above).
-- Both views use four-line text blocks with a stable prefix. The compactor folds its own `compactBytes` view of preceding messages, independent of the chat's `viewBytes`. Admission builds preceding parents before dispatching later requests whose context would exceed this budget. Pi 1.2's `Models.complete` accepts retention preferences, but `TextContent` exposes no per-block cache marker. Blocks are unmarked; provider-specific cache-hit guarantees are unavailable. There is no separate `fsync` log or single-writer lock; Pi's storage owns durability. Summaries are read from documents on every request and stats call.
+- Both views use four-line text blocks. Model compactor requests carry the entire last emitted turn view (bare lines), even when their job revisits older history; the admitted `upTo` still bounds the source they may summarize. Before the first turn, native documents retain one frozen, whole-block prefix within `compactBytes × foldHighWater`. Callback summarizers keep their past-only `compactBytes` context. The constant size example is in the summarizer system prompt rather than every job suffix. The first native task on a new conversation/model/prefix sends alone; other admitted tasks await its response, then send in parallel within the same `jobs` cap. Failure or cancellation releases the warmup wait. This is process-local request coordination, not a scheduler or proof a provider retained its cache.
+- `stats().compactor.requests` exposes provider-reported input, cache reads/writes, output and cost for the most recent 256 responses booked by settled native tasks, including retries, oldest first. The history limit is `COMPACTOR_REQUEST_HISTORY_LIMIT`, next to `DEFAULTS`; lifetime `calls` and aggregate `usage` totals are not truncated. These records survive restart; old state documents start with no request records. Interrupted tasks can lose uncommitted usage, as before. Pi 1.2 exposes no per-block cache marker: the fictional Anthropic replay supplies host breakpoints and measures cold starts plus 100 turns at 600/full-fold and 400/default messages. It enforces ≥95% cache reads, fewer uncached tokens and no extra summarizer calls; this is not live-provider qualification. Longer cached context increases total input. Qualify the host's chosen budgets, breakpoints and retention with its provider. There is no separate `fsync` log or single-writer lock; Pi's storage owns durability. Summaries are read from documents on every request and stats call.
 - Cost: the compactor makes about two calls per message, each carrying its small view as context. Choose a cheap summarizer model; the browser-agent example uses the chat model only because it has one.
 - The package runs in Node and in a browser (no Node-only imports). Fictional-data tests are in `test/packages/agent.test.mjs`; `examples/bot` and `examples/browser-agent` have journeys.
 
@@ -149,7 +150,7 @@ Deviations, honestly:
 
 `await memory.forgetMessage(conversationId, messageIndex)` forgets one message in Pi's current model context. `messageIndex` is its zero-based position in `(await conversation.context(context)).messages`, including system-message positions. System messages cannot be forgotten. This is distinct from the leaf id used by `zoom`; seeded notes and assistant tool calls can shift leaf ids.
 
-The operation replaces every leaf of that message with `(forgotten)` at the same positions. One native commit stores the tombstone and deletes those leaf summaries, their ancestors, later summaries and corresponding failure records. Later summaries can contain facts resolved from earlier context, so deleting only the logarithmic structural ancestors would leave copies behind. Rebuilding can therefore cost more than logarithmic work. Later leaves keep their ids and fingerprints. Native tasks rebuild the invalidated summaries. In-flight calls recheck the memory revision inside their commit and cannot restore old output. A chat request waiting on summaries also rechecks that revision before returning its rewritten messages. `nap` waits for rebuilding. Repeated forgetting is harmless. `forget(id)` only evicts the process cache and does not undo tombstones.
+The operation replaces every leaf of that message with `(forgotten)` at the same positions. One native commit stores the tombstone and deletes those leaf summaries, their ancestors, later summaries and corresponding failure records. Once a model summarizer has used the shared prefix, forgetting conservatively deletes all summaries: earlier nodes and transitive parents can also copy later facts. That fact is retained across restart and summarizer configuration changes. With only callback summaries, later summaries can contain facts resolved from earlier context, so deleting only the logarithmic structural ancestors would leave copies behind. Rebuilding can therefore cost more than logarithmic work. Later leaves keep their ids and fingerprints. Native tasks rebuild the invalidated summaries. In-flight calls recheck the memory revision inside their commit and cannot restore old output. A chat request waiting on summaries also rechecks that revision before returning its rewritten messages. `nap` waits for rebuilding. Repeated forgetting is harmless. `forget(id)` only evicts the process cache and does not undo tombstones.
 
 This changes active memory only, not requests already dispatched. Pi's session entries, native task history, underlying storage pages, provider logs, backups and copies quoted in other messages or existing forks are not erased. Indices address the current context, so a reset or history replacement requires the host to resolve positions again. Hosts must authorize the operation. Seed notes are stored once in `boring.memory.optchat.notes`; changing `options.notes` does not overwrite an existing conversation's notes.
 

package/dist/documents.js

@@ -147,7 +147,13 @@ async function readSpreadsheet(path, bytes, sheet, range) {
     const type = sniff(bytes, path);
     if (!SPREADSHEET_TYPES.includes(type))
         return refused(path, type, bytes.byteLength, 'Not a spreadsheet');
-    const XLSX = await import('xlsx');
+    let XLSX;
+    try {
+        XLSX = await import('xlsx');
+    }
+    catch {
+        return refused(path, type, bytes.byteLength, 'Spreadsheet reading needs the optional package xlsx (SheetJS); install it in the host');
+    }
     let book;
     try {
         book = type === TYPES.csv ? XLSX.read(textOf(bytes), { type: 'string', raw: true }) : XLSX.read(bytes, { type: 'array', cellDates: true });

package/dist/memory/optchat-core.d.ts

@@ -6,6 +6,9 @@ export declare const DEFAULTS: {
     readonly viewBytes: 128000;
     /** Independent view budget for summary requests. */
     readonly compactBytes: 32000;
+    /** Checkpoint thresholds, as fractions of each view budget. */
+    readonly foldHighWater: 1.25;
+    readonly foldLowWater: 0.75;
     /** Summarizer calls running at once. */
     readonly jobs: 64;
     /** Attempts per node to get a line under `nodeBytes`. */
@@ -17,6 +20,7 @@ export declare const DEFAULTS: {
     /** How long a request waits for the summaries it needs. */
     readonly settleTimeoutMs: 60000;
 };
+export declare const COMPACTOR_REQUEST_HISTORY_LIMIT = 256;
 /** What a view line shows for a message that has no summary yet. Only a fail-safe: requests wait for summaries. */
 export declare const PLACEHOLDER = "(not summarized yet: zoom it)";
 /** UTF-8 length without allocating. */
@@ -49,7 +53,8 @@ export interface LeafOptions {
     /** Non-system message ordinals mapped to original leaf counts. Positional system updates must not shift tombstones. */
     readonly forgotten?: Readonly<Record<string, number>>;
 }
-export declare function leavesOf(messages: readonly Message[], cap?: number, options?: LeafOptions): {
+/** Reuse prior leaves only after exact kind, capped-text and timestamp comparison; native contexts may contain fresh message objects. */
+export declare function leavesOf(messages: readonly Message[], cap?: number, options?: LeafOptions, previous?: readonly Leaf[]): {
     leaves: Leaf[];
     before: number[];
 };
@@ -96,14 +101,16 @@ export interface ViewFold {
     T: number;
     size: number;
     parts: Part[];
+    /** Leaves already sent: interior placeholders stay frozen until a checkpoint. */
+    stableThrough?: number;
+    /** A checkpoint waiting for its required summaries; its due priorities keep the original leaf count. */
+    checkpoint?: boolean;
+    checkpointAt?: number;
 }
 export declare const emptyFold: () => ViewFold;
 export declare const partStart: (part: Pick<Part, "level" | "index">) => number;
-/**
- * Append leaves up to `to` (one part each, fitting after every append) and refit with what has been built since. The view
- * only ever appends at its end and coarsens, so its start stays the same from one request to the next (§5.2).
- */
-export declare function advanceFold(fold: ViewFold, tree: OptChatTree, to: number, viewBytes: number): ViewFold;
+/** Append one leaf at a time; coarsen from the high mark to the low mark in a single checkpoint. */
+export declare function advanceFold(fold: ViewFold, tree: OptChatTree, to: number, viewBytes: number, highWater?: number, lowWater?: number): ViewFold;
 /** The view's parts that still wait for a summary. */
 export declare const openParts: (fold: ViewFold) => Part[];
 /** One line per part, `id+n|text`, newlines as spaces, wrapped in `<chat>` (§5.1). */

package/dist/memory/optchat-core.js

@@ -5,6 +5,9 @@ export const DEFAULTS = {
     viewBytes: 128_000,
     /** Independent view budget for summary requests. */
     compactBytes: 32_000,
+    /** Checkpoint thresholds, as fractions of each view budget. */
+    foldHighWater: 1.25,
+    foldLowWater: 0.75,
     /** Summarizer calls running at once. */
     jobs: 64,
     /** Attempts per node to get a line under `nodeBytes`. */
@@ -16,6 +19,7 @@ export const DEFAULTS = {
     /** How long a request waits for the summaries it needs. */
     settleTimeoutMs: 60_000,
 };
+export const COMPACTOR_REQUEST_HISTORY_LIMIT = 256;
 /** What a view line shows for a message that has no summary yet. Only a fail-safe: requests wait for summaries. */
 export const PLACEHOLDER = '(not summarized yet: zoom it)';
 /** UTF-8 length without allocating. */
@@ -72,13 +76,17 @@ export function capText(text, cap) {
     const head = Math.ceil(cap / 2), tail = Math.floor(cap / 2);
     return `${text.slice(0, head)}\n[… ${text.length - cap} characters cut …]\n${text.slice(text.length - tail)}`;
 }
-function leafOf(kind, text, at, cap) {
+function leafOf(kind, text, at, cap, previous) {
     const capped = capText(text, cap);
+    if (previous?.kind === kind && previous.text === capped && previous.at === at)
+        return previous;
     const line = `${kind}: ${capped}`;
     return { kind, text: capped, line, bytes: byteLength(line), hash: hashText(line), ...(at === undefined ? {} : { at }) };
 }
-export function leavesOf(messages, cap = DEFAULTS.cap, options = {}) {
-    const leaves = (options.notes ?? []).map(text => leafOf('note', text, undefined, cap)), before = [];
+/** Reuse prior leaves only after exact kind, capped-text and timestamp comparison; native contexts may contain fresh message objects. */
+export function leavesOf(messages, cap = DEFAULTS.cap, options = {}, previous = []) {
+    const leaves = (options.notes ?? []).map((text, index) => leafOf('note', text, undefined, cap, previous[index])), before = [];
+    const append = (kind, text, at) => leaves.push(leafOf(kind, text, at, cap, previous[leaves.length]));
     let messageIndex = 0;
     for (const message of messages) {
         before.push(leaves.length);
@@ -87,24 +95,24 @@ export function leavesOf(messages, cap = DEFAULTS.cap, options = {}) {
         const forgotten = options.forgotten?.[String(messageIndex++)];
         if (forgotten !== undefined) {
             for (let i = 0; i < forgotten; i++)
-                leaves.push(leafOf('note', '(forgotten)', undefined, cap));
+                append('note', '(forgotten)', undefined);
             continue;
         }
         const at = typeof message.timestamp === 'number' ? message.timestamp : undefined;
         if (options.workOf?.(message))
-            leaves.push(leafOf('work', textOf(message.content), at, cap));
+            append('work', textOf(message.content), at);
         else if (message.role === 'user')
-            leaves.push(leafOf('user', textOf(message.content), at, cap));
+            append('user', textOf(message.content), at);
         else if (message.role === 'assistant') {
             const talk = message.content.filter(part => part.type === 'text').map(part => part.text).join('\n');
             if (talk.trim())
-                leaves.push(leafOf('talk', talk, at, cap));
+                append('talk', talk, at);
             for (const part of message.content)
                 if (part.type === 'toolCall')
-                    leaves.push(leafOf('tool', `${part.name} ${JSON.stringify(part.arguments)}`, at, cap));
+                    append('tool', `${part.name} ${JSON.stringify(part.arguments)}`, at);
         }
         else if (message.role === 'toolResult')
-            leaves.push(leafOf('echo', `${message.toolName}${message.isError ? ' (error)' : ''}: ${textOf(message.content)}`, at, cap));
+            append('echo', `${message.toolName}${message.isError ? ' (error)' : ''}: ${textOf(message.content)}`, at);
     }
     before.push(leaves.length);
     return { leaves, before };
@@ -193,7 +201,7 @@ const label = (part) => `${partStart(part)}+${span(part.level)}`;
 const lineBytes = (part, text) => byteLength(label(part)) + 1 + byteLength(flat(text)) + 1;
 function settleOpen(fold, tree) {
     for (const part of fold.parts) {
-        if (!part.open)
+        if (!part.open || (!fold.checkpoint && partStart(part) < (fold.stableThrough ?? 0) && part !== fold.parts.at(-1)))
             continue;
         const text = tree.text(part.level, part.index);
         if (text === undefined)
@@ -204,37 +212,45 @@ function settleOpen(fold, tree) {
         part.open = false;
     }
 }
-/** Merge the most due adjacent pair until the view fits, never splitting; stop when no parent is built (§5.2). */
-function fit(fold, tree, viewBytes) {
-    while (fold.size > viewBytes) {
+/** Merge by due priority fixed at the checkpoint; never skip an unbuilt preferred parent. */
+function fit(fold, tree, high, low) {
+    if (fold.size > high && !fold.checkpoint) {
+        fold.checkpoint = true;
+        fold.checkpointAt = fold.T;
+    }
+    if (!fold.checkpoint)
+        return;
+    settleOpen(fold, tree);
+    while (fold.size > low) {
         let best = -1, bestDue = -Infinity;
         for (let p = 0; p + 1 < fold.parts.length; p++) {
             const a = fold.parts[p], b = fold.parts[p + 1];
             if (a.level !== b.level || a.index % 2 !== 0 || b.index !== a.index + 1)
                 continue;
-            if (tree.text(a.level + 1, a.index / 2) === undefined)
-                continue;
-            const due = (fold.T - partStart(a)) / 2 ** (a.level + 2);
+            const due = ((fold.checkpointAt ?? fold.T) - partStart(a)) / 2 ** (a.level + 2);
             if (due > bestDue) {
                 best = p;
                 bestDue = due;
             }
         }
         if (best < 0)
-            return;
+            break;
         const a = fold.parts[best], b = fold.parts[best + 1];
+        const text = tree.text(a.level + 1, a.index / 2);
+        if (text === undefined)
+            return;
         const merged = { level: a.level + 1, index: a.index / 2, bytes: 0, open: false };
-        merged.bytes = lineBytes(merged, tree.text(merged.level, merged.index));
+        merged.bytes = lineBytes(merged, text);
         fold.size += merged.bytes - a.bytes - b.bytes;
         fold.parts.splice(best, 2, merged);
     }
+    fold.checkpoint = false;
+    delete fold.checkpointAt;
 }
-/**
- * Append leaves up to `to` (one part each, fitting after every append) and refit with what has been built since. The view
- * only ever appends at its end and coarsens, so its start stays the same from one request to the next (§5.2).
- */
-export function advanceFold(fold, tree, to, viewBytes) {
+/** Append one leaf at a time; coarsen from the high mark to the low mark in a single checkpoint. */
+export function advanceFold(fold, tree, to, viewBytes, highWater = 1, lowWater = 1) {
     settleOpen(fold, tree);
+    fit(fold, tree, viewBytes * highWater, viewBytes * lowWater);
     while (fold.T < to) {
         const index = fold.T++;
         const text = tree.text(0, index);
@@ -242,22 +258,21 @@ export function advanceFold(fold, tree, to, viewBytes) {
         part.bytes = lineBytes(part, text ?? PLACEHOLDER);
         fold.parts.push(part);
         fold.size += part.bytes;
-        fit(fold, tree, viewBytes);
+        fit(fold, tree, viewBytes * highWater, viewBytes * lowWater);
     }
-    fit(fold, tree, viewBytes);
     return fold;
 }
 /** The view's parts that still wait for a summary. */
 export const openParts = (fold) => fold.parts.filter(part => part.open);
 /** One line per part, `id+n|text`, newlines as spaces, wrapped in `<chat>` (§5.1). */
 export function renderView(fold, tree) {
-    const lines = fold.parts.map(part => `${label(part)}|${flat(tree.text(part.level, part.index) ?? PLACEHOLDER)}`);
+    const lines = fold.parts.map(part => `${label(part)}|${flat(part.open ? PLACEHOLDER : tree.text(part.level, part.index) ?? PLACEHOLDER)}`);
     return `<chat>\n${lines.join('\n')}\n</chat>`;
 }
 /** The view's lines before `end` as bare text, one per line, with no ids: what a summarizer sees as context (§4.2). */
 export function bareLines(fold, tree, end) {
     return fold.parts.filter(part => partStart(part) + span(part.level) <= end)
-        .map(part => flat(tree.text(part.level, part.index) ?? PLACEHOLDER)).join('\n');
+        .map(part => flat(part.open ? PLACEHOLDER : tree.text(part.level, part.index) ?? PLACEHOLDER)).join('\n');
 }
 /** zoom(id, n): line `id+n` opened into its two children, or message `id` whole when n is 1 (§7.1). */
 export function zoomText(tree, id, n) {

package/dist/memory/optchat.d.ts

@@ -39,6 +39,8 @@ type Failure = {
 type StateDoc = {
     calls: number;
     usage: Ledger;
+    requests?: Ledger[];
+    sharedContext?: boolean;
     failures: Record<string, Failure>;
 };
 /** Compactor bookkeeping of one conversation: calls, spend and the last failure of each node, including its retry streak. */
@@ -81,8 +83,11 @@ export interface OptChatOptions {
     readonly agentName?: string;
     readonly nodeBytes?: number;
     readonly viewBytes?: number;
-    /** Budget of the compactor's separate, incrementally folded view. Default 32 000 UTF-8 bytes. */
+    /** Cold-start/callback context budget. Model jobs otherwise reuse the last turn view. Default 32 000 UTF-8 bytes. */
     readonly compactBytes?: number;
+    /** Checkpoint high/low marks as fractions of both view budgets. Defaults 1.25 / 0.75; set both to 1 for strict budgets. */
+    readonly foldHighWater?: number;
+    readonly foldLowWater?: number;
     /** Seed memories from the app brief or PM decisions. Captured at creation, stored once before this conversation's messages. */
     readonly notes?: readonly string[];
     /** Host classification of agent, subagent or job reports. A work message starts a turn like a user message. */
@@ -107,7 +112,7 @@ export interface OptChatStats {
     /** Stored summaries whose fingerprint matches the log. */
     readonly summaries: number;
     readonly view: {
-        /** Bytes of the view's lines (what `budget` limits), without the `<chat>` wrapper. */
+        /** Bytes of the view's lines, without the `<chat>` wrapper; checkpoints start above budget × foldHighWater. */
         readonly bytes: number;
         readonly budget: number;
         /** Lines the next request would show, oldest first. `id` and `n` as in `id+n`. */
@@ -132,6 +137,8 @@ export interface OptChatStats {
         readonly error?: string;
         /** Summarizer calls made for this conversation (from the state document, so it survives a restart). */
         readonly calls: number;
+        /** Latest 256 provider response usage records booked by settled tasks, including retries; retained across restart. Lifetime calls/usage are not truncated. Interrupted tasks and callback summarizers have no usage record. */
+        readonly requests: readonly Readonly<Ledger>[];
         /** Spend of those calls (kept by this extension; not in Pi's `pi.usage`). */
         readonly usage: {
             readonly input: number;
@@ -150,7 +157,7 @@ export interface OptChatStats {
         readonly logMessages: number;
         readonly viewBytes: number;
         readonly waitedMs: number;
-        /** False if the wait timed out and the view shows placeholders. */
+        /** False while the view still shows placeholders, including frozen ones from an earlier timeout. */
         readonly settled: boolean;
     };
 }
@@ -167,7 +174,7 @@ export interface OptChatMemory {
     date(conversationId: ConversationId, id: number): Promise<string>;
     /** Forget a Pi context message by its zero-based position, including system positions. Atomically tombstones its leaves and deletes summaries that could depend on them, including through context. Does not delete Pi's session entry. */
     forgetMessage(conversationId: ConversationId, messageIndex: number): Promise<void>;
-    /** Drop in-memory state of a conversation (stored summaries stay). */
+    /** Drop in-memory state of a conversation (stored summaries and the emitted tiling stay). */
     forget(conversationId: ConversationId): void;
     /** Stop this process's waits. Summary tasks belong to Pi, which keeps or recovers them; harmless to skip, and safe to call before closing the harness. */
     dispose(): Promise<void>;

package/dist/memory/optchat.js

@@ -10,7 +10,7 @@
 import { CompactionTask, GenerationTask, defineDoc, defineDocFamily, defineExtension, defineTask, defineTool, hook, section } from '@earendil-works/pi-durable';
 import { withAbortSignal, withCancel } from '@earendil-works/chord/context';
 import { Type } from '@earendil-works/pi-ai';
-import { DEFAULTS, OptChatLog, OptChatTree, advanceFold, bareLines, byteLength, compactPrompt, cutBytes, dateText, emptyFold, leavesOf, nodeKey, openParts, partStart, renderView, scaleLine, span, viewDoc, zoomText, } from './optchat-core.js';
+import { COMPACTOR_REQUEST_HISTORY_LIMIT, DEFAULTS, OptChatLog, OptChatTree, advanceFold, bareLines, byteLength, compactPrompt, cutBytes, dateText, emptyFold, leavesOf, nodeKey, openParts, partStart, renderView, scaleLine, span, viewDoc, zoomText, } from './optchat-core.js';
 export { DEFAULTS as OPTCHAT_DEFAULTS, PLACEHOLDER as OPTCHAT_PLACEHOLDER, compactPrompt, viewDoc } from './optchat-core.js';
 export { OptChatLog, OptChatTree, advanceFold, emptyFold, leavesOf, renderView, zoomText, dateText } from './optchat-core.js';
 /** Summaries of one conversation, in shards of 64 nodes per level (`<level>.<shard>`). A fork starts with the current ones; fingerprints reject any that no longer match. */
@@ -27,6 +27,11 @@ export const OptChatEdits = defineDoc({
     kind: 'boring.memory.optchat.edits', version: 1, scope: 'conversation', history: 'latest', fork: 'current',
     initial: () => ({ revision: 0, forgotten: {} }),
 });
+/** Only the emitted tiling and fingerprints are retained, never another copy of message or summary text. */
+const OptChatFold = defineDocFamily({
+    kind: 'boring.memory.optchat.fold', version: 1, family: true, scope: 'conversation', history: 'latest', fork: 'current',
+    initial: () => ({ configuration: '', revision: -1, fingerprints: [], fold: null }),
+});
 /** Compactor bookkeeping of one conversation: calls, spend and the last failure of each node, including its retry streak. */
 export const OptChatState = defineDoc({
     kind: 'boring.memory.optchat.state', version: 1, scope: 'conversation', history: 'latest', fork: 'initial',
@@ -59,6 +64,7 @@ class Session {
     }
 }
 const message = (text) => ({ role: 'user', content: text.map(part => ({ type: 'text', text: part })), timestamp: Date.now() });
+const copyFold = (fold) => ({ ...fold, parts: fold.parts.map(part => ({ ...part })) });
 const LIVE = ['pending', 'running', 'waiting'];
 function redacted(item) {
     if (item.role === 'system')
@@ -104,6 +110,12 @@ export function createOptChatMemory(options) {
     const nodeBytes = options.nodeBytes ?? DEFAULTS.nodeBytes, viewBytes = options.viewBytes ?? DEFAULTS.viewBytes, jobs = options.jobs ?? DEFAULTS.jobs;
     const tries = options.tries ?? DEFAULTS.tries, cap = options.cap ?? DEFAULTS.cap, retryMs = options.retryMs ?? DEFAULTS.retryMs, settleMs = options.settleTimeoutMs ?? DEFAULTS.settleTimeoutMs;
     const compactBytes = options.compactBytes ?? DEFAULTS.compactBytes;
+    const highWater = options.foldHighWater ?? DEFAULTS.foldHighWater, lowWater = options.foldLowWater ?? DEFAULTS.foldLowWater;
+    if (!Number.isFinite(highWater) || !Number.isFinite(lowWater) || lowWater <= 0 || lowWater > highWater)
+        throw new Error('Invalid OptChat fold watermarks');
+    const configuration = JSON.stringify([nodeBytes, viewBytes, compactBytes, highWater, lowWater, cap]);
+    const foldView = (fold, tree, to, budget) => advanceFold(fold, tree, to, budget, highWater, lowWater);
+    const missingParts = (fold, tree) => openParts(fold).filter(part => tree.text(part.level, part.index) === undefined);
     const agentName = options.agentName ?? 'OptChat';
     const extensionName = options.name ?? 'boring.memory.optchat';
     const taskKind = `${extensionName}.node`;
@@ -113,6 +125,8 @@ export function createOptChatMemory(options) {
     const logOptions = (seed, edits) => ({ notes: seed, forgotten: edits.forgotten, ...(options.workOf ? { workOf: options.workOf } : {}) });
     const sessions = new Map();
     const naps = new Map();
+    const warming = new Map();
+    const warmed = new Map();
     const retryDelay = (failure) => Math.min(60_000, retryMs * 2 ** Math.min(30, (failure.attempts ?? 1) - 1));
     const aborter = new AbortController();
     const hostContext = withAbortSignal(aborter.signal, options.context);
@@ -142,11 +156,28 @@ export function createOptChatMemory(options) {
             throw new Error(`No conversation ${String(id)}`);
         return found;
     }
-    /** The compactor has an independent fold. Copy before advancing: native tasks and requests can overlap. */
+    /** Earlier summary jobs take a prefix of the same tiling, never unfolding an already-sent checkpoint. */
     function compactView(s, to) {
-        const cached = session(s.id).compactFold;
-        const fold = cached.T > to ? emptyFold() : { T: cached.T, size: cached.size, parts: cached.parts.map(part => ({ ...part })) };
-        return advanceFold(fold, s.tree, to, compactBytes);
+        return foldView(copyFold(s.compactFold), s.tree, to, compactBytes);
+    }
+    async function restoreFold(s, revision, to, kind = 'turn') {
+        const saved = await options.harness().snapshot(OptChatFold, s.id, kind, options.context);
+        if (saved?.fold && saved.configuration === configuration && saved.revision === revision && saved.fold.T <= to
+            && saved.fingerprints.every((fp, i) => { const part = saved.fold.parts[i]; return fp === s.log.fp(part.level, part.index); }))
+            s[kind === 'compact' ? 'compactFold' : 'fold'] = copyFold(saved.fold);
+    }
+    async function saveFold(s, revision, kind, fold, context) {
+        await (await conversationOf(s.id)).commit(async (tx) => {
+            if ((await tx.doc(OptChatEdits, s.id)).revision !== revision)
+                return;
+            const doc = await tx.doc(OptChatFold, s.id, kind, null);
+            if (kind === 'compact' && doc.configuration === configuration && doc.revision === revision && doc.fold && doc.fold.T > fold.T)
+                return;
+            doc.configuration = configuration;
+            doc.revision = revision;
+            doc.fold = copyFold(fold);
+            doc.fingerprints = fold.parts.map(part => s.log.fp(part.level, part.index));
+        }, context);
     }
     /** Bring a session up to date with the committed context and its stored summaries. */
     async function refresh(id) {
@@ -154,8 +185,10 @@ export function createOptChatMemory(options) {
         const view = await (await conversationOf(id)).context(options.context);
         const seed = await seedNotes(id);
         const edits = await memoryEdits(id);
-        s.sync(leavesOf(view.messages, cap, logOptions(seed, edits)).leaves);
+        s.sync(leavesOf(view.messages, cap, logOptions(seed, edits), s.log.leaves).leaves);
         await loadStored(s, options.harness(), options.context);
+        await restoreFold(s, edits.revision, s.log.leaves.length);
+        await restoreFold(s, edits.revision, s.log.leaves.length, 'compact');
         return s;
     }
     async function memoryEdits(id) {
@@ -176,7 +209,7 @@ export function createOptChatMemory(options) {
     }
     // ---- The summarizer (runs inside the native task) ----------------------------------------------------------------------
     /** One node by the reference protocol (§4.2, §4.3): context first, then the step; over-long lines are resubmitted cut at the limit; the shortest wins. */
-    async function askModel(job, runtimeModels, signal) {
+    async function askModel(job, runtimeModels, signal, prefix = job.context) {
         const summarizer = options.summarizer;
         if ('summarize' in summarizer) {
             try {
@@ -194,20 +227,53 @@ export function createOptChatMemory(options) {
         if (!model)
             throw new Error(`Summarizer model ${ref.provider}/${ref.modelId} is not available`);
         const lines = job.source.map(line => line.replace(/\r?\n/g, ' '));
-        const step = `For scale, this line is exactly ${scale.bytes} bytes:\n${scale.text}\n\n` + (job.level === 0
+        const prompt = `${system}\n\nFor scale, this line is exactly ${scale.bytes} bytes:\n${scale.text}`;
+        const step = job.level === 0
             ? `Compress this message into one line, in at most ${nodeBytes} bytes:\n${job.source[0]}`
-            : `Merge these two lines into one, in at most ${nodeBytes} bytes:\n${lines.join('\n')}`);
-        const conversation = [message([...chatBlocks(job.context), step])];
+            : `Merge these two lines into one, in at most ${nodeBytes} bytes:\n${lines.join('\n')}`;
+        const conversation = [message([...chatBlocks(prefix), step])];
         const attempts = [];
+        const requests = [];
         let usage, calls = 0, spender;
+        const owner = `${String(job.conversationId)}:${ref.provider}/${ref.modelId}`;
+        const key = JSON.stringify([owner, prefix]);
+        let release;
         try {
+            // Tasks already occupy the host's admission cap. Only the first request on a new prefix warms it.
+            while (warmed.get(owner) !== prefix) {
+                const pending = warming.get(key);
+                if (!pending) {
+                    warming.set(key, new Promise(resolve => { release = resolve; }));
+                    break;
+                }
+                if (signal.aborted)
+                    throw signal.reason;
+                let abort;
+                try {
+                    await Promise.race([pending, new Promise((_resolve, reject) => {
+                            abort = () => reject(signal.reason);
+                            signal.addEventListener('abort', abort, { once: true });
+                        })]);
+                }
+                finally {
+                    if (abort)
+                        signal.removeEventListener('abort', abort);
+                }
+            }
             for (;;) {
-                const reply = await models.complete(model, { systemPrompt: system, messages: conversation }, { signal });
+                const reply = await models.complete(model, { systemPrompt: prompt, messages: conversation }, { signal });
                 calls++;
                 spender = `${reply.provider}/${reply.model}`;
                 usage = addUsage(usage, reply.usage);
+                requests.push({ input: reply.usage.input, output: reply.usage.output, cacheRead: reply.usage.cacheRead, cacheWrite: reply.usage.cacheWrite, cost: reply.usage.cost.total });
                 if (reply.stopReason === 'error' || reply.stopReason === 'aborted')
                     throw new Error(reply.errorMessage ?? `Summarizer stopped: ${reply.stopReason}`);
+                warmed.set(owner, prefix);
+                if (release) {
+                    warming.delete(key);
+                    release();
+                    release = undefined;
+                }
                 const line = reply.content.filter(part => part.type === 'text').map(part => part.text).join('\n').trim();
                 if (!line)
                     throw new Error('The summarizer returned nothing');
@@ -220,14 +286,24 @@ export function createOptChatMemory(options) {
         catch (error) {
             if (signal.aborted)
                 throw error;
-            return { error, ...(usage ? { usage } : {}), calls, ...(spender ? { spender } : {}) };
+            return { error, ...(usage ? { usage } : {}), calls, requests, ...(spender ? { spender } : {}) };
         }
-        return { text: attempts.reduce((best, line) => byteLength(line) < byteLength(best) ? line : best), ...(usage ? { usage } : {}), calls, ...(spender ? { spender } : {}) };
+        finally {
+            if (release) {
+                warming.delete(key);
+                release();
+            }
+        }
+        return { text: attempts.reduce((best, line) => byteLength(line) < byteLength(best) ? line : best), ...(usage ? { usage } : {}), calls, requests, ...(spender ? { spender } : {}) };
     }
     /** What one node task adds to the conversation's documents in its single commit: calls, spend and the node or its failure. */
     async function bookkeeping(tx, id, summary) {
         const state = await tx.doc(OptChatState, id);
         state.calls += summary.calls;
+        if (summary.requests?.length) {
+            state.requests = [...(state.requests ?? []), ...summary.requests].slice(-COMPACTOR_REQUEST_HISTORY_LIMIT);
+            state.sharedContext = true;
+        }
         if (summary.usage && summary.spender) {
             const u = summary.usage;
             state.usage.input += u.input;
@@ -254,27 +330,31 @@ export function createOptChatMemory(options) {
                 const load = async () => {
                     const seed = (await runtime.snapshot(OptChatNotes, id, context))?.notes ?? [];
                     const edits = await runtime.snapshot(OptChatEdits, id, context) ?? { revision: 0, forgotten: {} };
-                    s.sync(leavesOf((await runtime.context(id, context)).messages, cap, logOptions(seed, edits)).leaves);
+                    s.sync(leavesOf((await runtime.context(id, context)).messages, cap, logOptions(seed, edits), s.log.leaves.length ? s.log.leaves : session(id).log.leaves).leaves);
                     await loadStored(s, runtime, context);
+                    await restoreFold(s, edits.revision, s.log.leaves.length, 'compact');
                     return edits.revision;
                 };
                 const revision = await load();
                 const ready = () => s.log.fp(level, index) === of && (level === 0 || (s.tree.text(level - 1, 2 * index) !== undefined && s.tree.text(level - 1, 2 * index + 1) !== undefined));
-                if (!ready() || s.tree.text(level, index) !== undefined)
+                if ((index + 1) * span(level) > upTo || !ready() || s.tree.text(level, index) !== undefined)
                     return stale(); // the history changed, a child is missing, or the node needs no summary
                 const to = Math.min(upTo, index * span(level));
-                const shared = session(id);
-                shared.sync(s.log.leaves);
                 const fold = compactView(s, to);
-                if (fold.size > compactBytes)
+                if (fold.size > compactBytes * highWater || fold.checkpoint)
                     return stale(); // admission will build preceding parents before trying this node again
-                if (fold.T >= shared.compactFold.T)
-                    shared.compactFold = fold;
+                await saveFold(s, revision, 'compact', fold, context);
                 const source = level === 0 ? [s.log.leaves[index].line] : [s.tree.text(level - 1, 2 * index), s.tree.text(level - 1, 2 * index + 1)];
                 const job = { conversationId: id, level, index, range: [index * span(level), (index + 1) * span(level) - 1], source, context: bareLines(fold, s.tree, to), nodeBytes };
+                // Keep the callback's past-only context; model requests carry the last turn's full prefix, with the job after it.
+                await restoreFold(s, revision, s.log.leaves.length, 'bootstrap');
+                await restoreFold(s, revision, s.log.leaves.length);
+                const prefixFold = s.fold.stableThrough === undefined ? fold : s.fold;
+                await loadStored(s, runtime, context); // the selected turn may reference summaries committed since load()
+                const prefix = bareLines(prefixFold, s.tree, prefixFold.T);
                 let summary;
                 try {
-                    summary = await askModel(job, runtime.models, runtime.signal);
+                    summary = await askModel(job, runtime.models, runtime.signal, prefix);
                 }
                 catch (error) {
                     if (runtime.signal.aborted)
@@ -314,8 +394,8 @@ export function createOptChatMemory(options) {
     /** Nodes that could be built now: sources exist and the whole preceding view is summarized (§4.1), oldest first. */
     function candidates(s) {
         const upTo = Math.min(s.upTo, s.log.leaves.length);
-        advanceFold(s.fold, s.tree, upTo, viewBytes);
-        const open = openParts(s.fold)[0];
+        foldView(s.fold, s.tree, upTo, viewBytes);
+        const open = missingParts(s.fold, s.tree)[0];
         const first = open ? partStart(open) : s.fold.T;
         const found = [];
         for (let level = 0; span(level) <= upTo; level++) {
@@ -327,7 +407,8 @@ export function createOptChatMemory(options) {
                     continue;
                 if (level > 0 && (s.tree.text(level - 1, 2 * index) === undefined || s.tree.text(level - 1, 2 * index + 1) === undefined))
                     continue;
-                if (compactView(s, index * span(level)).size > compactBytes)
+                const preceding = compactView(s, index * span(level));
+                if (preceding.size > compactBytes * highWater || preceding.checkpoint)
                     break;
                 found.push({ level, index });
             }
@@ -344,6 +425,26 @@ export function createOptChatMemory(options) {
         const failures = (await options.harness().snapshot(OptChatState, s.id, options.context))?.failures ?? {};
         const now = Date.now(), upTo = Math.min(s.upTo, s.log.leaves.length);
         const conversation = await conversationOf(s.id);
+        // Before the first turn there is no sent view. Freeze one cold prefix for the seed's native tasks.
+        if ('model' in options.summarizer) {
+            const revision = (await memoryEdits(s.id)).revision;
+            const saved = await options.harness().snapshot(OptChatFold, s.id, 'bootstrap', options.context);
+            if (!saved?.fold || saved.revision !== revision || saved.configuration !== configuration
+                || !saved.fingerprints.every((fp, i) => fp === s.log.fp(saved.fold.parts[i].level, saved.fold.parts[i].index))) {
+                const prefix = emptyFold();
+                for (let i = 0; i + 4 <= s.fold.parts.length; i += 4) {
+                    const block = s.fold.parts.slice(i, i + 4), bytes = block.reduce((sum, part) => sum + part.bytes, 0);
+                    if (prefix.size + bytes > compactBytes * highWater)
+                        break;
+                    prefix.parts.push(...block.map(part => ({ ...part })));
+                    prefix.size += bytes;
+                    const last = block.at(-1);
+                    prefix.T = partStart(last) + span(last.level);
+                }
+                prefix.stableThrough = prefix.T;
+                await saveFold(s, revision, 'bootstrap', prefix, options.context);
+            }
+        }
         let waiting = 0, created = 0;
         const live = await conversation.commit(async (tx) => {
             const found = new Map();
@@ -432,10 +533,12 @@ export function createOptChatMemory(options) {
         const deadline = Date.now() + settleMs, skip = new Set();
         for (;;) {
             await loadStored(s, options.harness(), options.context);
-            advanceFold(s.fold, s.tree, covered, viewBytes);
-            const open = openParts(s.fold).length;
-            if (open === 0 && s.fold.size <= viewBytes)
-                return true;
+            if (s.fold.stableThrough === undefined)
+                s.fold = emptyFold();
+            foldView(s.fold, s.tree, covered, viewBytes);
+            const open = missingParts(s.fold, s.tree).length;
+            if (open === 0 && s.fold.size <= viewBytes * highWater && !s.fold.checkpoint)
+                return openParts(s.fold).length === 0;
             const { live } = await admit(s, skip);
             if (open === 0 && live.size === 0)
                 return true;
@@ -449,7 +552,7 @@ export function createOptChatMemory(options) {
                 skip.add(done.key);
         }
     }
-    /** Context can copy an earlier fact into a later summary. Invalidate those too, atomically with the tombstone. */
+    /** Context can copy facts across nodes. Shared-prefix model summaries may depend on any leaf, transitively. */
     async function forgetMessage(id, messageIndex) {
         if (!Number.isInteger(messageIndex) || messageIndex < 0)
             throw new Error('Invalid message index');
@@ -473,8 +576,10 @@ export function createOptChatMemory(options) {
                 doc.revision++;
                 const state = await tx.doc(OptChatState, id);
                 for (let level = 0; span(level) <= leaves.length; level++) {
-                    for (let index = Math.floor(start / span(level)); (index + 1) * span(level) <= leaves.length; index++) {
+                    for (let index = 0; (index + 1) * span(level) <= leaves.length; index++) {
                         const shard = await tx.doc(OptChatNodes, id, shardKey(level, index), 0);
+                        if (!state.sharedContext && index < Math.floor(start / span(level)))
+                            continue;
                         delete shard.nodes[String(index)];
                         delete state.failures[nodeKey(level, index)];
                     }
@@ -515,18 +620,17 @@ export function createOptChatMemory(options) {
                     for (;;) {
                         const seed = await seedNotes(api.conversationId);
                         const edits = await memoryEdits(api.conversationId);
-                        const { leaves, before } = leavesOf(request.messages, cap, logOptions(seed, edits));
+                        const cached = session(api.conversationId);
+                        const { leaves, before } = leavesOf(request.messages, cap, logOptions(seed, edits), cached.log.leaves);
                         const last = request.messages.findLastIndex(item => item.role === 'user' || (item.role !== 'system' && options.workOf?.(item)));
                         const covered = last < 0 ? 0 : before[last];
                         if (covered === 0 && Object.keys(edits.forgotten).length === 0 && !request.messages.some(item => options.workOf?.(item)))
                             return undefined;
-                        const cached = session(api.conversationId);
                         cached.sync(leaves);
                         const s = new Session(api.conversationId, nodeBytes);
                         s.sync(leaves);
                         s.upTo = covered;
-                        if (cached.fold.T <= covered)
-                            s.fold = { T: cached.fold.T, size: cached.fold.size, parts: cached.fold.parts.map(part => ({ ...part })) };
+                        await restoreFold(s, edits.revision, covered);
                         const started = Date.now();
                         const settled = await settle(s, covered, context);
                         if ((await memoryEdits(api.conversationId)).revision !== edits.revision)
@@ -556,6 +660,10 @@ export function createOptChatMemory(options) {
                             return [{ ...current, content: [...blocks, ...body] }];
                         });
                         cached.lastRequest = { at: new Date().toISOString(), covered, requestMessages: messages.length, logMessages: request.messages.length, viewBytes: byteLength(view), waitedMs: Date.now() - started, settled };
+                        s.fold.stableThrough = s.fold.T;
+                        await saveFold(s, edits.revision, 'turn', s.fold, context);
+                        if ((await memoryEdits(api.conversationId)).revision !== edits.revision)
+                            continue;
                         cached.fold = s.fold;
                         return { messages };
                     }
@@ -575,8 +683,8 @@ export function createOptChatMemory(options) {
         nap: id => nap(id),
         async stats(id) {
             const s = await refresh(id);
-            const fold = { T: s.fold.T, size: s.fold.size, parts: s.fold.parts.map(part => ({ ...part })) };
-            advanceFold(fold, s.tree, s.log.leaves.length, viewBytes);
+            const fold = copyFold(s.fold);
+            foldView(fold, s.tree, s.log.leaves.length, viewBytes);
             let summaries = 0;
             for (const [key, node] of s.stored) {
                 const [level, index] = key.split(':').map(Number);
@@ -591,15 +699,17 @@ export function createOptChatMemory(options) {
             return {
                 leaves: s.log.leaves.length, summaries,
                 view: { bytes: fold.size, budget: viewBytes, open: openParts(fold).length,
-                    parts: fold.parts.map(part => ({ id: partStart(part), n: span(part.level), level: part.level, text: s.tree.text(part.level, part.index) ?? '', open: part.open })) },
-                compactor: { busy, retrying, failed: failures.length, idle: busy === 0 && retrying === 0, ...(latest ? { error: latest.error } : {}), calls: state?.calls ?? 0, usage: { ...(state?.usage ?? { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, cost: 0 }) } },
+                    parts: fold.parts.map(part => ({ id: partStart(part), n: span(part.level), level: part.level, text: part.open ? '' : s.tree.text(part.level, part.index) ?? '', open: part.open })) },
+                compactor: { busy, retrying, failed: failures.length, idle: busy === 0 && retrying === 0, ...(latest ? { error: latest.error } : {}), calls: state?.calls ?? 0, requests: (state?.requests ?? []).slice(-COMPACTOR_REQUEST_HISTORY_LIMIT).map(request => ({ ...request })), usage: { ...(state?.usage ?? { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, cost: 0 }) } },
                 ...(s.lastRequest ? { lastRequest: s.lastRequest } : {}),
             };
         },
         async zoom(id, nth, n) { return zoomText((await refresh(id)).tree, nth, n); },
         async date(id, nth) { return dateText((await refresh(id)).log, nth); },
         forgetMessage,
-        forget(id) { sessions.delete(String(id)); },
+        forget(id) { sessions.delete(String(id)); for (const key of warmed.keys())
+            if (key.startsWith(`${String(id)}:`))
+                warmed.delete(key); },
         async dispose() { aborter.abort(); await Promise.allSettled(naps.values()); },
     };
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hachej/boring-agent",
-  "version": "0.4.5",
+  "version": "0.4.7",
   "license": "MIT",
   "private": false,
   "type": "module",
@@ -147,15 +147,18 @@
     }
   },
   "dependencies": {
-    "unpdf": "1.9.1",
+    "unpdf": "1.9.1"
+  },
+  "devDependencies": {
     "xlsx": "https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz"
   },
   "peerDependencies": {
     "@earendil-works/pi-durable": "~1.2.0",
     "@earendil-works/pi-ai": "~1.2.0",
     "@earendil-works/chord": "~1.2.0",
-    "@hachej/boring-files": "^0.4.5",
-    "@hachej/boring-ui-kit": "^0.4.5"
+    "@hachej/boring-files": "^0.4.7",
+    "@hachej/boring-ui-kit": "^0.4.7",
+    "xlsx": "^0.20.3"
   },
   "peerDependenciesMeta": {
     "@hachej/boring-files": {
@@ -163,6 +166,9 @@
     },
     "@hachej/boring-ui-kit": {
       "optional": true
+    },
+    "xlsx": {
+      "optional": true
     }
   },
   "engines": {