@hachej/boring-agent 0.4.4 → 0.4.6

package/README.md

@@ -114,13 +114,13 @@ await conversation.configure({ extensions: { add: [memory.extension] } }, contex
 await conversation.configure({ extensions: { remove: [memory.extension] } }, context); // off again
 ```
 
-Pi applies an extension's hooks only in conversations that select it, so memory is opt-in per conversation and there is no separate mode. A host that selects extensions by default selects it everywhere; give conversations an explicit `extensions` list to keep it opt-in. `createOptChatMemory(options)` returns `{ extension, nap(id), stats(id), zoom(id, leafId, n), date(id, leafId), forgetMessage(id, messageIndex), forget(id), dispose() }`. Options: `harness`, `context` (long-lived, for background work), `summarizer` (required: `{ model }`, where `model` may be a function `(conversationId) => ModelRef`, sync or async, for example that conversation's own model, and `models` defaults to the Harness's; or `{ summarize(job, signal) }` returning one line), `nodeBytes` (512), `viewBytes` (128000), `compactBytes` (32000, the compactor's separate view), `notes` (seed strings, stored once per conversation before the transcript), `workOf(message)` (host classification of subagent or job reports), `jobs` (64, how many summary tasks of a conversation the host keeps live at once), `tries` (5), `cap` (30000), `retryMs` (10000 initially, doubling after each failure to at most 60000), `settleTimeoutMs` (60000), `agentName`, `name`, `onError`. `dispose()` only stops this process's waits: the summary tasks belong to Pi, which keeps and recovers them, so it is harmless to skip and safe to call before `harness.close`.
+Pi applies an extension's hooks only in conversations that select it, so memory is opt-in per conversation and there is no separate mode. A host that selects extensions by default selects it everywhere; give conversations an explicit `extensions` list to keep it opt-in. `createOptChatMemory(options)` returns `{ extension, nap(id), stats(id), zoom(id, leafId, n), date(id, leafId), forgetMessage(id, messageIndex), forget(id), dispose() }`. Options: `harness`, `context` (long-lived, for background work), `summarizer` (required: `{ model }`, where `model` may be a function `(conversationId) => ModelRef`, sync or async, for example that conversation's own model, and `models` defaults to the Harness's; or `{ summarize(job, signal) }` returning one line), `nodeBytes` (512), `viewBytes` (128000), `compactBytes` (32000, the compactor's separate view), `foldHighWater` (1.25) and `foldLowWater` (0.75), fractions of both view budgets, `notes` (seed strings, stored once per conversation before the transcript), `workOf(message)` (host classification of subagent or job reports), `jobs` (64, how many summary tasks of a conversation the host keeps live at once), `tries` (5), `cap` (30000), `retryMs` (10000 initially, doubling after each failure to at most 60000), `settleTimeoutMs` (60000), `agentName`, `name`, `onError`. `dispose()` only stops this process's waits: the summary tasks belong to Pi, which keeps and recovers them, so it is harmless to skip and safe to call before `harness.close`.
 
 | Reference | Here |
 | --- | --- |
 | Append-only log of user, talk, tool and echo messages | The Pi transcript, never rewritten. Leaves are derived from the model context: a user message is `user`, assistant text `talk`, each tool call `tool`, each tool result `echo`; thinking is not logged. A leaf over `cap` characters keeps head and tail. Seeded `note` leaves precede them; messages marked by `workOf` become one `work` leaf |
 | Summary tree: node (l, i) covers messages [i·2^l, (i+1)·2^l), at most `nodeBytes` | The same, purely binary. A short message, or two children that fit together, are their own text and cost no call |
-| The view tiles the whole chat, one `id+n\|text` line per node, fit incrementally by merging the most due pair (age / 2^(level+2)), never splitting | The same fold (`advanceFold`), kept per conversation and advanced as the log grows, so the start of the view stays the same between requests. `renderView` produces the `<chat>` block |
+| The view tiles the whole chat, one `id+n\|text` line per node, fit incrementally by merging the most due pair (age / 2^(level+2)), never splitting | The checkpoint fold (`advanceFold`): append between checkpoints; above 125% of the budget, merge deterministically down to 75%. Both fractions are configurable. The public four-argument core helper retains strict fitting. `renderView` produces the `<chat>` block |
 | Each turn calls the model with [system][view][new message] | `GenerationTask.beforeRequest` replaces the messages before the current turn with the view, as four-line text blocks at the start of the current turn's first message. The turn in progress starts at the latest user or work message. Work reports and sibling results whose tool calls fell outside the turn become ordinary user-role input, with their full text preserved. Every positional system message is kept in order. The explanation of the view is a prompt section, so the system prompt and tools never change between calls |
 | Compactor: context first, no ids, scale line, resubmit with the cut at the limit, `JOBS` in parallel, patient retry | The same protocol (`askModel`), run by native pi-durable tasks: each node `(level, index, fingerprint)` is one `boring.memory.optchat.node` task (shipped in the extension's `tasks`, input `{ key, level, index, of, upTo }`, one `summarize` phase) that calls the summarizer through `runtime.models` (or your `summarize`) and, in one commit, writes the node and the summarizer's usage into the extension's state. Pi schedules, runs and recovers them. See below |
 | A turn waits until every line of the view is a summary and never shows cut text | The request waits up to `settleTimeoutMs`; after that the missing lines show `(not summarized yet: zoom it)`, never a clipped message, and `zoom(id, 1)` returns the message whole |
@@ -134,14 +134,14 @@ What is native, what is host-side:
 - **Waiting.** A request waits with `waitForTask` on the live node tasks, up to `settleTimeoutMs`, then shows the placeholder. `nap` waits until all nodes are built, including retries. These waits are in-process; if the process dies the tasks are not lost, only the cascade to parents waits for the next trigger.
 - **Retry.** A failed node is a failed native task (`outcome.failed`). Its failure time and streak length live in `boring.memory.optchat.state` with call count and spend. The host's idle admission loop waits from `retryMs`, doubling to at most 60 seconds, then creates another native task without a new turn. `dispose()` aborts those waits, not Pi's tasks. After a process restart, `nap` or another trigger restarts admission. Requests retain their bounded wait and can show placeholders during an outage. `onError` reports only the first failure of a streak.
 - **Usage.** In the commit that stores a node (or records its failure) the task adds the summarizer's usage, all attempts of the cut-and-resubmit loop included, to the extension's own state document (`boring.memory.optchat.state`), reported by `stats().compactor.usage`. It is not written into Pi's `pi.usage`: Pi has no public way for a task to record model usage (`recordUsage` is internal), so per BORING-PI-4 that guarantee is unavailable here rather than worked around; a public usage-recording seam is an upstream request. Hosts that budget model spend add `stats().compactor.usage` to `harness.usage()`. Task and document kinds are all in `boring.memory.optchat.*`; this package writes no `pi.*` document.
-- **Still host-side.** The trigger and admission logic, the in-memory fold and log per process (a cache; fingerprints keep it honest), and the summarizer prompt. A custom `summarize` function has no model usage to record. A model call interrupted by a crash is rerun from the start of the task, and its spend is not recorded.
+- **Still host-side.** The trigger and admission logic, the log cache and the summarizer prompt. Context refreshes reuse a prior leaf only after exact kind, capped-text and timestamp comparison, avoiding repeated byte counting and hashing without trusting native message identity. The emitted turn and compactor tilings, open flags and content fingerprints are retained in `boring.memory.optchat.fold` (no duplicated message or summary text), so cache eviction and restart restore the same prefix with unchanged options and history. A sent interior placeholder stays frozen until a checkpoint (the tail may settle); later summaries remain accessible through zoom. History changes, forgetting or changed fold options rebuild the view. A custom `summarize` function has no model usage to record. A model call interrupted by a crash is rerun from the start of the task, and its spend is not recorded.
 
 Deviations, honestly:
 
 - The log is Pi's model context, not an OptChat file, so ids are positions in that context. A native compaction, reset or edit that changes earlier messages invalidates the summaries it touches (fingerprints), and the view is refolded. `beforeCompact` declines while the extension is selected: the view replaces compaction, and a native summary would cut the transcript the view is built from. That is the only effect on Pi's lifecycle.
 - The current turn is verbatim (including its tool steps) instead of being summarized to one line each as soon as it is logged; it enters the view when the next user message arrives. A user message sent mid-run starts a new "current turn" for the next step.
 - A request is rewritten once per attempt from the full context. The summaries are a cache rebuilt from the log; the summarizer's calls are not Pi generations: they run as Pi tasks and their usage is in `stats().compactor.usage`, not `pi.usage` (see above).
-- Both views use four-line text blocks with a stable prefix. The compactor folds its own `compactBytes` view of preceding messages, independent of the chat's `viewBytes`. Admission builds preceding parents before dispatching later requests whose context would exceed this budget. Pi 1.2's `Models.complete` accepts retention preferences, but `TextContent` exposes no per-block cache marker. Blocks are unmarked; provider-specific cache-hit guarantees are unavailable. There is no separate `fsync` log or single-writer lock; Pi's storage owns durability. Summaries are read from documents on every request and stats call.
+- Both views use four-line text blocks with a stable prefix. The compactor folds its own `compactBytes` view, independent of the chat's `viewBytes`. Earlier summary jobs take only lines wholly before their source from that same tiling, rather than unfolding an older checkpoint. Admission builds preceding parents before dispatching later requests whose context would exceed this budget. Pi 1.2's `Models.complete` accepts retention preferences, but `TextContent` exposes no per-block cache marker. Blocks are unmarked; provider-specific cache-hit guarantees are unavailable. The 600-message/100-turn prefix regression uses 128 KB for both views. Smaller compactor budgets spend a larger share on uncached source/scale text; a stable fold alone does not guarantee 95% cache reads for complete requests. Qualify the host's chosen budgets with its provider. There is no separate `fsync` log or single-writer lock; Pi's storage owns durability. Summaries are read from documents on every request and stats call.
 - Cost: the compactor makes about two calls per message, each carrying its small view as context. Choose a cheap summarizer model; the browser-agent example uses the chat model only because it has one.
 - The package runs in Node and in a browser (no Node-only imports). Fictional-data tests are in `test/packages/agent.test.mjs`; `examples/bot` and `examples/browser-agent` have journeys.
 
@@ -305,6 +305,8 @@ Each evidence entry must link to a completed native ToolTask, its original assis
 
 `createSkillsExtension` tells the model that a user message starting with `/<skill-name>` is an explicit request to load that skill first, which is what the chat's `/` menu inserts. `defineAgent(...).skills` lists each skill's name and description for such a menu.
 
+With self-evolution, workspace skills may be flat `.agent/skills/<name>.md` files or `.agent/skills/<name>/SKILL.md` folders. Only `SKILL.md` defines a skill inside a folder; other Markdown and deeper files are companions, not skills. `load_skill` returns the body followed by “This skill's folder is `.agent/skills/<name>`; paths in it are relative to that folder.” (For flat files the folder is `.agent/skills`.) A skill can say `see portions.md` or `see tables/2026.csv`; the agent reads that path under the returned folder with Pi's native `read`, through the same workspace provider. Scanning and loading do not read the companions. No special companion folder, new file tool or additional access is granted. Host-given `Skill` data stays `{ name, description, body }`: with no source folder, `load_skill` returns its body unchanged.
+
 ## Sub-agents and schedules
 
 These follow [Pi Durable on Cloudflare](https://earendil.com/posts/pi-durable-on-cloudflare/) and stay host-agnostic: a Node process and a Durable Object mount the same tools.
@@ -323,7 +325,7 @@ These follow [Pi Durable on Cloudflare](https://earendil.com/posts/pi-durable-on
 
 `defineAgent({ …, selfEvolving: true, workspace })` lets the agent keep its own standing instructions, skills and tools in the `.agent/` folder of its workspace and apply them with a `reload` tool; the plan and its reasons are in [SELF-EVOLUTION.md](../../docs/architecture/SELF-EVOLUTION.md). Off by default.
 
-- One native extension per workspace, named `self-evolving:<workspace>`, so two workspaces in one host never replace each other's tools. Without a fixed `workspace` the agent works in a workspace per conversation: `inWorkspace(name)` returns the agent of workspace `name` (made once, then the same), whose configuration selects only `self-evolving:<name>`; the host creates each conversation with the agent of its owner's workspace and installs and reloads that workspace's extension when the workspace opens, so one harness and one registry never offer one workspace's agent-written tools to another's. It carries the skills (the host's, then `.agent/skills/*.md` parsed with `parseSkill`, behind the usual `load_skill`), `reload`, the agent-written tools and the `agent-written` prompt section. It is selected last; the host's `instructions` become the first section (`host-instructions`) of `agent.<id>`, because Pi renders native `instructions` after every extension section.
+- One native extension per workspace, named `self-evolving:<workspace>`, so two workspaces in one host never replace each other's tools. Without a fixed `workspace` the agent works in a workspace per conversation: `inWorkspace(name)` returns the agent of workspace `name` (made once, then the same), whose configuration selects only `self-evolving:<name>`; the host creates each conversation with the agent of its owner's workspace and installs and reloads that workspace's extension when the workspace opens, so one harness and one registry never offer one workspace's agent-written tools to another's. It carries the skills (the host's, then `.agent/skills/*.md` and `.agent/skills/<name>/SKILL.md` parsed with `parseSkill`, behind the usual `load_skill`), `reload`, the agent-written tools and the `agent-written` prompt section. It is selected last; the host's `instructions` become the first section (`host-instructions`) of `agent.<id>`, because Pi renders native `instructions` after every extension section.
 - `.agent/AGENTS.md` is rendered by that section on every request through the request's environment, labelled as agent-written and capped at `MAX_AGENT_INSTRUCTIONS` (16,000) characters.
 - `.agent/tools/<name>.json` is `{ name, description, parameters, run }`: `name` snake_case and not one of the agent's other tools, `parameters` a JSON Schema with `"type": "object"`, `run` one command line; a path under `.agent/` that `run` names must exist. Each becomes a native `ToolRegistration` (through `jsonSchemaTool`, so Pi refuses arguments that do not match `parameters` before `run` starts) whose `execute` runs `run` through the call's own `ExecutionEnv.exec` (the same provider instance as the files it was read from), with the arguments as one line of JSON on standard input (a quoted here-document); the output is the result and a non-zero exit is an error.
 - `reload` (and `DefinedAgent.reload(env, context)`, which a host calls on open and for the person's `/reload`) reads `.agent/` through the given environment's `FileSystem`, rebuilds the extension and calls the native `registry.install()` in every registry `install` was given (same name: replaced in place). Reloads are serialised. The report (`SelfEvolutionReport`, its `text` is the tool result) names tools added, changed and removed, the agent's skills, the instructions and every load error (invalid JSON, schema, missing script, name conflict, unreadable file). A reload never imports or evaluates anything. Without `exec` the tools are reported unavailable and skipped; instructions and skills still load.

package/dist/memory/optchat-core.d.ts

@@ -6,6 +6,9 @@ export declare const DEFAULTS: {
     readonly viewBytes: 128000;
     /** Independent view budget for summary requests. */
     readonly compactBytes: 32000;
+    /** Checkpoint thresholds, as fractions of each view budget. */
+    readonly foldHighWater: 1.25;
+    readonly foldLowWater: 0.75;
     /** Summarizer calls running at once. */
     readonly jobs: 64;
     /** Attempts per node to get a line under `nodeBytes`. */
@@ -49,7 +52,8 @@ export interface LeafOptions {
     /** Non-system message ordinals mapped to original leaf counts. Positional system updates must not shift tombstones. */
     readonly forgotten?: Readonly<Record<string, number>>;
 }
-export declare function leavesOf(messages: readonly Message[], cap?: number, options?: LeafOptions): {
+/** Reuse prior leaves only after exact kind, capped-text and timestamp comparison; native contexts may contain fresh message objects. */
+export declare function leavesOf(messages: readonly Message[], cap?: number, options?: LeafOptions, previous?: readonly Leaf[]): {
     leaves: Leaf[];
     before: number[];
 };
@@ -96,14 +100,16 @@ export interface ViewFold {
     T: number;
     size: number;
     parts: Part[];
+    /** Leaves already sent: interior placeholders stay frozen until a checkpoint. */
+    stableThrough?: number;
+    /** A checkpoint waiting for its required summaries; its due priorities keep the original leaf count. */
+    checkpoint?: boolean;
+    checkpointAt?: number;
 }
 export declare const emptyFold: () => ViewFold;
 export declare const partStart: (part: Pick<Part, "level" | "index">) => number;
-/**
- * Append leaves up to `to` (one part each, fitting after every append) and refit with what has been built since. The view
- * only ever appends at its end and coarsens, so its start stays the same from one request to the next (§5.2).
- */
-export declare function advanceFold(fold: ViewFold, tree: OptChatTree, to: number, viewBytes: number): ViewFold;
+/** Append one leaf at a time; coarsen from the high mark to the low mark in a single checkpoint. */
+export declare function advanceFold(fold: ViewFold, tree: OptChatTree, to: number, viewBytes: number, highWater?: number, lowWater?: number): ViewFold;
 /** The view's parts that still wait for a summary. */
 export declare const openParts: (fold: ViewFold) => Part[];
 /** One line per part, `id+n|text`, newlines as spaces, wrapped in `<chat>` (§5.1). */

package/dist/memory/optchat-core.js

@@ -5,6 +5,9 @@ export const DEFAULTS = {
     viewBytes: 128_000,
     /** Independent view budget for summary requests. */
     compactBytes: 32_000,
+    /** Checkpoint thresholds, as fractions of each view budget. */
+    foldHighWater: 1.25,
+    foldLowWater: 0.75,
     /** Summarizer calls running at once. */
     jobs: 64,
     /** Attempts per node to get a line under `nodeBytes`. */
@@ -72,13 +75,17 @@ export function capText(text, cap) {
     const head = Math.ceil(cap / 2), tail = Math.floor(cap / 2);
     return `${text.slice(0, head)}\n[… ${text.length - cap} characters cut …]\n${text.slice(text.length - tail)}`;
 }
-function leafOf(kind, text, at, cap) {
+function leafOf(kind, text, at, cap, previous) {
     const capped = capText(text, cap);
+    if (previous?.kind === kind && previous.text === capped && previous.at === at)
+        return previous;
     const line = `${kind}: ${capped}`;
     return { kind, text: capped, line, bytes: byteLength(line), hash: hashText(line), ...(at === undefined ? {} : { at }) };
 }
-export function leavesOf(messages, cap = DEFAULTS.cap, options = {}) {
-    const leaves = (options.notes ?? []).map(text => leafOf('note', text, undefined, cap)), before = [];
+/** Reuse prior leaves only after exact kind, capped-text and timestamp comparison; native contexts may contain fresh message objects. */
+export function leavesOf(messages, cap = DEFAULTS.cap, options = {}, previous = []) {
+    const leaves = (options.notes ?? []).map((text, index) => leafOf('note', text, undefined, cap, previous[index])), before = [];
+    const append = (kind, text, at) => leaves.push(leafOf(kind, text, at, cap, previous[leaves.length]));
     let messageIndex = 0;
     for (const message of messages) {
         before.push(leaves.length);
@@ -87,24 +94,24 @@ export function leavesOf(messages, cap = DEFAULTS.cap, options = {}) {
         const forgotten = options.forgotten?.[String(messageIndex++)];
         if (forgotten !== undefined) {
             for (let i = 0; i < forgotten; i++)
-                leaves.push(leafOf('note', '(forgotten)', undefined, cap));
+                append('note', '(forgotten)', undefined);
             continue;
         }
         const at = typeof message.timestamp === 'number' ? message.timestamp : undefined;
         if (options.workOf?.(message))
-            leaves.push(leafOf('work', textOf(message.content), at, cap));
+            append('work', textOf(message.content), at);
         else if (message.role === 'user')
-            leaves.push(leafOf('user', textOf(message.content), at, cap));
+            append('user', textOf(message.content), at);
         else if (message.role === 'assistant') {
             const talk = message.content.filter(part => part.type === 'text').map(part => part.text).join('\n');
             if (talk.trim())
-                leaves.push(leafOf('talk', talk, at, cap));
+                append('talk', talk, at);
             for (const part of message.content)
                 if (part.type === 'toolCall')
-                    leaves.push(leafOf('tool', `${part.name} ${JSON.stringify(part.arguments)}`, at, cap));
+                    append('tool', `${part.name} ${JSON.stringify(part.arguments)}`, at);
         }
         else if (message.role === 'toolResult')
-            leaves.push(leafOf('echo', `${message.toolName}${message.isError ? ' (error)' : ''}: ${textOf(message.content)}`, at, cap));
+            append('echo', `${message.toolName}${message.isError ? ' (error)' : ''}: ${textOf(message.content)}`, at);
     }
     before.push(leaves.length);
     return { leaves, before };
@@ -193,7 +200,7 @@ const label = (part) => `${partStart(part)}+${span(part.level)}`;
 const lineBytes = (part, text) => byteLength(label(part)) + 1 + byteLength(flat(text)) + 1;
 function settleOpen(fold, tree) {
     for (const part of fold.parts) {
-        if (!part.open)
+        if (!part.open || (!fold.checkpoint && partStart(part) < (fold.stableThrough ?? 0) && part !== fold.parts.at(-1)))
             continue;
         const text = tree.text(part.level, part.index);
         if (text === undefined)
@@ -204,37 +211,45 @@ function settleOpen(fold, tree) {
         part.open = false;
     }
 }
-/** Merge the most due adjacent pair until the view fits, never splitting; stop when no parent is built (§5.2). */
-function fit(fold, tree, viewBytes) {
-    while (fold.size > viewBytes) {
+/** Merge by due priority fixed at the checkpoint; never skip an unbuilt preferred parent. */
+function fit(fold, tree, high, low) {
+    if (fold.size > high && !fold.checkpoint) {
+        fold.checkpoint = true;
+        fold.checkpointAt = fold.T;
+    }
+    if (!fold.checkpoint)
+        return;
+    settleOpen(fold, tree);
+    while (fold.size > low) {
         let best = -1, bestDue = -Infinity;
         for (let p = 0; p + 1 < fold.parts.length; p++) {
             const a = fold.parts[p], b = fold.parts[p + 1];
             if (a.level !== b.level || a.index % 2 !== 0 || b.index !== a.index + 1)
                 continue;
-            if (tree.text(a.level + 1, a.index / 2) === undefined)
-                continue;
-            const due = (fold.T - partStart(a)) / 2 ** (a.level + 2);
+            const due = ((fold.checkpointAt ?? fold.T) - partStart(a)) / 2 ** (a.level + 2);
             if (due > bestDue) {
                 best = p;
                 bestDue = due;
             }
         }
         if (best < 0)
-            return;
+            break;
         const a = fold.parts[best], b = fold.parts[best + 1];
+        const text = tree.text(a.level + 1, a.index / 2);
+        if (text === undefined)
+            return;
         const merged = { level: a.level + 1, index: a.index / 2, bytes: 0, open: false };
-        merged.bytes = lineBytes(merged, tree.text(merged.level, merged.index));
+        merged.bytes = lineBytes(merged, text);
         fold.size += merged.bytes - a.bytes - b.bytes;
         fold.parts.splice(best, 2, merged);
     }
+    fold.checkpoint = false;
+    delete fold.checkpointAt;
 }
-/**
- * Append leaves up to `to` (one part each, fitting after every append) and refit with what has been built since. The view
- * only ever appends at its end and coarsens, so its start stays the same from one request to the next (§5.2).
- */
-export function advanceFold(fold, tree, to, viewBytes) {
+/** Append one leaf at a time; coarsen from the high mark to the low mark in a single checkpoint. */
+export function advanceFold(fold, tree, to, viewBytes, highWater = 1, lowWater = 1) {
     settleOpen(fold, tree);
+    fit(fold, tree, viewBytes * highWater, viewBytes * lowWater);
     while (fold.T < to) {
         const index = fold.T++;
         const text = tree.text(0, index);
@@ -242,22 +257,21 @@ export function advanceFold(fold, tree, to, viewBytes) {
         part.bytes = lineBytes(part, text ?? PLACEHOLDER);
         fold.parts.push(part);
         fold.size += part.bytes;
-        fit(fold, tree, viewBytes);
+        fit(fold, tree, viewBytes * highWater, viewBytes * lowWater);
     }
-    fit(fold, tree, viewBytes);
     return fold;
 }
 /** The view's parts that still wait for a summary. */
 export const openParts = (fold) => fold.parts.filter(part => part.open);
 /** One line per part, `id+n|text`, newlines as spaces, wrapped in `<chat>` (§5.1). */
 export function renderView(fold, tree) {
-    const lines = fold.parts.map(part => `${label(part)}|${flat(tree.text(part.level, part.index) ?? PLACEHOLDER)}`);
+    const lines = fold.parts.map(part => `${label(part)}|${flat(part.open ? PLACEHOLDER : tree.text(part.level, part.index) ?? PLACEHOLDER)}`);
     return `<chat>\n${lines.join('\n')}\n</chat>`;
 }
 /** The view's lines before `end` as bare text, one per line, with no ids: what a summarizer sees as context (§4.2). */
 export function bareLines(fold, tree, end) {
     return fold.parts.filter(part => partStart(part) + span(part.level) <= end)
-        .map(part => flat(tree.text(part.level, part.index) ?? PLACEHOLDER)).join('\n');
+        .map(part => flat(part.open ? PLACEHOLDER : tree.text(part.level, part.index) ?? PLACEHOLDER)).join('\n');
 }
 /** zoom(id, n): line `id+n` opened into its two children, or message `id` whole when n is 1 (§7.1). */
 export function zoomText(tree, id, n) {

package/dist/memory/optchat.d.ts

@@ -81,8 +81,11 @@ export interface OptChatOptions {
     readonly agentName?: string;
     readonly nodeBytes?: number;
     readonly viewBytes?: number;
-    /** Budget of the compactor's separate, incrementally folded view. Default 32 000 UTF-8 bytes. */
+    /** Target budget of the compactor's separate checkpoint-folded view. Default 32 000 UTF-8 bytes. */
     readonly compactBytes?: number;
+    /** Checkpoint high/low marks as fractions of both view budgets. Defaults 1.25 / 0.75; set both to 1 for strict budgets. */
+    readonly foldHighWater?: number;
+    readonly foldLowWater?: number;
     /** Seed memories from the app brief or PM decisions. Captured at creation, stored once before this conversation's messages. */
     readonly notes?: readonly string[];
     /** Host classification of agent, subagent or job reports. A work message starts a turn like a user message. */
@@ -107,7 +110,7 @@ export interface OptChatStats {
     /** Stored summaries whose fingerprint matches the log. */
     readonly summaries: number;
     readonly view: {
-        /** Bytes of the view's lines (what `budget` limits), without the `<chat>` wrapper. */
+        /** Bytes of the view's lines, without the `<chat>` wrapper; checkpoints start above budget × foldHighWater. */
         readonly bytes: number;
         readonly budget: number;
         /** Lines the next request would show, oldest first. `id` and `n` as in `id+n`. */
@@ -150,7 +153,7 @@ export interface OptChatStats {
         readonly logMessages: number;
         readonly viewBytes: number;
         readonly waitedMs: number;
-        /** False if the wait timed out and the view shows placeholders. */
+        /** False while the view still shows placeholders, including frozen ones from an earlier timeout. */
         readonly settled: boolean;
     };
 }
@@ -167,7 +170,7 @@ export interface OptChatMemory {
     date(conversationId: ConversationId, id: number): Promise<string>;
     /** Forget a Pi context message by its zero-based position, including system positions. Atomically tombstones its leaves and deletes summaries that could depend on them, including through context. Does not delete Pi's session entry. */
     forgetMessage(conversationId: ConversationId, messageIndex: number): Promise<void>;
-    /** Drop in-memory state of a conversation (stored summaries stay). */
+    /** Drop in-memory state of a conversation (stored summaries and the emitted tiling stay). */
     forget(conversationId: ConversationId): void;
     /** Stop this process's waits. Summary tasks belong to Pi, which keeps or recovers them; harmless to skip, and safe to call before closing the harness. */
     dispose(): Promise<void>;

package/dist/memory/optchat.js

@@ -27,6 +27,11 @@ export const OptChatEdits = defineDoc({
     kind: 'boring.memory.optchat.edits', version: 1, scope: 'conversation', history: 'latest', fork: 'current',
     initial: () => ({ revision: 0, forgotten: {} }),
 });
+/** Only the emitted tiling and fingerprints are retained, never another copy of message or summary text. */
+const OptChatFold = defineDocFamily({
+    kind: 'boring.memory.optchat.fold', version: 1, family: true, scope: 'conversation', history: 'latest', fork: 'current',
+    initial: () => ({ configuration: '', revision: -1, fingerprints: [], fold: null }),
+});
 /** Compactor bookkeeping of one conversation: calls, spend and the last failure of each node, including its retry streak. */
 export const OptChatState = defineDoc({
     kind: 'boring.memory.optchat.state', version: 1, scope: 'conversation', history: 'latest', fork: 'initial',
@@ -59,6 +64,7 @@ class Session {
     }
 }
 const message = (text) => ({ role: 'user', content: text.map(part => ({ type: 'text', text: part })), timestamp: Date.now() });
+const copyFold = (fold) => ({ ...fold, parts: fold.parts.map(part => ({ ...part })) });
 const LIVE = ['pending', 'running', 'waiting'];
 function redacted(item) {
     if (item.role === 'system')
@@ -104,6 +110,12 @@ export function createOptChatMemory(options) {
     const nodeBytes = options.nodeBytes ?? DEFAULTS.nodeBytes, viewBytes = options.viewBytes ?? DEFAULTS.viewBytes, jobs = options.jobs ?? DEFAULTS.jobs;
     const tries = options.tries ?? DEFAULTS.tries, cap = options.cap ?? DEFAULTS.cap, retryMs = options.retryMs ?? DEFAULTS.retryMs, settleMs = options.settleTimeoutMs ?? DEFAULTS.settleTimeoutMs;
     const compactBytes = options.compactBytes ?? DEFAULTS.compactBytes;
+    const highWater = options.foldHighWater ?? DEFAULTS.foldHighWater, lowWater = options.foldLowWater ?? DEFAULTS.foldLowWater;
+    if (!Number.isFinite(highWater) || !Number.isFinite(lowWater) || lowWater <= 0 || lowWater > highWater)
+        throw new Error('Invalid OptChat fold watermarks');
+    const configuration = JSON.stringify([nodeBytes, viewBytes, compactBytes, highWater, lowWater, cap]);
+    const foldView = (fold, tree, to, budget) => advanceFold(fold, tree, to, budget, highWater, lowWater);
+    const missingParts = (fold, tree) => openParts(fold).filter(part => tree.text(part.level, part.index) === undefined);
     const agentName = options.agentName ?? 'OptChat';
     const extensionName = options.name ?? 'boring.memory.optchat';
     const taskKind = `${extensionName}.node`;
@@ -142,11 +154,28 @@ export function createOptChatMemory(options) {
             throw new Error(`No conversation ${String(id)}`);
         return found;
     }
-    /** The compactor has an independent fold. Copy before advancing: native tasks and requests can overlap. */
+    /** Earlier summary jobs take a prefix of the same tiling, never unfolding an already-sent checkpoint. */
     function compactView(s, to) {
-        const cached = session(s.id).compactFold;
-        const fold = cached.T > to ? emptyFold() : { T: cached.T, size: cached.size, parts: cached.parts.map(part => ({ ...part })) };
-        return advanceFold(fold, s.tree, to, compactBytes);
+        return foldView(copyFold(s.compactFold), s.tree, to, compactBytes);
+    }
+    async function restoreFold(s, revision, to, kind = 'turn') {
+        const saved = await options.harness().snapshot(OptChatFold, s.id, kind, options.context);
+        if (saved?.fold && saved.configuration === configuration && saved.revision === revision && saved.fold.T <= to
+            && saved.fingerprints.every((fp, i) => { const part = saved.fold.parts[i]; return fp === s.log.fp(part.level, part.index); }))
+            s[kind === 'turn' ? 'fold' : 'compactFold'] = copyFold(saved.fold);
+    }
+    async function saveFold(s, revision, kind, fold, context) {
+        await (await conversationOf(s.id)).commit(async (tx) => {
+            if ((await tx.doc(OptChatEdits, s.id)).revision !== revision)
+                return;
+            const doc = await tx.doc(OptChatFold, s.id, kind, null);
+            if (kind === 'compact' && doc.configuration === configuration && doc.revision === revision && doc.fold && doc.fold.T > fold.T)
+                return;
+            doc.configuration = configuration;
+            doc.revision = revision;
+            doc.fold = copyFold(fold);
+            doc.fingerprints = fold.parts.map(part => s.log.fp(part.level, part.index));
+        }, context);
     }
     /** Bring a session up to date with the committed context and its stored summaries. */
     async function refresh(id) {
@@ -154,8 +183,10 @@ export function createOptChatMemory(options) {
         const view = await (await conversationOf(id)).context(options.context);
         const seed = await seedNotes(id);
         const edits = await memoryEdits(id);
-        s.sync(leavesOf(view.messages, cap, logOptions(seed, edits)).leaves);
+        s.sync(leavesOf(view.messages, cap, logOptions(seed, edits), s.log.leaves).leaves);
         await loadStored(s, options.harness(), options.context);
+        await restoreFold(s, edits.revision, s.log.leaves.length);
+        await restoreFold(s, edits.revision, s.log.leaves.length, 'compact');
         return s;
     }
     async function memoryEdits(id) {
@@ -254,8 +285,9 @@ export function createOptChatMemory(options) {
                 const load = async () => {
                     const seed = (await runtime.snapshot(OptChatNotes, id, context))?.notes ?? [];
                     const edits = await runtime.snapshot(OptChatEdits, id, context) ?? { revision: 0, forgotten: {} };
-                    s.sync(leavesOf((await runtime.context(id, context)).messages, cap, logOptions(seed, edits)).leaves);
+                    s.sync(leavesOf((await runtime.context(id, context)).messages, cap, logOptions(seed, edits), s.log.leaves.length ? s.log.leaves : session(id).log.leaves).leaves);
                     await loadStored(s, runtime, context);
+                    await restoreFold(s, edits.revision, s.log.leaves.length, 'compact');
                     return edits.revision;
                 };
                 const revision = await load();
@@ -263,13 +295,10 @@ export function createOptChatMemory(options) {
                 if (!ready() || s.tree.text(level, index) !== undefined)
                     return stale(); // the history changed, a child is missing, or the node needs no summary
                 const to = Math.min(upTo, index * span(level));
-                const shared = session(id);
-                shared.sync(s.log.leaves);
                 const fold = compactView(s, to);
-                if (fold.size > compactBytes)
+                if (fold.size > compactBytes * highWater || fold.checkpoint)
                     return stale(); // admission will build preceding parents before trying this node again
-                if (fold.T >= shared.compactFold.T)
-                    shared.compactFold = fold;
+                await saveFold(s, revision, 'compact', fold, context);
                 const source = level === 0 ? [s.log.leaves[index].line] : [s.tree.text(level - 1, 2 * index), s.tree.text(level - 1, 2 * index + 1)];
                 const job = { conversationId: id, level, index, range: [index * span(level), (index + 1) * span(level) - 1], source, context: bareLines(fold, s.tree, to), nodeBytes };
                 let summary;
@@ -314,8 +343,8 @@ export function createOptChatMemory(options) {
     /** Nodes that could be built now: sources exist and the whole preceding view is summarized (§4.1), oldest first. */
     function candidates(s) {
         const upTo = Math.min(s.upTo, s.log.leaves.length);
-        advanceFold(s.fold, s.tree, upTo, viewBytes);
-        const open = openParts(s.fold)[0];
+        foldView(s.fold, s.tree, upTo, viewBytes);
+        const open = missingParts(s.fold, s.tree)[0];
         const first = open ? partStart(open) : s.fold.T;
         const found = [];
         for (let level = 0; span(level) <= upTo; level++) {
@@ -327,7 +356,8 @@ export function createOptChatMemory(options) {
                     continue;
                 if (level > 0 && (s.tree.text(level - 1, 2 * index) === undefined || s.tree.text(level - 1, 2 * index + 1) === undefined))
                     continue;
-                if (compactView(s, index * span(level)).size > compactBytes)
+                const preceding = compactView(s, index * span(level));
+                if (preceding.size > compactBytes * highWater || preceding.checkpoint)
                     break;
                 found.push({ level, index });
             }
@@ -432,10 +462,12 @@ export function createOptChatMemory(options) {
         const deadline = Date.now() + settleMs, skip = new Set();
         for (;;) {
             await loadStored(s, options.harness(), options.context);
-            advanceFold(s.fold, s.tree, covered, viewBytes);
-            const open = openParts(s.fold).length;
-            if (open === 0 && s.fold.size <= viewBytes)
-                return true;
+            if (s.fold.stableThrough === undefined)
+                s.fold = emptyFold();
+            foldView(s.fold, s.tree, covered, viewBytes);
+            const open = missingParts(s.fold, s.tree).length;
+            if (open === 0 && s.fold.size <= viewBytes * highWater && !s.fold.checkpoint)
+                return openParts(s.fold).length === 0;
             const { live } = await admit(s, skip);
             if (open === 0 && live.size === 0)
                 return true;
@@ -515,18 +547,17 @@ export function createOptChatMemory(options) {
                     for (;;) {
                         const seed = await seedNotes(api.conversationId);
                         const edits = await memoryEdits(api.conversationId);
-                        const { leaves, before } = leavesOf(request.messages, cap, logOptions(seed, edits));
+                        const cached = session(api.conversationId);
+                        const { leaves, before } = leavesOf(request.messages, cap, logOptions(seed, edits), cached.log.leaves);
                         const last = request.messages.findLastIndex(item => item.role === 'user' || (item.role !== 'system' && options.workOf?.(item)));
                         const covered = last < 0 ? 0 : before[last];
                         if (covered === 0 && Object.keys(edits.forgotten).length === 0 && !request.messages.some(item => options.workOf?.(item)))
                             return undefined;
-                        const cached = session(api.conversationId);
                         cached.sync(leaves);
                         const s = new Session(api.conversationId, nodeBytes);
                         s.sync(leaves);
                         s.upTo = covered;
-                        if (cached.fold.T <= covered)
-                            s.fold = { T: cached.fold.T, size: cached.fold.size, parts: cached.fold.parts.map(part => ({ ...part })) };
+                        await restoreFold(s, edits.revision, covered);
                         const started = Date.now();
                         const settled = await settle(s, covered, context);
                         if ((await memoryEdits(api.conversationId)).revision !== edits.revision)
@@ -556,6 +587,10 @@ export function createOptChatMemory(options) {
                             return [{ ...current, content: [...blocks, ...body] }];
                         });
                         cached.lastRequest = { at: new Date().toISOString(), covered, requestMessages: messages.length, logMessages: request.messages.length, viewBytes: byteLength(view), waitedMs: Date.now() - started, settled };
+                        s.fold.stableThrough = s.fold.T;
+                        await saveFold(s, edits.revision, 'turn', s.fold, context);
+                        if ((await memoryEdits(api.conversationId)).revision !== edits.revision)
+                            continue;
                         cached.fold = s.fold;
                         return { messages };
                     }
@@ -575,8 +610,8 @@ export function createOptChatMemory(options) {
         nap: id => nap(id),
         async stats(id) {
             const s = await refresh(id);
-            const fold = { T: s.fold.T, size: s.fold.size, parts: s.fold.parts.map(part => ({ ...part })) };
-            advanceFold(fold, s.tree, s.log.leaves.length, viewBytes);
+            const fold = copyFold(s.fold);
+            foldView(fold, s.tree, s.log.leaves.length, viewBytes);
             let summaries = 0;
             for (const [key, node] of s.stored) {
                 const [level, index] = key.split(':').map(Number);
@@ -591,7 +626,7 @@ export function createOptChatMemory(options) {
             return {
                 leaves: s.log.leaves.length, summaries,
                 view: { bytes: fold.size, budget: viewBytes, open: openParts(fold).length,
-                    parts: fold.parts.map(part => ({ id: partStart(part), n: span(part.level), level: part.level, text: s.tree.text(part.level, part.index) ?? '', open: part.open })) },
+                    parts: fold.parts.map(part => ({ id: partStart(part), n: span(part.level), level: part.level, text: part.open ? '' : s.tree.text(part.level, part.index) ?? '', open: part.open })) },
                 compactor: { busy, retrying, failed: failures.length, idle: busy === 0 && retrying === 0, ...(latest ? { error: latest.error } : {}), calls: state?.calls ?? 0, usage: { ...(state?.usage ?? { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, cost: 0 }) } },
                 ...(s.lastRequest ? { lastRequest: s.lastRequest } : {}),
             };

package/dist/self-evolving.js

@@ -1,5 +1,6 @@
 import { defineExtension, defineTool } from '@earendil-works/pi-durable';
 import { Type } from '@earendil-works/pi-ai';
+import { skillInFolder } from './skills.js';
 import { jsonSchemaTool } from './json-schema-tool.js';
 /*
  * Self-evolution (docs/architecture/SELF-EVOLUTION.md, laws SELF-1..4 in this package's README): an agent keeps its own standing
@@ -102,6 +103,27 @@ async function filesIn(env, directory, suffix, context) {
         return listed.error.code === 'not_found' ? 'missing' : { error: listed.error.message };
     return listed.value.filter(entry => entry.kind === 'file' && entry.name.endsWith(suffix)).map(entry => `${directory}/${entry.name}`).sort();
 }
+/** Flat Markdown skills and `SKILL.md` in immediate child folders; companion files are not definitions. */
+async function skillFilesIn(env, context, errors) {
+    const listed = await env.listDir(SKILLS, context);
+    if (!listed.ok) {
+        if (listed.error.code !== 'not_found')
+            errors.push({ path: SKILLS, message: listed.error.message });
+        return [];
+    }
+    const paths = listed.value.filter(entry => entry.kind === 'file' && entry.name.endsWith('.md')).map(entry => `${SKILLS}/${entry.name}`);
+    for (const entry of listed.value.filter(entry => entry.kind === 'directory')) {
+        const folder = `${SKILLS}/${entry.name}`;
+        const contents = await env.listDir(folder, context);
+        if (!contents.ok) {
+            errors.push({ path: folder, message: contents.error.message });
+            continue;
+        }
+        if (contents.value.some(file => file.kind === 'file' && file.name === 'SKILL.md'))
+            paths.push(`${folder}/SKILL.md`);
+    }
+    return paths.sort();
+}
 export function createSelfEvolution(options) {
     const { name, parseSkill, skillsExtension } = options;
     const hostSkills = [...options.skills];
@@ -113,7 +135,7 @@ export function createSelfEvolution(options) {
         name: 'reload',
         description: `Apply the changes you made to your own ${AGENT_FOLDER}/ folder in the workspace, then return what was added, changed and removed and every load error. `
             + `${INSTRUCTIONS}: standing instructions for yourself, shown after the host's instructions on every request. `
-            + `${SKILLS}/<name>.md: skills (front matter with name and description between --- lines, then the instructions), loaded with load_skill. `
+            + `${SKILLS}/<name>.md or ${SKILLS}/<name>/SKILL.md: skills (front matter with name and description between --- lines, then the instructions), loaded with load_skill; paths in a skill are relative to its folder and read with read. `
             + `${TOOLS}/<name>.json: your own tools, {"name": "snake_case", "description": "...", "parameters": {a JSON Schema with "type": "object"}, "run": "a command run in the workspace, for example sh ${TOOLS}/<name>.sh"}; `
             + 'the arguments arrive as one line of JSON on standard input and the command output is the result. Restore or delete the files and reload to undo a change.',
         parameters: Type.Object({}, { additionalProperties: false }),
@@ -147,10 +169,8 @@ export function createSelfEvolution(options) {
         else if (read.error.code !== 'not_found')
             errors.push({ path: INSTRUCTIONS, message: read.error.message });
         const skills = [];
-        const skillFiles = await filesIn(env, SKILLS, '.md', context);
-        if (typeof skillFiles === 'object' && !Array.isArray(skillFiles))
-            errors.push({ path: SKILLS, message: skillFiles.error });
-        for (const path of Array.isArray(skillFiles) ? skillFiles : []) {
+        const skillFiles = await skillFilesIn(env, context, errors);
+        for (const path of skillFiles) {
             const text = await env.readTextFile(path, context);
             if (!text.ok) {
                 errors.push({ path, message: text.error.message });
@@ -168,7 +188,7 @@ export function createSelfEvolution(options) {
                 errors.push({ path, message: `a skill named ${skill.name} already exists` });
                 continue;
             }
-            skills.push(skill);
+            skills.push(skillInFolder(skill, path.slice(0, path.lastIndexOf('/'))));
         }
         const available = typeof env.exec === 'function';
         const tools = [];

package/dist/skills.d.ts

@@ -7,6 +7,7 @@ export interface Skill {
     readonly description: string;
     readonly body: string;
 }
+export declare function skillInFolder(skill: Skill, folder: string): Skill;
 /** Parse a `SKILL.md`-style document: `---` front matter with `name` and `description`, then the body. */
 export declare function parseSkill(markdown: string): Skill;
 export declare function validSkill(skill: Skill): Skill;

package/dist/skills.js

@@ -2,6 +2,11 @@ import { defineExtension, defineTool } from '@earendil-works/pi-durable';
 import { Type } from '@earendil-works/pi-ai';
 /** Names of skills and agents: a lowercase letter or digit, then lowercase letters, digits, `.`, `_` or `-` (at most 64). */
 export const NAME = /^[a-z0-9][a-z0-9._-]{0,63}$/;
+const skillFolders = new WeakMap();
+export function skillInFolder(skill, folder) {
+    skillFolders.set(skill, folder);
+    return skill;
+}
 /** Parse a `SKILL.md`-style document: `---` front matter with `name` and `description`, then the body. */
 export function parseSkill(markdown) {
     const match = /^---\r?\n([\s\S]*?)\r?\n---\r?\n?([\s\S]*)$/.exec(markdown);
@@ -23,7 +28,9 @@ export function validSkill(skill) {
         throw new TypeError(`Invalid skill name: ${skill.name}`);
     if (!skill.description.trim() || !skill.body.trim())
         throw new TypeError(`Skill ${skill.name} needs a description and a body`);
-    return Object.freeze({ name: skill.name, description: skill.description.trim(), body: skill.body });
+    const valid = Object.freeze({ name: skill.name, description: skill.description.trim(), body: skill.body });
+    const folder = skillFolders.get(skill);
+    return folder === undefined ? valid : skillInFolder(valid, folder);
 }
 /** A prompt section listing the skills and a `load_skill` tool returning one body. Usable without `defineAgent`. */
 export function createSkillsExtension(name, skills) {
@@ -44,7 +51,9 @@ export function createSkillsExtension(name, skills) {
                 replay: 'safe',
                 execute: async (args) => {
                     const skill = known.get(args.name);
-                    return { output: [{ type: 'text', text: skill ? skill.body : `Unknown skill "${args.name}". Available: ${[...known.keys()].join(', ')}` }] };
+                    const folder = skill && skillFolders.get(skill);
+                    const location = folder === undefined ? '' : `\n\nThis skill's folder is \`${folder}\`; paths in it are relative to that folder.`;
+                    return { output: [{ type: 'text', text: skill ? skill.body + location : `Unknown skill "${args.name}". Available: ${[...known.keys()].join(', ')}` }] };
                 },
             })],
     });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hachej/boring-agent",
-  "version": "0.4.4",
+  "version": "0.4.6",
   "license": "MIT",
   "private": false,
   "type": "module",
@@ -154,8 +154,8 @@
     "@earendil-works/pi-durable": "~1.2.0",
     "@earendil-works/pi-ai": "~1.2.0",
     "@earendil-works/chord": "~1.2.0",
-    "@hachej/boring-files": "^0.4.4",
-    "@hachej/boring-ui-kit": "^0.4.4"
+    "@hachej/boring-files": "^0.4.6",
+    "@hachej/boring-ui-kit": "^0.4.6"
   },
   "peerDependenciesMeta": {
     "@hachej/boring-files": {