@earendil-works/pi-durable 0.99.0 → 0.99.2

package/README.md

@@ -1,86 +1,475 @@
 # @earendil-works/pi-durable
 
-Durable conversation, task, and document runtime for Pi.
+> **Experimental.** The API changes without notice between releases.
 
-This package contains the Pico runtime. Its current public API provides durable record contracts and memory, JSONL, and SQLite storage implementations:
+A durable agent harness. Conversations, model turns, tool calls, and your own state are committed to storage before anything is shown. If the process dies mid-turn, reopening the storage picks the work up where it stopped.
 
-```ts
-import { MemoryStorage, ROOT_CONVERSATION_ID } from "@earendil-works/pi-durable";
-```
+Built on [`@earendil-works/pi-ai`](../ai/README.md) for model access and `@earendil-works/chord` for document state.
+
+## Table of Contents
 
-The root export is runtime-neutral. Storage implementations also have explicit subpaths:
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Concepts](#concepts)
+- [Persist and Resume](#persist-and-resume)
+- [Tools](#tools)
+- [System Prompt](#system-prompt)
+- [Conversation Settings](#conversation-settings)
+- [Watching a Conversation](#watching-a-conversation)
+- [Busy Conversations](#busy-conversations)
+- [Reset and Handoff](#reset-and-handoff)
+- [Compaction](#compaction)
+- [Agent Events (Experimental)](#agent-events-experimental)
+- [Hooks](#hooks)
+- [More Conversations and Forks](#more-conversations-and-forks)
+- [Abort and Subagents](#abort-and-subagents)
+- [Child Tasks](#child-tasks)
+- [Your Own State](#your-own-state)
+- [Usage and Cost](#usage-and-cost)
+- [Storage](#storage)
+- [Examples](#examples)
+- [Design Documents](#design-documents)
 
-```ts
-import { MemoryStorage } from "@earendil-works/pi-durable/storage/memory";
+## Installation
+
+```bash
+npm install @earendil-works/pi-durable @earendil-works/pi-ai @earendil-works/chord
 ```
 
-Node applications can open file-backed JSONL or SQLite storage through Node-only subpaths:
+## Quick Start
 
-```ts
+```typescript
 import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
-import { openNodeJsonlStorage } from "@earendil-works/pi-durable/storage/jsonl/node";
+import { createModels } from "@earendil-works/pi-ai/models";
+import { openaiProvider } from "@earendil-works/pi-ai/providers/openai";
+import { AssistantEntry, createRegistry, Harness, MemoryStorage } from "@earendil-works/pi-durable";
+
+const context = BACKGROUND_CONTEXT;
+
+const models = createModels();
+models.setProvider(openaiProvider()); // reads OPENAI_API_KEY
+
+const harness = await Harness.open(new MemoryStorage(), { models, registry: createRegistry() }, context);
+const root = await harness.root(context);
+await root.setModel({ provider: "openai", modelId: "gpt-6-sol" }, context);
+
+const submission = await root.submit({ type: "input", content: "What is the capital of France?" }, context);
+const settled = await submission.wait(context);
+if (settled.status === "done" && settled.type === "input") {
+	const answer = await root.commit((tx) => tx.entry(AssistantEntry, settled.answer), context);
+	console.log(answer?.model?.[0]);
+}
+await harness.close(context);
+```
+
+What happened:
+
+- `Harness.open()` opens a Session over a storage backend. `MemoryStorage` keeps everything in memory.
+- `root()` returns the root conversation, creating it on first use. A conversation is a transcript of immutable entries.
+- `submit()` durably admits your input and returns a `Submission`. A built-in generation task calls the model and appends the answer.
+- `wait()` resolves once the input is answered (`done`) or has failed (`unanswered`, with a reason).
+
+Every async call takes a Chord `Context`, which carries cancellation. `BACKGROUND_CONTEXT` never cancels. Cancelling a wait only cancels that wait, never the work.
+
+## Concepts
+
+- **Harness**: one open storage plus the machinery that runs agents on it. All changes go through one line of atomic commits, and nothing is shown before its commit is stored.
+- **Conversation**: a transcript. `root()` creates the root conversation on first use; you can create more and fork them. A `Conversation` handle holds no state; compare handles by `id`.
+- **Entry**: one immutable transcript record, such as a user message (`pi.user`), a model response (`pi.assistant`), a tool result (`pi.tool-result`), a system prompt change (`pi.system`), a reset (`pi.reset`), or your own kind. The model sees the entries from the newest reset onward.
+- **Commit**: an atomic write. `conversation.commit((tx) => ...)` can append entries, edit documents, and create tasks together; either all of it is stored or none of it.
+- **Document**: typed JSON state stored next to the transcript and changed in commits. Built-in ones hold each conversation's settings (`pi.conversation.config`), the running generation and tools (`pi.live`), queued submissions (`pi.inbox`), and spend (`pi.usage`).
+- **Task**: a durable state machine that saves a checkpoint at every step, so a restarted process continues from the last one. Every task has an owner: its conversation, or another task. The Harness runs answers as built-in tasks: `pi.generation` calls the model and owns the `pi.tool` tasks of its tool calls, waits for them, and hands the run to the next generation.
+- **Submission**: something you hand to a conversation, either user input or an entry to write, which you can wait for.
+- **Turn and run**: a turn is one model response and its tool calls; a run is the turns from an input to its final answer. A conversation is busy while a run is going.
+- **Registry**: your tools, system prompt sections, hooks, and tasks. It can change while the Harness runs; new work uses the new state.
+
+One answered input, as entries and tasks:
+
+```text
+submit(input) → pi.user
+  pi.generation → pi.system (only if the prompt or tools changed), pi.assistant (tool calls)
+    pi.tool × n → pi.tool-result × n   (owned by the generation, which waits for them)
+  pi.generation → pi.assistant (answer) → submission done
+```
+
+## Persist and Resume
+
+Use SQLite or JSONL storage to keep conversations across restarts:
+
+```typescript
 import { openNodeSqliteStorage } from "@earendil-works/pi-durable/storage/sqlite/node";
 
-const jsonl = await openNodeJsonlStorage("./session", BACKGROUND_CONTEXT);
-const sqlite = await openNodeSqliteStorage("./session.sqlite");
+const harness = await Harness.open(await openNodeSqliteStorage("./session.sqlite"), { models, registry }, context);
+const root = await harness.root(context); // the same root as last time
+harness.resume(); // continue any run the last process left unfinished
+```
+
+Work interrupted by a crash or close stays pending. `resume()` starts the task scheduler; submitting or waiting starts it too. A retried submission with the same `requestId` returns the existing submission instead of submitting twice:
+
+```typescript
+const submission = await root.submit({ type: "input", content: "Hello", requestId: "greeting-1" }, context);
+// After a restart: the same request ID finds the same submission.
+const again = await root.submit({ type: "input", content: "Hello", requestId: "greeting-1" }, context);
+// again.id === submission.id
+```
+
+`harness.submission(id)` reacquires a submission by ID, for example to wait for it after a restart.
+
+## Tools
+
+`@earendil-works/pi-durable/tools` provides `read`, `bash`, `edit`, and `write`. They touch files and processes only through the execution environment passed as `env`:
+
+```typescript
+import { NodeExecutionEnv } from "@earendil-works/pi-durable/env/node";
+import { createBashTool, createEditTool, createReadTool, createWriteTool } from "@earendil-works/pi-durable/tools";
+
+const registry = createRegistry();
+for (const tool of [createReadTool(), createBashTool(), createEditTool(), createWriteTool()]) registry.tools.add(tool);
+const env = new NodeExecutionEnv({ cwd: process.cwd() });
+const harness = await Harness.open(storage, { models, registry, env }, context);
 ```
 
-The portable JSONL core accepts the `FileSystem` capability exported from `@earendil-works/pi-durable/env`. Its `fsync` option defaults to `false`; enabling it flushes affected sidecars before appending the main commit marker. One `JsonlStorage` owner must serialize writes to a storage directory; cross-process locking and ID allocation are not supported.
+A new conversation activates every tool registered when it is created. Reading images is not supported yet.
 
-The portable SQLite core, minimal database facade, and ordered schema migrations are exported from `@earendil-works/pi-durable/storage/sqlite`. Adapters for synchronous SQLite environments such as Bun and Cloudflare Durable Objects can implement that facade without importing Node APIs. Remote asynchronous APIs such as Cloudflare D1 cannot implement this synchronous facade; they require a dedicated `Storage` backend.
+Define your own tool with a TypeBox schema. `api.output()` streams running output, which becomes the result when `execute()` returns no `content`:
+
+```typescript
+import { Type } from "@earendil-works/pi-ai";
+
+registry.tools.add({
+	name: "count",
+	description: "Count from 1 to n",
+	parameters: Type.Object({ n: Type.Number() }),
+	execute: async (args, api) => {
+		for (let i = 1; i <= (args as { n: number }).n; i++) api.output(`${i}\n`);
+		return {};
+	},
+});
+```
 
-The Node adapter uses WAL mode with `synchronous = NORMAL` and checkpoints the WAL on close. Acknowledged commits survive process crashes, but the newest commits may be lost after a power or host failure. One `SqliteStorage` owner must serialize writes to a database file; cross-process ID allocation is not supported.
+Each call runs as its own durable task. Its intent is committed before `execute()` runs. If the process dies mid-call, the tool reruns on reopen only when it is declared `replay: "safe"`; otherwise the model gets an `interrupted` error result with the output committed so far. Throwing from `execute()` gives the model an error result. A result can also return `usage`, which is added to the conversation's [usage](#usage-and-cost). It can also return `control: { terminate: true }`: when every result of the round asks for it, the run ends without another model request.
+
+## System Prompt
+
+The system prompt is built from named sections, rendered before each request:
+
+```typescript
+registry.systemPrompt.section("preamble", () => "You are a concise coding assistant.", { tag: false });
+registry.systemPrompt.section("cwd", () => process.cwd()); // rendered as <cwd>\n...\n</cwd>
+```
+
+Sections and tool changes are stored as positional system entries in the transcript. Only what changed is sent again, which keeps provider prompt caches warm. A section that returns something different every time, such as the current time, defeats that.
+
+## Conversation Settings
+
+Each conversation has durable settings, readable and writable through the handle:
+
+```typescript
+await root.setModel({ provider: "openai", modelId: "gpt-6-sol" }, context);
+await root.setThinkingLevel("high", context);
+await root.setActiveTools(["read", "bash"], context);
+await root.setToolExecution("sequential", context); // default "parallel"
+await root.setRetryPolicy({ enabled: true, maxRetries: 3, baseDelayMs: 2000 }, context);
+await root.setStreamOptions({ timeoutMs: 120_000 }, context);
+```
+
+Changes apply when the next model request is prepared, not in the middle of one.
+
+## Watching a Conversation
+
+Everything a UI needs is committed state. `viewState()` returns the conversation's structural view as a read-only Chord state, updated after every commit that touches it:
+
+```typescript
+const view = await root.viewState(context);
+view.subscribe((value) => {
+	// value.entries: the active transcript
+	// value.docs["pi.live"]: the running generation (streamed partial, retry, deferred) and tool calls (output, details)
+	// value.docs["pi.inbox"], value.docs["pi.usage"], value.docs["pi.conversation.config"]
+	render(value);
+});
+// later: view.dispose();
+```
+
+`watch()` delivers the same view with the exact Chord operations of each commit, one callback at a time:
+
+```typescript
+const watch = await root.watch(context);
+render(watch.value); // the state at attachment
+watch.start(async (value, ops) => {
+	await send(ops); // for example to a remote client that applies them
+});
+// later: await watch.stop();
+```
+
+A slow watch keeps at most 100 undelivered frames. After that, the pending frames are replaced by one frame holding the whole newest view. A client that joins late or reconnects starts from the current view; nothing is replayed.
+
+Partial answers and tool output are committed at most every 100 ms, so a crash loses at most that window.
+
+## Busy Conversations
+
+A conversation is busy while a run is working on an input. Submitting to a busy conversation queues the submission in the conversation's inbox, `docs["pi.inbox"]` in the view:
+
+```typescript
+await root.submit({ type: "input", content: "Also run the tests" }, context); // follow-up (default)
+await root.submit({ type: "input", content: "Use pnpm, not npm", whenBusy: "steer" }, context);
+await root.submit({ type: "input", content: "Only if idle", whenBusy: "reject" }, context); // throws ConversationBusy
+await root.submit({ type: "write", entry: { kind: "app.note", data: "user opened a file" } }, context);
+```
+
+- **Steers** are placed after the current tool round and join the running work.
+- **Follow-ups** are placed when the run answers, and start the next run.
+- **Writes** append an entry without asking the model anything.
+- `await submission.abort(context)` withdraws a queued submission.
+- `setSteeringMode("all")` and `setFollowUpMode("all")` place every queued item at once instead of one per turn.
+
+If a run fails, queued items stay in the inbox until the next submission places them, oldest first.
+
+## Reset and Handoff
+
+`reset()` starts a new context. The model no longer sees older entries, but they stay in storage:
+
+```typescript
+await root.reset(undefined, context);                                  // start from nothing
+await root.reset("We were fixing the flaky login test. Continue.", context); // start from a handoff note
+```
 
-## Storage conformance
+While busy, the reset is queued like a write. When it is placed during a tool round, the current run ends. A tool can request the same with `control: { handoff: "..." }`.
 
-Storage adapters can register the runner-independent conformance cases through the testing entry. The convenience adapter accepts Vitest/Jest-compatible runner functions without importing either package:
+## Compaction
 
-```ts
+Compaction shrinks what the model sees: it summarizes older entries and appends a `pi.compaction` entry that holds the summary and heads the first entry it keeps. Older entries stay in storage.
+
+```typescript
+const id = await root.compact("Keep the failing test names", context); // manual, with optional instructions
+const { outcome } = (await harness.waitForTask(id, context)).state;
+if (outcome.status === "completed" && outcome.result.submissionId !== undefined) {
+	const placed = await (await harness.submission(outcome.result.submissionId, context))!.wait(context);
+	console.log(placed.status); // "done", or "unanswered" with reason "stale"
+}
+```
+
+The conversation keeps working while the summary is made. The summary is placed at once when the conversation is idle, otherwise at the next turn boundary. Esc (`abort()`) cancels a manual compaction.
+
+Generation also compacts on its own, controlled per conversation:
+
+```typescript
+await root.setCompaction({
+	enabled: true, // automatic compaction; manual compact() always works
+	reserveTokens: 16384, // above contextWindow - reserveTokens, the next request waits for a compaction
+	keepRecentTokens: 20000, // roughly how much recent context stays verbatim
+	backgroundTokens: 32768, // this far below that, a compaction starts in the background; 0 disables it
+}, context);
+```
+
+When a provider rejects a request because the context is too long, generation compacts and retries once. A summary that would cut before the start of the current context settles as `stale` when it is placed, so when several are in flight, the furthest cut stays in effect. Summarization spend counts in `pi.usage`. A `beforeCompact` hook on `CompactionTask` can decline or supply its own summary.
+
+Running compactions are listed in `docs["pi.live"].compactions` with their reason, attempt, and retry backoff. The agent events add `compaction_start` and `compaction_end`, and a `compactions` field in the snapshot.
+
+## Agent Events (Experimental)
+
+For consumers that want coding-agent style events (`message_start`, `message_update`, `tool_execution_start`, ...) instead of structural state:
+
+```typescript
+import { watchEvents } from "@earendil-works/pi-durable";
+
+const stream = await watchEvents(harness, root.id, context);
+initialize(stream.snapshot); // entries, run, in-flight generation, tools, compactions, inbox, config, usage
+stream.start(async (events) => {
+	for (const event of events) console.log(JSON.stringify(event));
+});
+```
+
+Events are derived from commits, one batch per commit, and apply on top of the snapshot. Message and tool updates carry deltas: text and thinking appends, appended tool-call argument text, and output trims and appends. When a consumer falls more than 100 batches behind, it receives a fresh `snapshot` event instead. See `test/examples/19-json.ts` for the full stream of one run.
+
+## Hooks
+
+Hooks let extensions observe or adjust the built-in tasks:
+
+```typescript
+import { GenerationTask, ToolTask } from "@earendil-works/pi-durable";
+
+registry.hooks.add(ToolTask, {
+	beforeTool: (call) => (call.name === "bash" ? { block: "bash is disabled here" } : undefined),
+});
+registry.hooks.add(GenerationTask, {
+	onYield: (answer) => (needsMoreWork(answer) ? { continue: "Keep going." } : undefined),
+});
+```
+
+- **Generation:** `beforeRequest` (replace the messages of one request), `afterResponse`, `onYield` (continue the run with another user message), and `afterTools` (runs once a round's tools are done).
+- **Tools:** `beforeTool` (block or rewrite arguments) and `afterTool` (replace the result).
+
+Pass `{ scope: { conversationId } }` to limit a hook to one conversation.
+
+## More Conversations and Forks
+
+```typescript
+const other = await harness.createConversation({ ownership: { kind: "ownerless" } }, context);
+const fork = await root.fork(entryId, { ownership: { kind: "ownerless" } }, context);
+```
+
+A fork sees its parent's entries up to `entryId` and continues independently. It keeps the parent's settings as of that entry.
+
+## Abort and Subagents
+
+`await root.abort(context)` stops a conversation: queued inputs are withdrawn (queued writes stay), every task of its current work is aborted, and the call resolves once the conversation is idle.
+
+A conversation can be **owned** by a task. A subagent tool creates its child inside `api.commit()` with `ownership: { kind: "task", taskId: api.taskId }`, then drives it through `api.conversation(id)`:
+
+```typescript
+registry.tools.add({
+	name: "subagent",
+	description: "Delegate a self-contained task to a subagent and get its answer back.",
+	parameters: Type.Object({ task: Type.String() }),
+	replay: "safe", // a rerun after a crash finds the same child and submission
+	execute: async (args, api, context) => {
+		const child = await api.commit(async (tx) => {
+			// The ownership index remembers the child, so a rerun reuses it.
+			const existing = (await tx.scanConversations({ ownerTaskId: api.taskId }, 1)).items[0];
+			if (existing !== undefined) return existing.id;
+			const created = await tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } });
+			(await tx.doc(ConversationConfig, created.id)).model = { provider: "openai", modelId: "gpt-6-sol" };
+			return created.id;
+		}, context);
+		await api.details({ conversationId: child }, context); // lets a UI attach to the child
+		const request = { type: "input", content: (args as { task: string }).task, requestId: `subagent:${api.taskId}` } as const;
+		const settled = await (await (await api.conversation(child, context))!.submit(request, context)).wait(context);
+		return { content: [{ type: "text", text: settled.status }] };
+	},
+});
+```
+
+Owned work belongs to its owner:
+
+- Aborting the call aborts the child. So does the call failing: `execute()` throwing, or a crash that interrupts a call that is not replay-safe.
+- The parent is idle only once the child is.
+- A task created with `{ background: true }` is a boundary: work it owns survives the parent's abort and does not keep the parent busy. `root.abort(context, { background: true })` aborts it too.
+
+The examples show both patterns as product code:
+
+- [`22-subagent-foreground.ts`](test/examples/22-subagent-foreground.ts): the tool above, returning the child's answer. The UI finds the child through the call's `details` and prints the child's events indented under the call.
+- [`23-subagent-background.ts`](test/examples/23-subagent-background.ts): persistent subagents behind one `subagent` tool that spawns, messages (steer or follow-up), waits for, stops, and lists them. Each child is owned by a background anchor task, so the parent's Esc and idle waits never reach it. Each message is delivered by a background reporter task that posts the answer back to the parent as a follow-up input once it arrives; request IDs keep a restart from sending a message or a report twice.
+
+## Child Tasks
+
+A task can own child tasks, created with `ownership: { kind: "task", taskId }`, and wait for them by committing a `waiting` state:
+
+```typescript
+pay: async (task, runtime, context) => {
+	await runtime.commit(async (tx) => {
+		const payments = [];
+		for (const card of task.input.cards) {
+			payments.push(await tx.createTask(Payment, { card }, { ownership: { kind: "task", taskId: task.id } }));
+		}
+		// Resume in `decide` once every payment is done; the first failure aborts the rest.
+		return { status: "waiting", checkpoint: { phase: "decide", payments }, on: payments, policy: "failFast" };
+	}, context);
+},
+decide: async (task, runtime, context) => {
+	const outcomes = await runtime.outcomes(task.state.checkpoint.payments, context);
+	// ...commit the checkout's own outcome
+},
+```
+
+- **Waiting:** the task runs no code while it waits. With `allSettled` it resumes once every task in `on` is done; with `failFast` the first failed child also aborts the others. `on` may name other tasks too, with `allSettled`.
+- **Finishing:** a task that finishes while work it owns is still running is `completing`: its outcome is decided, but it becomes terminal, and `waitForTask()` returns, only once that work is done. A failed or aborted outcome aborts that work first.
+- **Aborting:** abort runs bottom-up. Aborting a task aborts the work it owns first, and its own abort handler starts only once that work is done, so each task undoes its own effects.
+
+[`24-child-tasks.ts`](test/examples/24-child-tasks.ts) runs a checkout with four payments: a declined card, a cancelled checkout, and a restart while the payments run.
+
+## Your Own State
+
+Documents are typed JSON objects committed together with entries. Define one, and edit it in a commit:
+
+```typescript
+import { defineDoc } from "@earendil-works/pi-durable";
+
+const Todos = defineDoc<{ items: string[] }>({
+	kind: "app.todos",
+	version: 1,
+	scope: "conversation",
+	history: "latest", // or "rewindable" to read old values with snapshotAsOf()
+	fork: "initial", // what a fork starts with: "initial", "current", or "asOf"
+	initial: () => ({ items: [] }),
+});
+
+await root.commit(async (tx) => {
+	(await tx.doc(Todos, root.id)).items.push("write docs");
+}, context);
+console.log(await harness.snapshot(Todos, root.id, context));
+```
+
+`harness.watchDoc()` and `harness.documentState()` observe one document like the view above. `registry.conversations.setup()` creates your documents in every new conversation, and a tool reads them through its `api`.
+
+## Usage and Cost
+
+Each conversation keeps token and cost totals in `docs["pi.usage"]`: per `provider/model` for model responses, and per tool name for tool results that report usage. Failed and aborted attempts count too. For the whole Session:
+
+```typescript
+const usage = await harness.usage(context); // { models: { "openai/gpt-6-sol": Usage }, tools: {...} }
+```
+
+## Storage
+
+| Backend | Import | Notes |
+|---|---|---|
+| Memory | `MemoryStorage` from the package root | Nothing is persisted. |
+| SQLite | `openNodeSqliteStorage(file)` from `@earendil-works/pi-durable/storage/sqlite/node` | One database file. WAL mode with `synchronous = NORMAL`: commits survive process crashes; the newest may be lost on power or host failure. |
+| JSONL | `openNodeJsonlStorage(directory, context)` from `@earendil-works/pi-durable/storage/jsonl/node` | Append-only files in one directory. Pass `{ fsync: true }` to flush before each commit marker. |
+
+One process owns a storage at a time; there is no cross-process locking. The portable SQLite and JSONL cores (`/storage/sqlite`, `/storage/jsonl`) run without Node APIs, for example on Bun or in Cloudflare Durable Objects, given a synchronous SQLite database or a `FileSystem` from `@earendil-works/pi-durable/env`.
+
+Custom backends can run the shared conformance suite with any Vitest- or Jest-compatible runner:
+
+```typescript
 import { registerStorageConformance } from "@earendil-works/pi-durable/testing";
 import { describe, expect, it } from "vitest";
 
-registerStorageConformance({ describe, expect, it }, "Some Custom Storage", async (use) => {
-	const storage = await openCustomStorage();
+registerStorageConformance({ describe, expect, it }, "My Storage", async (use) => {
+	const storage = await openMyStorage();
 	try {
 		await use(storage);
 	} finally {
-		await closeStorage(storage);
+		await closeMyStorage(storage);
 	}
 });
 ```
 
-The provider must call and await `use` exactly once with isolated storage. Other runners can use `createStorageConformance` with their own `StorageConformanceAssertions` implementation.
+The package root loads TypeBox, because the tool task validates arguments with pi-ai's `validateToolArguments()`. That costs about 23 MB of peak RSS unbundled, about 4 MB in a tree-shaken bundle.
 
-## Storage benchmarks
+## Examples
 
-From this package directory:
+Runnable examples live in [`test/examples`](test/examples). Run one from this package directory with:
 
-```sh
-npm run bench:storage
-npm run bench:storage:memory
+```bash
+node --conditions=source --experimental-strip-types test/examples/14-chat.ts
 ```
 
-The timing suite runs shared deterministic workloads against the built-in memory, JSONL, and SQLite adapters. Third-party adapters can import the same seeds and workload definitions from `@earendil-works/pi-durable/testing` and use their platform's timing runner:
+| Example | Shows |
+|---|---|
+| [14-chat](test/examples/14-chat.ts) | One question and answer |
+| [16-real-model](test/examples/16-real-model.ts) | Streaming an answer from OpenAI |
+| [17-coding-tools](test/examples/17-coding-tools.ts) | A tool-using turn on JSONL storage |
+| [18-print](test/examples/18-print.ts) | Print mode: submit a prompt, print the answer |
+| [19-json](test/examples/19-json.ts) | JSON mode: agent events or raw view operations, on SQLite, JSONL, or memory |
+| [20-inbox](test/examples/20-inbox.ts) | Steers, follow-ups, writes, and withdrawal while busy |
+| [21-late-join](test/examples/21-late-join.ts) | Attaching a view and an event stream mid-run |
+| [22-subagent-foreground](test/examples/22-subagent-foreground.ts) | A replay-safe subagent tool whose child the call owns, with the child's events under the call |
+| [23-subagent-background](test/examples/23-subagent-background.ts) | Persistent subagents: spawn, steer, stop, list, answers reported back, restart-safe |
+| [24-child-tasks](test/examples/24-child-tasks.ts) | A checkout that owns and waits for four payments: failFast, abort, restart |
+| [25-compaction](test/examples/25-compaction.ts) | A long chat compacted in the background, manually, and after a context overflow |
+| [00](test/examples/00-conversation.ts)–[13](test/examples/13-recovery.ts) | The layers underneath: sessions, documents, forks, watches, tasks, recovery |
 
-```ts
-import {
-	seedStorageBenchmark,
-	seedStorageWriteBenchmark,
-	STORAGE_READ_BENCHMARKS,
-	STORAGE_WRITE_BENCHMARKS,
-} from "@earendil-works/pi-durable/testing";
+Examples that call OpenAI need `OPENAI_API_KEY`; most use the faux provider otherwise.
 
-const dataset = await seedStorageBenchmark(readStorage);
-await STORAGE_READ_BENCHMARKS[0].run(readStorage, dataset);
+## Design Documents
 
-await seedStorageWriteBenchmark(freshWriteStorage);
-await STORAGE_WRITE_BENCHMARKS[0].run(freshWriteStorage);
-```
+- [`docs/pico-v5.md`](docs/pico-v5.md): the normative specification
+- [`docs/pico-v5-handoff.md`](docs/pico-v5-handoff.md): the implementation plan
+- [`docs/pico-v5-chord-usage.md`](docs/pico-v5-chord-usage.md): how the package uses Chord
 
-Read scenarios reuse one seeded store; each write sample requires a fresh seeded store. The workloads cover representative commits, indexed reads, pagination, fork traversal, document replay, historical reads, and persistent-backend reopen. The footprint suite measures each built-in adapter in a separate process at 1k and 10k scales and reports heap, RSS, external memory, file counts, and on-disk JSONL/SQLite size. These synthetic workloads are baselines for regression analysis, not production capacity limits or CI pass/fail thresholds.
+Benchmarks: `npm run bench:storage`, `npm run bench:storage:memory`, and `npm run bench:tool-output`.
 
-The normative design and implementation sequence are in:
+## License
 
-- [`docs/pico-v5.md`](docs/pico-v5.md)
-- [`docs/pico-v5-handoff.md`](docs/pico-v5-handoff.md)
-- [`docs/pico-v5-chord-usage.md`](docs/pico-v5-chord-usage.md)
+MIT

package/dist/entries.d.ts

@@ -1,4 +1,5 @@
 import type { JsonValue } from "@earendil-works/chord";
+import type { CompactionReason, ToolDiagnostic } from "./harness/types.ts";
 import type { Entry } from "./types.ts";
 /** Define a typed entry kind whose `is()` guard narrows by `EntryRecord.kind`. */
 export declare function defineEntry<D extends JsonValue = never>(kind: string): Entry<D>;
@@ -8,6 +9,23 @@ export declare const UserEntry: Entry<never>;
 export declare const AssistantEntry: Entry<never>;
 /** Positional prompt and tool change: `model` is `[SystemMessage]` with empty `content`. */
 export declare const SystemEntry: Entry<never>;
-/** Tool result: `model` is `[ToolResultMessage]`. Written by tool tasks. */
-export declare const ToolResultEntry: Entry<never>;
+/**
+ * Tool result: `model` is `[ToolResultMessage]`, whose content ends with the rendered diagnostics block; `data` holds
+ * the structured diagnostics, possibly none. Written by tool tasks, and by generation for calls it did not offer.
+ */
+export declare const ToolResultEntry: Entry<{
+    diagnostics: ToolDiagnostic[];
+}>;
+/**
+ * Start of a new context: always `head: "self"`, with `model` absent for a plain reset or `[UserMessage]` carrying the
+ * handoff text. Written by `Conversation.reset()` and the `handoff` tool control.
+ */
+export declare const ResetEntry: Entry<never>;
+/**
+ * Compaction summary: `model` is `[UserMessage]` with the wrapped summary, `head` the first kept entry. Written by
+ * compaction tasks, directly or through a write submission.
+ */
+export declare const CompactionEntry: Entry<{
+    reason: CompactionReason;
+}>;
 //# sourceMappingURL=entries.d.ts.map

package/dist/entries.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"entries.d.ts","sourceRoot":"","sources":["../src/entries.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,uBAAuB,CAAC;AACvD,OAAO,KAAK,EAAE,KAAK,EAA2B,MAAM,YAAY,CAAC;AAEjE,kFAAkF;AAClF,wBAAgB,WAAW,CAAC,CAAC,SAAS,SAAS,GAAG,KAAK,EAAE,IAAI,EAAE,MAAM,GAAG,KAAK,CAAC,CAAC,CAAC,CAM/E;AAED,sEAAsE;AACtE,eAAO,MAAM,SAAS,cAAyB,CAAC;AAChD,oGAAoG;AACpG,eAAO,MAAM,cAAc,cAA8B,CAAC;AAC1D,4FAA4F;AAC5F,eAAO,MAAM,WAAW,cAA2B,CAAC;AACpD,4EAA4E;AAC5E,eAAO,MAAM,eAAe,cAAgC,CAAC"}
+{"version":3,"file":"entries.d.ts","sourceRoot":"","sources":["../src/entries.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,uBAAuB,CAAC;AACvD,OAAO,KAAK,EAAE,gBAAgB,EAAE,cAAc,EAAE,MAAM,oBAAoB,CAAC;AAC3E,OAAO,KAAK,EAAE,KAAK,EAA2B,MAAM,YAAY,CAAC;AAEjE,kFAAkF;AAClF,wBAAgB,WAAW,CAAC,CAAC,SAAS,SAAS,GAAG,KAAK,EAAE,IAAI,EAAE,MAAM,GAAG,KAAK,CAAC,CAAC,CAAC,CAM/E;AAED,sEAAsE;AACtE,eAAO,MAAM,SAAS,cAAyB,CAAC;AAChD,oGAAoG;AACpG,eAAO,MAAM,cAAc,cAA8B,CAAC;AAC1D,4FAA4F;AAC5F,eAAO,MAAM,WAAW,cAA2B,CAAC;AACpD;;;GAGG;AACH,eAAO,MAAM,eAAe;iBAA8B,cAAc,EAAE;EAAqB,CAAC;AAChG;;;GAGG;AACH,eAAO,MAAM,UAAU,cAA0B,CAAC;AAClD;;;GAGG;AACH,eAAO,MAAM,eAAe;YAAyB,gBAAgB;EAAoB,CAAC"}

package/dist/entries.js

@@ -13,6 +13,19 @@ export const UserEntry = defineEntry("pi.user");
 export const AssistantEntry = defineEntry("pi.assistant");
 /** Positional prompt and tool change: `model` is `[SystemMessage]` with empty `content`. */
 export const SystemEntry = defineEntry("pi.system");
-/** Tool result: `model` is `[ToolResultMessage]`. Written by tool tasks. */
+/**
+ * Tool result: `model` is `[ToolResultMessage]`, whose content ends with the rendered diagnostics block; `data` holds
+ * the structured diagnostics, possibly none. Written by tool tasks, and by generation for calls it did not offer.
+ */
 export const ToolResultEntry = defineEntry("pi.tool-result");
+/**
+ * Start of a new context: always `head: "self"`, with `model` absent for a plain reset or `[UserMessage]` carrying the
+ * handoff text. Written by `Conversation.reset()` and the `handoff` tool control.
+ */
+export const ResetEntry = defineEntry("pi.reset");
+/**
+ * Compaction summary: `model` is `[UserMessage]` with the wrapped summary, `head` the first kept entry. Written by
+ * compaction tasks, directly or through a write submission.
+ */
+export const CompactionEntry = defineEntry("pi.compaction");
 //# sourceMappingURL=entries.js.map

package/dist/entries.js.map

@@ -1 +1 @@
-{"version":3,"file":"entries.js","sourceRoot":"","sources":["../src/entries.ts"],"names":[],"mappings":"AAGA,kFAAkF;AAClF,MAAM,UAAU,WAAW,CAA8B,IAAY;IACpE,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC;QAAE,MAAM,IAAI,SAAS,CAAC,uCAAuC,CAAC,CAAC;IAChH,OAAO;QACN,IAAI;QACJ,EAAE,EAAE,CAAC,KAA8B,EAA0B,EAAE,CAAC,KAAK,KAAK,SAAS,IAAI,KAAK,CAAC,IAAI,KAAK,IAAI;KAC1G,CAAC;AACH,CAAC;AAED,sEAAsE;AACtE,MAAM,CAAC,MAAM,SAAS,GAAG,WAAW,CAAC,SAAS,CAAC,CAAC;AAChD,oGAAoG;AACpG,MAAM,CAAC,MAAM,cAAc,GAAG,WAAW,CAAC,cAAc,CAAC,CAAC;AAC1D,4FAA4F;AAC5F,MAAM,CAAC,MAAM,WAAW,GAAG,WAAW,CAAC,WAAW,CAAC,CAAC;AACpD,4EAA4E;AAC5E,MAAM,CAAC,MAAM,eAAe,GAAG,WAAW,CAAC,gBAAgB,CAAC,CAAC","sourcesContent":["import type { JsonValue } from \"@earendil-works/chord\";\nimport type { Entry, EntryRecord, TypedEntry } from \"./types.ts\";\n\n/** Define a typed entry kind whose `is()` guard narrows by `EntryRecord.kind`. */\nexport function defineEntry<D extends JsonValue = never>(kind: string): Entry<D> {\n\tif (typeof kind !== \"string\" || kind.length === 0) throw new TypeError(\"Entry kind must be a non-empty string\");\n\treturn {\n\t\tkind,\n\t\tis: (entry: EntryRecord | undefined): entry is TypedEntry<D> => entry !== undefined && entry.kind === kind,\n\t};\n}\n\n/** User input: `model` is `[UserMessage]`. Written by submissions. */\nexport const UserEntry = defineEntry(\"pi.user\");\n/** Provider result with any stop reason: `model` is `[AssistantMessage]`. Written by generation. */\nexport const AssistantEntry = defineEntry(\"pi.assistant\");\n/** Positional prompt and tool change: `model` is `[SystemMessage]` with empty `content`. */\nexport const SystemEntry = defineEntry(\"pi.system\");\n/** Tool result: `model` is `[ToolResultMessage]`. Written by tool tasks. */\nexport const ToolResultEntry = defineEntry(\"pi.tool-result\");\n"]}
+{"version":3,"file":"entries.js","sourceRoot":"","sources":["../src/entries.ts"],"names":[],"mappings":"AAIA,kFAAkF;AAClF,MAAM,UAAU,WAAW,CAA8B,IAAY;IACpE,IAAI,OAAO,IAAI,KAAK,QAAQ,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC;QAAE,MAAM,IAAI,SAAS,CAAC,uCAAuC,CAAC,CAAC;IAChH,OAAO;QACN,IAAI;QACJ,EAAE,EAAE,CAAC,KAA8B,EAA0B,EAAE,CAAC,KAAK,KAAK,SAAS,IAAI,KAAK,CAAC,IAAI,KAAK,IAAI;KAC1G,CAAC;AACH,CAAC;AAED,sEAAsE;AACtE,MAAM,CAAC,MAAM,SAAS,GAAG,WAAW,CAAC,SAAS,CAAC,CAAC;AAChD,oGAAoG;AACpG,MAAM,CAAC,MAAM,cAAc,GAAG,WAAW,CAAC,cAAc,CAAC,CAAC;AAC1D,4FAA4F;AAC5F,MAAM,CAAC,MAAM,WAAW,GAAG,WAAW,CAAC,WAAW,CAAC,CAAC;AACpD;;;GAGG;AACH,MAAM,CAAC,MAAM,eAAe,GAAG,WAAW,CAAoC,gBAAgB,CAAC,CAAC;AAChG;;;GAGG;AACH,MAAM,CAAC,MAAM,UAAU,GAAG,WAAW,CAAC,UAAU,CAAC,CAAC;AAClD;;;GAGG;AACH,MAAM,CAAC,MAAM,eAAe,GAAG,WAAW,CAA+B,eAAe,CAAC,CAAC","sourcesContent":["import type { JsonValue } from \"@earendil-works/chord\";\nimport type { CompactionReason, ToolDiagnostic } from \"./harness/types.ts\";\nimport type { Entry, EntryRecord, TypedEntry } from \"./types.ts\";\n\n/** Define a typed entry kind whose `is()` guard narrows by `EntryRecord.kind`. */\nexport function defineEntry<D extends JsonValue = never>(kind: string): Entry<D> {\n\tif (typeof kind !== \"string\" || kind.length === 0) throw new TypeError(\"Entry kind must be a non-empty string\");\n\treturn {\n\t\tkind,\n\t\tis: (entry: EntryRecord | undefined): entry is TypedEntry<D> => entry !== undefined && entry.kind === kind,\n\t};\n}\n\n/** User input: `model` is `[UserMessage]`. Written by submissions. */\nexport const UserEntry = defineEntry(\"pi.user\");\n/** Provider result with any stop reason: `model` is `[AssistantMessage]`. Written by generation. */\nexport const AssistantEntry = defineEntry(\"pi.assistant\");\n/** Positional prompt and tool change: `model` is `[SystemMessage]` with empty `content`. */\nexport const SystemEntry = defineEntry(\"pi.system\");\n/**\n * Tool result: `model` is `[ToolResultMessage]`, whose content ends with the rendered diagnostics block; `data` holds\n * the structured diagnostics, possibly none. Written by tool tasks, and by generation for calls it did not offer.\n */\nexport const ToolResultEntry = defineEntry<{ diagnostics: ToolDiagnostic[] }>(\"pi.tool-result\");\n/**\n * Start of a new context: always `head: \"self\"`, with `model` absent for a plain reset or `[UserMessage]` carrying the\n * handoff text. Written by `Conversation.reset()` and the `handoff` tool control.\n */\nexport const ResetEntry = defineEntry(\"pi.reset\");\n/**\n * Compaction summary: `model` is `[UserMessage]` with the wrapped summary, `head` the first kept entry. Written by\n * compaction tasks, directly or through a write submission.\n */\nexport const CompactionEntry = defineEntry<{ reason: CompactionReason }>(\"pi.compaction\");\n"]}