pi-optchat 0.10.2 → 0.11.1

package/README.md

@@ -21,7 +21,7 @@ pi install npm:pi-web-access   # optional, for web search and page fetching
 
 Or from GitHub: `pi install git:github.com/jonaslsaa/pi-optchat`.
 
-Requirements: Pi 1.0.2 or compatible, Node.js 22.19+, and Git. Tested on macOS; the offline tests also run on Linux and Windows.
+Requirements: Pi 1.2 or later (tested on 1.4.0), Node.js 22.19+, and Git. Tested on macOS; the offline tests also run on Linux and Windows.
 
 Web access is not bundled. Subagents load the Pi extensions you have installed (except pi-optchat itself), so installing `pi-web-access` gives web tools to the main agent and every subagent.
 
@@ -213,6 +213,8 @@ Profile data lives in `~/.optchat/profiles/<name>/` (override the root with `OPT
 
 Git checkpoints and the pluggable `Store` seam (for memory on another machine): [docs/storage.md](docs/storage.md).
 
+OptChat's memory also runs outside Pi, for example in a Cloudflare Durable Object: `import ... from 'pi-optchat/core'`. See [docs/core.md](docs/core.md); `pi-optchat/durable` wires it into Pi Durable, see [docs/durable.md](docs/durable.md).
+
 ## Windows
 
 The same install and commands work on Windows, and the offline tests run there too. What differs:
@@ -242,7 +244,7 @@ OptChat keeps the recipe's numbers: 512-byte lines, binary merges, the 128,000
 
 | Area | Recipe | OptChat | Why |
 | --- | --- | --- | --- |
-| Prompts | The original and 2026-10-08 prompts | The same, lightly adapted, all in `src/prompts.ts`: reports start `[id]`, a note on `[OptChat status]`, "don't poll slow things", the size limit as a ceiling rather than a target | Fit OptChat's tools; the size wording gave less filler with no recall loss (#129) |
+| Prompts | The original and 2026-10-08 prompts | The same, lightly adapted, in `src/core/prompts.ts` and `src/prompts.ts`: reports start `[id]`, a note on `[OptChat status]`, "don't poll slow things", the size limit as a ceiling rather than a target | Fit OptChat's tools; the size wording gave less filler with no recall loss (#129) |
 | Compactor instructions | Gets your instructions | Doesn't | An A/B found they added nothing (#112) |
 | Summary size | Strict 512 bytes | Asked for 512, keeps up to 640 without a retry | Saves retries for a little view space |
 | Memory view | Sawtooth, 128,000 → 64,000 bytes | Same, saved to `view.json` with the compactions' view and reloaded as they were | A restart keeps both prompt prefixes cached |

package/docs/core.md

@@ -0,0 +1,38 @@
+# OptChat core
+
+`pi-optchat/core` is OptChat's memory without Pi: the log, the summary tree and its views, zoom/date/search, the compactor, cache marks and memory's prompts. It is meant for hosts such as Pi Durable in a Cloudflare Durable Object. The Pi extension (`pi-optchat`) is built on it and behaves exactly as before.
+
+Core imports nothing from Pi or Node at runtime: only `typebox` (the tools' schemas) and pi-ai's types. `test/core-boundary.test.ts` keeps it that way. Pi's packages are optional peer dependencies, so installing `pi-optchat` outside Pi pulls in none of them.
+
+It ships as TypeScript, like the extension, so the host needs a bundler (Wrangler, esbuild, tsx). A host that type-checks with `tsc` needs `allowImportingTsExtensions`.
+
+## What a host supplies
+
+- **A `Store`:** `load`, `append`, `appendNode`, `saveView`, `putImage` and `image`, sync or async, plus `text` and `summary`, which must be synchronous. Memory loads only metadata when it opens, reads texts through `text`/`summary` as it needs them, and issues its writes in order. Pi's is `FileStore` (`src/store.ts`); Pi Durable's is `SqliteStore` (`pi-optchat/durable`, see [durable.md](durable.md)).
+- **The compactor's model:** `createCompressor({ model, retryable })`, where `model()` returns `{ model, reasoning, stream }` for each call (pi-ai's `Models.streamSimple` or one with its shape) and `retryable` is pi-ai's `isRetryableAssistantError`.
+- **Image names**, if it keeps images: `logMessage` and `textContent` take an `imageRef` that names an image `[image <16 hex digits>]`; `loadImages` reads those back from the store.
+
+## What it exposes
+
+`src/core/index.ts` lists the exports by name: only these are public.
+
+```ts
+import { createCompressor, logMessage, memoryTools, openMemory, withView } from 'pi-optchat/core';
+
+const memory = await openMemory(store, createCompressor({ model: () => ({ model, stream: models.streamSimple.bind(models) }), retryable }));
+logMessage(memory, message);                       // each message of a turn, as the recipe's kinds
+await memory.settle();                             // until the view's lines are built
+const request = withView(run, memory.render());    // the view goes before the user's words
+const [zoom, date] = memoryTools(() => memory);    // VIEW_DOC tells the model how to use them
+```
+
+- `openMemory(store, compress, { warn, budget, jobs, retryMs })`, and `Memory`: `append`, `text`, `render`, `settle(signal?, 'turn' | 'tree' | 'ahead')`, `zoom`, `search`, `date`, `onChange`, `close`, plus `root`, `tree`, `view`, `pending`, `lastError`, `saveError`.
+- `Store`, `Saved`, `Lease`, `ImageStore` (a store's `putImage` and `image`, for keeping images elsewhere), `Mime`, `isMime`, and the record types `Entry`, `Summary`, `Origin`, `Kind`. An entry with an `origin` is imported history, which the summarizer treats as historical; `origin.source` is any label the host picks ("email").
+- `createCompressor({ model, retryable, onUsage?, accepted?, retryDelays?, system? })` (`system`: the compactor's prompt, `COMPACT` by default), `ACCEPTED_BYTES`.
+- `logMessage(memory, message, { kind?, receipt?, imageRef? })` (every entry it appends gets `receipt`, so a host can use it as a cursor), `textContent`, `withView`, `boundedMessage`, `completedExchange`, and the mid-turn refresh helpers (`refreshBudget`, `refreshPoint`, `refreshedRun`).
+- `memoryTools(memory, runs?)` (zoom and date; `runs` adds zooming into a subagent's chat), `searchTool`, `loadImages`, `isImage`, `decode`.
+- `cachePayload` (Anthropic: the view's cache marks, as a request's `onPayload`).
+- Imports: `planImport(existing, incoming, 'append' | 'rebuild', copyOf?)` dedupes by `receipt` (each starts with `import:`) and orders the new history by date: append keeps the existing log as the plan's first `kept` entries, rebuild orders everything and summarizes it again. `runImport({ entry, total, open, activate }, signal)` builds the memory `open` gives from the plan's `total` entries, read one at a time through `entry(i)` so the plan need not stay in RAM, resuming where it stopped, and calls `activate` once the whole tree is built and the memory is closed. The host keeps the plan, seeds the staging store (append: the kept entries, the summaries and the saved view), and switches to it in `activate`. Until then the live memory must take no messages, or the switch drops them: queue them and log them after. `deduplicate` alone tells what would be new, and `chronological` is the order rebuild uses. `pi-optchat/durable`'s `optchatImport` does all this on SQLite.
+- Prompts: `COMPACT`, `VIEW_DOC`, `CONTINUITY`, `SEARCH_DOC`, `allowSearch`, `TURN_REFRESHED`.
+
+Not in core: the turn itself (when to snapshot the view, refresh mid-turn, replay the previous exchange). Each host does it with the helpers above: Pi in `src/index.ts`, Pi Durable in `src/durable/optchat.ts`.

package/docs/durable.md

@@ -0,0 +1,35 @@
+# OptChat on Pi Durable
+
+`pi-optchat/durable` runs OptChat's memory in a [Pi Durable](https://www.npmjs.com/package/@earendil-works/pi-durable) agent, such as one in a Cloudflare Durable Object behind the Agents SDK's `PiHarness`. It is `SqliteStore`, a `Store` over SQLite (in a Durable Object, the object's), a Pi Durable extension that builds each request from OptChat's view, and `optchatImport` for importing old history. The Worker, its prompts and tools stay in the app. It needs `@earendil-works/pi-durable` 1.4.x (an optional peer dependency; tested on 1.4.0) and imports nothing from Pi's terminal app or Node; `test/core-boundary.test.ts` keeps it that way.
+
+## Wiring it up
+
+```ts
+const db = new DurableObjectSqliteDatabase(ctx.storage);         // the one Pi Durable's storage uses too
+const read = (sql, ...params) => ctx.storage.sql.exec(sql, ...params); // synchronous reads of memory's texts
+const memory = await openMemory(new SqliteStore(db, read), createCompressor({ model, retryable })); // or new SqliteStore(db, read, { images })
+const memoryExtension = optchat({ memory });
+registry.install(memoryExtension.extension);
+const harness = await Harness.open(storage, { models: withCacheMarks(models), registry, onWake, settings: { compaction: { enabled: false } } }, context);
+memoryExtension.attach(harness, context);
+```
+
+With the Agents SDK's `PiHarness` this goes in its `harness` factory, which supplies `storage`, `context` and `onWake` and resumes the Harness when an alarm wakes the object. A host without it passes an `onWake` that arms an alarm, and calls `harness.resume()` on open and on each alarm, or a task owed from before an eviction never runs.
+
+`examples/durable-object` is a Worker built this way, importing `pi-optchat/core` and `pi-optchat/durable` as an app would (linked with `file:../..`), with the counters and endpoints its `proof.mjs` uses: it runs the Worker under `wrangler dev` with real model calls and checks a multi-turn chat, restarts and the cache.
+
+## How it works
+
+- **Each request** (`beforeRequest`, in the configured conversation only) is the system prompt, OptChat's view, the previous exchange and Pi's current context. The view and previous exchange are fixed for one context, so the requests of a tool loop share a cached prefix. Pi's older transcript is never sent.
+- **Long runs** (`refreshAt`, 0.3 by default, 0 is off): once a run's messages since the last refresh pass that share of the model's window, the run up to its last answer is logged, the view is rebuilt, and the run goes on with its task and last step, as in Pi (#146). An earlier run still in the context, or a steer before the cut, moves into memory with the cut steps. The log's cursor says where the run is cut, so a restart sends the same request.
+- **When the conversation is idle** (no run, nothing in the inbox, checked again inside the commit), the extension resets Pi's context and creates a background task, in one commit. The task logs what the reset ended into memory and waits for its summaries. Pi Durable runs the task and, after an eviction, its alarm wakes the object to finish it.
+- **The log's cursor** is the log itself: each message logged from a Pi entry carries the receipt `pi:<entry id>`, written with the message. On open, the newest entry in memory is read again and the records it already has are skipped, so an entry cut off between its records (an answer with tool calls makes several) is completed. A restart logs nothing twice and summarizes nothing again. Only Pi's user, assistant and tool result entries are logged, plus a reset's note (`Conversation.reset(text)`, a tool's `handoff`) as `[Handoff note] …`, once the context it starts has ended. One endless chat with no resets of your own is still the recommended setup.
+- **A failed save** stops memory's log: requests keep the view it has, and after a restart logging resumes from the receipts and Pi's transcript. Cloudflare resets the object itself only when a commit fails after the code ran (the output gate, which drops Pi's writes too); a write that throws at once, like `SQLITE_FULL` at the storage limit, doesn't, so the restart waits for idle eviction, a deploy, or the host's `ctx.abort()`, which the example calls from `warn`. `SQLITE_FULL` stops Pi's writes as well; the errors only memory's writes could hit (a row over 2 MB, a message index saved twice) are ruled out by the store's parts and by one Memory per database.
+- **Compaction:** turn Pi's off; the extension also declines any that starts. OptChat's summaries replace it.
+- **Large tool results** are sent as their head and tail (`boundedMessage`), capped as Pi does.
+- **Images:** each image in a user message or tool result is named in memory's text as Pi's OptChat names it, `[image <16 hex digits>]` (SHA-256 of its base64, by Web Crypto), and kept in the store before the message is logged, so `zoom(id, 1)` returns it and the previous exchange names it instead of sending it again. OptChat doesn't resize images: Pi Durable's Photon processor (`createCloudflarePhotonImages()`) runs in Workers, but the memory it decodes in never shrinks again. Shrink photos when they arrive instead (Cloudflare Images, say), so Pi Durable's transcript and the model get the small copy too; Pi Durable keeps each transcript entry, base64 images included, in one row, so on Cloudflare a message's images must stay well under 2 MB anyway. `SqliteStore` keeps images in `optchat_images`; to keep large binary data out of the database (its row size limit and storage cost), give it `{ images }`, an `ImageStore` with `putImage` and `image`. The example keeps them in R2.
+- **Tools:** `zoom` and `date` (plus `search` with `search: true`), replay-safe.
+- **Cache marks:** `withCacheMarks(models)` adds OptChat's marks to Anthropic requests that carry a view, so a view that grows at its end is read from the cache. A request with its own `onPayload` is left alone.
+- **Summaries of a group chat:** a chat that isn't one user talking to OptChat, such as a housing board with several people, gives `createCompressor` its own `system` prompt and names each speaker in the text, like "[Kari] …".
+- **Imports** (`optchatImport`): `start(entries, 'append' | 'rebuild')` plans the import and saves the plan in `optchat_plan`; append keeps the log, its summaries and its view (so the cached prefix holds), rebuild orders everything by date and summarizes it all again. A Pi Durable task builds the new memory beside the live one, as a new generation of the same tables, and carries on after an eviction; it saves its progress every 50 summaries, so restarts during a long import don't park it. The chat goes on with the old memory meanwhile. Once the new one is complete it goes live in one transaction and `onActivated` is called: restart the object there, since the memory in RAM is still the old one: `ctx.abort()` once `ctx.storage.sync()` resolves, as the example does. Aborting at once drops the responses still waiting on the writes, such as that of the `start()` call itself when the import had nothing to summarize; a request that is mid-reply at the switch fails either way, and Pi Durable runs its turn again after the restart. On open, chat messages the old memory logged from Pi's transcript during the import are logged again; messages a host appended to memory directly meanwhile are not, so don't. The old generation's rows are deleted then too. `pending()` gives the job and its task's state; `discard()` stops it. A second import, or one after a failed save, is refused. Neither the plan nor the log's texts are held in RAM. Two memories summarize at once during an import: open the import's with fewer jobs (`jobs: 4`) to stay under the model's rate limit.
+- **Storage:** tables `optchat_log`, `optchat_tree`, `optchat_view` (each row with its generation), `optchat_state`, `optchat_plan` and `optchat_images` beside Pi's `pi_*`, in the same database and queue. A message is stored in parts of 512K UTF-16 units, under the 2 MB row limit; an image is one row. Memory keeps only metadata and node sizes in RAM (about 5 MB of heap for a 16,000-message log, measured in workerd) and reads texts through `read` as it needs them, synchronously, so zoom, search and views stay synchronous.

package/docs/recipe-differences.md

@@ -4,7 +4,7 @@ The README has the summary table. This page has the detail: what changed in each
 
 ## Prompts
 
-`src/prompts.ts` holds every prompt sent to a model (the recipe's four, the subagent instruction paragraphs, the compaction task and its retry, the handoff prompts, the import note), each with a comment on where it goes. Tool descriptions stay with their tools. The recipe's prompts are lightly adapted:
+`src/core/prompts.ts` holds memory's prompts (the compactor's and the view doc from the recipe, the compaction task and its retry, the import note) and `src/prompts.ts` Pi's (the main agent's, the subagent instruction paragraphs, the handoff prompts), each with a comment on where it goes. Tool descriptions stay with their tools. The recipe's prompts are lightly adapted:
 
 - **Main agent.** The turn rules are the revised recipe's (2026-10-08) word for word, with three changes:
   - its paragraph on computers is left out;

package/docs/storage.md

@@ -12,10 +12,10 @@ An import builds a new memory generation (`memories/<id>/`) and switches `active
 
 ## The `Store` seam
 
-Memory reaches disk through one seam, `Store` in `src/store.ts`: the log, summary nodes, saved view and images, which is what would travel between machines. The saved view (`view.json`) holds the chat's view and the compactions' view, each with its batch state; an older one without the compactions' view refolds it from the chat's, once.
+Memory reaches disk through one seam, `Store` in `src/core/store.ts` (Pi's `FileStore` is in `src/store.ts`): the log, summary nodes, saved view and images, which is what would travel between machines. The saved view (`view.json`) holds the chat's view and the compactions' view, each with its batch state; an older one without the compactions' view refolds it from the chat's, once.
 
 - Runs, recovery journals, imports and their pointer, usage, settings and `AGENTS.md` stay local files.
-- `FileStore` (the layout in the README) is the only store today. Another, such as Postgres, would implement the same few operations.
+- Pi uses `FileStore` (the layout in the README). `pi-optchat/durable` has `SqliteStore`, for a Durable Object's SQLite ([durable.md](durable.md)). Another, such as Postgres, would implement the same few operations.
 - Its lease (take, heartbeat, release) is optional: without one, OptChat keeps its machine-local profile lock.
 - Import records (`imports/*.json`, the staging generations and the `active-memory.json` pointer) are per machine, so a second machine with the same `~/.claude` or Codex history on disk may offer those sessions for import again.
 

package/docs/victor-recipe.md

@@ -6,13 +6,13 @@ Source: https://gist.github.com/VictorTaelin/91837951a5ce5b38f341ec1ba1df6449
 
 The implementation was compared with the recipe fetched on 2026-10-04. The reference content SHA-256 was `8f6997e8944d85e4df53b5704bf7c4e393e4da361071181e9cc2f7d9d1b6e430`.
 
-The source remains upstream rather than duplicating the full article here. Its prompt strings, from the original and the 2026-10-08 revision, are in `src/prompts.ts` with the small adaptations listed in `docs/recipe-differences.md`, and attribution in `THIRD_PARTY_NOTICES.md`.
+The source remains upstream rather than duplicating the full article here. Its prompt strings, from the original and the 2026-10-08 revision, are in `src/core/prompts.ts` and `src/prompts.ts` with the small adaptations listed in `docs/recipe-differences.md`, and attribution in `THIRD_PARTY_NOTICES.md`.
 
 ## Implementation mapping
 
-- `src/memory.ts`: append-only log, binary summary tree, compression scheduling (§4 of the 2026-10-08 revision: a message's summary starts once fewer than 8 lines before it are unbuilt, a merge once both halves are built, kept in a queue so the tree is never scanned for work), the compactions' own view (§4: the view merged further to 16,000-32,000 bytes, ending at the node; unlike the recipe, a line not built yet shows as a placeholder instead of ending it, so a summary still sees the built lines after it), the view (§3.2: the most due pair, measured from its last message, merges in one batch from 128,000 down to 64,000 bytes, and the view is saved to `view.json` rather than refolded at start), zoom/date, and the opt-in text search over original messages (`src/tools.ts` has the tool; not in the recipe, off by default).
-- `src/compactor.ts`: contextual compression and size retries, with the 2026-10-08 revision's task (a 512-dash ruler for the size) and its "Too long" retry, verbatim. The compactions' view shows each line under its `id+n|` head, as the task's line ids need, so a reply that copies a head has it removed.
-- `src/cache.ts`: Anthropic cache marks: the view in blocks of 4 lines, one mark on the last whole block and one at the request's end, as in recipe §3.3 (a compaction leaves its end unmarked, as Unii does: its task is never sent again), plus two marks 20 and 40 blocks before the last. Anthropic looks back only 20 blocks from a mark, so with the recipe's one view mark a turn that added more than 80 lines, such as 40 tool calls and their results, made the next turn rewrite the whole view. OpenAI requests get no marks: GPT-5.6 and GPT-6 Luna reject the recipe's `prompt_cache_breakpoint` with a 400 (found by @aaaxn), so they rely on implicit prefix caching. OptChat doesn't set `reasoning.context` either: GPT-5.6 already defaults to the recipe's `"all_turns"`, and OpenAI documents it only for GPT-5.6 and GPT-6.1 Sol.
+- `src/core/memory.ts`: append-only log, binary summary tree, compression scheduling (§4 of the 2026-10-08 revision: a message's summary starts once fewer than 8 lines before it are unbuilt, a merge once both halves are built, kept in a queue so the tree is never scanned for work), the compactions' own view (§4: the view merged further to 16,000-32,000 bytes, ending at the node; unlike the recipe, a line not built yet shows as a placeholder instead of ending it, so a summary still sees the built lines after it), the view (§3.2: the most due pair, measured from its last message, merges in one batch from 128,000 down to 64,000 bytes, and the view is saved to `view.json` rather than refolded at start), zoom/date, and the opt-in text search over original messages (`src/core/tools.ts` has the tool; not in the recipe, off by default).
+- `src/core/compactor.ts`: contextual compression and size retries, with the 2026-10-08 revision's task (a 512-dash ruler for the size) and its "Too long" retry, verbatim. The compactions' view shows each line under its `id+n|` head, as the task's line ids need, so a reply that copies a head has it removed.
+- `src/core/cache.ts`: Anthropic cache marks: the view in blocks of 4 lines, one mark on the last whole block and one at the request's end, as in recipe §3.3 (a compaction leaves its end unmarked, as Unii does: its task is never sent again), plus two marks 20 and 40 blocks before the last. Anthropic looks back only 20 blocks from a mark, so with the recipe's one view mark a turn that added more than 80 lines, such as 40 tool calls and their results, made the next turn rewrite the whole view. OpenAI requests get no marks: GPT-5.6 and GPT-6 Luna reject the recipe's `prompt_cache_breakpoint` with a 400 (found by @aaaxn), so they rely on implicit prefix caching. OptChat doesn't set `reasoning.context` either: GPT-5.6 already defaults to the recipe's `"all_turns"`, and OpenAI documents it only for GPT-5.6 and GPT-6.1 Sol.
 - `src/transcript.ts`: fresh context per parent run, current-run tool loop retained. The previous completed exchange is also retained in full text (left out if over 16,000 bytes by default), an intentional addition to the summary-only recipe for conversational continuity.
 - `src/agents.ts`: asynchronous Pi SDK children and automatic completion reports.
 - `src/settings.ts`: per-profile settings for the departures from the recipe. Defaults are the recipe's (no memory search), except the previous exchange (on), the summary size tolerance (640 bytes, against the recipe's strict 512) and the turn refresh threshold (30%, where the recipe has none). Grouped subagent reports are off by default, as in the revised recipe, which reports each subagent on its own.

package/package.json

@@ -1,7 +1,13 @@
 {
   "name": "pi-optchat",
-  "version": "0.10.2",
+  "version": "0.11.1",
   "type": "module",
+  "exports": {
+    ".": "./src/index.ts",
+    "./core": "./src/core/index.ts",
+    "./durable": "./src/durable/index.ts",
+    "./package.json": "./package.json"
+  },
   "pi": {
     "extensions": [
       "./src/index.ts"
@@ -14,22 +20,42 @@
     "test:live": "tsx test/live.ts"
   },
   "devDependencies": {
-    "@earendil-works/pi-agent-core": "1.0.2",
-    "@earendil-works/pi-ai": "1.0.2",
-    "@earendil-works/pi-coding-agent": "1.0.2",
-    "@earendil-works/pi-tui": "1.0.2",
-    "@types/node": "^24.0.0",
-    "tsx": "^4.20.0",
+    "@earendil-works/chord": "1.4.0",
+    "@earendil-works/pi-agent-core": "1.4.0",
+    "@earendil-works/pi-ai": "1.4.0",
+    "@earendil-works/pi-coding-agent": "1.4.0",
+    "@earendil-works/pi-durable": "1.4.0",
+    "@earendil-works/pi-tui": "1.4.0",
+    "@types/node": "^22.20.5",
+    "tsx": "^4.23.15",
     "typebox": "1.3.27",
-    "typescript": "^5.9.0"
+    "typescript": "^7.0.2"
   },
   "peerDependencies": {
-    "@earendil-works/pi-agent-core": "*",
-    "@earendil-works/pi-ai": "*",
-    "@earendil-works/pi-coding-agent": "*",
-    "@earendil-works/pi-tui": "*",
+    "@earendil-works/pi-agent-core": ">=1.2",
+    "@earendil-works/pi-ai": ">=1.2",
+    "@earendil-works/pi-coding-agent": ">=1.2",
+    "@earendil-works/pi-durable": "~1.4.0",
+    "@earendil-works/pi-tui": ">=1.2",
     "typebox": "*"
   },
+  "peerDependenciesMeta": {
+    "@earendil-works/pi-agent-core": {
+      "optional": true
+    },
+    "@earendil-works/pi-ai": {
+      "optional": true
+    },
+    "@earendil-works/pi-coding-agent": {
+      "optional": true
+    },
+    "@earendil-works/pi-durable": {
+      "optional": true
+    },
+    "@earendil-works/pi-tui": {
+      "optional": true
+    }
+  },
   "description": "OptChat memory for Pi: persistent profiles, zoomable history, background agents, and conversation imports.",
   "keywords": [
     "pi-package",

package/src/agent-view.ts

@@ -8,7 +8,7 @@ import type { AgentMessage } from '@earendil-works/pi-agent-core';
 import type { Children } from './agents.ts';
 import { isHandingOff, type RunInfo } from './runs.ts';
 import { textContent } from './transcript.ts';
-import { record } from './cache.ts';
+import { record } from './core/cache.ts';
 import { agentLabel, elapsed, fit, oneLine } from './inspector.ts';
 import { isReport } from './report-message.ts';
 

package/src/agents.ts

@@ -6,16 +6,19 @@ import { createAgentSession, DefaultResourceLoader, SessionManager, SettingsMana
 import * as sdk from '@earendil-works/pi-coding-agent';
 import type { AgentMessage } from '@earendil-works/pi-agent-core';
 import { StringEnum, type Api, type Model } from '@earendil-works/pi-ai';
-import { allowSearch, windowHandoff, CONNECTED, cutOffChildren, delegation, NO_DELEGATION, NOT_CONNECTED, RESTARTED, SEARCH_DOC, STEERABLE, SUBAGENT, VIEW_DOC } from './prompts.ts';
-import { memoryTools, searchTool } from './tools.ts';
-import { VIEW, type Memory } from './memory.ts';
+import { allowSearch, SEARCH_DOC, VIEW_DOC } from './core/prompts.ts';
+import { windowHandoff, CONNECTED, cutOffChildren, delegation, NO_DELEGATION, NOT_CONNECTED, RESTARTED, STEERABLE, SUBAGENT } from './prompts.ts';
+import { memoryTools } from './tools.ts';
+import { searchTool } from './core/tools.ts';
+import { VIEW } from './core/memory.ts';
+import type { Memory } from './memory.ts';
 import type { ModelChoice } from './compactor.ts';
-import { cachePayload } from './cache.ts';
+import { cachePayload } from './core/cache.ts';
 import { CUT_OFF, reportHead, RunHistory, isRunning, transition, sessionMessages, savedSession, undelivered, sentText, type Guidance, type RunInfo, type RunState, type FinishReason } from './runs.ts';
 import { UsageLedger } from './usage.ts';
 import { fullChat, refreshFraction, textContent } from './transcript.ts';
 import { Type, type Static } from 'typebox';
-import { result } from './tools.ts';
+import { result } from './core/tools.ts';
 import { handoffText, withUndelivered, type HandoffEvidence } from './handoff.ts';
 import { DEFAULT_SETTINGS, type Settings } from './settings.ts';
 import { THINKING } from './profiles.ts';

package/src/browser.ts

@@ -1,13 +1,17 @@
 import { join } from 'node:path';
 import { atomicWrite } from './store.ts';
-import { type Memory } from './memory.ts';
+import type { Memory } from './memory.ts';
+import type { Source } from './import/sources.ts';
 
 // One self-contained, read-only HTML snapshot of a profile's memory.
+// Each import source's name in the page; a source without one shows its label as is.
+const SOURCES = { claude: 'Claude Code', 'claude-memory': 'Claude Code memory', codex: 'Codex', pi: 'Pi / OMP', chatgpt: 'ChatGPT' } satisfies Record<Source, string>;
+
 export function exportBrowser(memory: Memory, profile: string, directory = memory.directory) {
   const data = JSON.stringify({
-    profile, budget: memory.budget, size: memory.size, view: memory.view,
-    root: memory.root.map(({ i, kind, text, date, origin }) => ({ i, kind, text, date, ...(origin ? { origin: { source: origin.source, title: origin.title } } : {}) })),
-    tree: [...memory.tree.values()].map(({ l, i, text }) => ({ l, i, text })),
+    profile, sources: SOURCES, budget: memory.budget, size: memory.size, view: memory.view,
+    root: memory.root.map(({ i, kind, date, origin }) => ({ i, kind, text: memory.text(i), date, ...(origin ? { origin: { source: origin.source, title: origin.title } } : {}) })),
+    tree: [...memory.tree.values()].map(({ l, i }) => ({ l, i, text: memory.summary({ l, i }) })),
   }).replace(/</g, '\\u003c');
   const path = join(directory, 'memory.html'); atomicWrite(path, PAGE.replace('__DATA__', () => data)); return path;
 }
@@ -82,7 +86,7 @@ function when(a, b) {
   if (lo === hi) return day(x) + ', ' + time(x);
   return x.toDateString() === y.toDateString() ? day(x) + ', ' + time(x) + '–' + time(y) : day(x) + ' ' + time(x) + ' – ' + day(y) + ' ' + time(y);
 }
-const SOURCE = { claude: 'Claude Code', 'claude-memory': 'Claude Code memory', codex: 'Codex', pi: 'Pi / OMP', chatgpt: 'ChatGPT' };
+const SOURCE = D.sources;
 // Imported text starts with a "[Historical …]" line for the model; the meta line already says it.
 const body = e => { const j = e.origin && e.text.startsWith('[Historical ') ? e.text.indexOf(']\n') : -1; return j < 0 ? e.text : e.text.slice(j + 2); };
 const short = (s, n) => { s = s.replace(/\s+/g, ' ').trim(); return s.length > n ? s.slice(0, n - 1) + '…' : s; };
@@ -103,7 +107,7 @@ function mix(a, b) {
 function head(e) {
   const [label] = who(e), h = $('div', 'who');
   h.append($('b', null, label), ' · ' + when(e.i, e.i));
-  if (e.origin) h.append(' · imported from ' + SOURCE[e.origin.source] + (e.origin.title ? ': ' + short(e.origin.title, 60) : ''));
+  if (e.origin) h.append(' · imported from ' + (Object.hasOwn(SOURCE, e.origin.source) ? SOURCE[e.origin.source] : e.origin.source) + (e.origin.title ? ': ' + short(e.origin.title, 60) : ''));
   return h;
 }
 function message(e) {

package/src/compactor.ts

@@ -1,11 +1,7 @@
-import { setTimeout as sleep } from 'node:timers/promises';
-import { clampThinkingLevel, isRetryableAssistantError, type Api, type AssistantMessage, type Message, type Model } from '@earendil-works/pi-ai';
+import { clampThinkingLevel, isRetryableAssistantError, type Api, type Model } from '@earendil-works/pi-ai';
 import type { ThinkingLevel } from '@earendil-works/pi-agent-core';
 import type { ModelRegistry } from '@earendil-works/pi-coding-agent';
-import { COMPACT, compaction, IMPORT_GUIDANCE, tooLong } from './prompts.ts';
-import { bytes, cut, Refused, type Compressor } from './memory.ts';
-import { cachePayload, splitView } from './cache.ts';
-import { DEFAULT_SETTINGS } from './settings.ts';
+import { createCompressor as compressor, type CompressorOptions } from './core/compactor.ts';
 
 export interface ModelChoice { provider: string; model: string; thinking: ThinkingLevel }
 /** A level the model can't take would be sent as no level, which Sonnet 5.5 runs at high effort; Pi's own sessions clamp the same way. */
@@ -13,96 +9,13 @@ export const reasoningFor = (model: Model<Api>, level: ThinkingLevel) => {
   const thinking = clampThinkingLevel(model, level);
   return thinking === 'off' ? undefined : thinking;
 };
-const WARM_MS = 4 * 60_000; // Anthropic's short cache lives 5 minutes from its last use.
 
-/** Parallel calls can't read a cache entry that isn't written yet, so one call primes a cold prefix and the rest wait until it answers.
- * Parallel compactions end their views at different messages, so a call also waits for a primer of a shorter prefix of its own view. */
-function primeFirst() {
-  const warm = new Map<string, number | Promise<void>>();
-  return async (prefix: string, signal: AbortSignal) => {
-    const priming = () => { for (const [k, state] of warm) if (typeof state !== 'number' && prefix.startsWith(k)) return state; };
-    for (let state = warm.get(prefix) ?? priming(); state !== undefined; state = warm.get(prefix) ?? priming()) {
-      if (typeof state === 'number') { if (Date.now() - state < WARM_MS) break; warm.delete(prefix); continue; }
-      // A cancelled waiter leaves at once instead of waiting for someone else's primer.
-      signal.throwIfAborted();
-      let wake = () => {};
-      const aborted = new Promise<void>(resolve => { wake = resolve; });
-      signal.addEventListener('abort', wake, { once: true });
-      await Promise.race([state, aborted]);
-      signal.removeEventListener('abort', wake);
-      signal.throwIfAborted();
-    }
-    let release = () => {};
-    const pending = warm.has(prefix) ? undefined : new Promise<void>(resolve => { release = resolve; });
-    if (pending) warm.set(prefix, pending);
-    return (ok: boolean) => {
-      if (ok) { for (const [k, at] of warm) if (typeof at === 'number' && Date.now() - at >= WARM_MS) warm.delete(k); warm.set(prefix, Date.now()); }
-      else if (warm.get(prefix) === pending) warm.delete(prefix);
-      release();
-    };
-  };
-}
-/** Pi's agent loop retries a dropped connection itself; streamSimple doesn't, so a transient error is asked again before it surfaces. */
-export async function retryTransient(ask: () => Promise<AssistantMessage>, onReply: (reply: AssistantMessage) => void,
-  delays: readonly number[] = [1000, 2000], signal?: AbortSignal) {
-  for (let retry = 0; ; retry++) {
-    const reply = await ask();
-    onReply(reply);
-    if (retry >= delays.length || !isRetryableAssistantError(reply)) return reply;
-    await sleep(delays[retry], undefined, { signal });
-  }
-}
-
-/** The model is asked for 512 bytes; `accepted` is the longest line kept without a retry (the profile's summary size tolerance). */
-export function createCompressor(registry: ModelRegistry, choice: () => ModelChoice,
-  onUsage: (message: AssistantMessage) => void = () => {}, accepted = () => DEFAULT_SETTINGS.summaryAcceptBytes,
-  retryDelays?: readonly number[]): Compressor {
-  const gate = primeFirst();
-  return async (input, signal) => {
-    const selected = choice();
-    const model = registry.find(selected.provider, selected.model);
+/** The profile's compactor: `choice` is read for each call, through Pi's registry and its keys. `accepted` is the profile's
+ * summary size tolerance. */
+export const createCompressor = (registry: ModelRegistry, choice: () => ModelChoice,
+  options: Omit<CompressorOptions, 'model' | 'retryable'> = {}) => compressor({ ...options, retryable: isRetryableAssistantError,
+  model: () => {
+    const selected = choice(), model = registry.find(selected.provider, selected.model);
     if (!model) throw new Error(`Compactor model unavailable: ${selected.provider}/${selected.model}. Use /optchat model.`);
-    const thinking = reasoningFor(model, selected.thinking);
-    const step = `${input.historical ? IMPORT_GUIDANCE + '\n\n' : ''}${compaction(input)}`;
-    // OpenAI joins text blocks with nothing between them, so the task starts its own paragraph after the view.
-    const messages: Message[] = [{ role: 'user', content: [{ type: 'text', text: input.context }, { type: 'text', text: `\n\n${step}` }], timestamp: Date.now() }];
-    const view = splitView(input.context);
-    const prefix = model.api === 'anthropic-messages' && view.length > 1 ? `${model.provider}/${model.id}/${thinking ?? 'off'}\n${view.slice(0, -1).join('')}` : undefined;
-    const tries: string[] = [];
-    const ask = async () => {
-      const warmed = prefix ? await gate(prefix, signal) : () => {};
-      try {
-        const stream = registry.streamSimple(model, { systemPrompt: COMPACT, messages }, {
-          // A shared session id is the OpenAI prompt-cache key; SSE because over a websocket Codex would chain unrelated parallel calls on one cached connection.
-          sessionId: 'optchat-compactor', transport: 'sse',
-          reasoning: thinking, signal, cacheRetention: 'short',
-          onPayload: payload => model.api === 'anthropic-messages' ? cachePayload(payload, { end: false }) : payload,
-        });
-        // The cache entry is usable once the model starts answering.
-        for await (const event of stream) if (event.type !== 'start') { warmed(event.type !== 'error'); break; }
-        return await stream.result();
-      } finally { warmed(false); }
-    };
-    const answer = () => retryTransient(ask, onUsage, retryDelays, signal);
-    for (let attempt = 0; attempt < 5; attempt++) {
-      const reply = await answer();
-      // A stop reason means the model answered: a refusal or a content filter, which asking again repeats.
-      // A provider's own 'error' stop (Mistral) is a failure on its side, not an answer: a plain error, tried again later.
-      if (reply.stopReason === 'error' && reply.rawStopReason !== undefined && reply.rawStopReason !== 'error' && !isRetryableAssistantError(reply))
-        throw new Refused(reply.errorMessage ?? `Compactor stopped: ${reply.rawStopReason}`);
-      if (reply.stopReason === 'error' || reply.stopReason === 'aborted') throw new Error(reply.errorMessage ?? `Compactor ${reply.stopReason}`);
-      // A reply cut off at the token limit, or one that calls a tool, is no summary: ask again.
-      if (reply.stopReason === 'length' || reply.content.some(c => c.type === 'toolCall')) continue;
-      // The compactions' view shows each line under its id+n| head, which a line can copy.
-      const line = reply.content.filter(c => c.type === 'text').map(c => c.text).join('').trim().replace(/^\d+\+\d+\|\s*/, '');
-      if (!line) throw new Refused('Compactor returned no text.');
-      tries.push(line);
-      // A merge of two short lines can come back nearly as big as both, so a line must also shrink what it replaces.
-      if (bytes(line) <= accepted() && bytes(line) < bytes(input.source)) break;
-      messages.push(reply);
-      messages.push({ role: 'user', content: tooLong(bytes(line), cut(line)), timestamp: Date.now() });
-    }
-    if (!tries.length) throw new Refused('Compactor returned no whole line in 5 tries.');
-    return tries.reduce((a, b) => bytes(a) <= bytes(b) ? a : b);
-  };
-}
+    return { model, reasoning: reasoningFor(model, selected.thinking), stream: (model, context, options) => registry.streamSimple(model, context, options) };
+  } });

package/src/core/compactor.ts

@@ -0,0 +1,119 @@
+import type { Api, AssistantMessage, Message, Model, Models, SimpleStreamOptions } from '@earendil-works/pi-ai';
+import { COMPACT, compaction, IMPORT_GUIDANCE, tooLong } from './prompts.ts';
+import { bytes, cut, NODE, Refused, type Compressor } from './memory.ts';
+import { cachePayload, splitView } from './cache.ts';
+
+/** The longest line kept without asking again, unless the host sets its own (Pi's "Summary size tolerance"). */
+export const ACCEPTED_BYTES = NODE * 1.25;
+const WARM_MS = 4 * 60_000; // Anthropic's short cache lives 5 minutes from its last use.
+
+/** Parallel calls can't read a cache entry that isn't written yet, so one call primes a cold prefix and the rest wait until it answers.
+ * Parallel compactions end their views at different messages, so a call also waits for a primer of a shorter prefix of its own view. */
+function primeFirst() {
+  const warm = new Map<string, number | Promise<void>>();
+  return async (prefix: string, signal: AbortSignal) => {
+    const priming = () => { for (const [k, state] of warm) if (typeof state !== 'number' && prefix.startsWith(k)) return state; };
+    for (let state = warm.get(prefix) ?? priming(); state !== undefined; state = warm.get(prefix) ?? priming()) {
+      if (typeof state === 'number') { if (Date.now() - state < WARM_MS) break; warm.delete(prefix); continue; }
+      // A cancelled waiter leaves at once instead of waiting for someone else's primer.
+      signal.throwIfAborted();
+      let wake = () => {};
+      const aborted = new Promise<void>(resolve => { wake = resolve; });
+      signal.addEventListener('abort', wake, { once: true });
+      await Promise.race([state, aborted]);
+      signal.removeEventListener('abort', wake);
+      signal.throwIfAborted();
+    }
+    let release = () => {};
+    const pending = warm.has(prefix) ? undefined : new Promise<void>(resolve => { release = resolve; });
+    if (pending) warm.set(prefix, pending);
+    return (ok: boolean) => {
+      if (ok) { for (const [k, at] of warm) if (typeof at === 'number' && Date.now() - at >= WARM_MS) warm.delete(k); warm.set(prefix, Date.now()); }
+      else if (warm.get(prefix) === pending) warm.delete(prefix);
+      release();
+    };
+  };
+}
+const sleep = (ms: number, signal?: AbortSignal) => new Promise<void>((resolve, reject) => {
+  signal?.throwIfAborted();
+  const abort = () => { clearTimeout(timer); reject(signal?.reason); };
+  const timer = setTimeout(() => { signal?.removeEventListener('abort', abort); resolve(); }, ms);
+  signal?.addEventListener('abort', abort, { once: true });
+});
+/** Whether a failed reply is worth asking again: pi-ai's `isRetryableAssistantError`, passed in by the host. */
+export type Retryable = (reply: AssistantMessage) => boolean;
+/** Pi's agent loop retries a dropped connection itself; streamSimple doesn't, so a transient error is asked again before it surfaces. */
+export async function retryTransient(ask: () => Promise<AssistantMessage>, onReply: (reply: AssistantMessage) => void, retryable: Retryable,
+  delays: readonly number[] = [1000, 2000], signal?: AbortSignal) {
+  for (let retry = 0; ; retry++) {
+    const reply = await ask();
+    onReply(reply);
+    if (retry >= delays.length || !retryable(reply)) return reply;
+    await sleep(delays[retry], signal);
+  }
+}
+
+/** The compactor's model for one call, as the host resolves it: `stream` is pi-ai's `Models.streamSimple`, or Pi's
+ * `ModelRegistry.streamSimple`, which has its shape. */
+export interface CompactorModel { model: Model<Api>; reasoning?: SimpleStreamOptions['reasoning']; stream: Models['streamSimple'] }
+export interface CompressorOptions {
+  /** Resolves the model for each call, so a settings change applies to the next summary. Throws if it is unavailable. */
+  model: () => CompactorModel;
+  retryable: Retryable;
+  onUsage?: (reply: AssistantMessage) => void;
+  /** The longest line kept without a retry; ACCEPTED_BYTES unless the host has a setting for it. */
+  accepted?: () => number;
+  retryDelays?: readonly number[];
+  /** The compactor's system prompt: COMPACT, unless the host's chat isn't one user and OptChat. */
+  system?: string;
+}
+
+/** Builds summary lines with the model `model` resolves. The model is asked for 512 bytes. */
+export function createCompressor({ model: resolve, retryable, onUsage = () => {}, accepted = () => ACCEPTED_BYTES, retryDelays,
+  system = COMPACT }: CompressorOptions): Compressor {
+  const gate = primeFirst();
+  return async (input, signal) => {
+    const { model, reasoning: thinking, stream: streamSimple } = resolve();
+    const step = `${input.historical ? IMPORT_GUIDANCE + '\n\n' : ''}${compaction(input)}`;
+    // OpenAI joins text blocks with nothing between them, so the task starts its own paragraph after the view.
+    const messages: Message[] = [{ role: 'user', content: [{ type: 'text', text: input.context }, { type: 'text', text: `\n\n${step}` }], timestamp: Date.now() }];
+    const view = splitView(input.context);
+    const prefix = model.api === 'anthropic-messages' && view.length > 1 ? `${model.provider}/${model.id}/${thinking ?? 'off'}\n${view.slice(0, -1).join('')}` : undefined;
+    const tries: string[] = [];
+    const ask = async () => {
+      const warmed = prefix ? await gate(prefix, signal) : () => {};
+      try {
+        const stream = streamSimple(model, { systemPrompt: system, messages }, {
+          // A shared session id is the OpenAI prompt-cache key; SSE because over a websocket Codex would chain unrelated parallel calls on one cached connection.
+          sessionId: 'optchat-compactor', transport: 'sse',
+          reasoning: thinking, signal, cacheRetention: 'short',
+          onPayload: payload => model.api === 'anthropic-messages' ? cachePayload(payload, { end: false }) : payload,
+        });
+        // The cache entry is usable once the model starts answering.
+        for await (const event of stream) if (event.type !== 'start') { warmed(event.type !== 'error'); break; }
+        return await stream.result();
+      } finally { warmed(false); }
+    };
+    const answer = () => retryTransient(ask, onUsage, retryable, retryDelays, signal);
+    for (let attempt = 0; attempt < 5; attempt++) {
+      const reply = await answer();
+      // A stop reason means the model answered: a refusal or a content filter, which asking again repeats.
+      // A provider's own 'error' stop (Mistral) is a failure on its side, not an answer: a plain error, tried again later.
+      if (reply.stopReason === 'error' && reply.rawStopReason !== undefined && reply.rawStopReason !== 'error' && !retryable(reply))
+        throw new Refused(reply.errorMessage ?? `Compactor stopped: ${reply.rawStopReason}`);
+      if (reply.stopReason === 'error' || reply.stopReason === 'aborted') throw new Error(reply.errorMessage ?? `Compactor ${reply.stopReason}`);
+      // A reply cut off at the token limit, or one that calls a tool, is no summary: ask again.
+      if (reply.stopReason === 'length' || reply.content.some(c => c.type === 'toolCall')) continue;
+      // The compactions' view shows each line under its id+n| head, which a line can copy.
+      const line = reply.content.filter(c => c.type === 'text').map(c => c.text).join('').trim().replace(/^\d+\+\d+\|\s*/, '');
+      if (!line) throw new Refused('Compactor returned no text.');
+      tries.push(line);
+      // A merge of two short lines can come back nearly as big as both, so a line must also shrink what it replaces.
+      if (bytes(line) <= accepted() && bytes(line) < bytes(input.source)) break;
+      messages.push(reply);
+      messages.push({ role: 'user', content: tooLong(bytes(line), cut(line)), timestamp: Date.now() });
+    }
+    if (!tries.length) throw new Refused('Compactor returned no whole line in 5 tries.');
+    return tries.reduce((a, b) => bytes(a) <= bytes(b) ? a : b);
+  };
+}

package/src/core/images.ts

@@ -0,0 +1,24 @@
+import type { ImageContent } from '@earendil-works/pi-ai';
+import type { Store } from './store.ts';
+
+/** How a message's image reads in memory: `[image <16 hex digits>]`, named by the host (Pi: a hash of the image as sent). */
+const REF = /\[image ([0-9a-f]{16})\]/g;
+
+export const isImage = (part: unknown): part is ImageContent => typeof part === 'object' && part !== null
+  && 'type' in part && part.type === 'image' && 'data' in part && typeof part.data === 'string' && 'mimeType' in part && typeof part.mimeType === 'string';
+
+/** Base64 in chunks: one String.fromCharCode call per image would pass the engine's argument limit. */
+const base64 = (data: Uint8Array) => {
+  let binary = '';
+  for (let i = 0; i < data.length; i += 0x8000) binary += String.fromCharCode(...data.subarray(i, i + 0x8000));
+  return btoa(binary);
+};
+/** An image's base64 as bytes. */
+export const decode = (data: string) => Uint8Array.from(atob(data), char => char.charCodeAt(0));
+
+/** The kept images a text refers to, in order, each once; ones never kept are skipped. */
+export async function loadImages(store: Store, text: string): Promise<ImageContent[]> {
+  const names = [...new Set(Array.from(text.matchAll(REF), match => match[1]))];
+  return (await Promise.all(names.map(name => store.image(name))))
+    .flatMap(kept => kept ? [{ type: 'image' as const, data: base64(kept.data), mimeType: kept.mimeType }] : []);
+}