@earendil-works/chord 1.3.0 → 1.5.0

package/docs/next/store/walkthrough.md

@@ -0,0 +1,193 @@
+---
+title: One Store
+kind: walkthrough
+kicker: A walkthrough · Chord v2 and p3 · 2026-10-21
+subtitle: How one transactional store keeps a p3 session's transcript, documents and plugin tables consistent, taught one concept at a time
+model: anthropic/claude-sonnet-5-5
+sources:
+  - packages/chord/docs/next/store/book.md
+  - packages/chord/docs/next/decisions.md
+  - apps/p2/docs/p3-reviews/store-review-summary.md
+  - apps/p2/docs/p3-reviews/store-bulk-migrations.md
+  - apps/p2/docs/p3-reviews/session-data-scenarios.md
+  - apps/p2/docs/p3-reviews/store-proposal.md
+  - packages/durable/src/storage/sqlite/storage.ts
+  - packages/durable/src/session/session.ts
+truth: The decision log (redesign-decisions.md, entries "The store", "S1" to "S17", "Context last") wins over everything; the book (book.md) is written from it; store-proposal.md and session-data-scenarios.md are earlier drafts and the discussion, superseded where the log differs; the Durable sources show today's code, before the S1 change.
+---
+
+:::tutor
+Audience: the owner of p3 and Chord, who decided this design in one long discussion and wants to understand it end to
+end, or a developer joining the work who knows TypeScript, SQLite and pi-durable but not this store. Use the book's
+running example throughout: Mario's session S1 on his laptop, the file-history plugin with its `edits` table, entry
+812, the rename of `a/` to `b/` over 10,000 rows, continuing on the desktop. Read the matching book chapter before each
+concept and check decisions in the log. Be concrete: code, short traces with seq numbers, small tables. Short messages;
+one figure per concept at most. When the reader asks why something was not done another way, answer from the book's
+"What We Tried and Dropped" chapter and the log.
+:::
+
+:::intro
+#### What this is about
+
+A p3 session must be self-contained: everything it owns lives in its own store and directory, so it can be exported,
+imported into another p3, synced to Radius and continued on another device. Three kinds of owners write into a
+session: Durable (the transcript), Chord objects (persisted documents) and plugins (their own tables). This walkthrough
+explains the store they all share and why it is built that way.
+
+#### The example
+
+Mario works in session S1 on his laptop. A plugin called **file-history** records every file edit in its table
+`edits`: path, the conversation entry that made it (say entry 812), the time. He renames the directory `a/` to `b/`
+(10,000 rows change). In the evening he continues S1 on his desktop; another day he exports it and a colleague imports
+it. Every concept follows this session through one mechanism.
+
+#### How it progresses
+
+Eleven concepts in four parts: the problem and what was dropped; how the store writes (commit queue, seq, adoption,
+completion); how it reads and stores documents and tables (text order, the documents engine, bulk operations and
+migrations; blobs and derived data); and how a session moves (sync, takeover, export) plus the rules left to plugin authors.
+:::
+
+# The problem {#problem part="Why one store"}
+Book "Before we begin" and chapter I. Must cover: P1 (a session is self-contained; the server only mirrors); three
+owners (Durable, Chord objects, plugins); the trace where file-history's own `history.sqlite` and Durable's log sync
+separately and entry 812 dangles or points to the wrong entry after takeover; what is missing is one order across
+storages. A short trace or a steps figure helps.
+
+:::check
+1. Both of S1's storages are consistent on their own. Why can the desktop still end up in a state the laptop never had?
++++ answers
+1. Because nothing orders the changes of one storage relative to the other: the desktop can receive history.sqlite up to the edit but Durable's log only up to the commit before it (or the reverse), a combination that never existed on the laptop.
+:::
+
+# What was dropped {#dropped}
+Book chapter II; log entry "The store" (its list of rejected options) and S16. Must cover the main dropped options and
+the reason for each: derived-only plugin data, opaque directories, export hooks, plugin record logs with apply, several
+storages with a shared position counter, recorded SQLite statements, raw SQL reads, an IndexedDB-only API; inside the
+store: generic JSON rows, store-maintained index entries, code point order with replaced surrogates, text equality only.
+Point out what the final design kept from them (one order, recorded operations, a small query language).
+
+:::check
+1. Why was "record the plugin's SQL statements and replay them on the replica" dropped, while bulk operations are now recorded as operations and replayed?
++++ answers
+1. Arbitrary SQL can produce different results on a replica (random(), datetime('now'), unordered INSERT ... SELECT, other SQLite versions); the store's bulk operations use only the store's own expressions, specified once and identical on every backend, plus a row count check.
+:::
+
+# The store and its tables {#store part="Writing"}
+Book chapter III; log "The store", S14, S16. Must cover: layers (owners, store with documents engine, tables and commit
+step, backends); a table belongs to one object (address host, object type, table name); `defineTable` with typed key
+and indexed columns (`str` or `int`) in a flat `columns` record of real columns, nesting only inside `json<T>()` columns (typed but not queryable); one transaction API
+`self.tx(fn, ctx)`; index definitions are recorded in the outbox (contents never travel); Durable keeps its storages and gains `StoreStorage`; context last.
+
+:::check
+1. In `edits`, which fields can a query filter on, and where does the rest of the row live?
++++ answers
+1. Every scalar column (str, int, bool, num) is a real column and can be filtered; keys and indexes need str or int; json(...) columns hold JSON text, typed when read but not queryable or indexable.
+:::
+
+# Lines, the commit queue and the seq {#commit}
+Book chapter IV; log S1, S3, S9, S10. Must cover: lines (per Chord object, per Durable Session) serialize whole
+transactions, the commit queue serializes only the short final step; one seq per store, Durable sees gaps and only
+compares seqs; adoption inside the step with `Storage.commit(writes, adopt, context)`; why (the trace where Durable
+adopts 41 before file-history adopts 40); listeners still run afterwards and read a complete prefix; a commit is done
+only on the backend's terminal success; ambiguous outcomes stop the store; ids derived at open. The book's sequence
+figure of two objects and a Durable turn helps.
+
+:::check
+1. Object A commits seq 40 and Durable commits 41. Without S1, how could a routine see 41 but not 40?
+2. Why is it harmless that Durable's seqs now have gaps?
++++ answers
+1. Each owner adopted in its own continuation after the disk commit; Durable's continuation ran first, so a routine triggered by 41 read A's in-memory state still at 39. With S1 both adoptions run inside the commit step, in seq order.
+2. Durable only compares seqs (created_at <= at, seq > base); it never computes seq + 1 to find a neighbouring commit.
+:::
+
+# Reading and text order {#reading part="Reading and storing"}
+Book chapter V; log S4, S12, S16. Must cover: the small query language (where operators, orderBy, limit, cursor; no
+joins, grouping, projections at first); JavaScript and SQLite order strings differently (the ～ vs 😀 example); the
+decision: JavaScript order everywhere, SQLite stores key and indexed text as UTF-16 BLOBs (big-endian), lossless,
+measured costs; no normalization; cursors carry direction and logical values; multi-query reads in one read
+transaction, no global snapshot.
+
+:::check
+1. A plugin pages through `edits` by path with a cursor. Why would it break if SQLite stored paths as ordinary TEXT while the memory backend compared JS strings?
+2. What does the UTF-16 BLOB encoding cost, according to the measurements?
++++ answers
+1. SQLite would order by UTF-8 bytes and JS by UTF-16 code units; they disagree for characters between U+E000 and U+FFFF versus emoji, so a cursor taken from one order skips or repeats rows in the other (and tests on the memory backend would not show what SQLite does).
+2. Mostly JS encoding and decoding plus twice the bytes for ASCII: about +40% on bulk inserts, +62% file size in the test; SQLite's own range scans are about 4% slower when values are not decoded from the BLOBs.
+:::
+
+# The documents engine {#documents}
+Book chapter VI; log S11, S13. Must cover: Durable and Chord persist documents the same way, so the store has one
+engine; address (namespace, owner, kind, key) with an opaque owner string filled by each adapter (Durable: session,
+conversation:7, task:31; Chord: host/object type); find, load, list outside the step, create, change, retire inside it;
+load reconstructs base plus deltas once, the engine keeps nothing in memory; Durable's Session caches and evicts as
+today; fork copies are reconstructed by StoreStorage before the commit step.
+
+:::check
+1. Who decides that a conversation's documents fork and rewind with it: the documents engine or Durable's adapter?
++++ answers
+1. Durable's adapter (StoreStorage): the engine only sees an opaque owner string such as "conversation:7"; the meaning of that owner stays with Durable.
+:::
+
+# Bulk operations and migrations {#bulk}
+Book chapter VII; log S17; store-bulk-migrations.md. Must cover: the five writes (put, update, delete, updateWhere,
+deleteWhere); the rename of a/ to b/ as one recorded operation with a row count (about 150 bytes instead of about 40
+MB), compiled to one SQL UPDATE on the UTF-16 BLOB column; the replica runs the same operation and compares the count;
+the set expressions and why they are limited (identical behaviour on every backend; no JSON operands yet); migrations:
+changes inside json(...) columns on access via migrate(row, from), column changes as recorded alters, key and indexed columns eagerly by recorded table changes plus an update
+step or a rows step (scan + tx.update in chunks, migrating then ready, resume after a crash). The book's migration steps
+figure helps.
+
+:::check
+1. Version 4 of `edits` adds an indexed column `dir` computed with dirname(path). Why can this not migrate on access, and how does it run instead?
+2. A replica applies the rename and affects 9,998 rows instead of 10,000. What happens?
++++ answers
+1. An index over a column filled only on access would silently miss unmigrated rows; so it runs eagerly on the driver before the object goes live: a recorded alter adds the column, then a rows step scans in key order and writes dir with tx.update, chunk by chunk, until the table is marked ready.
+2. The counts differ, so the replica stops applying that store and restores it from a snapshot: divergence is detected, not silent.
+:::
+
+# Blobs and derived data {#easy}
+Book chapter VIII "The Easy Parts"; log "Blobs and derived data". Must cover: blobs are app-layer files named by the
+hash of their content, written before the row or document that names them; export copies them, import checks hashes;
+sync uploads every blob present and not yet uploaded before each batch (no reference tracking); no deletion before the
+session is deleted; derived data is never shipped and is repaired by a catch-up routine (the artifacts full-text index
+example: normal save, crash after commit, import).
+
+:::check
+1. Why does the sync layer not need to know which rows reference which blobs?
+2. The process dies between the artifact document's commit and the index update. What fixes the index?
++++ answers
+1. A blob is always written before the commit that names it, so uploading every blob present before each batch guarantees that any hash a batch names has its file on Radius already.
+2. The catch-up routine on the next start: it compares the index with the documents, finds version 3 missing and indexes it from the blob.
+:::
+
+# Sync and takeover {#sync part="Moving a session"}
+Book chapter IX; log S3, S5, S6, S7. Must cover: the outbox record is written in the same backend transaction;
+parallel uploads, resubmit until acknowledged, trim after acknowledgement; a replica applies whole batches in order
+(from equals its applied head), later batches wait; compaction is safe because batches apply whole; pull-back after a
+power loss; losing the driver role splits the session (new id) without rollback, and why rollback is impossible.
+
+:::check
+1. The laptop committed 40 and 41 locally after the desktop took over. What happens to those commits?
++++ answers
+1. Nothing is rolled back: when Radius refuses the laptop's upload for an old epoch, the laptop's copy becomes a separate session with a new id, keeps 40 and 41, and never syncs with S1 again; S1 continues on the desktop.
+:::
+
+# Export, import and the rules left to plugins {#export}
+Book chapters IX ("Export and import") and X; log S2, S4, S8. Must cover: export holds the commit queue so the
+archive is one head, includes tables of plugins that are not installed, never derived data; import restores into a
+fresh store and the server mirrors the catalog; rules: nothing leaves a transaction before it commits (reference the
+tool-call entry, not the tool-result entry); work that finishes out of order is the app's concern (routine handlers
+never overlap: chain or drop stale results); text compares like JS strings; prefer updateWhere over loops of puts.
+
+:::check
+1. The file-history plugin wants to store which entry made an edit. Which entry can it safely reference, and why?
++++ answers
+1. The assistant entry with the tool call: it is committed before the tool runs. The tool-result entry does not exist yet while the tool runs, and its id is handed out only inside the harness's later transaction.
+:::
+
+# What comes next {#next part="Where it stands"}
+Book "What's Next"; log entries. Must cover: everything decided on 2026-10-21; the pi change (adopt argument, Session
+rejection handling, mintId wording, traceIn context last); the store in packages/chord/src/next (about 1,650 lines
+estimated); StoreStorage passing Durable's conformance suite; the sync layer later; open points (new id of a split
+session, chunk size, whether IndexedDB is needed while sqlite-wasm covers the browser).

package/docs/spec.md

@@ -3,7 +3,7 @@
 This document specifies the runtime in `src`. Where it and the code disagree, the disagreement is a bug. [Known limitations and decisions](#12-known-limitations-and-decisions) lists what the runtime does not do yet and the choices the sections below leave open.
 
 Hard constraints:
-- Chord's runtime depends only on its own `delta` module (tracker, applyImmutable, encoder/decoder, diff) and Node/browser built-ins plus `typebox` for schemas. No other Pi packages.
+- Chord's runtime depends only on its own `delta` module (tracker, applyImmutable, encoder/decoder, diff) and Node/browser built-ins, plus `@earendil-works/pi-telemetry` for telemetry and `@earendil-works/pi-schema` for schemas. No other Pi packages.
 - Built-in plugins are trusted, loaded from source in-process. The plugin loader uses dynamic `import()` with a cache-busting query (owner-approved exception, `TODO(loader)`: replace with an isolating, evicting loader; old module copies leak; TUI plugins need isolation).
 - Hot reload is required. Host restart only for Chord/runtime code changes or host-level crash recovery.
 
@@ -29,23 +29,23 @@ Contracts are runtime values for every type (shipped bundles have no TypeScript;
 ```ts
 export const Chat = remote("pi.chat", {
 	version: 1,                                   // contract version: a plain number, bumped only on breaking changes
-	input: Type.Object({ session: Type.String(), conversation: Type.Number() }),
+	input: obj({ session: str(), conversation: num() }),
 	host: hostFamily("session:", "session"),       // where its objects live: `session:` + input.session
 	variants: {
 		viewer: { state: { view: ViewSchema } },
 		participant: {
 			state: { view: ViewSchema },
-			methods: { send: method(SendSchema, Type.Object({ submissionId: Type.Number() })) },
+			methods: { send: method(SendSchema, obj({ submissionId: num() })) },
 		},
 	},
 });
 
 export const Index = local("pi.index", {
 	version: 1,
-	input: Type.Object({ session: Type.String() }),
+	input: obj({ session: str() }),
 	members: {
-		search: method(Type.Object({ query: Type.String() }), Type.Array(Hit)),
-		onChange: method(Type.Function([Change], Type.Void()), Unsubscribe),   // non-JSON allowed for locals
+		search: method(obj({ query: str() }), arr(Hit)),
+		onChange: method({ type: "function", parameters: [Change], returnType: { type: "void" } }, Unsubscribe),   // non-JSON allowed for locals
 		harness: opaque("pi.durable-harness@1"),                               // native object, by name+version
 	},
 });
@@ -81,7 +81,7 @@ Factories are synchronous; returning a promise throws. A declaration has a name
 host.serve(StatusStore, ({ session }, _host, self) => {
 	const status = self.state("status", StatusSchema, { running: false, tasks: 0 },  // schema required, initial value
 		{ version: 2, migrate: (old, from) => upgrade(old, from) });                     // per cell; default version 1
-	const errors = self.state("errors", Type.Array(Type.String()), []);
+	const errors = self.state("errors", arr(str()), []);
 	const catalog = self.get(Catalog, {});                         // reference; identity = canonical address (+ at)
 	const watch = self.resource("watch", {
 		version: 1, deps: [catalog], exclusive: false,
@@ -278,8 +278,8 @@ The tests cover at least: lifetime/references/linger/forced disposal/gone; activ
 - **Joining during a reload.** Participants are the child hosts in step when a reload starts. A child whose hello arrives while it runs is decided when it ends: in step, or reconciled to the committed code version; a child already behind is fenced at once, so the calls its roots make meanwhile are admitted in the parent's code version. A reconciliation that is running when a reload starts syncs its child only after that reload ended, against the code version it committed. A reconciliation announces the parent's retired code versions before its `prepare`.
 - **Two-version rule.** An object of a changed type created after a commit has only new code; a previous-code-version root calling it gets `superseded`.
 - **Validation.** Remote calls are validated at reference dispatch, in-host too: a direct binding against the provider's schema, a binding through an adapter against the consumer's. A provider validates each snapshot and reset of a remote state member once per (re)subscription, against the consumer's contract through an `accepts` adapter and the provider's otherwise; ops are validated only in dev mode. At reload, the new provider's description must cover the old provider's when the version is unchanged, in addition to the check against live consumers and facets' `uses`.
-- **Contracts.** Local contracts list their members, so a loading reference knows whether a member is a method, state or opaque. Canonical identity omits input fields equal to their default; with TypeBox, use `Type.Optional(Type.X({ default }))` to let callers omit them. Provider variants must cover consumer variants; members the consumer's contract does not list are hidden. Schema change detection compares canonical JSON, so reordering `anyOf` branches counts as a change. Fingerprints use a non-cryptographic 64-bit hash: synchronous and browser-safe; contracts are trusted.
-- **Principals.** A root ctx at ingress acts as its principal (`host.root(fn, { principal })`); contexts inside objects act as `host`. The acceptor decides a link's kind and name (`accept(link, { principal, kind, name })`); only configured host links are addressable, carry roots and may vote, and control frames are accepted only from the parent. Clients never carry roots or code versions. Relaying needs a matching `route`; `authorize(principal, hostName)` runs before `resolve` and again after, and revoking or re-stamping a principal closes its relays at once. `invalidatePrincipal(match)` re-stamps or closes matching links and propagates pattern matches to child hosts.
+- **Contracts.** Local contracts list their members, so a loading reference knows whether a member is a method, state or opaque. Canonical identity omits input fields equal to their default; use `opt(str({ default }))` (pi-schema) to let callers omit them. Provider variants must cover consumer variants; members the consumer's contract does not list are hidden. Schema change detection compares canonical JSON, so reordering `anyOf` branches counts as a change. Fingerprints use a non-cryptographic 64-bit hash: synchronous and browser-safe; contracts are trusted.
+- **Principals.** A root ctx at ingress acts as its principal (`host.root(fn, { principal })`); contexts inside objects act as `host`. The acceptor decides a link's kind and name (`accept(link, { principal, kind, name })`); only configured host links are addressable, carry roots and may vote, and control frames are accepted only from the parent. Clients never carry roots or code versions. Relaying needs a matching `route`; `authorize(principal, hostName)` runs before `resolve` and again after, and revoking or re-stamping a principal closes its relays at once. The host `resolve` returns is admitted before that second check, so a refused, revoked or released relay never leaves a spawned host's link unread. `invalidatePrincipal(match)` re-stamps or closes matching links and propagates pattern matches to child hosts.
 - **Cells and readables.** A value returned from `change` replaces the value and is always validated; `replace` transfers ownership without copying; by default (`{ diff }` overrides) it publishes a diff when the new value reuses a container of the old one at the same key, else `r`. `subscribe(fn)` delivers values, starting with the current one; `events(fn)` delivers `snap`, then `ops`, or `reset`. Reference state members are stable readables that throw `not_ready` while loading and `not_in_variant` outside the variant, and reset when their source moves; a listener added before they are readable gets its snapshot then. Remote state members stay hydrated while the reference is held.
 - **References and calls.** A reference's identity is its address; a second `self.get` of the same address in one run returns the same reference. A reference call without ctx, or with a task ctx, starts a fresh root in the current code version; a ctx whose call or root already settled is refused (`aborted/expired`), one from another host `invalid/foreign_ctx`. Calling a reference inside a factory throws `not_ready`.
 - **Resources and disposal.** New and replaced exclusive resources open at the gate, and so do non-exclusive resources of the same object that depend on them. A new object for an address that is still being disposed waits for that disposal. A resource open that completes after its signal aborted closes its value at once.

package/docs/tutorial.md

@@ -45,11 +45,12 @@ shuts down its objects and connections.
 
 Before callers can use an object, they need to agree on its name and members.
 That agreement is a **contract**: a runtime value that also gives TypeScript its
-types. A **schema** describes a value's shape. We use TypeBox's `Type` to write
-schemas. `method(args, result)` declares a method by its argument and result
-schemas: `method()` takes no arguments and returns nothing, and
-`method(undefined, Type.String())`, used below, takes no arguments and returns a
-string.
+types. A **schema** describes a value's shape: plain JSON Schema, which we write
+with the builders of `@earendil-works/pi-schema` (`obj`, `str`, `num`, ...;
+`{ open: true }` lets an object carry fields it does not name).
+`method(args, result)` declares a method by its argument and result schemas:
+`method()` takes no arguments and returns nothing, and
+`method(undefined, str())`, used below, takes no arguments and returns a string.
 
 Start with a label for a named list. `local` means only code inside this host
 can reach it. The input `{ list: "family" }` identifies which list we want;
@@ -59,8 +60,8 @@ it is not an argument to `label`.
 ```ts
 const TodoLabel = local("tutorial.todos", {
 	version: 1,
-	input: Type.Object({ list: Type.String() }),
-	members: { label: method(undefined, Type.String()) },
+	input: obj({ list: str() }, { open: true }),
+	members: { label: method(undefined, str()) },
 });
 ```
 
@@ -116,17 +117,17 @@ name, because it holds nothing a reload must keep.
 
 <!-- example:03-contract -->
 ```ts
-const Todo = Type.Object({ text: Type.String(), done: Type.Boolean() });
-const Rows = Type.Array(Todo);
+const Todo = obj({ text: str(), done: bool() }, { open: true });
+const Rows = arr(Todo);
 const TodoStore = local("tutorial.todos", {
 	version: 1,
-	input: Type.Object({ list: Type.String() }),
+	input: obj({ list: str() }, { open: true }),
 	members: {
-		label: method(undefined, Type.String()),
+		label: method(undefined, str()),
 		rows: Rows,
-		openCount: Type.Number(),
-		add: method(Type.Object({ text: Type.String() })),
-		finish: method(Type.Object({ index: Type.Integer() })),
+		openCount: num(),
+		add: method(obj({ text: str() }, { open: true })),
+		finish: method(obj({ index: int() }, { open: true })),
 		reset: method(Rows),
 	},
 });
@@ -230,13 +231,13 @@ to a declared resource, without reading the resource during the factory.
 ```ts
 const Database = local("tutorial.database", {
 	version: 1,
-	input: Type.Object({}),
+	input: obj({}, { open: true }),
 	members: { db: opaque<{ saved: string[] }>("tutorial.database@1") },
 });
 const SavedTodos = local("tutorial.saved-todos", {
 	version: 1,
-	input: Type.Object({ list: Type.String() }),
-	members: { add: method(Type.Object({ text: Type.String() })) },
+	input: obj({ list: str() }, { open: true }),
+	members: { add: method(obj({ text: str() }, { open: true })) },
 });
 const host = createHost({ self: "server" });
 let opens = 0;
@@ -338,13 +339,13 @@ puts every `Todos` object on the host named `server`. Section 6 uses that.
 ```ts
 const Todos = remote("tutorial.todo-view", {
 	version: 1,
-	input: Type.Object({ list: Type.String() }),
+	input: obj({ list: str() }, { open: true }),
 	host: hostNamed("server"),
 	variants: {
 		viewer: { state: { rows: Rows } },
 		editor: {
 			state: { rows: Rows },
-			methods: { add: method(Type.Object({ text: Type.String() })) },
+			methods: { add: method(obj({ text: str() }, { open: true })) },
 		},
 	},
 });
@@ -499,8 +500,8 @@ test; they are not stored application state.
 ```ts
 const Activity = local("tutorial.activity", {
 	version: 1,
-	input: Type.Object({}),
-	members: { seen: Type.Number() },
+	input: obj({}, { open: true }),
+	members: { seen: num() },
 });
 const host = createHost({ self: "server" });
 host.serve(TodoStore, todoFactory);
@@ -509,7 +510,7 @@ const seeded = Promise.withResolvers<void>();
 const stopped = Promise.withResolvers<void>();
 host.serve(Activity, (_input, _principal, self) => {
 	const todos = self.get(TodoStore, { list: "family" });
-	const seen = self.state("seen", Type.Number(), 0);
+	const seen = self.state("seen", num(), 0);
 	self.on("rows", todos.rows, (rows, ctx) => {
 		seen.replace(ctx, rows?.length ?? 0);
 		if (rows?.length === 1) observed.resolve();
@@ -570,7 +571,7 @@ default lifetime, `"referenced"`, are still created only when referenced.
 <!-- example:08 -->
 ```ts
 const Actions = defineRegistry<(ctx: Ctx) => Promise<number>>("tutorial.actions");
-const Menu = local("tutorial.menu", { version: 1, input: Type.Object({}) });
+const Menu = local("tutorial.menu", { version: 1, input: obj({}, { open: true }) });
 let click: (() => Promise<number>) | undefined;
 const storeFacet: Facet = {
 	name: "todo-store",
@@ -745,7 +746,7 @@ const { label: _label, ...members } = TodoStore.def.members;
 const RenamedStore = local("tutorial.todos", {
 	version: 2,
 	input: TodoStore.def.input,
-	members: { ...members, heading: method(undefined, Type.String()) },
+	members: { ...members, heading: method(undefined, str()) },
 });
 await host.reload([
 	{
@@ -890,7 +891,7 @@ tutorial's source-matching check, not by the application.
 ```ts
 import assert from "node:assert/strict";
 import { readFileSync } from "node:fs";
-import { type Static, Type } from "typebox";
+import { arr, bool, int, num, obj, type Static, str } from "@earendil-works/pi-schema";
 import { it, onTestFinished } from "vitest";
 import {
 	type Ctx,

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@earendil-works/chord",
-	"version": "1.3.0",
+	"version": "1.5.0",
 	"description": "Typed host runtime for plugins, contracts, replicated state, and RPC over JSON links",
 	"type": "module",
 	"main": "./dist/index.js",
@@ -31,6 +31,31 @@
 			"types": "./dist/node.d.ts",
 			"import": "./dist/node.js"
 		},
+		"./telemetry-sqlite": {
+			"source": "./src/telemetry-sqlite/index.ts",
+			"types": "./dist/telemetry-sqlite/index.d.ts",
+			"import": "./dist/telemetry-sqlite/index.js"
+		},
+		"./next": {
+			"source": "./src/next/index.ts",
+			"types": "./dist/next/index.d.ts",
+			"import": "./dist/next/index.js"
+		},
+		"./sqlite": {
+			"source": "./src/sqlite/index.ts",
+			"types": "./dist/sqlite/index.d.ts",
+			"import": "./dist/sqlite/index.js"
+		},
+		"./sqlite/node": {
+			"source": "./src/sqlite/node.ts",
+			"types": "./dist/sqlite/node.d.ts",
+			"import": "./dist/sqlite/node.js"
+		},
+		"./sqlite/cloudflare": {
+			"source": "./src/sqlite/cloudflare.ts",
+			"types": "./dist/sqlite/cloudflare.d.ts",
+			"import": "./dist/sqlite/cloudflare.js"
+		},
 		"./package.json": "./package.json"
 	},
 	"sideEffects": false,
@@ -42,7 +67,7 @@
 		"src/delta/README.md"
 	],
 	"scripts": {
-		"clean": "shx rm -rf dist",
+		"clean": "node ../../scripts/package-files.mjs clean dist",
 		"build": "tsc -p tsconfig.build.json",
 		"test": "vitest --run",
 		"prepublishOnly": "npm run clean && npm run build"
@@ -65,12 +90,24 @@
 		"node": ">=22.19.0"
 	},
 	"dependencies": {
-		"typebox": "1.3.27"
+		"@earendil-works/pi-schema": "1.5.0",
+		"@earendil-works/pi-telemetry": "1.5.0"
+	},
+	"peerDependencies": {
+		"esbuild": "0.28.2",
+		"ws": "8.21.0"
+	},
+	"peerDependenciesMeta": {
+		"esbuild": {
+			"optional": true
+		},
+		"ws": {
+			"optional": true
+		}
 	},
 	"devDependencies": {
 		"@types/ws": "8.18.2",
-		"shx": "0.3.4",
-		"vitest": "4.1.11",
+		"vitest": "4.1.12",
 		"ws": "8.21.0"
 	}
 }

package/src/delta/README.md

@@ -82,6 +82,35 @@ Parsing a separately serialized message per replica is also a valid detachment.
 `decode()` does not copy payloads, so decoding one in-memory `WireOp[]` for two
 mutable replicas makes them alias each other.
 
+## Slow consumers
+
+`PendingOps` holds the batches one consumer has not taken yet (a watch listener
+that is still running, a link whose window is full) and folds them while they
+wait, so what it hands over replays to the same value with fewer ops:
+
+```ts
+import { PendingOps } from "@earendil-works/chord/delta";
+
+const pending = new PendingOps();
+pending.push(prepared.ops); // for each adopted batch, while the consumer is busy
+// When the consumer is ready again:
+const batch = pending.take();
+const ops = batch.kind === "ops" ? batch.ops : [["r", tracker.value]];
+```
+
+- A leaf op (an append, or a set of a primitive) folds into the last pending leaf
+  op on its path: `a` + `a` join, `s` replaces `s` or `a`, an `a` after a string `s`
+  appends into it. `["a", ["text"], "he"]`, `["s", ["n"], 1]`, `["a", ["text"], "y"]`
+  becomes `["a", ["text"], "hey"]`, `["s", ["n"], 1]`.
+- Any other op (`p`, `m`, `d`, `t`, `s` of an object or array) ends folding into
+  the ops before it, as it may change what a path names; `r` drops them.
+- Past 10,000 pending ops after folding, only `replace` stays: the owner sends the
+  current value as `[["r", value]]`.
+- One batch is handed over as received; received batches and op tuples are never
+  mutated, so the same batch can feed other consumers.
+- It folds ops of the tracker, `diffRevisions()` and `appendText()`; it does not
+  validate or normalize hand-written ops.
+
 ## Do and don't
 
 - Do build a fresh root and then transfer it; validate untrusted roots at your ingestion boundary.
@@ -125,6 +154,8 @@ Growing `length` inserts `null`; shrinking it removes elements.
 
 `tracker.value` is the latest adopted revision. `beginChange()` opens an overlay
 draft over it and never modifies it. A draft may stay open across `await`.
+A draft dropped without `prepare()` or `abort()` stays referenced by its
+tracker until the tracker's next `beginChange()` or `adopt()`.
 `prepare()` materializes the candidate and ops and does not change authority.
 `adopt()` checks the preparation and swaps the root pointer.
 
@@ -181,9 +212,47 @@ and restores `Op` tuples. Use one encoder/decoder pair per ordered state stream.
 Path IDs span batches and an `r` resets both dictionaries. After a decode or
 apply error, discard the decoder and replica and recover from a later `r`.
 
+## Growing strings: use `appendText`
+
+The footgun: appending to a string through a draft costs the string's whole length on every change, not the length of
+what you append.
+
+```ts
+const change = tracker.beginChange();
+change.state.message.text += chunk;   // looks like O(chunk)
+const prepared = change.prepare();     // emits ["a", ["message", "text"], chunk], but costs O(text.length)
+```
+
+The draft's proxy only sees the finished new string, never the chunk, so `prepare()` recovers the append by checking
+that the new string starts with the old one. That check reads the whole string, and V8 first copies a string built by
+`+=` (a rope) into one flat piece. There is no fast path: measured on Node 24 and 26, 20,000 changes appending 40
+characters each, up to 800 KB, take about 2.4 s with the check and about 1 ms without it. A stream of N chunks into one
+string therefore costs O(N x length) in total: quadratic.
+
+The fix: hand Delta the chunk.
+
+```ts
+import { appendText } from "@earendil-works/chord/delta";
+
+appendText(change.state.message, "text", chunk);   // records ["a", path, chunk]: O(chunk), no compare, no copy
+```
+
+- `appendText(parent, key, text)` takes the parent draft and a key whose type is exactly `string` (not optional, not a
+  string literal type), so the compiler refuses `appendText(block, "type", …)`.
+- Several appends to one key in one change become one op. A plain assignment or deletion of the same key in that change
+  falls back to the compare, as does an append after a plain write.
+- The appended text is copied into a string of its own, so neither the op nor the new value keeps a larger string
+  alive.
+- Use it wherever a string grows by appends across many changes (streamed answers, tool output, logs). A one-off
+  assignment is fine with `=`.
+- Derives cannot use it: `diffRevisions()` compares two revisions it did not write, so a derive producing a growing
+  string pays O(length) per delivery.
+
 ## Limits and footguns
 
 - No freezing. Mutating any immutable value corrupts state silently.
+- Appending to a string through a draft (`draft.text += chunk`) costs O(string length) per change; use `appendText`
+  (section "Growing strings").
 - In-process sharing: local handles, cell subscribers, and `replicatedState` subscribers receive the provider's containers. A consumer mutation corrupts authority and makes remote replicas diverge.
 - A change inside a large flat array copies that array's pointer storage for the new revision.
 - Placement validation costs time and memory on bulk inserts. In a benchmark unshifting 100k three-field objects, it added about 19 ms and 20 MiB of transient heap.