@earendil-works/chord 1.5.0 → 1.5.1

package/docs/store/walkthrough.md

@@ -0,0 +1,193 @@
+---
+title: One Store
+kind: walkthrough
+kicker: A walkthrough · Chord v2 and p3 · 2026-10-21
+subtitle: How one transactional store keeps a p3 session's transcript, documents and plugin tables consistent, taught one concept at a time
+model: anthropic/claude-sonnet-5-5
+sources:
+  - packages/chord/docs/store/book.md
+  - packages/chord/docs/decisions.md
+  - apps/p2/docs/p3-reviews/store-review-summary.md
+  - apps/p2/docs/p3-reviews/store-bulk-migrations.md
+  - apps/p2/docs/p3-reviews/session-data-scenarios.md
+  - apps/p2/docs/p3-reviews/store-proposal.md
+  - packages/durable/src/storage/sqlite/storage.ts
+  - packages/durable/src/session/session.ts
+truth: The decision log (redesign-decisions.md, entries "The store", "S1" to "S17", "Context last") wins over everything; the book (book.md) is written from it; store-proposal.md and session-data-scenarios.md are earlier drafts and the discussion, superseded where the log differs; the Durable sources show today's code, before the S1 change.
+---
+
+:::tutor
+Audience: the owner of p3 and Chord, who decided this design in one long discussion and wants to understand it end to
+end, or a developer joining the work who knows TypeScript, SQLite and pi-durable but not this store. Use the book's
+running example throughout: Mario's session S1 on his laptop, the file-history plugin with its `edits` table, entry
+812, the rename of `a/` to `b/` over 10,000 rows, continuing on the desktop. Read the matching book chapter before each
+concept and check decisions in the log. Be concrete: code, short traces with seq numbers, small tables. Short messages;
+one figure per concept at most. When the reader asks why something was not done another way, answer from the book's
+"What We Tried and Dropped" chapter and the log.
+:::
+
+:::intro
+#### What this is about
+
+A p3 session must be self-contained: everything it owns lives in its own store and directory, so it can be exported,
+imported into another p3, synced to Radius and continued on another device. Three kinds of owners write into a
+session: Durable (the transcript), Chord objects (persisted documents) and plugins (their own tables). This walkthrough
+explains the store they all share and why it is built that way.
+
+#### The example
+
+Mario works in session S1 on his laptop. A plugin called **file-history** records every file edit in its table
+`edits`: path, the conversation entry that made it (say entry 812), the time. He renames the directory `a/` to `b/`
+(10,000 rows change). In the evening he continues S1 on his desktop; another day he exports it and a colleague imports
+it. Every concept follows this session through one mechanism.
+
+#### How it progresses
+
+Eleven concepts in four parts: the problem and what was dropped; how the store writes (commit queue, seq, adoption,
+completion); how it reads and stores documents and tables (text order, the documents engine, bulk operations and
+migrations; blobs and derived data); and how a session moves (sync, takeover, export) plus the rules left to plugin authors.
+:::
+
+# The problem {#problem part="Why one store"}
+Book "Before we begin" and chapter I. Must cover: P1 (a session is self-contained; the server only mirrors); three
+owners (Durable, Chord objects, plugins); the trace where file-history's own `history.sqlite` and Durable's log sync
+separately and entry 812 dangles or points to the wrong entry after takeover; what is missing is one order across
+storages. A short trace or a steps figure helps.
+
+:::check
+1. Both of S1's storages are consistent on their own. Why can the desktop still end up in a state the laptop never had?
++++ answers
+1. Because nothing orders the changes of one storage relative to the other: the desktop can receive history.sqlite up to the edit but Durable's log only up to the commit before it (or the reverse), a combination that never existed on the laptop.
+:::
+
+# What was dropped {#dropped}
+Book chapter II; log entry "The store" (its list of rejected options) and S16. Must cover the main dropped options and
+the reason for each: derived-only plugin data, opaque directories, export hooks, plugin record logs with apply, several
+storages with a shared position counter, recorded SQLite statements, raw SQL reads, an IndexedDB-only API; inside the
+store: generic JSON rows, store-maintained index entries, code point order with replaced surrogates, text equality only.
+Point out what the final design kept from them (one order, recorded operations, a small query language).
+
+:::check
+1. Why was "record the plugin's SQL statements and replay them on the replica" dropped, while bulk operations are now recorded as operations and replayed?
++++ answers
+1. Arbitrary SQL can produce different results on a replica (random(), datetime('now'), unordered INSERT ... SELECT, other SQLite versions); the store's bulk operations use only the store's own expressions, specified once and identical on every backend, plus a row count check.
+:::
+
+# The store and its tables {#store part="Writing"}
+Book chapter III; log "The store", S14, S16. Must cover: layers (owners, store with documents engine, tables and commit
+step, backends); a table belongs to one object (address host, object type, table name); `defineTable` with typed key
+and indexed columns (`str` or `int`) in a flat `columns` record of real columns, nesting only inside `json<T>()` columns (typed but not queryable); one transaction API
+`self.tx(fn, ctx)`; index definitions are recorded in the outbox (contents never travel); Durable keeps its storages and gains `StoreStorage`; context last.
+
+:::check
+1. In `edits`, which fields can a query filter on, and where does the rest of the row live?
++++ answers
+1. Every scalar column (str, int, bool, num) is a real column and can be filtered; keys and indexes need str or int; json(...) columns hold JSON text, typed when read but not queryable or indexable.
+:::
+
+# Lines, the commit queue and the seq {#commit}
+Book chapter IV; log S1, S3, S9, S10. Must cover: lines (per Chord object, per Durable Session) serialize whole
+transactions, the commit queue serializes only the short final step; one seq per store, Durable sees gaps and only
+compares seqs; adoption inside the step with `Storage.commit(writes, adopt, context)`; why (the trace where Durable
+adopts 41 before file-history adopts 40); listeners still run afterwards and read a complete prefix; a commit is done
+only on the backend's terminal success; ambiguous outcomes stop the store; ids derived at open. The book's sequence
+figure of two objects and a Durable turn helps.
+
+:::check
+1. Object A commits seq 40 and Durable commits 41. Without S1, how could a routine see 41 but not 40?
+2. Why is it harmless that Durable's seqs now have gaps?
++++ answers
+1. Each owner adopted in its own continuation after the disk commit; Durable's continuation ran first, so a routine triggered by 41 read A's in-memory state still at 39. With S1 both adoptions run inside the commit step, in seq order.
+2. Durable only compares seqs (created_at <= at, seq > base); it never computes seq + 1 to find a neighbouring commit.
+:::
+
+# Reading and text order {#reading part="Reading and storing"}
+Book chapter V; log S4, S12, S16. Must cover: the small query language (where operators, orderBy, limit, cursor; no
+joins, grouping, projections at first); JavaScript and SQLite order strings differently (the ～ vs 😀 example); the
+decision: JavaScript order everywhere, SQLite stores key and indexed text as UTF-16 BLOBs (big-endian), lossless,
+measured costs; no normalization; cursors carry direction and logical values; multi-query reads in one read
+transaction, no global snapshot.
+
+:::check
+1. A plugin pages through `edits` by path with a cursor. Why would it break if SQLite stored paths as ordinary TEXT while the memory backend compared JS strings?
+2. What does the UTF-16 BLOB encoding cost, according to the measurements?
++++ answers
+1. SQLite would order by UTF-8 bytes and JS by UTF-16 code units; they disagree for characters between U+E000 and U+FFFF versus emoji, so a cursor taken from one order skips or repeats rows in the other (and tests on the memory backend would not show what SQLite does).
+2. Mostly JS encoding and decoding plus twice the bytes for ASCII: about +40% on bulk inserts, +62% file size in the test; SQLite's own range scans are about 4% slower when values are not decoded from the BLOBs.
+:::
+
+# The documents engine {#documents}
+Book chapter VI; log S11, S13. Must cover: Durable and Chord persist documents the same way, so the store has one
+engine; address (namespace, owner, kind, key) with an opaque owner string filled by each adapter (Durable: session,
+conversation:7, task:31; Chord: host/object type); find, load, list outside the step, create, change, retire inside it;
+load reconstructs base plus deltas once, the engine keeps nothing in memory; Durable's Session caches and evicts as
+today; fork copies are reconstructed by StoreStorage before the commit step.
+
+:::check
+1. Who decides that a conversation's documents fork and rewind with it: the documents engine or Durable's adapter?
++++ answers
+1. Durable's adapter (StoreStorage): the engine only sees an opaque owner string such as "conversation:7"; the meaning of that owner stays with Durable.
+:::
+
+# Bulk operations and migrations {#bulk}
+Book chapter VII; log S17; store-bulk-migrations.md. Must cover: the five writes (put, update, delete, updateWhere,
+deleteWhere); the rename of a/ to b/ as one recorded operation with a row count (about 150 bytes instead of about 40
+MB), compiled to one SQL UPDATE on the UTF-16 BLOB column; the replica runs the same operation and compares the count;
+the set expressions and why they are limited (identical behaviour on every backend; no JSON operands yet); migrations:
+changes inside json(...) columns on access via migrate(row, from), column changes as recorded alters, key and indexed columns eagerly by recorded table changes plus an update
+step or a rows step (scan + tx.update in chunks, migrating then ready, resume after a crash). The book's migration steps
+figure helps.
+
+:::check
+1. Version 4 of `edits` adds an indexed column `dir` computed with dirname(path). Why can this not migrate on access, and how does it run instead?
+2. A replica applies the rename and affects 9,998 rows instead of 10,000. What happens?
++++ answers
+1. An index over a column filled only on access would silently miss unmigrated rows; so it runs eagerly on the driver before the object goes live: a recorded alter adds the column, then a rows step scans in key order and writes dir with tx.update, chunk by chunk, until the table is marked ready.
+2. The counts differ, so the replica stops applying that store and restores it from a snapshot: divergence is detected, not silent.
+:::
+
+# Blobs and derived data {#easy}
+Book chapter VIII "The Easy Parts"; log "Blobs and derived data". Must cover: blobs are app-layer files named by the
+hash of their content, written before the row or document that names them; export copies them, import checks hashes;
+sync uploads every blob present and not yet uploaded before each batch (no reference tracking); no deletion before the
+session is deleted; derived data is never shipped and is repaired by a catch-up routine (the artifacts full-text index
+example: normal save, crash after commit, import).
+
+:::check
+1. Why does the sync layer not need to know which rows reference which blobs?
+2. The process dies between the artifact document's commit and the index update. What fixes the index?
++++ answers
+1. A blob is always written before the commit that names it, so uploading every blob present before each batch guarantees that any hash a batch names has its file on Radius already.
+2. The catch-up routine on the next start: it compares the index with the documents, finds version 3 missing and indexes it from the blob.
+:::
+
+# Sync and takeover {#sync part="Moving a session"}
+Book chapter IX; log S3, S5, S6, S7. Must cover: the outbox record is written in the same backend transaction;
+parallel uploads, resubmit until acknowledged, trim after acknowledgement; a replica applies whole batches in order
+(from equals its applied head), later batches wait; compaction is safe because batches apply whole; pull-back after a
+power loss; losing the driver role splits the session (new id) without rollback, and why rollback is impossible.
+
+:::check
+1. The laptop committed 40 and 41 locally after the desktop took over. What happens to those commits?
++++ answers
+1. Nothing is rolled back: when Radius refuses the laptop's upload for an old epoch, the laptop's copy becomes a separate session with a new id, keeps 40 and 41, and never syncs with S1 again; S1 continues on the desktop.
+:::
+
+# Export, import and the rules left to plugins {#export}
+Book chapters IX ("Export and import") and X; log S2, S4, S8. Must cover: export holds the commit queue so the
+archive is one head, includes tables of plugins that are not installed, never derived data; import restores into a
+fresh store and the server mirrors the catalog; rules: nothing leaves a transaction before it commits (reference the
+tool-call entry, not the tool-result entry); work that finishes out of order is the app's concern (routine handlers
+never overlap: chain or drop stale results); text compares like JS strings; prefer updateWhere over loops of puts.
+
+:::check
+1. The file-history plugin wants to store which entry made an edit. Which entry can it safely reference, and why?
++++ answers
+1. The assistant entry with the tool call: it is committed before the tool runs. The tool-result entry does not exist yet while the tool runs, and its id is handed out only inside the harness's later transaction.
+:::
+
+# What comes next {#next part="Where it stands"}
+Book "What's Next"; log entries. Must cover: everything decided on 2026-10-21; the pi change (adopt argument, Session
+rejection handling, mintId wording, traceIn context last); the store in packages/chord/src (about 1,650 lines
+estimated); StoreStorage passing Durable's conformance suite; the sync layer later; open points (new id of a split
+session, chunk size, whether IndexedDB is needed while sqlite-wasm covers the browser).

package/docs/theming/book.md

@@ -0,0 +1,223 @@
+---
+title: Looks and Themes: how an app shares its styles and components with plugins
+cover: Looks / and Themes
+kicker: A proposal · Chord v2 · 2026-11-02
+subtitle: One app, a notes plugin and a stranger's theme plugin, on a browser page and in a terminal
+stats:
+  - 1 | registry for themes, shared by web and terminal
+  - 0 | copies of the app's components inside plugins
+  - 0 | page reloads to switch a theme
+note: |
+  A proposal by the redesign session, asked for by the owner on 2026-11-02 ("can you write a small explainer about
+  what you envision"). It builds on decided rules: the app's versioned API split by host kind, plugins importing only
+  that API, registries for what plugins add (decisions of 2026-11-01 in `decisions.md`). Nothing here is built; slice
+  11 (web and TUI) and slice 13 (the app kit) are where it would land.
+colophon: Looks and Themes · proposal · 2026-11-02
+---
+
+# Before we begin {.preface}
+
+Acme's notes app has a look: a sidebar, cards with rounded corners, an orange accent, a dark mode. Its own plugins
+draw panels inside that look, and so should a stranger's plugin. Then someone wants a Solarized theme, and writes it as
+a plugin. And the same app also runs in a terminal, where there are no CSS files and no React, only text and colours.
+
+Three questions follow. How does a plugin get the app's components and styles without carrying its own copy? How does
+it style its own parts so they fit, without stepping on other plugins? And how does a plugin contribute a theme that
+the app, every other plugin and both surfaces pick up?
+
+## Five things to know
+
+:::findings
+1. **Shared components come from the app's API.** The web module of the app's API (`@acme/app/v3/web`) exports them;
+   it is in the base bundle once, and every plugin uses that instance.
+2. **Looks are design tokens.** Colours, radii and fonts are CSS custom properties on the page (`--accent`, `--bg`);
+   the app's components and every plugin's CSS use them.
+3. **A plugin's own CSS is scoped.** CSS modules (`notes.module.css`) keep its class names its own; the build already
+   collects and ships each facet's stylesheet.
+4. **A theme is data in one registry.** `Themes` lives in the app's shared API module; an entry names colours for
+   roles (accent, background, text, ...). The web turns the active one into CSS variables, the terminal into colours.
+5. **A plugin contributes a theme like anything else.** It contributes an entry to `Themes`; the picker shows it,
+   choosing it restyles the page and the terminal at once, editing the plugin restyles them on its rollout.
+:::
+
+## Words used throughout
+
+The **app's API** is the versioned set of modules plugins may import (`@acme/app/v3`, shared by every host kind, and
+`@acme/app/v3/web`, `/v3/tui` for one kind). A **host package** is a module the running process provides once to all
+plugins. A **design token** is a named value of the look (the accent colour), written as a **CSS custom property**
+(`--accent`) on the web. A **role** is what a colour is for (accent, background, error), as opposed to the colour
+itself. A **theme** is one set of values for every role. A **registry** is an extension point plugins contribute to.
+
+# Shared Components {#components lede="Why plugins import the app's components instead of bundling their own."}
+
+A plugin's script is built without the app's API: imports of it are left as names and answered by the page at load
+time (decision "Plugins import only the app's versioned API"). So when the app exports components from its web module,
+every plugin gets the very same functions, from the base bundle the page loaded once:
+
+```ts
+// src/api/v3/web.ts (the app)
+export { Sheet, Button, List, Toolbar } from "../../web/components.tsx";
+export { Panels } from "./panels.web.ts";
+
+// plugins/notes/web.tsx
+import { defineFacet } from "@acme/app/v3";
+import { Panels, Sheet, Button } from "@acme/app/v3/web";
+
+export default defineFacet((host, self) => {
+	self.contribute(Panels, { title: "Notes", Component: () => <Sheet><Button>New note</Button></Sheet> });
+});
+```
+
+Two things follow. A plugin never ships a copy of the app's components, so a page with twenty plugins loads them once.
+And when the app improves a component, every plugin shows the improvement after the app's next restart, without being
+rebuilt. Components that change shape incompatibly get a new API version, as any API does.
+
+# Looks as Tokens {#tokens lede="One set of variables that the app's components and every plugin's CSS read."}
+
+The app's stylesheet, part of the base bundle, defines the look as CSS custom properties on the page's root:
+
+```css
+:root {
+	--accent: #e8590c;
+	--bg: #111317;
+	--fg: #e8e8e8;
+	--muted: #8a8f98;
+	--radius: 10px;
+	--font: "Inter", system-ui, sans-serif;
+}
+```
+
+The app's components use only these variables, never fixed colours. A plugin does the same in its own CSS:
+
+```css
+/* plugins/notes/notes.module.css */
+.card { background: color-mix(in srgb, var(--bg) 92%, var(--fg)); border-radius: var(--radius); }
+.title { color: var(--accent); font-family: var(--font); }
+```
+
+```tsx
+import styles from "./notes.module.css";
+<div className={styles.card}><h2 className={styles.title}>Shopping</h2></div>
+```
+
+Because everything reads the variables, changing the variables changes everything: the app's components and every
+plugin's parts, at once, without reloading anything. That is what makes themes cheap.
+
+## Keeping plugins apart
+
+Two plugins both writing `.card { ... }` would overwrite each other's styles. CSS modules solve that: esbuild gives each
+class in a `*.module.css` file a name unique to that file, and the import returns the mapping. The build already
+collects each web facet's CSS into one stylesheet, served next to its script and swapped when the plugin reloads, so
+nothing new is needed except the convention. Plain CSS files still work; their class names should then start with the
+plugin's name.
+
+# The Terminal {#tui lede="The same idea without CSS: components and colour roles from the app."}
+
+A terminal has no stylesheets, but the same split works. The app's terminal module exports its components, built on
+whichever terminal UI library the app chose, and a theme lookup by role:
+
+```ts
+// plugins/notes/tui.ts
+import { defineFacet } from "@acme/app/v3";
+import { Panels, Box, theme } from "@acme/app/v3/tui";
+
+const render = (notes) => Box({ title: theme.fg("accent", "Notes"), lines: notes.map((n) => theme.fg("fg", n.text)) });
+```
+
+A plugin never writes raw escape codes; it asks for a role (`accent`, `muted`, `error`), and the app's lookup turns
+the active theme's colour into what the terminal supports: true colour, or the nearest of 256 colours.
+
+# Themes as a Registry {#themes lede="One extension point, data only, read by both surfaces."}
+
+A theme names a colour for each role. Roles, not surfaces: an accent is an accent on a page and in a terminal. So the
+registry lives in the app's **shared** API module, and its entries are plain data, safe to import from any host kind:
+
+```ts
+// src/api/v3/index.ts (the app)
+export type Theme = {
+	id: string;
+	name: string;
+	mode: "dark" | "light";
+	colors: { accent: string; bg: string; fg: string; muted: string; error: string; warning: string; success: string };
+	web?: Readonly<Record<string, string>>;   // extra CSS variables: radius, fonts, shadows
+	tui?: Readonly<Record<string, string>>;   // terminal-only overrides: a 256-colour fallback for a role
+};
+export const Themes = defineRegistry<Theme, readonly Theme[]>("acme.themes", { version: 1 });
+```
+
+The app contributes its default themes from its own plugins, like any other contribution. Its theme picker lists the
+registry's current entries; the choice is a user preference.
+
+:::flow wide animate | A theme from a plugin reaches the page and the terminal through one registry.
+row "Plugins" muted
+box sol "solarized plugin" "contributes a Theme"
+box app "app's theme plugin" "contributes the defaults"
+row "Shared API" accent
+box reg "Themes registry" "data: roles to colours" accent
+row "Surfaces" blue
+box web "web page" "active theme → CSS variables on :root" blue
+box tui "terminal" "active theme → colours by role" blue
+arrow sol -> reg "contribute"
+arrow app -> reg "contribute"
+arrow reg -> web "watch"
+arrow reg -> tui "watch"
+:::
+
+**On the web**, the page watches `Themes` and the user's choice; when either changes it writes the active theme's
+colours (and its `web` extras) as CSS variables on `:root`. Nothing reloads; the browser repaints.
+
+**In the terminal**, the app's theme lookup reads the same entry; the next redraw uses the new colours.
+
+# A Theme Plugin {#plugin lede="What a stranger writes to add a theme, and what happens when they edit it."}
+
+A theme plugin is the smallest kind of plugin there is: data, contributed. As single-file plugins (decision
+"Single-file plugins"), one per surface it wants to be visible on, sharing the data:
+
+```ts
+// plugins/solarized.web.ts
+import { defineFacet, Themes } from "@acme/app/v3";
+import { solarized } from "./solarized-data.ts";   // or inline: single-file plugins import only the API
+export default defineFacet((_host, self) => self.contribute(Themes, solarized));
+```
+
+Since a single-file plugin may import only the app's API, a theme with two surfaces is either two files with the data
+inlined, or a small directory plugin (`plugins/solarized/` with `theme.ts`, `web.ts`, `tui.ts`).
+
+When its author changes a colour, the watcher marks the plugin dirty; the next rollout reloads it on the server and in
+every open page and terminal; the registry recomposes; pages that use Solarized repaint. No page reloads, nothing on
+screen is lost.
+
+## When the app changes its roles
+
+The app adds a role (`link`) in a later release. Old themes do not name it. The registry's version guards this: the app
+bumps `Themes` to version 2 and maps version 1 entries by filling the new role from the default theme. An old theme
+keeps working, slightly less complete; nobody's page breaks.
+
+# What a Plugin Cannot Do {#limits lede="Where the app's look ends and a plugin's begins."}
+
+A plugin cannot change the app's base stylesheet or its components: those are base code, changed by the app's
+release. It can contribute a theme (the app's looks, re-valued), style its own parts with scoped CSS that reads the
+tokens, and place UI through registries like `Panels`. That line keeps a page coherent with twenty plugins from
+twenty authors, and keeps one bad plugin from breaking everyone's layout.
+
+# Open Questions {.epilogue lede="What this proposal leaves to decide."}
+
+1. **The roles.** A fixed set the kit defines (accent, background, text, muted, error, warning, success) or each app's
+   own. Recommendation: the kit defines a base set and an app may add roles; themes fill the base set at least.
+2. **Where `Themes` lives.** In the app kit (every kit app gets it and the CSS-variable writer) or in each app.
+   Recommendation: the kit ships it, with the picker's preference and the writer; apps add roles and their defaults.
+3. **Light and dark.** A theme carries one mode; following the operating system's preference picks the user's chosen
+   dark or light theme. Recommendation: yes, two choices per user (one dark, one light).
+4. **Fonts and assets in themes.** Allowed through `web` extras with URLs to the plugin's assets (the build serves
+   them), or colours only. Recommendation: allowed, under the page's CSP (`font-src`, `img-src` self).
+
+# Reference {.appendix}
+
+| Piece | Where | What |
+| --- | --- | --- |
+| shared components (web) | `@acme/app/v3/web` | React components, one instance in the base bundle |
+| shared components (terminal) | `@acme/app/v3/tui` | the app's terminal components and `theme` lookup |
+| design tokens | the app's base stylesheet | CSS custom properties on `:root` |
+| plugin styles | `*.module.css` in the plugin | scoped class names, shipped with the facet's script |
+| `Themes` | `@acme/app/v3` (shared) | registry of theme data, read by both surfaces |
+| theme plugin | `plugins/<name>.web.ts`, `.tui.ts` or a directory | contributes a `Theme` |