@earendil-works/chord 1.5.0 → 1.5.1

package/docs/slices.md

@@ -0,0 +1,241 @@
+# Chord v2: the slice plan
+
+Status: approved (owner, 2026-10-22); kept current. `design.md` states the rules; this file says which slice builds
+what, and (section "Carry-over") what a slice deferred to a later one.
+
+## Why slices
+
+Building bottom-up made each layer guess the needs of callers that did not exist yet; L2 grew generic machinery
+(nodes, sinks, deliveries, publications) that nobody could follow, and was deleted. A **slice** is the thinnest
+end-to-end path through every layer a feature needs, so every API has a real caller from the first line. Later slices
+extend earlier code; reworking it when a real need appears is expected, guessing ahead is not.
+
+Per slice (owner, 2026-10-22): a one-page spec written by this session (`slice-N.md`, including the carry-over
+entries it takes), the owner's approval, a build by a fresh Opus 5.5 subagent, reviews of the uncommitted code by this
+session and by the reviewers Astra and Fable (`v2-review-astra`, `v2-review-fable`), fixes by the builder, the owner's
+review of the uncommitted diff, commit. Deferred work goes into the section "Carry-over" below, under the slice
+that must do it, in the same change (owner, 2026-10-22; moved here from `carry-over.md` on 2026-10-24). Every slice that touches a hot path (links, the store, relays) also adds a light benchmark script
+(`packages/chord/test/bench/`, run by hand) for timing, memory and wire size of realistic payloads, with results
+in `benchmarks.md` (owner, 2026-10-22). Every slice adds one or more runnable
+examples (`packages/chord/test/examples/NN-topic.ts`, numbered across slices, in the format of Durable's
+`test/examples/`), each on one topic: for the owner to run and dig around in, and as references for Chord's users
+(owner, 2026-10-22). Implementers and reviewers read
+the slice spec, the feature map below and the whole of `design.md` in full, and check the work against all three; a
+slice spec's section citations are for orientation, not a reading limit (owner, 2026-10-22).
+
+## What exists
+
+- **L1** (`src/contracts`, `src/data`): contracts, params and identity, argument checks, principal keys,
+  errors. `index.ts` for apps, `internal.ts` for Chord's own code.
+- **Slices 1, 2, 3a, 3b, 4a, 4b, 5, 6a, 6b, 8, 7a** (committed; 5 `4a5b45c45`, 6a and 6b 2026-10-27/28, 8 2026-10-28, 7a
+  2026-10-29): `src/documents`, `state`, `host`, `jobs`, `link`, `remote`, `store`, `registries`; examples 01-32
+  (Durable's 35-37); benchmarks in `benchmarks.md`.
+- **Slice 7b** (built 2026-10-29, `slice-7b.md`): `host.reload`, staging (`host/staging.ts`), fates, adapted members
+  (`host/adapted.ts`), references, jobs and registries across a reload; examples 33-34; `test/bench/reload.ts`.
+- **Slice 7c** (committed 2026-10-30, `slice-7c.md`): `src/node/plugins/` (discovery, builds, evaluation,
+  `pluginFile`, `mapStack`, `installDependencies`, the watcher, `rollout`), per-plugin `load` and `reload`; examples 35-39.
+- **Slice 10** (built 2026-10-31, `slice-10.md`): `src/node/http/` (a `fetch` core with the security baseline,
+  Chord's Node listener, `nodeRequest`, plugin sockets, links), `src/node/workers/`, `createPluginRuntime`, kinds,
+  the kernel's `RouteHandle` and anonymous `host.root`, origins only from relays; examples 40-48.
+- **Slice 12** (committed 2026-11-01, `slice-12.md`): the plugin model in `src/node/plugins/` (the app's versioned
+  API, per-kind files, single-file plugins, installs from git and npm, `testPlugin`); examples 49-52.
+- **Slice 13** (committed 2026-11-02, `slice-13.md`): the app kit, `src/node/app/` (`startServer`, `startWorker`,
+  sessions, owner link, pairing, `Files`, `exportData`/`importData`); examples 53-55.
+- **Slice 11** (committed 2026-11-03, `slice-11.md`): `src/surfaces/` (scripts objects, registered scripts, the
+  connection loop, clients, themes), `src/web/`, `src/react/`, `src/tui/`, `src/node/surfaces/`, `src/node/tui/` (the
+  launcher and its child-process sandbox); examples 56-59.
+- **Live queries** (built 2026-11-03, decision "Live queries over an object's own tables"): `self.query` in
+  `src/store/live-query.ts`; examples 59 and 61 use it; `test/bench/live-query.ts`.
+- **Slice 9** (built 2026-11-04, `slice-9.md`): telemetry, `src/telemetry.ts` (`CHORD_TELEMETRY_SCHEMA`, the span sites'
+  NOOP rule, open spans, typed instruments), spans of calls, methods, relays, holds, activations, natives, loads,
+  reloads, resolves, revocations, routines, job phases and HTTP requests, carriers on `hold` and `call`, `ctx.trace`,
+  commit events and the host-wide metrics, `telemetry` options of `createHttp`, `startServer` and `startWorker`;
+  example 62; `test/bench/telemetry.ts` (with R24's container width).
+- **Reload restarts every affected object** (built 2026-11-05, decision "Reload restarts every affected object; old
+  work finishes on old code"): the pure swap removed; old work through the call chain (`host/old-work.ts`) and
+  `call.old`, a restart's readiness (`host/readiness.ts`), reload cycles (`host/reload-cycle.ts`),
+  `reloadDrainTimeout`; example 64.
+- **Chord's shared code** (upstream pi): `src/watch.ts` (`Watch`, `NativeWatch`, `WatchHandle`, `WatchEnd`,
+  `NativeWatchable`), Delta (`track`, `diffRevisions`, `applyImmutable`, encoders, `PendingOps`, `appendText`),
+  context, JSON, output, SQLite drivers.
+
+## The slices
+
+Order (owner, 2026-10-22: routines and jobs before the store and reload; 2026-10-27: slice 8 before slice 7, whose
+reload has to reload real registries; 2026-10-29: slice 7 split into 7a, 7b, 7c; the app kit as slices 10 and 11;
+telemetry last, since it instruments the app kit too; 2026-11-01: the plugin model (12) and the app kit (13) before web and TUI (11), which builds into the kit).
+
+| # | Slice | In short |
+| --- | --- | --- |
+| 1 | **One host, local objects** | host, facets, local objects, references, in-memory documents, transactions, derives, watches |
+| 2 | **Failures and natives** | `self.open`, activation failures and recovery, backoff, the drain, timeouts |
+| 3a | **Remote objects over one link** | remote contracts, principals, variants, a memory link, the message format, holds, calls, one message per commit (no relay, no drops) |
+| 3b | **Surviving drops** | re-holds, parking, `ref.state`, `host.hold`, `ref.retry()`, `linger` |
+| 4a | **Links and relays** | host kinds, host addresses (`placement` then), routes, two-sided links, relays and principals, IPC and WebSocket, flow control, heartbeat |
+| 4b | **Authority and housekeeping** | revocation, cycles across hosts, `host.references`, `host.activity`, size limits, compression |
+| 5 | **Routines and jobs** | `self.routines`, `self.job` |
+| 6a | **The store** | the store (backends, tables, queries, writes, commit queue, outbox, documents engine, migrations, export), persisted documents, `self.table`, keyed watches and retirement |
+| 6b | **Durable and jobs on the store** | persisted jobs, Durable's `StoreStorage`, the Chord source over Durable's views (`self.follow`) |
+| 8 | **Registries and `host.root`** | registries, `self.writable`, `retryOn`, Durable's `registryReader`, `host.root` (`chord(fn, context)` and the harness object are p3's base code) |
+| 7a | **The dependency model** | requests that outlive providers, parked attempts without timers, wakes, the publication gate, getters, registries as dependencies, `parked_long` (`slice-7a.md`, `dependencies/model.md`) |
+| 7b | **Reload in the kernel** | `host.reload`: staging, commit, rollback; restarts (swaps until 2026-11-05), removal and re-serving; holds and `accepts` across a reload; jobs and registries across a reload; per-name wakes (`slice-7b/` brief) |
+| 7c | **Plugins** | `chord/plugins` (discovery, `buildFacet`, `evaluateFacet`, `pluginFile`, `installDependencies`, the watcher, `rollout`, `mapStack`; no dev/prod split; the kernel's `load`/`reload` per plugin; `slice-7c.md`) |
+| 10 | **HTTP and workers** | `chord/http` (server, `Http` registry, responses as descriptions, WebSocket entry) and `chord/workers` (a worker per address, IPC, idle stop) |
+| 12 | **The plugin model** | imports (relative within a repository, package names pinned across), versioned app API with Chord re-exported, per-kind API modules and contract files with the build's import check, single-file plugins, git installs, `testPlugin`, a required root (`slice-12.md`) |
+| 13 | **The app kit** | `chord/app`: `startServer`, `startWorker`, layout, worker kinds, owner link, deny by default, sessions, CLI tokens, storage, the files plugin, export per address, plugin roots and "apply changes", base code as API, bootstrap and a start-data facet (`slice-13.md`) |
+| 11 | **Web and TUI** | `chord/web` (page host), `chord/react` (React hooks), `chord/tui` with `chord/tui/launcher` (sandbox, launcher) and `chord/surfaces` (their server side), from p2's code |
+| 9 | **Telemetry** | spans, events, metrics across hosts |
+
+The playground (`apps/chord-playground`) moves to the new code once slice 4 exists. Round-3 cuts (C3-C16) and
+carry-over entries are decided in the slice that first needs them.
+
+## Feature map
+
+Every feature of `design.md`, by section, assigned to the slice that builds it. A slice's spec covers every entry
+assigned to it; **implementers and reviewers check the slice spec, this map and the whole design** (owner, 2026-10-22). A feature missing here is a planning bug: add it before the slice that needs it. "Doc" means a documented
+pattern, no code. Decided cuts are noted where they apply.
+
+| § | Feature | Slice |
+| --- | --- | --- |
+| 3 | `createHost` with `telemetry`, `onError`; host without address | 1 |
+| 3 | `createHost` with `principal`, `principalKey` (3a), `address` (4a), `revokedBy` (4b), the store (6a) | 3a, 4a, 4b, 6a |
+| 3 | `load([{ name, facet }])` with static facets; start barrier (host-lifetime objects live before `load` resolves) | 1 (failed plugins reported: 2) |
+| 3 | `connect`, `accept`, memory link pair, incarnations in the handshake, protocol-version handshake | 3a |
+| 3 | host kinds, addresses, `route` (`authorize`, `resolve`), relayed holds and calls, re-resolve after a drop | 4a |
+| 3 | IPC and WebSocket links, trusted links, the ack-clocked window per origin (K unacked state messages, pending queues, call window N per origin), relays with flow control per origin (`origin`, `closed`, senders FIFO), heartbeat | 4a |
+| 3 | timeouts: `closeTimeout`, call drain, `readyTimeout` (C13: two host timeouts) | 2 |
+| 3 | provider-side `linger` (C7: kept) | 3b |
+| 3 | park deadline (= `readyTimeout`, C13) | 3b |
+| 3 | `host.activity` (keep-alive counts) | 4b |
+| 3 | `host.root(fn, …)` for code outside objects | 8 |
+| 3 | requests that outlive providers: the destination by address, the publication gate, the unbound index, the check against the selected provider only; the start barrier's first outcomes (decision (a)) | 7a |
+| 4 | principal type `P`, links stamp principals, principal keys in identity | 3a |
+| 4 | revocation (`revokedBy`, `host.revoke`, propagation to children) | 4b |
+| 5 | `local` types: params, members, checked arguments (Q15), results unchecked | 1 |
+| 5 | version and fingerprint match at acquisition; reserved member names (`then`, `release`, `state`, `retry`) | 1 |
+| 5 | `remote` types, variants chosen by the factory (`{ variant, members }`), identity with principal key, calls checked against the binding's variant | 3a |
+| 5 | `accepts` mappings at runtime (old consumers), keyed members per key, `contract_mismatch` | 7b |
+| 6 | lifecycle, one object per identity, lifetime from kind and params (Q-A), factory, activation, ending | 1 |
+| 6 | failure causes, backoff per identity, dependency recovery (R18) | 2 |
+| 6 | `retryOn` | 8 (first independently living source: registries) |
+| 6 | parked attempts (no dependency deadline or backoff, wakes on every change, retry passing on), `FailureCause` `waiting`, `parked_long` and the authoring rules against cycles | 7a |
+| 6 | cycle rejection on one host (Q5) | 1 |
+| 6 | cycle rejection across hosts (activation chain in hold headers) | 4b |
+| 6 | shared store with per-principal views; worker-wide native owners; natives following a relocation | doc (1, 3, 2) |
+| 7 | `self.get`, `host.get`, hydrated references, members as watchables, `release` | 1 |
+| 7 | per-reference watches (released with their reference) | 1 |
+| 7 | `ctx.get` (call-scoped references) | 3a |
+| 7 | `ref.state` (tagged), `ref.retry()`, `host.hold`, calls parking, failed references | 3b |
+| 7 | getters wait through `waiting`; `host.hold` throws only for invalid params | 7a |
+| 7 | `ref.variant` as discriminant, `RefIn` (C15: no `inVariant`/`expectVariant`) | 3a |
+| 7 | `state: false` (C8: cut) | - |
+| 7 | optional dependencies as registries | doc (8) |
+| 8 | `ctx` with telemetry and signal | 1 |
+| 8 | `ctx.principal`, `ctx.incarnation` (R20), `ctx.get` (3a); relayed incarnations (Q13, 4a) | 3a, 4a |
+| 8 | `ctx.trace` (C11) | 9 |
+| 8 | a job's `ctx.jobId`, `ctx.state` | 5 |
+| 9 | `defineState`, singleton and keyed handles, `self.state`, in-memory documents (P2) | 1 |
+| 9 | `self.tx`, `tx.draft`, `tx.retire`, drafts revoked (R12), no existence check (R13), lines per object | 1 |
+| 9 | `handle.change`/`handle.retire` shorthands (C14: kept) | 1 |
+| 9 | `initial()` checked once per definition (R8, D3) | 1 |
+| 9 | `persist`, `checkpointWhen` (required for persisted documents), persisted only in local empty-params objects (P3), eviction | 6a |
+| 9 | keyed watches (`handle.watch(key, ctx)`), `handle.at(key)`, retirement as a flag (decision "One retirement mechanism") | 6a |
+| 9 | versions, `migrate`, schema fingerprints on records | 6a (in-memory migrate on reload: 7b) |
+| 9 | ~~facet documents, a served object's transaction spanning its facet's documents~~ (removed: C16 accepted, owner, 2026-10-26) | 1, removed in 6a |
+| 9 | rules across owners, reconciling routines | doc (5) |
+| 10 | the store: backends, tables, query runner, writes, commit queue, outbox, migrations, export/import | 6a |
+| 11 | `Watchable`, watches (Chord's `Watch`), the batch (W1), reentrancy | 1 |
+| 11 | `NativeWatchable`, `self.writable` | 8 |
+| 11 | Durable's views as sources (`self.follow` over Durable's `observeXXX`) | 6b |
+| 11 | live queries (`self.query(table, q, limit)`: run before live, re-run per touching commit, diffed) | after 11 |
+| 12 | `self.derive`, failures per member, lazy when unwatched, `mapShared` | 1 |
+| 12 | container-width benchmark (R24) | 9 |
+| 13 | `self.open`, close reasons, order, restart patterns (R16) | 2 (restarts on reload: 7b) |
+| 14 | `self.on` (W2b), `self.onLive`, `sleep`, `untilAborted` (C4 decided); routines across a restart | 5 (reload: 7b) |
+| 15 | `defineJob`, `self.job`, intent/run/finish, state documents, dedupe (Q9), `busy`, `running`, blocked `finish` (C3 decided); persistence, resume, restart limit; `accepts` in use | 5 (persistence: 6b; `accepts` and reload: 7b) |
+| 16 | `defineFacet<P>`, facet as declarations only (documents and routines removed in 6a) | 1, 6a |
+| 17 | registries: `defineRegistry`, contributions, `compose`, versions (Q8), use-site guards | 8 |
+| 17 | registry readers as parked dependencies (`waiting`), `host.registry` waiting, the last good value dropped when a contribution leaves | 7a |
+| 18 | `constant(value)`, start data as a core object (`SessionInfo`) | 1 (core facet pattern: 4a) |
+| 18 | platform bindings as core objects | doc (8) |
+| 19 | `host.references` (presence) | 4b |
+| 20 | flows kept by their owner, the `steps` helper, the Questions service | doc (5); deferred decisions #9, #11 |
+| 21 | `host.reload`: staging, atomic commit, restarts with old work on old code, removal and re-serving | 7b, restart-only 2026-11-05 |
+| 21 | loading plugins (loader D), dev loop; the rollout across hosts is the app's (decision "Slice 7b's open points" (3)) | 7c |
+| 22 | message format, header schema, unknown ids, one message per commit per link, consumers apply ops, plugin errors | 3a |
+| 22 | relays, size limits per part, compression at the origin, adapted bindings get diffs | 4a (relays), 4b (limits, compression), 7b |
+| 22 | trace carriers (R5) | 9 |
+| 23 | HTTP (routes, wraps, responses as descriptions, request bodies, plugin sockets), workers; channels and chunked transfers stay owner patterns until a second user | 10 |
+| 23 | the app kit: `startServer`, `startWorker`, start-data objects, deny by default, sessions, owner link and token file, device pairing, files (with chunked transfers), export and import per address, plugin roots and "apply changes", optional users, invite and ephemeral-login plugins | 13 |
+| 23 | web and TUI: base bundles, registered plugin scripts and the combined file, `WebScripts`/`TuiScripts`, `startPage` with the connection loop and in-place reloads, React bindings, the sandboxed terminal (launcher, sandbox, `TuiScope`), themes (`Themes`, `ThemeChoice`, CSS variables, terminal colours), CSS modules | 11 |
+| 24 | Durable integration: `chord(fn, context)` (Q6), a `RegistryReader` over composed registries; the harness object and its model and env registries are p3's (base code, owner, 2026-10-27) | 8 (Chord parts), p3 |
+| 24 | `StoreStorage` | 6b |
+| 27 | versions table: contracts (L1 done), registries 8, documents 6, jobs 5, link protocol 3a | as listed |
+| 29 | telemetry spans, events, metrics, carriers | 9 (cost counters: every slice) |
+| 30 | cost guards per hop | every slice |
+
+## Size
+
+About 9-10k source lines in total, as estimated for the layers (owner: fine, on the order of Durable).
+
+## Carry-over
+
+Obligations a slice deliberately left to a later one, under the slice that must do them, with where they came from.
+Rules (owner, 2026-10-22): whoever defers something (this session, a subagent, a reviewer) adds it here in the same
+change; a slice's spec includes every entry of its section, and its build and review check them off; a done entry is
+deleted (the slice's spec and code are the record). Moved here from `carry-over.md` (owner, 2026-10-24).
+
+
+### Any slice (protocol and checks)
+
+- **Bump `PROTOCOL_VERSION`** for any message or reason change once p3 has shipped (slice 3b added `retry`, `loading`
+  and `failed`'s `retryIn`; slice 4a parts, the list `bound`, `hello`'s `calls`, `ack`, `ping`, `pong`, `closed`,
+  `origin` and the hold's `address`, `principal` and `incarnation`; the reload build `call`'s `old` and the
+  `unavailable` reasons `version` and `reload_cycle`; all without a bump: nothing is deployed). (Owner, slice 3b.)
+- **The park deadline's per-type override** is not built: a parked call waits the reference's `readyTimeout`. Build
+  it only if a real case needs it. (Slice 3b spec.)
+- No CBOR; revisit only if benchmarks show encoding as a bottleneck (slice 3a's and 4a's do not). (L2 decisions.)
+- **One N on every host:** a relay forwards calls at once, so a worker announcing a smaller N than the relay's could
+  see more running calls of one origin than it accepts (a protocol error). Every host announces 256; per-link N, if
+  ever needed, must bound the relay's forwarding too. (Slice 4a rework.)
+- **The Durable conversation view's size on re-hold** is a known cost until pi #11327 (windowed view) lands. (Slice 4.)
+
+### Later: reload (when a case needs it)
+
+- **`rebuild` of tables** (key and column-type changes): `evolveTable` has no edit for them; build one only when a
+  real case needs it. (Slice 6a's second spec review; out of slice 7b's scope, `slice-7b.md` §12.)
+
+- **Content-compared plugin marks:** a plugin's `dirty` mark stays latched until the plugin is applied (a rewrite with
+  the same bytes, or a change and its revert, dirties it); compare a plugin's files with the bytes its last applied
+  build read (`BuiltFacet.inputs` already holds their hashes), as `base` now does. (p3 step 0a,
+  `apps/p3/docs/step-0a/book.md`.)
+
+### Later: live queries (when a case needs it)
+
+- **Incremental maintenance of a live query's result** (decision "Live queries over an object's own tables"): an
+  ordered, limited result kept from row-level changes in memory, re-running only when a removal empties the window
+  or a bulk write hides which rows changed; today every touching commit re-runs the query. (Live queries build.)
+
+### After slice 11
+
+- **Small examples** (owner, 2026-10-30: the 7c examples are complete but complex): once HTTP, workers, web and TUI
+  exist, a pass of small, easy-to-read examples, each showing one thing a plugin author or app does.
+
+### When v2 goes upstream
+
+- `mapShared` lives in `src/state/derive.ts` and moves into Delta. (L2 decisions.)
+- `withLinkedAbortSignal` lives in `src/host/ctx.ts` and moves next to `withAbortSignal` in `src/context` (pi
+  first): a call's context linked to long-lived abort signals until the call settles, instead of `AbortSignal.any`,
+  which keeps cleanup entries per call while its sources live before Node 24.16.0 (nodejs/node#62367). (Slice 4a.)
+- An export for `nodeCompression` (`src/node/compression.ts`; no subpath exports it yet). (Slice 4b.)
+
+### The sync layer (not a slice; when Radius sync is built)
+
+- Upload, acknowledgement and `store.outbox.trim`, epochs, pull-back, the split (S5, S7); a store opened with an
+  outbox (decision "The outbox is optional, internal and versioned") starts its sync from an archive.
+- Snapshot reads for large exports on a second SQLite connection (S4), if exports get large.
+
+### Outside the slices
+
+- The playground's runtime tests fail on a stale `node_modules` copy of chord 1.3.0; relink with
+  `npm install --ignore-scripts` when the playground is used again. (Frozen-kernel move.)