@earendil-works/chord 1.5.0 → 1.5.1

package/docs/next/slice-6b.md

@@ -1,332 +0,0 @@
-# Slice 6b: Durable and jobs on the store (spec)
-
-Status: **built** (2026-10-28, pi `14b4c56e5`: one build, a build review and two fix rounds by the four spec reviewers;
-reopen, the one-entry commit's mean and the resume watch's CPU accepted by the owner). Was: ready to build (2026-10-27).
-
-The single source for building 6b, together with `design.md` read in full; the decision log `decisions.md` (later
-entries win), in particular "The store", S1, S2, S4, S10, S11, S13, S15, W1, W12, "Slice 6 first decisions", "Durable's
-synchronous observers", "Text is plain UTF-8", "Failures are watch calls", the "One vocabulary" entries, "The performance
-and memory bar", and every "Slice 6b" entry; `slice-6a.md` (what 6a built); the feature map and the carry-over of
-`slices.md` (6b); the root `AGENTS.md` "Chord v2 Code Style".
-
-6b puts Durable on the store (`StoreStorage`), follows Durable's views as Chord sources (`self.follow`), persists jobs,
-and extends the applier to every write kind. A worker then has one store: Chord's objects and Durable's Session commit
-through one queue, and a commit of either is one seq in one outbox.
-
-## What an app can do after this slice
-
-```ts
-// p3 worker base code
-const store = await openStore(sqliteBackend(db), ctx);
-const host = createHost<P>({ address: Session({ id }), principal, store, onError });
-await host.load([{ name: "@p3/core", facet: coreWorkerFacet({ init, store }) }, ...plugins], ctx);
-
-// the core facet: the harness object opens Durable on the same store, its data under the harness's own type
-host.serve(Harness, (_, self) => {                     // local, empty params (P3)
-  const harness = self.open(async (context) =>
-    Harness.open(await StoreStorage.open(store, Harness.name, context), { models, registry, … }, context),
-    (h, _reason, context) => h.close(context));
-  const view = self.follow((observer, ctx) => harness.value.observeConversation(rootId, observer, ctx, { min: 50 }));
-  return { view };
-});
-
-// a persisted job: survives a crash, a clean close and its object's retry
-const ImportJob = defineJob("notes.import", {
-  version: 1, args: obj({ path: str() }), result: obj({ count: int() }),
-  state: ImportState /* persist: true, checkpointWhen */, persist: { resume: "rerun" },
-});
-const imports = self.job(ImportJob, { run, finish });
-await imports.abort(id, ctx);                          // "marked" | "terminal", stored
-await ctx.state.change((d) => { d.output = text; }, ctx, { persistence: { kind: "deferred" } });
-```
-
-## 1. Changes to 6a's store
-
-Every item below changes 6a's code; nothing is deployed, so the schema and the archive format change in place.
-
-### 1.1 Names and claims (decisions "R2", "owner names and claims")
-
-- Tables are `"<owner>:<table>"` (`"notes.store:edits"`), document owners `"<owner>"`; no host kind. The applier and
-  import check two name parts.
-- **Claims** (owner, 2026-10-27, decision "owner names and claims", refined): what can collide is claimed once per
-  store, in memory: a table by its SQL name, documents by `(owner, namespace)`. An object claims its tables and
-  `(type, "chord")` (plus `chord.jobs` and `chord.job-states` with persisted jobs) when it declares them;
-  `StoreStorage.open` claims its five tables and its document namespaces (Durable's scopes contain a `:`, Chord's
-  namespaces never do). A second claim is refused `invalid/args`; a claim is released by its owner's end
-  (`StoreStorage.close` after its calls underway drained, the object's end), so a new Session can reopen over the same
-  store. So the object that opens Durable can declare its own persisted data under the same owner name, and two
-  harnesses can share a store under two owner names.
-
-### 1.2 One id allocator (D3)
-
-- The store owns one synchronous allocator: `mintId()` takes the next id; the commit step takes its documents' ids from
-  the same allocator when a creation applies (no copy in the `Step`, no overwrite after `COMMIT`); explicit ids in
-  staged writes raise it at staging (`max`); a refused step leaves its ids unused (gaps, S10). Each commit persists
-  the allocator's current value.
-- It starts at 2 (id 1 is Durable's root conversation). It saturates at `Number.MAX_SAFE_INTEGER`: an explicit write of
-  that id is accepted, the persisted value stays `MAX_SAFE_INTEGER`, every later `mintId` rejects "ID space is
-  exhausted", and a Chord creation at the saturated allocator refuses its own commit (1.4).
-- Each commit takes the allocator's value once, after its writes applied, for the outbox commit (`{ seq, nextId,
-  writes }`) and the metadata row (mints meanwhile only raise it further); an export reads it once with `lastSeq`; the
-  archive manifest's `nextDocumentId` is `nextId`; import and batch apply restore it, so a replica mints above every id
-  it holds.
-
-### 1.3 The commit step
-
-- **A staged write returns the writes it recorded:** `StagedWrite.apply(step): Promise<readonly StoreWrite[]>`.
-  `StoreStorage` stages one composite write per Durable commit (1.3 of section 2).
-- **Order:** within the step, writes apply in the order staged, except that a staged write that frees a live address
-  (a retirement of another incarnation) is applied before the creation at that address (rule 9's reason, the partial
-  unique index); Chord's own staging keeps 6a's order. The outbox records the writes in applied order; the applier
-  replays them in that order (no re-sort).
-- **Stamps:** a staged write can take values only the step knows (`step.seq`): Durable's `entries.commit_seq`. The
-  outbox records the resolved values.
-- **Adopt in the owner's batch:** the host gives the store its batch when it attaches (`attach({ failed, batch })`);
-  an adapter's fence runs its `adopt` in `batch.within(…)` (joins a running batch, opens one otherwise); Chord's own
-  transactions keep their `batch.commit`. Without a host, `adopt` runs plain.
-
-### 1.4 Errors: Durable's Storage semantics (D4)
-
-- The step records the error its own code threw inside the transaction callback. A rejection of the backend
-  transaction that is that very error is a **refusal**: the facade rolled back (a failed `ROLLBACK` reaches us as
-  another error, an `AggregateError` on Node), the commit rejects, nothing is adopted, the allocator keeps its raised
-  value, the store goes on. Any other rejection (`BEGIN`, `COMMIT`, `ROLLBACK`, a throwing `adopt`) is an unknown
-  outcome: the store stops (`storage_failed`). The Cloudflare facade is checked to keep the same identity; a test
-  injects failures at `BEGIN`, in the body, at `COMMIT` and at `ROLLBACK` on both drivers' facades.
-- **The refusal's error reaches the owner unchanged** (Durable's suite matches its messages); the store classifies it
-  for Chord's owners: an `int` `CHECK` failure (SQLite's extended code `SQLITE_CONSTRAINT_CHECK` on Node; the error
-  message on the Durable Object facade, which has no extended codes) is `invalid/args` and
-  fails only that transaction; every other refusal of a Chord commit (the engine's counted `UPDATE`/`DELETE`, the
-  live-address index, a primary key, the saturated allocator, a backend error in a statement) fails the object live
-  (`object_failed`, `broken`: memory and disk disagree).
-- **6a's owners:** a refused `self.tx` rejects with the error above and the object fails as above; a refused
-  migration commit (a chunk, (A) or (Z)) is rolled back whole, fails the activation `broken`, the table stays
-  `migrating`, earlier committed chunks stay; a refused job start rejects the start; a refused outcome or finish commit fails the object (records and
-  memory disagree); a refused commit that carried held changes (D8) fails the object.
-- **Reads:** a read's callback returns data, never throws for a request: an adapter's request checks (an unknown
-  conversation, a cursor, history a document does not keep) happen after the guarded read returned, from tagged
-  results, so only a backend error fails the store. The store offers a read transaction for reads of several queries
-  that must see one state (S4).
-
-### 1.5 The documents engine
-
-- **Addresses:** `(owner, namespace, kind, key?)`: `owner` the owner name; `namespace` the adapter's partition, claimed
-  with the owner (Chord: `chord` for documents, `chord.jobs` and `chord.job-states` for persisted runs; Durable: the
-  document's scope, `session`, `conversation:<id>`, `task:<id>`, which Durable's reads by scope need); `kind` and `key` as the adapter writes them (Durable JSON-encodes both,
-  as its indexed strings). One live record per address (the partial unique index).
-- **`history` per record (R1):** `none` (Chord: older revisions dropped at a new base; row and revisions deleted at
-  retirement), `latest` (Durable's session, task and latest conversation documents: older revisions dropped; the row
-  kept with `retired_at` at retirement), `rewindable` (nothing dropped; the row kept). `retireDocument` reads the record
-  and applies its rule.
-- **Explicit ids:** a creation may carry its id (Durable mints ahead); otherwise the allocator gives one.
-- **Changes without a known revision:** an adapter whose writes carry no revision (Durable) stages a change that reads
-  the record in the step and uses its revision; the counted conditional `UPDATE` stays the guard.
-- **`meta`:** Durable's `fork` policy and `history` details live in the record's `meta`.
-- **Reads at a point:** `find(address, at)`, `load(id, at)` (the base at or below `at` plus the deltas up to it, its
-  version the base's), `list(owner, namespace, at, kind?, limit, after?)` by id, with the half-open lifetime
-  (`createdAt <= at < retiredAt`); records carry `retiredAt`. New indexes (store schema version 2):
-  `(owner, namespace, kind, key, created_at)`, `(owner, namespace, id)`, revisions `(document_id, kind, seq)`; their
-  cost on Chord's commit gate is measured.
-
-### 1.6 The archive and batch apply
-
-- **Format 1 gains** (nothing deployed): `document.create` carries `history`, `retiredAt?` and `meta`; a
-  `document.revision { id, seq, content }` (`content` a `DocumentContent`: `{ kind: "base", version, value }` or `{ kind:
-  "delta", version, ops }`) part per kept revision, in seq order after its create; for
-  live `none` and `latest` records only the current base and its deltas are exported (as 6a); a retired `latest` record
-  is exported as its record alone (no base: its content was dropped at retirement); for `rewindable` every revision,
-  retired records included. Import checks: revisions in seq order, a base first, `seq <= lastSeq`, lifetimes
-  at one address never overlapping (a second live record is still refused), versions per revision; the `end` count
-  includes revisions.
-- **Batch apply** (`applyBatch(store, commits, ctx)`, internal): refused while any owner holds a claim on the store
-  (an object or a `StoreStorage`: its memory would go stale) and run through the store's commit queue; `commits[0].seq` follows `lastSeq` (S5); one backend transaction for the batch, every write applied
-  with its commit's seq, `lastSeq` and `nextId` set from the last commit; the applier handles every `StoreWrite` kind
-  (`alter`, `dropIndex`, `update`/`updateWhere`/`delete`/`deleteWhere` with their recorded counts and versions,
-  `document.change`, `document.retire`, `document.revision`), checked against the store's records (a trust boundary);
-  a count that differs from the recorded one stops applying and marks the store diverged (S17): every later
-  `applyBatch` rejects. A test replays a store's outbox into an empty store and compares the exports, manifest
-  included.
-
-## 2. `StoreStorage`: Durable's `Storage` on the store (D1, D2)
-
-In Durable's `src` (an ordinary pi module), importing `@earendil-works/chord/next` (Chord's `./next` export comes
-with it: the store and the adapter surface below, marked internal for adapters). It passes Durable's storage
-conformance suite on SQLite `:memory:` and across reopen (a wrapper that closes and reopens the store file after
-every commit, Durable's reopen harness), and Durable's Session and harness test files run on it.
-
-### 2.1 Opening and the adapter surface
-
-`StoreStorage.open(store, owner, ctx)`: claims `owner` (1.1); declares its tables through the store's table
-preparation (the path `self.table` takes: created at the newest version or migrated); raises the allocator to
-`max(record_ids.id) + 1`, saturating at `Number.MAX_SAFE_INTEGER` (one index read; covers stores built by import or
-batch apply); attaches to the store's host
-batch if there is one. Chord exports for it: `StoreOwnerTables` (declare and prepare tables under an owner),
-staged table writes and engine writes, the step reader, the read transaction, `mintId`, the commit with a fence. Its
-`close(context)` refuses new calls ("StoreStorage is closed"), waits for its calls underway, then releases its claims; the store stays open (the
-app owns it).
-
-### 2.2 Tables
-
-Every row carries the record's indexed columns and the whole record as `json(unknown(), { version: 1 })`; indexed
-strings that must stay lossless (`kind`, `request_id`) hold Durable's JSON encoding; index columns are non-`opt`, so
-absent values take sentinels that no encoded value can be (`0` for an absent owner id, ids start at 2; `""` for an
-absent request id, JSON-encoded strings start with `"`). SQLite's partial head index is the table `entry_heads`
-(builder, 2026-10-27: an index on `(conversation_id, has_head, id)` cost about 5.5 µs per commit; the table is written
-only for entries with a head and is faster on both paths).
-
-| Table | Key | Columns | Indexes |
-| --- | --- | --- | --- |
-| `record_ids` | `id` | `record_type str` | |
-| `conversations` | `id` | `owner_conversation_id int`, `owner_task_id int`, `record` | `(owner_conversation_id, id)`, `(owner_task_id, id)` |
-| `entries` | `id` | `conversation_id int`, `commit_seq int` (stamped), `record` | `(conversation_id, id)` |
-| `entry_heads` | `id` | `conversation_id int` (one row per entry with a `head`, written with it) | `(conversation_id, id)` |
-| `tasks` | `id` | `conversation_id int`, `kind str`, `status str`, `abort_requested int`, `background int`, `record` | `(status, id)`, `(conversation_id, id)`, `(kind, id)`, `(abort_requested, id)`, `(background, id)` |
-| `submissions` | `id` | `conversation_id int`, `request_id str`, `status str`, `record` | `(conversation_id, request_id)`, `(conversation_id, id)`, `(status, id)` |
-
-### 2.3 Commits
-
-`commit(writes, adopt, ctx)` stages one composite write and calls the store's commit with `{ admit: () => true,
-adopt: (seq) => adopt(seq) }` (adopted in the host's batch, 1.3). Inside the step the composite write does what
-`SqliteStorage` does inside its transaction, with the step's reader: `checkGlobalIds` (an id already claimed by
-another record type, a conversation, entry or document written twice: Durable's messages), the per-document actions
-of `prepareDocumentActions`/`checkDocumentActions` (one content command per document, a delta on its version, one live
-incarnation per address: "already has a current incarnation"), then the table `put`s (`record_ids` only for ids not yet
-claimed by the same type), then the documents in two passes: first every document that already exists (its content,
-then its retirement), which frees the addresses a creation of this commit takes, then the creations and copies (each
-with its retirement, a create-plus-retire being an empty lifetime, `createdAt = retiredAt = seq`). Durable's
-"reconstructs rewindable documents" case (`create secondId` at address A, `retire firstId` at A, `change firstId`)
-needs this order. Copies (`document.copy`) are resolved before
-the step (S11): the source read at its point, checked against the copy's record ("changed in the copy batch",
-kind, key, history, fork), entered as a creation carrying the base. A refused check rejects with Durable's message (a
-refusal, 1.4): the store goes on, Durable's Session fails (`GuardedStorage`).
-
-### 2.4 Reads
-
-`conversation`, `entry` (both overloads, `commitSeq` from its column), `findLatestHeadMarker`, `scanEntries` (both
-orders, the fork chain walked as `SqliteStorage` does), `task`, `scanTasks`, `submission`, `scanSubmissions`,
-`submissionByRequest`, `scanConversations`: queries over the tables, Durable's cursors (`scan.ts`) as they are;
-`findDocument`, `document` (one read transaction), `scanDocuments`: the engine's reads at a point (1.5). Request errors
-(`StorageRequestError`) are thrown after the guarded read (1.4).
-
-### 2.5 Cost
-
-Statements per commit, as the gate measures them: an entry append, a `record_ids` check read, the `record_ids` put,
-the entry put, the meta update (4; `SqliteStorage` 5); a document delta, the record read, the revision insert, the
-counted update, the meta update (about 4-5; `SqliteStorage` about 7). Per `put` one compiled row check and the JSON
-encoding of indexed strings.
-
-## 3. `self.follow`: Durable's views as Chord sources (D9, R4)
-
-`self.follow(attach)` in a factory returns a `Watchable` (a state member, a derive source, an `on` source);
-`attach(observer, ctx)` returns `Promise<{ value, detach } | undefined>` (Durable's `Observation`; `undefined` for an
-absent `observeDoc` address). In 6b it is used inside the object that opened Durable (a native's value); reaching
-Durable from other objects is slice 8's (the harness in a registry entry, carry-over).
-
-- **Activation:** after the natives opened, before the members are checked and derives compute: Chord makes the
-  observer and calls `attach`; calls arriving before it resolves are queued and applied in order on `value`; a rejected
-  or `undefined` attach fails the activation like a failed dependency (`transient`); a failed activation detaches
-  what attached. The attempt's end detaches.
-- **Calls:** a call inside a running batch (Durable's `adopt` on the store) applies the queued calls first, then marks
-  the source in that batch; a call outside any batch (Durable's deferred commits, `#hold`, and the task graph's
-  `blocked` refresh, even on the store; any Durable storage that is not the store) is queued and one batch on the next
-  microtask applies everything queued. A derive over two views of one commit sees both changes in one computation.
-- **Ends (R4):** `failed` fails the live object `transient` (reported once as `object_failed` with that failure, its
-  identity's backoff, several sources ending at once fail it once; its retry reopens Durable and attaches again); `ended` is final:
-  a pending retirement call reaches the source's watches first (Durable's `endAfterDelivery`), then they end `ended`;
-  the member keeps its last value; a new `watch` on an ended follow source opens at that value and ends `ended` at once.
-
-## 4. Persisted jobs (D5-D8, R3)
-
-`persist: { resume: "rerun" | "abandon" }` only in `local` types with empty params (P3, at `self.job`); a persisted job
-needs a state definition with `persist: true` and `checkpointWhen` if it has state, and a JSON-only result (no `bytes()`,
-refused at declaration).
-
-### 4.1 Records
-
-Each unsettled run is a keyed persisted document in namespace `chord.jobs` (kind the job name, key the run id,
-`history: none`); its state is a keyed persisted document in namespace `chord.job-states` (same kind and key), so two
-jobs sharing a state definition never meet. The record:
-
-```ts
-{ version, fingerprint, args, key?, trace?,
-  phase: "pending" | "running" | "stopped" | "finishing" | "blocked",
-  restarts, finishFailures, abort?: true, blocked?: "finish_failed" | "version",
-  outcome?: { kind: "done", result } | { kind: "failed", error: StoredJobError } }
-// StoredJobError: { kind: "threw", error: WireError } | { kind: "aborted", reason } | { kind: "restart_limit", restarts }
-```
-
-Errors are stored in the wire error format (a `PluginError`'s code, message and details survive); a stored outcome is
-rebuilt as the in-memory one.
-
-### 4.2 Transitions
-
-| Event | Writes (one commit unless noted) | After |
-| --- | --- | --- |
-| start | record `pending`, state `initial()` | invoked on the live attempt |
-| reservation (before `run`) | record `running` | `run` invoked (a crash after it counts) |
-| `ctx.state.change` | state; the record's `restarts: 0` only when it was not 0 | |
-| abort (D7) | record `abort: true` (on the line); the signal fires after the commit, outside the batch; the line is released, then `abort` joins `run` | `"marked"` |
-| `run` settles | held state flushed (D8), record `finishing` with the outcome | `finish` on the live attempt |
-| `finish` fails | record `finishFailures + 1`; at 5 `blocked: "finish_failed"` | retried after the backoff, a blocked run on reload |
-| `finish` succeeds | record deleted, state retired (Q9), with what `finish` wrote | the run settles |
-| clean stop (host close, a native restart, R3's seal) | the stopped runs written `stopped`, held state flushed, by the close writer | |
-| activation (records loaded once per object; each live attempt resumes what waits) | each `running` record: `restarts + 1` (or, at 5, `failed { restart_limit }` and `finishing`, `job_interrupted` reported); `stopped` and `pending`: back to `pending` | maps, keys, active count, `running` rebuilt before anything runs; `rerun` invokes, `abandon` records `failed { aborted, restart }`; an abort-marked run finishes `aborted` without `run`; `finishing` resumes `finish`; `blocked` stays; a record of another job version, or of the same version with another fingerprint, is `blocked: "version"` |
-
-- **A retry after a seal (R3):** the runs the seal stopped are resumed by the next live attempt only once their old
-  invocation settled (bounded by `closeTimeout`, as the drain; an invocation outliving it is fenced, its run resumes
-  then), so a `run` that ignores its signal never runs twice at once.
-- **The close writer:** at the object's end, after the drain and before the documents drop, one commit past the seal
-  (Durable's `writeAtClose`): the runs stopped by the end written `stopped`, held state flushed; a run whose `run`
-  outlived the drain is fenced and written `stopped` too; blocked persisted runs keep their records (dropped only by
-  `abort`). In-memory runs keep 6a's rules.
-- **Pinning:** records and state documents of unsettled runs stay in memory (never evicted): `ctx.state.value` and
-  `running` read them synchronously.
-- **Fingerprint:** the job's fingerprint (args, result, state) is stored; a mismatch is `blocked: "version"` (resolved
-  by slice 7). A run ending at the limit reports `job_interrupted { object, job, id, restarts }` (a new `HostError`).
-
-### 4.3 Deferred state (D8)
-
-As Durable's holds: the slot adopts and publishes the change at once (`valueVersion` ahead of what is stored), keeps
-the held ops in a `PendingOps` (a base flag when `checkpointWhen` selected one), and the run's documents stay pinned;
-the next immediate change of the state document stores the held ops and its own as one delta on the stored revision
-(or a base); one foreground timer per object for the earliest deadline (`withinMs`, default 1,000 ms, `Infinity` waits
-for the others; a negative or non-number `withinMs` is `invalid/args`); the outcome's commit and the close writer
-store what is held; a held change of a fenced invocation is still stored; a crash loses it. A deferred change that
-would reset `restarts` is stored at once (the reset must be durable). On an in-memory job the option is accepted and
-does nothing.
-
-## 5. Benchmarks and tests
-
-- **The gate** ("The performance and memory bar"): Durable's storage benchmark (`test/storage.bench.ts`, every read and
-  write case, 1k and 10k) and `test/conversation-view-bench.ts` on `StoreStorage` against `SqliteStorage`, outbox off
-  (its cost reported separately): each hot path within about 5%, allocation per op and retained memory no worse beyond
-  noise; Chord's 6a commit gate again with the new indexes and the allocator; persisted job start and state change
-  reported against in-memory ones (a disk commit is not held to the in-memory number).
-- **Tests:** the conformance suite (both fixtures); the D4 matrix (`BEGIN`, body, `COMMIT`, `ROLLBACK` failures; a
-  refused Durable commit keeps the store; a refused Chord commit fails its object; a `CHECK` failure only its
-  transaction; a refused migration chunk; a request error inside a multi-query read keeps the store); the allocator (a
-  mint during a running Chord step, saturation across reopen, a refused step's gap, replay restoring `nextId`);
-  create-plus-retire and change-plus-retire through the outbox and the applier; reads at a point on retired `latest`
-  and `rewindable` records; export and import of rewindable history and mixed Chord and Durable data; `applyBatch`
-  (every kind, the replay equality, a diverged count); `self.follow` (one frame per commit, a derive over two views,
-  calls outside a batch on the store, early calls, an absent `observeDoc`, `failed` and `ended`, a retirement before
-  the end); persisted jobs (every transition of 4.2, a crash between reservation and `run`, a crash during the close
-  writer, a seal and retry, restart counting and its reset, the limit, the finish failure count across a crash, abort
-  joined and stored, abort racing an outcome, deferred state merged, flushed at the outcome and at close, pinned past
-  the retention); GC (closed `StoreStorage`, ended follow sources, cancelled abort waiters, deferred timers, settled and
-  dropped runs).
-- **Examples:** Durable's `test/examples/35-durable-on-the-store.ts` (a worker with Durable's Session and a Chord store
-  object on one store, exporting the archive) and `36-durable-views.ts` (Chord's tests import nothing of Durable);
-  Chord's `28-persisted-jobs.ts`.
-
-## Carry-over taken
-
-From `slices.md` "Slice 6b": persisted jobs (all of it, the finish failure count included), `StoreStorage` (ids, fork
-copies, `adopt` in the batch), the Chord source over Durable's views (`self.follow`), the applier for batch apply;
-`job.abort` async and stored.
-
-## Carry-over left
-
-Slice 7: `blocked: "version"` resolved on reload; runs of jobs the code no longer declares; blocked runs retried on
-reload. Slice 8: Durable's views from other objects (the harness in a registry entry). Slice 9: the trace carrier's
-use.