@smthrs/engine-store 0.0.0-stage → 1.0.0-rc.3

package/docs/guides/reclaim-runs-from-a-dead-host.md

@@ -0,0 +1,133 @@
+---
+title: "Reclaim runs from a dead host"
+description: "Choose an isAlive check for your deployment, understand what evidence each takeover records, and read the journal to see why a steal was admitted or refused."
+sidebar:
+  order: 2
+---
+
+A process that is killed with SIGKILL leaves its runs `running` with a frozen
+heartbeat. Nothing else revisits such a run on its own: it has no waiting row,
+no pending clock, and no future deferred completion. This guide is about
+choosing how aggressively another process may take those runs over.
+
+## Choose an isAlive check
+
+`isAlive` on `EngineStore.Options` is consulted before this store steals a run
+whose lease has expired. Answering `true` refuses the takeover.
+
+**Take the default when you have no better evidence.** Omit the option and the
+store uses `Ownership.leaseLiveness(Ownership.heartbeatStaleAfter)`: the owner
+is alive while its persisted heartbeat is younger than the staleness cutoff,
+and gone once it is not.
+
+```ts
+import { EngineStore } from "@smthrs/engine-store"
+
+const engine = EngineStore.layer({
+  owner: { hostId: "worker-a" },
+  journalSource: "worker-a-engine"
+})
+```
+
+This is enough to reclaim from a hard kill without any application code, and
+the steal it admits carries `lease-expired` evidence, which `RunStore.steal`
+re-checks against the row inside the same write. A claimant that lies about the
+lease loses the compare-and-swap.
+
+**Probe the process table on a single machine.** When every engine over a
+database runs on one host, ask that host:
+
+```ts
+import { Ownership } from "@smthrs/run-store"
+
+const engine = EngineStore.layer({
+  owner: { hostId: "worker-a" },
+  journalSource: "worker-a-engine",
+  isAlive: Ownership.sameHostPidProbe
+})
+```
+
+A second process over the same file then cannot take a run out of a live one,
+even while that process is busy enough to miss a heartbeat.
+
+**Write your own for a multi-host deployment.** A supplied check receives the
+recorded owner and a `LivenessContext` of `{ claimant, heartbeatAtMs, nowMs }`.
+Guard any pid read with `Ownership.sameHostIncarnation(owner, claimant)`,
+because a pid names a process only inside its own host's namespace. An
+orchestrator that reports pod liveness is the other shape this takes.
+
+Do not write a check that returns `false` without asking anything. That says
+"the owner is gone" about an owner it never looked at, and the engine steals
+runs out of live processes on the strength of it.
+
+## Read the journal to see what happened
+
+Every arbitration is recorded. Query the run's entries and look for two event
+types:
+
+| Record                      | `evidence`                     | Meaning                                                        |
+| --------------------------- | ------------------------------ | -------------------------------------------------------------- |
+| `steal-refused-owner-alive` | `lease-fresh`                  | The lease was still inside the window; no probe was consulted. |
+| `steal-refused-owner-alive` | `probe`                        | Your `isAlive` check answered for the owner.                   |
+| `stolen-and-activated`      | `lease-expired`                | The lease expired and nothing refused it.                      |
+| `stolen-and-activated`      | `same-host-pid-dead`           | A same-host probe found the process gone.                      |
+| `stolen-and-activated`      | `cross-host-unreachable-stale` | The owner was on another host and unreachable past the window. |
+
+Refusals are deduplicated by run, refused owner, lease, and evidence, so a run
+refused every second for an hour produces one record per distinct reason, not
+thousands. Seeing both a `lease-fresh` and a `probe` refusal for one lease is
+normal: the first is a wake arriving while the owner still pulses, the second
+is the same owner alive but stalled past the window.
+
+## What the sweep does with a stale run
+
+The driver's periodic sweep runs on the heartbeat cadence and re-drives three
+shapes through the ordinary claim, steal, and activate path: parked runs whose
+cancellation was durably requested, runs parked with reason `released` by a
+shutdown, and stale `running` rows left by a hard-killed owner.
+
+The stale sweep reads a batch of 64 rows, oldest heartbeat first, and reads past
+exactly the rows it is going to skip. Each refusal defers that run for two
+heartbeat ticks, doubling per consecutive refusal against the same lease, capped
+at the staleness cutoff. A deferral is forgotten as soon as the row leaves the
+stale window, so a fresh stall under a new lease is probed on the first tick
+that sees it.
+
+## Check a run's own view
+
+`DurableEngineState` answers directly, which is useful in a diagnostic tool:
+
+```ts
+import { DurableEngineState } from "@smthrs/engine-store"
+import * as Effect from "effect/Effect"
+
+const stalled = (staleBeforeMs: number) =>
+  Effect.gen(function*() {
+    const state = yield* DurableEngineState.DurableEngineState
+    const stale = yield* state.staleRunningRuns(staleBeforeMs, 64)
+    const parked = yield* state.waitingRuns({ reason: "released" })
+    return { stale, parked }
+  })
+```
+
+`staleRunningRuns` returns runs still `running` whose heartbeat froze before
+the horizon, each with that heartbeat. `waitingRuns(filter)` returns parked runs matching a reason, a
+`dueBeforeMs` bound, or a `cancelRequested` predicate. Results sort by ascending
+`wakeAt`, with untimed waits (`wakeAt: null`) after every timed wait and ties
+broken by `runId`. A `dueBeforeMs` bound includes matching deadlines and excludes
+untimed waits.
+
+## A restored backup needs an explicit fence
+
+Restoring does not clear ownership: a restored database still carries the owner
+tokens that were live when the backup was taken. Run
+`DisasterRecovery.fence(manifest)` against the restored file before any engine
+adopts it, or use `restoreAndFence`, which does both. See
+[Back up and restore the store](./back-up-and-restore.md).
+
+## Related
+
+- [Ownership and fencing](../concepts/ownership-and-fencing.md): the model
+  behind this task.
+- [Ownership](/docs/concepts/ownership/) on smithers.sh: the same story from
+  the run store's side.

package/docs/guides/share-a-cache-across-machines.md

@@ -0,0 +1,134 @@
+---
+title: "Share a step cache across machines"
+description: "Wire ArtifactSync and CacheSync onto a shared tier so one machine's sealed result serves another, and choose how eagerly a replay downloads the artifacts it references."
+sidebar:
+  order: 4
+---
+
+By default a recorded step result is local. Two seams turn it into a shared one:
+`ArtifactSync` publishes the blobs the result references, and `CacheSync`
+publishes the result entry itself. Both are absent by default, which is what
+keeps a purely local engine free of any remote-cache machinery.
+
+## Trust every writer of the shared tier
+
+A shared step-result tier is trusted to write into your workspace. A hit
+replays the entry's boundary evidence through `StepBoundary.replayOutputs`: it
+writes each recorded output, deletes each path recorded with `digest: null`,
+and prunes files under a recorded tree that the entry does not list. The
+integrity checks are self-consistent, not authenticating: inline bytes are
+checked against the digest recorded in the same entry, and blobs against their
+own address. Replay refuses paths outside the workspace, but inside it an entry
+chooses both the path and the content that lands on every machine that hits it.
+
+Only producers you trust as much as your own machines may write to the tier.
+This is the trust model of Bazel's remote cache.
+
+- Give consumers read-only credentials, and grant write access only to the
+  machines whose results you would run anyway, such as CI.
+- Use a separate tier per tenant or trust level rather than one tier shared
+  across them.
+
+A stronger property, accepting entries only from named producers, needs signed
+provenance on the entry. Another digest check cannot provide it.
+
+## Publish artifacts before entries
+
+Provide `ArtifactSync` with a shared `ArtifactStore`. The two tiers cannot both
+be layers, because they inhabit the same tag and composing both would shadow
+one, so the shared tier arrives as an effect:
+
+```ts
+import { ArtifactSync } from "@smthrs/engine-store"
+
+const artifacts = ArtifactSync.layer(remoteArtifactStore, { downloadPolicy: "all" })
+```
+
+`layer` takes the local tier from the `ArtifactStore` tag and the shared tier
+from the effect. `ArtifactSync.make({ local, remote, downloadPolicy })` is the
+same thing without the layer.
+
+`publish(digests)` runs `findMissing` on the shared tier, uploads what is
+missing, and re-probes to confirm. It is called immediately before the
+transaction that records the cache entry, and never inside it. A publication
+that cannot make the artifacts durable fails with `ArtifactPublicationFailed`
+and the shared entry is withheld.
+
+## Publish entries after they are durable
+
+```ts
+import { CacheSync } from "@smthrs/engine-store"
+
+const entries = CacheSync.layer(remoteCacheStore)
+```
+
+`publishEntry(entry)` reports a refusal rather than failing: `Option.none()`
+means the entry is now shared, and `Option.some(error)` means it is not, and
+why. A `Conflict` from the shared tier is not reported at all, because it means
+another machine recorded the key first, which is exactly the first-writer-wins
+outcome a shared tier exists to arbitrate.
+
+Compose it with `CombinedCacheStore` in `"deferred"` publication mode, which is
+the mode that leaves the shared write to this seam. Lookups stay read-through
+either way.
+
+## Neither seam can fail a run
+
+Both run after `attempts.finish`, so the result is already durably recorded on
+this host. A refusal withholds the shared copy, never the local row, and
+journals a `cache-provenance` record with `action: "unpublished"` carrying the
+stage (`artifacts` or `entry`) and the reason. A missing shared entry is
+therefore explainable from the journal rather than inferred from its absence.
+
+## Choose a download policy
+
+`hydrate(digests)` establishes that this host can resolve every referenced
+artifact and reports whether the replay is now worth retrying. It never fails a
+run: a shared tier that is down must not stop work that can simply be done.
+
+How eagerly it materializes is `downloadPolicy`. Declare it once on the shared
+tier as `RemoteArtifacts.Options.downloadPolicy` and both seams read it; pass it
+explicitly to `make` or `layer` to override.
+
+| Policy          | Choose it when                                                                                           |
+| --------------- | -------------------------------------------------------------------------------------------------------- |
+| `all` (default) | You want every later read to be local and want a shared tier outage after admission to cost nothing.     |
+| `toplevel`      | You read a small subset of a large output set and compose `CombinedArtifacts` over the same remote tier. |
+| `minimal`       | The same, and you do not want this host accumulating other machines' artifacts.                          |
+
+The two lazy policies are sound only when the store the replay reads through can
+reach the shared tier, which means `CombinedArtifacts` with the same remote
+tier. Under a purely local `ArtifactStore` an admitted lazy replay would later
+read an artifact this host never fetched.
+
+## Fall back to local explicitly
+
+`ArtifactSync.layerLocal` and `CacheSync.layerLocal` are the single-tier
+implementations: publish is a no-op and hydrate reports nothing arrived. They
+are what the engine falls back to when the tags are absent, and providing them
+explicitly says "this composition is local on purpose".
+
+## Make sure the results are shareable at all
+
+Publishing does nothing for a result that was never admitted to the cache. A
+record is admitted only when the action is sealed, the boundary is hard, no
+deviation occurred, and the evidence carries `wholeTreeWritesVerified: true`.
+Under the production composition that flag comes from
+`WorkspaceSandbox.layerFileSystem()`, so a composition with `StepBoundary.layer`
+and no sandbox produces run-local results however the sync seams are wired. See
+[Cache admission](../concepts/cache-admission.md).
+
+## What a replay does when the bytes are missing
+
+A verified hit calls `replayOutputs` before returning the stored result. When
+that refuses with `MissingArtifact`, the normal first answer for a row recorded
+on a machine whose artifacts this one has never seen, the dispatch hydrates from
+the shared tier and retries the replay exactly once before falling through to a
+real execution.
+
+## Related
+
+- [Cache admission](../concepts/cache-admission.md): the ordering argument and
+  the verification that runs before a hit is served.
+- [Artifacts and the cache](/docs/guides/artifacts-cache/) on smithers.sh: the
+  operator-facing version.

package/docs/guides/testing.md

@@ -0,0 +1,123 @@
+---
+title: "Test against a durable store"
+description: "Build every durable engine service over one migrated database, choose between the in-memory and file-backed bundles, and use the deterministic layers each seam ships."
+sidebar:
+  order: 11
+---
+
+Everything nondeterministic in this package is a service, so a test swaps the
+service rather than the code. This guide is the checklist.
+
+## Start from TestStores
+
+`@smthrs/engine-store/test/TestStores` builds every durable engine service over
+one database, with this package's composed migration set already installed:
+
+```ts
+import * as TestStores from "@smthrs/engine-store/test/TestStores"
+
+const stores = TestStores.layer()
+```
+
+`layer(options?)` uses a private in-memory database and keeps `SqlClient` to
+itself, which is what a case that only needs an engine wants. It provides the
+SQL journal, the run store, the attempt store, the cache store, the plan store,
+and `OwnerIdentity.layer`.
+
+`OwnerIdentity.layer` rides along rather than a pinned identity, so a test
+observes the same fresh-per-incarnation owner the production composition mints.
+Pin it with `OwnerIdentity.layerConstant(owner)` when a test needs to assert on
+the token itself.
+
+`TestStoresOptions` forwards `capacity`, `overflow`, and `batchSize` to the
+journal, for a test that wants to observe a saturated queue.
+
+## Take the connection when you need it
+
+`TestStores.databaseAt(filename)` is the migrated database alone, and
+`TestStores.layerAt(filename, options?)` is the full bundle with the `SqlClient`
+connection and `DurableEngineState` re-exported. Two shapes need it:
+
+- Adding another SQL-backed service, a control runtime for instance, over the
+  same database as the engine.
+- Pointing two independently constructed bundles at one file, which gives two
+  connections, two engines, and no shared object graph. That is what a second
+  process actually has.
+
+Use a real file for the second case. `:memory:` gives each connection its own
+private database, so it cannot prove anything durable across compositions. Pass
+`:memory:` to `layerAt` only when you want the cheap variant and still need the
+connection.
+
+`TestStores.database` is the in-memory migrated database exposed as `SqlClient`
+and `DurableWriter`.
+
+## The deterministic layer for each seam
+
+| Seam                 | Deterministic layer        | What it does                                                                                                                        |
+| -------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| `DurableEngineState` | `layerMemory`              | An in-memory implementation that, given a `runs` lookup, enforces the same park, wake, and clock ownership fences as the SQL layer. |
+| `StepBoundary`       | `layerTest(options)`       | Supports explicit `failure` or `deviation` fixtures, replay, and `readSnapshot` assertions.                                         |
+| `WorkspaceSandbox`   | `makeMemory(initialFiles)` | Deterministic and browser-safe, and the conformance implementation.                                                                 |
+| `StepSandbox`        | `layerTest(initialFiles)`  | The same transaction behind the scope-safe front door.                                                                              |
+| `StepSandbox`        | `layerNoop`                | Fails closed with `UnsupportedBoundary`, which models a host that cannot sandbox.                                                   |
+| `WakeBus`            | `layerNoop(overrides?)`    | Drops every wake, so every resume travels the polling fallback.                                                                     |
+| `Inconsistency`      | `layerNoop(overrides?)`    | Journals nothing and tolerates everything.                                                                                          |
+| `Selection`          | `layerNoop`                | Admits every candidate.                                                                                                             |
+| `Reconciliation`     | `layerDefault`             | The deterministic verdict function.                                                                                                 |
+| `ArtifactSync`       | `layerLocal`               | Publish is a no-op; hydrate reports nothing arrived.                                                                                |
+| `CacheSync`          | `layerLocal`               | A recorded entry is already everywhere it will be.                                                                                  |
+| `OwnerIdentity`      | `layerConstant(owner)`     | Pins the whole owner token.                                                                                                         |
+
+Both `DurableEngineState` implementations answer one behavior contract, so the
+memory twin cannot drift from the SQL one and a test written against the twin
+holds for the SQL store. The suite that pins them,
+[`DurableEngineStateContract.ts`](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/engine-store/test/contract/DurableEngineStateContract.ts),
+is not part of the published package. Copy it into your own project to run it
+against a third implementation you write.
+
+## What not to stub
+
+Do not replace `StepBoundary.layer` with `layerTest` and then assert on cache
+sharing. `layerTest` defaults `wholeTreeWriteDetection` to `true`, which is a
+fixture claim rather than a proof, and nothing composed that way ever proves the
+cross-run admission path. Use `StepBoundary.layer` with
+`WorkspaceSandbox.layerFileSystem()` when the cache behavior is what you are
+testing.
+
+Do not stub the liveness check with a function that returns `false` without
+asking. That is not a test double, it is a different, wrong policy: it says the
+owner is gone about an owner it never looked at.
+
+## Assert on the durable evidence
+
+The journal is the record, so most assertions read it back:
+
+- Claim arbitration writes `steal-refused-owner-alive` or
+  `stolen-and-activated`, each with its `evidence` kind.
+- A refused cache hit writes `cache-provenance` with
+  `action: "stale_read_set"`; a withheld shared entry writes the same record
+  with `action: "unpublished"` and the stage that refused.
+- The scheduler writes `plan-recorded`, `subgraph-appended`, `node-scheduled`,
+  `node-settled`, `node-invalidated`, and `selection-deferred`. A node record
+  names the tag its node dispatches under `action`, and a plan record carries
+  the graph under `graph.nodes`.
+- An interpreted flow writes the same records through `FlowRuntime.recordNode`,
+  addressed by a replay-stable `sourceId`, so a resumed walk holds one row per
+  node instead of one per observation. `@smthrs/journal`'s `EngineEvent` module
+  publishes the payload schema for every one of them.
+- A workspace transaction writes `diff-bundle-captured` and
+  `copy-back-settled`.
+
+`EngineStoreMetrics` is the other observable surface, and its counters are
+attributed by outcome, so a test can assert that a dispatch settled as a
+`VerifiedHit` rather than inferring it from timing.
+
+## Related
+
+- [Coordinate two processes](./coordinate-two-processes.md): the two-connection
+  pattern in full.
+- [Compose a durable engine](./compose-a-durable-engine.md): the production
+  composition these layers substitute into.
+
+Test filesystem classification with `StepBoundary.layer` over an in-memory `FileSystem` and `ArtifactStore`. `layerTest` returns supplied settlement evidence or failure verbatim; it does not classify changed paths, missing outputs, or surviving removals.

package/docs/installation.md

@@ -0,0 +1,102 @@
+---
+title: "Installation"
+description: "Install @smthrs/engine-store, its runtime requirements and import forms, the storage packages a runnable composition adds, and the browser boundary."
+sidebar:
+  order: 1
+---
+
+## Install the package
+
+Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
+
+The package requires Node.js 26.4.0 or later and ships as both ESM and
+CommonJS with TypeScript declarations. Its runtime dependencies, including
+[`effect`](https://effect.website) and the `@smthrs/*` storage packages it
+composes, install with it. Every surface on this site is an Effect service:
+`layer` values you compose and `Effect` values you run.
+
+## Import forms
+
+The root entry point re-exports every module as a namespace:
+
+```ts
+import { DurableEngineState, EngineStore, Migrations, StepBoundary, WorkspaceSandbox } from "@smthrs/engine-store"
+```
+
+Each module is also importable from its own subpath, which is the form the API
+reference uses:
+
+```ts
+import * as EngineStore from "@smthrs/engine-store/EngineStore"
+import * as Retention from "@smthrs/engine-store/Retention"
+```
+
+Three subpath forms are not public and are blocked in the export map:
+`@smthrs/engine-store/internal/*`, `@smthrs/engine-store/migrations/*`, and
+`@smthrs/engine-store/*/index`. `@smthrs/engine-store/package.json` is exported.
+
+The deterministic store bundle for tests lives at
+`@smthrs/engine-store/test/TestStores`. See
+[Test against a durable store](./guides/testing.md).
+
+## What a runnable composition adds
+
+This package composes storage rather than providing it. A composition that
+executes a flow supplies the four stores, a database, an artifact store, a
+workspace root, and a platform.
+
+| Package                                 | What it supplies                                                                                                            |
+| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
+| [`@smthrs/journal`](/api/journal)       | `Journal`, the durable event log every engine decision is written to.                                                       |
+| [`@smthrs/run-store`](/api/run-store)   | `RunStore` and `AttemptStore`, the run rows and attempt rows ownership is fenced on, plus the `Ownership` liveness helpers. |
+| [`@smthrs/step-cache`](/api/step-cache) | `CacheStore`, the content-addressed step result cache.                                                                      |
+| [`@smthrs/database`](/api/database)     | `SqlClient` and `DurableWriter`, and the `Migrations` runner the schema installs through.                                   |
+| [`@smthrs/artifacts`](/api/artifacts)   | `ArtifactStore`, where outputs too large to inline are spilled by digest.                                                   |
+| [`@smthrs/kernel`](/api/kernel)         | `Jj` for snapshot boundaries and `Workspace` for the workspace root.                                                        |
+| [`@smthrs/plan`](/api/plan)             | `PlanStore`, required only when you drive plans with `PlanScheduler`.                                                       |
+
+[`@smthrs/flows`](/api/flows) ships `NodeRuntime`, which wires all of the above
+over one SQLite file. Reach for it when you want the composition rather than
+control over it; reach for `EngineStore.layer` when you are building a host.
+
+## Install the schema before anything reads it
+
+`Migrations.sets` is the complete durable engine schema in dependency order
+(journal, run store, step cache, this package's own tables, then the plan
+store), and `Migrations.layer` installs it:
+
+```ts
+import { DurableWriter } from "@smthrs/database"
+import * as NodeDatabase from "@smthrs/database/node/NodeDatabase"
+import { Migrations } from "@smthrs/engine-store"
+import * as Layer from "effect/Layer"
+
+const database = (filename: string) =>
+  Layer.provideMerge(
+    Migrations.layer,
+    Layer.provideMerge(DurableWriter.layer(), NodeDatabase.layer({ filename }))
+  )
+```
+
+Provide the stores over that layer, never beside it: a store built before the
+migration runs reads a table that does not exist yet.
+
+## The browser boundary
+
+This entry point bundles for a browser. The two host reads it once made
+directly, `process.pid` and `randomUUID` from `node:crypto`, enter through the
+injectable `OwnerIdentity` service, and every package it composes is
+browser-bundleable. A release that broke the bundle would fail the build before
+it shipped.
+
+Bundling is not running. The only durable backing shipped here is local SQLite
+through Node.js `node:sqlite` and `@effect/sql-sqlite-node`. A browser or edge
+deployment can import the types and the browser-safe in-memory helpers
+(`DurableEngineState.layerMemory`, `WorkspaceSandbox.makeMemory`,
+`StepSandbox.layerNoop`), but cannot execute durable flows. Supplying an
+alternative browser SQL client is not a supported runtime.
+
+## Next step
+
+Compose the engine and run a flow twice over one file in the
+[Quickstart](./quickstart.md).