@smthrs/run-store 0.0.0-stage → 1.0.0-rc.4

package/docs/concepts/durable-values.md

@@ -0,0 +1,140 @@
+---
+title: "Durable values"
+description: "How run state, checkpoints, errors, outcomes, and metadata cross the persistence boundary: inert copies, shape limits rather than byte limits, the one configurable ceiling, and why nothing is redacted."
+sidebar:
+  order: 4
+---
+
+Every value these stores persist is executable resume data. A run's
+`stateJson` is what a restarted process re-enters; an attempt's checkpoint is
+where a retried step picks up. That single fact decides every rule on this
+page.
+
+## Values are copied inert before anything can yield
+
+A JavaScript object handed to a store is not trusted to hold still. Before
+persistence can yield, the boundary walks the value and builds a detached copy,
+reading only own enumerable data properties. What it refuses:
+
+- **Accessors.** A getter would run application code inside the persistence
+  path, and could return different values to the validator and to the writer.
+- **`toJSON` and non-plain objects.** A prototype other than `Object.prototype`
+  or `null` is refused, so a class instance never persists as whatever its
+  serializer felt like emitting.
+- **Cycles**, sparse arrays, enumerable non-index array members, enumerable
+  symbols, and non-finite numbers.
+- **Ill-formed text.** A lone UTF-16 surrogate or an embedded NUL is refused,
+  because it would not survive the round trip through SQLite unchanged.
+
+The copy is frozen, so what was validated is what gets written. Stored rows are
+validated again on the way out: a row edited by something other than these
+stores fails with `decode_failed` rather than being handed back as state.
+
+## The bounds are on shape, not on size
+
+Run state, metadata, errors, and outcomes have no byte ceiling, on purpose.
+These rows are what a resume re-enters, so a multi-megabyte state a flow
+legitimately produced has to persist or the run cannot continue. What is
+bounded is shape, and the limits are public constants:
+
+| Constant                        | Value   | Applies to            |
+| ------------------------------- | ------- | --------------------- |
+| `RunStore.maximumRunJsonDepth`  | 128     | Executable run state. |
+| `RunStore.maximumRunJsonNodes`  | 100,000 | Executable run state. |
+| `AttemptStore.maximumJsonDepth` | 128     | Every attempt value.  |
+| `AttemptStore.maximumJsonNodes` | 100,000 | Every attempt value.  |
+
+Depth bounds the recursion the boundary itself performs. Node and member counts
+bound the work one admission can cost. A value past either is refused with
+`invalid_run` or `invalid_attempt` and a complaint naming the limit it broke.
+
+## The one byte ceiling is the checkpoint
+
+A checkpoint is the exception, because it is a convenience rather than a
+requirement: a step that cannot checkpoint still runs, it just restarts from
+the beginning. So it takes a policy:
+
+```ts
+import { AttemptStore } from "@smthrs/run-store"
+
+const attempts = AttemptStore.layerWith({ maxCheckpointBytes: 4 * 1024 * 1024 })
+```
+
+`Options.maxCheckpointBytes` defaults to 1 MiB and may not exceed
+`AttemptStore.maximumCheckpointBytes`, which is 16 MiB. A larger value is
+refused when the store is built, not when the first oversized checkpoint
+arrives.
+
+## Identifiers are bounded durable text
+
+Run ids, step key digests, lineage ids, owner host ids and nonces, and attempt
+state names are all validated the same way: non-empty, at most 1,024 UTF-16
+units, no NUL, and no lone surrogate. `U+FFFD` is ordinary text and is
+accepted; an astral character round-trips byte for byte.
+
+## Timestamps are checked independently
+
+Every timestamp must be a non-negative safe integer, and each is range-checked
+on its own. None is compared against another. An attempt whose `finishedAtMs`
+precedes its `startedAtMs` persists exactly as written, because the store does
+not adjudicate the caller's timeline: it records what the caller observed.
+
+The two exceptions are the ones that are lease predicates rather than records.
+Heartbeats are monotonic, and lease readings are bounded by the skew allowance.
+Both are covered in [The heartbeat lease](./leases.md).
+
+## Nothing is redacted
+
+The stores rewrite no field on the way through. A redactor here would silently
+change what the flow re-reads on resume, which corrupts the run rather than
+protecting it: a field named `token` in executable state is state, and a run
+that resumes without it is broken.
+
+`Schema.Redacted` hides inspection but allows schema JSON encoding by default.
+`Schema.encodeSync(Schema.toCodecJson(schema))` unwraps a default `Redacted`
+field, and the store persists the resulting credential bytes. Refuse JSON
+encoding explicitly for values that must never be persisted:
+
+```ts
+const RuntimeState = Schema.Struct({
+  apiKey: Schema.Redacted(Schema.String, { disallowJsonEncode: true })
+})
+```
+
+This rejects serialization; it does not omit the field. For resumable state,
+use a persistence schema that excludes credentials and stores a secret
+reference instead:
+
+```ts
+const PersistedState = Schema.Struct({
+  secretRef: Schema.String,
+  pageToken: Schema.String
+})
+```
+
+Resolve `secretRef` from a secret provider on resume. Persist only the
+reference and flow data, never the resolved credential. Values that must not
+be published are handled on the journal-event and export surfaces, which is
+where the observability boundary lives.
+
+### The run file is a secret store
+
+Two columns keep run values verbatim because resume re-reads them byte for
+byte:
+
+- `flows_runs.state_json` holds the run's payload and result.
+- `flows_attempts.outcome_json` holds each action's encoded result, which an
+  action replays instead of executing again.
+
+A credential in a payload, or spliced into an action's success value, is
+stored in cleartext in the run's SQLite file (`.flows/*.db`). Handle that file
+as a secret store: restrict its permissions, keep it out of commits, bug
+reports, and artifacts, and delete it when the run is no longer needed. The
+journal table is redacted on the write path; the fault suite's case 22 scans
+every table and fails on a credential in any other column.
+
+The stores hold that line in their own diagnostics as well. Failure causes
+reach logs, spans, and telemetry, so they carry field names, lengths, and
+validity flags, and never the rejected payload. Malformed timestamp causes
+contain only the field name and a fixed validation complaint. Clock-skew
+diagnostics include timestamp readings only after numeric validation.

package/docs/concepts/execution-revisions.md

@@ -0,0 +1,53 @@
+---
+title: "Execution revisions and cancellation acknowledgement"
+description: "Database-owned monotonic run revisions, retained deletion evidence, and fenced acknowledgement of cancellation intent."
+---
+
+Migration `0003_execution_revisions` adds `flows_run_source`,
+`flows_run_changes`, and nullable `cancel_acknowledgement_json` on `flows_runs`.
+It is additive: old rows remain intact and receive a baseline revision in
+creation-time/run-ID order. That ordering does not claim to reconstruct their
+unknown historical mutations. Old acknowledgement values remain null.
+
+The source table holds a random 128-bit database identity and a non-negative
+safe-integer revision. Triggers advance it for every inserted, updated or deleted
+run row, including claims, activation, heartbeat, wait fields, cancellation,
+parent/lineage changes and terminal settlement. Trigger effects roll back with
+the mutation. Safe-integer overflow refuses the mutation instead of wrapping.
+Run IDs are immutable; deletion followed by creation is a new, higher revision.
+
+`flows_run_changes` holds the latest revision and deletion flag for each run ID.
+Multiple mutations coalesce; revisions are not journal event sequence numbers.
+Deleted IDs retain their tombstones indefinitely. No ordinary retention pass or
+consumer acknowledgement removes them. This policy protects consumers offline
+for any duration without assuming a consumer lease. It costs one retained entry
+per distinct deleted ID. Recreating an ID replaces the tombstone with a newer
+live revision. A future compaction policy must first provide registered-consumer
+watermarks and a durable rebuild protocol.
+
+`RunStore.acknowledgeCancel(runId, owner, nowMs)` records the first observing
+owner and time. The SQL mutation requires both durable intent and an exact
+running owner fence. It returns true for a matching owner, preserving the first
+acknowledgement on repetition, or false when intent or ownership is absent.
+Invalid inputs and storage failures use the existing `RunStoreError` channel.
+`makeNoop` returns false for this operation.
+
+Intent, acknowledgement, and terminal cancellation are distinct facts. A request
+can exist before the owner observes it. An acknowledgement can exist while user
+cleanup is still running. Neither acknowledgement nor a cancelled lifecycle
+proves an arbitrary external effect was undone.
+
+The engine owns the public snapshot and change-feed ports that consume this
+tracking state. Its wait payload already lives on the run row, so parking and
+waking advance the same revision. Engine-store's additive migration binds
+durable spawn-parent changes to the affected child's revision as well.
+
+The identity survives connection and process restarts. Backups copy that identity;
+independently writing divergent copies or restoring an older revision requires
+explicit source rotation and projection rebuild by the host. Revisions are local
+to one database. They do not establish a transaction with the separate control
+database, and a projection's applied watermark can lag the engine's watermark.
+
+See [engine observation and listing](https://engine-store.smithers.sh/guides/observe-executions/)
+for bounded snapshots, keyset cursor semantics, live-page concurrency and the
+revision comparison that rejects stale projection updates.

package/docs/concepts/fencing.md

@@ -0,0 +1,146 @@
+---
+title: "Fencing and ownership"
+description: "How a run row decides which process may mutate it: the OwnerId fence, the exact-snapshot compare-and-swap, the two claim paths, and why competition is a success value rather than an error."
+sidebar:
+  order: 1
+---
+
+A durable run outlives the process that started it. Two processes over one
+database will at some point both believe they should be running the same run,
+and one of them has to lose without corrupting anything. That arbitration is
+what this package is for.
+
+## The fence is an identity, not a lock
+
+`OwnerId` is three fields, defined by [`@smthrs/journal`](/api/journal) and
+re-exported from `Ownership` because it is the same token the journal accepts
+on durable appends:
+
+```ts
+import type { OwnerId } from "@smthrs/run-store/Ownership"
+
+const owner: OwnerId = { hostId: "host-a", pid: 4102, nonce: "3f9c-8b21" }
+```
+
+`hostId` names the machine, `pid` names the process, and `nonce` distinguishes
+one incarnation of that process from the next. The nonce is what makes a
+restarted process a different owner from its predecessor, and what lets two
+engines composed inside one process tell each other apart.
+
+Nothing is locked. Every owned write asks the injected `Consensus` strategy
+whether the complete triple still holds the run inside the same serialized
+write transaction as the mutation, and the run row is updated only after the
+strategy answers:
+
+```ts
+const held = yield * consensus.guard(runId, owner) // fails fence_lost otherwise
+yield * sql`UPDATE flows_runs SET ... WHERE run_id = ${runId}`
+```
+
+The durable writer serializes write transactions, so there is no window
+between checking who owns the run and writing to it, and a displaced owner's
+late write fails instead of racing. `RunStore.layer` arbitrates through the
+journal's database-backed `SqlConsensus`, whose lease lives in
+`flows_consensus_leases`; `RunStore.layerWith` takes any strategy, and the
+journal must be built over the same instance so both agree on who holds the
+run.
+
+## The snapshot is the compare-and-swap
+
+`RunSnapshot` is the exact triple a claim guards: `status`, `owner`, and
+`heartbeatAtMs`. You read the row, restate those three fields, and hand them
+back with your write. The store admits the write only while the row still
+matches, and then only if the strategy grants the claim: the row admits, the
+lease decides, and a refusal is reported from the row exactly as a failed
+compare-and-swap always was.
+
+```ts
+import type { RunRow, RunSnapshot } from "@smthrs/run-store/RunStore"
+
+const snapshot = (row: RunRow): RunSnapshot => ({
+  status: row.status,
+  owner: row.owner,
+  heartbeatAtMs: row.heartbeatAtMs
+})
+```
+
+Partial snapshots are refused as invalid input rather than treated as a miss.
+`running` requires both an owner and a heartbeat; every other status requires
+neither. That invariant is also a SQL `CHECK` on the table, so no writer,
+including one issuing raw SQL, can leave a half-owned row behind.
+
+## Two ways to take a run
+
+Claiming is deliberately two phase for the engine's path, and atomic for the
+control plane's.
+
+**Two phase.** `claim` writes the claim columns (`claim_host_id`, `claim_pid`,
+`claim_nonce`, `claimed_at_ms`) on a `pending` or `suspended` row and returns
+the `claimedAtMs` that becomes your fence token. `activate` then trades that
+token for ownership: it sets `status = 'running'`, writes the owner columns and
+the first heartbeat, and clears the claim. The gap between them is where a
+caller does work that must not be repeated, such as recording in the journal
+who is taking the run. If activation loses, `abandonClaim` releases the
+reservation so the next sweeper is not blocked by a claim nobody is using.
+
+**Atomic.** `claimAndOwn` does both in one compare-and-swap. It admits
+`pending`, `suspended`, and `running` rows, and replacing a different running
+owner additionally requires matching liveness evidence. Use it when there is no
+work to do between reserving and owning.
+
+`steal` is the third entry, and it produces a claim rather than ownership: a
+successful steal writes the claim columns of a stale running row, and the
+caller follows with `activate`. That is why the takeover path and the ordinary
+claim path converge on the same second step.
+
+Every transition the strategy grants — `claimed`, `activated`, `released`,
+`stolen`, `expired` — is appended as a `flows.consensus.*` fact on the run's
+`run-store` companion journal stream in the same transaction, beside the run's
+own stream rather than in it. Heartbeats renew the strategy's lease and never
+enter the journal.
+
+## Competition is a value, failure is an error
+
+Every operation splits its answers into two channels, and the split is the
+contract:
+
+- **Success values** describe the race. `AlreadyClaimed`, `HeartbeatFresh`,
+  `SnapshotChanged`, `FenceLost`, `ClaimLost`, `ClaimFresh`, `ClaimChanged`,
+  `EvidenceRequired`, and `LivenessUnconfirmed` all mean the store worked
+  correctly and your write did not win.
+- **`RunStoreError`** describes a defect. Invalid input (`invalid_run`), a
+  corrupt durable row (`decode_failed`), a constraint violation
+  (`constraint`), a missing row on a direct read (`not_found_row`), and a
+  database failure (`persistence_failed`).
+
+A caller can therefore retry contention without retrying corruption, and can
+type its error channel without swallowing races. Each outcome and its cause is
+listed in [Troubleshooting](../troubleshooting.md).
+
+## Leaving a run
+
+`transitionOwned` is the only way to move a run you own, and the target decides
+what happens to the ownership columns:
+
+- `running` keeps the owner and updates only the executable state. It is how a
+  run records progress mid-flight.
+- `suspended` clears the owner, the heartbeat, and the claim, and leaves
+  `finished_at_ms` null. The run is parked and claimable again.
+- `completed`, `failed`, and `cancelled` clear the same columns and stamp
+  `finished_at_ms`. A terminal row is never claimed, activated, or reopened.
+- `pending` is refused with `invalid_run`. A run does not go back to unstarted.
+
+A transition may also carry a `TransitionGuard`, an extra predicate evaluated
+in the same write transaction as the fence check. `{ cancelRequested:
+"absent" }` is the "do not finalize a run somebody asked to cancel" rule,
+expressed inside the serialized transaction rather than as a read-then-write
+race. See [Cancel a run](../guides/cancel-a-run.md).
+
+## Attempts inherit the run's fence
+
+`AttemptStore` never carries its own ownership. `put`, `heartbeat`, `finish`,
+and `patch` each embed an `EXISTS` subquery over `flows_runs` requiring that
+the run is `running` and owned by the caller. One consequence is worth knowing
+before you debug it: once a run reaches a terminal status its owner columns are
+cleared, so every later attempt write on that run reports `FenceLost`. A
+delayed write from a displaced owner cannot rewrite the winning row.

package/docs/concepts/leases.md

@@ -0,0 +1,154 @@
+---
+title: "The heartbeat lease"
+description: "The four heartbeat durations and how they relate, the two clock sources that stamp one run row, the skew allowance the store enforces, and what a wall-clock lease can and cannot promise."
+sidebar:
+  order: 2
+---
+
+An owner proves it is still working by writing `heartbeat_at_ms`. A peer that
+wants the run reads that stamp and decides whether the owner is gone. Every
+number in that exchange is a constant this package defines, and they are
+related rather than independently chosen.
+
+## The four durations
+
+They live in `@smthrs/run-store/Heartbeat` and are re-exported from
+`Ownership`. All four are `Duration` values:
+
+| Constant                  | Value      | What it governs                                                                           |
+| ------------------------- | ---------- | ----------------------------------------------------------------------------------------- |
+| `heartbeatInterval`       | 1 second   | How often the supervision loop pulses.                                                    |
+| `heartbeatStaleAfter`     | 30 seconds | How old a persisted heartbeat must be before a peer may steal the run.                    |
+| `heartbeatSkewAllowance`  | 10 seconds | How far the owner's wall clock may lag a peer's before the lease reasoning stops holding. |
+| `heartbeatWriteTolerance` | 19 seconds | How long an owner may keep working through failing or stalled heartbeat writes.           |
+
+The last one is derived, not chosen:
+`heartbeatStaleAfter - heartbeatSkewAllowance - heartbeatInterval`. A peer may
+steal the run the instant the persisted heartbeat is `heartbeatStaleAfter` old.
+The peer's clock may already read `heartbeatSkewAllowance` later than the
+owner's, so an owner that tolerated write failures for the full staleness
+window would still be running side effects when the steal was admitted.
+Subtracting the skew allowance and an extra pulse interval reserves one
+interval to reconfirm ownership before interrupting.
+
+They live in a leaf module because `Ownership` imports `RunStore` and
+`RunStore` needs the staleness cutoff for its own predicates. Neither could own
+the constants without the other restating them, and a restated constant is a
+constant that drifts.
+
+## The supervision loop
+
+`Ownership.heartbeatLoop` is the owner's half of the exchange. It pulses
+forever and interrupts itself when the fence is gone, so race it against the
+owned work with `Effect.raceFirst`, which settles on the first outcome of
+either side, interruption included:
+
+```ts
+import { Ownership, RunStore } from "@smthrs/run-store"
+import * as Effect from "effect/Effect"
+
+const owned = <A, E>(
+  runId: string,
+  owner: Ownership.OwnerId,
+  work: Effect.Effect<A, E, RunStore.RunStore>
+): Effect.Effect<A, E, RunStore.RunStore> => Effect.raceFirst(work, Ownership.heartbeatLoop(runId, owner))
+```
+
+The loop pulses every `heartbeatInterval`, reading the injected Effect `Clock`,
+so a test drives it with `TestClock` and never waits in wall time. It
+distinguishes two failures on purpose:
+
+- **A lost fence** is any heartbeat outcome other than `Updated`. That is
+  durable evidence that another owner holds the run, so the loop interrupts
+  immediately and takes the raced work with it.
+- **A failed write** is not evidence of anything. The persisted heartbeat is
+  still on the row and no peer may steal the run until it is
+  `heartbeatStaleAfter` old, so transient database errors are tolerated for
+  `heartbeatWriteTolerance`.
+
+When that budget expires, an independent deadline calls `RunStore.reconfirm`,
+bounded by one `heartbeatInterval`. It delegates to the same `Consensus`
+strategy as ordinary heartbeats and mirrors the renewed stamp in the same
+transaction. Reconfirm succeeds only when the owner still matches and there
+is no pending claim. A peer that reserved the stale lease therefore wins
+even before it activates.
+
+Reconfirmation keeps its one-heartbeat-interval deadline. When it succeeds,
+supervision resets the confirmed-pulse time from the timestamp sent to the
+store. `onReconfirm(unconfirmedMs)` must finish within the remaining renewed
+`heartbeatWriteTolerance` budget; reconfirmation time is subtracted, and receipt
+completion does not extend the persisted lease. A blocked or failed receipt
+lapses the lease. The engine journals
+`run-decision` with decision `lease-reconfirmed` and detail `{ unconfirmedMs }`;
+`runs show` reports a warning. Owner identity and journal generation do not
+change. A lost lease, write error, or timeout calls `onLapse` and interrupts
+the pulse loop and its pending write, including when that write never returns. A successful pulse
+re-arms the deadline from the timestamp supplied to the store, not the time the
+write completes. The deadline reads the current clock after each wait; delayed
+successes and failures cannot extend the lease beyond that persisted timestamp's
+budget. Until the first successful pulse, the budget starts when supervision
+begins.
+
+## Heartbeats only move forward
+
+`heartbeat` writes `MAX(heartbeat_at_ms, :nowMs)`. A pulse that arrives after a
+newer one from the same owner still reports `Updated`, because the fence held
+and the write proved liveness, but it does not move the stamp backwards and
+make a live run look stale to a peer's cutoff. The same rule applies to attempt
+heartbeats.
+
+## Two clocks stamp one row
+
+A run row carries readings from two different sources, and the split is
+deliberate.
+
+**The caller's `nowMs`** is used by the operations that judge or record a
+lease: `claim`, `claimAndOwn`, `steal`, `heartbeat`, `reconfirm`, `requestCancel`, and
+`recoverClaim`. It is the cutoff their predicates compare against and the value
+they persist, and it is taken literally.
+
+**The Effect `Clock`** stamps the lifecycle. `create` writes `created_at_ms`,
+`activate` writes `started_at_ms` and `heartbeat_at_ms`, and `transitionOwned`
+writes `finished_at_ms`.
+
+One row can therefore hold readings from both, so a composition must give both
+sources the same reading. In practice that means taking `nowMs` from
+`Clock.currentTimeMillis` rather than from `Date.now()`, which is also what
+makes the whole store driveable under `TestClock`.
+
+## The skew allowance is enforced
+
+Every `nowMs` is validated as a non-negative safe integer. The lease operations
+(`claim`, `claimAndOwn`, `steal`, `heartbeat`, `reconfirm`, and `recoverClaim`) then bound it
+from above: a reading more than `heartbeatSkewAllowance` ahead of the store's
+own `Clock` fails with `invalid_run` before any predicate runs.
+
+No composition produces such a reading honestly, because the caller and the
+store share a process, and it is the one lever that steals a fresh owner or
+pins a lease past the cutoff. A reading behind the clock is admitted: it makes
+every staleness judgment more conservative, and the monotonic heartbeat absorbs
+it.
+
+Two operations sit outside that bound on purpose. `requestCancel` keeps its
+literal reading because its timestamp is request data rather than a lease
+predicate. The `claimedAtMs` fence tokens passed to `activate`, `abandonClaim`,
+and `recoverClaim` are compared against the row rather than bounded, because
+they are tokens the store itself issued.
+
+Inside the allowance the reading is trusted completely: the store cannot tell a
+slow clock from a lie. That is the right contract for an in-process library over
+a local SQLite file whose caller can issue raw SQL anyway, and it must not
+cross a trust boundary.
+
+## What the lease does not promise
+
+Beyond `heartbeatSkewAllowance` of clock offset between two hosts, a peer can
+be admitted while the previous owner is still running. Durable writes stay safe
+regardless, because they are fenced by the consensus strategy's commit-time
+guard and the displaced owner's writes fail rather than corrupt. Non-durable external side
+effects, such as an HTTP call or a spawned process, can genuinely overlap.
+
+That is inherent to any wall-clock lease. A caller that cannot tolerate any
+overlap needs a fencing token at the side effect itself, not a larger timeout.
+The next lever after the lease is evidence: see
+[Liveness evidence](./liveness-evidence.md).

package/docs/concepts/liveness-evidence.md

@@ -0,0 +1,142 @@
+---
+title: "Liveness evidence"
+description: "The three kinds of evidence that admit a takeover, why evidence binds to one instant, the difference between a LivenessCheck and a LivenessProbe, and why every probe failure is read as life."
+sidebar:
+  order: 3
+---
+
+An expired lease says the owner stopped writing. It does not say the owner
+stopped working. Evidence is the second lever: a claim about the recorded owner
+that the store can check before it admits a takeover.
+
+## Three kinds, three scopes
+
+`Ownership.LivenessEvidence` carries the owner it is about, the instant it was
+observed, and what kind of claim it is:
+
+```ts
+import type { LivenessEvidence } from "@smthrs/run-store/Ownership"
+
+const evidence: LivenessEvidence = {
+  expectedOwner: { hostId: "host-a", pid: 4102, nonce: "3f9c" },
+  checkedAtMs: 1757000030000,
+  kind: "same-host-pid-dead"
+}
+```
+
+Each kind is admitted only where it means something:
+
+- **`same-host-pid-dead`** is a local process probe. A pid names a process only
+  inside one process namespace, so the store accepts this kind only when the
+  observer and the recorded owner share a `hostId`.
+- **`cross-host-unreachable-stale`** is a reachability judgment the deployment
+  makes. It is accepted only when the hosts differ: on the owner's own host it
+  would be an unprobed guess dressed as evidence.
+- **`lease-expired`** asserts only that the persisted heartbeat is older than
+  the staleness cutoff. That is the one claim the store can verify for itself,
+  and `steal` does verify it: the write refuses any row whose `heartbeat_at_ms`
+  is still inside the window. It is therefore accepted from any host.
+
+Two of the three are collected outside this package, because the store never
+probes a process or a network itself. It validates what it is handed.
+
+## Evidence binds to one instant
+
+`evidence.checkedAtMs` must equal the `nowMs` of the call that consumes it,
+exactly. A probe taken at T is refused by a call made at T plus one
+millisecond, and the caller has to build fresh evidence.
+
+That rule is what stops a probe from being replayed. Without it, a caller could
+hold one "the owner is dead" observation and spend it against a run whose owner
+came back. `claimAndOwn`, `recoverClaim`, and `steal` all enforce it.
+
+`steal` also separates two refusals that look alike from outside.
+`LivenessUnconfirmed` means the evidence did not match the snapshot's owner,
+the host relation its `kind` requires, or `nowMs`, so no compare-and-swap ran
+at all. `SnapshotChanged` is reserved for matching evidence whose comparison
+lost to a row that moved. A caller can tell "I was not allowed to try" from "I
+tried and lost".
+
+## A check answers, a probe produces
+
+Two types sit either side of the store, and they answer different questions.
+
+**`LivenessCheck`** is the question the arbitration asks before it decides to
+steal: is the recorded owner still working? It takes the owner and a
+`LivenessContext` carrying the claimant, the persisted `heartbeatAtMs`, and the
+`nowMs` the decision is made against, and answers a boolean. Answering `true`
+refuses the takeover. The engine consults it only for a run whose lease has
+already expired.
+
+**`LivenessProbe`** is the evidence factory: given the expected owner, the
+claimant, and the instant, it produces `LivenessEvidence` or `undefined` for
+`steal`, `claimAndOwn`, and `recoverClaim` to verify.
+
+The package ships two checks.
+
+`Ownership.leaseLiveness(staleAfter?)` is the default and the honest floor: an
+owner is alive for as long as its persisted heartbeat is younger than the
+cutoff, and gone once it is not. It is the weakest answer, and the one every
+host can give. A fresh process with no application code at all can reclaim a
+hard-killed owner's runs once the lease it stopped renewing expires. Browser
+compositions keep it, because a tab has no process table to ask.
+
+`Ownership.sameHostPidProbe` asks the operating system instead. It is what a
+Node host hands the engine, so this snippet also installs
+[`@smthrs/flows`](/api/flows):
+
+```ts
+import type * as NodeRuntime from "@smthrs/flows/NodeRuntime"
+import { Ownership } from "@smthrs/run-store"
+
+const options: NodeRuntime.Options = {
+  filename: ".smithers/smithers.db",
+  workspaceRoot: "/srv/workspace",
+  owner: { hostId: "host-a" },
+  isAlive: Ownership.sameHostPidProbe
+}
+```
+
+It exists because the lease alone is only a timeout: two engine processes over
+one database steal each other's running rows `heartbeatStaleAfter` after any
+heartbeat stall, whether that stall was a dead process, a stop-the-world pause,
+or a disk that blocked too long. The probe answers the question the lease is
+standing in for.
+
+## Every unknown answer is life
+
+`sameHostPidProbe` sends no signal. `process.kill(pid, 0)` performs only the
+delivery checks, and exactly three answers matter:
+
+- It returns: the process exists. The owner is alive.
+- It throws `ESRCH`: no such process. This is the sole death answer.
+- It throws anything else, including `EPERM` and `EINVAL`, or the pid is not a
+  positive safe integer: the owner is treated as alive, because an unknown
+  answer is not death.
+
+Failing closed is the whole point. A check that returns `false` without asking
+says "that owner is gone" about an owner it never looked at, and the engine
+steals runs out of live processes on the strength of it. That is why the
+engine's `NodeRuntime` requires `isAlive` rather than defaulting it.
+
+Two limits are inherent to asking a pid, and they bound what reclaim can
+promise:
+
+- **An owner recorded with the claimant's own pid is always alive.** A previous
+  incarnation of this process, or a second engine composed inside it, differs
+  from the claimant only by `nonce`, and the process it names is this one. Such
+  a row is never stolen while the process lives, so an embedded host that
+  re-creates its engine in place should keep `leaseLiveness`, whose timeout does
+  expire.
+- **A reused pid reports the unrelated process that now holds it.** The dead
+  owner's row stays refused for as long as that process lives, which delays
+  reclaim rather than breaking it: the row is still `running` under an expired
+  lease, and the next probe after the pid is free reclaims it.
+
+A recorded owner on another host is never probed. The answer is `false`, which
+adds no evidence and lets the expired lease decide, so a host that dies for
+good does not strand its runs where no other machine could ever produce
+evidence about its pids.
+
+For the sequence a takeover actually runs, see
+[Take over a stalled run](../guides/recover-a-stalled-run.md).