@smthrs/engine-store 0.0.0-stage → 1.0.0-rc.3

package/docs/concepts/durable-waits.md

@@ -0,0 +1,175 @@
+---
+title: "Durable waits"
+description: "How a parked run survives a restart: durable deferreds and clocks, the waiting taxonomy, the sweeps that re-drive stranded runs, and the wake bus that makes the common case fast."
+sidebar:
+  order: 5
+---
+
+A flow that waits is the hard case for durability. The process holding the
+timer can die, the completion can arrive while nothing is listening, and the run
+can be left parked with no one scheduled to wake it. `DurableEngineState` is the
+state that makes waiting survive all three.
+
+## Two durable primitives
+
+A **deferred** is addressed by flow name, execution id, and deferred name. It is
+completed once, first writer wins, and its `exit` is the value the resumed run
+receives.
+
+A **clock** is addressed the same way and carries an absolute `dueAtMs`.
+Scheduling is first writer wins and fenced against the active run owner, so a
+process that no longer owns a run cannot arm a timer against it.
+
+Both report outcome unions rather than booleans, so a caller can tell a fresh
+write from an idempotent repeat:
+
+| Call               | Outcomes                                       |
+| ------------------ | ---------------------------------------------- |
+| `completeDeferred` | `Completed` or `Existing`                      |
+| `scheduleClock`    | `Scheduled` or `Existing`                      |
+| `completeClock`    | `Completed`, `AlreadyCompleted`, or `NotFound` |
+| `park`             | `Parked` or `NotFound`                         |
+| `wake`             | `Woken`, `NotWaiting`, or `NotFound`           |
+
+A successful mutation means the row is durable. Callers may therefore journal
+and schedule a wake only after the mutation returns.
+
+The additive [state event schemas](https://journal.smithers.sh/concepts/state-event-authority/)
+describe execution, deferred and clock history with explicit lineage. Existing
+writers retain their bytes and these stores remain authoritative. A retained
+attempt-history projection cannot replace deferred, clock or ownership state.
+
+## The waiting taxonomy
+
+A run that suspends parks first. The driver writes a `Waiting` payload with a
+`reason`, an optional `wakeAt`, and an optional `token`:
+
+- reason `timer`, with the earliest pending clock deadline as `wakeAt`, when a
+  durable clock is outstanding.
+- reason `event` otherwise.
+- reason `released` for a run interrupt-released by shutdown or a lapsed lease;
+  its `interrupt-released` decision records which as `cause`.
+- reason `quarantine` for a run parked on corrupt attempt evidence.
+
+Every resume clears the payload when the run re-enters `running`. Because the
+driver populates the taxonomy on the execution path rather than only through the
+store API, `waitingRuns` and the waiting-row partial index match real
+suspensions.
+
+`WaitingRunsFilter` narrows a sweep to actionable rows by `reason`,
+`dueBeforeMs`, and a `cancelRequested` predicate over the run row, so a sweeper
+fetches only what it can act on instead of scanning every parked run.
+
+## Three sweeps, on the heartbeat cadence
+
+The driver re-drives three durable shapes periodically, and each one re-enters
+the ordinary claim, steal, and activate path:
+
+1. Parked runs whose cancellation was durably requested.
+2. Runs parked with reason `released`, left by a shutdown that interrupted them.
+3. Stale `running` rows left by a hard-killed owner.
+   `staleRunningRuns(staleBeforeMs, limit?)` lists runs whose heartbeat froze
+   before the horizon, with that heartbeat, oldest first, so a capped sweep drains a mass
+   owner death across ticks rather than waking every stale run in every driver
+   on every tick.
+
+Registration does its own recovery: `pendingClocks(scope)` re-arms outstanding
+timers and `completedDeferreds(flowName)` recovers unconsumed completions. Both exclude rows
+whose run has settled, so a restart arms timers for runs that can still make
+progress and for no others. Before serving a persisted deferred result, the
+engine marks it consumed. Later registration sweeps leave that completion out,
+so a run parked on a subsequent approval, timer, or deferred stays parked. The
+original result remains available for replay. Migration 0008 starts existing
+completions as unconsumed, allowing recovery after an upgrade.
+
+One corrupt row costs its own row and nothing else. Every list read
+(`dueClocks`, `pendingClocks`, `completedDeferreds`, `waitingRuns`,
+`runParents`, and `runChildren`) skips a row that will not decode and logs a
+storage-integrity warning naming its primary key, so a single unreadable row
+cannot stop a registration, a timer sweep, or a cancellation cascade. The point
+reads `deferred`, `clock`, and `waiting` still fail on such a row: they answer a
+question about one row, and reporting "no row" for a completion or deadline that
+is durably recorded but unreadable would re-run work whose side effects already
+ran.
+
+A wake for a flow the sweeping process has not registered is not dropped
+silently. The driver logs a once-per-run structured warning naming the run id
+and the flow, and leaves the durable waiting row parked, so any process that
+does register the flow still reclaims the run.
+
+## Cancellation is observed, not just recorded
+
+While a run executes, the driver polls `cancel_requested_at_ms` on the heartbeat
+cadence and cancels the run, interrupting the flow fiber, when another process
+called `RunStore.requestCancel`. A terminal transition additionally asserts
+`{ cancelRequested: "absent" }` inside the ownership compare-and-swap, so a
+request that races past the last poll turns finalize into a cancellation rather
+than a `completed` or `failed` write.
+
+`completeRunClocks(executionId, completedAtMs)` closes every uncompleted clock
+row of one run in a single statement, inside the terminal transaction. A run
+that settled has no future, so a timer firing against it could only schedule a
+resume nothing will drive.
+
+## Retrying a clock that failed to fire
+
+A durable clock whose fire fails transiently is redispatched with capped
+exponential backoff rather than being lost until process restart. The policy is
+`EngineStore.Options.clockFireRetryPolicy`, defaulting to exponential from 100ms
+capped at 30s, forever. It is the same option shape as the engine's
+`suspendedRetryPolicy`: the built-in behavior is the default, and a deployment
+that wants a different backoff supplies one rather than patching the store.
+
+## The wake bus makes the common case fast
+
+Polling is correct and slow. `WakeBus` is an in-process, edge-triggered bus:
+`wake(executionId)` resumes every waiter parked on that execution, `awaitWake`
+parks until the next one, and `waiters` reports the current count for tests and
+diagnostics.
+
+Nothing about it is durable, and that is the design. A wake with no waiters is
+dropped, and the polling fallback covers the run. Registration is removed when
+a waiting fiber is interrupted, including scope closure and losing a race
+against the polling sleep, so an abandoned wait leaks nothing. Cross-process
+delivery stays store-driven.
+
+An engine composition resolves the bus optionally. Provide `WakeBus.layer` to
+share one bus between the engine and your own wake sources; provide nothing and
+the composition builds a private one. `WakeBus.layerNoop()` drops every wake,
+which is what a test asserting the polling fallback provides.
+
+## The run DAG
+
+`DurableEngineState` also owns the parent edges between runs, because a run
+row's `state_json` carries only the first creating parent and a diamond gives a
+run a second one.
+
+`recordRunParent(childId, parentId)` is first writer wins per pair, and the
+cycle check runs inside the same write transaction as the insert: the edge is
+inserted, the child's ancestor chain is walked over the durable edges, and on a
+hit the transaction rolls back and the call fails with `RunParentCycleError`. A
+rejected edge therefore leaves no durable trace, and of two concurrent writers
+whose edges jointly close a cycle exactly one fails. The driver maps that error
+to `FlowCycleDetected`.
+
+`transaction(effect)` runs several store operations atomically. The driver wraps
+the parent-edge record and the run-row creation it guards in one, so a crash
+between them cannot leave a durable orphan edge for a run that was never
+created. Serialized write transactions are a documented requirement of the
+`DurableWriter.write` contract, not a SQLite artifact: a Postgres-backed
+implementation must use `SERIALIZABLE`.
+
+`runParents(childId)` and `runChildren(parentId)` list the edges oldest first.
+The child direction is the only instance-independent way to find the runs a
+cancelled parent linked to itself, so cancellation cascade reads it rather than
+an in-process map. `removeRunParentsForRun(runId)` deletes every edge naming a
+run as child or parent; an `AFTER DELETE` trigger on `flows_runs` prunes the
+same edges in the same transaction, so a lane that never calls the hook still
+cannot leave ghost edges.
+
+## Related
+
+- [Coordinate two processes](../guides/coordinate-two-processes.md): sharing a
+  wake bus and following a workspace's runs.
+- [Ownership and fencing](./ownership-and-fencing.md): why parking and clock
+  scheduling take an owner.

package/docs/concepts/execution-facts.md

@@ -0,0 +1,90 @@
+# Native lifecycle observations
+
+The native driver records `executionFact: { version: 1, baseline, observation }`
+in its authoritative `flows.engine.run-decision` and `flows.engine.interrupted`
+records. This is a producer contract implemented by `RunDriver`, separate from
+the generic `EngineEvent` schema vocabulary. Adding a schema alone does not
+migrate a writer.
+
+An observation contains the execution ID, flow name, lifecycle status, creation,
+start and finish timestamps, effective parent, trampoline lineage and ordinal,
+actual cancellation-request timestamp, and waiting reason/deadline/token digest.
+Ownership, claims, heartbeats and cancellation acknowledger identity remain
+operational store state; a heartbeat does not invalidate a semantic observation.
+The existing encoded `state` on run decisions remains unchanged.
+
+A wait's `tokenDigest` is SHA-256 of the exact opaque token, or null when absent.
+The protected native waiting row remains execution authority for resolving the
+wait. The event contains a comparison commitment, **not a replayable resolver
+credential**. The journal redactor is not bypassed. Other application strings
+that the redactor changes cannot establish exact replay equality and correctly
+remain observations rather than verified event state.
+
+## Commit boundary
+
+Creation, parent admission, activation, resume clearing, suspension, release,
+handoff and terminal settlement capture their fact in the transaction that
+changes the row. Parking and the guarded status CAS are one transaction. A lost
+CAS rolls the park back, preserving an earlier committed waiting row. Resume
+clears the waiting row before capturing the running fact and before executing
+work. Engine-state is the outer transaction and the journal write transaction
+is inside it, matching deferred completion's state-first lock order. The memory
+state adapter holds its own gate: taking the journal writer first can deadlock against a
+state-first fiber waiting for that writer. The memory state gate is released
+at SQL commit before post-commit journal work; publication failures after
+commit cannot undo either committed boundary.
+
+Cancellation records the actual `cancel_requested_at_ms` only when a guarded
+request changes a live row. Repeat requests, missing rows and terminal rows
+invent no new intent. Lineage cancellation reads membership, updates live rounds
+and records each changed round in one native transaction. Descendant and inherited
+requests use the same producer. A completed predecessor does not acquire a fake
+request timestamp just because its successor is cancelled.
+
+The Node and Bun host capture the native journal, run store and engine state
+before `AgentSession` selects the separate control journal. Its direct
+cancellation callback executes in the captured host context, outside any caller's
+control SQL transaction. The native commit precedes the control receipt; the two
+databases are not one distributed transaction. A later control write failure
+cannot undo already accepted native intent, which remains visible in the native
+observation and copied facts. Low-level `RunStore` calls and standalone legacy
+`AgentSession.requestCancel` do not promise this event contract.
+
+## Replay and migration
+
+`ExecutionFact.fold` in `@smthrs/journal` is browser safe and pure. New creation
+facts establish a `created` baseline. A first fact for an older row, an unknown
+producer version, an explicit omission, or a generation boundary cannot invent
+the missing prefix. A later complete fact establishes a `legacy` baseline.
+If a low-level/corrupt row cannot encode a valid observation, the writer records
+`{ version: 1, unavailable: "invalid-observation" }`. Existing invalid-round or
+corruption handling can still settle the execution; this marker never creates
+verified coverage and does not normalize malformed ancestry into invented facts.
+The exposed sequence/generation range names the current round's native stream;
+root and current are both compared, and either legacy baseline makes the logical
+view's baseline legacy. Numeric sequence holes alone are not omissions: rolled-back SQL journal
+reservations can leave them without losing a committed event.
+
+The native supervisor authenticates a wrapper through its native flow, approved
+plan and root ancestry, then the bridge writes a separate `control.engine.bound`
+fact. It never binds a control run by trusting an input `runId` field. Existing
+`control.engine.event` payloads and producer identities are unchanged. The bridge
+walks both durable spawn edges and native trampoline membership; a handoff is not
+a spawn edge. A legacy adapter without the lineage port cannot establish coverage
+for a missing latest round.
+
+Gateway run reads and subscriptions use the same control fold and native fold.
+`lifecycleProvenance` continues to describe control replay and whether lifecycle
+was engine observed. The additive `executionProvenance` separately identifies
+native `events`, `legacy-observation` or `unverified-observation`. Native facts are
+compared against a coherent observation of both the requested root and latest
+round. The root retains ancestry and round identity; the current round supplies
+status and waiting. A missing binding, gap, unsupported version or mismatch keeps
+an honest observed fallback. Bridge lag is possible because this is a read-side
+copy between databases, not a distributed snapshot.
+
+This stream verifies the semantic run/approval view. It does not rebuild every
+execution table: action outcomes, clocks, deferred values and resolver credentials
+retain their existing native authorities. [Native call facts](./call-facts.md)
+now use the owning action transaction and durable journal as their outbox.
+Other agent trace records still use a best-effort channel.

package/docs/concepts/ownership-and-fencing.md

@@ -0,0 +1,128 @@
+---
+title: "Ownership and fencing"
+description: "How one engine claims a run, how every durable write is fenced on an OwnerId, and how a run is reclaimed from an owner that stopped heartbeating."
+sidebar:
+  order: 1
+---
+
+Two processes can open the same database. Nothing in SQLite stops the second
+one from driving a run the first one is already driving, and nothing in the
+first one notices. Ownership is the answer: exactly one incarnation owns a run
+at a time, every durable write it makes is conditioned on still being that
+owner, and a run whose owner went away can be taken over on evidence rather
+than on a guess.
+
+## The OwnerId
+
+An `OwnerId` is a triple: the `hostId` the composition declares, a `pid`, and a
+fresh `nonce`. `OwnerIdentity` mints it, once per incarnation:
+
+```ts
+import { OwnerIdentity } from "@smthrs/engine-store"
+
+const identity = OwnerIdentity.layer
+```
+
+The default reads a process id off `globalThis` where the platform has one, and
+draws an incarnation number from `Random` where it does not, so a browser tab
+can mint a token even though it cannot execute durable flows. The nonce is a
+UUIDv4 from the injected `Crypto` service. `layerConstant(owner)` pins the whole
+token, which is what a test wants, and what a host that already holds a lease
+from somewhere else wants.
+
+The nonce is what makes the token an incarnation rather than a machine. A
+process that restarts on the same host with the same pid mints a different
+token, so its writes are correctly refused against runs the previous
+incarnation still owns.
+
+## What fencing means in practice
+
+The engine claims a run before driving it, and from then on every durable write
+carries the owner:
+
+- Run state transitions go through the run store, which asks the journal's
+  consensus strategy whether the owner still holds the run inside the same
+  write transaction. A stale owner gets `FenceLost`.
+- Attempt lifecycle writes pass the owner to the journal, which fences the
+  append. A reclaimed owner fails with `fence_lost` and self-interrupts rather
+  than appending.
+- `DurableEngineState.park` and `scheduleClock` take the owner, so a process
+  that no longer runs a run cannot park it or arm a timer against it.
+- Terminal transitions additionally assert `{ cancelRequested: "absent" }`
+  inside the same compare-and-swap, so a cancellation that arrives after the
+  last poll turns finalize into a cancellation rather than a `completed` write.
+
+The consequence is that a zombie process is harmless. It can hold a live fiber
+and a live database handle, and every write it attempts against a reclaimed run
+is refused by the row it is writing to.
+
+## The lease, and the evidence to break it
+
+An owner heartbeats while it drives. `isAlive` on `EngineStore.Options` decides
+whether a run whose lease has expired may be stolen; answering `true` refuses
+the takeover.
+
+The default is `Ownership.leaseLiveness(Ownership.heartbeatStaleAfter)`: the
+owner counts as alive while its persisted `heartbeat_at_ms` is younger than the
+staleness cutoff, and gone once it is not. That is the weakest honest answer,
+and the only evidence every host has. It is also enough: a process killed with
+SIGKILL stops heartbeating, so a fresh process reclaims its runs with no
+application code at all.
+
+A deployment that knows more supplies its own check and refuses the takeover
+for longer. A `process.kill(pid, 0)` probe is one such check;
+`Ownership.sameHostPidProbe` is the shipped version. Guard any pid read with
+`Ownership.sameHostIncarnation`, because a pid names a process only inside its
+own host's namespace. A supplied check receives the recorded owner and a
+`LivenessContext` of `{ claimant, heartbeatAtMs, nowMs }`.
+
+## The journal says which answer was used
+
+Every arbitration is recorded, so a takeover is auditable after the fact:
+
+| Record                                                                 | Meaning                                                        |
+| ---------------------------------------------------------------------- | -------------------------------------------------------------- |
+| `steal-refused-owner-alive` with `evidence: "lease-fresh"`             | The lease was still inside the staleness window.               |
+| `steal-refused-owner-alive` with `evidence: "probe"`                   | The supplied `isAlive` check answered for the owner.           |
+| `stolen-and-activated` with `evidence: "lease-expired"`                | The lease had expired and no check refused it.                 |
+| `stolen-and-activated` with `evidence: "same-host-pid-dead"`           | A same-host probe found the process gone.                      |
+| `stolen-and-activated` with `evidence: "cross-host-unreachable-stale"` | The owner was on another host and unreachable past the window. |
+
+A refusal is journaled once, not once per sweep tick. The record is addressed
+by the run, the refused owner, the lease it was refused against, and the
+evidence that refused it, and it carries source sequence 0, so the journal's
+first-writer admission collapses every repeat about an unchanged lease into one
+record. The evidence is part of the address because one lease can be refused
+twice for different reasons: a wake arriving while the owner is still pulsing
+is refused by the lease, and the same owner, alive but stalled past the window,
+is refused by the probe. Both records stand.
+
+## Backoff, so arbitration keeps moving
+
+The driver stops re-probing a run it was just refused. Each refusal defers that
+run for two heartbeat ticks, doubling per consecutive refusal against the same
+lease up to `heartbeatStaleAfter`. The stale-running sweep reads past exactly
+the rows it is going to skip, so its batch of 64 keeps advancing. Without that,
+the oldest stale rows sort first every second and a run behind them is never
+arbitrated at all.
+
+A deferral is forgotten as soon as its row leaves the stale window: the owner
+resumed heartbeating, the run was stolen, or the run settled. A fresh stall
+under a new lease is therefore probed on the first tick that sees it, rather
+than waiting out a backoff it did not earn.
+
+## Restoring a backup resets the fence
+
+A restored database still carries the owner tokens that were live when the
+backup was taken. `DisasterRecovery.fence` clears every pending claim and
+suspends every run that was `running`, in one serialized write transaction, so
+a surviving pre-backup owner fails its compare-and-swap immediately and the
+restored runs are claimable without waiting out the staleness cutoff. See
+[Back up and restore the store](../guides/back-up-and-restore.md).
+
+## Related
+
+- [Reclaim runs from a dead host](../guides/reclaim-runs-from-a-dead-host.md):
+  the task-shaped version of this page.
+- [Attempts and replay](./attempts-and-replay.md): what a reclaiming process
+  does with the attempt rows the previous owner left behind.

package/docs/concepts/step-boundaries.md

@@ -0,0 +1,148 @@
+---
+title: "Step boundaries"
+description: "The declared read and write sets of a step, what prepare, settle, and replayOutputs each measure, and why only whole-tree evidence may enter the shared cache."
+sidebar:
+  order: 3
+---
+
+A step declares which files it reads and which it writes. `StepBoundary` is the
+service that measures that declaration against reality: before the step runs,
+after it runs, and again when a later replay has to reproduce its outputs on a
+workspace that never executed anything.
+
+The declaration itself is `FileBoundary`, from
+[`@smthrs/flow`](/api/flow)'s `Action` namespace: a `readSet`, a `writeSet`,
+optional `removes`, and a `boundaryMode` of `hard` or `expected`.
+
+## The three calls
+
+```ts
+interface Service {
+  readonly prepare: (descriptor: FileBoundary) => Effect<PreparedBoundary, UnsupportedBoundary, Crypto>
+  readonly settle: (prepared: PreparedBoundary) => Effect<BoundaryEvidence, SettleError, Crypto>
+  readonly replayOutputs: (evidence: BoundaryEvidence) => Effect<void, ReplayError, Crypto>
+}
+```
+
+`prepare` measures the declared read set's real digests and returns a
+`PreparedBoundary` carrying the descriptor plus a `readSnapshot`. The declared
+digests in the descriptor are caller metadata folded into the step key; the
+snapshot is the evidence that they still describe reality.
+
+`settle` measures the writes afterwards and turns them into
+`BoundaryEvidence`: the declared outputs (as a replayable payload), a
+`diffIdentity` naming the post-state, and two optional honest flags,
+`wholeTreeWritesVerified` and `hermeticReadsVerified`.
+
+`replayOutputs` reproduces those outputs from the evidence.
+
+## Hard mode refuses, expected mode records
+
+The same observation produces a refusal in `hard` mode and a `BoundaryDeviation`
+in `expected` mode:
+
+| Observation                                                            | Hard mode                  | Expected mode                        |
+| ---------------------------------------------------------------------- | -------------------------- | ------------------------------------ |
+| Wrote outside the declared write set                                   | `UndeclaredWrite`          | `ExpectedSetDeviation`               |
+| Did not produce a declared output, and did not declare it as a removal | `MissingDeclaredOutput`    | `MissingDeclaredOutput` deviation    |
+| Left a declared removal in place                                       | `SurvivingDeclaredRemoval` | `SurvivingDeclaredRemoval` deviation |
+
+The two absence cases exist for a specific reason. Recording `digest: null` as
+valid evidence caches the claim "this file should not exist", which
+`replayOutputs` then acts on by deleting the path on a workspace that never ran
+the step. A step that crashed after declaring its writes would poison the cache
+with an eraser. Declaring a path in `removes` is how a deliberate deletion says
+so, and it is the only thing that makes an absence legitimate.
+
+A deviation of any variant bars the evidence from the shared cache, so an
+unexplained absence never reaches another host either way. Deviations go to
+[`Reconciliation`](../guides/drive-a-plan.md), which answers with a verdict.
+
+## Bounded host measurement
+
+`prepare` and `settle` use the kernel's optional guarded batch capability when
+the host provides it. Enumeration batches directory and stat requests while
+retaining the plan's glob matcher, dotfile behavior, pruning, sorted output,
+and entry budget. Hosts without batching use the existing operations with at
+most four in flight.
+
+Read digests are SHA-256 of the original file bytes. Tree-artifact and diff
+identities still use the same canonical key inputs and absence sentinels, so
+batching introduces no persisted key or evidence migration. The declaration
+snapshot stays immutable, and all measurements use the same guarded filesystem
+whose root descriptor identity is pinned at composition.
+
+Settlement first obtains digest/size measurements, then groups content reads
+by both path count and the host's response-byte budget. A change between the
+two measurements refuses settlement. Inline decisions and artifact spills
+remain in sorted path order, preserving the aggregate inline budget and the
+resulting manifest. Helper errors retain their original cause through
+`UnsupportedBoundary`. Losing the workspace root fails the boundary instead
+of turning every file into an absence.
+
+Batching does not freeze the workspace, attest whole-tree writes, or change
+cache admission. Each helper measures through checked descriptors; later
+verification still has to consume fresh measurements through that boundary.
+
+## Inline, or spilled by digest
+
+Outputs are recorded by content digest, never inlined without bound. Two limits
+on `FileSystemOptions` decide:
+
+- `maxInlineBytes`, the largest single output carried inline in the evidence.
+  Defaults to 1 MiB.
+- `maxTotalInlineBytes`, the largest aggregate inline payload one settle may
+  fold into its evidence. Defaults to 8 MiB. A per-output bound alone still let
+  a wide write set multiply many individually small payloads into an unbounded
+  evidence row.
+
+Anything past either bound is handed to the `ArtifactStore` and recorded by
+digest reference only. `referencedDigests(evidence)` names exactly those
+digests: the set a shared tier must hold before the evidence's cache entry
+becomes observable there, and the set a replay on a fresh host has to fetch.
+
+The blob mechanics, content addressing, atomic publication, digest
+verification, and dedupe, belong to [`@smthrs/artifacts`](/api/artifacts). What
+stays here is the policy that decides which outputs become blobs at all.
+
+## Why the production boundary cannot claim the whole tree by itself
+
+`StepBoundary.layer` is the filesystem-backed boundary over the kernel
+`FileSystem` seam and the artifact store. It measures declared paths. It cannot
+see a write elsewhere in the tree, so it never claims
+`wholeTreeWritesVerified` on its own, and a composition with a boundary but no
+sandbox keeps the honest outcome: run-local results only.
+
+The whole-tree proof comes from running the body somewhere else. That is
+[`WorkspaceSandbox`](./workspace-transactions.md): the transaction is the tree,
+so a write outside the declared set is a map comparison rather than an
+inference, and the engine sets the flag structurally.
+
+`StepBoundary.layerTest(options)` is deterministic and supports
+explicit `failure` or `deviation` fixtures, replay, and `readSnapshot` assertions, but it does not
+enforce a real sandbox. Its `wholeTreeWriteDetection` default of `true` is a
+fixture, not a proof.
+
+## Replay refusals a caller can act on
+
+`replayOutputs` distinguishes three failures on purpose:
+
+- `MissingArtifact`: the bytes are simply not on this host. A shared artifact
+  tier can repair it, so `ArtifactSync.hydrate` fetches and the dispatch
+  retries the replay exactly once before falling through to a real execution.
+- `BoundaryCorruption`: the bytes are here and no longer hash to their recorded
+  digest. That is an integrity violation, not a transient failure.
+- `UnsupportedBoundary`: the host could not honour the boundary at all. It is
+  the catch-all, and it carries the refusing host or store failure whole in
+  `cause` rather than flattened into the message.
+
+## Related
+
+- [Workspace transactions](./workspace-transactions.md): where the whole-tree
+  proof actually comes from.
+- [Cache admission](./cache-admission.md): the full list of conditions a record
+  must meet before another run may use it.
+- [Share a cache across machines](../guides/share-a-cache-across-machines.md):
+  publishing the digests this evidence references.
+
+Test filesystem classification with `StepBoundary.layer` over an in-memory `FileSystem` and `ArtifactStore`. `layerTest` returns supplied settlement evidence or failure verbatim; it does not classify changed paths, missing outputs, or surviving removals.