@smthrs/run-store 0.0.0-stage → 1.0.0-rc.3

package/docs/guides/cancel-a-run.md

@@ -0,0 +1,156 @@
+---
+title: "Cancel a run"
+description: "Record cancellation intent from any observer, read it from the owner, and settle the run under a transition guard so a completion and a cancellation can never both win."
+sidebar:
+  order: 4
+---
+
+Cancellation is two halves that never share a fence. Any observer records the
+intent with `requestCancel`. Only the owner acts on it, with a guarded
+`transitionOwned`. Nothing interrupts the owner for you: the request is durable
+intent, and the owner decides where its flow can stop cleanly.
+
+## Record the request
+
+For a logical run that can hand off, use `requestCancelLineage`: a completed
+round is not necessarily a completed job. It resolves every round and records
+intent atomically, including a successor that exists when the transaction wins.
+The engine's handoff transaction refuses to admit a successor if cancellation
+won first. `requestCancel` remains the lower-level, single-round operation used
+by owners coordinating lifecycle transitions and child cleanup.
+
+```ts
+import { RunStore } from "@smthrs/run-store"
+import * as Clock from "effect/Clock"
+import * as Effect from "effect/Effect"
+
+const ask = (runId: string) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    const nowMs = yield* Clock.currentTimeMillis
+    return yield* runs.requestCancelLineage(runId, nowMs)
+  })
+```
+
+Neither request operation takes an owner, on purpose. An operator, a parent run, or a
+control plane asks for a cancellation without holding the run, and the write is
+first-writer-wins, so asking twice is harmless.
+
+| Outcome            | What it means                                                                                 |
+| ------------------ | --------------------------------------------------------------------------------------------- |
+| `CancelRequested`  | This call wrote `cancel_requested_at_ms`, and `requestedAtMs` is the value it wrote.          |
+| `AlreadyRequested` | A request was already recorded. `requestedAtMs` is the original time, not the one you passed. |
+| `Terminal`         | The run had already settled, so nothing was recorded. `status` says which ending it lost to.  |
+| `NotFound`         | No such run row.                                                                              |
+
+For each round, the status predicate is part of the same statement that writes the column, so a
+run that settles between your read and your request loses the write instead of
+racing it. A settled run records nothing because it has no owner left to observe
+the intent, and a reader that took the column as live intent would go on
+cancelling children the finished parent had already dealt with.
+
+For `requestCancelLineage`, the outcome summarizes the transaction: any newly
+recorded request yields `CancelRequested`; otherwise an existing request yields
+`AlreadyRequested`. `Terminal` means every round is settled and reports the
+latest round's status. Terminal predecessor rows are never rewritten.
+
+`nowMs` is taken literally and is not bounded by `heartbeatSkewAllowance`: this
+timestamp is request data rather than a lease predicate, so the only rule is
+that it must be a non-negative safe integer. See
+[The heartbeat lease](../concepts/leases.md).
+
+## Read the request as the owner
+
+The request is a column on the run row, so the owner reads it with `get`:
+
+```ts
+const cancelRequested = (runId: string) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    const row = yield* runs.get(runId)
+    return row.cancelRequestedAtMs !== null
+  })
+```
+
+Poll it at a point where the flow can stop without leaving work half done.
+[`examples/src/19-cancel-and-child-cleanup.ts`](https://github.com/smithersai/smithers/blob/main/examples/src/19-cancel-and-child-cleanup.ts)
+polls exactly this column and cleans up its children before it settles.
+
+## Settle under a guard
+
+A `TransitionGuard` is an extra predicate compiled into the same `UPDATE` as the
+ownership fence, so a request that arrives while you are deciding cannot slip
+between a check and a terminal write.
+
+```ts
+import type { OwnerId } from "@smthrs/run-store/Ownership"
+
+const settleCancelled = (runId: string, owner: OwnerId, state: unknown) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    return yield* runs.transitionOwned(runId, owner, "cancelled", JSON.stringify(state), {
+      cancelRequested: "present"
+    })
+  })
+```
+
+`{ cancelRequested: "present" }` admits the write only while a request is on the
+row, so a run is never recorded as cancelled without one. Put the mirror image
+on the completion:
+
+```ts
+const settleCompleted = (runId: string, owner: OwnerId, state: unknown) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    return yield* runs.transitionOwned(runId, owner, "completed", JSON.stringify(state), {
+      cancelRequested: "absent"
+    })
+  })
+```
+
+Together the two guards make the pair exclusive in SQL rather than in your
+control flow.
+
+`GuardFailed` means you still own the run and the predicate refused the write.
+That is a different situation from `FenceLost`, which means another owner holds
+the run: the store checks ownership first and only reports `GuardFailed` for a
+row you genuinely own.
+
+## Let the guard tell you
+
+A guard on the mid-flight progress write turns the next checkpoint into the
+notification:
+
+```ts
+const recordProgress = (runId: string, owner: OwnerId, state: unknown) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    return yield* runs.transitionOwned(runId, owner, "running", JSON.stringify(state), {
+      cancelRequested: "absent"
+    })
+  })
+```
+
+While no request exists the write reports `Transitioned`. The moment one lands,
+the same call reports `GuardFailed` and the owner learns about the request
+without a separate read.
+
+## What a cancellation does not do
+
+- It does not interrupt the owner. A request is a row, not a signal, and an
+  owner that never reads it runs to completion.
+- It does not clear the column. `transitionOwned` clears the owner, the
+  heartbeat, and the claim, and never touches `cancel_requested_at_ms`, so a
+  settled run keeps the record of what was asked.
+- It does not decide the ending. A run whose owner honored a request settles as
+  `cancelled`; one that finished first settles as `completed`, and the next
+  request reports `Terminal`.
+
+## Next steps
+
+- [Claim a run and finish it](./claim-and-finish-a-run.md): the transition this
+  guard rides on.
+- [Fencing and ownership](../concepts/fencing.md): why every refused write is a
+  named outcome.
+- [Troubleshooting](../troubleshooting.md): what to change when a guard or a
+  request keeps refusing.

package/docs/guides/claim-and-finish-a-run.md

@@ -0,0 +1,175 @@
+---
+title: "Claim a run and finish it"
+description: "Take ownership of a run, keep its lease alive while you work, record progress, and settle it: the two-phase claim, the atomic form, heartbeat supervision, and the fenced transition."
+sidebar:
+  order: 2
+---
+
+This is the path a durable executor walks on every run it picks up. Each step
+is a compare-and-swap you can lose, and losing is normal.
+
+## Read the row and choose the snapshot
+
+Every claim guards three fields. Read the row; you can restate those three as
+the `expected` snapshot:
+
+```ts
+import type { RunRow, RunSnapshot } from "@smthrs/run-store/RunStore"
+
+const snapshot = (row: RunRow): RunSnapshot => ({
+  status: row.status,
+  owner: row.owner,
+  heartbeatAtMs: row.heartbeatAtMs
+})
+```
+
+The three-field helper is optional: a complete `RunRow` returned by `get` can
+also be passed directly as `expected`. The store copies only the three guarded
+fields; extra own data fields are ignored. Missing or inherited required fields,
+enumerable accessors on any key, and non-plain prototypes are refused.
+
+## Claim, then activate
+
+`claim` reserves a `pending` or `suspended` row and hands back the
+`claimedAtMs` that is your fence token for the second step. `activate` trades
+that token for ownership:
+
+```ts
+import { RunStore } from "@smthrs/run-store"
+import type { OwnerId } from "@smthrs/run-store/Ownership"
+import * as Clock from "effect/Clock"
+import * as Effect from "effect/Effect"
+
+const take = (runId: string, owner: OwnerId) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    const row = yield* runs.get(runId)
+    const expected = snapshot(row)
+    const nowMs = yield* Clock.currentTimeMillis
+
+    const claim = yield* runs.claim(runId, expected, owner, nowMs)
+    if (claim._tag !== "Claimed") return claim
+
+    const activation = yield* runs.activate(runId, owner, claim.claimedAtMs, expected)
+    if (activation._tag !== "Activated") {
+      yield* runs.abandonClaim(runId, owner, claim.claimedAtMs)
+    }
+    return activation
+  })
+```
+
+Take `nowMs` from `Clock.currentTimeMillis`, not from `Date.now()`. It is the
+value the store persists as `claimed_at_ms`, and a reading more than
+`heartbeatSkewAllowance` ahead of the store's own clock fails with
+`invalid_run`.
+
+Release the claim when activation loses. The gap between the two calls exists
+so a caller can do work that must not be repeated, such as recording in the
+journal who is taking the run over. If that work or the activation fails, a
+claim nobody is using would block the next sweeper until it went stale.
+
+## Or take it in one call
+
+When there is nothing to do between reserving and owning, `claimAndOwn` is one
+compare-and-swap:
+
+```ts
+const takeAtomically = (runId: string, owner: OwnerId) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    const row = yield* runs.get(runId)
+    const nowMs = yield* Clock.currentTimeMillis
+    return yield* runs.claimAndOwn(runId, snapshot(row), owner, nowMs)
+  })
+```
+
+It admits `pending`, `suspended`, and `running` rows. Re-entering a run you
+already own needs nothing extra. Replacing a _different_ running owner needs
+matching liveness evidence as the fifth argument, and reports
+`EvidenceRequired` when the snapshot is still current, the other owner's
+heartbeat is stale, and no evidence was supplied. Re-reading and retrying
+cannot make progress from there; see
+[Take over a stalled run](./recover-a-stalled-run.md).
+
+## Keep the lease alive
+
+An owner that stops writing `heartbeat_at_ms` is stealable after
+`heartbeatStaleAfter`. `Ownership.heartbeatLoop` pulses until the fence is lost
+and then interrupts itself, so racing it against the work makes losing
+ownership interrupt the work:
+
+```ts
+import { Ownership } from "@smthrs/run-store"
+
+const runOwned = <A, E>(
+  runId: string,
+  owner: OwnerId,
+  work: Effect.Effect<A, E, RunStore.RunStore>
+) => Effect.raceFirst(work, Ownership.heartbeatLoop(runId, owner))
+```
+
+Use `Effect.raceFirst` rather than `Effect.race`: the loop signals a lost fence
+by interrupting itself, and `raceFirst` settles on the first outcome of either
+side, interruption included.
+
+## Record progress without giving up ownership
+
+`transitionOwned` with the target `running` keeps the owner and rewrites only
+the executable state. It is how a mid-flight run persists what a resume would
+re-enter:
+
+```ts
+const checkpointRun = (runId: string, owner: OwnerId, state: unknown) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    return yield* runs.transitionOwned(runId, owner, "running", JSON.stringify(state))
+  })
+```
+
+Omit the state argument to change status alone. Passing it rewrites
+`state_json`; the column is never partially merged.
+
+## Settle it
+
+A terminal transition clears the owner, the heartbeat, and any claim, and
+stamps `finished_at_ms`:
+
+```ts
+const finish = (runId: string, owner: OwnerId, state: unknown) =>
+  Effect.gen(function*() {
+    const runs = yield* RunStore.RunStore
+    const outcome = yield* runs.transitionOwned(
+      runId,
+      owner,
+      "completed",
+      JSON.stringify(state),
+      { cancelRequested: "absent" }
+    )
+    return outcome
+  })
+```
+
+The guard is optional and is compiled into the same `UPDATE`, so a cancellation
+request that arrives while you are deciding cannot slip between a check and a
+terminal write. `GuardFailed` means you still own the run and the guard refused
+it, which is a different situation from `FenceLost`.
+
+`suspended` is the other useful target: it clears ownership without stamping
+`finished_at_ms`, parking the run so any host may claim it again.
+
+## The outcomes you have to handle
+
+| Outcome            | Which call                      | What it means                                                          |
+| ------------------ | ------------------------------- | ---------------------------------------------------------------------- |
+| `NotFound`         | any                             | No such run row.                                                       |
+| `AlreadyClaimed`   | `claim`, `claimAndOwn`, `steal` | Another process holds the claim columns.                               |
+| `HeartbeatFresh`   | `claim`, `claimAndOwn`, `steal` | The row is `running` under a live lease.                               |
+| `SnapshotChanged`  | claim family, `activate`        | The row moved between your read and your write.                        |
+| `EvidenceRequired` | `claimAndOwn`                   | The snapshot is current and its stale owner needs evidence to replace. |
+| `ClaimLost`        | `activate`, `abandonClaim`      | The claim columns no longer match your token.                          |
+| `FenceLost`        | `heartbeat`, `transitionOwned`  | Another owner holds the run. Stop working on it.                       |
+| `GuardFailed`      | `transitionOwned`               | You own the run; the guard predicate refused the write.                |
+
+Every one of these is a success value. `RunStoreError` is reserved for invalid
+input, corrupt rows, and database failures, so a caller can retry contention
+without retrying corruption. See [Troubleshooting](../troubleshooting.md).

package/docs/guides/compose-the-stores.md

@@ -0,0 +1,142 @@
+---
+title: "Compose the stores into a host"
+description: "Wire RunStore and AttemptStore over a SQL driver: the layer order that satisfies migrations, the attempt-store policy options, the stubs, and where the composed storage ladder already exists."
+sidebar:
+  order: 1
+---
+
+Both stores require a `SqlClient` and a `DurableWriter`, and both require their
+tables to exist. This guide builds that composition from the bottom up.
+
+## Check whether you need to build it
+
+If you are running the Smithers engine, the composition already exists and
+building a second one over the same database is a way to get two engines
+fighting over one set of rows. [`@smthrs/engine-store`](/api/engine-store)
+composes these stores with the journal, the step cache, and the durable engine
+state, and `@smthrs/flows/NodeRuntime` builds that ladder over one SQLite file.
+Reach for the layers here when you are writing a host, an adapter, or a sweeper
+that has to agree with the engine about ownership.
+
+## The layer order
+
+Migrations must finish before either store is exposed, and both stores need the
+writer and the client:
+
+```ts
+import * as DurableWriter from "@smthrs/database/DurableWriter"
+import * as NodeDatabase from "@smthrs/database/node/NodeDatabase"
+import { AttemptStore, Migrations, RunStore } from "@smthrs/run-store"
+import * as Layer from "effect/Layer"
+
+export const storage = (filename: string) =>
+  Layer.mergeAll(RunStore.layer, AttemptStore.layer).pipe(
+    Layer.provideMerge(Migrations.layer),
+    Layer.provideMerge(DurableWriter.layer()),
+    Layer.provideMerge(NodeDatabase.layer({ filename }))
+  )
+```
+
+`Layer.provideMerge` is what keeps the database and the writer in the output
+context, so the rest of the host can reach them too.
+
+The SQLite and PostgreSQL adapters share this package's migrations and
+statements. The composition must satisfy the `DurableWriter` serialization
+contract: concurrent write transactions cannot both commit from snapshots
+that exclude each other's writes. Use `NodeDatabase`, `BunDatabase`, or
+`PostgresDatabase` from `@smthrs/database`; a bare PostgreSQL READ COMMITTED
+client does not provide the required serialization.
+
+Store writes nested inside `DurableWriter.write` join it as savepoints. A
+transient database conflict retries the entire outer transaction. Persistence
+errors retain a payload-free `DatabaseError` in their cause chain so redaction
+does not remove that retry classification. SQL text and bound values are not
+included.
+
+## Compose the migration set, do not run it twice
+
+`Migrations.layer` installs this package's tables and nothing else. A host that
+also runs the journal, the step cache, or the engine state should compose the
+migration sets instead, so one migrator applies them in one pass:
+
+```ts
+import * as DatabaseMigrations from "@smthrs/database/Migrations"
+import { Migrations } from "@smthrs/run-store"
+
+const runAll = DatabaseMigrations.run([Migrations.set /* , the other packages' sets */])
+```
+
+`Migrations.set` reserves id block 1000, so its ids can never collide with
+another package's. `@smthrs/engine-store` already composes the full set list.
+
+## Choose the attempt-store policy
+
+`AttemptStore.layer` takes the defaults: only `running` counts as in progress,
+checkpoints cap at 1 MiB, and `put` is first-writer-wins. `layerWith` takes a
+policy instead:
+
+```ts
+import { AttemptStore } from "@smthrs/run-store"
+
+const attempts = AttemptStore.layerWith({
+  inProgressStates: ["running", "retrying"],
+  maxCheckpointBytes: 4 * 1024 * 1024,
+  putMode: "upsert"
+})
+```
+
+- `inProgressStates` names the states that mean "this attempt is still moving".
+  `heartbeat` and `finish` fence on membership, and `finish` refuses them as
+  targets. The names must be unique, non-empty durable text.
+- `maxCheckpointBytes` must be between 1 and `AttemptStore.maximumCheckpointBytes`.
+- `putMode: "upsert"` lets a re-`put` overwrite an attempt that is still in
+  progress and report `Upserted`. A terminal attempt stays immutable in both
+  modes.
+
+All three are validated, detached, and frozen when the store is built, so an
+invalid policy fails at composition time with `AttemptStoreError`.
+
+## Stubs for compositions that must not persist
+
+Both services ship an explicit absence, so a composition that cannot reach a
+database still type-checks and still says so out loud:
+
+| Layer                      | Behavior                                                                                                                                                                |
+| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `RunStore.layerNoop()`     | `create` and `get` fail with `persistence_failed`. Every compare-and-swap reports its "you lost" outcome: `NotFound`, or `ClaimLost` for `activate` and `abandonClaim`. |
+| `AttemptStore.layerNoop()` | Every operation fails with the `unknown` code.                                                                                                                          |
+
+Both take partial overrides, which is how a test replaces one operation without
+building a whole store:
+
+```ts
+import { RunStore } from "@smthrs/run-store"
+import * as Effect from "effect/Effect"
+
+const alwaysFenced = RunStore.layerNoop({
+  heartbeat: () => Effect.succeed({ _tag: "FenceLost" })
+})
+```
+
+## Verify the composition
+
+Read a run back through the layer you built. If migrations did not run, `get`
+fails with `persistence_failed` rather than returning nothing:
+
+```ts
+import { RunStore } from "@smthrs/run-store"
+import * as Effect from "effect/Effect"
+
+const check = Effect.gen(function*() {
+  const runs = yield* RunStore.RunStore
+  yield* runs.create("composition-check", "{}")
+  return yield* runs.get("composition-check")
+})
+```
+
+## Next steps
+
+- [Claim a run and finish it](./claim-and-finish-a-run.md): what to do with the
+  composed services.
+- [Test against the real stores](./testing.md): the in-memory layer that skips
+  all of this.

package/docs/guides/observe-outcomes.md

@@ -0,0 +1,182 @@
+---
+title: "Observe store outcomes"
+description: "Read the fencing counters and spans the stores emit: the three counters and their attributes, what a span carries, how to wire an exporter, and how to assert on a counter in a test."
+sidebar:
+  order: 6
+---
+
+A lost fence is a normal success value, so it never reaches an error channel and
+never appears in a log by default. The stores publish it as telemetry instead:
+every claim, heartbeat, and transition updates an attributed counter and closes a
+span that says how it ended. Wire an exporter once and "who keeps stealing this
+run" becomes a query rather than a grep.
+
+## The three counters
+
+`RunStoreMetrics` defines the handles, and `RunStore` updates them as it decides
+each outcome. Every driver that writes through the store lands in the same
+counters.
+
+| Counter                 | Attributes      | What it counts                                                     |
+| ----------------------- | --------------- | ------------------------------------------------------------------ |
+| `flows_run_claims`      | `op`, `outcome` | Every ownership compare-and-swap, across all six claim operations. |
+| `flows_run_heartbeats`  | `outcome`       | Every fenced ownership heartbeat.                                  |
+| `flows_run_transitions` | `outcome`, `to` | Every fenced owned transition, by requested target status.         |
+
+`op` is the operation in snake case: `claim`, `claim_and_own`, `activate`,
+`abandon_claim`, `recover_claim`, or `steal`. `outcome` is the result tag in
+snake case, so `HeartbeatFresh` is recorded as `heartbeat_fresh`. `to` on a
+transition is the target status you asked for, whether or not the write won.
+A call that fails or is interrupted still counts, under `outcome="failure"` or
+`outcome="interrupt"`, so a database refusing writes shows up as a failure rate
+rather than a silent drop.
+
+Four of those series are the ones worth alerting on:
+
+- `flows_run_heartbeats{outcome="fence_lost"}` is the fencing event: an owner
+  discovered another process holds its run.
+- `flows_run_claims{op="activate",outcome="claim_lost"}` means a held claim was
+  recovered or replaced before its activation ran.
+- `flows_run_heartbeats{outcome="failure"}` is a lease write the database
+  refused. `heartbeatLoop` also logs `run heartbeat write failed` on the first
+  failure of an outage, and `run lease lapsed; interrupting owned work` when
+  the write tolerance runs out.
+- `flows_run_claims{op="steal",outcome="claimed"}` is a takeover. A steady rate
+  of these is a host that keeps dying, or two hosts with clocks far enough apart
+  to take runs from each other.
+
+A terminal transition also advances `runThroughput` from
+[`@smthrs/observability`](/api/observability), published as
+`flows_run_throughput`, so finished runs are counted in the same place as the
+rest of the runtime. See
+[Read the runtime metrics](/pkg/observability/guides/read-runtime-metrics).
+
+Throughput advances only after the outermost managed `DurableWriter.write`
+commits. Rolled-back savepoints and retried transactions do not count; the
+final committed transition counts once. Use the shared writer for this metric:
+raw SQL transactions do not provide a managed commit notification.
+
+## Wire an exporter
+
+This package ships no exporter. It defines counters and updates them, so they
+appear in whatever registry the composition provides:
+
+```ts
+import * as Layer from "effect/Layer"
+import * as Metric from "effect/Metric"
+
+const registry = Layer.succeed(Metric.MetricRegistry)(new Map())
+```
+
+Provide `@smthrs/observability`'s exporter layer instead when you want the
+counters to leave the process. Nothing about the stores changes either way: an
+unprovided registry costs a map write and is dropped.
+
+## Read a counter directly
+
+Each exported record maps an outcome tag to an attributed view of its counter, so
+reading one is a lookup rather than a filter:
+
+```ts
+import { RunStoreMetrics } from "@smthrs/run-store"
+import * as Effect from "effect/Effect"
+import * as Metric from "effect/Metric"
+
+const fenceLosses = Effect.map(
+  Metric.value(RunStoreMetrics.heartbeat.FenceLost),
+  (state) => state.count
+)
+```
+
+The same shape works for `RunStoreMetrics.claim`, `claimAndOwn`, `activate`,
+`abandonClaim`, `recoverClaim`, `steal`, and `transition`. Each record's keys are
+exactly the outcome tags of the matching operation, so a new outcome cannot be
+added without a key to count it under.
+
+`transition` is the one exception to "look it up and read it": `RunStore` adds the
+`to` attribute at the update site, because the target status is call input rather
+than a closed set the metrics module should restate. Add the same attribute to
+read a single target:
+
+```ts
+const completedTransitions = Metric.value(
+  Metric.withAttributes(RunStoreMetrics.transition.Transitioned, { to: "completed" })
+)
+```
+
+## Read the spans
+
+Every store operation runs inside a span named after it, `RunStore.claim`,
+`AttemptStore.finish`, and so on. Each span carries the identity it operated
+on, the acting host when the call is owner or claimant fenced, and an `outcome`
+written when the span closes. A success carries its outcome tag in snake case,
+or `success` for operations with no domain outcome. A failed effect carries
+`failure`, and an interrupted one carries `interrupt`, so no span ever closes
+without saying how it ended. Identity attributes are written after the inputs
+are validated, so a call rejected as invalid input closes with `outcome` alone.
+
+| Span                            | Identity                            | Acting host      | `outcome` on success                                                               |
+| ------------------------------- | ----------------------------------- | ---------------- | ---------------------------------------------------------------------------------- |
+| `RunStore.create`               | `runId`                             | none             | `success`                                                                          |
+| `RunStore.get`                  | `runId`                             | none             | `success`                                                                          |
+| `RunStore.lineage`              | `runId`                             | none             | `success`                                                                          |
+| `RunStore.latestRound`          | `runId`                             | none             | `success`                                                                          |
+| `RunStore.requestCancel`        | `runId`                             | none             | request-cancel tag                                                                 |
+| `RunStore.requestCancelLineage` | `runId`                             | none             | request-cancel tag                                                                 |
+| `RunStore.acknowledgeCancel`    | `runId`                             | `ownerHostId`    | `success`                                                                          |
+| `RunStore.claim`                | `runId`                             | `claimantHostId` | claim tag                                                                          |
+| `RunStore.claimAndOwn`          | `runId`                             | `ownerHostId`    | claim-and-own tag                                                                  |
+| `RunStore.activate`             | `runId`                             | `claimantHostId` | activate tag                                                                       |
+| `RunStore.abandonClaim`         | `runId`                             | `claimantHostId` | abandon-claim tag                                                                  |
+| `RunStore.recoverClaim`         | `runId`                             | `observerHostId` | recover-claim tag                                                                  |
+| `RunStore.heartbeat`            | `runId`                             | `ownerHostId`    | heartbeat tag                                                                      |
+| `RunStore.transitionOwned`      | `runId`, `to`                       | `ownerHostId`    | transition tag                                                                     |
+| `RunStore.steal`                | `runId`                             | `claimantHostId` | steal tag                                                                          |
+| `AttemptStore.put`              | `runId`, `stepKeyDigest`, `attempt` | `ownerHostId`    | `inserted`, `upserted`, `existing_same`, `conflict`, `fence_lost`, `run_not_found` |
+| `AttemptStore.get`              | `runId`, `stepKeyDigest`, `attempt` | none             | `success`                                                                          |
+| `AttemptStore.heartbeat`        | `runId`, `stepKeyDigest`, `attempt` | `ownerHostId`    | `updated`, `fence_lost`, `not_found`, `state_changed`                              |
+| `AttemptStore.finish`           | `runId`, `stepKeyDigest`, `attempt` | `ownerHostId`    | `finished`, `fence_lost`, `not_found`, `state_changed`                             |
+| `AttemptStore.patch`            | `runId`, `stepKeyDigest`, `attempt` | `ownerHostId`    | `patched`, `not_found`, `fence_lost`                                               |
+
+A `RunStore` outcome tag is the snake case form of the matching result type's
+`_tag`, `cancel_requested` for `requestCancel` or `fence_lost` for `heartbeat`.
+`RunStore.acknowledgeCancel` returns a boolean rather than a tagged result, so
+its span closes with `success` whether or not the guarded update matched. The
+`SpanAnnotations` test in this package asserts the exact attribute set of each
+`AttemptStore` span and of `acknowledgeCancel`; change both together.
+
+Failure causes are published to logs, spans, and telemetry, so they carry field
+names, lengths, and validity flags and never the value that failed. A
+`decode_failed` cause names the invariants the row broke, not its executable
+state. See [Durable values](../concepts/durable-values.md).
+
+## Assert on a counter in a test
+
+Provide a registry, run the work, and read the count:
+
+```ts
+import { RunStore, RunStoreMetrics } from "@smthrs/run-store"
+import * as TestRunStore from "@smthrs/run-store/test/TestRunStore"
+import * as Effect from "effect/Effect"
+import * as Metric from "effect/Metric"
+
+const lostFences = Effect.gen(function*() {
+  const runs = yield* RunStore.RunStore
+  yield* runs.heartbeat("missing-run", { hostId: "host-a", pid: 1, nonce: "n" }, 0)
+  return yield* Metric.value(RunStoreMetrics.heartbeat.NotFound)
+}).pipe(
+  Effect.provide(TestRunStore.layer),
+  Effect.provideService(Metric.MetricRegistry, new Map()),
+  Effect.scoped
+)
+```
+
+Counting outcomes is how a test proves a race resolved the way it claims: a
+takeover test that asserts one `steal` `claimed` and one `heartbeat`
+`fence_lost` has pinned both halves of the exchange.
+
+## Next steps
+
+- [Test against the real stores](./testing.md): the layer used here.
+- [Troubleshooting](../troubleshooting.md): what each outcome means when it is
+  the one you did not expect.