@smthrs/run-store 0.0.0-stage → 1.0.0-rc.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +112 -0
- package/LICENSE +21 -0
- package/README.md +195 -2
- package/dist/cjs/AttemptStore.d.ts +359 -0
- package/dist/cjs/AttemptStore.d.ts.map +1 -0
- package/dist/cjs/AttemptStore.js +731 -0
- package/dist/cjs/AttemptStore.js.map +7 -0
- package/dist/cjs/Heartbeat.d.ts +46 -0
- package/dist/cjs/Heartbeat.d.ts.map +1 -0
- package/dist/cjs/Heartbeat.js +28 -0
- package/dist/cjs/Heartbeat.js.map +7 -0
- package/dist/cjs/Migrations.d.ts +42 -0
- package/dist/cjs/Migrations.d.ts.map +1 -0
- package/dist/cjs/Migrations.js +55 -0
- package/dist/cjs/Migrations.js.map +7 -0
- package/dist/cjs/Ownership.d.ts +275 -0
- package/dist/cjs/Ownership.d.ts.map +1 -0
- package/dist/cjs/Ownership.js +129 -0
- package/dist/cjs/Ownership.js.map +7 -0
- package/dist/cjs/RunStore.d.ts +571 -0
- package/dist/cjs/RunStore.d.ts.map +1 -0
- package/dist/cjs/RunStore.js +1079 -0
- package/dist/cjs/RunStore.js.map +7 -0
- package/dist/cjs/RunStoreMetrics.d.ts +151 -0
- package/dist/cjs/RunStoreMetrics.d.ts.map +1 -0
- package/dist/cjs/RunStoreMetrics.js +91 -0
- package/dist/cjs/RunStoreMetrics.js.map +7 -0
- package/dist/cjs/index.d.ts +47 -0
- package/dist/cjs/index.d.ts.map +1 -0
- package/dist/cjs/index.js +43 -0
- package/dist/cjs/index.js.map +7 -0
- package/dist/cjs/internal/Boundary.d.ts +80 -0
- package/dist/cjs/internal/Boundary.d.ts.map +1 -0
- package/dist/cjs/internal/Boundary.js +92 -0
- package/dist/cjs/internal/Boundary.js.map +7 -0
- package/dist/cjs/internal/Companion.d.ts +44 -0
- package/dist/cjs/internal/Companion.d.ts.map +1 -0
- package/dist/cjs/internal/Companion.js +50 -0
- package/dist/cjs/internal/Companion.js.map +7 -0
- package/dist/cjs/internal/SpanOutcome.d.ts +37 -0
- package/dist/cjs/internal/SpanOutcome.d.ts.map +1 -0
- package/dist/cjs/internal/SpanOutcome.js +62 -0
- package/dist/cjs/internal/SpanOutcome.js.map +7 -0
- package/dist/cjs/migrations/0001_initial.d.ts +15 -0
- package/dist/cjs/migrations/0001_initial.d.ts.map +1 -0
- package/dist/cjs/migrations/0001_initial.js +110 -0
- package/dist/cjs/migrations/0001_initial.js.map +7 -0
- package/dist/cjs/migrations/0002_lineage.d.ts +27 -0
- package/dist/cjs/migrations/0002_lineage.d.ts.map +1 -0
- package/dist/cjs/migrations/0002_lineage.js +43 -0
- package/dist/cjs/migrations/0002_lineage.js.map +7 -0
- package/dist/cjs/migrations/0003_execution_revisions.d.ts +15 -0
- package/dist/cjs/migrations/0003_execution_revisions.d.ts.map +1 -0
- package/dist/cjs/migrations/0003_execution_revisions.js +77 -0
- package/dist/cjs/migrations/0003_execution_revisions.js.map +7 -0
- package/dist/cjs/migrations/0004_waiting_request.d.ts +32 -0
- package/dist/cjs/migrations/0004_waiting_request.d.ts.map +1 -0
- package/dist/cjs/migrations/0004_waiting_request.js +42 -0
- package/dist/cjs/migrations/0004_waiting_request.js.map +7 -0
- package/dist/cjs/package.json +1 -0
- package/dist/cjs/test/TestRunStore.d.ts +17 -0
- package/dist/cjs/test/TestRunStore.d.ts.map +1 -0
- package/dist/cjs/test/TestRunStore.js +42 -0
- package/dist/cjs/test/TestRunStore.js.map +7 -0
- package/dist/esm/AttemptStore.d.ts +359 -0
- package/dist/esm/AttemptStore.d.ts.map +1 -0
- package/dist/esm/AttemptStore.js +835 -0
- package/dist/esm/AttemptStore.js.map +1 -0
- package/dist/esm/Heartbeat.d.ts +46 -0
- package/dist/esm/Heartbeat.d.ts.map +1 -0
- package/dist/esm/Heartbeat.js +46 -0
- package/dist/esm/Heartbeat.js.map +1 -0
- package/dist/esm/Migrations.d.ts +42 -0
- package/dist/esm/Migrations.d.ts.map +1 -0
- package/dist/esm/Migrations.js +56 -0
- package/dist/esm/Migrations.js.map +1 -0
- package/dist/esm/Ownership.d.ts +275 -0
- package/dist/esm/Ownership.d.ts.map +1 -0
- package/dist/esm/Ownership.js +299 -0
- package/dist/esm/Ownership.js.map +1 -0
- package/dist/esm/RunStore.d.ts +571 -0
- package/dist/esm/RunStore.d.ts.map +1 -0
- package/dist/esm/RunStore.js +1287 -0
- package/dist/esm/RunStore.js.map +1 -0
- package/dist/esm/RunStoreMetrics.d.ts +151 -0
- package/dist/esm/RunStoreMetrics.d.ts.map +1 -0
- package/dist/esm/RunStoreMetrics.js +120 -0
- package/dist/esm/RunStoreMetrics.js.map +1 -0
- package/dist/esm/index.d.ts +47 -0
- package/dist/esm/index.d.ts.map +1 -0
- package/dist/esm/index.js +47 -0
- package/dist/esm/index.js.map +1 -0
- package/dist/esm/internal/Boundary.d.ts +80 -0
- package/dist/esm/internal/Boundary.d.ts.map +1 -0
- package/dist/esm/internal/Boundary.js +109 -0
- package/dist/esm/internal/Boundary.js.map +1 -0
- package/dist/esm/internal/Companion.d.ts +44 -0
- package/dist/esm/internal/Companion.d.ts.map +1 -0
- package/dist/esm/internal/Companion.js +53 -0
- package/dist/esm/internal/Companion.js.map +1 -0
- package/dist/esm/internal/SpanOutcome.d.ts +37 -0
- package/dist/esm/internal/SpanOutcome.d.ts.map +1 -0
- package/dist/esm/internal/SpanOutcome.js +48 -0
- package/dist/esm/internal/SpanOutcome.js.map +1 -0
- package/dist/esm/migrations/0001_initial.d.ts +15 -0
- package/dist/esm/migrations/0001_initial.d.ts.map +1 -0
- package/dist/esm/migrations/0001_initial.js +88 -0
- package/dist/esm/migrations/0001_initial.js.map +1 -0
- package/dist/esm/migrations/0002_lineage.d.ts +27 -0
- package/dist/esm/migrations/0002_lineage.d.ts.map +1 -0
- package/dist/esm/migrations/0002_lineage.js +33 -0
- package/dist/esm/migrations/0002_lineage.js.map +1 -0
- package/dist/esm/migrations/0003_execution_revisions.d.ts +15 -0
- package/dist/esm/migrations/0003_execution_revisions.d.ts.map +1 -0
- package/dist/esm/migrations/0003_execution_revisions.js +58 -0
- package/dist/esm/migrations/0003_execution_revisions.js.map +1 -0
- package/dist/esm/migrations/0004_waiting_request.d.ts +32 -0
- package/dist/esm/migrations/0004_waiting_request.d.ts.map +1 -0
- package/dist/esm/migrations/0004_waiting_request.js +37 -0
- package/dist/esm/migrations/0004_waiting_request.js.map +1 -0
- package/dist/esm/test/TestRunStore.d.ts +17 -0
- package/dist/esm/test/TestRunStore.d.ts.map +1 -0
- package/dist/esm/test/TestRunStore.js +19 -0
- package/dist/esm/test/TestRunStore.js.map +1 -0
- package/docs/README.md +142 -0
- package/docs/api.md +901 -0
- package/docs/concepts/durable-values.md +140 -0
- package/docs/concepts/execution-revisions.md +53 -0
- package/docs/concepts/fencing.md +146 -0
- package/docs/concepts/leases.md +154 -0
- package/docs/concepts/liveness-evidence.md +142 -0
- package/docs/guides/cancel-a-run.md +156 -0
- package/docs/guides/claim-and-finish-a-run.md +175 -0
- package/docs/guides/compose-the-stores.md +142 -0
- package/docs/guides/observe-outcomes.md +182 -0
- package/docs/guides/record-step-attempts.md +170 -0
- package/docs/guides/recover-a-stalled-run.md +173 -0
- package/docs/guides/testing.md +141 -0
- package/docs/installation.md +109 -0
- package/docs/quickstart.md +201 -0
- package/docs/troubleshooting.md +255 -0
- package/package.json +183 -3
- package/src/AttemptStore.ts +1195 -0
- package/src/Heartbeat.ts +47 -0
- package/src/Migrations.ts +59 -0
- package/src/Ownership.ts +407 -0
- package/src/RunStore.ts +2063 -0
- package/src/RunStoreMetrics.ts +165 -0
- package/src/index.ts +51 -0
- package/src/internal/Boundary.ts +130 -0
- package/src/internal/Companion.ts +64 -0
- package/src/internal/SpanOutcome.ts +71 -0
- package/src/migrations/0001_initial.ts +114 -0
- package/src/migrations/0002_lineage.ts +38 -0
- package/src/migrations/0003_execution_revisions.ts +69 -0
- package/src/migrations/0004_waiting_request.ts +39 -0
- package/src/test/TestRunStore.ts +22 -0
package/docs/api.md
ADDED
|
@@ -0,0 +1,901 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "API reference"
|
|
3
|
+
description: "Every public export of @smthrs/run-store: the RunStore and AttemptStore services and their outcomes, ownership arbitration, the heartbeat constants, the fencing metrics, the migration set, and the test layer."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
`@smthrs/run-store` exports five namespaces from its root entry point, and each
|
|
7
|
+
is also importable from `@smthrs/run-store/<Module>`:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
import { AttemptStore, Migrations, Ownership, RunStore, RunStoreMetrics } from "@smthrs/run-store"
|
|
11
|
+
// or
|
|
12
|
+
import * as RunStore from "@smthrs/run-store/RunStore"
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Two subpaths are not namespaces of the root barrel:
|
|
16
|
+
`@smthrs/run-store/Heartbeat`, the lease-constant leaf, and
|
|
17
|
+
`@smthrs/run-store/test/TestRunStore`, which binds a Node database.
|
|
18
|
+
`@smthrs/run-store/package.json` is exported. `internal/*`, `migrations/*`, and
|
|
19
|
+
nested `*/index` are blocked in the export map and are not public API.
|
|
20
|
+
|
|
21
|
+
Services and tags are Effect constructs: a `Layer` provides a service, and an
|
|
22
|
+
effect reads it from context. `RunStore.layer` and `AttemptStore.layer` each
|
|
23
|
+
require a `SqlClient` and a `DurableWriter` from
|
|
24
|
+
[`@smthrs/database`](/api/database).
|
|
25
|
+
|
|
26
|
+
## Example
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
import { RunStore } from "@smthrs/run-store"
|
|
30
|
+
const owner = { hostId: "worker-1", pid: process.pid, nonce: "9c31-af02" }
|
|
31
|
+
const program = Effect.gen(function*() {
|
|
32
|
+
const runs = yield* RunStore.RunStore
|
|
33
|
+
yield* runs.create("build-42", JSON.stringify({ step: "checkout" }))
|
|
34
|
+
const taken = yield* runs.claimAndOwn(
|
|
35
|
+
"build-42",
|
|
36
|
+
{ status: "pending", owner: null, heartbeatAtMs: null },
|
|
37
|
+
owner,
|
|
38
|
+
Date.now()
|
|
39
|
+
)
|
|
40
|
+
if (taken._tag === "Activated") yield* runs.transitionOwned("build-42", owner, "completed")
|
|
41
|
+
})
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
## Entry points
|
|
45
|
+
|
|
46
|
+
| Import | Source | Platform |
|
|
47
|
+
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------- |
|
|
48
|
+
| `@smthrs/run-store` | [src/index.ts](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/run-store/src/index.ts) | any |
|
|
49
|
+
| `@smthrs/run-store/Heartbeat` | [src/Heartbeat.ts](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/run-store/src/Heartbeat.ts) | any |
|
|
50
|
+
| `@smthrs/run-store/test/TestRunStore` | [src/test/TestRunStore.ts](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/run-store/src/test/TestRunStore.ts) | Node |
|
|
51
|
+
|
|
52
|
+
The root is driver-neutral and browser-bundleable: nothing under it imports a
|
|
53
|
+
`node:` built-in.
|
|
54
|
+
|
|
55
|
+
## RunStore
|
|
56
|
+
|
|
57
|
+
One row per durable run: its status, its owner, its heartbeat, its cancellation
|
|
58
|
+
intent, and the executable state a resume re-enters.
|
|
59
|
+
|
|
60
|
+
### RunStore.RunStore
|
|
61
|
+
|
|
62
|
+
```ts
|
|
63
|
+
class RunStore extends Context.Service<RunStore, Service>()("@smthrs/run-store/RunStore")
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The service tag. The identity string equals the defining module path.
|
|
67
|
+
|
|
68
|
+
### RunStore.Service
|
|
69
|
+
|
|
70
|
+
Fourteen operations. Every one validates its input first and fails with
|
|
71
|
+
`RunStoreError` on a defect; competition is reported as a success value.
|
|
72
|
+
|
|
73
|
+
#### create
|
|
74
|
+
|
|
75
|
+
```ts
|
|
76
|
+
const create: (runId: string, stateJson: string, options?: CreateOptions) => Effect<void, RunStoreError>
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Inserts a `pending` row. `stateJson` is JSON text: it is parsed, bounded, and
|
|
80
|
+
stored as the caller's own bytes. `created_at_ms` is stamped from the Effect
|
|
81
|
+
`Clock`. A `runId` that begins with the journal's `companionPrefix` names a
|
|
82
|
+
companion stream and fails `invalid_run`.
|
|
83
|
+
|
|
84
|
+
#### get
|
|
85
|
+
|
|
86
|
+
```ts
|
|
87
|
+
const get: (runId: string) => Effect<RunRow, RunStoreError>
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Reads one row and re-validates it. Fails with `not_found_row` when there is no
|
|
91
|
+
such run, and with `decode_failed` when the row breaks the durable invariants.
|
|
92
|
+
`latestRound` also reports a missing row as a failure.
|
|
93
|
+
|
|
94
|
+
#### lineage and latestRound
|
|
95
|
+
|
|
96
|
+
```ts
|
|
97
|
+
const lineage: (runId: string) => Effect<ReadonlyArray<RunRow>, RunStoreError>
|
|
98
|
+
const latestRound: (runId: string) => Effect<RunRow, RunStoreError>
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Resolve trampoline membership from any existing round. `lineage` returns all
|
|
102
|
+
rounds in ordinal order, or an empty array for an unknown ID. `latestRound`
|
|
103
|
+
returns the highest ordinal using the lineage index, or `not_found_row`.
|
|
104
|
+
Pre-lineage roots with null lineage columns are included as round zero. A run
|
|
105
|
+
whose own ID equals the lineage ID joins only when its lineage columns are
|
|
106
|
+
null or it explicitly names that lineage, so a same-named run in an
|
|
107
|
+
independent lineage is excluded.
|
|
108
|
+
Fork ancestry in `parentRunId` does not join otherwise independent lineages.
|
|
109
|
+
These are individual snapshot reads; compose several reads with the owning
|
|
110
|
+
database transaction when they must describe one coherent state.
|
|
111
|
+
|
|
112
|
+
#### requestCancelLineage
|
|
113
|
+
|
|
114
|
+
```ts
|
|
115
|
+
const requestCancelLineage: (runId: string, nowMs: number) => Effect<RequestCancelOutcome, RunStoreError>
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Records cancellation for every nonterminal round of the named logical run in
|
|
119
|
+
one write transaction: one guarded UPDATE over the lineage members, and on a
|
|
120
|
+
miss one read of their status and request columns. Cost follows the number of
|
|
121
|
+
rounds and never their state payloads, so a long settled history adds nothing.
|
|
122
|
+
A completed predecessor does not hide its live handoff successor. Existing
|
|
123
|
+
request times and completed-round history are unchanged.
|
|
124
|
+
Outcome precedence: `CancelRequested` if any round gets a new request;
|
|
125
|
+
otherwise `AlreadyRequested` with the latest live round's earlier request;
|
|
126
|
+
otherwise, with every round settled, `Terminal` with the latest round's status;
|
|
127
|
+
`NotFound` when no round exists. A live round without a request that the
|
|
128
|
+
UPDATE did not reach fails with `persistence_failed`, since the writer
|
|
129
|
+
serializes the transaction and no peer can clear the column mid-call.
|
|
130
|
+
This method does not traverse child ownership edges or interrupt local fibers;
|
|
131
|
+
the engine coordinates those operations.
|
|
132
|
+
|
|
133
|
+
#### requestCancel
|
|
134
|
+
|
|
135
|
+
```ts
|
|
136
|
+
const requestCancel: (runId: string, nowMs: number) => Effect<RequestCancelOutcome, RunStoreError>
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Records unfenced cancellation intent for exactly one round that a later guarded transition observes.
|
|
140
|
+
Any observer may call it, and it is first-writer-wins, so a repeat reports the
|
|
141
|
+
original time. A settled run records nothing. The call is one guarded UPDATE
|
|
142
|
+
and, on a miss, one read of the status and request columns; it never retries,
|
|
143
|
+
because the durable writer serializes the whole call. `nowMs` is request data rather than
|
|
144
|
+
a lease predicate, so it is checked as a non-negative safe integer and not bound
|
|
145
|
+
by the skew allowance.
|
|
146
|
+
|
|
147
|
+
#### acknowledgeCancel
|
|
148
|
+
|
|
149
|
+
```ts
|
|
150
|
+
const acknowledgeCancel: (runId: string, owner: OwnerId, nowMs: number) => Effect<boolean, RunStoreError>
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
Records the first owner observation of cancellation intent under the running
|
|
154
|
+
owner fence. Returns false when the run is missing, not running, has no intent,
|
|
155
|
+
or belongs to another owner. Repeated acknowledgements preserve the original
|
|
156
|
+
owner and timestamp. This does not make the run terminal or prove cleanup ended.
|
|
157
|
+
Engine snapshots expose the record separately from request and finish.
|
|
158
|
+
|
|
159
|
+
#### claim
|
|
160
|
+
|
|
161
|
+
```ts
|
|
162
|
+
const claim: (
|
|
163
|
+
runId: string,
|
|
164
|
+
expected: RunSnapshot,
|
|
165
|
+
claimant: OwnerId,
|
|
166
|
+
nowMs: number
|
|
167
|
+
) => Effect<ClaimOutcome, RunStoreError>
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
Reserves an exact `pending` or `suspended` snapshot for a later `activate`.
|
|
171
|
+
`nowMs` becomes `claimed_at_ms` and is the token `activate` compares against.
|
|
172
|
+
|
|
173
|
+
#### claimAndOwn
|
|
174
|
+
|
|
175
|
+
```ts
|
|
176
|
+
const claimAndOwn: (
|
|
177
|
+
runId: string,
|
|
178
|
+
expected: RunSnapshot,
|
|
179
|
+
owner: OwnerId,
|
|
180
|
+
nowMs: number,
|
|
181
|
+
evidence?: LivenessEvidence
|
|
182
|
+
) => Effect<ClaimAndOwnOutcome, RunStoreError>
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
Claims and activates in one compare-and-swap. It admits `pending`, `suspended`,
|
|
186
|
+
and `running` rows; replacing a different running owner additionally requires
|
|
187
|
+
matching evidence, whose `checkedAtMs` must equal `nowMs` exactly. `nowMs` is the
|
|
188
|
+
lease cutoff and the first heartbeat.
|
|
189
|
+
|
|
190
|
+
#### activate
|
|
191
|
+
|
|
192
|
+
```ts
|
|
193
|
+
const activate: (
|
|
194
|
+
runId: string,
|
|
195
|
+
claimant: OwnerId,
|
|
196
|
+
claimedAtMs: number,
|
|
197
|
+
expected: RunSnapshot
|
|
198
|
+
) => Effect<ActivateOutcome, RunStoreError>
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Trades a held claim for ownership: sets `status = 'running'`, writes the owner
|
|
202
|
+
columns and the first heartbeat from the Effect `Clock`, stamps `started_at_ms`
|
|
203
|
+
if it is not already set, and clears the claim. `claimedAtMs` is the fence token
|
|
204
|
+
`claim` or `steal` returned.
|
|
205
|
+
|
|
206
|
+
#### abandonClaim
|
|
207
|
+
|
|
208
|
+
```ts
|
|
209
|
+
const abandonClaim: (
|
|
210
|
+
runId: string,
|
|
211
|
+
claimant: OwnerId,
|
|
212
|
+
claimedAtMs: number
|
|
213
|
+
) => Effect<AbandonClaimOutcome, RunStoreError>
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
Releases a claim you hold, so a failed activation does not block the next
|
|
217
|
+
claimant until the claim goes stale.
|
|
218
|
+
|
|
219
|
+
#### recoverClaim
|
|
220
|
+
|
|
221
|
+
```ts
|
|
222
|
+
const recoverClaim: (
|
|
223
|
+
runId: string,
|
|
224
|
+
staleClaimant: OwnerId,
|
|
225
|
+
claimedAtMs: number,
|
|
226
|
+
observer: OwnerId,
|
|
227
|
+
nowMs: number,
|
|
228
|
+
evidence: LivenessEvidence
|
|
229
|
+
) => Effect<RecoverClaimOutcome, RunStoreError>
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
Clears an exact stale claim after matching its claimant and liveness evidence.
|
|
233
|
+
The claim must be older than `heartbeatStaleAfter` relative to `nowMs`.
|
|
234
|
+
`claimedAtMs` is compared against the row and is not bound by the skew allowance,
|
|
235
|
+
because the store issued it.
|
|
236
|
+
|
|
237
|
+
#### heartbeat
|
|
238
|
+
|
|
239
|
+
```ts
|
|
240
|
+
const heartbeat: (runId: string, owner: OwnerId, nowMs: number) => Effect<HeartbeatOutcome, RunStoreError>
|
|
241
|
+
const reconfirm: (runId: string, owner: OwnerId, nowMs: number) => Effect<HeartbeatOutcome, RunStoreError>
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Renews the owner's lease. The write is `MAX(heartbeat_at_ms, nowMs)`, so a
|
|
245
|
+
reading behind the persisted stamp still reports `Updated` without moving the
|
|
246
|
+
lease backwards.
|
|
247
|
+
|
|
248
|
+
#### transitionOwned
|
|
249
|
+
|
|
250
|
+
```ts
|
|
251
|
+
const transitionOwned: (
|
|
252
|
+
runId: string,
|
|
253
|
+
owner: OwnerId,
|
|
254
|
+
toStatus: RunStatus,
|
|
255
|
+
stateJson?: string,
|
|
256
|
+
guard?: TransitionGuard
|
|
257
|
+
) => Effect<TransitionOutcome, RunStoreError>
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
The only way to move a run you own. `running` keeps the owner and rewrites only
|
|
261
|
+
the state. `suspended` clears the owner, the heartbeat, and the claim and leaves
|
|
262
|
+
`finished_at_ms` null. `completed`, `failed`, and `cancelled` clear the same
|
|
263
|
+
columns and stamp `finished_at_ms`. `pending` fails with `invalid_run`. An
|
|
264
|
+
omitted `stateJson` leaves the column as recorded; a supplied one replaces it
|
|
265
|
+
whole. The guard is compiled into the same `UPDATE` as the ownership fence.
|
|
266
|
+
|
|
267
|
+
#### steal
|
|
268
|
+
|
|
269
|
+
```ts
|
|
270
|
+
const steal: (
|
|
271
|
+
runId: string,
|
|
272
|
+
expected: RunSnapshot,
|
|
273
|
+
claimant: OwnerId,
|
|
274
|
+
nowMs: number,
|
|
275
|
+
evidence: LivenessEvidence
|
|
276
|
+
) => Effect<StealOutcome, RunStoreError>
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
Writes the claim columns of a stale `running` row after verifying evidence,
|
|
280
|
+
leaving status, owner, and heartbeat untouched so the same `expected` snapshot
|
|
281
|
+
activates. The write refuses any row whose `heartbeat_at_ms` is within
|
|
282
|
+
`heartbeatStaleAfter` of `nowMs`, and any row that already carries a claim.
|
|
283
|
+
|
|
284
|
+
### RunStore models
|
|
285
|
+
|
|
286
|
+
#### RunStatus
|
|
287
|
+
|
|
288
|
+
```ts
|
|
289
|
+
const RunStatus: Schema.Literals<["pending", "running", "suspended", "completed", "failed", "cancelled"]>
|
|
290
|
+
type RunStatus = "pending" | "running" | "suspended" | "completed" | "failed" | "cancelled"
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
#### TerminalRunStatus and isTerminalRunStatus
|
|
294
|
+
|
|
295
|
+
```ts
|
|
296
|
+
const TerminalRunStatus: Schema.Literals<["completed", "failed", "cancelled"]>
|
|
297
|
+
type TerminalRunStatus = "completed" | "failed" | "cancelled"
|
|
298
|
+
const isTerminalRunStatus: (status: RunStatus) => status is TerminalRunStatus
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
The states a run never leaves. A settled run refuses new cancellation intent and
|
|
302
|
+
is never claimed, activated, or swept.
|
|
303
|
+
|
|
304
|
+
#### RunSnapshot
|
|
305
|
+
|
|
306
|
+
```ts
|
|
307
|
+
interface RunSnapshot {
|
|
308
|
+
readonly status: RunStatus
|
|
309
|
+
readonly owner: OwnerId | null
|
|
310
|
+
readonly heartbeatAtMs: number | null
|
|
311
|
+
}
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
The triple every claim guards. `running` requires both an owner and a
|
|
315
|
+
heartbeat; every other status requires neither. Admission is structural: the
|
|
316
|
+
store reads only these three fields as own enumerable data properties and
|
|
317
|
+
copies them, so a `RunRow` from `get` passes directly as `expected`. Other own
|
|
318
|
+
data fields are ignored. A non-plain prototype, an enumerable accessor on any
|
|
319
|
+
key, or a missing or inherited required field is refused as invalid input.
|
|
320
|
+
|
|
321
|
+
#### RunRow
|
|
322
|
+
|
|
323
|
+
```ts
|
|
324
|
+
interface RunRow extends RunSnapshot {
|
|
325
|
+
readonly runId: string
|
|
326
|
+
readonly createdAtMs: number
|
|
327
|
+
readonly startedAtMs: number | null
|
|
328
|
+
readonly finishedAtMs: number | null
|
|
329
|
+
readonly claim: OwnerId | null
|
|
330
|
+
readonly claimedAtMs: number | null
|
|
331
|
+
readonly parentRunId: string | null
|
|
332
|
+
readonly cancelRequestedAtMs: number | null
|
|
333
|
+
readonly lineageId?: string | null | undefined
|
|
334
|
+
readonly roundOrdinal?: number | null | undefined
|
|
335
|
+
readonly stateJson: string
|
|
336
|
+
}
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
A decoded `flows_runs` row, frozen. `lineageId` and `roundOrdinal` are optional
|
|
340
|
+
on the interface because they arrived in an append-only migration, so a
|
|
341
|
+
hand-built row still compiles.
|
|
342
|
+
|
|
343
|
+
#### CreateOptions
|
|
344
|
+
|
|
345
|
+
```ts
|
|
346
|
+
interface CreateOptions {
|
|
347
|
+
readonly parentRunId?: string | undefined
|
|
348
|
+
readonly lineageId?: string | undefined
|
|
349
|
+
readonly roundOrdinal?: number | undefined
|
|
350
|
+
}
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
`parentRunId` is the ancestry edge a fork, rewind, child, or trampoline round
|
|
354
|
+
carries. It is a column rather than a state field because ancestry is walked in
|
|
355
|
+
SQL, and it is a foreign key, so the parent must exist. `lineageId` and
|
|
356
|
+
`roundOrdinal` name the trampoline lineage; both absent reads back as round 0 of
|
|
357
|
+
a lineage of one.
|
|
358
|
+
|
|
359
|
+
#### TransitionGuard
|
|
360
|
+
|
|
361
|
+
```ts
|
|
362
|
+
const TransitionGuard: Schema.Struct<{ cancelRequested: Schema.optional<Schema.Literals<["absent", "present"]>> }>
|
|
363
|
+
type TransitionGuard = { readonly cancelRequested?: "absent" | "present" }
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
An extra compare-and-swap predicate over first-class run metadata, compiled into
|
|
367
|
+
the transition's own `UPDATE`.
|
|
368
|
+
|
|
369
|
+
### RunStore outcomes
|
|
370
|
+
|
|
371
|
+
Every tag below is a success value. Union members carrying data are noted.
|
|
372
|
+
|
|
373
|
+
| Type | Tags |
|
|
374
|
+
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
375
|
+
| `RequestCancelOutcome` | `CancelRequested { requestedAtMs }`, `AlreadyRequested { requestedAtMs }`, `NotFound`, `Terminal { status }` |
|
|
376
|
+
| `ClaimOutcome` | `Claimed { claimedAtMs }`, `NotFound`, `AlreadyClaimed`, `HeartbeatFresh`, `SnapshotChanged` |
|
|
377
|
+
| `StealOutcome` | `ClaimOutcome` plus `LivenessUnconfirmed` |
|
|
378
|
+
| `ClaimAndOwnOutcome` | `Activated`, `NotFound`, `AlreadyClaimed`, `HeartbeatFresh`, `SnapshotChanged`, `EvidenceRequired` |
|
|
379
|
+
| `ActivateOutcome` | `Activated`, `ClaimLost`, `SnapshotChanged` |
|
|
380
|
+
| `AbandonClaimOutcome` | `Abandoned`, `ClaimLost` |
|
|
381
|
+
| `RecoverClaimOutcome` | `Recovered`, `NotFound`, `ClaimFresh`, `ClaimChanged`, `LivenessUnconfirmed` |
|
|
382
|
+
| `HeartbeatOutcome` | `Updated`, `FenceLost`, `NotFound` |
|
|
383
|
+
| `TransitionOutcome` | `Transitioned`, `FenceLost`, `NotFound`, `GuardFailed` |
|
|
384
|
+
|
|
385
|
+
`LivenessUnconfirmed` means the evidence did not match, so no compare-and-swap
|
|
386
|
+
ran. `SnapshotChanged` is reserved for matching evidence whose comparison lost to
|
|
387
|
+
a row that moved. `EvidenceRequired` means the snapshot is current, its different
|
|
388
|
+
owner is stale, and no evidence was supplied: retrying cannot help.
|
|
389
|
+
`GuardFailed` means you own the run and the guard refused the write. What to do
|
|
390
|
+
about each is in [Troubleshooting](./troubleshooting.md).
|
|
391
|
+
|
|
392
|
+
### RunStore errors
|
|
393
|
+
|
|
394
|
+
```ts
|
|
395
|
+
const RunStoreErrorCode: Schema.Literals<
|
|
396
|
+
["invalid_run", "not_found_row", "constraint", "decode_failed", "persistence_failed"]
|
|
397
|
+
>
|
|
398
|
+
|
|
399
|
+
class RunStoreError extends Schema.TaggedError<RunStoreError>()("@smthrs/run-store/RunStoreError", {
|
|
400
|
+
code: RunStoreErrorCode
|
|
401
|
+
method: Schema.String
|
|
402
|
+
message: Schema.String
|
|
403
|
+
cause: Schema.Unknown
|
|
404
|
+
})
|
|
405
|
+
```
|
|
406
|
+
|
|
407
|
+
`message` is prefixed with the code and the method, as in
|
|
408
|
+
`invalid_run: RunStore.claim: run input is invalid`. `cause` carries field names,
|
|
409
|
+
lengths, and validity flags, never the value that failed.
|
|
410
|
+
|
|
411
|
+
### RunStore constants
|
|
412
|
+
|
|
413
|
+
| Constant | Value | Bounds |
|
|
414
|
+
| --------------------- | ------- | --------------------- |
|
|
415
|
+
| `maximumRunJsonDepth` | 128 | Executable run state. |
|
|
416
|
+
| `maximumRunJsonNodes` | 100,000 | Executable run state. |
|
|
417
|
+
|
|
418
|
+
Run state has no byte ceiling by design: it is what a resume re-enters, so a
|
|
419
|
+
large state has to persist.
|
|
420
|
+
|
|
421
|
+
### Ownership transitions
|
|
422
|
+
|
|
423
|
+
```ts
|
|
424
|
+
const OwnershipTransition: Schema<"claimed" | "activated" | "released" | "stolen" | "expired">
|
|
425
|
+
const ownershipEventType: (transition: OwnershipTransition) => string // "flows.consensus.<transition>"
|
|
426
|
+
const companionStream: string // "run-store"
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
Every transition the consensus strategy grants is appended as a
|
|
430
|
+
`flows.consensus.<transition>` fact, with payload `{ runId, owner,
|
|
431
|
+
grantedAtMs }`, to the run's companion stream
|
|
432
|
+
`JournalEvent.companionRunId(companionStream, runId)`, in the transaction that
|
|
433
|
+
made it. `claim` records `claimed`; `activate` records `activated`, or
|
|
434
|
+
`released` when the snapshot moved on; `claimAndOwn` records `claimed` or
|
|
435
|
+
`stolen` and then `activated`, with a `released` first when an owner re-owns
|
|
436
|
+
its own stale run; `steal` records `stolen`; `abandonClaim` and an owner's
|
|
437
|
+
non-running `transitionOwned` record `released`; `recoverClaim` records
|
|
438
|
+
`expired`. A refused operation and a heartbeat record nothing, and a
|
|
439
|
+
rolled-back transaction leaves no fact.
|
|
440
|
+
|
|
441
|
+
The facts sit beside the run's own stream, so a consumer that reads that stream
|
|
442
|
+
by position never sees one. They are recorded when the caller runs with a
|
|
443
|
+
`Journal` in its context, which the ordinary composition
|
|
444
|
+
(`Layer.mergeAll(SqlJournal.layer(...), RunStore.layer)`) provides.
|
|
445
|
+
|
|
446
|
+
### RunStore constructors and layers
|
|
447
|
+
|
|
448
|
+
```ts
|
|
449
|
+
const make: Effect<Service, never, DurableWriter | SqlClient.SqlClient>
|
|
450
|
+
const makeNoop: (overrides?: Partial<Service>) => Service
|
|
451
|
+
const layer: Layer<RunStore, never, DurableWriter | SqlClient.SqlClient>
|
|
452
|
+
const layerNoop: (overrides?: Partial<Service>) => Layer<RunStore>
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
`makeNoop` is an explicit absence: `create` and `get` fail with
|
|
456
|
+
`persistence_failed`, and every compare-and-swap reports a loss, `NotFound` or
|
|
457
|
+
`ClaimLost` for `activate` and `abandonClaim`.
|
|
458
|
+
|
|
459
|
+
## AttemptStore
|
|
460
|
+
|
|
461
|
+
One row per step attempt, addressed by `(runId, stepKeyDigest, attempt)`. Every
|
|
462
|
+
write is fenced on the run being `running` under the caller's owner; there is no
|
|
463
|
+
unfenced write surface.
|
|
464
|
+
|
|
465
|
+
### AttemptStore.AttemptStore
|
|
466
|
+
|
|
467
|
+
```ts
|
|
468
|
+
class AttemptStore extends Context.Service<AttemptStore, Service>()("@smthrs/run-store/AttemptStore")
|
|
469
|
+
```
|
|
470
|
+
|
|
471
|
+
### AttemptStore.Service
|
|
472
|
+
|
|
473
|
+
#### put
|
|
474
|
+
|
|
475
|
+
```ts
|
|
476
|
+
const put: (attempt: Attempt, owner: OwnerId) => Effect<PutResult, AttemptStoreError>
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
Inserts the attempt row. `PutResult` is `Inserted`, `Upserted`, `ExistingSame`,
|
|
480
|
+
`Conflict`, `FenceLost`, or `RunNotFound`. `ExistingSame` is a byte-equivalent
|
|
481
|
+
replay, which makes a repeated `put` safe; object key order is ignored in that
|
|
482
|
+
comparison and array order is not. `Upserted` occurs only under
|
|
483
|
+
`putMode: "upsert"` and only while the existing row is still in progress.
|
|
484
|
+
|
|
485
|
+
#### get
|
|
486
|
+
|
|
487
|
+
```ts
|
|
488
|
+
const get: (id: AttemptId) => Effect<Option<Attempt>, AttemptStoreError>
|
|
489
|
+
```
|
|
490
|
+
|
|
491
|
+
The one unfenced operation. Absent optional columns come back as absent keys
|
|
492
|
+
rather than nulls, and every value is re-validated on the way out.
|
|
493
|
+
|
|
494
|
+
Every `AttemptId` parameter (`get`, `patch`) is admitted structurally: the store
|
|
495
|
+
reads only `runId`, `stepKeyDigest`, and `attempt` as own enumerable data
|
|
496
|
+
properties and ignores other own data fields, so an `Attempt` from `put` or
|
|
497
|
+
`get` passes directly as the id. A non-plain prototype, an enumerable accessor
|
|
498
|
+
on any key, or a missing or inherited id field is refused with
|
|
499
|
+
`invalid_attempt`.
|
|
500
|
+
|
|
501
|
+
#### heartbeat
|
|
502
|
+
|
|
503
|
+
```ts
|
|
504
|
+
const heartbeat: (
|
|
505
|
+
runId: string,
|
|
506
|
+
stepKeyDigest: string,
|
|
507
|
+
attempt: number,
|
|
508
|
+
owner: OwnerId,
|
|
509
|
+
nowMs: number,
|
|
510
|
+
checkpoint?: JsonValue
|
|
511
|
+
) => Effect<HeartbeatResult, AttemptStoreError>
|
|
512
|
+
```
|
|
513
|
+
|
|
514
|
+
Proves the attempt is still moving and optionally saves the value it would resume
|
|
515
|
+
from. The checkpoint column is written with `COALESCE`, so omitting the argument
|
|
516
|
+
leaves the stored checkpoint alone. The stamp is monotonic. `HeartbeatResult` is
|
|
517
|
+
`Updated`, `FenceLost`, `NotFound`, or `StateChanged`, where `StateChanged` means
|
|
518
|
+
you own the run but the attempt is no longer in `inProgressStates`.
|
|
519
|
+
|
|
520
|
+
#### finish
|
|
521
|
+
|
|
522
|
+
```ts
|
|
523
|
+
const finish: (attempt: FinishAttempt, owner: OwnerId) => Effect<FinishResult, AttemptStoreError>
|
|
524
|
+
```
|
|
525
|
+
|
|
526
|
+
Moves the attempt to a terminal state. The target state must not be one of
|
|
527
|
+
`inProgressStates`. `FinishResult` is `Finished`, `FenceLost`, `NotFound`, or
|
|
528
|
+
`StateChanged`, so a second `finish` never overwrites the winning row.
|
|
529
|
+
|
|
530
|
+
#### patch
|
|
531
|
+
|
|
532
|
+
```ts
|
|
533
|
+
const patch: (id: AttemptId, patch: AttemptPatch, owner: OwnerId) => Effect<PatchResult, AttemptStoreError>
|
|
534
|
+
```
|
|
535
|
+
|
|
536
|
+
Rewrites the opaque fields without touching the lifecycle, on running and
|
|
537
|
+
terminal rows alike. `PatchResult` is `Patched`, `NotFound`, or `FenceLost`. It
|
|
538
|
+
is still fenced on run ownership, so a patch after the run settles reports
|
|
539
|
+
`FenceLost`.
|
|
540
|
+
|
|
541
|
+
### AttemptStore models
|
|
542
|
+
|
|
543
|
+
#### AttemptId, Attempt, FinishAttempt, AttemptPatch
|
|
544
|
+
|
|
545
|
+
Each is a `Schema.Struct` exported under its own name, with a `type` alias of the
|
|
546
|
+
same name for the decoded value:
|
|
547
|
+
|
|
548
|
+
```ts
|
|
549
|
+
interface AttemptId {
|
|
550
|
+
readonly runId: string
|
|
551
|
+
readonly stepKeyDigest: string
|
|
552
|
+
readonly attempt: number
|
|
553
|
+
}
|
|
554
|
+
|
|
555
|
+
interface Attempt extends AttemptId {
|
|
556
|
+
readonly state: string
|
|
557
|
+
readonly startedAtMs: number
|
|
558
|
+
readonly finishedAtMs?: number
|
|
559
|
+
readonly heartbeatAtMs?: number
|
|
560
|
+
readonly checkpoint?: JsonValue
|
|
561
|
+
readonly error?: JsonValue
|
|
562
|
+
readonly outcome?: JsonValue
|
|
563
|
+
readonly meta: JsonValue
|
|
564
|
+
}
|
|
565
|
+
|
|
566
|
+
interface FinishAttempt extends AttemptId {
|
|
567
|
+
readonly state: string
|
|
568
|
+
readonly finishedAtMs: number
|
|
569
|
+
readonly error?: JsonValue
|
|
570
|
+
readonly outcome?: JsonValue
|
|
571
|
+
readonly meta?: JsonValue
|
|
572
|
+
}
|
|
573
|
+
|
|
574
|
+
interface AttemptPatch {
|
|
575
|
+
readonly checkpoint?: JsonValue
|
|
576
|
+
readonly error?: JsonValue
|
|
577
|
+
readonly outcome?: JsonValue
|
|
578
|
+
readonly meta?: JsonValue
|
|
579
|
+
}
|
|
580
|
+
```
|
|
581
|
+
|
|
582
|
+
`runId`, `stepKeyDigest`, and `state` are durable text: non-empty, at most 1,024
|
|
583
|
+
UTF-16 units, no NUL, no lone surrogate. `attempt` and every timestamp are
|
|
584
|
+
non-negative safe integers, range-checked independently and never against each
|
|
585
|
+
other, so an attempt whose `finishedAtMs` precedes its `startedAtMs` persists as
|
|
586
|
+
written.
|
|
587
|
+
|
|
588
|
+
`meta` is required on `Attempt` and opaque: its shape belongs to the step
|
|
589
|
+
executor, and the store carries it unchanged. On `FinishAttempt` and
|
|
590
|
+
`AttemptPatch` an omitted field is left as recorded rather than cleared, so a
|
|
591
|
+
terminal transition never erases a value written mid-flight. `AttemptPatch` never
|
|
592
|
+
moves `state`, `startedAtMs`, or `finishedAtMs`.
|
|
593
|
+
|
|
594
|
+
#### JsonValue
|
|
595
|
+
|
|
596
|
+
```ts
|
|
597
|
+
const JsonValue: Schema.declare<Json>
|
|
598
|
+
```
|
|
599
|
+
|
|
600
|
+
The strict JSON value accepted as durable attempt data: plain objects and arrays
|
|
601
|
+
of finite numbers, well-formed strings, booleans, and null. Accessors, `toJSON`,
|
|
602
|
+
non-plain prototypes, cycles, sparse arrays, enumerable symbols, and ill-formed
|
|
603
|
+
text are refused. The store still takes an inert snapshot at effect start; this
|
|
604
|
+
schema is the declaration contract the other schemas share.
|
|
605
|
+
|
|
606
|
+
#### Options
|
|
607
|
+
|
|
608
|
+
```ts
|
|
609
|
+
interface Options {
|
|
610
|
+
readonly inProgressStates?: ReadonlyArray<string> | undefined
|
|
611
|
+
readonly maxCheckpointBytes?: number | undefined
|
|
612
|
+
readonly putMode?: "insert" | "upsert" | undefined
|
|
613
|
+
}
|
|
614
|
+
```
|
|
615
|
+
|
|
616
|
+
| Field | Default | Meaning |
|
|
617
|
+
| -------------------- | ------------- | ------------------------------------------------------------------------------------------------- |
|
|
618
|
+
| `inProgressStates` | `["running"]` | The states meaning "still moving". `heartbeat` and `finish` fence on membership. |
|
|
619
|
+
| `maxCheckpointBytes` | 1 MiB | Largest encoded checkpoint. Must be between 1 and `maximumCheckpointBytes`. |
|
|
620
|
+
| `putMode` | `"insert"` | `"insert"` is first-writer-wins. `"upsert"` overwrites an in-progress row and reports `Upserted`. |
|
|
621
|
+
|
|
622
|
+
All three are validated, detached, and frozen when the store is built, so an
|
|
623
|
+
invalid policy fails at composition time.
|
|
624
|
+
|
|
625
|
+
### AttemptStore errors
|
|
626
|
+
|
|
627
|
+
```ts
|
|
628
|
+
const AttemptStoreErrorCode: Schema.Literals<
|
|
629
|
+
["invalid_attempt", "constraint", "decode_failed", "persistence_failed", "unknown"]
|
|
630
|
+
>
|
|
631
|
+
|
|
632
|
+
class AttemptStoreError extends Schema.TaggedError<AttemptStoreError>()(
|
|
633
|
+
"@smthrs/run-store/AttemptStoreError",
|
|
634
|
+
{ code: AttemptStoreErrorCode; method: Schema.String; message: Schema.String; cause?: Schema.Unknown }
|
|
635
|
+
)
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
`unknown` is what `layerNoop` reports.
|
|
639
|
+
|
|
640
|
+
### AttemptStore constants
|
|
641
|
+
|
|
642
|
+
| Constant | Value | Bounds |
|
|
643
|
+
| ------------------------ | ------- | -------------------------------------------------------- |
|
|
644
|
+
| `maximumCheckpointBytes` | 16 MiB | The ceiling `Options.maxCheckpointBytes` may not exceed. |
|
|
645
|
+
| `maximumJsonDepth` | 128 | Every attempt value. |
|
|
646
|
+
| `maximumJsonNodes` | 100,000 | Every attempt value. |
|
|
647
|
+
|
|
648
|
+
Metadata, errors, and outcomes have no byte ceiling. Only the checkpoint does,
|
|
649
|
+
because a step that cannot checkpoint still runs.
|
|
650
|
+
|
|
651
|
+
### AttemptStore constructors and layers
|
|
652
|
+
|
|
653
|
+
```ts
|
|
654
|
+
const makeWith: (options?: Options) => Effect<Service, AttemptStoreError, DurableWriter | SqlClient.SqlClient>
|
|
655
|
+
const make: Effect<Service, never, DurableWriter | SqlClient.SqlClient>
|
|
656
|
+
const makeNoop: (overrides?: Partial<Service>) => Service
|
|
657
|
+
const layer: Layer<AttemptStore, never, DurableWriter | SqlClient.SqlClient>
|
|
658
|
+
const layerWith: (options: Options) => Layer<AttemptStore, AttemptStoreError, DurableWriter | SqlClient.SqlClient>
|
|
659
|
+
const layerNoop: (overrides?: Partial<Service>) => Layer<AttemptStore>
|
|
660
|
+
```
|
|
661
|
+
|
|
662
|
+
`make` is `makeWith()` with the defaults, whose validation cannot fail.
|
|
663
|
+
|
|
664
|
+
## Ownership
|
|
665
|
+
|
|
666
|
+
Liveness evidence, the checks that produce it, and the heartbeat supervision loop.
|
|
667
|
+
|
|
668
|
+
### OwnerId
|
|
669
|
+
|
|
670
|
+
```ts
|
|
671
|
+
interface OwnerId {
|
|
672
|
+
readonly hostId: string
|
|
673
|
+
readonly pid: number
|
|
674
|
+
readonly nonce: string
|
|
675
|
+
}
|
|
676
|
+
```
|
|
677
|
+
|
|
678
|
+
Defined by [`@smthrs/journal`](/api/journal), because it is the same token the
|
|
679
|
+
journal accepts on durable appends, and re-exported here so ownership callers
|
|
680
|
+
read one vocabulary. All three fields are compared in the same SQL statement as
|
|
681
|
+
every owned mutation.
|
|
682
|
+
|
|
683
|
+
### LivenessEvidence
|
|
684
|
+
|
|
685
|
+
```ts
|
|
686
|
+
const LivenessEvidence: Schema.Struct<{
|
|
687
|
+
expectedOwner: OwnerId
|
|
688
|
+
checkedAtMs: Schema.Number
|
|
689
|
+
kind: Schema.Literals<["same-host-pid-dead", "cross-host-unreachable-stale", "lease-expired"]>
|
|
690
|
+
}>
|
|
691
|
+
```
|
|
692
|
+
|
|
693
|
+
| Kind | Accepted when |
|
|
694
|
+
| ------------------------------ | ---------------------------------------------------------------------------------- |
|
|
695
|
+
| `same-host-pid-dead` | The observer and the recorded owner share a `hostId`. |
|
|
696
|
+
| `cross-host-unreachable-stale` | The hosts differ. |
|
|
697
|
+
| `lease-expired` | Any host. The strategy verifies this claim itself against the persisted heartbeat. |
|
|
698
|
+
|
|
699
|
+
Defined by [`@smthrs/journal`](/api/journal)'s `Consensus`, because steal
|
|
700
|
+
requires staleness plus liveness evidence in every strategy, and re-exported
|
|
701
|
+
here. `checkedAtMs` must equal the consuming call's `nowMs` exactly, so a probe
|
|
702
|
+
cannot be replayed into a later decision.
|
|
703
|
+
|
|
704
|
+
### LivenessProbe, LivenessContext, LivenessCheck
|
|
705
|
+
|
|
706
|
+
```ts
|
|
707
|
+
type LivenessProbe<E = never, R = never> = (
|
|
708
|
+
expectedOwner: OwnerId,
|
|
709
|
+
claimant: OwnerId,
|
|
710
|
+
checkedAtMs: number
|
|
711
|
+
) => Effect<LivenessEvidence | undefined, E, R>
|
|
712
|
+
|
|
713
|
+
interface LivenessContext {
|
|
714
|
+
readonly claimant: OwnerId
|
|
715
|
+
readonly heartbeatAtMs: number | null
|
|
716
|
+
readonly nowMs: number
|
|
717
|
+
}
|
|
718
|
+
|
|
719
|
+
type LivenessCheck = (expectedOwner: OwnerId, context: LivenessContext) => Effect<boolean>
|
|
720
|
+
```
|
|
721
|
+
|
|
722
|
+
A `LivenessCheck` answers "is the recorded owner still working?", and answering
|
|
723
|
+
`true` refuses the takeover. A `LivenessProbe` is the evidence factory the store's
|
|
724
|
+
evidence-taking operations consume.
|
|
725
|
+
|
|
726
|
+
### sameHostIncarnation
|
|
727
|
+
|
|
728
|
+
```ts
|
|
729
|
+
const sameHostIncarnation: (expectedOwner: OwnerId, claimant: OwnerId) => boolean
|
|
730
|
+
```
|
|
731
|
+
|
|
732
|
+
Whether two identities are incarnations on the same host. The predicate a probe
|
|
733
|
+
applies before it inspects a pid.
|
|
734
|
+
|
|
735
|
+
### leaseLiveness
|
|
736
|
+
|
|
737
|
+
```ts
|
|
738
|
+
const leaseLiveness: (staleAfter?: Duration.Input) => LivenessCheck
|
|
739
|
+
```
|
|
740
|
+
|
|
741
|
+
The default check and the honest floor: an owner is alive while its persisted
|
|
742
|
+
heartbeat is younger than `staleAfter`, which defaults to `heartbeatStaleAfter`.
|
|
743
|
+
An owner with no recorded heartbeat is reported gone. Browser compositions keep
|
|
744
|
+
it, because a tab has no process table to ask.
|
|
745
|
+
|
|
746
|
+
### sameHostPidProbe
|
|
747
|
+
|
|
748
|
+
```ts
|
|
749
|
+
const sameHostPidProbe: LivenessCheck
|
|
750
|
+
```
|
|
751
|
+
|
|
752
|
+
Asks this machine's process table with `process.kill(pid, 0)`, which sends no
|
|
753
|
+
signal. Only `ESRCH` is read as death; `EPERM`, any other error, and a pid that
|
|
754
|
+
is not a positive safe integer are read as life. A recorded owner on another host
|
|
755
|
+
is never probed and answers `false`, so the expired lease decides instead of
|
|
756
|
+
stranding that host's runs forever. Node hosts only.
|
|
757
|
+
|
|
758
|
+
### heartbeatLoop
|
|
759
|
+
|
|
760
|
+
```ts
|
|
761
|
+
const heartbeatLoop: (
|
|
762
|
+
runId: string,
|
|
763
|
+
owner: OwnerId,
|
|
764
|
+
options?: HeartbeatLoopOptions
|
|
765
|
+
) => Effect<never, never, RunStore>
|
|
766
|
+
```
|
|
767
|
+
|
|
768
|
+
Pulses every `heartbeatInterval` on the injected `Clock`, each pulse renewing
|
|
769
|
+
the `Consensus` strategy's lease through `RunStore.heartbeat`, and interrupts
|
|
770
|
+
itself when the fence is gone, so race it against the owned work with
|
|
771
|
+
`Effect.raceFirst`. A heartbeat outcome other than `Updated` is durable evidence
|
|
772
|
+
and interrupts immediately. An independent deadline bounds failing or stalled
|
|
773
|
+
writes by `heartbeatWriteTolerance`. At expiry it tries `RunStore.reconfirm`
|
|
774
|
+
for at most one `heartbeatInterval`: the same owner with no pending claim renews
|
|
775
|
+
the lease. Its `onReconfirm` receipt must finish within the remaining renewed
|
|
776
|
+
`heartbeatWriteTolerance` budget, measured from the timestamp sent to the store;
|
|
777
|
+
loss, error, or timeout interrupts.
|
|
778
|
+
Successful pulses re-arm that deadline from the timestamp supplied to the store,
|
|
779
|
+
not their completion time.
|
|
780
|
+
|
|
781
|
+
`RunStore.reconfirm` uses `Consensus.reconfirm` and mirrors the heartbeat
|
|
782
|
+
transactionally, with the same input validation and outcome type as `heartbeat`.
|
|
783
|
+
`options.onReconfirm(unconfirmedMs)` records each successful reconfirmation;
|
|
784
|
+
it leaves the owner and generation unchanged.
|
|
785
|
+
|
|
786
|
+
`options.onLapse(unconfirmedMs)` runs once when the lease lapses, before the
|
|
787
|
+
self-interrupt, with how long the lease went unconfirmed. A lost fence does not
|
|
788
|
+
call it. The engine's run driver uses it to record a `lease-lapsed` release cause.
|
|
789
|
+
|
|
790
|
+
### The heartbeat constants
|
|
791
|
+
|
|
792
|
+
`heartbeatInterval`, `heartbeatStaleAfter`, `heartbeatSkewAllowance`, and
|
|
793
|
+
`heartbeatWriteTolerance` are re-exported from `Ownership`. See
|
|
794
|
+
[Heartbeat](#heartbeat).
|
|
795
|
+
|
|
796
|
+
## RunStoreMetrics
|
|
797
|
+
|
|
798
|
+
Metric handles only; `RunStore` updates them as it decides each outcome, and the
|
|
799
|
+
counters appear in whatever registry the composition provides.
|
|
800
|
+
|
|
801
|
+
```ts
|
|
802
|
+
const claims: Metric.Metric<number, Metric.CounterState<number>> // "flows_run_claims"
|
|
803
|
+
const transitions: Metric.Metric<number, Metric.CounterState<number>> // "flows_run_transitions"
|
|
804
|
+
const heartbeats: Metric.Metric<number, Metric.CounterState<number>> // "flows_run_heartbeats"
|
|
805
|
+
```
|
|
806
|
+
|
|
807
|
+
Eight records map an outcome `_tag` to an attributed view of its counter, so an
|
|
808
|
+
update or a read is a lookup rather than a branch:
|
|
809
|
+
|
|
810
|
+
| Record | Counter | Keys |
|
|
811
|
+
| -------------- | ------------- | --------------------------------------------------------------------------------------------------- |
|
|
812
|
+
| `claim` | `claims` | `Claimed`, `NotFound`, `AlreadyClaimed`, `HeartbeatFresh`, `SnapshotChanged` |
|
|
813
|
+
| `claimAndOwn` | `claims` | the above with `Activated` for `Claimed`, plus `EvidenceRequired` |
|
|
814
|
+
| `activate` | `claims` | `Activated`, `ClaimLost`, `SnapshotChanged` |
|
|
815
|
+
| `abandonClaim` | `claims` | `Abandoned`, `ClaimLost` |
|
|
816
|
+
| `recoverClaim` | `claims` | `Recovered`, `NotFound`, `ClaimFresh`, `ClaimChanged`, `LivenessUnconfirmed` |
|
|
817
|
+
| `steal` | `claims` | `Claimed`, `NotFound`, `AlreadyClaimed`, `HeartbeatFresh`, `SnapshotChanged`, `LivenessUnconfirmed` |
|
|
818
|
+
| `heartbeat` | `heartbeats` | `Updated`, `FenceLost`, `NotFound` |
|
|
819
|
+
| `transition` | `transitions` | `Transitioned`, `FenceLost`, `NotFound`, `GuardFailed` |
|
|
820
|
+
|
|
821
|
+
Each view carries `outcome` in snake case, and the `claims` views also carry `op`
|
|
822
|
+
(`claim`, `claim_and_own`, `activate`, `abandon_claim`, `recover_claim`,
|
|
823
|
+
`steal`). `RunStore` adds the `to` attribute to a transition at the update site,
|
|
824
|
+
because the target status is call input. A terminal transition also advances
|
|
825
|
+
`runThroughput` from [`@smthrs/observability`](/api/observability). See
|
|
826
|
+
[Observe store outcomes](./guides/observe-outcomes.md).
|
|
827
|
+
|
|
828
|
+
## Migrations
|
|
829
|
+
|
|
830
|
+
```ts
|
|
831
|
+
const set: DatabaseMigrations.MigrationSet
|
|
832
|
+
const run: Effect<ReadonlyArray<readonly [id: number, name: string]>, MigrationError | SqlError, SqlClient.SqlClient>
|
|
833
|
+
const layer: Layer<never, MigrationError | SqlError, SqlClient.SqlClient>
|
|
834
|
+
```
|
|
835
|
+
|
|
836
|
+
`set` owns `flows_runs`, `flows_attempts`, `flows_run_source`, and
|
|
837
|
+
`flows_run_changes` under the namespace `run-store` and
|
|
838
|
+
reserves migration id block 1000, so its ids can never collide with another
|
|
839
|
+
package's. `run` and `layer` install [`@smthrs/journal`](/api/journal)'s set
|
|
840
|
+
ahead of it, because the SQL `RunStore` arbitrates ownership through the
|
|
841
|
+
journal's `SqlConsensus` and its `flows_consensus_leases` table. Compose `set`
|
|
842
|
+
with the other storage packages' sets and run them in one pass rather than
|
|
843
|
+
layering several migrators; [`@smthrs/engine-store`](/api/engine-store)
|
|
844
|
+
already does.
|
|
845
|
+
|
|
846
|
+
The schema enforces the ownership invariants as SQL `CHECK` constraints, so no
|
|
847
|
+
writer, including one issuing raw SQL, can leave a half-owned row behind.
|
|
848
|
+
|
|
849
|
+
Migration 1003 adds a per-database source identity, monotonic revision triggers,
|
|
850
|
+
and cancellation acknowledgement. Every run-row mutation advances the revision
|
|
851
|
+
in its transaction. Existing rows receive baseline revisions; earlier mutation
|
|
852
|
+
order and acknowledgement are unknown. Deleted run IDs remain permanent
|
|
853
|
+
tombstones. See [Execution revisions](./concepts/execution-revisions.md).
|
|
854
|
+
|
|
855
|
+
Migration 1004 (`0004_waiting_request`) adds nullable `flows_runs.waiting_request`
|
|
856
|
+
as JSON text, enforced by `json_valid`. Existing rows read as having no declared
|
|
857
|
+
request. The durable engine writes and clears it with the other waiting fields;
|
|
858
|
+
human tasks use it to publish the prompt, answer kind, and attempt budget.
|
|
859
|
+
|
|
860
|
+
## Heartbeat
|
|
861
|
+
|
|
862
|
+
`@smthrs/run-store/Heartbeat` is a leaf module re-exporting the four lease
|
|
863
|
+
durations from [`@smthrs/journal`](/api/journal)'s `Consensus`, where every
|
|
864
|
+
strategy judges staleness against them. `RunStore` needs the staleness cutoff
|
|
865
|
+
and `Ownership` needs all four, and `Ownership` imports `RunStore`, so neither
|
|
866
|
+
could own them without the other restating them.
|
|
867
|
+
|
|
868
|
+
| Constant | Value | What it governs |
|
|
869
|
+
| ------------------------- | ---------- | ----------------------------------------------------------------------------------- |
|
|
870
|
+
| `heartbeatInterval` | 1 second | How often the supervision loop pulses. |
|
|
871
|
+
| `heartbeatStaleAfter` | 30 seconds | How old a persisted heartbeat must be before a peer may steal the run. |
|
|
872
|
+
| `heartbeatSkewAllowance` | 10 seconds | How far the owner's wall clock may lag a peer's before the reasoning stops holding. |
|
|
873
|
+
| `heartbeatWriteTolerance` | 19 seconds | How long an owner may keep working through failing heartbeat writes. |
|
|
874
|
+
|
|
875
|
+
All four are `Duration.Duration`. The last is derived rather than chosen:
|
|
876
|
+
`heartbeatStaleAfter - heartbeatSkewAllowance - heartbeatInterval`, so an owner
|
|
877
|
+
interrupts itself before a peer may steal the run. See
|
|
878
|
+
[The heartbeat lease](./concepts/leases.md).
|
|
879
|
+
|
|
880
|
+
## TestRunStore
|
|
881
|
+
|
|
882
|
+
The module exports one layer, which provides both services at once:
|
|
883
|
+
|
|
884
|
+
```ts
|
|
885
|
+
import * as TestRunStore from "@smthrs/run-store/test/TestRunStore"
|
|
886
|
+
import * as Effect from "effect/Effect"
|
|
887
|
+
|
|
888
|
+
const tested = program.pipe(Effect.provide(TestRunStore.layer), Effect.scoped)
|
|
889
|
+
```
|
|
890
|
+
|
|
891
|
+
`@smthrs/run-store/test/TestRunStore` provides the production run and attempt
|
|
892
|
+
stores over a fresh in-memory SQLite database, with the migrations applied before
|
|
893
|
+
either service is exposed. Node only, and scoped: the database closes with the
|
|
894
|
+
scope. See [Test against the real stores](./guides/testing.md).
|
|
895
|
+
|
|
896
|
+
## Related reading
|
|
897
|
+
|
|
898
|
+
- [Fencing and ownership](./concepts/fencing.md) for why competition is a value.
|
|
899
|
+
- [Durable values](./concepts/durable-values.md) for the admission boundary.
|
|
900
|
+
- [`@smthrs/journal`](/api/journal) for the history half of durability, and
|
|
901
|
+
[`@smthrs/engine-store`](/api/engine-store) for the composed storage ladder.
|