@marianmeres/workflow 1.0.0 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +207 -94
- package/API.md +277 -157
- package/README.md +182 -83
- package/dist/correlator.d.ts +31 -14
- package/dist/correlator.js +130 -73
- package/dist/definition.d.ts +5 -1
- package/dist/definition.js +54 -1
- package/dist/driver.d.ts +28 -23
- package/dist/driver.js +223 -59
- package/dist/migrations/1_1_0.d.ts +11 -0
- package/dist/migrations/1_1_0.js +61 -0
- package/dist/migrations/1_2_0.d.ts +16 -0
- package/dist/migrations/1_2_0.js +44 -0
- package/dist/migrations/index.js +5 -1
- package/dist/mod.d.ts +8 -5
- package/dist/mod.js +7 -4
- package/dist/persistence/history.d.ts +1 -1
- package/dist/persistence/history.js +3 -3
- package/dist/persistence/inbox.d.ts +9 -3
- package/dist/persistence/inbox.js +16 -7
- package/dist/persistence/instances.d.ts +46 -13
- package/dist/persistence/instances.js +78 -33
- package/dist/scheduler.d.ts +33 -9
- package/dist/scheduler.js +45 -21
- package/dist/types.d.ts +126 -13
- package/dist/types.js +28 -2
- package/dist/workflow.d.ts +76 -4
- package/dist/workflow.js +237 -35
- package/docs/_archive/analysis/00-overview-and-roadmap.md +233 -0
- package/docs/_archive/analysis/01-durability.md +404 -0
- package/docs/_archive/analysis/02-correlation.md +228 -0
- package/docs/_archive/analysis/03-definition-and-api.md +248 -0
- package/docs/_archive/analysis/04-docs-tests-ops.md +191 -0
- package/docs/_archive/analysis/PROGRESS.md +172 -0
- package/docs/_archive/analysis/README.md +38 -0
- package/package.json +8 -7
|
@@ -0,0 +1,404 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
GENERATED ANALYSIS — @marianmeres/workflow (durability & idempotency)
|
|
3
|
+
Produced 2026-08-26 by an inline single-agent review: full read of src/, tests/, docs, and the
|
|
4
|
+
resolved sources of @marianmeres/fsm 3.1.0, steve 3.0.0, cron 3.2.0 → self-verification pass
|
|
5
|
+
re-opening every cited line. Claims verified against the codebase at commit 02a7635.
|
|
6
|
+
Planning artifact; no code was changed.
|
|
7
|
+
-->
|
|
8
|
+
|
|
9
|
+
# Durability & Idempotency — the driver, steve, and the crash windows
|
|
10
|
+
|
|
11
|
+
> This document is about the contract between the driver and `@marianmeres/steve`. Steve
|
|
12
|
+
> delivers jobs **at least once**: it re-runs a job whose handler already committed if the
|
|
13
|
+
> completion write fails (`_execute.ts:44–53` → `_handle-failure.ts:59–68`), and it never
|
|
14
|
+
> retries a job whose worker crashed — `cleanup()` marks such jobs `expired`, which is
|
|
15
|
+
> terminal by design (`_mark-expired.ts:6–8`), and only when the consumer turned
|
|
16
|
+
> `autoCleanup` on. The driver, however, is written as if delivery were exactly-once.
|
|
17
|
+
|
|
18
|
+
> The single most important takeaway: **there is no fencing token.** Nothing on an advance
|
|
19
|
+
> or effect job says which version of the instance row it was issued against, so a
|
|
20
|
+
> duplicate or stale job is not ignored — it is applied. A stale outcome is rejected by the
|
|
21
|
+
> FSM and marks a healthy instance `failed`; a replayed initial advance dispatches the same
|
|
22
|
+
> effect twice; an expired effect job from a step the instance already left marks the
|
|
23
|
+
> instance `failed`. Separately, the scheduler's claim-then-enqueue and `create()`'s
|
|
24
|
+
> insert-then-enqueue have windows that leave rows `pending` with no job that will ever
|
|
25
|
+
> move them, and `workflow.advance` failures are not observed at all.
|
|
26
|
+
|
|
27
|
+
> Headline recommendation: add a per-instance sequence number (`seq`) that every advance
|
|
28
|
+
> bumps and every job carries as `expected_seq`; make the scheduler and correlator ticks
|
|
29
|
+
> read-only "pokes" that the advance re-validates under the row lock; observe advance-job
|
|
30
|
+
> failures and re-dispatch `expired` jobs (bounded). With those three, every crash window
|
|
31
|
+
> collapses to "the next tick pokes again" and every duplicate collapses to a no-op. A
|
|
32
|
+
> `cancel()`/`retry()` pair then falls out almost for free.
|
|
33
|
+
|
|
34
|
+
> What is fine and needs no change: the per-advance transaction with `SELECT … FOR UPDATE`
|
|
35
|
+
> (`driver.ts:104–105`), the terminal-state guard (`:111–120`), the `fromSnapshot` resume
|
|
36
|
+
> (no `onEnter` on restore, `fsm.ts:1005–1048`), and `FOR UPDATE SKIP LOCKED` batch claiming.
|
|
37
|
+
|
|
38
|
+
## Summary of recommendations
|
|
39
|
+
|
|
40
|
+
| # | Recommendation | Value | Effort | Risk |
|
|
41
|
+
| - | -------------------------------------------------------------------------------------------- | ----- | ------ | ------------------------------------------ |
|
|
42
|
+
| 1 | Fencing token `seq` on the instance; `expected_seq` + `kind` on every advance; fenced effect | high | M | med — migration + payload shape (shimmed) |
|
|
43
|
+
| 2 | Ticks become read-only pokes; stale-`pending` re-poke; `create()` inside a transaction | high | S–M | low — behavior-preserving on healthy paths |
|
|
44
|
+
| 3 | Observe `workflow.advance` failures; bounded re-dispatch of `expired` jobs | high | S | low |
|
|
45
|
+
| 4 | `cancel()` / `retry()` admin API | med | S | low — additive |
|
|
46
|
+
|
|
47
|
+
## Findings & recommendations (detailed)
|
|
48
|
+
|
|
49
|
+
### 1. No fencing token: duplicate or late jobs corrupt instances
|
|
50
|
+
|
|
51
|
+
- **Problem / observation.** Three concrete paths, all reachable in production, none
|
|
52
|
+
reachable in the current test suite:
|
|
53
|
+
|
|
54
|
+
**(a) The documented crash-after-enqueue scenario fails a healthy instance.**
|
|
55
|
+
`AGENTS.md:232` describes it as recoverable: the effect handler succeeds, `runEffect`
|
|
56
|
+
enqueues the completion advance (`driver.ts:375–380`), the process dies before steve
|
|
57
|
+
writes the job's completion. On restart the advance runs and the instance moves on — say
|
|
58
|
+
to `running` at `classify_reply`. Five minutes later `jobs.cleanup()` (via
|
|
59
|
+
`autoCleanup`) marks the old effect job `expired` and publishes `onDone`
|
|
60
|
+
(`jobs.ts:920–922`); the workflow's listener (`workflow.ts:131–144`) calls
|
|
61
|
+
`failEffectJob` (`driver.ts:389–415`), which checks only that the row is non-terminal
|
|
62
|
+
and marks it `failed` with `effect_failed { handler: "sendOrderEmail" }`. A stale job
|
|
63
|
+
from a step the instance already left kills it.
|
|
64
|
+
|
|
65
|
+
**(b) A re-run effect produces a second completion advance, which the FSM rejects.**
|
|
66
|
+
If steve's completion write fails after the handler returned, `_executeJob` falls into
|
|
67
|
+
its `catch` and `_handleJobFailure` puts the job back to `pending`
|
|
68
|
+
(`_execute.ts:51–53`, `_handle-failure.ts:59–68`). The handler runs again (idempotency is
|
|
69
|
+
documented, fine) and enqueues **a second advance with the same outcome**. `runAdvance`
|
|
70
|
+
applies it at the _new_ cursor; `fsm.transition` returns `null`
|
|
71
|
+
(`driver.ts:146–150`), so the instance gets `transition_rejected` and `failed`
|
|
72
|
+
(`:151–165`). The same re-run of the _initial_ advance is worse: the row is already
|
|
73
|
+
`running`, the dispatch loop lands on the effectful case again (`:211–233`) and enqueues
|
|
74
|
+
a second effect job for the same node — the handler runs twice, both completions
|
|
75
|
+
advance, the second is rejected.
|
|
76
|
+
|
|
77
|
+
**(c) `runEffect` runs the handler unconditionally.** It reads the row without a lock and
|
|
78
|
+
without looking at `execution_state` or the cursor (`driver.ts:349–373`). After a
|
|
79
|
+
future `cancel()` (#4), or on any stale dispatch from (b), the side effect still fires.
|
|
80
|
+
|
|
81
|
+
Also latent: an advance _without_ an outcome at a `waiting` cursor recomputes `wake_at`
|
|
82
|
+
(`driver.ts:235–246`) — i.e. resets the timer. Today only a duplicate can trigger it; once
|
|
83
|
+
ticks re-poke (#2) it becomes a real hazard unless advances carry a precondition.
|
|
84
|
+
|
|
85
|
+
The comment at `workflow.ts:152–154` — "The advance is idempotent (cursor-aware) so an
|
|
86
|
+
enqueue-but-no-commit race is recoverable" — is the assumption this finding refutes:
|
|
87
|
+
nothing compares a job's origin against the row it lands on.
|
|
88
|
+
|
|
89
|
+
- **Evidence.** `driver.ts:389–415`:
|
|
90
|
+
```ts
|
|
91
|
+
export async function failEffectJob(pool, payload, reason) {
|
|
92
|
+
await withTransaction(pool, async (client) => {
|
|
93
|
+
const row = await lockInstance(client, payload.instance_id);
|
|
94
|
+
if (!row) return;
|
|
95
|
+
if (row.execution_state === COMPLETED || FAILED || CANCELLED) return;
|
|
96
|
+
await updateInstance(client, row.id, { execution_state: EXECUTION_STATE.FAILED });
|
|
97
|
+
// ... history effect_failed
|
|
98
|
+
```
|
|
99
|
+
`driver.ts:349–373` (`runEffect`): plain `SELECT … WHERE id = $1`, then `handler(...)` —
|
|
100
|
+
no state check. `types.ts:151–169`: `AdvanceJobPayload` carries `outcome`, `outcome_data`,
|
|
101
|
+
`timeout?: boolean`; `EffectJobPayload` carries `handler` — neither carries anything that
|
|
102
|
+
identifies the row version they were issued against.
|
|
103
|
+
|
|
104
|
+
- **Proposed change.**
|
|
105
|
+
1. Migration `1.2.0` (append-only ledger, `src/migrations/1_2_0.ts`):
|
|
106
|
+
```sql
|
|
107
|
+
ALTER TABLE __workflow_instances ADD COLUMN IF NOT EXISTS seq integer NOT NULL DEFAULT 0;
|
|
108
|
+
-- down: ALTER TABLE __workflow_instances DROP COLUMN IF EXISTS seq;
|
|
109
|
+
```
|
|
110
|
+
(`DEFAULT 0` on a non-volatile default is a metadata-only change on PG ≥ 11.)
|
|
111
|
+
2. `WorkflowInstanceRow.seq: number`. Every row write issued by `runAdvance`,
|
|
112
|
+
`failInstance`, and (later) `cancel`/`retry` bumps `seq = seq + 1` in the same
|
|
113
|
+
`UPDATE` as the state change — one settle-point write per advance, so a single
|
|
114
|
+
`bumpSeq: true` flag on `updateInstance` (or a dedicated `advanceInstance` helper) is
|
|
115
|
+
enough.
|
|
116
|
+
3. Payload shapes (`types.ts`):
|
|
117
|
+
```ts
|
|
118
|
+
export type AdvanceKind = "start" | "effect" | "timeout" | "signal";
|
|
119
|
+
export interface AdvanceJobPayload {
|
|
120
|
+
tenant_id: string;
|
|
121
|
+
instance_id: string;
|
|
122
|
+
kind: AdvanceKind; // replaces `timeout?: boolean`
|
|
123
|
+
expected_seq: number; // fence
|
|
124
|
+
outcome?: string; // kind=effect (handler's label)
|
|
125
|
+
outcome_data?: Record<string, unknown>;
|
|
126
|
+
handler?: string; // kind=effect, for history
|
|
127
|
+
inbox_id?: string; // kind=signal (see 02 #1)
|
|
128
|
+
redispatch?: number; // see #3
|
|
129
|
+
}
|
|
130
|
+
export interface EffectJobPayload {
|
|
131
|
+
tenant_id: string;
|
|
132
|
+
instance_id: string;
|
|
133
|
+
handler: string;
|
|
134
|
+
seq: number; // fence
|
|
135
|
+
cursor: string; // for history/diagnostics
|
|
136
|
+
redispatch?: number;
|
|
137
|
+
}
|
|
138
|
+
```
|
|
139
|
+
4. `runAdvance`, inside the row lock, after the terminal check:
|
|
140
|
+
```ts
|
|
141
|
+
if (payload.expected_seq !== undefined && row.seq !== payload.expected_seq) {
|
|
142
|
+
clog.debug?.(
|
|
143
|
+
`advance: stale (row.seq=${row.seq}, expected=${payload.expected_seq}); no-op`,
|
|
144
|
+
);
|
|
145
|
+
return;
|
|
146
|
+
}
|
|
147
|
+
// per-kind preconditions — violation is a debug no-op, never a failure
|
|
148
|
+
switch (payload.kind) {
|
|
149
|
+
case "start":
|
|
150
|
+
if (row.execution_state !== PENDING) return;
|
|
151
|
+
break;
|
|
152
|
+
case "effect":
|
|
153
|
+
if (row.execution_state !== RUNNING) return;
|
|
154
|
+
break;
|
|
155
|
+
case "timeout":
|
|
156
|
+
if (
|
|
157
|
+
row.execution_state !== WAITING || !row.wake_at ||
|
|
158
|
+
row.wake_at > new Date()
|
|
159
|
+
) return;
|
|
160
|
+
break;
|
|
161
|
+
case "signal":
|
|
162
|
+
if (row.execution_state !== WAITING) return;
|
|
163
|
+
break;
|
|
164
|
+
}
|
|
165
|
+
```
|
|
166
|
+
A payload without `expected_seq`/`kind` is a job queued by 2.0.x: treat it as
|
|
167
|
+
unfenced and infer `kind` (`timeout: true` → `timeout`; `outcome` present →
|
|
168
|
+
`effect`; else `start`). Same transitional status as `payloadTenantId`
|
|
169
|
+
(`driver.ts:34–38`); remove both together.
|
|
170
|
+
5. `runEffect`: `if (row.seq !== payload.seq || row.execution_state !== RUNNING) return
|
|
171
|
+
{ skipped: "stale" };` — no handler call, no advance. Completion advance carries
|
|
172
|
+
`{ kind: "effect", expected_seq: payload.seq, outcome, outcome_data, handler }`.
|
|
173
|
+
6. `failEffectJob`: `if (row.seq !== payload.seq) return;` before anything else.
|
|
174
|
+
7. `create()`: wrap insert + `created` history + enqueue in `withTransaction`. Steve's
|
|
175
|
+
insert is autocommit on its own connection (`_create.ts:21,43`), so if our commit then
|
|
176
|
+
fails the advance finds no row and no-ops (`driver.ts:106–109`); if `jobs.create`
|
|
177
|
+
throws, our insert rolls back. The residual "insert committed, enqueue never happened"
|
|
178
|
+
window is what #2's stale-`pending` re-poke covers.
|
|
179
|
+
8. Emit the currently dead `HISTORY_EVENT.EFFECT_COMPLETED` (`types.ts:118`) for
|
|
180
|
+
`kind: "effect"` advances, before the `transition` row, with `{ handler, outcome }`.
|
|
181
|
+
|
|
182
|
+
- **Done when** — `tests/durability.test.ts` (new) passes with: (1) enqueuing the initial
|
|
183
|
+
advance payload twice yields exactly one `workflow.effect.checkInventory` row in `__job`;
|
|
184
|
+
(2) enqueuing a second, identical effect-completion advance leaves the instance
|
|
185
|
+
non-failed with the cursor advanced once; (3) `failEffectJob` with a stale `seq` leaves a
|
|
186
|
+
moved-on instance untouched; and `tests/migrations.test.ts` asserts `seq` exists after
|
|
187
|
+
`up("latest")` and is absent after `down("1.1.0")`.
|
|
188
|
+
|
|
189
|
+
- **Affected files.** `src/migrations/1_2_0.ts` (new), `src/migrations/index.ts`,
|
|
190
|
+
`src/types.ts`, `src/persistence/instances.ts`, `src/driver.ts`, `src/workflow.ts`,
|
|
191
|
+
`src/scheduler.ts` + `src/correlator.ts` (payload shape only — their redesign is #2 and
|
|
192
|
+
02 #1), `tests/migrations.test.ts`, `tests/durability.test.ts` (new). Docs in 04 #1.
|
|
193
|
+
|
|
194
|
+
- **Effort / Value / Risk.** M / high / med. The migration is additive; the payload
|
|
195
|
+
change is shimmed for jobs queued before the upgrade. Consumers calling
|
|
196
|
+
`enqueueAdvance`/`enqueueEffect` directly (documented as internal, `API.md:102–104`)
|
|
197
|
+
must now pass a fenced payload.
|
|
198
|
+
|
|
199
|
+
- **Implementation notes.** Keep the fence check _inside_ the row lock — it is only
|
|
200
|
+
meaningful against the locked row. Log stale no-ops at debug and do not write a history
|
|
201
|
+
row for them: once ticks re-poke (#2) they are routine, not events. While there, drop
|
|
202
|
+
the ignored `_client` parameter from `JobEnqueuer` (`driver.ts:62–72`,
|
|
203
|
+
`workflow.ts:148–168`): steve always inserts on its own connection, so the parameter
|
|
204
|
+
only suggests an atomicity that does not exist (the README even claims it — 04 #1).
|
|
205
|
+
Column name is bikeshed-able (`seq` / `step` / `revision`); `seq` avoids reading as
|
|
206
|
+
"workflow step".
|
|
207
|
+
|
|
208
|
+
### 2. Ticks claim-then-enqueue and `create()` inserts-then-enqueues — crash windows leave rows `pending` forever
|
|
209
|
+
|
|
210
|
+
- **Problem / observation.** The scheduler tick (`scheduler.ts:87–108`) calls
|
|
211
|
+
`claimDueWakeUps` — an autocommit `UPDATE … SET execution_state = 'pending', wake_at =
|
|
212
|
+
NULL` on the pool (`instances.ts:137–161`) — and only _then_ enqueues one advance per
|
|
213
|
+
row via `jobs.create` on steve's connection. Die between the two, or let `jobs.create`
|
|
214
|
+
throw for row 3 of 5, and the affected rows are `pending` with `wake_at = NULL` and no
|
|
215
|
+
job: `claimDueWakeUps` requires `waiting`, the correlator requires `waiting`, nothing
|
|
216
|
+
will ever touch them again. `create()` (`workflow.ts:187–205`) is three autocommits;
|
|
217
|
+
die after the first and the instance is `pending` forever. Steve cannot enlist in our
|
|
218
|
+
transaction (`Jobs.create` → `_create` always uses `context.db`, `_create.ts:21,43`), so
|
|
219
|
+
"enqueue inside the tx" is not an option — although `README.md:195` claims that is what
|
|
220
|
+
happens (04 #1).
|
|
221
|
+
|
|
222
|
+
> **Cut from the draft:** the correlator was listed here too. Its flip is inside the tick
|
|
223
|
+
> transaction and the enqueue precedes the commit (`correlator.ts:198–210`), so its crash
|
|
224
|
+
> windows are benign — a rollback either undoes everything or leaves a job that applies
|
|
225
|
+
> `MATCHED` against a still-`waiting` row. Its real problems are semantic (02 #1).
|
|
226
|
+
|
|
227
|
+
- **Evidence.** `instances.ts:143–158`:
|
|
228
|
+
```sql
|
|
229
|
+
UPDATE __workflow_instances SET execution_state = 'pending', wake_at = NULL, updated_at = now()
|
|
230
|
+
WHERE id IN (SELECT id FROM __workflow_instances WHERE tenant_id = $1 AND execution_state = 'waiting'
|
|
231
|
+
AND wake_at IS NOT NULL AND wake_at <= now() ORDER BY wake_at LIMIT $4 FOR UPDATE SKIP LOCKED)
|
|
232
|
+
RETURNING id, correlation_token
|
|
233
|
+
```
|
|
234
|
+
followed at `scheduler.ts:97–105` by a `for` loop of `enqueueAdvance` calls.
|
|
235
|
+
|
|
236
|
+
- **Proposed change.** With #1's fence in place, make the tick a read-only poke and let the
|
|
237
|
+
advance do every write under the lock:
|
|
238
|
+
```ts
|
|
239
|
+
// scheduler tick — no UPDATE
|
|
240
|
+
const due = await selectDueWakeUps(db, tenantId, batch); // SELECT id, seq ... WHERE waiting AND wake_at <= now() ORDER BY wake_at LIMIT n
|
|
241
|
+
for (const r of due) {
|
|
242
|
+
await wf.enqueueAdvance({
|
|
243
|
+
tenant_id,
|
|
244
|
+
instance_id: r.id,
|
|
245
|
+
kind: "timeout",
|
|
246
|
+
expected_seq: r.seq,
|
|
247
|
+
});
|
|
248
|
+
}
|
|
249
|
+
|
|
250
|
+
const stale = await selectStalePending(db, tenantId, stalePendingSec, batch); // SELECT id, seq ... WHERE pending AND updated_at < now() - $sec
|
|
251
|
+
for (const r of stale) {
|
|
252
|
+
await wf.enqueueAdvance({
|
|
253
|
+
tenant_id,
|
|
254
|
+
instance_id: r.id,
|
|
255
|
+
kind: "start",
|
|
256
|
+
expected_seq: r.seq,
|
|
257
|
+
});
|
|
258
|
+
}
|
|
259
|
+
```
|
|
260
|
+
The advance re-checks `waiting && wake_at <= now()` (the `timeout` precondition from #1),
|
|
261
|
+
applies `TIMEOUT`, and clears `wake_at` itself. A row poked but not yet advanced is
|
|
262
|
+
poked again next tick — a fenced no-op. Delete `claimDueWakeUps`. The stale-`pending`
|
|
263
|
+
re-poke is safe _only_ because, after #1 and 02 #1, `create()` is the sole producer of
|
|
264
|
+
`pending` (hence the T02 → T03 sequencing in `PROGRESS.md`). Add a partial index in
|
|
265
|
+
migration `1.2.0` for the second scan:
|
|
266
|
+
```sql
|
|
267
|
+
CREATE INDEX IF NOT EXISTS __workflow_instances_stale_pending_idx
|
|
268
|
+
ON __workflow_instances (tenant_id, updated_at) WHERE execution_state = 'pending';
|
|
269
|
+
```
|
|
270
|
+
(`pending` rows are transient, so the index stays tiny.) New option
|
|
271
|
+
`WorkflowSchedulerOptions.stalePendingSec` (default `300`; `0` disables).
|
|
272
|
+
|
|
273
|
+
- **Done when** — tests: "two `scheduler.tickOnce()` calls for one due row apply `TIMEOUT`
|
|
274
|
+
once (exactly one `timeout` history row; cursor `_end_timeout`)"; "a `pending` row with
|
|
275
|
+
no job and `updated_at` older than `stalePendingSec` is advanced by the next tick";
|
|
276
|
+
`grep -rn claimDueWakeUps src/` returns nothing. The existing TIMEOUT test
|
|
277
|
+
(`tests/stock_replenishment.test.ts:173–174`, `assertEquals(claimed, 1)`) is adjusted to
|
|
278
|
+
the new return shape.
|
|
279
|
+
|
|
280
|
+
- **Affected files.** `src/scheduler.ts`, `src/persistence/instances.ts`, `src/driver.ts`
|
|
281
|
+
(precondition from #1), `src/migrations/1_2_0.ts` (index), `tests/stock_replenishment.test.ts`,
|
|
282
|
+
`tests/durability.test.ts`.
|
|
283
|
+
|
|
284
|
+
- **Effort / Value / Risk.** S–M / high / low. Healthy paths behave identically; only the
|
|
285
|
+
moment `wake_at` is cleared moves from the tick to the advance.
|
|
286
|
+
|
|
287
|
+
- **Implementation notes.** `tickOnce()` can return `{ woken, repoked }` (or keep a number
|
|
288
|
+
= sum). Keep `ORDER BY wake_at` and the batch size. An in-process "poked recently"
|
|
289
|
+
`Map<id, ts>` to suppress duplicate pokes between ticks is a nice-to-have, not needed for
|
|
290
|
+
correctness. The `timeout: true` flag becomes `kind: "timeout"`.
|
|
291
|
+
|
|
292
|
+
### 3. `workflow.advance` failures are unobserved; `expired` is terminal in steve and the docs say "retry"
|
|
293
|
+
|
|
294
|
+
- **Problem / observation.** `workflow.ts:131–144` subscribes `onDone` for effect types
|
|
295
|
+
only. A `workflow.advance` job that throws — a guard or `action` throwing inside
|
|
296
|
+
`fsm.transition` (fsm wraps and rethrows, `fsm.ts:414–449`), or any DB error — is
|
|
297
|
+
retried by steve three times with 2 s / 4 s backoff (`_handle-failure.ts:56`) and then
|
|
298
|
+
marked `failed`, silently. The instance stays `pending`/`running` with no job. After #2,
|
|
299
|
+
a `pending` row is re-poked every `stalePendingSec` forever (a permanently-throwing guard
|
|
300
|
+
becomes an infinite retry); a `running` row whose outcome died with the advance is
|
|
301
|
+
stuck for good.
|
|
302
|
+
|
|
303
|
+
Steve never retries a crashed job: `cleanup()` flips `running` rows older than 5 min to
|
|
304
|
+
`expired`, terminal by design (`_mark-expired.ts:6–8`), and publishes `onDone`
|
|
305
|
+
(`jobs.ts:920–922`). That reaper runs only with `autoCleanup: true`
|
|
306
|
+
(`jobs.ts:451–454`, `:697–700`) or a manual `cleanup()`; the default is off. So a worker
|
|
307
|
+
crash mid-effect means: `autoCleanup` on → instance `failed` after ~5 min via
|
|
308
|
+
`failEffectJob`; off → instance `running` forever. `AGENTS.md:231` and `README.md:196`
|
|
309
|
+
say steve "retries" — it does not. (04 #1 fixes the text; this finding fixes the
|
|
310
|
+
behavior.)
|
|
311
|
+
|
|
312
|
+
- **Evidence.** `workflow.ts:131–144`:
|
|
313
|
+
```ts
|
|
314
|
+
for (const name of handlerNames) {
|
|
315
|
+
const type = effectJobType(name);
|
|
316
|
+
this.jobs.onDone(type, (job) => {
|
|
317
|
+
if (job.status === JOB_STATUS.FAILED || job.status === JOB_STATUS.EXPIRED) {
|
|
318
|
+
failEffectJob(this.db, payload, `steve: ${job.status}`).catch(...)
|
|
319
|
+
```
|
|
320
|
+
No subscription for `JOB_TYPE_ADVANCE`. `_mark-expired.ts:6–8`: "we don't auto-retry
|
|
321
|
+
because the work may now be stale".
|
|
322
|
+
|
|
323
|
+
- **Proposed change.**
|
|
324
|
+
1. `jobs.onDone(JOB_TYPE_ADVANCE, cb)`: `failed` → `failInstance` with the last attempt's
|
|
325
|
+
`error_message` (`jobs.find(job.uid, true)` → `attempts.at(-1)?.error_message`,
|
|
326
|
+
`jobs.ts:806–824`), history `failed { reason, job_uid }`; `expired` → re-enqueue the
|
|
327
|
+
identical payload with `redispatch: (payload.redispatch ?? 0) + 1` while
|
|
328
|
+
`< redispatchLimit`, else fail. If the original advance did commit before the crash,
|
|
329
|
+
the re-dispatch is a fenced no-op (#1).
|
|
330
|
+
2. Effect types: `expired` → same bounded re-dispatch of the identical effect payload
|
|
331
|
+
(`seq` intact — a zombie handler that later completes produces a fenced duplicate;
|
|
332
|
+
handler idempotency is already required); `failed` → `failEffectJob` as today (fenced).
|
|
333
|
+
3. New options: `advanceMaxAttempts` (default `10`: exponential backoff sums to ≈17 min
|
|
334
|
+
of DB-outage tolerance before an instance is failed; today's implicit `3` gives ≈6 s)
|
|
335
|
+
passed as `max_attempts` where advance jobs are created (`workflow.ts:155`,
|
|
336
|
+
`:202–205`); `redispatchLimit` (default `3`).
|
|
337
|
+
|
|
338
|
+
- **Done when** — tests: "an advance whose transition `action` throws every time ends with
|
|
339
|
+
the instance `failed` and a `failed` history row whose `reason` contains the error
|
|
340
|
+
message"; "an effect job marked `expired` via `jobs.cleanup(0)` is re-dispatched and the
|
|
341
|
+
workflow completes"; "after `redispatchLimit` expiries the instance is `failed` with
|
|
342
|
+
`effect_failed`".
|
|
343
|
+
|
|
344
|
+
- **Affected files.** `src/workflow.ts`, `src/driver.ts` (reason plumbing into
|
|
345
|
+
`failInstance`), `src/types.ts` (`redispatch`), `API.md` (options).
|
|
346
|
+
|
|
347
|
+
- **Effort / Value / Risk.** S / high / low.
|
|
348
|
+
|
|
349
|
+
- **Implementation notes.** `jobs.cleanup(0)` reaps immediately in tests
|
|
350
|
+
(`jobs.ts:904–916`; `Math.max(0, …)` at `_mark-expired.ts:20`). Steve wraps `onDone`
|
|
351
|
+
callbacks in try/catch and only logs (`jobs.ts:1099–1105`) — keep the
|
|
352
|
+
`.catch(clog.error)` pattern. Document that `autoCleanup: true` (or a periodic
|
|
353
|
+
`cleanup()`) is what makes crash recovery happen at all; the README example should set
|
|
354
|
+
it (04 #1).
|
|
355
|
+
|
|
356
|
+
### 4. No way to cancel or retry an instance
|
|
357
|
+
|
|
358
|
+
- **Problem / observation.** `EXECUTION_STATE.CANCELLED` and `HISTORY_EVENT.CANCELLED`
|
|
359
|
+
exist (`types.ts:16`, `:126`) and the driver treats `cancelled` as terminal
|
|
360
|
+
(`driver.ts:111–120`, `:397–403`), but nothing sets it: `Workflow` exposes
|
|
361
|
+
`create`/`find`/`appendInbox`/`enqueueAdvance`/`enqueueEffect` only
|
|
362
|
+
(`workflow.ts:177–232`). An instance waiting three days for a reply cannot be aborted
|
|
363
|
+
except by SQL, and a `failed` instance (an API outage exhausted `effectMaxAttempts`)
|
|
364
|
+
cannot be resumed — the only recovery is a new instance, which re-runs every side effect
|
|
365
|
+
from the start.
|
|
366
|
+
|
|
367
|
+
- **Evidence.** `grep -rn "CANCELLED" src/` → only the two terminal checks in `driver.ts`
|
|
368
|
+
and the constant definitions; no writer.
|
|
369
|
+
|
|
370
|
+
- **Proposed change.** On `Workflow`, tenant-scoped like `find`:
|
|
371
|
+
```ts
|
|
372
|
+
/** Non-terminal → cancelled. Clears wake_at/correlation_token, bumps seq (in-flight jobs become stale). */
|
|
373
|
+
async cancel(id: string, reason?: string): Promise<boolean>;
|
|
374
|
+
/** failed → pending at the current cursor, bumps seq, enqueues { kind: "start" }. `force` also allows `running`. */
|
|
375
|
+
async retry(id: string, opts?: { force?: boolean }): Promise<boolean>;
|
|
376
|
+
```
|
|
377
|
+
Both return `false` when the state does not allow the operation. History: `cancelled
|
|
378
|
+
{ reason }`, and a new `HISTORY_EVENT.RETRIED { from_state }`. `force: true` is the
|
|
379
|
+
operator asserting the effect is dead — the `seq` bump (#1) makes any zombie effect job
|
|
380
|
+
stale, so this is safe by construction.
|
|
381
|
+
|
|
382
|
+
- **Done when** — tests: "`cancel()` on a `waiting` instance → `cancelled`; a later
|
|
383
|
+
matching signal is marked processed without delivery; `scheduler.tickOnce()` ignores
|
|
384
|
+
it"; "`cancel()` while an effect job is queued → the job completes with
|
|
385
|
+
`{ skipped: "stale" }` and the handler is never called"; "`retry()` on a `failed`
|
|
386
|
+
instance re-dispatches the cursor node's effect and the workflow completes".
|
|
387
|
+
|
|
388
|
+
- **Affected files.** `src/workflow.ts`, `src/persistence/instances.ts`, `src/types.ts`,
|
|
389
|
+
`API.md`, `README.md` (Failure Modes).
|
|
390
|
+
|
|
391
|
+
- **Effort / Value / Risk.** S / med / low. Depends on #1.
|
|
392
|
+
|
|
393
|
+
- **Implementation notes.** `retry` from a `transition_rejected` failure re-dispatches the
|
|
394
|
+
cursor node's effect and will fail again unless the handler was fixed — acceptable for a
|
|
395
|
+
manual tool. If a "force an outcome" escape hatch is ever wanted, `enqueueAdvance` with
|
|
396
|
+
`{ kind: "effect", expected_seq: row.seq, outcome }` is it; today's unfenced
|
|
397
|
+
`enqueueAdvance` is that hatch by accident, and #1 closes it deliberately.
|
|
398
|
+
|
|
399
|
+
## Open questions / decisions needed
|
|
400
|
+
|
|
401
|
+
All resolved in the owner interview of 2026-08-26 (see `PROGRESS.md` → Decisions log):
|
|
402
|
+
column name `seq`; `advanceMaxAttempts = 10` and `redispatchLimit = 3`; `expired` →
|
|
403
|
+
bounded re-dispatch of the identical payload (advance and effect), then fail;
|
|
404
|
+
stale-`pending` re-poke on by default at `300` s (`0` disables).
|
|
@@ -0,0 +1,228 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
GENERATED ANALYSIS — @marianmeres/workflow (inbox correlation)
|
|
3
|
+
Produced 2026-08-26 by an inline single-agent review: full read of src/, tests/, docs, and the
|
|
4
|
+
resolved sources of @marianmeres/fsm 3.1.0, steve 3.0.0, cron 3.2.0 → self-verification pass
|
|
5
|
+
re-opening every cited line. Claims verified against the codebase at commit 02a7635.
|
|
6
|
+
Planning artifact; no code was changed.
|
|
7
|
+
-->
|
|
8
|
+
|
|
9
|
+
# Inbox Correlation — signal delivery semantics
|
|
10
|
+
|
|
11
|
+
> The inbox and correlator are how the outside world reaches a sleeping instance. For the
|
|
12
|
+
> motivating use case — supplier email replies — this is the main event, not an edge. The
|
|
13
|
+
> split "token is the index, matcher is the gate" (`AGENTS.md:202`) is right, the
|
|
14
|
+
> append-only inbox decoupled from matching is right, and the `FOR UPDATE SKIP LOCKED`
|
|
15
|
+
> batch claim is right. The delivery _rules_ are where the problems are.
|
|
16
|
+
|
|
17
|
+
> The single most important takeaway: **the correlator only looks for instances that are
|
|
18
|
+
> already `waiting`, and it consumes every inbox row it looks at.** A reply that lands
|
|
19
|
+
> while the instance is still `running` its send step — or sitting in a timer-only delay
|
|
20
|
+
> node — is marked processed and silently thrown away; the instance then waits out its
|
|
21
|
+
> full timeout for a reply that already arrived. Worse, at a timer-only suspending node the
|
|
22
|
+
> signal _is_ delivered as `MATCHED`, which the node has no transition for, and the
|
|
23
|
+
> instance is marked `failed`. Two replies inside one tick lose the second; a matcher that
|
|
24
|
+
> throws once consumes the signal permanently.
|
|
25
|
+
|
|
26
|
+
> Headline recommendation: deliver only when the instance is `waiting` at a node that
|
|
27
|
+
> accepts `MATCHED` and the matcher agrees; otherwise leave the row **unprocessed** (early
|
|
28
|
+
> signal — try again next tick) and mark it processed only when no live instance owns the
|
|
29
|
+
> token. Move "mark processed" into the advance transaction so delivery and transition are
|
|
30
|
+
> atomic, and let effect handlers set the correlation token so a wait point can key on
|
|
31
|
+
> something that only exists after the effect ran (a `Message-ID`, a provider session id).
|
|
32
|
+
|
|
33
|
+
## Summary of recommendations
|
|
34
|
+
|
|
35
|
+
| # | Recommendation | Value | Effort | Risk |
|
|
36
|
+
| - | ------------------------------------------------------------------------------------------------------------------ | ----- | ------ | ---- |
|
|
37
|
+
| 1 | Defer early/late signals instead of dropping them; deliver only to `MATCHED`-accepting waits; processed-in-advance | high | M | low |
|
|
38
|
+
| 2 | Duplicate / concurrent signals and matcher exceptions | med | S | low |
|
|
39
|
+
| 3 | Handler-settable correlation token (per wait point) | med | S | low |
|
|
40
|
+
|
|
41
|
+
## Findings & recommendations (detailed)
|
|
42
|
+
|
|
43
|
+
### 1. Signals that arrive before the wait point are dropped; signals at timer-only waits fail the instance
|
|
44
|
+
|
|
45
|
+
- **Problem / observation.** `#processOne` looks the token up with
|
|
46
|
+
`findWaitingByCorrelation`, which filters `execution_state = 'waiting'`
|
|
47
|
+
(`instances.ts:175–177`), and on a miss marks the inbox row processed and returns
|
|
48
|
+
(`correlator.ts:130–133`). Timeline for the README flow: `sendOrderEmail` is `running`
|
|
49
|
+
(or the following advance has not yet flipped the row to `waiting`); the supplier's
|
|
50
|
+
auto-reply, or a fast human, is appended to the inbox by a webhook that has no idea where
|
|
51
|
+
the instance is; the correlator tick runs first → row processed, nothing delivered → the
|
|
52
|
+
instance reaches `await_reply` and waits three days for a reply that already came. The
|
|
53
|
+
window is the whole duration of whatever sits between "token goes out" and "instance is
|
|
54
|
+
waiting": a slow AI step, a retry with backoff, any timer-only node.
|
|
55
|
+
|
|
56
|
+
The timer-only case is not a drop but a failure: a `suspending` node without `matcher`
|
|
57
|
+
takes the `matched = true` default (`correlator.ts:144–146`), an advance with
|
|
58
|
+
`outcome: "MATCHED"` is enqueued (`:205–210`), `positioned.transition("MATCHED")`
|
|
59
|
+
returns `null` because the node's `on` has neither `MATCHED` nor `*`
|
|
60
|
+
(`fsm.ts:536–546`), and the driver writes `transition_rejected` and marks the instance
|
|
61
|
+
`failed` (`driver.ts:151–165`). Example: a "wait 1 h, then send a reminder" delay node
|
|
62
|
+
between `send_order` and `await_reply` — a reply during the hour kills the instance.
|
|
63
|
+
The validator cannot catch this and should not try (03 #1 covers `matcher` without
|
|
64
|
+
`MATCHED`): a delay node legitimately has no `MATCHED` edge.
|
|
65
|
+
|
|
66
|
+
- **Evidence.** `correlator.ts:124–133`:
|
|
67
|
+
```ts
|
|
68
|
+
async #processOne(client, row) {
|
|
69
|
+
const instance = await findWaitingByCorrelation(client, this.tenantId, row.correlation_token);
|
|
70
|
+
if (!instance) {
|
|
71
|
+
await markProcessed(client, row.id); // <-- early signal is consumed here
|
|
72
|
+
return;
|
|
73
|
+
}
|
|
74
|
+
```
|
|
75
|
+
`correlator.ts:144–146`: `let matched = true; if (meta && meta.kind === "suspending" &&
|
|
76
|
+
meta.matcher) { … }` — no check that the node accepts `MATCHED`.
|
|
77
|
+
|
|
78
|
+
- **Proposed change.** Replace the lookup and the decision table; move the processed-mark
|
|
79
|
+
into the advance.
|
|
80
|
+
1. `findByCorrelation(exec, tenant_id, token)` → the live (non-terminal) instance,
|
|
81
|
+
`ORDER BY created_at LIMIT 1`.
|
|
82
|
+
2. Decision table in the tick (inside the existing batch transaction):
|
|
83
|
+
|
|
84
|
+
| Instance | Action |
|
|
85
|
+
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
|
|
86
|
+
| none (unknown token) / terminal | mark processed; `clog.warn` (unknown) or `signal_rejected { reason }` |
|
|
87
|
+
| live but not `waiting` | **defer** — leave unprocessed, debug log |
|
|
88
|
+
| `waiting`, node lacks `MATCHED` and `*` | **defer** |
|
|
89
|
+
| `waiting`, accepts `MATCHED`, matcher → `false` | `signal_rejected` history; mark processed |
|
|
90
|
+
| `waiting`, accepts `MATCHED`, matcher → `true` | enqueue `{ kind: "signal", inbox_id, expected_seq: instance.seq }` — do **not** flip, do **not** mark processed |
|
|
91
|
+
|
|
92
|
+
"Accepts `MATCHED`" = `"MATCHED" in stateCfg.on || "*" in stateCfg.on` (resolution
|
|
93
|
+
order per `fsm.ts:510–534`).
|
|
94
|
+
3. `runAdvance`, `kind: "signal"` (01 #1 gives the fence and the `waiting` precondition):
|
|
95
|
+
```ts
|
|
96
|
+
const inbox = await lockInboxRow(client, payload.inbox_id!); // SELECT ... FOR UPDATE
|
|
97
|
+
if (!inbox || inbox.processed_at) return; // delivered by a duplicate poke
|
|
98
|
+
const result = positioned.transition("MATCHED", inbox.payload, false);
|
|
99
|
+
// ... rejected → transition_rejected + failInstance (unchanged)
|
|
100
|
+
await appendHistory(client, {
|
|
101
|
+
event_type: SIGNAL_RECEIVED,
|
|
102
|
+
from_node,
|
|
103
|
+
data: { inbox_id, source },
|
|
104
|
+
});
|
|
105
|
+
await markProcessed(client, inbox.id); // same transaction as the transition
|
|
106
|
+
```
|
|
107
|
+
4. Deferred rows are re-examined every tick. Starvation note: `claimUnprocessed` orders
|
|
108
|
+
by `received_at` with a `LIMIT` (`inbox.ts:49–55`); a backlog of deferred rows ≥
|
|
109
|
+
`tickBatchSize` would shadow newer tokens. At the stated scale that is a non-issue;
|
|
110
|
+
document it, and keep an `inboxMaxAgeSec` knob ("rows older than N seconds are marked
|
|
111
|
+
processed with `signal_expired`") as a later addition if it ever bites.
|
|
112
|
+
|
|
113
|
+
- **Done when** — `tests/correlator.test.ts` (new): (a) append the signal _before_ the
|
|
114
|
+
instance reaches `await_reply` (gate the `sendOrderEmail` handler on a promise), tick →
|
|
115
|
+
the row is still unprocessed and the instance is not failed; release the handler, wait
|
|
116
|
+
for `waiting`, tick → delivered, cursor `_end_ok`; (b) a fixture variant with a
|
|
117
|
+
timer-only `cooldown` node before `await_reply`: signal during cooldown → deferred,
|
|
118
|
+
instance survives, later delivered; (c) after delivery, the `signal_received` history
|
|
119
|
+
row and `processed_at` were written together (both present; the instance's `seq`
|
|
120
|
+
advanced by exactly one for the signal step).
|
|
121
|
+
|
|
122
|
+
- **Affected files.** `src/correlator.ts`, `src/persistence/instances.ts`
|
|
123
|
+
(`findByCorrelation`), `src/persistence/inbox.ts` (`lockInboxRow`), `src/driver.ts`
|
|
124
|
+
(`kind: "signal"` branch), `src/types.ts`, `tests/correlator.test.ts` (new),
|
|
125
|
+
`tests/fixtures/stock-replenishment.ts` (cooldown variant), `API.md` "Per-tick behavior"
|
|
126
|
+
(via 04 #1).
|
|
127
|
+
|
|
128
|
+
- **Effort / Value / Risk.** M / high / low–med. The semantic change is that early
|
|
129
|
+
signals now wait instead of vanishing; a signal for an instance that never waits again
|
|
130
|
+
lingers unprocessed until that instance terminates — bounded by the instance's lifetime.
|
|
131
|
+
|
|
132
|
+
- **Implementation notes.** Keep the matcher call inside the tick transaction as today
|
|
133
|
+
(`correlator.ts:105–118`): the `SKIP LOCKED` claim is what stops two correlator workers
|
|
134
|
+
from double-poking the same row within a tick, and matchers are cheap by contract.
|
|
135
|
+
Depends on 01 #1 for `expected_seq` and the `waiting` precondition. Delete the
|
|
136
|
+
instance `UPDATE` at `correlator.ts:198–203` — with the fence it is no longer needed as
|
|
137
|
+
a "claimed" marker (see #2 for why it looked useful).
|
|
138
|
+
|
|
139
|
+
### 2. Concurrent / duplicate signals and matcher exceptions
|
|
140
|
+
|
|
141
|
+
- **Problem / observation.**
|
|
142
|
+
**(a) Same tick, same token, two rows.** The first match flips the instance to
|
|
143
|
+
`pending` (`correlator.ts:198–203`); the second row then finds no `waiting` instance and
|
|
144
|
+
is marked processed — dropped. Under #1 the flip is gone, so the second row would be
|
|
145
|
+
poked too; its advance is a fenced no-op (01 #1) and, because processed-marking now
|
|
146
|
+
happens in the advance, the row stays unprocessed and is deferred to the next wait point
|
|
147
|
+
— correct, _provided_ the tick does not poke the same instance twice in one batch.
|
|
148
|
+
**(b) Matcher throws.** `catch → matched = false` (`correlator.ts:169–172`) leads to
|
|
149
|
+
`signal_rejected` + mark processed: a transient failure in an async matcher (a lookup, an
|
|
150
|
+
HTTP call) permanently consumes the signal.
|
|
151
|
+
**(c) `LIMIT 1` without `ORDER BY`** (`instances.ts:174–178`): if two live instances
|
|
152
|
+
share a token (not enforced anywhere), delivery is arbitrary.
|
|
153
|
+
|
|
154
|
+
- **Evidence.** `correlator.ts:169–172`:
|
|
155
|
+
```ts
|
|
156
|
+
} catch (e) {
|
|
157
|
+
clog.error?.(`correlator: matcher "${meta.matcher}" threw: ${e}`);
|
|
158
|
+
matched = false;
|
|
159
|
+
}
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
- **Proposed change.** (a) A per-tick `Set<instance_id>`; a second row for an instance
|
|
163
|
+
already poked in this tick is deferred. (b) On throw: log at error, leave the row
|
|
164
|
+
unprocessed, continue with the next row. A deterministic throw retries every tick with
|
|
165
|
+
an error line — visible rather than silent, bounded by the instance's lifetime. (c)
|
|
166
|
+
`ORDER BY created_at` in `findByCorrelation`; document "one live instance per token" as
|
|
167
|
+
the contract. No unique index: the terminal path clears the token (`driver.ts:201`) and a
|
|
168
|
+
partial unique index on `waiting` rows would turn a definition mistake into a failed
|
|
169
|
+
advance rather than a matcher decision.
|
|
170
|
+
|
|
171
|
+
- **Done when** — tests: "two signals for one token appended before a tick: the first is
|
|
172
|
+
delivered, the second remains unprocessed and is delivered at the next wait point (or
|
|
173
|
+
marked processed once the instance terminates)"; "a matcher that throws on its first
|
|
174
|
+
call and succeeds on the second delivers the signal on the second tick".
|
|
175
|
+
|
|
176
|
+
- **Affected files.** `src/correlator.ts`, `src/persistence/instances.ts`,
|
|
177
|
+
`tests/correlator.test.ts`. Part of the same task as #1.
|
|
178
|
+
|
|
179
|
+
- **Effort / Value / Risk.** S / med / low.
|
|
180
|
+
|
|
181
|
+
### 3. The correlation token cannot be set after creation
|
|
182
|
+
|
|
183
|
+
- **Problem / observation.** The token lives on the instance row and is settable only via
|
|
184
|
+
`create({ correlationToken })` (`workflow.ts:177–194`). `updateInstance` is not exported
|
|
185
|
+
(`mod.ts:38–77`), `HandlerResult` is `{ outcome, data? }` (`types.ts:49–52`), and the
|
|
186
|
+
suspending branch deliberately leaves the token alone (`driver.ts:238–239`). So: one
|
|
187
|
+
token per instance, chosen before any effect runs. That fits plus-addressing
|
|
188
|
+
(`README.md:140`) and nothing else — an SMTP `Message-ID` for `In-Reply-To` threading, a
|
|
189
|
+
payment-provider session id, a ticket number returned by an API all exist only _after_
|
|
190
|
+
the effect. Two sequential wait points must share one token and rely on the matcher to
|
|
191
|
+
tell their signals apart.
|
|
192
|
+
|
|
193
|
+
- **Evidence.** `types.ts:49–52`:
|
|
194
|
+
```ts
|
|
195
|
+
export interface HandlerResult {
|
|
196
|
+
outcome: string;
|
|
197
|
+
data?: Record<string, unknown>;
|
|
198
|
+
}
|
|
199
|
+
```
|
|
200
|
+
`driver.ts:238–239`: "correlation_token already on the row (set at create-time …). We
|
|
201
|
+
don't auto-generate one here."
|
|
202
|
+
|
|
203
|
+
- **Proposed change.** `HandlerResult.correlationToken?: string | null`, carried on the
|
|
204
|
+
effect-completion advance payload (01 #1 already reshapes it) and applied in
|
|
205
|
+
`runAdvance` at the settle-point write, so the token is in place the moment the row
|
|
206
|
+
becomes `waiting`. `null` clears it. Record it in the `transition` history row's data.
|
|
207
|
+
`create({ correlationToken })` stays as the up-front option.
|
|
208
|
+
|
|
209
|
+
- **Done when** — test: `sendOrderEmail` returns `{ outcome: "SENT", correlationToken:
|
|
210
|
+
"msg-42" }` on an instance created without a token → `find()` shows `msg-42` → a signal
|
|
211
|
+
with that token is delivered and the workflow reaches `_end_ok`.
|
|
212
|
+
|
|
213
|
+
- **Affected files.** `src/types.ts`, `src/driver.ts`, `src/workflow.ts` (payload),
|
|
214
|
+
`API.md`.
|
|
215
|
+
|
|
216
|
+
- **Effort / Value / Risk.** S / med / low. Depends on 01 #1 (payload shape).
|
|
217
|
+
|
|
218
|
+
- **Implementation notes.** An instance with `correlation_token IS NULL` entering a
|
|
219
|
+
`MATCHED`-accepting wait can never be signalled; the validator cannot see it (tokens are
|
|
220
|
+
runtime), so `clog.warn` at that moment in the suspending branch.
|
|
221
|
+
|
|
222
|
+
## Open questions / decisions needed
|
|
223
|
+
|
|
224
|
+
Resolved 2026-08-26 (see `PROGRESS.md` → Decisions log): inbox rows whose token no live
|
|
225
|
+
instance owns — unknown token, or the instance is terminal — are marked processed on the
|
|
226
|
+
first tick with a warn log, plus a `signal_rejected` history row when a terminal instance
|
|
227
|
+
exists. No grace period. The deferral semantics for early signals stand as proposed in #1
|
|
228
|
+
and #2.
|