@marianmeres/workflow 1.0.1 → 2.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,404 @@
1
+ <!--
2
+ GENERATED ANALYSIS — @marianmeres/workflow (durability & idempotency)
3
+ Produced 2026-08-26 by an inline single-agent review: full read of src/, tests/, docs, and the
4
+ resolved sources of @marianmeres/fsm 3.1.0, steve 3.0.0, cron 3.2.0 → self-verification pass
5
+ re-opening every cited line. Claims verified against the codebase at commit 02a7635.
6
+ Planning artifact; no code was changed.
7
+ -->
8
+
9
+ # Durability & Idempotency — the driver, steve, and the crash windows
10
+
11
+ > This document is about the contract between the driver and `@marianmeres/steve`. Steve
12
+ > delivers jobs **at least once**: it re-runs a job whose handler already committed if the
13
+ > completion write fails (`_execute.ts:44–53` → `_handle-failure.ts:59–68`), and it never
14
+ > retries a job whose worker crashed — `cleanup()` marks such jobs `expired`, which is
15
+ > terminal by design (`_mark-expired.ts:6–8`), and only when the consumer turned
16
+ > `autoCleanup` on. The driver, however, is written as if delivery were exactly-once.
17
+
18
+ > The single most important takeaway: **there is no fencing token.** Nothing on an advance
19
+ > or effect job says which version of the instance row it was issued against, so a
20
+ > duplicate or stale job is not ignored — it is applied. A stale outcome is rejected by the
21
+ > FSM and marks a healthy instance `failed`; a replayed initial advance dispatches the same
22
+ > effect twice; an expired effect job from a step the instance already left marks the
23
+ > instance `failed`. Separately, the scheduler's claim-then-enqueue and `create()`'s
24
+ > insert-then-enqueue have windows that leave rows `pending` with no job that will ever
25
+ > move them, and `workflow.advance` failures are not observed at all.
26
+
27
+ > Headline recommendation: add a per-instance sequence number (`seq`) that every advance
28
+ > bumps and every job carries as `expected_seq`; make the scheduler and correlator ticks
29
+ > read-only "pokes" that the advance re-validates under the row lock; observe advance-job
30
+ > failures and re-dispatch `expired` jobs (bounded). With those three, every crash window
31
+ > collapses to "the next tick pokes again" and every duplicate collapses to a no-op. A
32
+ > `cancel()`/`retry()` pair then falls out almost for free.
33
+
34
+ > What is fine and needs no change: the per-advance transaction with `SELECT … FOR UPDATE`
35
+ > (`driver.ts:104–105`), the terminal-state guard (`:111–120`), the `fromSnapshot` resume
36
+ > (no `onEnter` on restore, `fsm.ts:1005–1048`), and `FOR UPDATE SKIP LOCKED` batch claiming.
37
+
38
+ ## Summary of recommendations
39
+
40
+ | # | Recommendation | Value | Effort | Risk |
41
+ | - | -------------------------------------------------------------------------------------------- | ----- | ------ | ------------------------------------------ |
42
+ | 1 | Fencing token `seq` on the instance; `expected_seq` + `kind` on every advance; fenced effect | high | M | med — migration + payload shape (shimmed) |
43
+ | 2 | Ticks become read-only pokes; stale-`pending` re-poke; `create()` inside a transaction | high | S–M | low — behavior-preserving on healthy paths |
44
+ | 3 | Observe `workflow.advance` failures; bounded re-dispatch of `expired` jobs | high | S | low |
45
+ | 4 | `cancel()` / `retry()` admin API | med | S | low — additive |
46
+
47
+ ## Findings & recommendations (detailed)
48
+
49
+ ### 1. No fencing token: duplicate or late jobs corrupt instances
50
+
51
+ - **Problem / observation.** Three concrete paths, all reachable in production, none
52
+ reachable in the current test suite:
53
+
54
+ **(a) The documented crash-after-enqueue scenario fails a healthy instance.**
55
+ `AGENTS.md:232` describes it as recoverable: the effect handler succeeds, `runEffect`
56
+ enqueues the completion advance (`driver.ts:375–380`), the process dies before steve
57
+ writes the job's completion. On restart the advance runs and the instance moves on — say
58
+ to `running` at `classify_reply`. Five minutes later `jobs.cleanup()` (via
59
+ `autoCleanup`) marks the old effect job `expired` and publishes `onDone`
60
+ (`jobs.ts:920–922`); the workflow's listener (`workflow.ts:131–144`) calls
61
+ `failEffectJob` (`driver.ts:389–415`), which checks only that the row is non-terminal
62
+ and marks it `failed` with `effect_failed { handler: "sendOrderEmail" }`. A stale job
63
+ from a step the instance already left kills it.
64
+
65
+ **(b) A re-run effect produces a second completion advance, which the FSM rejects.**
66
+ If steve's completion write fails after the handler returned, `_executeJob` falls into
67
+ its `catch` and `_handleJobFailure` puts the job back to `pending`
68
+ (`_execute.ts:51–53`, `_handle-failure.ts:59–68`). The handler runs again (idempotency is
69
+ documented, fine) and enqueues **a second advance with the same outcome**. `runAdvance`
70
+ applies it at the _new_ cursor; `fsm.transition` returns `null`
71
+ (`driver.ts:146–150`), so the instance gets `transition_rejected` and `failed`
72
+ (`:151–165`). The same re-run of the _initial_ advance is worse: the row is already
73
+ `running`, the dispatch loop lands on the effectful case again (`:211–233`) and enqueues
74
+ a second effect job for the same node — the handler runs twice, both completions
75
+ advance, the second is rejected.
76
+
77
+ **(c) `runEffect` runs the handler unconditionally.** It reads the row without a lock and
78
+ without looking at `execution_state` or the cursor (`driver.ts:349–373`). After a
79
+ future `cancel()` (#4), or on any stale dispatch from (b), the side effect still fires.
80
+
81
+ Also latent: an advance _without_ an outcome at a `waiting` cursor recomputes `wake_at`
82
+ (`driver.ts:235–246`) — i.e. resets the timer. Today only a duplicate can trigger it; once
83
+ ticks re-poke (#2) it becomes a real hazard unless advances carry a precondition.
84
+
85
+ The comment at `workflow.ts:152–154` — "The advance is idempotent (cursor-aware) so an
86
+ enqueue-but-no-commit race is recoverable" — is the assumption this finding refutes:
87
+ nothing compares a job's origin against the row it lands on.
88
+
89
+ - **Evidence.** `driver.ts:389–415`:
90
+ ```ts
91
+ export async function failEffectJob(pool, payload, reason) {
92
+ await withTransaction(pool, async (client) => {
93
+ const row = await lockInstance(client, payload.instance_id);
94
+ if (!row) return;
95
+ if (row.execution_state === COMPLETED || FAILED || CANCELLED) return;
96
+ await updateInstance(client, row.id, { execution_state: EXECUTION_STATE.FAILED });
97
+ // ... history effect_failed
98
+ ```
99
+ `driver.ts:349–373` (`runEffect`): plain `SELECT … WHERE id = $1`, then `handler(...)` —
100
+ no state check. `types.ts:151–169`: `AdvanceJobPayload` carries `outcome`, `outcome_data`,
101
+ `timeout?: boolean`; `EffectJobPayload` carries `handler` — neither carries anything that
102
+ identifies the row version they were issued against.
103
+
104
+ - **Proposed change.**
105
+ 1. Migration `1.2.0` (append-only ledger, `src/migrations/1_2_0.ts`):
106
+ ```sql
107
+ ALTER TABLE __workflow_instances ADD COLUMN IF NOT EXISTS seq integer NOT NULL DEFAULT 0;
108
+ -- down: ALTER TABLE __workflow_instances DROP COLUMN IF EXISTS seq;
109
+ ```
110
+ (`DEFAULT 0` on a non-volatile default is a metadata-only change on PG ≥ 11.)
111
+ 2. `WorkflowInstanceRow.seq: number`. Every row write issued by `runAdvance`,
112
+ `failInstance`, and (later) `cancel`/`retry` bumps `seq = seq + 1` in the same
113
+ `UPDATE` as the state change — one settle-point write per advance, so a single
114
+ `bumpSeq: true` flag on `updateInstance` (or a dedicated `advanceInstance` helper) is
115
+ enough.
116
+ 3. Payload shapes (`types.ts`):
117
+ ```ts
118
+ export type AdvanceKind = "start" | "effect" | "timeout" | "signal";
119
+ export interface AdvanceJobPayload {
120
+ tenant_id: string;
121
+ instance_id: string;
122
+ kind: AdvanceKind; // replaces `timeout?: boolean`
123
+ expected_seq: number; // fence
124
+ outcome?: string; // kind=effect (handler's label)
125
+ outcome_data?: Record<string, unknown>;
126
+ handler?: string; // kind=effect, for history
127
+ inbox_id?: string; // kind=signal (see 02 #1)
128
+ redispatch?: number; // see #3
129
+ }
130
+ export interface EffectJobPayload {
131
+ tenant_id: string;
132
+ instance_id: string;
133
+ handler: string;
134
+ seq: number; // fence
135
+ cursor: string; // for history/diagnostics
136
+ redispatch?: number;
137
+ }
138
+ ```
139
+ 4. `runAdvance`, inside the row lock, after the terminal check:
140
+ ```ts
141
+ if (payload.expected_seq !== undefined && row.seq !== payload.expected_seq) {
142
+ clog.debug?.(
143
+ `advance: stale (row.seq=${row.seq}, expected=${payload.expected_seq}); no-op`,
144
+ );
145
+ return;
146
+ }
147
+ // per-kind preconditions — violation is a debug no-op, never a failure
148
+ switch (payload.kind) {
149
+ case "start":
150
+ if (row.execution_state !== PENDING) return;
151
+ break;
152
+ case "effect":
153
+ if (row.execution_state !== RUNNING) return;
154
+ break;
155
+ case "timeout":
156
+ if (
157
+ row.execution_state !== WAITING || !row.wake_at ||
158
+ row.wake_at > new Date()
159
+ ) return;
160
+ break;
161
+ case "signal":
162
+ if (row.execution_state !== WAITING) return;
163
+ break;
164
+ }
165
+ ```
166
+ A payload without `expected_seq`/`kind` is a job queued by 2.0.x: treat it as
167
+ unfenced and infer `kind` (`timeout: true` → `timeout`; `outcome` present →
168
+ `effect`; else `start`). Same transitional status as `payloadTenantId`
169
+ (`driver.ts:34–38`); remove both together.
170
+ 5. `runEffect`: `if (row.seq !== payload.seq || row.execution_state !== RUNNING) return
171
+ { skipped: "stale" };` — no handler call, no advance. Completion advance carries
172
+ `{ kind: "effect", expected_seq: payload.seq, outcome, outcome_data, handler }`.
173
+ 6. `failEffectJob`: `if (row.seq !== payload.seq) return;` before anything else.
174
+ 7. `create()`: wrap insert + `created` history + enqueue in `withTransaction`. Steve's
175
+ insert is autocommit on its own connection (`_create.ts:21,43`), so if our commit then
176
+ fails the advance finds no row and no-ops (`driver.ts:106–109`); if `jobs.create`
177
+ throws, our insert rolls back. The residual "insert committed, enqueue never happened"
178
+ window is what #2's stale-`pending` re-poke covers.
179
+ 8. Emit the currently dead `HISTORY_EVENT.EFFECT_COMPLETED` (`types.ts:118`) for
180
+ `kind: "effect"` advances, before the `transition` row, with `{ handler, outcome }`.
181
+
182
+ - **Done when** — `tests/durability.test.ts` (new) passes with: (1) enqueuing the initial
183
+ advance payload twice yields exactly one `workflow.effect.checkInventory` row in `__job`;
184
+ (2) enqueuing a second, identical effect-completion advance leaves the instance
185
+ non-failed with the cursor advanced once; (3) `failEffectJob` with a stale `seq` leaves a
186
+ moved-on instance untouched; and `tests/migrations.test.ts` asserts `seq` exists after
187
+ `up("latest")` and is absent after `down("1.1.0")`.
188
+
189
+ - **Affected files.** `src/migrations/1_2_0.ts` (new), `src/migrations/index.ts`,
190
+ `src/types.ts`, `src/persistence/instances.ts`, `src/driver.ts`, `src/workflow.ts`,
191
+ `src/scheduler.ts` + `src/correlator.ts` (payload shape only — their redesign is #2 and
192
+ 02 #1), `tests/migrations.test.ts`, `tests/durability.test.ts` (new). Docs in 04 #1.
193
+
194
+ - **Effort / Value / Risk.** M / high / med. The migration is additive; the payload
195
+ change is shimmed for jobs queued before the upgrade. Consumers calling
196
+ `enqueueAdvance`/`enqueueEffect` directly (documented as internal, `API.md:102–104`)
197
+ must now pass a fenced payload.
198
+
199
+ - **Implementation notes.** Keep the fence check _inside_ the row lock — it is only
200
+ meaningful against the locked row. Log stale no-ops at debug and do not write a history
201
+ row for them: once ticks re-poke (#2) they are routine, not events. While there, drop
202
+ the ignored `_client` parameter from `JobEnqueuer` (`driver.ts:62–72`,
203
+ `workflow.ts:148–168`): steve always inserts on its own connection, so the parameter
204
+ only suggests an atomicity that does not exist (the README even claims it — 04 #1).
205
+ Column name is bikeshed-able (`seq` / `step` / `revision`); `seq` avoids reading as
206
+ "workflow step".
207
+
208
+ ### 2. Ticks claim-then-enqueue and `create()` inserts-then-enqueues — crash windows leave rows `pending` forever
209
+
210
+ - **Problem / observation.** The scheduler tick (`scheduler.ts:87–108`) calls
211
+ `claimDueWakeUps` — an autocommit `UPDATE … SET execution_state = 'pending', wake_at =
212
+ NULL` on the pool (`instances.ts:137–161`) — and only _then_ enqueues one advance per
213
+ row via `jobs.create` on steve's connection. Die between the two, or let `jobs.create`
214
+ throw for row 3 of 5, and the affected rows are `pending` with `wake_at = NULL` and no
215
+ job: `claimDueWakeUps` requires `waiting`, the correlator requires `waiting`, nothing
216
+ will ever touch them again. `create()` (`workflow.ts:187–205`) is three autocommits;
217
+ die after the first and the instance is `pending` forever. Steve cannot enlist in our
218
+ transaction (`Jobs.create` → `_create` always uses `context.db`, `_create.ts:21,43`), so
219
+ "enqueue inside the tx" is not an option — although `README.md:195` claims that is what
220
+ happens (04 #1).
221
+
222
+ > **Cut from the draft:** the correlator was listed here too. Its flip is inside the tick
223
+ > transaction and the enqueue precedes the commit (`correlator.ts:198–210`), so its crash
224
+ > windows are benign — a rollback either undoes everything or leaves a job that applies
225
+ > `MATCHED` against a still-`waiting` row. Its real problems are semantic (02 #1).
226
+
227
+ - **Evidence.** `instances.ts:143–158`:
228
+ ```sql
229
+ UPDATE __workflow_instances SET execution_state = 'pending', wake_at = NULL, updated_at = now()
230
+ WHERE id IN (SELECT id FROM __workflow_instances WHERE tenant_id = $1 AND execution_state = 'waiting'
231
+ AND wake_at IS NOT NULL AND wake_at <= now() ORDER BY wake_at LIMIT $4 FOR UPDATE SKIP LOCKED)
232
+ RETURNING id, correlation_token
233
+ ```
234
+ followed at `scheduler.ts:97–105` by a `for` loop of `enqueueAdvance` calls.
235
+
236
+ - **Proposed change.** With #1's fence in place, make the tick a read-only poke and let the
237
+ advance do every write under the lock:
238
+ ```ts
239
+ // scheduler tick — no UPDATE
240
+ const due = await selectDueWakeUps(db, tenantId, batch); // SELECT id, seq ... WHERE waiting AND wake_at <= now() ORDER BY wake_at LIMIT n
241
+ for (const r of due) {
242
+ await wf.enqueueAdvance({
243
+ tenant_id,
244
+ instance_id: r.id,
245
+ kind: "timeout",
246
+ expected_seq: r.seq,
247
+ });
248
+ }
249
+
250
+ const stale = await selectStalePending(db, tenantId, stalePendingSec, batch); // SELECT id, seq ... WHERE pending AND updated_at < now() - $sec
251
+ for (const r of stale) {
252
+ await wf.enqueueAdvance({
253
+ tenant_id,
254
+ instance_id: r.id,
255
+ kind: "start",
256
+ expected_seq: r.seq,
257
+ });
258
+ }
259
+ ```
260
+ The advance re-checks `waiting && wake_at <= now()` (the `timeout` precondition from #1),
261
+ applies `TIMEOUT`, and clears `wake_at` itself. A row poked but not yet advanced is
262
+ poked again next tick — a fenced no-op. Delete `claimDueWakeUps`. The stale-`pending`
263
+ re-poke is safe _only_ because, after #1 and 02 #1, `create()` is the sole producer of
264
+ `pending` (hence the T02 → T03 sequencing in `PROGRESS.md`). Add a partial index in
265
+ migration `1.2.0` for the second scan:
266
+ ```sql
267
+ CREATE INDEX IF NOT EXISTS __workflow_instances_stale_pending_idx
268
+ ON __workflow_instances (tenant_id, updated_at) WHERE execution_state = 'pending';
269
+ ```
270
+ (`pending` rows are transient, so the index stays tiny.) New option
271
+ `WorkflowSchedulerOptions.stalePendingSec` (default `300`; `0` disables).
272
+
273
+ - **Done when** — tests: "two `scheduler.tickOnce()` calls for one due row apply `TIMEOUT`
274
+ once (exactly one `timeout` history row; cursor `_end_timeout`)"; "a `pending` row with
275
+ no job and `updated_at` older than `stalePendingSec` is advanced by the next tick";
276
+ `grep -rn claimDueWakeUps src/` returns nothing. The existing TIMEOUT test
277
+ (`tests/stock_replenishment.test.ts:173–174`, `assertEquals(claimed, 1)`) is adjusted to
278
+ the new return shape.
279
+
280
+ - **Affected files.** `src/scheduler.ts`, `src/persistence/instances.ts`, `src/driver.ts`
281
+ (precondition from #1), `src/migrations/1_2_0.ts` (index), `tests/stock_replenishment.test.ts`,
282
+ `tests/durability.test.ts`.
283
+
284
+ - **Effort / Value / Risk.** S–M / high / low. Healthy paths behave identically; only the
285
+ moment `wake_at` is cleared moves from the tick to the advance.
286
+
287
+ - **Implementation notes.** `tickOnce()` can return `{ woken, repoked }` (or keep a number
288
+ = sum). Keep `ORDER BY wake_at` and the batch size. An in-process "poked recently"
289
+ `Map<id, ts>` to suppress duplicate pokes between ticks is a nice-to-have, not needed for
290
+ correctness. The `timeout: true` flag becomes `kind: "timeout"`.
291
+
292
+ ### 3. `workflow.advance` failures are unobserved; `expired` is terminal in steve and the docs say "retry"
293
+
294
+ - **Problem / observation.** `workflow.ts:131–144` subscribes `onDone` for effect types
295
+ only. A `workflow.advance` job that throws — a guard or `action` throwing inside
296
+ `fsm.transition` (fsm wraps and rethrows, `fsm.ts:414–449`), or any DB error — is
297
+ retried by steve three times with 2 s / 4 s backoff (`_handle-failure.ts:56`) and then
298
+ marked `failed`, silently. The instance stays `pending`/`running` with no job. After #2,
299
+ a `pending` row is re-poked every `stalePendingSec` forever (a permanently-throwing guard
300
+ becomes an infinite retry); a `running` row whose outcome died with the advance is
301
+ stuck for good.
302
+
303
+ Steve never retries a crashed job: `cleanup()` flips `running` rows older than 5 min to
304
+ `expired`, terminal by design (`_mark-expired.ts:6–8`), and publishes `onDone`
305
+ (`jobs.ts:920–922`). That reaper runs only with `autoCleanup: true`
306
+ (`jobs.ts:451–454`, `:697–700`) or a manual `cleanup()`; the default is off. So a worker
307
+ crash mid-effect means: `autoCleanup` on → instance `failed` after ~5 min via
308
+ `failEffectJob`; off → instance `running` forever. `AGENTS.md:231` and `README.md:196`
309
+ say steve "retries" — it does not. (04 #1 fixes the text; this finding fixes the
310
+ behavior.)
311
+
312
+ - **Evidence.** `workflow.ts:131–144`:
313
+ ```ts
314
+ for (const name of handlerNames) {
315
+ const type = effectJobType(name);
316
+ this.jobs.onDone(type, (job) => {
317
+ if (job.status === JOB_STATUS.FAILED || job.status === JOB_STATUS.EXPIRED) {
318
+ failEffectJob(this.db, payload, `steve: ${job.status}`).catch(...)
319
+ ```
320
+ No subscription for `JOB_TYPE_ADVANCE`. `_mark-expired.ts:6–8`: "we don't auto-retry
321
+ because the work may now be stale".
322
+
323
+ - **Proposed change.**
324
+ 1. `jobs.onDone(JOB_TYPE_ADVANCE, cb)`: `failed` → `failInstance` with the last attempt's
325
+ `error_message` (`jobs.find(job.uid, true)` → `attempts.at(-1)?.error_message`,
326
+ `jobs.ts:806–824`), history `failed { reason, job_uid }`; `expired` → re-enqueue the
327
+ identical payload with `redispatch: (payload.redispatch ?? 0) + 1` while
328
+ `< redispatchLimit`, else fail. If the original advance did commit before the crash,
329
+ the re-dispatch is a fenced no-op (#1).
330
+ 2. Effect types: `expired` → same bounded re-dispatch of the identical effect payload
331
+ (`seq` intact — a zombie handler that later completes produces a fenced duplicate;
332
+ handler idempotency is already required); `failed` → `failEffectJob` as today (fenced).
333
+ 3. New options: `advanceMaxAttempts` (default `10`: exponential backoff sums to ≈17 min
334
+ of DB-outage tolerance before an instance is failed; today's implicit `3` gives ≈6 s)
335
+ passed as `max_attempts` where advance jobs are created (`workflow.ts:155`,
336
+ `:202–205`); `redispatchLimit` (default `3`).
337
+
338
+ - **Done when** — tests: "an advance whose transition `action` throws every time ends with
339
+ the instance `failed` and a `failed` history row whose `reason` contains the error
340
+ message"; "an effect job marked `expired` via `jobs.cleanup(0)` is re-dispatched and the
341
+ workflow completes"; "after `redispatchLimit` expiries the instance is `failed` with
342
+ `effect_failed`".
343
+
344
+ - **Affected files.** `src/workflow.ts`, `src/driver.ts` (reason plumbing into
345
+ `failInstance`), `src/types.ts` (`redispatch`), `API.md` (options).
346
+
347
+ - **Effort / Value / Risk.** S / high / low.
348
+
349
+ - **Implementation notes.** `jobs.cleanup(0)` reaps immediately in tests
350
+ (`jobs.ts:904–916`; `Math.max(0, …)` at `_mark-expired.ts:20`). Steve wraps `onDone`
351
+ callbacks in try/catch and only logs (`jobs.ts:1099–1105`) — keep the
352
+ `.catch(clog.error)` pattern. Document that `autoCleanup: true` (or a periodic
353
+ `cleanup()`) is what makes crash recovery happen at all; the README example should set
354
+ it (04 #1).
355
+
356
+ ### 4. No way to cancel or retry an instance
357
+
358
+ - **Problem / observation.** `EXECUTION_STATE.CANCELLED` and `HISTORY_EVENT.CANCELLED`
359
+ exist (`types.ts:16`, `:126`) and the driver treats `cancelled` as terminal
360
+ (`driver.ts:111–120`, `:397–403`), but nothing sets it: `Workflow` exposes
361
+ `create`/`find`/`appendInbox`/`enqueueAdvance`/`enqueueEffect` only
362
+ (`workflow.ts:177–232`). An instance waiting three days for a reply cannot be aborted
363
+ except by SQL, and a `failed` instance (an API outage exhausted `effectMaxAttempts`)
364
+ cannot be resumed — the only recovery is a new instance, which re-runs every side effect
365
+ from the start.
366
+
367
+ - **Evidence.** `grep -rn "CANCELLED" src/` → only the two terminal checks in `driver.ts`
368
+ and the constant definitions; no writer.
369
+
370
+ - **Proposed change.** On `Workflow`, tenant-scoped like `find`:
371
+ ```ts
372
+ /** Non-terminal → cancelled. Clears wake_at/correlation_token, bumps seq (in-flight jobs become stale). */
373
+ async cancel(id: string, reason?: string): Promise<boolean>;
374
+ /** failed → pending at the current cursor, bumps seq, enqueues { kind: "start" }. `force` also allows `running`. */
375
+ async retry(id: string, opts?: { force?: boolean }): Promise<boolean>;
376
+ ```
377
+ Both return `false` when the state does not allow the operation. History: `cancelled
378
+ { reason }`, and a new `HISTORY_EVENT.RETRIED { from_state }`. `force: true` is the
379
+ operator asserting the effect is dead — the `seq` bump (#1) makes any zombie effect job
380
+ stale, so this is safe by construction.
381
+
382
+ - **Done when** — tests: "`cancel()` on a `waiting` instance → `cancelled`; a later
383
+ matching signal is marked processed without delivery; `scheduler.tickOnce()` ignores
384
+ it"; "`cancel()` while an effect job is queued → the job completes with
385
+ `{ skipped: "stale" }` and the handler is never called"; "`retry()` on a `failed`
386
+ instance re-dispatches the cursor node's effect and the workflow completes".
387
+
388
+ - **Affected files.** `src/workflow.ts`, `src/persistence/instances.ts`, `src/types.ts`,
389
+ `API.md`, `README.md` (Failure Modes).
390
+
391
+ - **Effort / Value / Risk.** S / med / low. Depends on #1.
392
+
393
+ - **Implementation notes.** `retry` from a `transition_rejected` failure re-dispatches the
394
+ cursor node's effect and will fail again unless the handler was fixed — acceptable for a
395
+ manual tool. If a "force an outcome" escape hatch is ever wanted, `enqueueAdvance` with
396
+ `{ kind: "effect", expected_seq: row.seq, outcome }` is it; today's unfenced
397
+ `enqueueAdvance` is that hatch by accident, and #1 closes it deliberately.
398
+
399
+ ## Open questions / decisions needed
400
+
401
+ All resolved in the owner interview of 2026-08-26 (see `PROGRESS.md` → Decisions log):
402
+ column name `seq`; `advanceMaxAttempts = 10` and `redispatchLimit = 3`; `expired` →
403
+ bounded re-dispatch of the identical payload (advance and effect), then fail;
404
+ stale-`pending` re-poke on by default at `300` s (`0` disables).
@@ -0,0 +1,228 @@
1
+ <!--
2
+ GENERATED ANALYSIS — @marianmeres/workflow (inbox correlation)
3
+ Produced 2026-08-26 by an inline single-agent review: full read of src/, tests/, docs, and the
4
+ resolved sources of @marianmeres/fsm 3.1.0, steve 3.0.0, cron 3.2.0 → self-verification pass
5
+ re-opening every cited line. Claims verified against the codebase at commit 02a7635.
6
+ Planning artifact; no code was changed.
7
+ -->
8
+
9
+ # Inbox Correlation — signal delivery semantics
10
+
11
+ > The inbox and correlator are how the outside world reaches a sleeping instance. For the
12
+ > motivating use case — supplier email replies — this is the main event, not an edge. The
13
+ > split "token is the index, matcher is the gate" (`AGENTS.md:202`) is right, the
14
+ > append-only inbox decoupled from matching is right, and the `FOR UPDATE SKIP LOCKED`
15
+ > batch claim is right. The delivery _rules_ are where the problems are.
16
+
17
+ > The single most important takeaway: **the correlator only looks for instances that are
18
+ > already `waiting`, and it consumes every inbox row it looks at.** A reply that lands
19
+ > while the instance is still `running` its send step — or sitting in a timer-only delay
20
+ > node — is marked processed and silently thrown away; the instance then waits out its
21
+ > full timeout for a reply that already arrived. Worse, at a timer-only suspending node the
22
+ > signal _is_ delivered as `MATCHED`, which the node has no transition for, and the
23
+ > instance is marked `failed`. Two replies inside one tick lose the second; a matcher that
24
+ > throws once consumes the signal permanently.
25
+
26
+ > Headline recommendation: deliver only when the instance is `waiting` at a node that
27
+ > accepts `MATCHED` and the matcher agrees; otherwise leave the row **unprocessed** (early
28
+ > signal — try again next tick) and mark it processed only when no live instance owns the
29
+ > token. Move "mark processed" into the advance transaction so delivery and transition are
30
+ > atomic, and let effect handlers set the correlation token so a wait point can key on
31
+ > something that only exists after the effect ran (a `Message-ID`, a provider session id).
32
+
33
+ ## Summary of recommendations
34
+
35
+ | # | Recommendation | Value | Effort | Risk |
36
+ | - | ------------------------------------------------------------------------------------------------------------------ | ----- | ------ | ---- |
37
+ | 1 | Defer early/late signals instead of dropping them; deliver only to `MATCHED`-accepting waits; processed-in-advance | high | M | low |
38
+ | 2 | Duplicate / concurrent signals and matcher exceptions | med | S | low |
39
+ | 3 | Handler-settable correlation token (per wait point) | med | S | low |
40
+
41
+ ## Findings & recommendations (detailed)
42
+
43
+ ### 1. Signals that arrive before the wait point are dropped; signals at timer-only waits fail the instance
44
+
45
+ - **Problem / observation.** `#processOne` looks the token up with
46
+ `findWaitingByCorrelation`, which filters `execution_state = 'waiting'`
47
+ (`instances.ts:175–177`), and on a miss marks the inbox row processed and returns
48
+ (`correlator.ts:130–133`). Timeline for the README flow: `sendOrderEmail` is `running`
49
+ (or the following advance has not yet flipped the row to `waiting`); the supplier's
50
+ auto-reply, or a fast human, is appended to the inbox by a webhook that has no idea where
51
+ the instance is; the correlator tick runs first → row processed, nothing delivered → the
52
+ instance reaches `await_reply` and waits three days for a reply that already came. The
53
+ window is the whole duration of whatever sits between "token goes out" and "instance is
54
+ waiting": a slow AI step, a retry with backoff, any timer-only node.
55
+
56
+ The timer-only case is not a drop but a failure: a `suspending` node without `matcher`
57
+ takes the `matched = true` default (`correlator.ts:144–146`), an advance with
58
+ `outcome: "MATCHED"` is enqueued (`:205–210`), `positioned.transition("MATCHED")`
59
+ returns `null` because the node's `on` has neither `MATCHED` nor `*`
60
+ (`fsm.ts:536–546`), and the driver writes `transition_rejected` and marks the instance
61
+ `failed` (`driver.ts:151–165`). Example: a "wait 1 h, then send a reminder" delay node
62
+ between `send_order` and `await_reply` — a reply during the hour kills the instance.
63
+ The validator cannot catch this and should not try (03 #1 covers `matcher` without
64
+ `MATCHED`): a delay node legitimately has no `MATCHED` edge.
65
+
66
+ - **Evidence.** `correlator.ts:124–133`:
67
+ ```ts
68
+ async #processOne(client, row) {
69
+ const instance = await findWaitingByCorrelation(client, this.tenantId, row.correlation_token);
70
+ if (!instance) {
71
+ await markProcessed(client, row.id); // <-- early signal is consumed here
72
+ return;
73
+ }
74
+ ```
75
+ `correlator.ts:144–146`: `let matched = true; if (meta && meta.kind === "suspending" &&
76
+ meta.matcher) { … }` — no check that the node accepts `MATCHED`.
77
+
78
+ - **Proposed change.** Replace the lookup and the decision table; move the processed-mark
79
+ into the advance.
80
+ 1. `findByCorrelation(exec, tenant_id, token)` → the live (non-terminal) instance,
81
+ `ORDER BY created_at LIMIT 1`.
82
+ 2. Decision table in the tick (inside the existing batch transaction):
83
+
84
+ | Instance | Action |
85
+ | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
86
+ | none (unknown token) / terminal | mark processed; `clog.warn` (unknown) or `signal_rejected { reason }` |
87
+ | live but not `waiting` | **defer** — leave unprocessed, debug log |
88
+ | `waiting`, node lacks `MATCHED` and `*` | **defer** |
89
+ | `waiting`, accepts `MATCHED`, matcher → `false` | `signal_rejected` history; mark processed |
90
+ | `waiting`, accepts `MATCHED`, matcher → `true` | enqueue `{ kind: "signal", inbox_id, expected_seq: instance.seq }` — do **not** flip, do **not** mark processed |
91
+
92
+ "Accepts `MATCHED`" = `"MATCHED" in stateCfg.on || "*" in stateCfg.on` (resolution
93
+ order per `fsm.ts:510–534`).
94
+ 3. `runAdvance`, `kind: "signal"` (01 #1 gives the fence and the `waiting` precondition):
95
+ ```ts
96
+ const inbox = await lockInboxRow(client, payload.inbox_id!); // SELECT ... FOR UPDATE
97
+ if (!inbox || inbox.processed_at) return; // delivered by a duplicate poke
98
+ const result = positioned.transition("MATCHED", inbox.payload, false);
99
+ // ... rejected → transition_rejected + failInstance (unchanged)
100
+ await appendHistory(client, {
101
+ event_type: SIGNAL_RECEIVED,
102
+ from_node,
103
+ data: { inbox_id, source },
104
+ });
105
+ await markProcessed(client, inbox.id); // same transaction as the transition
106
+ ```
107
+ 4. Deferred rows are re-examined every tick. Starvation note: `claimUnprocessed` orders
108
+ by `received_at` with a `LIMIT` (`inbox.ts:49–55`); a backlog of deferred rows ≥
109
+ `tickBatchSize` would shadow newer tokens. At the stated scale that is a non-issue;
110
+ document it, and keep an `inboxMaxAgeSec` knob ("rows older than N seconds are marked
111
+ processed with `signal_expired`") as a later addition if it ever bites.
112
+
113
+ - **Done when** — `tests/correlator.test.ts` (new): (a) append the signal _before_ the
114
+ instance reaches `await_reply` (gate the `sendOrderEmail` handler on a promise), tick →
115
+ the row is still unprocessed and the instance is not failed; release the handler, wait
116
+ for `waiting`, tick → delivered, cursor `_end_ok`; (b) a fixture variant with a
117
+ timer-only `cooldown` node before `await_reply`: signal during cooldown → deferred,
118
+ instance survives, later delivered; (c) after delivery, the `signal_received` history
119
+ row and `processed_at` were written together (both present; the instance's `seq`
120
+ advanced by exactly one for the signal step).
121
+
122
+ - **Affected files.** `src/correlator.ts`, `src/persistence/instances.ts`
123
+ (`findByCorrelation`), `src/persistence/inbox.ts` (`lockInboxRow`), `src/driver.ts`
124
+ (`kind: "signal"` branch), `src/types.ts`, `tests/correlator.test.ts` (new),
125
+ `tests/fixtures/stock-replenishment.ts` (cooldown variant), `API.md` "Per-tick behavior"
126
+ (via 04 #1).
127
+
128
+ - **Effort / Value / Risk.** M / high / low–med. The semantic change is that early
129
+ signals now wait instead of vanishing; a signal for an instance that never waits again
130
+ lingers unprocessed until that instance terminates — bounded by the instance's lifetime.
131
+
132
+ - **Implementation notes.** Keep the matcher call inside the tick transaction as today
133
+ (`correlator.ts:105–118`): the `SKIP LOCKED` claim is what stops two correlator workers
134
+ from double-poking the same row within a tick, and matchers are cheap by contract.
135
+ Depends on 01 #1 for `expected_seq` and the `waiting` precondition. Delete the
136
+ instance `UPDATE` at `correlator.ts:198–203` — with the fence it is no longer needed as
137
+ a "claimed" marker (see #2 for why it looked useful).
138
+
139
+ ### 2. Concurrent / duplicate signals and matcher exceptions
140
+
141
+ - **Problem / observation.**
142
+ **(a) Same tick, same token, two rows.** The first match flips the instance to
143
+ `pending` (`correlator.ts:198–203`); the second row then finds no `waiting` instance and
144
+ is marked processed — dropped. Under #1 the flip is gone, so the second row would be
145
+ poked too; its advance is a fenced no-op (01 #1) and, because processed-marking now
146
+ happens in the advance, the row stays unprocessed and is deferred to the next wait point
147
+ — correct, _provided_ the tick does not poke the same instance twice in one batch.
148
+ **(b) Matcher throws.** `catch → matched = false` (`correlator.ts:169–172`) leads to
149
+ `signal_rejected` + mark processed: a transient failure in an async matcher (a lookup, an
150
+ HTTP call) permanently consumes the signal.
151
+ **(c) `LIMIT 1` without `ORDER BY`** (`instances.ts:174–178`): if two live instances
152
+ share a token (not enforced anywhere), delivery is arbitrary.
153
+
154
+ - **Evidence.** `correlator.ts:169–172`:
155
+ ```ts
156
+ } catch (e) {
157
+ clog.error?.(`correlator: matcher "${meta.matcher}" threw: ${e}`);
158
+ matched = false;
159
+ }
160
+ ```
161
+
162
+ - **Proposed change.** (a) A per-tick `Set<instance_id>`; a second row for an instance
163
+ already poked in this tick is deferred. (b) On throw: log at error, leave the row
164
+ unprocessed, continue with the next row. A deterministic throw retries every tick with
165
+ an error line — visible rather than silent, bounded by the instance's lifetime. (c)
166
+ `ORDER BY created_at` in `findByCorrelation`; document "one live instance per token" as
167
+ the contract. No unique index: the terminal path clears the token (`driver.ts:201`) and a
168
+ partial unique index on `waiting` rows would turn a definition mistake into a failed
169
+ advance rather than a matcher decision.
170
+
171
+ - **Done when** — tests: "two signals for one token appended before a tick: the first is
172
+ delivered, the second remains unprocessed and is delivered at the next wait point (or
173
+ marked processed once the instance terminates)"; "a matcher that throws on its first
174
+ call and succeeds on the second delivers the signal on the second tick".
175
+
176
+ - **Affected files.** `src/correlator.ts`, `src/persistence/instances.ts`,
177
+ `tests/correlator.test.ts`. Part of the same task as #1.
178
+
179
+ - **Effort / Value / Risk.** S / med / low.
180
+
181
+ ### 3. The correlation token cannot be set after creation
182
+
183
+ - **Problem / observation.** The token lives on the instance row and is settable only via
184
+ `create({ correlationToken })` (`workflow.ts:177–194`). `updateInstance` is not exported
185
+ (`mod.ts:38–77`), `HandlerResult` is `{ outcome, data? }` (`types.ts:49–52`), and the
186
+ suspending branch deliberately leaves the token alone (`driver.ts:238–239`). So: one
187
+ token per instance, chosen before any effect runs. That fits plus-addressing
188
+ (`README.md:140`) and nothing else — an SMTP `Message-ID` for `In-Reply-To` threading, a
189
+ payment-provider session id, a ticket number returned by an API all exist only _after_
190
+ the effect. Two sequential wait points must share one token and rely on the matcher to
191
+ tell their signals apart.
192
+
193
+ - **Evidence.** `types.ts:49–52`:
194
+ ```ts
195
+ export interface HandlerResult {
196
+ outcome: string;
197
+ data?: Record<string, unknown>;
198
+ }
199
+ ```
200
+ `driver.ts:238–239`: "correlation_token already on the row (set at create-time …). We
201
+ don't auto-generate one here."
202
+
203
+ - **Proposed change.** `HandlerResult.correlationToken?: string | null`, carried on the
204
+ effect-completion advance payload (01 #1 already reshapes it) and applied in
205
+ `runAdvance` at the settle-point write, so the token is in place the moment the row
206
+ becomes `waiting`. `null` clears it. Record it in the `transition` history row's data.
207
+ `create({ correlationToken })` stays as the up-front option.
208
+
209
+ - **Done when** — test: `sendOrderEmail` returns `{ outcome: "SENT", correlationToken:
210
+ "msg-42" }` on an instance created without a token → `find()` shows `msg-42` → a signal
211
+ with that token is delivered and the workflow reaches `_end_ok`.
212
+
213
+ - **Affected files.** `src/types.ts`, `src/driver.ts`, `src/workflow.ts` (payload),
214
+ `API.md`.
215
+
216
+ - **Effort / Value / Risk.** S / med / low. Depends on 01 #1 (payload shape).
217
+
218
+ - **Implementation notes.** An instance with `correlation_token IS NULL` entering a
219
+ `MATCHED`-accepting wait can never be signalled; the validator cannot see it (tokens are
220
+ runtime), so `clog.warn` at that moment in the suspending branch.
221
+
222
+ ## Open questions / decisions needed
223
+
224
+ Resolved 2026-08-26 (see `PROGRESS.md` → Decisions log): inbox rows whose token no live
225
+ instance owns — unknown token, or the instance is terminal — are marked processed on the
226
+ first tick with a warn log, plus a `signal_rejected` history row when a terminal instance
227
+ exists. No grace period. The deferral semantics for early signals stand as proposed in #1
228
+ and #2.