@adonis-agora/durable 0.40.0 → 0.41.1

package/CHANGELOG.md

@@ -1,5 +1,69 @@
 # @adonis-agora/durable
 
+## 0.41.1
+
+### Patch Changes
+
+- [#202](https://github.com/DavideCarvalho/adonis-agora-durable/pull/202) [`573a198`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/573a198ae5e1e69503f3a37d3b9817da07b038c4) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - A run is no longer counted, dead-lettered or executed twice when its turn settles while the orphan sweep is looking at it.
+  
+  Port of the same race fixed in the NestJS core (nestjs-durable#331), seen there as intermittently hung durable chat turns where an in-process worker answers a dispatched step in a couple of milliseconds:
+  
+  - **A turn releases its lease only after its settled state is written.** `runExecution` returned `this.settleRun(...)` (and `this.parkBlocked(...)`) without awaiting it inside `try … finally { releaseRunLock }`, so the `finally` ran first: a first turn that completed, failed or parked `blocked` sat unlocked while its row still read `running`, which is what `listOrphanedRuns` returns for a crashed turn. Every settle in that block is now `return await`ed. (The suspend settle was already awaited.)
+  - **`recoverIncomplete` decides on the run it locked, not the one it listed.** It trusted the listed `running` status: a run that settled between the listing and `tryLockRun` got a recovery attempt counted and, past `maxRecoveryAttempts`, an unconditional `dead` written over a completed or suspended run. It now re-reads the run under the lease and releases it again unless it still reads `running`. The dead-letter write is also conditional on `running`, so a lease-free `cancel()` landing after that re-read is not overwritten with `dead`.
+  - **A replay re-reads a `pending` remote step before acting on it.** With `remoteRedispatchMs` set, a replay that found its step pending in its start-of-execution snapshot stamped the lease by writing `{ ...snapshot, wakeAt }` back, overwriting a result that had landed since with `pending` and parking the run on a lease an hour out. The step is now re-read from the store first; a result that has landed is replayed instead.
+
+## 0.41.0
+
+### Minor Changes
+
+- [#198](https://github.com/DavideCarvalho/adonis-agora-durable/pull/198) [`2b74239`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/2b74239efc6defb62d7f978d8df060384791447a) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Re-drive a `gather_calls` step whose worker was killed mid-step, instead of orphaning it forever.
+  
+  A polyglot workflow's turn declares its fan-out as `call` commands; `applyCommands` writes a
+  `pending` checkpoint and dispatches each one. On every later turn the worker's replay RE-EMITS the
+  calls it is still waiting on, and the engine skips a command whose checkpoint already exists — the
+  guard that stops a partially-settled fan-out from double-dispatching its live siblings.
+  
+  That guard had no notion of a LOST job. A worker killed mid-step (an OOM kill) takes its in-flight
+  job with it, and the transport cannot always put it back: the `bullmq` transport's terminal-failure
+  bridge (`worker.on('failed')`) only exists inside a JS consumer, so a first-class Python fleet (see
+  `docs/python.mdx`) has none; the `queue` transport's stalled-claim sweep gives up past
+  `maxStalledCount`, and never runs at all on an adapter without `recoverStalledJobs`. Nothing then
+  publishes a `StepResult`, so the checkpoint stayed `pending` — "out for delivery" — and every turn
+  skipped it: the run woke on the `reconcileMs` sweep, dispatched a turn, had the calls re-emitted,
+  skipped them, and slept. Forever, with `attempts` never leaving 1 and the checkpoint's `wakeAt` NULL.
+  
+  Those two tells named the gap exactly. `callRemote` (the `ctx.step` path) stamps a re-dispatch
+  deadline on the pending checkpoint's `wakeAt` and honours `remoteRedispatchMs`; the `call` path did
+  neither — so the self-heal `docs/reliability/failure-modes.mdx` documents as acting "on the
+  checkpoint regardless of which transport lost the job" **did not exist for a fan-out**, no matter how
+  it was configured.
+  
+  Now the two paths share one policy, and a pending remote checkpoint's `wakeAt` means the same thing
+  in both: the step's LEASE.
+  
+  - A settled checkpoint always wins — a step that completed just before the crash is never re-run.
+  - `remoteRedispatchMs` unset (still the default) keeps the by-design "re-suspend, never
+    re-dispatch": a merely-slow worker is never double-run.
+  - Set it and the dispatched step carries a lease, and the run suspends ON it (never later than the
+    `reconcileMs` sweep would have woken it anyway). Only a LAPSED lease re-dispatches, bounded by
+    `remoteRedispatchMax` (default 10); past the bound the step is failed `remote_step_lost`, which
+    enters the run's history so the workflow's own error path surfaces it rather than the engine
+    looping.
+  - A step-scoped heartbeat now RENEWS that lease durably, so a worker still holding a long step keeps
+    it — the in-memory rearm only ever protected a `timeoutMs` step, and only on the instance that
+    dispatched it. The renewal is throttled by the lease itself (a write only once the window is half
+    spent), so a worker beating every few seconds costs one read + one write per half
+    `remoteRedispatchMs`, not one per beat — the same concern that throttles the liveness write in
+    `persistHeartbeat`. And it is best-effort end to end: the renewal runs on the transport's SERIAL
+    heartbeat handler, so no read or write of it can reject into that loop — a store hiccup that took the
+    beat loop down would lapse every lease at once.
+  - Observability: a re-drive emits `step.started` with `redispatched: true` and appends a `warn`
+    `step.redispatched` event to the checkpoint's own trail (`step.lost` at the bound), so "re-driven
+    after a lost worker" reads differently from a failure retry — in the dashboard and in the database.
+  
+  No schema change and no new config knob: `remoteRedispatchMs` / `remoteRedispatchMax` (already
+  carried through from `config/durable.ts`) now simply mean what they say for a fan-out too.
+
 ## 0.40.0
 
 ### Minor Changes