@adonis-agora/durable 0.15.2 → 0.16.1

package/CHANGELOG.md

@@ -1,5 +1,95 @@
 # @adonis-agora/durable
 
+## 0.16.1
+
+### Patch Changes
+
+- [#23](https://github.com/DavideCarvalho/adonis-durable/pull/23) [`6e4a3f1`](https://github.com/DavideCarvalho/adonis-durable/commit/6e4a3f12417e7ba30b970f7078e6baf972948549) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Corrige problemas de robustez no engine e no bootstrap:
+
+  - **Efeitos pós-settle agora são aguardados pelo `drain()`.** Depois que um run era persistido no
+    estado terminal, `settleRun` disparava `notifyParent` (que pode acordar e retomar um run PAI
+    suspenso) e o `execute`/`cancel` disparavam o `wakeNext` do singleton — ambos fire-and-forget, fora
+    do conjunto `inflight`. O `drain()` só esperava o `inflight`, então essas escritas no store
+    escapavam da drenagem: em apps que rodam testes sob uma transação global do Lucid, o `updateRun` do
+    pai/singleton disparava após o rollback, virando um unhandled rejection ("Transaction query already
+    complete") depois da suíte passar — CI vermelho com testes verdes. Esses efeitos agora entram num
+    registry `postSettle` que o `drain()` aguarda junto com o `inflight` (em laço, pois um efeito pode
+    retomar outro run), sem que o caminho do `execute()` passe a bloquear neles — o retorno ao chamador
+    continua imediato ao persistir o status.
+
+  - **Mais três efeitos fire-and-forget internos passam a ser aguardados pelo `drain()`.** Eram da mesma
+    classe do `notifyParent`/`wakeNext` — `queueMicrotask(...)` sem tracking, com escrita no store,
+    disparados de dentro de um run que settla/cancela — e pré-existiam desde antes deste PR, expondo
+    consumidores de cancelamento de child, entidades duráveis e compensação saga ao mesmo
+    "Transaction query already complete" pós-teardown que este PR combate:
+
+    - **`ctx.cancelChild`** (usado pelo `ctx.all` failFast p/ cancelar os irmãos sobreviventes): o
+      `cancel()` — que escreve o filho e toda a cascata de cancelamento no store — agora entra no
+      `postSettle` (defer de reentrância preservado, promise registrada de forma síncrona).
+    - **`ctx.signalEntity`** (entidades duráveis): o `entities.dispatch` persiste via `signalWithStart`
+      (createRun/signal do run da entidade), então a op passa a ser aguardada pelo `drain()`.
+    - **Resume da compensação no cancel-compensate** (`engine.cancel({ compensate: true })`): o resume em
+      background que replaya o run e roda as compensações até o `cancelled` terminal era um
+      `queueMicrotask` não-rastreado, e o `resume()` só entra no `inflight` quando de fato é chamado (um
+      microtask depois) — então o `drain()` podia ver os dois registries vazios nessa janela pré-`inflight`
+      e retornar antes das escritas de compensação. Agora o resume deferido é segurado no `postSettle`
+      (não é um `handoffRun`: aqui um run EXISTENTE é retomado, e sua execução já se auto-rastreia no
+      `inflight` — só a janela antes da chamada precisava ser coberta).
+
+  - **Os handoffs internos de run (`continue-as-new` e child deferido) também são aguardados pelo
+    `drain()`.** Antes, ambos entravam por um `queueMicrotask(() => void this.start(...))` fire-and-forget:
+    entre o settle do pai e o novo run entrar no `inflight` havia hops de microtask + I/O de store, e essa
+    ponte ficava fora dos dois registries. O `drain()` podia observar `inflight` e `postSettle` vazios
+    nessa janela e retornar cedo, deixando a persistência (`createRun`) e o processamento do run
+    continuado/filho escaparem — o mesmo hazard "Transaction query already complete" pós-rollback, agora
+    para quem usa continue-as-new/child workflows. Agora cada handoff é registrado no `postSettle` de
+    forma síncrona no settle do pai (via `handoffRun`), preservando o defer de reentrância; com o
+    dispatcher in-process padrão ele mesmo conduz o pickup (`leaseAndResume`, sem o guard de `draining`,
+    pois é trabalho já em voo que o `drain()` aguarda), então a promise só resolve quando o run entrou no
+    `inflight` e settlou. O run fica brevemente nos DOIS conjuntos, o que é inofensivo porque o laço do
+    `drain()` re-snapshota ambos a cada iteração.
+
+  - **`whenBootedApp()` agora falha com mensagem clara em vez de pendurar pra sempre.** O top-level
+    `const app = await whenBootedApp()` do `services/main` ficava pendente SILENCIOSAMENTE se o
+    `DurableProvider` não estivesse registrado nos providers — DX pior que um erro explícito. Agora um
+    timeout (padrão 5s) rejeita com uma mensagem acionável apontando pra adicionar
+    `"@adonis-agora/durable/durable_provider"` no `adonisrc.ts`. O caminho normal não é afetado: o
+    provider registra antes do await, então o fast path devolve uma promise já resolvida (sem timer
+    armado); mesmo quando um timer é armado ele é limpo — e `unref`'d — assim que o app chega.
+
+  - **`services/main` não captura mais o singleton `app` do core de forma eager.** O
+    `import app from '@adonisjs/core/services/app'` no topo do módulo é o mesmo dual-package hazard que
+    já quebrou produção com o `@adonisjs/lucid`: com duas cópias físicas de `@adonisjs/core` na árvore
+    (pnpm/hoist), a cópia importada pode não ser a que o `bin/server` bootou, deixando o `app` como
+    `undefined`. Agora o `DurableProvider` alimenta a instância booted que RECEBE no `register()` para
+    um módulo `services/booted_app`, e o `services/main` lê dali — imune ao split de cópias. O
+    comportamento observável para os consumidores atuais (`import engine`, `import { runGateway }`) é
+    idêntico.
+
+## 0.16.0
+
+### Minor Changes
+
+- [#28](https://github.com/DavideCarvalho/adonis-durable/pull/28) [`024b7b5`](https://github.com/DavideCarvalho/adonis-durable/commit/024b7b54929284f2e88b782d108d2051af3d9d44) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - `static workflow` now accepts `singleton` (a `SingletonConfig`), and `app/workflows` discovery carries it through to `engine.register`. Before this, per-key run serialization was reachable only via a manual `engine.register(name, version, fn, { singleton })` — which forced anyone who needed a mutexed _scheduled_ workflow to bypass the discovery convention entirely, because a colocated `static schedule` fires a new run per window whether or not the previous one is still active. Declaring `singleton: { key: () => '...' }` next to the schedule now serializes those windows natively: excess runs gate (suspended) and admit in creation order when the slot frees.
+
+- [#24](https://github.com/DavideCarvalho/adonis-durable/pull/24) [`390a7c1`](https://github.com/DavideCarvalho/adonis-durable/commit/390a7c120e32f9f859f8b31936eb0a7b8471a63d) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - `defineConfig` now carries `remoteRedispatchMs` / `remoteRedispatchMax` through to the engine. The engine has always implemented this store-driven net for a remote step whose dispatched job was LOST (worker crashed after claiming it, or the transport dropped the job) — but the AdonisJS provider built the engine from an explicit allowlist that omitted both keys, so the net could not be turned on from `config/durable.ts` at all, leaving `engine.redispatchPending(runId)` as the only (manual) recovery.
+
+  Off by default, unchanged semantics: when set, a reconcile pass that finds a remote step still `pending` past the window re-dispatches it, bounded by `remoteRedispatchMax` (default 10). The window must exceed the longest legitimate run of the step, and steps must be idempotent — re-dispatch can double-run a step whose original job is merely slow.
+
+- [#25](https://github.com/DavideCarvalho/adonis-durable/pull/25) [`a0dae13`](https://github.com/DavideCarvalho/adonis-durable/commit/a0dae13be123aebbd29b6cdafcf827b3191e410e) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - The `queue` transport now reclaims stalled jobs. It drives the `@adonisjs/queue` adapter directly (`pushOn`/`popFrom`/`completeJob`) instead of the broker's `Worker` class — but `recoverStalledJobs` was only ever called from that same `Worker` class, so a worker that died after claiming a job left it in the broker's `active` state forever, with no re-delivery and no error (observed in production: jobs orphaned across container restarts, their steps `pending` indefinitely).
+
+  A coarse background sweep (default every 30s) now calls `adapter.recoverStalledJobs` for every queue this instance pops from — per-handler task queues on the worker side, and the results/heartbeats/control queues on the engine side (a dead engine orphans claimed result jobs the same way). New `QueueTransportOptions`: `stalledCheckIntervalMs` (default 30s; `0` disables), `stalledThresholdMs` (default 30min — the claim's `acquiredAt` is never renewed while a worker processes, so the threshold must exceed your longest legitimate step; re-delivery double-runs a merely-slow worker's step, which the durable idempotency contract makes safe but not free), `maxStalledCount` (default 3 — bounds a poison job). Adapters without `recoverStalledJobs` are detected and skipped.
+
+- [#26](https://github.com/DavideCarvalho/adonis-durable/pull/26) [`fc1f9b7`](https://github.com/DavideCarvalho/adonis-durable/commit/fc1f9b7ff33ad0cc4abc4a611401de7545487ec6) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - `durable:runs` now surfaces liveness signals so a `suspended` run's listing stops looking identical whether it's mid-step or stranded. `suspended` is the run's normal resting state while a remote step is in flight — it's also the only symptom a lost dispatch (worker died after claiming the job) ever produces, and nothing auto-redrives it (see `redispatchPending`'s own doc).
+
+  Each row now shows RECOVERY (`recovery_attempts`, blank unless > 0) and PENDING (the age + attempt count of the oldest `pending` REMOTE checkpoint, for `running`/`suspended` runs) alongside the existing columns; UPDATED now renders as a compact duration (`4h32m`) instead of `"4h ago"`.
+
+  A new `--stale[=<duration>]` flag (default threshold 15m, e.g. `--stale=1h`) narrows the listing to runs whose pending remote step exceeds that age — the "these are probably stranded" view — and prints a hint pointing at the two recovery paths that actually exist: `engine.redispatchPending(runId)` and `node ace durable:retry <runId>`.
+
+  The dashboard's `GET /runs` list payload also gains `recoveryAttempts` per run (free — already on the row); the oldest-pending-checkpoint age was deliberately left off that endpoint to avoid an N+1 `listCheckpoints` per row — `GET /runs/:id` already returns the full checkpoint timeline for that.
+
+  New exports from `@adonis-agora/durable/*` command surface: `attachLiveness`, `filterStale`, `parseDurationMs`, `staleHint`, `DEFAULT_STALE_MS`, `RunLiveness`, `StalePendingStep`. `RunLister` now also requires `listCheckpoints` (both `WorkflowEngine` and every `StateStore` already implement it) and `renderRunsTable` now takes `RunLiveness[]` instead of `WorkflowRun[]`.
+
 ## 0.15.2
 
 ### Patch Changes

package/dist/commands/runs.d.ts

@@ -2,8 +2,13 @@ import { BaseCommand } from '@adonisjs/core/ace';
 import type { CommandOptions } from '@adonisjs/core/types/ace';
 /**
  * `node ace durable:runs` — list recent workflow runs, optionally filtered by status and workflow.
- * Reads through the engine's read API (`engine.listRuns`); for the listing to show runs across
- * processes the engine must be backed by a persistent store (config/durable.ts).
+ * Reads through the engine's read API (`engine.listRuns`/`engine.listCheckpoints`); for the listing to
+ * show runs across processes the engine must be backed by a persistent store (config/durable.ts).
+ *
+ * Every row also carries liveness signals (UPDATED age, RECOVERY attempts, PENDING remote-step age) —
+ * `suspended` alone can't tell a run mid-step apart from one whose dispatch was lost (see
+ * `src/commands/runs.ts`'s module doc). `--stale` narrows the listing to exactly the runs that look
+ * stranded by that signal.
  */
 export default class DurableRuns extends BaseCommand {
     static commandName: string;
@@ -12,6 +17,7 @@ export default class DurableRuns extends BaseCommand {
     status?: string;
     workflow?: string;
     limit: number;
+    stale?: string;
     run(): Promise<void>;
 }
 //# sourceMappingURL=runs.d.ts.map

package/dist/commands/runs.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"runs.d.ts","sourceRoot":"","sources":["../../commands/runs.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAS,MAAM,oBAAoB,CAAC;AACxD,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,0BAA0B,CAAC;AAI/D;;;;GAIG;AACH,MAAM,CAAC,OAAO,OAAO,WAAY,SAAQ,WAAW;IAClD,OAAgB,WAAW,SAAkB;IAC7C,OAAgB,WAAW,SAAuC;IAClE,OAAgB,OAAO,EAAE,cAAc,CAAsB;IAKrD,MAAM,CAAC,EAAE,MAAM,CAAC;IAGhB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAGlB,KAAK,EAAE,MAAM,CAAC;IAEP,GAAG,IAAI,OAAO,CAAC,IAAI,CAAC;CASpC"}
+{"version":3,"file":"runs.d.ts","sourceRoot":"","sources":["../../commands/runs.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAS,MAAM,oBAAoB,CAAC;AACxD,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,0BAA0B,CAAC;AAY/D;;;;;;;;;GASG;AACH,MAAM,CAAC,OAAO,OAAO,WAAY,SAAQ,WAAW;IAClD,OAAgB,WAAW,SAAkB;IAC7C,OAAgB,WAAW,SAAuC;IAClE,OAAgB,OAAO,EAAE,cAAc,CAAsB;IAKrD,MAAM,CAAC,EAAE,MAAM,CAAC;IAGhB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAGlB,KAAK,EAAE,MAAM,CAAC;IAQd,KAAK,CAAC,EAAE,MAAM,CAAC;IAER,GAAG,IAAI,OAAO,CAAC,IAAI,CAAC;CA2BpC"}

package/dist/commands/runs.js

@@ -5,12 +5,17 @@ var __decorate = (this && this.__decorate) || function (decorators, target, key,
     return c > 3 && r && Object.defineProperty(target, key, r), r;
 };
 import { BaseCommand, flags } from '@adonisjs/core/ace';
-import { listRuns, renderRunsTable } from '../src/commands/runs.js';
+import { DEFAULT_STALE_MS, attachLiveness, filterStale, listRuns, parseDurationMs, renderRunsTable, staleHint, } from '../src/commands/runs.js';
 import { WorkflowEngine } from '../src/index.js';
 /**
  * `node ace durable:runs` — list recent workflow runs, optionally filtered by status and workflow.
- * Reads through the engine's read API (`engine.listRuns`); for the listing to show runs across
- * processes the engine must be backed by a persistent store (config/durable.ts).
+ * Reads through the engine's read API (`engine.listRuns`/`engine.listCheckpoints`); for the listing to
+ * show runs across processes the engine must be backed by a persistent store (config/durable.ts).
+ *
+ * Every row also carries liveness signals (UPDATED age, RECOVERY attempts, PENDING remote-step age) —
+ * `suspended` alone can't tell a run mid-step apart from one whose dispatch was lost (see
+ * `src/commands/runs.ts`'s module doc). `--stale` narrows the listing to exactly the runs that look
+ * stranded by that signal.
  */
 export default class DurableRuns extends BaseCommand {
     static commandName = 'durable:runs';
@@ -18,12 +23,28 @@ export default class DurableRuns extends BaseCommand {
     static options = { startApp: true };
     async run() {
         const engine = await this.app.container.make(WorkflowEngine);
+        const wantsStale = this.stale !== undefined;
         const runs = await listRuns(engine, {
             ...(this.status ? { status: this.status } : {}),
+            // No explicit --status: scope the query to running/suspended, the only statuses a stranded
+            // dispatch can show up in, so a stale candidate isn't pushed out of the window by newer
+            // terminal runs.
+            ...(!this.status && wantsStale ? { statuses: ['running', 'suspended'] } : {}),
             ...(this.workflow ? { workflow: this.workflow } : {}),
             limit: this.limit,
         });
-        this.logger.log(renderRunsTable(runs));
+        const liveRuns = await attachLiveness(engine, runs);
+        if (!wantsStale) {
+            this.logger.log(renderRunsTable(liveRuns));
+            return;
+        }
+        const thresholdMs = this.stale === ''
+            ? DEFAULT_STALE_MS
+            : (parseDurationMs(this.stale ?? '') ?? DEFAULT_STALE_MS);
+        const stale = filterStale(liveRuns, thresholdMs);
+        this.logger.log(renderRunsTable(stale));
+        for (const { run } of stale)
+            this.logger.warning(staleHint(run.id));
     }
 }
 __decorate([
@@ -37,4 +58,11 @@ __decorate([
 __decorate([
     flags.number({ description: 'Max runs to list', default: 50 })
 ], DurableRuns.prototype, "limit", void 0);
+__decorate([
+    flags.string({
+        description: 'List only running/suspended runs whose oldest pending remote step exceeds this age — the ' +
+            '"probably stranded" view. Pass bare --stale for the default threshold (15m), or a duration ' +
+            'like --stale=1h / --stale=90s for a custom one.',
+    })
+], DurableRuns.prototype, "stale", void 0);
 //# sourceMappingURL=runs.js.map

package/dist/commands/runs.js.map

@@ -1 +1 @@
-{"version":3,"file":"runs.js","sourceRoot":"","sources":["../../commands/runs.ts"],"names":[],"mappings":";;;;;;AAAA,OAAO,EAAE,WAAW,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAExD,OAAO,EAAE,QAAQ,EAAE,eAAe,EAAE,MAAM,yBAAyB,CAAC;AACpE,OAAO,EAAkB,cAAc,EAAE,MAAM,iBAAiB,CAAC;AAEjE;;;;GAIG;AACH,MAAM,CAAC,OAAO,OAAO,WAAY,SAAQ,WAAW;IAClD,MAAM,CAAU,WAAW,GAAG,cAAc,CAAC;IAC7C,MAAM,CAAU,WAAW,GAAG,mCAAmC,CAAC;IAClE,MAAM,CAAU,OAAO,GAAmB,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC;IAapD,KAAK,CAAC,GAAG;QAChB,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAC7D,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,MAAM,EAAE;YAClC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAmB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC5D,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACrD,KAAK,EAAE,IAAI,CAAC,KAAK;SAClB,CAAC,CAAC;QACH,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,eAAe,CAAC,IAAI,CAAC,CAAC,CAAC;IACzC,CAAC;;AAhBO;IAHP,KAAK,CAAC,MAAM,CAAC;QACZ,WAAW,EAAE,8EAA8E;KAC5F,CAAC;2CACsB;AAGhB;IADP,KAAK,CAAC,MAAM,CAAC,EAAE,WAAW,EAAE,yBAAyB,EAAE,CAAC;6CAC/B;AAGlB;IADP,KAAK,CAAC,MAAM,CAAC,EAAE,WAAW,EAAE,kBAAkB,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;0CACzC"}
+{"version":3,"file":"runs.js","sourceRoot":"","sources":["../../commands/runs.ts"],"names":[],"mappings":";;;;;;AAAA,OAAO,EAAE,WAAW,EAAE,KAAK,EAAE,MAAM,oBAAoB,CAAC;AAExD,OAAO,EACL,gBAAgB,EAChB,cAAc,EACd,WAAW,EACX,QAAQ,EACR,eAAe,EACf,eAAe,EACf,SAAS,GACV,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAkB,cAAc,EAAE,MAAM,iBAAiB,CAAC;AAEjE;;;;;;;;;GASG;AACH,MAAM,CAAC,OAAO,OAAO,WAAY,SAAQ,WAAW;IAClD,MAAM,CAAU,WAAW,GAAG,cAAc,CAAC;IAC7C,MAAM,CAAU,WAAW,GAAG,mCAAmC,CAAC;IAClE,MAAM,CAAU,OAAO,GAAmB,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC;IAqBpD,KAAK,CAAC,GAAG;QAChB,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAC7D,MAAM,UAAU,GAAG,IAAI,CAAC,KAAK,KAAK,SAAS,CAAC;QAC5C,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,MAAM,EAAE;YAClC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAmB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC5D,2FAA2F;YAC3F,wFAAwF;YACxF,iBAAiB;YACjB,GAAG,CAAC,CAAC,IAAI,CAAC,MAAM,IAAI,UAAU,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,CAAC,SAAS,EAAE,WAAW,CAAgB,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YAC5F,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;YACrD,KAAK,EAAE,IAAI,CAAC,KAAK;SAClB,CAAC,CAAC;QACH,MAAM,QAAQ,GAAG,MAAM,cAAc,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;QAEpD,IAAI,CAAC,UAAU,EAAE,CAAC;YAChB,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,eAAe,CAAC,QAAQ,CAAC,CAAC,CAAC;YAC3C,OAAO;QACT,CAAC;QAED,MAAM,WAAW,GACf,IAAI,CAAC,KAAK,KAAK,EAAE;YACf,CAAC,CAAC,gBAAgB;YAClB,CAAC,CAAC,CAAC,eAAe,CAAC,IAAI,CAAC,KAAK,IAAI,EAAE,CAAC,IAAI,gBAAgB,CAAC,CAAC;QAC9D,MAAM,KAAK,GAAG,WAAW,CAAC,QAAQ,EAAE,WAAW,CAAC,CAAC;QACjD,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC,CAAC;QACxC,KAAK,MAAM,EAAE,GAAG,EAAE,IAAI,KAAK;YAAE,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC;IACtE,CAAC;;AA1CO;IAHP,KAAK,CAAC,MAAM,CAAC;QACZ,WAAW,EAAE,8EAA8E;KAC5F,CAAC;2CACsB;AAGhB;IADP,KAAK,CAAC,MAAM,CAAC,EAAE,WAAW,EAAE,yBAAyB,EAAE,CAAC;6CAC/B;AAGlB;IADP,KAAK,CAAC,MAAM,CAAC,EAAE,WAAW,EAAE,kBAAkB,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;0CACzC;AAQd;IANP,KAAK,CAAC,MAAM,CAAC;QACZ,WAAW,EACT,2FAA2F;YAC3F,6FAA6F;YAC7F,iDAAiD;KACpD,CAAC;0CACqB"}

package/dist/providers/durable_provider.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"durable_provider.d.ts","sourceRoot":"","sources":["../../providers/durable_provider.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,kBAAkB,EAAE,MAAM,sBAAsB,CAAC;AA8D/D;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AACH,MAAM,CAAC,OAAO,OAAO,eAAe;;IAQtB,SAAS,CAAC,GAAG,EAAE,kBAAkB;gBAAvB,GAAG,EAAE,kBAAkB;IAE7C,QAAQ;IA+LR;;;;;;;;;;;OAWG;IACG,IAAI;IA+IV;;;;;;;OAOG;IACG,KAAK;IASL,QAAQ;CAcf"}
+{"version":3,"file":"durable_provider.d.ts","sourceRoot":"","sources":["../../providers/durable_provider.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,kBAAkB,EAAE,MAAM,sBAAsB,CAAC;AA+D/D;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AACH,MAAM,CAAC,OAAO,OAAO,eAAe;;IAQtB,SAAS,CAAC,GAAG,EAAE,kBAAkB;gBAAvB,GAAG,EAAE,kBAAkB;IAE7C,QAAQ;IA0MR;;;;;;;;;;;OAWG;IACG,IAAI;IA+IV;;;;;;;OAOG;IACG,KAAK;IASL,QAAQ;CAcf"}

package/dist/providers/durable_provider.js

@@ -1,6 +1,7 @@
 import { pathToFileURL } from 'node:url';
 import { InMemoryStateStore, InMemoryTransport, ProxyRunGateway, RunRequestResponder, StoreRunGateway, WorkflowEngine, attachDurableDiagnostics, registerStepsFromBarrel, registerStepsFromDir, registerWorkflowsFromBarrel, registerWorkflowsFromDir, } from '../src/index.js';
 import { DURABLE_RUN_GATEWAY, DURABLE_RUN_REQUEST_RESPONDER, DURABLE_TRANSPORT, DURABLE_WORKER_RUNTIME, NOOP_RUN_DISPATCHER, descriptorRedisFrom, hasProxyCapability, hasResponderCapability, } from '../src/role_bindings.js';
+import { setBootedApp } from '../src/services/booted_app.js';
 import { NoopWorkerRegistry, RedisWorkerRegistry, WorkerRuntime, } from '../src/worker-runtime/index.js';
 const CONTEXT_ACCESSOR = Symbol.for('@agora/context:accessor');
 /**
@@ -49,6 +50,9 @@ export default class DurableProvider {
         this.app = app;
     }
     register() {
+        // Hand the booted app to `services/main` so its eager population reads the SAME @adonisjs/core
+        // copy bin/server booted, never a pnpm-split non-booted one (see services/booted_app.ts).
+        setBootedApp(this.app);
         const config = this.app.config.get('durable', {});
         const role = config.role ?? 'standalone';
         if (role === 'tenant') {
@@ -129,6 +133,14 @@ export default class DurableProvider {
                 ...(config.compensationRetries !== undefined
                     ? { compensationRetries: config.compensationRetries }
                     : {}),
+                // The store-driven lost-dispatch net (see BaseDurableConfig.remoteRedispatchMs) — off by
+                // default, since re-dispatch can double-run a step whose original job is merely slow.
+                ...(config.remoteRedispatchMs !== undefined
+                    ? { remoteRedispatchMs: config.remoteRedispatchMs }
+                    : {}),
+                ...(config.remoteRedispatchMax !== undefined
+                    ? { remoteRedispatchMax: config.remoteRedispatchMax }
+                    : {}),
                 // Where a freshly-started run executes: explicit config wins; else `control-plane` leaves it
                 // pending for the poll loop (no inline execution), and `standalone` uses the engine's in-process
                 // default (embedded worker).

package/dist/providers/durable_provider.js.map

@@ -1 +1 @@
-{"version":3,"file":"durable_provider.js","sourceRoot":"","sources":["../../providers/durable_provider.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AAKzC,OAAO,EAEL,kBAAkB,EAClB,iBAAiB,EACjB,eAAe,EAEf,mBAAmB,EAKnB,eAAe,EAGf,cAAc,EAGd,wBAAwB,EACxB,uBAAuB,EACvB,oBAAoB,EACpB,2BAA2B,EAC3B,wBAAwB,GACzB,MAAM,iBAAiB,CAAC;AACzB,OAAO,EACL,mBAAmB,EACnB,6BAA6B,EAC7B,iBAAiB,EACjB,sBAAsB,EACtB,mBAAmB,EACnB,mBAAmB,EACnB,kBAAkB,EAClB,sBAAsB,GACvB,MAAM,yBAAyB,CAAC;AACjC,OAAO,EACL,kBAAkB,EAClB,mBAAmB,EAEnB,aAAa,GAEd,MAAM,gCAAgC,CAAC;AAMxC,MAAM,gBAAgB,GAAG,MAAM,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;AAE/D;;;;;GAKG;AACH,MAAM,gBAAgB,GAAG,MAAM,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;AAE/D,6GAA6G;AAC7G,MAAM,gBAAgB,GAAG,MAAM,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;AAE/D;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AACH,MAAM,CAAC,OAAO,OAAO,eAAe;IAQZ;IAPtB,kBAAkB,GAAwB,IAAI,CAAC;IAC/C,UAAU,GAAyD,IAAI,CAAC;IACxE,aAAa,GAA4D,IAAI,CAAC;IAC9E,UAAU,GAA+B,IAAI,CAAC;IAC9C,oGAAoG;IACpG,uBAAuB,GAAgE,IAAI,CAAC;IAE5F,YAAsB,GAAuB;QAAvB,QAAG,GAAH,GAAG,CAAoB;IAAG,CAAC;IAEjD,QAAQ;QACN,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAgB,SAAS,EAAE,EAAE,CAAC,CAAC;QACjE,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,YAAY,CAAC;QACzC,IAAI,IAAI,KAAK,QAAQ,EAAE,CAAC;YACtB,IAAI,CAAC,mBAAmB,CAAC,MAAsB,CAAC,CAAC;QACnD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,kBAAkB,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;QACxC,CAAC;IACH,CAAC;IAED,8EAA8E;IAC9E,2CAA2C;IAC3C,8EAA8E;IAE9E;iGAC6F;IAC7F,kBAAkB,CAAC,MAAqB,EAAE,IAAoC;QAC5E,0FAA0F;QAC1F,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,mBAAmB,EAAE,KAAK,IAAI,EAAE;YAC3D,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;YAC7D,OAAO,IAAI,eAAe,CAAC,MAAM,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC;QAC/C,CAAC,CAAC,CAAC;QAEH,gGAAgG;QAChG,qGAAqG;QACrG,sEAAsE;QACtE,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAC1B,6BAA6B,EAC7B,KAAK,IAAyC,EAAE;YAC9C,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;YAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC;YAClC,IAAI,CAAC,sBAAsB,CAAC,SAAS,CAAC;gBAAE,OAAO,IAAI,CAAC;YACpD,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,mBAAmB,CAAC,CAAC;YACnE,MAAM,YAAY,GAAI,MAA6B,CAAC,YAAY,CAAC;YACjE,OAAO,IAAI,mBAAmB,CAAC,SAAS,EAAE,OAAO,EAAE;gBACjD,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACzC,0FAA0F;gBAC1F,qBAAqB,EAAE,CAAC,QAAQ,EAAE,EAAE,CAAC,MAAM,CAAC,SAAS,CAAC,QAAQ,CAAC;aAChE,CAAC,CAAC;QACL,CAAC,CACF,CAAC;QAEF,uGAAuG;QACvG,uGAAuG;QACvG,0GAA0G;QAC1G,wGAAwG;QACxG,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,iBAAiB,EAAE,KAAK,IAAwB,EAAE;YAC7E,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;YAC9C,kGAAkG;YAClG,oCAAoC;YACpC,OAAO,IAAI,CAAC,UAAuB,CAAC;QACtC,CAAC,CAAC,CAAC;QAEH,mGAAmG;QACnG,6FAA6F;QAC7F,iGAAiG;QACjG,uDAAuD;QACvD,IAAI,CAAC,eAAe,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;IACrC,CAAC;IAED;;4GAEwG;IACxG,eAAe,CAAC,MAAqB,EAAE,IAAoC;QACzE,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,cAAc,EAAE,KAAK,IAAI,EAAE;YACtD,MAAM,QAAQ,GAAI,UAAsC,CAAC,gBAAgB,CAE5D,CAAC;YACd,MAAM,eAAe,GAAI,UAAsC,CAAC,gBAAgB,CAEnE,CAAC;YAEd,MAAM,GAAG,GAA0D,EAAE,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC;YACrF,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;YACpD,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,iBAAiB,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;YAC5D,qFAAqF;YACrF,IAAI,CAAC,UAAU,GAAG,SAAS,CAAC;YAC5B,MAAM,YAAY,GAAG,MAAM,IAAI,CAAC,oBAAoB,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;YAClE,0FAA0F;YAC1F,IAAI,CAAC,aAAa,GAAG,YAAY,CAAC;YAElC,MAAM,IAAI,GAAuB;gBAC/B,KAAK;gBACL,SAAS;gBACT,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACzC,GAAG,CAAC,MAAM,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACpE,GAAG,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC/D,GAAG,CAAC,MAAM,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC1E,GAAG,CAAC,MAAM,CAAC,mBAAmB,KAAK,SAAS;oBAC1C,CAAC,CAAC,EAAE,mBAAmB,EAAE,MAAM,CAAC,mBAAmB,EAAE;oBACrD,CAAC,CAAC,EAAE,CAAC;gBACP,GAAG,CAAC,MAAM,CAAC,mBAAmB,KAAK,SAAS;oBAC1C,CAAC,CAAC,EAAE,mBAAmB,EAAE,MAAM,CAAC,mBAAmB,EAAE;oBACrD,CAAC,CAAC,EAAE,CAAC;gBACP,6FAA6F;gBAC7F,iGAAiG;gBACjG,6BAA6B;gBAC7B,GAAG,CAAC,MAAM,CAAC,aAAa;oBACtB,CAAC,CAAC,EAAE,aAAa,EAAE,MAAM,CAAC,aAAa,EAAE;oBACzC,CAAC,CAAC,IAAI,KAAK,eAAe;wBACxB,CAAC,CAAC,EAAE,aAAa,EAAE,mBAAmB,EAAE;wBACxC,CAAC,CAAC,EAAE,CAAC;gBACT,4FAA4F;gBAC5F,yFAAyF;gBACzF,4FAA4F;gBAC5F,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,GAAG,EAAE,CAAC,QAAQ,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACtD,yFAAyF;gBACzF,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,eAAe,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;aAC7D,CAAC;YAEF,OAAO,IAAI,cAAc,CAAC,IAAI,CAAC,CAAC;QAClC,CAAC,CAAC,CAAC;IACL,CAAC;IAED,8EAA8E;IAC9E,yDAAyD;IACzD,8EAA8E;IAE9E;;;;;OAKG;IACH,mBAAmB,CAAC,MAAoB;QACtC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,mBAAmB,EAAE,KAAK,IAAI,EAAE;YAC3D,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,gBAAgB,CAAC,MAAM,CAAC,CAAC;YACtD,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,EAAE,CAAC;gBACnC,MAAM,IAAI,KAAK,CACb,2KAA2K,MAAM,CAAC,SAAS,0BAA0B,CACtN,CAAC;YACJ,CAAC;YACD,OAAO,IAAI,eAAe,CAAC,SAAS,EAAE;gBACpC,SAAS,EAAE,MAAM,CAAC,SAAS;gBAC3B,GAAG,CAAC,MAAM,CAAC,MAAM,EAAE,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC7E,GAAG,CAAC,MAAM,CAAC,gBAAgB,KAAK,SAAS;oBACvC,CAAC,CAAC,EAAE,gBAAgB,EAAE,MAAM,CAAC,gBAAgB,EAAE;oBAC/C,CAAC,CAAC,EAAE,CAAC;aACR,CAAC,CAAC;QACL,CAAC,CAAC,CAAC;QAEH,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,sBAAsB,EAAE,KAAK,IAAI,EAAE;YAC9D,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,gBAAgB,CAAC,MAAM,CAAC,CAAC;YACtD,IAAI,OAAQ,SAAsC,CAAC,MAAM,KAAK,UAAU,EAAE,CAAC;gBACzE,MAAM,IAAI,KAAK,CACb,uHAAuH,MAAM,CAAC,SAAS,YAAY,CACpJ,CAAC;YACJ,CAAC;YACD,OAAO,IAAI,aAAa,CAAC;gBACvB,SAAS,EAAE,SAAuC;gBAClD,SAAS,EAAE,MAAM,CAAC,SAAS;gBAC3B,GAAG,CAAC,MAAM,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC1E,GAAG,CAAC,MAAM,CAAC,UAAU,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC7E,GAAG,CAAC,MAAM,CAAC,YAAY,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,MAAM,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACnF,QAAQ,EAAE,IAAI,CAAC,oBAAoB,CAAC,SAAS,CAAC;aAC/C,CAAC,CAAC;QACL,CAAC,CAAC,CAAC;IACL,CAAC;IAED;;;;OAIG;IACH,oBAAoB,CAAC,SAAoB;QACvC,MAAM,KAAK,GAAG,mBAAmB,CAAC,SAAS,CAAC,CAAC;QAC7C,IAAI,KAAK;YAAE,OAAO,IAAI,mBAAmB,CAAC,KAAK,EAAE,EAAE,cAAc,EAAE,IAAI,EAAE,CAAC,CAAC;QAC3E,oFAAoF;QACpF,oGAAoG;QACpG,4FAA4F;QAC5F,OAAO,IAAI,kBAAkB,EAAE,CAAC;IAClC,CAAC;IAED;8GAC0G;IAC1G,gBAAgB,CAAC,MAAoB;QACnC,IAAI,CAAC,IAAI,CAAC,uBAAuB,EAAE,CAAC;YAClC,IAAI,CAAC,uBAAuB,GAAG,CAAC,KAAK,IAAI,EAAE;gBACzC,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,iBAAiB,CAAC,MAAM,EAAE,EAAE,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;gBAC1E,SAAS,CAAC,YAAY,EAAE,CAAC,MAAM,CAAC,SAAS,IAAI,SAAS,CAAC,CAAC;gBACxD,IAAI,CAAC,UAAU,GAAG,SAAS,CAAC;gBAC5B,OAAO,SAAS,CAAC;YACnB,CAAC,CAAC,EAAE,CAAC;QACP,CAAC;QACD,OAAO,IAAI,CAAC,uBAAuB,CAAC;IACtC,CAAC;IAED,8EAA8E;IAC9E,8DAA8D;IAC9D,8EAA8E;IAE9E;;;;;;;;;;;OAWG;IACH,KAAK,CAAC,IAAI;QACR,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAgB,SAAS,EAAE,EAAE,CAAC,CAAC;QACjE,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,YAAY,CAAC;QACzC,IAAI,IAAI,KAAK,QAAQ;YAAE,OAAO;QAC9B,IAAI,MAAM,CAAC,aAAa,KAAK,KAAK;YAAE,OAAO;QAC3C,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAE7D,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,6BAA6B,EAAE,CAAC;QAC1D,IAAI,MAAM,EAAE,CAAC;YACX,MAAM,2BAA2B,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QACpD,CAAC;aAAM,CAAC;YACN,4EAA4E;YAC5E,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,MAAM,CAAC,aAAa,IAAI,eAAe,CAAC,CAAC;YACvE,MAAM,wBAAwB,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;QAC9C,CAAC;QAED,qGAAqG;QACrG,iGAAiG;QACjG,aAAa;QACb,IAAI,IAAI,KAAK,YAAY;YAAE,MAAM,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,CAAC;QAE7D,MAAM,IAAI,CAAC,eAAe,EAAE,CAAC;IAC/B,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe;QACnB,6FAA6F;QAC7F,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAC9C,IAAI,CAAC,sBAAsB,CAAC,IAAI,CAAC,UAAU,CAAC;YAAE,OAAO;QACrD,MAAM,SAAS,GAAG,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAC9C,6BAA6B,CAC9B,CAA+B,CAAC;QACjC,IAAI,CAAC,SAAS;YAAE,OAAO;QACvB,SAAS,CAAC,KAAK,EAAE,CAAC;QAClB,IAAI,CAAC,UAAU,GAAG,SAAS,CAAC;IAC9B,CAAC;IAED;;;;;;OAMG;IACH,KAAK,CAAC,cAAc,CAAC,MAAqB;QACxC,IAAI,MAAM,CAAC,SAAS,KAAK,KAAK;YAAE,OAAO;QACvC,MAAM,MAAM,GAAG,IAAI,CAAC,UAAsD,CAAC;QAC3E,IAAI,CAAC,MAAM,IAAI,OAAO,MAAM,CAAC,MAAM,KAAK,UAAU;YAAE,OAAO;QAC3D,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,yBAAyB,EAAE,CAAC;QACtD,IAAI,MAAM,EAAE,CAAC;YACX,MAAM,uBAAuB,CAAC,MAAoB,EAAE,MAAM,CAAC,CAAC;YAC5D,OAAO;QACT,CAAC;QACD,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,IAAI,WAAW,CAAC,CAAC;QAC/D,MAAM,oBAAoB,CAAC,MAAoB,EAAE,GAAG,CAAC,CAAC;IACxD,CAAC;IAED,qGAAqG;IACrG,KAAK,CAAC,yBAAyB;QAC7B,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,4BAA4B,CAAC,CAAC;QAC7D,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,CAA4B,CAAC;YAChF,OAAO,GAAG,CAAC,KAAK,IAAI,IAAI,CAAC;QAC3B,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,MAAM,IAAI,GAAI,GAA6B,CAAC,IAAI,CAAC;YACjD,IAAI,IAAI,KAAK,sBAAsB,IAAI,IAAI,KAAK,QAAQ;gBAAE,OAAO,IAAI,CAAC;YACtE,MAAM,GAAG,CAAC;QACZ,CAAC;IACH,CAAC;IAED;;;;;;OAMG;IACH,KAAK,CAAC,6BAA6B;QACjC,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,gCAAgC,CAAC,CAAC;QACjE,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,CAAoC,CAAC;YACxF,OAAO,GAAG,CAAC,SAAS,IAAI,IAAI,CAAC;QAC/B,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,MAAM,IAAI,GAAI,GAA6B,CAAC,IAAI,CAAC;YACjD,IAAI,IAAI,KAAK,sBAAsB,IAAI,IAAI,KAAK,QAAQ;gBAAE,OAAO,IAAI,CAAC;YACtE,MAAM,GAAG,CAAC;QACZ,CAAC;IACH,CAAC;IAED,+FAA+F;IAC/F,KAAK,CAAC,aAAa,CAAC,MAAqB,EAAE,GAAiB;QAC1D,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC;QAC1B,IAAI,CAAC,IAAI;YAAE,OAAO,IAAI,kBAAkB,EAAE,CAAC;QAC3C,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,CAAC;QACtC,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,MAAM,IAAI,KAAK,CACb,oCAAoC,IAAI,wBAAwB,IAAI,iBAAiB,CACtF,CAAC;QACJ,CAAC;QACD,MAAM,KAAK,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,CAAC;QACjC,4EAA4E;QAC5E,uFAAuF;QACvF,6FAA6F;QAC7F,iGAAiG;QACjG,wBAAwB;QACxB,IAAI,MAAM,CAAC,UAAU,KAAK,KAAK,EAAE,CAAC;YAChC,MAAM,KAAK,CAAC,YAAY,EAAE,EAAE,CAAC;QAC/B,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC;IAED,iGAAiG;IACjG,KAAK,CAAC,iBAAiB,CAAC,MAAqB,EAAE,GAAqB;QAClE,MAAM,IAAI,GAAG,MAAM,CAAC,SAAS,CAAC;QAC9B,IAAI,CAAC,IAAI;YAAE,OAAO,IAAI,iBAAiB,EAAE,CAAC;QAC1C,MAAM,OAAO,GAAG,MAAM,CAAC,UAAU,EAAE,CAAC,IAAI,CAAC,CAAC;QAC1C,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,MAAM,IAAI,KAAK,CACb,wCAAwC,IAAI,4BAA4B,IAAI,iBAAiB,CAC9F,CAAC;QACJ,CAAC;QACD,OAAO,OAAO,CAAC,GAAG,CAAC,CAAC;IACtB,CAAC;IAED;;;;OAIG;IACH,KAAK,CAAC,oBAAoB,CACxB,MAAqB,EACrB,GAAwB;QAExB,MAAM,EAAE,GAAG,MAAM,CAAC,YAAY,CAAC;QAC/B,IAAI,CAAC,EAAE;YAAE,OAAO,IAAI,CAAC;QACrB,OAAO,OAAO,EAAE,KAAK,UAAU,CAAC,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;IACjD,CAAC;IAED;;;;;;;OAOG;IACH,KAAK,CAAC,KAAK;QACT,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAgB,SAAS,EAAE,EAAE,CAAC,CAAC;QACjE,IAAI,CAAC,MAAM,CAAC,IAAI,IAAI,YAAY,CAAC,KAAK,QAAQ;YAAE,OAAO;QACvD,MAAM,IAAI,GAAI,UAAsC,CAAC,gBAAgB,CAAC,CAAC;QACvE,IAAI,OAAO,IAAI,KAAK,UAAU;YAAE,OAAO;QACvC,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAC7D,IAAI,CAAC,kBAAkB,GAAG,wBAAwB,CAAC,MAAM,CAAC,CAAC;IAC7D,CAAC;IAED,KAAK,CAAC,QAAQ;QACZ,yEAAyE;QACzE,IAAI,CAAC,UAAU,EAAE,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC;QACvB,IAAI,CAAC,kBAAkB,EAAE,EAAE,CAAC;QAC5B,IAAI,CAAC,kBAAkB,GAAG,IAAI,CAAC;QAC/B,+FAA+F;QAC/F,MAAM,IAAI,CAAC,UAAU,EAAE,KAAK,EAAE,EAAE,CAAC;QACjC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC;QACvB,IAAI,CAAC,uBAAuB,GAAG,IAAI,CAAC;QACpC,sFAAsF;QACtF,MAAM,IAAI,CAAC,aAAa,EAAE,KAAK,EAAE,EAAE,CAAC;QACpC,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC;IAC5B,CAAC;CACF"}
+{"version":3,"file":"durable_provider.js","sourceRoot":"","sources":["../../providers/durable_provider.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AAKzC,OAAO,EAEL,kBAAkB,EAClB,iBAAiB,EACjB,eAAe,EAEf,mBAAmB,EAKnB,eAAe,EAGf,cAAc,EAGd,wBAAwB,EACxB,uBAAuB,EACvB,oBAAoB,EACpB,2BAA2B,EAC3B,wBAAwB,GACzB,MAAM,iBAAiB,CAAC;AACzB,OAAO,EACL,mBAAmB,EACnB,6BAA6B,EAC7B,iBAAiB,EACjB,sBAAsB,EACtB,mBAAmB,EACnB,mBAAmB,EACnB,kBAAkB,EAClB,sBAAsB,GACvB,MAAM,yBAAyB,CAAC;AACjC,OAAO,EAAE,YAAY,EAAE,MAAM,+BAA+B,CAAC;AAC7D,OAAO,EACL,kBAAkB,EAClB,mBAAmB,EAEnB,aAAa,GAEd,MAAM,gCAAgC,CAAC;AAMxC,MAAM,gBAAgB,GAAG,MAAM,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;AAE/D;;;;;GAKG;AACH,MAAM,gBAAgB,GAAG,MAAM,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;AAE/D,6GAA6G;AAC7G,MAAM,gBAAgB,GAAG,MAAM,CAAC,GAAG,CAAC,yBAAyB,CAAC,CAAC;AAE/D;;;;;;;;;;;;;;;;;;;;;;;;GAwBG;AACH,MAAM,CAAC,OAAO,OAAO,eAAe;IAQZ;IAPtB,kBAAkB,GAAwB,IAAI,CAAC;IAC/C,UAAU,GAAyD,IAAI,CAAC;IACxE,aAAa,GAA4D,IAAI,CAAC;IAC9E,UAAU,GAA+B,IAAI,CAAC;IAC9C,oGAAoG;IACpG,uBAAuB,GAAgE,IAAI,CAAC;IAE5F,YAAsB,GAAuB;QAAvB,QAAG,GAAH,GAAG,CAAoB;IAAG,CAAC;IAEjD,QAAQ;QACN,+FAA+F;QAC/F,0FAA0F;QAC1F,YAAY,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QACvB,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAgB,SAAS,EAAE,EAAE,CAAC,CAAC;QACjE,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,YAAY,CAAC;QACzC,IAAI,IAAI,KAAK,QAAQ,EAAE,CAAC;YACtB,IAAI,CAAC,mBAAmB,CAAC,MAAsB,CAAC,CAAC;QACnD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,kBAAkB,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;QACxC,CAAC;IACH,CAAC;IAED,8EAA8E;IAC9E,2CAA2C;IAC3C,8EAA8E;IAE9E;iGAC6F;IAC7F,kBAAkB,CAAC,MAAqB,EAAE,IAAoC;QAC5E,0FAA0F;QAC1F,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,mBAAmB,EAAE,KAAK,IAAI,EAAE;YAC3D,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;YAC7D,OAAO,IAAI,eAAe,CAAC,MAAM,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC;QAC/C,CAAC,CAAC,CAAC;QAEH,gGAAgG;QAChG,qGAAqG;QACrG,sEAAsE;QACtE,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAC1B,6BAA6B,EAC7B,KAAK,IAAyC,EAAE;YAC9C,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;YAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC;YAClC,IAAI,CAAC,sBAAsB,CAAC,SAAS,CAAC;gBAAE,OAAO,IAAI,CAAC;YACpD,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,mBAAmB,CAAC,CAAC;YACnE,MAAM,YAAY,GAAI,MAA6B,CAAC,YAAY,CAAC;YACjE,OAAO,IAAI,mBAAmB,CAAC,SAAS,EAAE,OAAO,EAAE;gBACjD,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACzC,0FAA0F;gBAC1F,qBAAqB,EAAE,CAAC,QAAQ,EAAE,EAAE,CAAC,MAAM,CAAC,SAAS,CAAC,QAAQ,CAAC;aAChE,CAAC,CAAC;QACL,CAAC,CACF,CAAC;QAEF,uGAAuG;QACvG,uGAAuG;QACvG,0GAA0G;QAC1G,wGAAwG;QACxG,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,iBAAiB,EAAE,KAAK,IAAwB,EAAE;YAC7E,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;YAC9C,kGAAkG;YAClG,oCAAoC;YACpC,OAAO,IAAI,CAAC,UAAuB,CAAC;QACtC,CAAC,CAAC,CAAC;QAEH,mGAAmG;QACnG,6FAA6F;QAC7F,iGAAiG;QACjG,uDAAuD;QACvD,IAAI,CAAC,eAAe,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;IACrC,CAAC;IAED;;4GAEwG;IACxG,eAAe,CAAC,MAAqB,EAAE,IAAoC;QACzE,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,cAAc,EAAE,KAAK,IAAI,EAAE;YACtD,MAAM,QAAQ,GAAI,UAAsC,CAAC,gBAAgB,CAE5D,CAAC;YACd,MAAM,eAAe,GAAI,UAAsC,CAAC,gBAAgB,CAEnE,CAAC;YAEd,MAAM,GAAG,GAA0D,EAAE,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC;YACrF,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,aAAa,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;YACpD,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,iBAAiB,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;YAC5D,qFAAqF;YACrF,IAAI,CAAC,UAAU,GAAG,SAAS,CAAC;YAC5B,MAAM,YAAY,GAAG,MAAM,IAAI,CAAC,oBAAoB,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;YAClE,0FAA0F;YAC1F,IAAI,CAAC,aAAa,GAAG,YAAY,CAAC;YAElC,MAAM,IAAI,GAAuB;gBAC/B,KAAK;gBACL,SAAS;gBACT,GAAG,CAAC,YAAY,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACzC,GAAG,CAAC,MAAM,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACpE,GAAG,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC/D,GAAG,CAAC,MAAM,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC1E,GAAG,CAAC,MAAM,CAAC,mBAAmB,KAAK,SAAS;oBAC1C,CAAC,CAAC,EAAE,mBAAmB,EAAE,MAAM,CAAC,mBAAmB,EAAE;oBACrD,CAAC,CAAC,EAAE,CAAC;gBACP,GAAG,CAAC,MAAM,CAAC,mBAAmB,KAAK,SAAS;oBAC1C,CAAC,CAAC,EAAE,mBAAmB,EAAE,MAAM,CAAC,mBAAmB,EAAE;oBACrD,CAAC,CAAC,EAAE,CAAC;gBACP,yFAAyF;gBACzF,sFAAsF;gBACtF,GAAG,CAAC,MAAM,CAAC,kBAAkB,KAAK,SAAS;oBACzC,CAAC,CAAC,EAAE,kBAAkB,EAAE,MAAM,CAAC,kBAAkB,EAAE;oBACnD,CAAC,CAAC,EAAE,CAAC;gBACP,GAAG,CAAC,MAAM,CAAC,mBAAmB,KAAK,SAAS;oBAC1C,CAAC,CAAC,EAAE,mBAAmB,EAAE,MAAM,CAAC,mBAAmB,EAAE;oBACrD,CAAC,CAAC,EAAE,CAAC;gBACP,6FAA6F;gBAC7F,iGAAiG;gBACjG,6BAA6B;gBAC7B,GAAG,CAAC,MAAM,CAAC,aAAa;oBACtB,CAAC,CAAC,EAAE,aAAa,EAAE,MAAM,CAAC,aAAa,EAAE;oBACzC,CAAC,CAAC,IAAI,KAAK,eAAe;wBACxB,CAAC,CAAC,EAAE,aAAa,EAAE,mBAAmB,EAAE;wBACxC,CAAC,CAAC,EAAE,CAAC;gBACT,4FAA4F;gBAC5F,yFAAyF;gBACzF,4FAA4F;gBAC5F,GAAG,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,GAAG,EAAE,CAAC,QAAQ,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACtD,yFAAyF;gBACzF,GAAG,CAAC,eAAe,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,eAAe,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;aAC7D,CAAC;YAEF,OAAO,IAAI,cAAc,CAAC,IAAI,CAAC,CAAC;QAClC,CAAC,CAAC,CAAC;IACL,CAAC;IAED,8EAA8E;IAC9E,yDAAyD;IACzD,8EAA8E;IAE9E;;;;;OAKG;IACH,mBAAmB,CAAC,MAAoB;QACtC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,mBAAmB,EAAE,KAAK,IAAI,EAAE;YAC3D,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,gBAAgB,CAAC,MAAM,CAAC,CAAC;YACtD,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,EAAE,CAAC;gBACnC,MAAM,IAAI,KAAK,CACb,2KAA2K,MAAM,CAAC,SAAS,0BAA0B,CACtN,CAAC;YACJ,CAAC;YACD,OAAO,IAAI,eAAe,CAAC,SAAS,EAAE;gBACpC,SAAS,EAAE,MAAM,CAAC,SAAS;gBAC3B,GAAG,CAAC,MAAM,CAAC,MAAM,EAAE,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC7E,GAAG,CAAC,MAAM,CAAC,gBAAgB,KAAK,SAAS;oBACvC,CAAC,CAAC,EAAE,gBAAgB,EAAE,MAAM,CAAC,gBAAgB,EAAE;oBAC/C,CAAC,CAAC,EAAE,CAAC;aACR,CAAC,CAAC;QACL,CAAC,CAAC,CAAC;QAEH,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,SAAS,CAAC,sBAAsB,EAAE,KAAK,IAAI,EAAE;YAC9D,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,gBAAgB,CAAC,MAAM,CAAC,CAAC;YACtD,IAAI,OAAQ,SAAsC,CAAC,MAAM,KAAK,UAAU,EAAE,CAAC;gBACzE,MAAM,IAAI,KAAK,CACb,uHAAuH,MAAM,CAAC,SAAS,YAAY,CACpJ,CAAC;YACJ,CAAC;YACD,OAAO,IAAI,aAAa,CAAC;gBACvB,SAAS,EAAE,SAAuC;gBAClD,SAAS,EAAE,MAAM,CAAC,SAAS;gBAC3B,GAAG,CAAC,MAAM,CAAC,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC1E,GAAG,CAAC,MAAM,CAAC,UAAU,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,UAAU,EAAE,MAAM,CAAC,UAAU,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC7E,GAAG,CAAC,MAAM,CAAC,YAAY,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,YAAY,EAAE,MAAM,CAAC,YAAY,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBACnF,QAAQ,EAAE,IAAI,CAAC,oBAAoB,CAAC,SAAS,CAAC;aAC/C,CAAC,CAAC;QACL,CAAC,CAAC,CAAC;IACL,CAAC;IAED;;;;OAIG;IACH,oBAAoB,CAAC,SAAoB;QACvC,MAAM,KAAK,GAAG,mBAAmB,CAAC,SAAS,CAAC,CAAC;QAC7C,IAAI,KAAK;YAAE,OAAO,IAAI,mBAAmB,CAAC,KAAK,EAAE,EAAE,cAAc,EAAE,IAAI,EAAE,CAAC,CAAC;QAC3E,oFAAoF;QACpF,oGAAoG;QACpG,4FAA4F;QAC5F,OAAO,IAAI,kBAAkB,EAAE,CAAC;IAClC,CAAC;IAED;8GAC0G;IAC1G,gBAAgB,CAAC,MAAoB;QACnC,IAAI,CAAC,IAAI,CAAC,uBAAuB,EAAE,CAAC;YAClC,IAAI,CAAC,uBAAuB,GAAG,CAAC,KAAK,IAAI,EAAE;gBACzC,MAAM,SAAS,GAAG,MAAM,IAAI,CAAC,iBAAiB,CAAC,MAAM,EAAE,EAAE,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;gBAC1E,SAAS,CAAC,YAAY,EAAE,CAAC,MAAM,CAAC,SAAS,IAAI,SAAS,CAAC,CAAC;gBACxD,IAAI,CAAC,UAAU,GAAG,SAAS,CAAC;gBAC5B,OAAO,SAAS,CAAC;YACnB,CAAC,CAAC,EAAE,CAAC;QACP,CAAC;QACD,OAAO,IAAI,CAAC,uBAAuB,CAAC;IACtC,CAAC;IAED,8EAA8E;IAC9E,8DAA8D;IAC9D,8EAA8E;IAE9E;;;;;;;;;;;OAWG;IACH,KAAK,CAAC,IAAI;QACR,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAgB,SAAS,EAAE,EAAE,CAAC,CAAC;QACjE,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,IAAI,YAAY,CAAC;QACzC,IAAI,IAAI,KAAK,QAAQ;YAAE,OAAO;QAC9B,IAAI,MAAM,CAAC,aAAa,KAAK,KAAK;YAAE,OAAO;QAC3C,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAE7D,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,6BAA6B,EAAE,CAAC;QAC1D,IAAI,MAAM,EAAE,CAAC;YACX,MAAM,2BAA2B,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QACpD,CAAC;aAAM,CAAC;YACN,4EAA4E;YAC5E,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,MAAM,CAAC,aAAa,IAAI,eAAe,CAAC,CAAC;YACvE,MAAM,wBAAwB,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;QAC9C,CAAC;QAED,qGAAqG;QACrG,iGAAiG;QACjG,aAAa;QACb,IAAI,IAAI,KAAK,YAAY;YAAE,MAAM,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,CAAC;QAE7D,MAAM,IAAI,CAAC,eAAe,EAAE,CAAC;IAC/B,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe;QACnB,6FAA6F;QAC7F,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAC9C,IAAI,CAAC,sBAAsB,CAAC,IAAI,CAAC,UAAU,CAAC;YAAE,OAAO;QACrD,MAAM,SAAS,GAAG,CAAC,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAC9C,6BAA6B,CAC9B,CAA+B,CAAC;QACjC,IAAI,CAAC,SAAS;YAAE,OAAO;QACvB,SAAS,CAAC,KAAK,EAAE,CAAC;QAClB,IAAI,CAAC,UAAU,GAAG,SAAS,CAAC;IAC9B,CAAC;IAED;;;;;;OAMG;IACH,KAAK,CAAC,cAAc,CAAC,MAAqB;QACxC,IAAI,MAAM,CAAC,SAAS,KAAK,KAAK;YAAE,OAAO;QACvC,MAAM,MAAM,GAAG,IAAI,CAAC,UAAsD,CAAC;QAC3E,IAAI,CAAC,MAAM,IAAI,OAAO,MAAM,CAAC,MAAM,KAAK,UAAU;YAAE,OAAO;QAC3D,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,yBAAyB,EAAE,CAAC;QACtD,IAAI,MAAM,EAAE,CAAC;YACX,MAAM,uBAAuB,CAAC,MAAoB,EAAE,MAAM,CAAC,CAAC;YAC5D,OAAO;QACT,CAAC;QACD,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,IAAI,WAAW,CAAC,CAAC;QAC/D,MAAM,oBAAoB,CAAC,MAAoB,EAAE,GAAG,CAAC,CAAC;IACxD,CAAC;IAED,qGAAqG;IACrG,KAAK,CAAC,yBAAyB;QAC7B,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,4BAA4B,CAAC,CAAC;QAC7D,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,CAA4B,CAAC;YAChF,OAAO,GAAG,CAAC,KAAK,IAAI,IAAI,CAAC;QAC3B,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,MAAM,IAAI,GAAI,GAA6B,CAAC,IAAI,CAAC;YACjD,IAAI,IAAI,KAAK,sBAAsB,IAAI,IAAI,KAAK,QAAQ;gBAAE,OAAO,IAAI,CAAC;YACtE,MAAM,GAAG,CAAC;QACZ,CAAC;IACH,CAAC;IAED;;;;;;OAMG;IACH,KAAK,CAAC,6BAA6B;QACjC,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,gCAAgC,CAAC,CAAC;QACjE,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,CAAC,MAAM,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,CAAoC,CAAC;YACxF,OAAO,GAAG,CAAC,SAAS,IAAI,IAAI,CAAC;QAC/B,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,MAAM,IAAI,GAAI,GAA6B,CAAC,IAAI,CAAC;YACjD,IAAI,IAAI,KAAK,sBAAsB,IAAI,IAAI,KAAK,QAAQ;gBAAE,OAAO,IAAI,CAAC;YACtE,MAAM,GAAG,CAAC;QACZ,CAAC;IACH,CAAC;IAED,+FAA+F;IAC/F,KAAK,CAAC,aAAa,CAAC,MAAqB,EAAE,GAAiB;QAC1D,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC;QAC1B,IAAI,CAAC,IAAI;YAAE,OAAO,IAAI,kBAAkB,EAAE,CAAC;QAC3C,MAAM,OAAO,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,IAAI,CAAC,CAAC;QACtC,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,MAAM,IAAI,KAAK,CACb,oCAAoC,IAAI,wBAAwB,IAAI,iBAAiB,CACtF,CAAC;QACJ,CAAC;QACD,MAAM,KAAK,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,CAAC;QACjC,4EAA4E;QAC5E,uFAAuF;QACvF,6FAA6F;QAC7F,iGAAiG;QACjG,wBAAwB;QACxB,IAAI,MAAM,CAAC,UAAU,KAAK,KAAK,EAAE,CAAC;YAChC,MAAM,KAAK,CAAC,YAAY,EAAE,EAAE,CAAC;QAC/B,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC;IAED,iGAAiG;IACjG,KAAK,CAAC,iBAAiB,CAAC,MAAqB,EAAE,GAAqB;QAClE,MAAM,IAAI,GAAG,MAAM,CAAC,SAAS,CAAC;QAC9B,IAAI,CAAC,IAAI;YAAE,OAAO,IAAI,iBAAiB,EAAE,CAAC;QAC1C,MAAM,OAAO,GAAG,MAAM,CAAC,UAAU,EAAE,CAAC,IAAI,CAAC,CAAC;QAC1C,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,MAAM,IAAI,KAAK,CACb,wCAAwC,IAAI,4BAA4B,IAAI,iBAAiB,CAC9F,CAAC;QACJ,CAAC;QACD,OAAO,OAAO,CAAC,GAAG,CAAC,CAAC;IACtB,CAAC;IAED;;;;OAIG;IACH,KAAK,CAAC,oBAAoB,CACxB,MAAqB,EACrB,GAAwB;QAExB,MAAM,EAAE,GAAG,MAAM,CAAC,YAAY,CAAC;QAC/B,IAAI,CAAC,EAAE;YAAE,OAAO,IAAI,CAAC;QACrB,OAAO,OAAO,EAAE,KAAK,UAAU,CAAC,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;IACjD,CAAC;IAED;;;;;;;OAOG;IACH,KAAK,CAAC,KAAK;QACT,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAgB,SAAS,EAAE,EAAE,CAAC,CAAC;QACjE,IAAI,CAAC,MAAM,CAAC,IAAI,IAAI,YAAY,CAAC,KAAK,QAAQ;YAAE,OAAO;QACvD,MAAM,IAAI,GAAI,UAAsC,CAAC,gBAAgB,CAAC,CAAC;QACvE,IAAI,OAAO,IAAI,KAAK,UAAU;YAAE,OAAO;QACvC,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAC7D,IAAI,CAAC,kBAAkB,GAAG,wBAAwB,CAAC,MAAM,CAAC,CAAC;IAC7D,CAAC;IAED,KAAK,CAAC,QAAQ;QACZ,yEAAyE;QACzE,IAAI,CAAC,UAAU,EAAE,IAAI,EAAE,CAAC;QACxB,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC;QACvB,IAAI,CAAC,kBAAkB,EAAE,EAAE,CAAC;QAC5B,IAAI,CAAC,kBAAkB,GAAG,IAAI,CAAC;QAC/B,+FAA+F;QAC/F,MAAM,IAAI,CAAC,UAAU,EAAE,KAAK,EAAE,EAAE,CAAC;QACjC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC;QACvB,IAAI,CAAC,uBAAuB,GAAG,IAAI,CAAC;QACpC,sFAAsF;QACtF,MAAM,IAAI,CAAC,aAAa,EAAE,KAAK,EAAE,EAAE,CAAC;QACpC,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC;IAC5B,CAAC;CACF"}

package/dist/src/commands/index.d.ts

@@ -1,3 +1,3 @@
 export { type TickOptions, type TickResult, type WorkerLogger, type WorkerLoopOptions, runTick, runWorkerLoop, } from './worker.js';
-export { type ListRunsOptions, type RunLister, listRuns, renderRunsTable, retryRun, } from './runs.js';
+export { DEFAULT_STALE_MS, type ListRunsOptions, type RunLister, type RunLiveness, type StalePendingStep, attachLiveness, filterStale, listRuns, parseDurationMs, renderRunsTable, retryRun, staleHint, } from './runs.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/src/commands/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/commands/index.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,WAAW,EAChB,KAAK,UAAU,EACf,KAAK,YAAY,EACjB,KAAK,iBAAiB,EACtB,OAAO,EACP,aAAa,GACd,MAAM,aAAa,CAAC;AACrB,OAAO,EACL,KAAK,eAAe,EACpB,KAAK,SAAS,EACd,QAAQ,EACR,eAAe,EACf,QAAQ,GACT,MAAM,WAAW,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/commands/index.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,WAAW,EAChB,KAAK,UAAU,EACf,KAAK,YAAY,EACjB,KAAK,iBAAiB,EACtB,OAAO,EACP,aAAa,GACd,MAAM,aAAa,CAAC;AACrB,OAAO,EACL,gBAAgB,EAChB,KAAK,eAAe,EACpB,KAAK,SAAS,EACd,KAAK,WAAW,EAChB,KAAK,gBAAgB,EACrB,cAAc,EACd,WAAW,EACX,QAAQ,EACR,eAAe,EACf,eAAe,EACf,QAAQ,EACR,SAAS,GACV,MAAM,WAAW,CAAC"}

package/dist/src/commands/index.js

@@ -1,3 +1,3 @@
 export { runTick, runWorkerLoop, } from './worker.js';
-export { listRuns, renderRunsTable, retryRun, } from './runs.js';
+export { DEFAULT_STALE_MS, attachLiveness, filterStale, listRuns, parseDurationMs, renderRunsTable, retryRun, staleHint, } from './runs.js';
 //# sourceMappingURL=index.js.map

package/dist/src/commands/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/commands/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAKL,OAAO,EACP,aAAa,GACd,MAAM,aAAa,CAAC;AACrB,OAAO,EAGL,QAAQ,EACR,eAAe,EACf,QAAQ,GACT,MAAM,WAAW,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../../src/commands/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAKL,OAAO,EACP,aAAa,GACd,MAAM,aAAa,CAAC;AACrB,OAAO,EACL,gBAAgB,EAKhB,cAAc,EACd,WAAW,EACX,QAAQ,EACR,eAAe,EACf,eAAe,EACf,QAAQ,EACR,SAAS,GACV,MAAM,WAAW,CAAC"}

package/dist/src/commands/runs.d.ts

@@ -1,24 +1,75 @@
-import type { RunQuery, RunResult, RunStatus, WorkflowEngine, WorkflowRun } from '../index.js';
+import type { RunQuery, RunResult, RunStatus, StepCheckpoint, WorkflowEngine, WorkflowRun } from '../index.js';
 export interface ListRunsOptions {
     /** Filter by run status (pending | running | suspended | completed | failed | cancelled | dead). */
     status?: RunStatus | undefined;
+    /**
+     * Match any of these statuses (`status IN (...)`) — used internally by the `--stale` filter to scope
+     * the query to `running`/`suspended` (the only statuses that can carry a stranded pending step)
+     * instead of scanning every terminal run. Ignored when `status` is also set (the single status wins,
+     * matching {@link RunQuery}'s own precedence).
+     */
+    statuses?: RunStatus[] | undefined;
     /** Filter by workflow name. */
     workflow?: string | undefined;
     /** Max rows. Default 50. */
     limit?: number | undefined;
 }
-/** Anything that can list runs — both a {@link WorkflowEngine} and a raw `StateStore` satisfy it. */
+/** Anything that can list runs and their checkpoints — both a {@link WorkflowEngine} and a raw
+ *  `StateStore` satisfy it. */
 export interface RunLister {
     listRuns(query: RunQuery): Promise<WorkflowRun[]>;
+    listCheckpoints(runId: string): Promise<StepCheckpoint[]>;
 }
 /** Query recent runs, newest activity first (the source returns its own order). */
 export declare function listRuns(source: RunLister, opts: ListRunsOptions): Promise<WorkflowRun[]>;
-/** Render a runs list as a plain-text table for the terminal. */
-export declare function renderRunsTable(runs: WorkflowRun[]): string;
 /**
  * Retry a run: re-enqueue it for a worker to (re-)execute via the engine's `requeue` (the
  * dispatch-model retry — sets it back to `pending`, clears any stale lease, replays its checkpoints).
  * Returns the enqueued result, or null if the run is unknown.
  */
 export declare function retryRun(engine: WorkflowEngine, runId: string): Promise<RunResult | null>;
+/** The oldest unresolved REMOTE checkpoint of a run — the stranded-dispatch signature when it's old. */
+export interface StalePendingStep {
+    seq: number;
+    name: string;
+    /** How many times this step has been (re-)dispatched. A lost dispatch typically sits at 1 forever. */
+    attempts: number;
+    /** Milliseconds since the step was enqueued to the transport. */
+    ageMs: number;
+}
+/** A run plus the liveness signals that tell "working" apart from "stranded" (see module doc above). */
+export interface RunLiveness {
+    run: WorkflowRun;
+    /** Milliseconds since `run.updatedAt`. */
+    ageMs: number;
+    /** `run.recoveryAttempts`, defaulted to 0 for display convenience. */
+    recoveryAttempts: number;
+    /** Oldest pending remote checkpoint, only computed for `running`/`suspended` runs — every other
+     *  status either has none in flight or is terminal. `null` when there is none. */
+    stalePending: StalePendingStep | null;
+}
+/** Default `--stale` threshold: a remote step pending this long is treated as "probably stranded". */
+export declare const DEFAULT_STALE_MS: number;
+/**
+ * Attach liveness signals to a list of runs. Issues one `listCheckpoints` per `running`/`suspended`
+ * run (bounded by the caller's `limit` — this is a CLI/dashboard read, not a hot path) so the listing
+ * can show the age of an in-flight run's oldest pending remote step, not just its own `updatedAt`.
+ */
+export declare function attachLiveness(source: RunLister, runs: WorkflowRun[], now?: number): Promise<RunLiveness[]>;
+/** Keep only runs whose oldest pending remote step has been unresolved for at least `thresholdMs` —
+ *  the "these are probably stranded" view behind `--stale`. */
+export declare function filterStale(liveRuns: RunLiveness[], thresholdMs?: number): RunLiveness[];
+/** Parse a compact single-unit duration (`90s`, `15m`, `4h`, `2d`) into milliseconds. Returns
+ *  `undefined` for anything else so a fat-fingered `--stale=15mins` in an ops one-liner falls back to
+ *  the default threshold instead of throwing. */
+export declare function parseDurationMs(input: string): number | undefined;
+/** An actionable hint for an operator looking at a stranded run — the two recovery paths that
+ *  actually exist for a lost dispatch (see `redispatchPending`'s own doc: no automatic recovery
+ *  re-drives one). */
+export declare function staleHint(runId: string): string;
+/** Render a runs list as a plain-text table for the terminal. Additive over the original
+ *  RUN/WORKFLOW/STATUS/UPDATED columns: RECOVERY (blank unless `recoveryAttempts > 0`) and PENDING
+ *  (the age + attempts of the oldest pending remote checkpoint, blank outside running/suspended or when
+ *  there is none) — see the liveness module doc above for why these matter. */
+export declare function renderRunsTable(liveRuns: RunLiveness[]): string;
 //# sourceMappingURL=runs.d.ts.map

package/dist/src/commands/runs.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"runs.d.ts","sourceRoot":"","sources":["../../../src/commands/runs.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,QAAQ,EAAE,SAAS,EAAE,SAAS,EAAE,cAAc,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAE/F,MAAM,WAAW,eAAe;IAC9B,oGAAoG;IACpG,MAAM,CAAC,EAAE,SAAS,GAAG,SAAS,CAAC;IAC/B,+BAA+B;IAC/B,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B,4BAA4B;IAC5B,KAAK,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAC5B;AAED,qGAAqG;AACrG,MAAM,WAAW,SAAS;IACxB,QAAQ,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC,CAAC;CACnD;AAED,mFAAmF;AACnF,wBAAsB,QAAQ,CAAC,MAAM,EAAE,SAAS,EAAE,IAAI,EAAE,eAAe,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC,CAM/F;AAED,iEAAiE;AACjE,wBAAgB,eAAe,CAAC,IAAI,EAAE,WAAW,EAAE,GAAG,MAAM,CAI3D;AAED;;;;GAIG;AACH,wBAAsB,QAAQ,CAAC,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC,CAE/F"}
+{"version":3,"file":"runs.d.ts","sourceRoot":"","sources":["../../../src/commands/runs.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACV,QAAQ,EACR,SAAS,EACT,SAAS,EACT,cAAc,EACd,cAAc,EACd,WAAW,EACZ,MAAM,aAAa,CAAC;AAErB,MAAM,WAAW,eAAe;IAC9B,oGAAoG;IACpG,MAAM,CAAC,EAAE,SAAS,GAAG,SAAS,CAAC;IAC/B;;;;;OAKG;IACH,QAAQ,CAAC,EAAE,SAAS,EAAE,GAAG,SAAS,CAAC;IACnC,+BAA+B;IAC/B,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B,4BAA4B;IAC5B,KAAK,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAC5B;AAED;+BAC+B;AAC/B,MAAM,WAAW,SAAS;IACxB,QAAQ,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC,CAAC;IAClD,eAAe,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC;CAC3D;AAED,mFAAmF;AACnF,wBAAsB,QAAQ,CAAC,MAAM,EAAE,SAAS,EAAE,IAAI,EAAE,eAAe,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC,CAM/F;AAED;;;;GAIG;AACH,wBAAsB,QAAQ,CAAC,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC,CAE/F;AAeD,wGAAwG;AACxG,MAAM,WAAW,gBAAgB;IAC/B,GAAG,EAAE,MAAM,CAAC;IACZ,IAAI,EAAE,MAAM,CAAC;IACb,sGAAsG;IACtG,QAAQ,EAAE,MAAM,CAAC;IACjB,iEAAiE;IACjE,KAAK,EAAE,MAAM,CAAC;CACf;AAED,wGAAwG;AACxG,MAAM,WAAW,WAAW;IAC1B,GAAG,EAAE,WAAW,CAAC;IACjB,0CAA0C;IAC1C,KAAK,EAAE,MAAM,CAAC;IACd,sEAAsE;IACtE,gBAAgB,EAAE,MAAM,CAAC;IACzB;sFACkF;IAClF,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC;CACvC;AAED,sGAAsG;AACtG,eAAO,MAAM,gBAAgB,QAAc,CAAC;AAwB5C;;;;GAIG;AACH,wBAAsB,cAAc,CAClC,MAAM,EAAE,SAAS,EACjB,IAAI,EAAE,WAAW,EAAE,EACnB,GAAG,GAAE,MAAmB,GACvB,OAAO,CAAC,WAAW,EAAE,CAAC,CAexB;AAED;+DAC+D;AAC/D,wBAAgB,WAAW,CACzB,QAAQ,EAAE,WAAW,EAAE,EACvB,WAAW,GAAE,MAAyB,GACrC,WAAW,EAAE,CAEf;AAED;;iDAEiD;AACjD,wBAAgB,eAAe,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,GAAG,SAAS,CAOjE;AAED;;sBAEsB;AACtB,wBAAgB,SAAS,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAK/C;AAED;;;+EAG+E;AAC/E,wBAAgB,eAAe,CAAC,QAAQ,EAAE,WAAW,EAAE,GAAG,MAAM,CAW/D"}

package/dist/src/commands/runs.js

@@ -1,18 +1,11 @@
 /** Query recent runs, newest activity first (the source returns its own order). */
 export async function listRuns(source, opts) {
     return source.listRuns({
-        ...(opts.status ? { status: opts.status } : {}),
+        ...(opts.status ? { status: opts.status } : opts.statuses ? { statuses: opts.statuses } : {}),
         ...(opts.workflow ? { workflow: opts.workflow } : {}),
         limit: opts.limit ?? 50,
     });
 }
-/** Render a runs list as a plain-text table for the terminal. */
-export function renderRunsTable(runs) {
-    if (runs.length === 0)
-        return 'No runs.';
-    const rows = runs.map((r) => [r.id, r.workflow, r.status, relTime(r.updatedAt)]);
-    return table(['RUN', 'WORKFLOW', 'STATUS', 'UPDATED'], rows);
-}
 /**
  * Retry a run: re-enqueue it for a worker to (re-)execute via the engine's `requeue` (the
  * dispatch-model retry — sets it back to `pending`, clears any stale lease, replays its checkpoints).
@@ -21,19 +14,106 @@ export function renderRunsTable(runs) {
 export async function retryRun(engine, runId) {
     return engine.requeue(runId);
 }
+/** Default `--stale` threshold: a remote step pending this long is treated as "probably stranded". */
+export const DEFAULT_STALE_MS = 15 * 60_000;
+/** Find the oldest `pending` REMOTE checkpoint of a run, if any — the one data point that separates a
+ *  run mid-step from a run whose dispatch was lost. */
+async function oldestPendingRemoteStep(source, runId, now) {
+    const checkpoints = await source.listCheckpoints(runId);
+    let oldest;
+    for (const cp of checkpoints) {
+        if (cp.kind !== 'remote' || cp.status !== 'pending')
+            continue;
+        if (!oldest || cp.enqueuedAt.getTime() < oldest.enqueuedAt.getTime())
+            oldest = cp;
+    }
+    if (!oldest)
+        return null;
+    return {
+        seq: oldest.seq,
+        name: oldest.name,
+        attempts: oldest.attempts,
+        ageMs: now - oldest.enqueuedAt.getTime(),
+    };
+}
+/**
+ * Attach liveness signals to a list of runs. Issues one `listCheckpoints` per `running`/`suspended`
+ * run (bounded by the caller's `limit` — this is a CLI/dashboard read, not a hot path) so the listing
+ * can show the age of an in-flight run's oldest pending remote step, not just its own `updatedAt`.
+ */
+export async function attachLiveness(source, runs, now = Date.now()) {
+    return Promise.all(runs.map(async (run) => {
+        const stalePending = run.status === 'running' || run.status === 'suspended'
+            ? await oldestPendingRemoteStep(source, run.id, now)
+            : null;
+        return {
+            run,
+            ageMs: now - run.updatedAt.getTime(),
+            recoveryAttempts: run.recoveryAttempts ?? 0,
+            stalePending,
+        };
+    }));
+}
+/** Keep only runs whose oldest pending remote step has been unresolved for at least `thresholdMs` —
+ *  the "these are probably stranded" view behind `--stale`. */
+export function filterStale(liveRuns, thresholdMs = DEFAULT_STALE_MS) {
+    return liveRuns.filter((rl) => rl.stalePending !== null && rl.stalePending.ageMs >= thresholdMs);
+}
+/** Parse a compact single-unit duration (`90s`, `15m`, `4h`, `2d`) into milliseconds. Returns
+ *  `undefined` for anything else so a fat-fingered `--stale=15mins` in an ops one-liner falls back to
+ *  the default threshold instead of throwing. */
+export function parseDurationMs(input) {
+    const match = /^(\d+)(ms|s|m|h|d)$/.exec(input.trim());
+    if (!match)
+        return undefined;
+    const value = Number(match[1]);
+    const unit = match[2];
+    const multiplier = { ms: 1, s: 1000, m: 60_000, h: 3_600_000, d: 86_400_000 }[unit];
+    return value * multiplier;
+}
+/** An actionable hint for an operator looking at a stranded run — the two recovery paths that
+ *  actually exist for a lost dispatch (see `redispatchPending`'s own doc: no automatic recovery
+ *  re-drives one). */
+export function staleHint(runId) {
+    return (`Run ${runId} looks stranded (pending remote step exceeds the threshold). Recover it with ` +
+        `\`engine.redispatchPending('${runId}')\` or \`node ace durable:retry ${runId}\`.`);
+}
+/** Render a runs list as a plain-text table for the terminal. Additive over the original
+ *  RUN/WORKFLOW/STATUS/UPDATED columns: RECOVERY (blank unless `recoveryAttempts > 0`) and PENDING
+ *  (the age + attempts of the oldest pending remote checkpoint, blank outside running/suspended or when
+ *  there is none) — see the liveness module doc above for why these matter. */
+export function renderRunsTable(liveRuns) {
+    if (liveRuns.length === 0)
+        return 'No runs.';
+    const rows = liveRuns.map(({ run, ageMs, recoveryAttempts, stalePending }) => [
+        run.id,
+        run.workflow,
+        run.status,
+        compactAge(ageMs),
+        recoveryAttempts > 0 ? String(recoveryAttempts) : '-',
+        stalePending ? `${compactAge(stalePending.ageMs)} (attempt ${stalePending.attempts})` : '-',
+    ]);
+    return table(['RUN', 'WORKFLOW', 'STATUS', 'UPDATED', 'RECOVERY', 'PENDING'], rows);
+}
 function table(headers, rows) {
     const widths = headers.map((h, i) => Math.max(h.length, ...rows.map((r) => (r[i] ?? '').length)));
     const fmt = (cells) => cells.map((c, i) => c.padEnd(widths[i] ?? 0)).join('  ');
     return [fmt(headers), ...rows.map(fmt)].join('\n');
 }
-function relTime(date) {
-    const s = Math.round((Date.now() - date.getTime()) / 1000);
-    if (s < 60)
-        return `${s}s ago`;
-    if (s < 3600)
-        return `${Math.round(s / 60)}m ago`;
-    if (s < 86_400)
-        return `${Math.round(s / 3600)}h ago`;
-    return `${Math.round(s / 86_400)}d ago`;
+/** Compact duration for the liveness columns, e.g. `4h32m`, `12m3s`, `45s`, `2d6h` — denser than a
+ *  `"4h ago"` phrase so it stays readable next to the RECOVERY/PENDING columns. */
+function compactAge(ms) {
+    const totalSeconds = Math.max(0, Math.round(ms / 1000));
+    const days = Math.floor(totalSeconds / 86_400);
+    const hours = Math.floor((totalSeconds % 86_400) / 3600);
+    const minutes = Math.floor((totalSeconds % 3600) / 60);
+    const seconds = totalSeconds % 60;
+    if (days > 0)
+        return `${days}d${hours}h`;
+    if (hours > 0)
+        return `${hours}h${minutes}m`;
+    if (minutes > 0)
+        return `${minutes}m${seconds}s`;
+    return `${seconds}s`;
 }
 //# sourceMappingURL=runs.js.map

package/dist/src/commands/runs.js.map

@@ -1 +1 @@
-{"version":3,"file":"runs.js","sourceRoot":"","sources":["../../../src/commands/runs.ts"],"names":[],"mappings":"AAgBA,mFAAmF;AACnF,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,MAAiB,EAAE,IAAqB;IACrE,OAAO,MAAM,CAAC,QAAQ,CAAC;QACrB,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC/C,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACrD,KAAK,EAAE,IAAI,CAAC,KAAK,IAAI,EAAE;KACxB,CAAC,CAAC;AACL,CAAC;AAED,iEAAiE;AACjE,MAAM,UAAU,eAAe,CAAC,IAAmB;IACjD,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,UAAU,CAAC;IACzC,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC;IACjF,OAAO,KAAK,CAAC,CAAC,KAAK,EAAE,UAAU,EAAE,QAAQ,EAAE,SAAS,CAAC,EAAE,IAAI,CAAC,CAAC;AAC/D,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,MAAsB,EAAE,KAAa;IAClE,OAAO,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC/B,CAAC;AAED,SAAS,KAAK,CAAC,OAAiB,EAAE,IAAgB;IAChD,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;IAClG,MAAM,GAAG,GAAG,CAAC,KAAe,EAAE,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC1F,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACrD,CAAC;AAED,SAAS,OAAO,CAAC,IAAU;IACzB,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,OAAO,EAAE,CAAC,GAAG,IAAI,CAAC,CAAC;IAC3D,IAAI,CAAC,GAAG,EAAE;QAAE,OAAO,GAAG,CAAC,OAAO,CAAC;IAC/B,IAAI,CAAC,GAAG,IAAI;QAAE,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,CAAC,OAAO,CAAC;IAClD,IAAI,CAAC,GAAG,MAAM;QAAE,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC;IACtD,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,MAAM,CAAC,OAAO,CAAC;AAC1C,CAAC"}
+{"version":3,"file":"runs.js","sourceRoot":"","sources":["../../../src/commands/runs.ts"],"names":[],"mappings":"AAgCA,mFAAmF;AACnF,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,MAAiB,EAAE,IAAqB;IACrE,OAAO,MAAM,CAAC,QAAQ,CAAC;QACrB,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC7F,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACrD,KAAK,EAAE,IAAI,CAAC,KAAK,IAAI,EAAE;KACxB,CAAC,CAAC;AACL,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,MAAsB,EAAE,KAAa;IAClE,OAAO,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC/B,CAAC;AAqCD,sGAAsG;AACtG,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,GAAG,MAAM,CAAC;AAE5C;uDACuD;AACvD,KAAK,UAAU,uBAAuB,CACpC,MAAiB,EACjB,KAAa,EACb,GAAW;IAEX,MAAM,WAAW,GAAG,MAAM,MAAM,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC;IACxD,IAAI,MAAkC,CAAC;IACvC,KAAK,MAAM,EAAE,IAAI,WAAW,EAAE,CAAC;QAC7B,IAAI,EAAE,CAAC,IAAI,KAAK,QAAQ,IAAI,EAAE,CAAC,MAAM,KAAK,SAAS;YAAE,SAAS;QAC9D,IAAI,CAAC,MAAM,IAAI,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,GAAG,MAAM,CAAC,UAAU,CAAC,OAAO,EAAE;YAAE,MAAM,GAAG,EAAE,CAAC;IACpF,CAAC;IACD,IAAI,CAAC,MAAM;QAAE,OAAO,IAAI,CAAC;IACzB,OAAO;QACL,GAAG,EAAE,MAAM,CAAC,GAAG;QACf,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,QAAQ,EAAE,MAAM,CAAC,QAAQ;QACzB,KAAK,EAAE,GAAG,GAAG,MAAM,CAAC,UAAU,CAAC,OAAO,EAAE;KACzC,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,MAAiB,EACjB,IAAmB,EACnB,MAAc,IAAI,CAAC,GAAG,EAAE;IAExB,OAAO,OAAO,CAAC,GAAG,CAChB,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,GAAG,EAAE,EAAE;QACrB,MAAM,YAAY,GAChB,GAAG,CAAC,MAAM,KAAK,SAAS,IAAI,GAAG,CAAC,MAAM,KAAK,WAAW;YACpD,CAAC,CAAC,MAAM,uBAAuB,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,EAAE,GAAG,CAAC;YACpD,CAAC,CAAC,IAAI,CAAC;QACX,OAAO;YACL,GAAG;YACH,KAAK,EAAE,GAAG,GAAG,GAAG,CAAC,SAAS,CAAC,OAAO,EAAE;YACpC,gBAAgB,EAAE,GAAG,CAAC,gBAAgB,IAAI,CAAC;YAC3C,YAAY;SACb,CAAC;IACJ,CAAC,CAAC,CACH,CAAC;AACJ,CAAC;AAED;+DAC+D;AAC/D,MAAM,UAAU,WAAW,CACzB,QAAuB,EACvB,cAAsB,gBAAgB;IAEtC,OAAO,QAAQ,CAAC,MAAM,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,YAAY,KAAK,IAAI,IAAI,EAAE,CAAC,YAAY,CAAC,KAAK,IAAI,WAAW,CAAC,CAAC;AACnG,CAAC;AAED;;iDAEiD;AACjD,MAAM,UAAU,eAAe,CAAC,KAAa;IAC3C,MAAM,KAAK,GAAG,qBAAqB,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC;IACvD,IAAI,CAAC,KAAK;QAAE,OAAO,SAAS,CAAC;IAC7B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IAC/B,MAAM,IAAI,GAAG,KAAK,CAAC,CAAC,CAAiC,CAAC;IACtD,MAAM,UAAU,GAAG,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,UAAU,EAAE,CAAC,IAAI,CAAC,CAAC;IACpF,OAAO,KAAK,GAAG,UAAU,CAAC;AAC5B,CAAC;AAED;;sBAEsB;AACtB,MAAM,UAAU,SAAS,CAAC,KAAa;IACrC,OAAO,CACL,OAAO,KAAK,+EAA+E;QAC3F,+BAA+B,KAAK,oCAAoC,KAAK,KAAK,CACnF,CAAC;AACJ,CAAC;AAED;;;+EAG+E;AAC/E,MAAM,UAAU,eAAe,CAAC,QAAuB;IACrD,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,UAAU,CAAC;IAC7C,MAAM,IAAI,GAAG,QAAQ,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,KAAK,EAAE,gBAAgB,EAAE,YAAY,EAAE,EAAE,EAAE,CAAC;QAC5E,GAAG,CAAC,EAAE;QACN,GAAG,CAAC,QAAQ;QACZ,GAAG,CAAC,MAAM;QACV,UAAU,CAAC,KAAK,CAAC;QACjB,gBAAgB,GAAG,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC,GAAG;QACrD,YAAY,CAAC,CAAC,CAAC,GAAG,UAAU,CAAC,YAAY,CAAC,KAAK,CAAC,aAAa,YAAY,CAAC,QAAQ,GAAG,CAAC,CAAC,CAAC,GAAG;KAC5F,CAAC,CAAC;IACH,OAAO,KAAK,CAAC,CAAC,KAAK,EAAE,UAAU,EAAE,QAAQ,EAAE,SAAS,EAAE,UAAU,EAAE,SAAS,CAAC,EAAE,IAAI,CAAC,CAAC;AACtF,CAAC;AAED,SAAS,KAAK,CAAC,OAAiB,EAAE,IAAgB;IAChD,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;IAClG,MAAM,GAAG,GAAG,CAAC,KAAe,EAAE,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC1F,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACrD,CAAC;AAED;mFACmF;AACnF,SAAS,UAAU,CAAC,EAAU;IAC5B,MAAM,YAAY,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,EAAE,GAAG,IAAI,CAAC,CAAC,CAAC;IACxD,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,GAAG,MAAM,CAAC,CAAC;IAC/C,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,YAAY,GAAG,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC;IACzD,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,YAAY,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;IACvD,MAAM,OAAO,GAAG,YAAY,GAAG,EAAE,CAAC;IAClC,IAAI,IAAI,GAAG,CAAC;QAAE,OAAO,GAAG,IAAI,IAAI,KAAK,GAAG,CAAC;IACzC,IAAI,KAAK,GAAG,CAAC;QAAE,OAAO,GAAG,KAAK,IAAI,OAAO,GAAG,CAAC;IAC7C,IAAI,OAAO,GAAG,CAAC;QAAE,OAAO,GAAG,OAAO,IAAI,OAAO,GAAG,CAAC;IACjD,OAAO,GAAG,OAAO,GAAG,CAAC;AACvB,CAAC"}

package/dist/src/dashboard/handlers.js

@@ -141,18 +141,26 @@ function summarizeRun(run) {
         tags: run.tags ?? [],
         createdAt: run.createdAt.toISOString(),
         updatedAt: run.updatedAt.toISOString(),
+        // Liveness signal #1 of 2 (see `src/commands/runs.ts`'s module doc): `suspended` alone can't tell
+        // a run mid-step apart from one stuck on a lost dispatch. `updatedAt` (above) already lets a
+        // client derive age; `recoveryAttempts` is the other cheap tell — it's already on the run row, so
+        // exposing it here is free. The other signal, the oldest pending REMOTE checkpoint's age, is
+        // deliberately NOT added to this list endpoint: it would need one `listCheckpoints` call per row
+        // (an N+1 the list view doesn't currently pay), whereas `GET /runs/:id` already returns the full
+        // checkpoint `timeline` a client can scan for it.
+        recoveryAttempts: run.recoveryAttempts ?? 0,
     };
 }
 /** Fuller run shape for the detail view. */
 function detailRun(run) {
     return {
+        // recoveryAttempts already comes through from summarizeRun.
         ...summarizeRun(run),
         input: run.input,
         output: run.output,
         error: run.error,
         searchAttributes: run.searchAttributes,
         wakeAt: run.wakeAt,
-        recoveryAttempts: run.recoveryAttempts,
     };
 }
 /** Compact checkpoint shape for the timeline. */