@adonis-agora/durable 0.49.0 → 0.50.0

package/CHANGELOG.md

@@ -1,5 +1,266 @@
 # @adonis-agora/durable
 
+## 0.50.0
+
+### Minor Changes
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`de4aee3`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/de4aee3583e4bb0f86fea14e3075f62e803bbea3) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Singleton admission is now an atomic, durable store claim. The gate used to re-scan the key's `running`/`suspended` runs by `(createdAt, id)` on every check, so two replicas could admit past `limit`, an older run picked up after a younger one was admitted alongside it, a same-millisecond start whose id sorted first displaced the holder, and a denial landing after a concurrent `cancel` wrote the run back to `suspended` (resurrecting it).
+  
+  - New optional `StateStore.tryAdmitSingleton(runId, tag, workflow, limit, retryWakeAt?)`. In one transaction it reads the key's active (`pending`/`running`/`suspended`/`blocked`) runs and claims a slot for the oldest waiter, or parks a denied run `suspended` on its retry timer (never a run that is no longer active). The Lucid store locks the rows with `SELECT … FOR UPDATE` on Postgres/MySQL and takes the write lock first on SQLite. The in-memory and `CodecStateStore` stores support it too.
+  - A slot is recorded on the run as the reserved tag `durable:singleton:admitted`. It is held through sleeps, signals, `blocked`, recovery on another instance and an expired lease, and released by the terminal settle (`updateRun`/`updateRunIf` drop it). Fresh runs can't supply or inherit the tag (`retryWithInput` copies tags), and `requeue` drops a stale one.
+  - Notify-on-release no longer wakes a sleeping holder. `maxQueueDepth` also counts `blocked` runs.
+  - The decision helpers `singletonAdmissionTags`, `singletonAdmissionDenialPatch`, `SINGLETON_ADMITTED_TAG` and `SINGLETON_ACTIVE_STATUSES` are exported for custom stores, and the state-store conformance suite covers the claim.
+  
+  No schema migration, and not breaking: a custom store without `tryAdmitSingleton` keeps the previous scan-based admission (not atomic) and the engine prints a one-time warning naming `StateStore.tryAdmitSingleton`. Implement it to get atomic admission. Runs already in flight carry no admission tag. The claim counts the ones an old replica's scan treats as admitted as holders, so old and new replicas can be mixed during a rolling upgrade.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`522f51a`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/522f51a5d0231ff4ecc5f15ce61a60a86982d0de) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - BullMQ, run-gateway and observability fixes and features ported from the NestJS library.
+  
+  **BullMQ transport**
+  - Pub/sub subscribers heal after a silent connection loss. The heartbeat, run-reply and tenant-event subscribers never write on their own, so a VPN/NAT/idle timeout that dropped their TCP connection left them deaf (Redis had dropped the subscription while ioredis still reported `ready`) until the process restarted. A shared watchdog now PINGs each one every `pingIntervalMs` (default `30_000`, `0`/`false` disables) and reconnects a dead one, so ioredis resubscribes. Each subscriber also gets an `error` listener (the first error of a burst goes to `onError`) instead of risking an unhandled `error` event.
+  - New opt-in `stepTimeoutMs`: a step handler still running at the ceiling is failed RETRYABLE, so a wedged `await` can no longer hold its job forever (BullMQ renews the lock on a timer, so such a job never stalls).
+  - New `readAllWorkerDescriptors()` / `readAllWorkerHeartbeats()` fleet-wide reads.
+  
+  **Run gateway**
+  - `RunRequestResponder` always replies. A gateway that threw in `listRuns`, `workerHealth` or the pre-verb ownership read, or a tenant verifier that threw, left the request unanswered and the tenant waited out `requestTimeoutMs`. An unknown verb is now answered with the `unsupported-request` code before the gateway is touched.
+  - Run detail survives version skew: against a control plane older than `getRunDetail` / `getFanMembers`, `ProxyRunGateway` builds the detail and fan pages from `getRun` + `getCheckpoints` + `getRunChildren` and summarizes on the tenant, remembering the old control plane for a minute.
+  - New gateway verbs, in the NestJS shapes: `retry`, `continue`, `retryWithInput`, `runFacets`, `runValueFacets`, `waitingFor` (optional on the interface, implemented by both gateways, tenant-scoped by the responder). A tenant pod's console now retries in place, continues breakpoints, fixes and replays, fills its value pickers server-side and names what a suspended run waits on.
+  
+  **Observability**
+  - A fleet with a control plane counts each lifecycle event once. Every pod used to forward the events it heard from other pods too, so Telescope, `collectMetrics`, `attachDurableMetrics` and `attachDurableOtel` counted (or traced) each event once per pod. They now handle only local events (`isRemoteEvent(event)`; `attachDurableDiagnostics(engine, { includeRemote: true })` restores the old behaviour), and Telescope's entry-backed providers collapse copies already recorded (`dedupeRunEvents`).
+  - Telescope: the state-breakdown donut showed completed in red and failed in blue; the "Starved worker groups" panel is renamed "Worker health"; a new per-instance **Workers** table (`durable.workerStatus`).
+  - Dashboard: `GET <path>/api/metrics` (Prometheus counters plus backlog and worker gauges), `GET <path>/api/runs/:id/events/:key` (the latest `ctx.setEvent` value) and `POST <path>/api/webhooks/:token` (resumes a run parked on a `ctx.webhook()`; webhook tokens only, only while a run waits on it, behind the console's guard).
+  - `currentStep()`: the step logger, reachable from code a step calls without being handed it, on local and remote steps.
+  
+  **Engine**
+  - `StartOptions.version` pins a start to a registered version and throws for an unregistered one, never falling back to the newest.
+  - `engine.remote(name, { group, version? })` registers a remote workflow without building the executor by hand.
+  - `engine.workflowDirectory()` / `engine.announcedWorkflows()`: every workflow the live fleet can run in this engine's partition, declared or observed, and why the list is empty when it is.
+
+- [#358](https://github.com/DavideCarvalho/adonis-agora-durable/pull/358) [`43a7c70`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/43a7c70afbf63d6585e407c88412b26cd42176d3) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - The `db` transport no longer runs a step twice when a step outlives `leaseMs`. It also gains `concurrency`, `groupHealth`, a poison-task bound, hand-back on close and dispatch priority, as on the `queue` transport.
+  
+  **Fix.** The task loop claimed up to `batchSize` (20) rows per poll, ran them one at a time, and renewed only the row it was running. The rest of the batch kept its claim-time timestamp. When an earlier step outlived `leaseMs`, an idle worker reclaimed those rows and ran them, and the first worker then ran them again and wrote a second result. One worker could also hold 20 rows while others sat idle. The results, heartbeat and control loops had the same flaw: a slow result handler let another engine take the rest of its batch. Now:
+  
+  - the task loop claims only as many rows as it has free slots;
+  - every held claim is renewed every `leaseMs / 3`;
+  - a row runs only after its claim is re-checked;
+  - the task delete and result insert happen in one transaction, and only while `claimed_by` is still the worker's token. A worker that lost its claim drops its result and reports it to `onError`.
+  
+  **New options on `transports.db()` / `DbTransport`:**
+  
+  - `concurrency`: a number (default 1), `'adaptive'`, or `{ mode: 'adaptive', ... }`. It sets how many task rows a worker runs at once. One poll loop serves every handler name, so more slots add no polling. It uses the same adaptive controller as `queue` and `bullmq`.
+  - `resultConcurrency` (default 1): how many results the engine side settles at once.
+  - `idleBackoffMax` (default 4): idle task, heartbeat and control loops double their sleep up to this many poll intervals. The results loop never backs off.
+  - `maxStalledCount` (default 3): past this many takeovers of a lapsed claim, the row is deleted. A retryable failed result (`remote step worker failed: claim lapsed more than N times`) is published, so the engine's retry policy decides what happens next. Before, such a row was re-delivered forever.
+  - `closeTimeoutMs` (default 5000): after this drain, `close()` hands every claim it still holds back. That covers running tasks and any result, heartbeat or control batch. Each claim is cleared only where it is still the worker's own, and not counted, so another worker picks the row up at once instead of after `leaseMs`. A deploy therefore never pushes a long step toward the poison bound. A worker that dies without closing still leaves a lapsed claim, and that still counts.
+  - Dispatch priority: the engine's `priority` (higher = more urgent) is now stored on the task row, and rows are claimed highest priority first, then oldest. An absent priority is `0`, "normal", which is where the `queue` transport places an unprioritised job too.
+  - `workerHeartbeatIntervalMs` (default 10s) and `onError` (default `console.error`).
+  
+  **Worker health.** Workers now beat `durable-worker-heartbeat:<token>:<instance>` keys carrying their `WorkerStatus` into `durable_transport_worker_keys`. `DbTransport` implements `groupHealth`, so `engine.workerHealth()`, the dashboard and Telescope report db groups. `depth` counts pending and claimed task rows.
+  
+  **Schema.** `durable_transport_tasks` gets two columns, `reclaims` and `priority` (both integer, not null, default 0). `autoCreate` (on by default) and `createDurableTransportTables` add them to existing tables, so no new migration is needed. The published migration stub includes them for new apps. With `autoCreate: false`, add them yourself. Until you do, tasks run as before: without `reclaims` the poison bound is off, without `priority` rows are claimed oldest first, and the transport reports each missing column once through `onError`.
+  
+  **Behaviour change.** An unconfigured worker used to claim up to 20 rows per poll. It now claims one row at a time (default `concurrency: 1`). Raise `concurrency` to run steps in parallel.
+
+- [#358](https://github.com/DavideCarvalho/adonis-agora-durable/pull/358) [`cf3bd6f`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/cf3bd6f3d4c5b4e513f46b833dd6733226accfd8) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - The `db` transport now carries remote workflow turns and store-less `tenant` pods, which used to need `bullmq` or `queue`.
+  
+  - Remote workflows: `dispatchWorkflowTask` / `handleWorkflow` / `onDecision` and streamed step events. A turn is a task row of kind `workflow` on the workflow's routing token, next to the steps of the same name. It is claimed by priority, runs in a `concurrency` slot, and is renewed, handed back on close and bounded by `maxStalledCount` like a step. Its decision is a row addressed to the engine instance that dispatched the turn, so with several engine replicas a decision never lands on one that is not waiting for it. Every `onDecision` handler on that instance sees it.
+  - Tenant pods: run starts and read/control requests are rows any control-plane instance may take. A reply is a row addressed to the requester only. A live-tail event is one row per pod subscribed to its tenant. A pod on `db` reaches the control plane's database by definition, so it writes its own subscription and inbox keys to `durable_transport_worker_keys` and nothing is relayed through the control plane. A janitor on every instance that addresses rows deletes rows older than `inboxTtlMs` whose recipient is gone. New options: `inboxTtlMs`, `tenantSubscribersCacheMs`, `inboxSweepIntervalMs`.
+  - Schema: a new `durable_transport_messages` table and a `kind` column on `durable_transport_tasks` (default `'step'`). `autoCreate` adds both. `node ace configure` publishes a new `create_durable_transport_messages` migration, which creates the table and adds `kind` to an existing tasks table. Without them, steps run as before; the transport says once through `onError` that the protocol is off the first time something uses it, and run starts, requests and remote turns fail at once instead of timing out.
+  - `autoCreate` no longer fails when several processes create the transport tables at the same moment: a `CREATE` or `ALTER` that loses the race to another process now counts as success. With four processes booting at once on an empty Postgres schema, all but one used to fail.
+  - A `standalone` or `control-plane` engine on `db` now starts the run-request responder, as it already did on `bullmq` and `queue`: it polls the start-run and run-request rows (with idle backoff), and logs a warning at boot unless `verifyTenant` is set.
+  
+  Compared with `bullmq`, replies and events are polled rather than pushed, a new subscription can miss events published in its first second or so, and each event costs one row per subscribed pod. The rows are this package's own wire: Python and NestJS workers speak `bullmq` only.
+
+- [#358](https://github.com/DavideCarvalho/adonis-agora-durable/pull/358) [`9710315`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/971031593d1b6954a1fd12d4d3b2f23f5f642d06) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - `consumers` (`'auto'` / `'never'` / `'always'`) now works on the `bullmq` and `db` transports, not only `queue`.
+  
+  Both started consuming in every process that booted them. A `node ace` command or REPL could claim step jobs and die holding them, and could take results meant for the long-lived engine — the bug the `queue` transport fixed in 0.17. With `bullmq` a claimed job stays held until BullMQ's stall check; with `db`, until the row lease expires.
+  
+  They now implement `deferConsumers()` / `startConsumers()`:
+  
+  - `bullmq` creates no `Worker` until consumption starts: the task, results, decisions, step-events, start-run and run-request workers. Their `failed` bridges and adaptive `concurrency` assignments are replayed onto the real workers. Pub/sub subscriptions (heartbeats, run replies, tenant events) are broadcast and still start right away.
+  - `db` parks its task, result, heartbeat and control poll loops.
+  
+  With the default `'auto'`, console and REPL processes are pure producers on every broker transport, and `durable:work` re-enables consumption for itself.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`623415e`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/623415e8beab4144f6c8a2fb80d951fdc945c48a) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - New `config/durable.ts` options and one behaviour change.
+  
+  - **`drive: false`**: the process never executes the runs it starts, requeues or recovers; they stay `pending` for `node ace durable:work` (which drives through its poll whatever the value) or a control plane. Pair it with `consumers: 'never'` on web pods whose workflow bodies are heavy, instead of a hand-written no-op `runDispatcher`. An explicit `runDispatcher` still wins, and `drive: false` together with `worker: { embedded: true }` throws at boot.
+  - **Drain on shutdown, always.** The provider now drains the engine on every app shutdown before it closes the transport and control plane, not only when the embedded worker runs. A web process mid-turn no longer has its transport closed under it. The budget is `worker.drainTimeoutMs` when set, else the new `drainTimeoutMs` (default 10 000), and the embedded loop is not drained twice. The `app.terminating(...)` hook the production docs used to suggest is no longer needed.
+  - **`queues`**: flow-control queues declared in config, registered on the engine at boot (`engine.registerQueue` shape). A queue named twice throws.
+  - **`deadLetterWorkflow`**: a fleet-default dead-letter workflow in config, a per-workflow `deadLetterWorkflow` (`static workflow`, `register`/`registerRemote` options), or an inline `deadLetter(ctx, dead)` method on a `BaseWorkflow` (registered as `<name>.dlq`). A run that goes `dead` starts it as `dlq:<runId>` with a `DeadLetter` input (`{ deadRunId, workflow, input, error }`), in the dead run's namespace. Declaring both the method and the option throws at boot.
+  - **Retention by count**: a `retention.policies` entry takes `maxCount` (keep the N newest runs of its status set within its scope), alone or with `maxAge` (a run goes once it is past either). A policy with neither now throws at boot.
+  
+  **Behaviour change: an absent `namespace` is the `'default'` pool on a standalone engine.** It used to make the engine an operator that drove, recovered and resumed the runs of every namespace, so a standalone process with no namespace sharing a store with other pools (a laptop pointed at a shared database, a second app) would execute their runs. It now drives only `'default'` runs and rows written before namespaces existed (which read back as `'default'`). Only multi-pool deployments sharing one store are affected: a standalone process that was meant to drive every pool must become a `role: 'control-plane'`, which keeps the operator meaning when its namespace is unset. `scopeReads: true` with no `namespace` on a standalone engine now fences reads to `'default'` (it fenced nothing before).
+  
+  The `config/durable_dashboard.ts` stub now says what the dashboard actually does (it fails closed in every environment unless `DURABLE_DASHBOARD_TOKEN`, `authorize`, `dashboardAuth` or `allowUnauthenticated` says otherwise), and carries the `#start/env` import its commented example needs.
+
+- [#363](https://github.com/DavideCarvalho/adonis-agora-durable/pull/363) [`52f1200`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/52f1200429bd354dac7cfcfe03d9b50ff3ea85c8) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Workflow and step discovery can scan several directories, for modular apps that keep one folder per bounded context. `workflowsPath` and `stepsPath` in `config/durable.ts` now accept `string | string[] | false`, and an empty array is the same as `false`. The provider's runtime scan and `node ace durable:worker` walk every directory.
+  
+  The `workflowsHook` / `stepsHook` codegen hooks take `source: string | string[]`. With several directories each hook still writes a single barrel at the same output path. The barrel is an array with one entry per root, each module imported through a relative path, and the file watcher regenerates it on a change in any root. A single directory produces the same barrel as before, byte for byte.
+  
+  When two directories define the same workflow name and version, or the same step name, with different classes or handlers, boot now fails with a `DuplicateDiscoveryError` that names both files, and nothing is registered. Duplicates inside one directory behave as before.
+  
+  `WorkflowEngine.handle(name, fn)` serves a step on the engine's transport, so app code can register an extra directory from a preload with `registerStepsFromDir(engine, dir)`. `registerWorkflowsFromDir` and `registerStepsFromDir` accept an array of directories too.
+  
+  Also fixed: a single-directory barrel with a sub-directory (`app/workflows/billing/refund_workflow.ts`) crashed the provider at boot, because the generator nests those entries and the loader called the nested object as a function. Nested entries now register.
+
+- [#351](https://github.com/DavideCarvalho/adonis-agora-durable/pull/351) [`c6c7550`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/c6c7550f8a1b498f48f13bfcbc1fd5d003983fd3) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - The `queue` transport can run several jobs of one step at once, and idle workers stop hammering the backend.
+  
+  - `transports.queue({ concurrency })`: each step name's task loop awaited one job before popping the next, so a worker ran one job of a step at a time however deep the backlog. With `concurrency: N` the loop keeps up to N jobs in flight. The slots share the single poll loop, so they add no polling, and every in-flight claim is renewed and drained on `close()`. The default stays 1.
+  - `transports.queue({ resultConcurrency })`: the engine settles up to N results at once (default 1), as on `bullmq`.
+  - `transports.queue({ idleBackoffMax })`: every loop polled at `pollIntervalMs` forever, so an idle worker serving 30 step names hit the backend about 150 times a second. Idle task, heartbeat and control loops now double their sleep after each empty poll, up to `idleBackoffMax × pollIntervalMs` (default 4, so 800ms), and the first job found resets them. That costs up to 800ms of pickup latency on the first job after a quiet spell; `1` restores the fixed cadence. The results loop never backs off.
+
+- [#358](https://github.com/DavideCarvalho/adonis-agora-durable/pull/358) [`da958fa`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/da958fac7ef253fc5e14b1a256af61fce4c3d26d) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - The `queue` transport now has worker health, adaptive concurrency and the poison-job bridge, as on `bullmq`.
+  
+  - Worker health: a queue worker now writes a liveness key for each step name it serves into the worker-key store (refreshed every 10s, expires after 35s). The key carries the worker's `WorkerStatus`: concurrency mode and live limit, in-flight count, RSS, CPU, throughput, p95 and the last adjustment. The transport implements `groupHealth` from these keys, so `engine.workerHealth()`, the dashboard's `/api/workers` and `/api/health` and the Telescope worker panel now report queue groups and render the worker cards. Before, they were empty on this transport. `depth` is the task queue's pending count (`adapter.sizeOf`). It is narrower than BullMQ's depth: the `@adonisjs/queue` adapter contract cannot count jobs in flight or delayed retries. Without a worker-key store (`workerKeys: false`, or no database), nothing can report liveness, so `groupHealth` is not offered. A process with deferred consumers (`node ace` console/REPL) writes no liveness key until it starts consuming.
+  - `transports.queue({ concurrency: 'adaptive' })`, or `{ mode: 'adaptive', min, max, start, tickMs, ramCeilingPct, cpuCeilingPct }`, uses the same controller as `transports.bullmq({ concurrency })`. Each step name's task loop reads the live limit on every poll and wakes as soon as the limit rises. Before this change the option was not accepted and the transport fell back to a fixed 1. A numeric `concurrency` behaves as before.
+  - Poison jobs: when the stalled sweep fails a task that stalled more than `maxStalledCount` times, the transport now publishes a retryable failed `StepResult` for that step (`remote step worker failed: stalled more than N times`), like BullMQ's terminal-failure bridge. The engine re-dispatches the step while it has retries left, then fails it into the workflow. Before, the run waited on that step's `pending` checkpoint forever. This works on `@adonisjs/queue` 0.7. On 0.6 the adapter deletes such a job inside its own sweep and reports only a count, so nothing can be published; use `timeoutMs` or `remoteRedispatchMs` there. Stalled result, heartbeat and control jobs are still only failed.
+
+- [#349](https://github.com/DavideCarvalho/adonis-agora-durable/pull/349) [`7096117`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/7096117e37f4f558599138fa94239844304ad67e) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - The `queue` transport now renews the claim of every job it is still working on, so a long step is never re-delivered and a dead worker's job is recovered in minutes instead of half an hour.
+  
+  The transport drives the `@adonisjs/queue` adapter directly and bypassed the `Worker` class's heartbeat along with its stalled recovery, so a claim's `acquiredAt` was never refreshed: its age was the step's runtime. The stalled threshold therefore had to exceed the longest step — a 30-minute default that left a crashed worker's job unrecovered for 30 minutes, while a step that ran longer than the threshold was re-delivered and ran twice. The transport now calls `adapter.renewJobs` every half `stalledThresholdMs` for its in-flight claims (leases on `@adonisjs/queue` 0.7, job ids on 0.6), so a claim's age measures how long its worker has been silent.
+  
+  - `stalledThresholdMs` defaults to 2 minutes when the adapter can renew (every `@adonisjs/queue` 0.6/0.7 driver), and keeps the 30-minute default on an adapter without `renewJobs`. An explicit value is unchanged.
+  - A renewal that finds a claim already re-delivered (the worker was silent past the threshold, e.g. a synchronous step blocked the event loop) is reported to `onError`, since the step may be running twice.
+  - Rolling upgrade: an old worker that does not renew can have a long step reclaimed by a new instance's 2-minute sweep. Durable steps are idempotent, so this is the usual at-least-once double run; pin `stalledThresholdMs` to the old value until every worker is upgraded if repeats are expensive.
+
+- [#358](https://github.com/DavideCarvalho/adonis-agora-durable/pull/358) [`9745d7e`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/9745d7e1a599c6e83ceceb6d428fa9a4523cd98f) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - The `queue` transport now carries remote workflow turns and store-less `tenant` pods, which used to need `bullmq`.
+  
+  - Remote workflows: `dispatchWorkflowTask` / `handleWorkflow` / `onDecision` and streamed step events. A turn rides the workflow name's task queue next to its steps (told apart by shape); decisions return on `<prefix>:decisions`, local steps on `<prefix>:step-events`. Every `onDecision` handler sees each decision, so several remote workflows can share one transport.
+  - Tenant pods: run starts and read/control requests are jobs on `<prefix>:start-run` and `<prefix>:run-request`. Replies, which `bullmq` broadcasts on Redis pub/sub, go to a queue of the requesting instance only (`<prefix>:run-replies:<instanceId>`), so two pods never receive each other's replies. Live-tail events are pushed to `<prefix>:tenant-events:<instanceId>` of every pod subscribed to that tenant: a pod announces its subscriptions on `<prefix>:inbox-registrations` and the control plane records them in its worker-key store, so the pod needs no database. A pod with no key store forwards its worker advertisement the same way, so the control plane still sees its workers. A control-plane janitor drains the queues of pods that died. New options: `inboxTtlMs`, `tenantSubscribersCacheMs`, `inboxSweepIntervalMs`.
+  - The provider already picks a transport for the `tenant` and `control-plane` roles by its methods, so `transport: 'queue'` now works for both. It also calls a new optional `Transport.useTenantRole()` hook on a tenant pod's transport; `bullmq` does not implement it and is unchanged.
+  - `verifyTenant` is now typed on the `standalone` role too. A `standalone` or `control-plane` engine whose transport carries this protocol starts the run-request responder, as it already did on `bullmq` — so a `standalone` app on `queue` now starts it, and logs a warning at boot unless `verifyTenant` is set.
+  
+  Compared with `bullmq`: replies and events are polled rather than pushed; a new subscription can miss events published in its first second or so; each event costs one job per subscribed pod; and live-tail needs a worker-key store on the control plane.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`64a3251`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/64a32511043a943ed69f58016db22e101fbda2d1) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Remote workflow turns are now safe with several engine replicas, and a turn whose worker died is re-driven instead of hanging or failing the run.
+  
+  **Fix.** `RemoteWorkflowExecutor` waited for each turn's decision in an in-memory map. On `bullmq` and `queue`, `<P>-decisions` is a point-to-point queue every replica consumes, so a decision was often taken by a replica that had not dispatched the turn. It found no waiter and dropped the decision, and the dispatcher waited forever with the run leased. The executor also numbered turns with a per-process counter, so two replicas could send the same `taskId` (the `db` transport then ignored the second turn). Now:
+  
+  - The engine mints a random `taskId`, records it on the run (`WorkflowRun.awaitingDecisionTaskId`), suspends the run and releases its lease, and only then enqueues the turn. A decision cannot arrive ahead of its marker.
+  - Every engine that registers a remote workflow consumes decisions. The replica that takes one leases the run, checks that it still awaits that `taskId`, and applies the decision durably. A stale or duplicate decision is dropped.
+  - A wake that lands while a turn is out (a step result, a signal) no longer dispatches a second turn over it.
+  - `waitForRun` does not resolve on the suspend of a run awaiting a turn.
+  
+  **New: `remoteAdvanceSilenceMs`** (engine option and `config/durable.ts`). A turn with no decision and no run-scoped heartbeat for this long is re-dispatched by the timer poller under a fresh `taskId`, with the same history. A run-scoped heartbeat (`{ runId, seq: 0, group }`, no `stepId`) pushes the deadline forward, throttled to one write per half window; the Python `durable-worker` already sends them, and the `bullmq`, `queue` and `db` transports now send one every 5 s while a TypeScript worker replays a turn. A `RemoteWorkflowExecutor`'s `timeoutMs` is now the per-workflow version of this window (the shorter wins) and re-drives the turn instead of failing the run. An inline executor (one that returns its decision in-process) that goes silent past the window releases its lease to crash recovery rather than failing; it throws the new `RemoteWorkflowTimeout`. Unset, a turn waits for its decision as before.
+  
+  **Convention routing is per run and tenant-aware.** A workflow started by name with no registration routes to the group of the run's namespace, `<name>@<tenant>` (bare for the default namespace), the same rule steps follow. An operator starting a tenant's run reaches the tenant's worker, and the group is re-checked on every start. In a transport pool the turn goes out on the transport that reports the live group.
+  
+  **Breaking for direct callers of the executor:** `WorkflowExecutor.advance` is now optional and joined by `dispatch(run, history, taskId, pendingSignals?)`, `subscribeDecisions(handler)` and `decisionTimeoutMs`. `RemoteWorkflowExecutor` implements `dispatch` and no longer has `advance`. `Heartbeat.stepId` is optional.
+  
+  **Schema.** `durable_workflow_runs` gains a nullable `awaiting_decision_task_id` column. `createDurableTables` creates it, and `ensureSchema()` adds it to an existing table at boot (reported once, like any repair).
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`2e9e85a`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/2e9e85aa0088888743cb48ab79159cf28d326de5) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Schemas are now [Standard Schema](https://standardschema.dev) based instead of zod-specific, so any library implementing `~standard` works: zod 3.24+ / 4, valibot, arktype, effect schema and others. **`zod` is no longer a runtime dependency** of the package, and neither the emitted JavaScript nor the `.d.ts` files import it. Existing zod 4 schemas keep working with no code changes. The Standard Schema V1 type is vendored (types only) and exported as `StandardSchemaV1`, together with a `validateStandard(schema, value, label)` helper.
+  
+  - `@Step({ input, output })` and `defineStep` accept any Standard Schema. Sync and async validators both work. The handler receives the input schema's output value (parsed or coerced), and the caller receives the output schema's output value. When the handler is unannotated, `defineStep` infers its types from the schemas: the ref takes the input schema's input type and resolves to the output schema's output type. An annotated handler keeps its own types, as before.
+  - A value that fails a schema now throws `SchemaValidationError`, a non-retryable `FatalError` with code `schema_validation`. The issues are in `issues` and `details.issues` as `{ message, path: string[] }[]`. Before this change a zod step-schema failure threw a retryable `ZodError`, so a bad payload was retried until `retries` ran out. It now fails on the first attempt.
+  - New `inputSchema` option on `static workflow`, `engine.register` and `engine.registerRemote`. The input is validated at `start` before a run is created, and the schema's output value becomes the run's input. It runs before the existing `validateInput` function, which is kept and receives the parsed value. An `onEvent` subscriber whose schema rejects a payload is skipped, as with `validateInput`.
+  - Typed search attributes. Declare `static workflow = { searchAttributes: schema }` (or `searchAttributesSchema` on `engine.register`) and every `ctx.upsertSearchAttributes` validates the merged attributes; a write that doesn't match throws `SchemaValidationError`, and nothing is written or recorded. The schema must validate synchronously. `WorkflowCtx<A>` (and `DynamicWorkflowCtx<A>`) narrows `upsertSearchAttributes` to `Partial<A>`, and the default `WorkflowCtx` behaves as before. `InferSearchAttributes`, `readSearchAttributes(schema, run)` and `attributesOf(WorkflowClass, run)` read attributes back typed. Reads are lenient: history that fails the schema reads as `{}`.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`f79db4c`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/f79db4c5985cd4ed107f5fd8e4dc9ddd78cd2c22) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Lucid store schema fixes, and store-level batching and aggregates.
+  
+  **Schema (Lucid).**
+  
+  - On MySQL the JSON columns (run and checkpoint `input`/`output`/`error`, `events`, `tags`, `search_attributes`, `heartbeat_progress`, the buffered payloads, the schedule documents) are `longtext`. MySQL `text` stops at 64KB: a larger output or event log was rejected (strict mode) or cut off and then failed to parse on read. An existing `text` column is widened with one `ALTER TABLE … MODIFY` per table and reported in the repair warning. MySQL rebuilds the table for it, so run `node ace migration:run` in a maintenance window for a large checkpoints table. Postgres and SQLite DDL is unchanged.
+  - `createDurableTables` records a fingerprint (`DURABLE_SCHEMA_REVISION` plus the collation) in a new one-row table, `durable_schema_meta`. A boot against a current schema reads that row and returns: two statements instead of a probe per table, column and index. A missing or stale row runs the full check under a cross-process lock (`pg_advisory_xact_lock` on Postgres, `GET_LOCK` on MySQL, none on SQLite), so pods booting together against a fresh database no longer race the same `CREATE TABLE` (on the previous release, 7 of 8 concurrent boots failed on both Postgres and MySQL). `{ force: true }` runs the check behind a current marker. `dropDurableTables` drops the marker too.
+  - New `collation` option (`stores.lucid({ collation })`, `createDurableTables(db, name, { collation })`), MySQL only: new durable tables are created with it, and an existing one with another collation is converted, so JOINs with the app's tables stop failing with `Illegal mix of collations`. Defaults to the Knex connection's `collate` setting when it has one.
+  - New index `durable_step_checkpoints_group_idx` on `(run_id, parallel_group, seq)`, for the run detail's per-fan reads.
+  - A JSON column that does not parse now reads back as `undefined` for that field instead of throwing. One damaged row used to fail `listRuns`, the run detail and every other read that touched it. Each damaged row and column is reported once with `console.warn`; route it with `onCorruptJsonColumn(reporter)`.
+  
+  **New optional `StateStore` members**, implemented by the Lucid and in-memory stores, forwarded by `CodecStateStore`, and required by the conformance kit:
+  
+  - `saveCheckpointsIfPending(checkpoints)`: `saveCheckpointIfPending` for a batch, atomic as a whole. A run's step results that land together (`resultConcurrency > 1`) are settled in batches of up to 256: one batched write, the step fan's barrier asked once, one resume. A batch of one takes the single-result path unchanged. A rejected batch write is retried row by row, so one bad row fails only its own result. Without the method each result gets its own write.
+  - `checkpointStats(runId, { group? })` and `queryCheckpoints(runId, query)`: a `GROUP BY` over a run's checkpoints, and paged, filtered reads. `engine.getRunDetail` / `engine.getFanMembers` (new) and the gateways build the summarized run detail and fan pages from them, reading a few rows per wide fan instead of the whole history.
+  - `deleteRuns(runIds)`: retention and `purgeRuns` delete a pass with one call (a few `IN (...)` statements in one transaction) instead of four statements per run.
+  - `runFacets(query)`: `(status, origin)` counts in one `GROUP BY`; `engine.runFacets(query)` uses it.
+  - `pruneBufferedEvents(publishedBefore, limit)`: deletes expired buffered events across every name.
+  
+  **Dashboard.** `GET /runs/:id` and `GET /runs/:id/fans/:group` send an `ETag` with `Cache-Control: private, no-cache` when the store has the aggregates. A poll whose `If-None-Match` still names it gets a bare `304`, answered without reading a checkpoint row. The browser adds the header to the console's polls itself. `?full=true` and stores without the aggregates answer in full as before.
+  
+  **Config.** `eventBufferTtl` (duration): a buffered event older than this is never delivered, and the worker tick's retention phase deletes expired ones. `scopeReads: true` (with `namespace`): the engine reads through `store.withScope({ namespace })`, a view whose run reads only see that namespace; writes are unaffected. `LucidStateStore` and `InMemoryStateStore` have `withScope`.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`667a004`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/667a004a2fe2dc0224985bd8f04cb58ddcc6d4c7) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - The dashboard's `POST /webhooks/:token` buffers a callback that lands before the run reaches `wait()` (`202 { buffered: true }`) when the token's run exists, is not terminal and has not consumed that webhook yet; the later `wait()` consumes it. Unknown or terminal runs and duplicates still answer `404` and buffer nothing.
+  
+  Entities: an operation stored in an entity's buffer whose wake was lost (the process died between storing it and waking the entity) is now applied by the worker loop's entity reconcile, at most once a minute (`engine.reconcileEntityWakes()` runs a pass by hand). Stores gain an optional `listBufferedSignalTokens(prefix, limit)`, implemented by the Lucid and in-memory stores and covered by the conformance kit.
+
+### Patch Changes
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`8954aec`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/8954aecf1e19a08a6ff9121cff0e5e72ad5dc7b1) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Fixes for bugs found while auditing the docs against the 0.51 code.
+  
+  - `ctx.webhook().wait()` consumes a callback that arrived before the wait. `engine.signal` had buffered it, but the wait never read the buffer, so the callback was lost and the run waited forever. The wait now checks the buffer, registers, then checks again, like `ctx.waitForSignal`. The dashboard webhook route still answers `404` to a callback that arrives before the wait.
+  - `ctx.callEntity` and `ctx.signalEntity` are exactly once across replay. A replay of a `callEntity` still waiting for its reply used to send the operation again, and a turn that died after a `signalEntity` dispatch sent it twice. The operation is now stored durably, tagged with the caller's `<runId>:<seq>`, before the caller checkpoints. `callEntity` records a pending intent that it marks as sent with a conditional write. The entity skips a copy re-sent with the same id.
+  - A child moved to `dead` by recovery (`maxRecoveryAttempts`) now notifies its parent right away as a failed child (`ctx.child` throws, `ctx.all`/`allSettled` record it). Before, the parent waited for its reconcile wake.
+  - An empty-string namespace is the default pool. An engine configured with `namespace: ''` stamps and polls `'default'`. The Lucid and in-memory stores write `''` as `'default'`, read legacy `''` rows back as `'default'`, and match those rows with every `'default'` filter (polls, recovery, `listRuns`, scoped views, schedules).
+  - `node ace make:workflow OrderWorkflow` (or `order_workflow`) no longer generates `OrderWorkflowWorkflow`.
+  - Event triggers: a start or publish that fails (unregistered workflow, invalid input, quota, store error) is reported through the bridge's new `onError` option (default `console.warn`). Before, it was an unhandled rejection. Other background promises now catch their rejections too: the run-request responder's tenant-event re-publish, the Redis admission backend's first liveness write, the bullmq heartbeat handler, the dashboard's audit and hook-warning logging (including an async `audit` hook), and the provider's unserved-onError-routes report.
+  - A tenant pod's top-level `partition` and `instanceId` now reach its transport (queue, db, bullmq, eventEmitter), so it consumes the `<name>@<tenant>` queues the control plane dispatches to. A value set on the transport still wins.
+  - New `transportPool` config: transports listed there are built next to `transport`, in failover order, with the same consumer deferral, namespace and shutdown. `ctx.step(..., { transport })` can pin any of them by name. Pinning a transport that isn't built now fails with an error that lists the built ones, and `default` still means the primary transport.
+  - bullmq: a process with deferred consumers (`node ace`, `consumers: 'auto'` in the console or REPL, or `'never'`) no longer advertises worker liveness until it starts consuming.
+  - db transport: payload columns are `longtext` on MySQL instead of `text`, which capped them at 64KB. Existing tables are widened at boot, and the migration stubs are updated.
+  - `transports.db()` forwards `workerKeysCacheMs`.
+  - `controlPlanes.redis()` (an `@adonisjs/redis` connection) gets the subscriber PING watchdog and a `pingIntervalMs` option.
+  - The bullmq transport and `RedisControlPlane` fall back to the default `pingIntervalMs` when given a negative, NaN, Infinity or out-of-range value. Before, the watchdog pinged about every millisecond.
+  - `workflowsPath: false` no longer skips registering `app/steps` or starting the run-request responder.
+  - Lucid store: `cancel` and `requeue` no longer fail with an unknown column on a runs table that predates `awaiting_decision_task_id` (`autoSchema: false` with an older migration).
+  - queue transport: a control plane without a worker-key store now warns once that it drops forwarded worker advertisements.
+  - queue, db and bullmq transports: when one `onDecision` handler fails, every handler still runs to completion and each failure is reported before the decision is redelivered. Before, the redelivery started while the other handlers were still running, so they ran again concurrently.
+  - `InMemoryStateStore.withScope` views share the store's state by reference, so counters no longer drift between a view and its base.
+  - The `create_durable_tables` migration stub comment now lists all seven tables.
+
+- [#358](https://github.com/DavideCarvalho/adonis-agora-durable/pull/358) [`6cc6e20`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/6cc6e20d197bf61d3cf214d0cd3e08940b8250b5) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - On the `bullmq` transport, a second (or later) remote workflow no longer hangs. The engine builds one remote-workflow executor per remote workflow, and each subscribes to decisions, ignoring the turns it isn't waiting for. `onDecision` kept only the first subscriber, so decisions for every other remote workflow went to an executor that dropped them, and those runs never advanced. Every subscriber now receives every decision.
+
+- [#369](https://github.com/DavideCarvalho/adonis-agora-durable/pull/369) [`4db7864`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/4db786434d2a47cf55a097bf9c1dc4984a54dd5e) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - A run whose turn fails on a store error no longer raises an unhandled rejection, `drain()` now waits for an in-process pickup that has not started executing yet, the `db` transport never runs more rows than its live adaptive limit, and worker heartbeats no longer pile up writes behind a slow key store.
+  
+  - The engine dropped a finished execution from its in-flight set with a `finally` chain nothing awaited, so a turn whose store write failed (a lost connection, a missing table) was handled by its caller and ALSO surfaced as a process-level unhandled rejection — fatal under Node's default `--unhandled-rejections=throw`.
+  - The default in-process dispatcher's pickup (lease, read, then execute) is tracked from the moment of dispatch, so `drain()` can no longer return while it is about to write onto a connection being closed. A failed pickup is reported with an `[adonis-durable]` warning instead of being swallowed silently; the run is retried by the pending-run poll or recovery, as before.
+  - The `db` transport sized each claim round from the adaptive limit read before the claim's round trips. When the RAM brake lowered the limit during a round, every row the round won still ran. Rows beyond the live limit are now handed back before running.
+  - The `db` and `queue` transports' worker heartbeat no longer starts a liveness-key write while the previous one for the same routing token is still in flight. A key store slower than `workerHeartbeatIntervalMs` used to pile up writes without bound, each holding a pool connection that the claim loop and `groupHealth` then waited behind.
+
+- [#350](https://github.com/DavideCarvalho/adonis-agora-durable/pull/350) [`137028a`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/137028abb9159d67c5ace93f22c3ac25ab23405b) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - A remote step with `timeoutMs` no longer times out on a live worker when its heartbeats are consumed by another engine instance.
+  
+  On a point-to-point transport (the `queue` transport) every instance competes for the heartbeat queue, so a worker's beat can land on an instance that did not dispatch the step. That instance persisted it on the checkpoint, but only the dispatcher holds the liveness timer — it never heard the beat, fired, and failed or re-dispatched a step that was still running. When the window runs out, the dispatcher now reads the checkpoint's `lastHeartbeatAt` first: a beat from this attempt still inside `timeoutMs` re-arms the timer for the rest of the window, and only a step silent on every instance times out. Because another instance persists a step's beats at most once every 10s, the beat on the checkpoint can trail the newest one by that much: once the dispatcher sees a persisted beat it did not hear itself, it allows up to 10s of extra silence, so a dead worker is detected within `timeoutMs` + 10s on a multi-instance fleet and within `timeoutMs` on a single instance, as before. The throttle is configurable as the `heartbeatPersistThrottleMs` engine option.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`de4aee3`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/de4aee3583e4bb0f86fea14e3075f62e803bbea3) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Recovery, cancellation and shutdown fixes:
+  
+  - **Orphan recovery no longer dead-letters a run whose remote step is in flight.** A run found `running` with a free lease while a remote step is still `pending` on the transport is parked `suspended` on the step's deadline or the `reconcileMs` wake, and `recoveryAttempts` is not counted. Before, every pass counted an attempt, so a slow step could reach `maxRecoveryAttempts` and be moved to `dead` while its worker was still busy.
+  - **`requeue` resets `recoveryAttempts` to 0 when it resurrects a `failed`/`dead` run**, so a retried dead-letter isn't killed again by the next recovery pass. Requeueing a live run keeps its count.
+  - **One bad run no longer aborts a recovery, timer or pending sweep.** It is logged with its id and `workflow@version` and skipped, and the rest of the batch runs. A lease taken while drain began is handed back instead of starting the run.
+  - **A compensating cancel is never lost when the turn doesn't suspend.** If the turn running when `cancel(runId, { compensate: true })` lands returns, throws or continues-as-new, it now runs the compensations and settles `cancelled`, instead of completing, failing or forking the next run. This also works when that turn runs on another instance (through the durable `cancel:<id>` marker).
+  - **A failed turn no longer causes an unhandled rejection.** The in-flight bookkeeping used `p.finally(...)`, which re-rejected.
+  - **Shutdown stays within one `drainTimeoutMs`.** `engine.beginDrain(spentMs)` reports wait already spent and it is deducted from `drain()`, including a drain already waiting. Non-finite or negative values are ignored. The worker loop (`durable:work`, embedded worker) charges the drain for the time spent waiting for its in-flight tick, and stops waiting for a tick that outlives the whole budget.
+  - **`reconcileMs` and `blockedPollMs` are now `config/durable.ts` fields.** A non-finite or negative `reconcileMs`, or a non-finite or sub-1 `blockedPollMs`, falls back to the default instead of silently disabling the reconcile net or hot-looping blocked runs.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`9d5da23`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/9d5da23d3c6cdf073731d189f4abd42ccb897f1e) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - A remote workflow run awaiting a dispatched turn's decision now honours `cancel(id, { compensate: true })`: it settles `cancelled` (a remote workflow has no engine-side saga compensations) instead of the cancel being swallowed by the turn-in-flight hold, and the late decision is dropped. `requeue` and `cancel` clear the run's `awaitingDecisionTaskId`. `DURABLE_SCHEMA_REVISION` is 3 (the `awaiting_decision_task_id` column), so every Lucid database runs the full schema check once on its next boot.
+  
+  A remote workflow run whose turn was lost with a call still in flight (an inline `advance` that hit `remoteAdvanceSilenceMs`, or a replica that died applying a decision) is now re-driven by the timer pass about a second after orphan recovery parks it, instead of after the whole `reconcileMs` window; the in-flight call is not dispatched again. `ProxyRunGateway.runFacets` degrades against an older control plane: it counts the cells from `runValueFacets`, or from `listRuns` on one older still, instead of rejecting with `unsupported-request`.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`feb4e57`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/feb4e57088b09b4ffe3946a668d0a46fd40f7c33) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - An instance restarted under the same stable `instanceId` (a pod name) after a `SIGKILL`/OOM kill now reclaims the `running` runs its previous process left leased to that id at once, instead of waiting up to `leaseMs` for those leases to expire.
+  
+  - For the first `leaseMs` after boot, `recoverIncomplete()` also lists the runs leased to its own `instanceId`. It reclaims one only when nothing in the process is executing it (no execution in its run gate, no tracked execution lease) and the lease was written before the engine started. The stale lease is released inside the run's in-process gate, so a resume landing at the same moment queues behind it, and the run then goes through normal orphan recovery (it counts a `recoveryAttempts`). A live instance never reclaims a run it is executing. Other instances still wait for the lease to expire.
+  - `StateStore.listOrphanedRuns(nowMs, limit, namespace?, owner?)` gains an optional `owner`: also return `running` runs whose lease is held by that owner. Implemented by the Lucid store, the in-memory store and `CodecStateStore` (passthrough, and the in-process fallback). A custom store that ignores it keeps the expiry-only behaviour. The state-store conformance suite has a case for it.
+  
+  This needs `instanceId` to be unique per live process, as the docs already required: two live processes sharing an id read each other's leases as their own.
+
+- [#368](https://github.com/DavideCarvalho/adonis-agora-durable/pull/368) [`feb4e57`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/feb4e57088b09b4ffe3946a668d0a46fd40f7c33) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Singleton admission counts holders admitted by 0.50-and-older replicas, so mixing old and new replicas over one store during a rolling upgrade never puts a second run on a key beside an old holder — including one an old replica admitted out of order, past an older `pending` run.
+  
+  A 0.51 replica now stamps the reserved `durable:singleton:tracked` tag on a singleton run's first execution (in the same write that sets it `running`, before it competes) and on every denial; admission swaps it for `durable:singleton:admitted`, and terminal settlement and requeue drop both (`withoutSingletonMarkers`). Both are stripped from user start tags. `singletonAdmissionTags` — and so `tryAdmitSingleton` on every bundled store — counts as holders the untagged `running`/`suspended` runs of the key that lack the tracked marker (an old replica executed them) or sit in the old scan's `limit`-oldest window (every tagged holder keeping its seat), never the candidate itself, inside the same locked claim. FIFO and handover are unchanged. A waiter an old replica parked may wait about one retry longer, until a new replica marks it. A custom store built on the exported helpers gets this for free; one that writes tags itself must drop both markers on a terminal status.
+
+- [#358](https://github.com/DavideCarvalho/adonis-agora-durable/pull/358) [`2631706`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/2631706ed3c4084f1c4126e33bae236277adfdd1) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Transport hardening found while auditing the transports against each other:
+  
+  - **`bullmq`: a connection string was ignored.** `transports.bullmq({ connection: 'redis://host:port/db' })` passed validation, but the string was then spread into the ioredis/BullMQ options character by character. Every client silently connected to `127.0.0.1:6379`. A `redis://`/`rediss://` URL is now parsed into connection options (host, port, credentials, db, TLS), and a malformed URL throws at boot.
+  - **`eventEmitter`: a failing result handler could crash the process.** The engine's result and heartbeat handlers were called as `void handler(...)`, so a rejection (the store briefly unreachable while settling a checkpoint) became an unhandled rejection. That is fatal under Node's default, and the result was lost. Failures now go to a new `onError` option (default `console.error`). A failed result is redelivered from memory with backoff until `close()`, and a failed heartbeat is dropped.
+  - **`queue`: `closeTimeoutMs`.** The close drain was a hardcoded 5s. It is now the `closeTimeoutMs` option (default 5000), as on `bullmq` and `db`.
+
+## 0.49.1
+
+### Patch Changes
+
+- [#345](https://github.com/DavideCarvalho/adonis-agora-durable/pull/345) [`4d65110`](https://github.com/DavideCarvalho/adonis-agora-durable/commit/4d651102f09715daaecce4987ff25078adb2c6ff) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - BullMQ adaptive concurrency controller fixes, mirrored from `@dudousxd/durable-worker`:
+  
+  - The worker heartbeat's `throughputPerMin` is a rate over the trailing minute. It decays while a worker is idle and reads 0 once a minute passes with no completed step, instead of reporting the last burst's rate forever.
+  - With a fixed `concurrency` the controller no longer runs its control-loop timer. The limit stays the fixed value, and the heartbeat samples CPU itself, at most once every 2s, so `cpuPct` is still reported.
+  - `AdaptiveController` takes optional `readCpuUsage` and `readCpuCount` readers (defaults `process.cpuUsage()` and `os.cpus().length`), so the CPU ceiling can be driven in tests.
+
 ## 0.49.0
 
 ### Minor Changes

package/README.md

@@ -2,29 +2,48 @@
 
 [![npm version](https://img.shields.io/npm/v/@adonis-agora/durable)](https://www.npmjs.com/package/@adonis-agora/durable) [![npm downloads](https://img.shields.io/npm/dm/@adonis-agora/durable)](https://www.npmjs.com/package/@adonis-agora/durable) [![CI](https://github.com/DavideCarvalho/adonis-agora-durable/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/DavideCarvalho/adonis-agora-durable/actions/workflows/ci.yml) [![license](https://img.shields.io/npm/l/@adonis-agora/durable)](https://www.npmjs.com/package/@adonis-agora/durable)
 
-AdonisJS binding for [`@adonis-agora/durable-core`](https://www.npmjs.com/package/@adonis-agora/durable-core)
-— a container-resolved durable workflow engine.
+Durable, resumable workflows for AdonisJS: write a workflow as plain async code; every step is
+checkpointed, so a run survives crashes and deploys and resumes from its last completed step.
+Steps can run in-process or on remote workers (another Adonis process, a NestJS worker, or a
+Python worker), and it stays one run with one timeline.
 
 ```sh
-npm i @adonis-agora/durable
-node ace configure @adonis-agora/durable
+node ace add @adonis-agora/durable
 ```
 
+Define a workflow under `app/workflows/` (auto-registered at boot):
+
 ```ts
-import { WorkflowEngine } from '@adonis-agora/durable'
+// app/workflows/order_workflow.ts
+import { BaseWorkflow, type WorkflowCtx } from '@adonis-agora/durable'
+
+export default class OrderWorkflow extends BaseWorkflow {
+  static workflow = { name: 'order', version: '1' }
+
+  async run(ctx: WorkflowCtx, order: { id: number; total: number }) {
+    const charge = await ctx.localStep('charge', async () => chargeCard(order))
+    await ctx.localStep('ship', async () => ship(charge))
+    return { status: 'shipped' }
+  }
+}
+```
 
-const engine = await app.container.make(WorkflowEngine)
-engine.register('order', '1', async (ctx) => {
-  const charge = await ctx.step('charge', () => chargeCard(ctx.input))
-  await ctx.step('ship', () => ship(charge))
-})
-await engine.start('order', input, runId)
+Start it from a controller, and use the engine service for everything else:
+
+```ts
+import OrderWorkflow from '#workflows/order_workflow'
+import engine from '@adonis-agora/durable/services/main'
+
+const { runId } = await OrderWorkflow.dispatch({ id: 1, total: 4200 }, { runId: 'order:1' })
+const result = await engine.waitForRun(runId)
 ```
 
-Defaults to an in-process store + transport (single-process, zero infra). Supply a
-persistent store and a broker-backed transport in `config/durable.ts` for
-production. When `@adonis-agora/context` is installed, the originating tenant/user/
-correlation carrier rides each dispatched task (best-effort, no hard dependency).
+Defaults to the in-memory store + the test-only `memory` transport (single process, zero infra).
+Supply a persistent store and a cross-process transport in `config/durable.ts` for production.
+When `@adonis-agora/context` is installed, the originating tenant/user/correlation carrier rides
+each dispatched task (best-effort, no hard dependency).
+
+Full docs: https://davidecarvalho.github.io/agora/docs/durable
 
 ## Transports
 
@@ -37,6 +56,9 @@ Select a transport by name in `config/durable.ts`:
 - `transports.queue({ connection })` — cross-process over `@adonisjs/queue`.
 - `transports.db({ connection })` — cross-process over the app database
   (`@adonisjs/lucid`), no broker.
+- `transports.bullmq({ connection })` — cross-process over BullMQ on Redis, wire-compatible
+  with the NestJS (`nestjs-durable`) BullMQ transport and the Python worker. Pair it with
+  `controlPlanes.redis(...)`.
 - `transports.memory()` — test-only (drives `dispatch` straight into the handler).
 
 ```ts
@@ -52,26 +74,35 @@ export default defineConfig({
 
 ## Workflows codegen (Assembler hook)
 
-`node ace configure` registers an AdonisJS **Assembler `init` hook** that generates
-a typed barrel of `app/workflows/` at build/dev time — exactly how core generates
+`node ace configure` registers two AdonisJS **Assembler `init` hooks** that generate
+typed barrels of `app/workflows/` and `app/steps/` at build/dev time — exactly how core generates
 the controllers/events/listeners barrels. The provider imports the generated
-`.adonisjs/durable/workflows.ts` at boot and registers every `BaseWorkflow` class,
-instead of scanning the directory with `readdir` at runtime. The file watcher
-regenerates the barrel whenever a workflow file changes.
+barrels at boot and registers every `BaseWorkflow` class and step, instead of scanning
+the directories with `readdir` at runtime. The file watcher regenerates a barrel
+whenever a workflow or step file changes. The barrels import through the `#workflows/*`
+and `#steps/*` subpath imports, which `configure` adds to your `package.json`.
 
 Register it in `adonisrc.ts` (the `configure` command does this for you):
 
 ```ts
 export default defineConfig({
   hooks: {
-    init: [() => import('@adonis-agora/durable/hooks/workflows')],
+    init: [
+      () => import('@adonis-agora/durable/hooks/workflows'),
+      () => import('@adonis-agora/durable/hooks/steps'),
+    ],
   },
 })
 ```
 
 If the hook is not registered (or the barrel hasn't been generated yet) the
 provider **falls back** to the runtime directory scan, so apps that don't opt in
-keep working unchanged. Opt out of discovery entirely with `workflowsPath: false`.
+keep working unchanged. Opt out of discovery entirely with `workflowsPath: false`
+(or `stepsPath: false`).
+
+A modular app can list several directories: `workflowsPath` / `stepsPath` take
+an array, and so does the hooks' `source` option. Names must be unique across
+those directories, and a clash fails boot naming both files.
 
 ## License