@adonis-agora/durable 0.49.1 → 0.51.0

package/skills/durable-transports-stores/SKILL.md

@@ -14,7 +14,7 @@ license: MIT
 metadata:
   type: core
   library: "@adonis-agora/durable"
-  library_version: "0.25.0"
+  library_version: "0.51.0"
   framework: adonisjs
 sources:
   - 'DavideCarvalho/adonis-durable:docs/transports/index.mdx'
@@ -59,6 +59,11 @@ export default defineConfig({
 
 Switching transports is a one-line change; workflow code never changes.
 
+`transports.eventEmitter({ onError })` reports a failing result/heartbeat
+handler (default `console.error`) and redelivers a failed result from memory
+with backoff until `close()`. Before 0.50 such a failure was an unhandled
+rejection — fatal under Node's default — and the result was lost.
+
 ## Core patterns
 
 ### Pattern 1 — cross-process steps without new infrastructure (`db`)
@@ -75,10 +80,51 @@ const transport = new DbTransport({ db }) // { partition } isolates this pool
 transport.handle('pipeline:extract', async (input) => extract(input))
 ```
 
+A worker runs ONE task row at a time unless `concurrency` is raised
+(`transports.db({ concurrency: 8, resultConcurrency: 8 })`, or `'adaptive'` with
+the same controller and fields as `queue`/`bullmq`); one poll loop serves every
+handler name and claims only as many rows as it has free slots. Every held
+claim is renewed every `leaseMs / 3`, and a row is recorded (task delete +
+result insert, one transaction) only while `claimed_by` is still the worker's
+token — a worker whose claim was taken over drops its result and reports it to
+`onError`. (Before 0.50 the loop claimed 20 rows per poll, renewed only the
+running one, and double-ran the rest of the batch when a step outlived
+`leaseMs` — upgrade if `db` steps run twice.) Idle task/heartbeat/control loops back off up to `idleBackoffMax`
+(default 4) poll intervals; results never back off.
+
+Workers beat `durable[-<ns>]-worker-heartbeat:<token>:<instance>` into
+`durable_transport_worker_keys` with their `WorkerStatus`, and `DbTransport`
+implements `groupHealth`: `depth` = pending + claimed task rows of the group.
+Each takeover of a lapsed claim bumps the row's `reclaims` column; past
+`maxStalledCount` (default 3) the row is deleted and a retryable failed
+`StepResult` (`remote step worker failed: claim lapsed more than N times`) is
+published. `close()` (after `closeTimeoutMs`, default 5s) clears every claim it
+still holds, uncounted, so another worker takes the rows at once and a deploy
+never pushes a step toward the poison bound; only a crash (no close) counts.
+Dispatch `priority` is stored on the row (`0` when absent = normal) and rows
+are claimed highest priority first, then oldest. `autoCreate` /
+`createDurableTransportTables` add `reclaims` and `priority` to an existing
+table; with `autoCreate: false` add them yourself (integer, not null, default
+0), or the poison bound / priority ordering stay off.
+
+`db` also carries remote workflow turns (task rows of kind `workflow`, claimed
+with the token's steps; each decision is a `durable_transport_messages` row
+addressed to the engine replica that dispatched the turn, though any replica
+could apply it) and store-less tenant
+pods that reach the same database: start-run/run-request/step-event rows go to
+any consumer, a reply is a row addressed to the requester, a tenant event is one
+row per live subscriber (pods write their own `-inbox:` / `-tenant-subscriber:`
+keys to `durable_transport_worker_keys`). A janitor on every instance that
+addresses rows deletes those older than `inboxTtlMs` whose recipient's inbox key
+is gone. Ships as the `create_durable_transport_messages` migration (also adds
+the tasks `kind` column) or via `autoCreate`; without it the protocol is off and
+said once through `onError`. Tune with `inboxTtlMs`, `tenantSubscribersCacheMs`,
+`inboxSweepIntervalMs`. See `durable-cluster`.
+
 Trade-off: throughput is bounded by polling + row contention — right for
 workflow-scale modest rates, not high-fanout firehoses. Delivery is
-at-least-once (crash between insert-result and delete-task), so handlers should
-be idempotent on `step_id`. Source: `docs/transports/db.mdx`.
+at-least-once (a worker that dies after its handler ran re-runs elsewhere), so
+handlers should be idempotent on `step_id`. Source: `docs/transports/db.mdx`.
 
 ### Pattern 2 — real brokers (`queue`, `bullmq`) and stalled-claim tuning
 
@@ -91,9 +137,9 @@ export default defineConfig({
   transports: {
     queue: transports.queue({
       connection: 'redis',
-      // A claim is NEVER renewed while a worker processes it — its age is the
-      // step's elapsed time. Raise the threshold above your longest step:
-      stalledThresholdMs: 2 * 60 * 60 * 1000, // default 30min
+      // Claims are renewed while a step runs, so this bounds worker SILENCE
+      // (crash, or a blocked event loop), not step length. Default 2min:
+      stalledThresholdMs: 5 * 60 * 1000,
       maxStalledCount: 3, // bounds a poison job
       onError: (err) => logger.error({ err }, 'durable queue transport'),
     }),
@@ -102,11 +148,37 @@ export default defineConfig({
 ```
 
 One instance consumes engine-side results; run one or more `node ace
-durable:work` workers serving the step names. Source:
+durable:work` workers serving the step names. A worker runs ONE job of a step
+name at a time unless `concurrency` is raised (`transports.queue({ connection,
+concurrency: 8, resultConcurrency: 8 })`); the slots share one poll loop.
+`concurrency: 'adaptive'` (or `{ mode: 'adaptive', min, max, start, tickMs,
+ramCeilingPct, cpuCeilingPct }`) runs the same controller as `bullmq` (below);
+each step name's loop reads the live limit every poll. Idle
+task/heartbeat/control loops back off up to `idleBackoffMax × pollIntervalMs`
+(default 4 × 200ms; `1` disables).
+
+A `queue` worker beats `<prefix>-worker-heartbeat:<token>:<instance>` into the
+worker-key store (10s, live 35s) with its `WorkerStatus`, and the transport
+implements `groupHealth` when a store is configured (the default with a database;
+absent with `workerKeys: false`). Its `depth` is the PENDING count (`sizeOf`) only —
+the adapter contract can't count jobs in flight or delayed retries. A task that
+stalls past `maxStalledCount` is failed AND bridged into a retryable failed
+`StepResult` (`remote step worker failed: stalled more than N times`) on
+`@adonisjs/queue` 0.7; on 0.6 the adapter deletes it internally and nothing is
+published, so rely on `timeoutMs` / `remoteRedispatchMs` there. Source:
 `docs/transports/queue.mdx`.
 
+`queue` also carries remote workflow turns (`<prefix>:decisions`,
+`<prefix>:step-events`) and store-less tenant pods: start-run/run-request are
+plain jobs to the control plane, replies go to a per-pod
+`<prefix>:run-replies:<instanceId>` queue, and tenant events are pushed to each
+subscribed pod's `<prefix>:tenant-events:<instanceId>` (subscriptions announced
+on `<prefix>:inbox-registrations`, recorded in the control plane's key store;
+tune with `inboxTtlMs`, `tenantSubscribersCacheMs`, `inboxSweepIntervalMs`).
+See `durable-cluster`.
+
 For wide fans (`ctx.all` over thousands of steps), raise how many results an
-instance settles at once on `bullmq` (`queue` and `db` have no such option),
+instance settles at once on `bullmq`, `queue` or `db` (`resultConcurrency`),
 let its workers size their step concurrency, and size the engine's due-timer
 batch:
 
@@ -115,7 +187,7 @@ export default defineConfig({
   timerBatchSize: 1000, // due runs per timer pass, most overdue first (default 500)
   transports: {
     bullmq: transports.bullmq({
-      connection: { host: '127.0.0.1', port: 6379 },
+      connection: { host: '127.0.0.1', port: 6379 }, // or 'redis://…' / 'rediss://…' (0.50+)
       resultConcurrency: 16, // default 1
       // A number pins each task queue's concurrency (default 1). 'adaptive' moves it
       // every tickMs: +1 while steps saturate the limit at steady latency, down by the
@@ -133,6 +205,17 @@ work and progress); a non-finite or out-of-range field falls back to its default
 throughput covers the trailing minute, decaying while the worker is idle and reading 0 once a minute passes with no completed step; with a fixed limit the heartbeat still reports `cpuPct`, sampled at most once every 2s. In adaptive mode the decisions, step-events, start-run and run-request consumers run
 at 1 (results run at `resultConcurrency`). Sources: `docs/transports/index.mdx`, `docs/cluster/interop.mdx`.
 
+Two `bullmq` safety nets: `pingIntervalMs` (default 30s; `0`/`false` off) PINGs
+every pub/sub subscriber (heartbeats, run replies, tenant events) and reconnects
+one a VPN/NAT/idle timeout dropped silently, so autoResubscribe restores it; each
+subscriber also gets an `error` listener (reported once per burst to `onError`).
+`stepTimeoutMs` (off by default) fails a step handler still running at the
+ceiling as RETRYABLE, so a wedged `await` cannot hold its job forever (BullMQ
+renews the lock on a timer, so the job never stalls). A `redis://` /
+`rediss://` connection string is parsed into ioredis options since 0.50; before
+0.50 it was silently ignored and every client connected to `127.0.0.1:6379`.
+Source: `docs/transports/bullmq.mdx`.
+
 ### Pattern 3 — multi-replica fan-out: the control plane
 
 The task transport is point-to-point; lifecycle events and cancellation need a
@@ -214,30 +297,56 @@ without it the engine loads the run's checkpoints per result), implement
 (without it a blind save can put a settled step back to `pending`), and return
 `listDueTimers` most overdue first, with or without a `limit`. Run
 `runStateStoreContract` from `@adonis-agora/durable/testing/conformance`, which
-covers all three and fails a store missing either method.
+covers all three and fails a store missing either method. The contract also
+requires `saveCheckpointsIfPending` (batched result settling), `deleteRuns`
+(batched retention/purge deletes), `runFacets`, `pruneBufferedEvents`
+(`eventBufferTtl`) and the `checkpointStats` + `queryCheckpoints` pair (the run
+detail from aggregates, with an ETag the console polls with `If-None-Match`).
+
+The Lucid schema check runs at every boot but is gated by a fingerprint row in
+`durable_schema_meta`: a current schema costs two statements. A stale or missing
+marker runs the full check under `pg_advisory_xact_lock` / MySQL `GET_LOCK`, so
+pods booting together don't race `CREATE TABLE`. Bump `DURABLE_SCHEMA_REVISION`
+with any DDL change; a hand change behind the marker needs `{ force: true }`. On
+MySQL the JSON columns are `longtext` (an old `text` column is widened in place;
+a table rebuild, so run the migration in a maintenance window for big tables),
+and `stores.lucid({ collation })` aligns the tables to the app's collation. A
+JSON column that does not parse reads back as `undefined` and is reported once
+(`onCorruptJsonColumn`) instead of failing every read that touches the row.
+
+To run singleton workflows a custom store also needs `tryAdmitSingleton` (one transaction that
+locks before it reads, built on the exported `singletonAdmissionTags` /
+`singletonAdmissionDenialPatch` including the pre-0.51 holder count, and a terminal `updateRun`/`updateRunIf` that
+drops both singleton markers (`durable:singleton:admitted` and `durable:singleton:tracked`, via `withoutSingletonMarkers`)); without it singletons fall back
+to the old non-atomic scan and the engine warns once naming `StateStore.tryAdmitSingleton`.
 
 Source: `docs/stores/lucid.mdx`, `docs/stores/index.mdx`.
 
 ## Common mistakes
 
-### HIGH setting `stalledThresholdMs` below the longest legitimate step
+### HIGH blocking the event loop longer than `stalledThresholdMs`
 
 Wrong:
 
 ```ts
-transports.queue({ connection: 'redis', stalledThresholdMs: 60_000 }) // imports take ~90min
+// a step body that renders a report synchronously for ~3 minutes, default 2min threshold
+transports.queue({ connection: 'redis' })
 ```
 
 Correct:
 
 ```ts
-transports.queue({ connection: 'redis', stalledThresholdMs: 2 * 60 * 60 * 1000 })
+transports.queue({ connection: 'redis', stalledThresholdMs: 5 * 60 * 1000 })
+// or keep the heavy work async / off the main thread
 ```
 
-Mechanism: a claim's age equals the step's elapsed runtime because nothing
-renews it mid-processing; a threshold below the longest step redelivers jobs
-from merely-slow workers and double-runs the step. For tighter detection use
-per-step `timeoutMs`, which IS heartbeat-aware.
+Mechanism: the transport renews every in-flight claim each half threshold
+(`adapter.renewJobs`), so a long but async step keeps its claim forever. A
+synchronous body starves the renewal timer; once the claim goes unrenewed past
+the threshold the sweep re-delivers it and the step runs twice — reported to
+`onError` as `lost N of M claim(s)`. Raising the threshold to cover step LENGTH
+is no longer needed: before 0.50 the `queue` transport never renewed claims
+(30min default threshold, long steps re-delivered); upgrade instead.
 Source: `docs/transports/queue.mdx` ("Stalled-job reclaim").
 
 ### HIGH hand-copying the DDL into the migration
@@ -309,7 +418,8 @@ transports.db({ partition: 'tenant-a' })
 Mechanism: routing is by handler name — an instance serves whatever names it
 registers, so there is no group to declare. Isolation comes from `partition`
 (suffixes routing tokens `<name>@<partition>`) plus `namespace` (scopes which
-runs this engine polls). Source: `docs/transports/db.mdx` options table,
+runs this engine polls; absent = `'default'` on standalone, every pool on a
+control plane). Source: `docs/transports/db.mdx` options table,
 `docs/concepts/tenancy.mdx`.
 
 ### MEDIUM wrapping a store in CodecStateStore and expecting heartbeats

package/skills/durable-workflows/SKILL.md

@@ -17,7 +17,7 @@ license: MIT
 metadata:
   type: core
   library: "@adonis-agora/durable"
-  library_version: "0.25.0"
+  library_version: "0.51.0"
   framework: adonisjs
 sources:
   - 'DavideCarvalho/adonis-durable:docs/authoring/app-workflows.mdx'
@@ -37,10 +37,17 @@ and registers every exported class — you never call `engine.register` by hand
 (it remains the low-level escape hatch). The convention mirrors `@adonisjs/queue`
 jobs under `app/jobs`; scaffold with `node ace make:workflow order`.
 
+A modular app (one folder per bounded context) lists several directories:
+`workflowsPath` and `stepsPath` in `config/durable.ts` take an array, and so
+does the `source` option of the `workflowsHook` / `stepsHook` codegen hooks.
+Names must be unique across those directories. To register one more directory
+from a preload, call `registerWorkflowsFromDir(engine, dir)` and
+`registerStepsFromDir(engine, dir)`. The engine serves steps through its
+transport (`engine.handle`).
+
 ## Setup
 
 ```ts title="app/workflows/checkout_workflow.ts"
-import { inject } from '@adonisjs/core'
 import { BaseWorkflow } from '@adonis-agora/durable'
 import type { WorkflowCtx } from '@adonis-agora/durable'
 
@@ -62,7 +69,8 @@ export default class CheckoutWorkflow extends BaseWorkflow {
 ```
 
 Run it from anywhere: `CheckoutWorkflow.dispatch(input, { runId })` enqueues;
-`CheckoutWorkflow.start(input)` awaits the settled result. Both statics are
+`CheckoutWorkflow.start(input)` blocks until the run is terminal and returns its
+output. Both statics are
 context-aware — inside a running body they become a linked child /
 fire-and-forget child instead of a top-level run.
 
@@ -75,7 +83,7 @@ Steps are discovered from `app/steps` and served by name automatically. A
 
 ```ts title="app/steps/payment_steps.ts"
 import { Step } from '@adonis-agora/durable'
-import { z } from 'zod'
+import { z } from 'zod' // any Standard Schema library: zod, valibot, arktype, …
 
 export default class PaymentSteps {
   @Step({
@@ -120,6 +128,28 @@ constructor as wiring; keep per-run state in input and step outputs, never on
 `this`. Injected services are for **composing**, not doing work: reach them
 through step refs so the work is checkpointed.
 
+Schemas are **any Standard Schema** (zod 3.24+/4, valibot, arktype, effect
+schema, …) — the package depends on none of them. They fit every schema slot:
+`@Step({ input, output })` / `defineStep` (with an unannotated handler the types
+are inferred from the schema), `static workflow = { inputSchema }` (validated at
+`start`, nothing created on failure; the run stores the input as sent and
+`run` receives the schema's output, re-parsed before every turn — keep the schema
+deterministic and bump `version` when it changes), and `static workflow = { searchAttributes }` (every
+`ctx.upsertSearchAttributes` is validated against the merged attributes; type
+the ctx as `WorkflowCtx<InferSearchAttributes<typeof schema>>` and read back with
+`attributesOf(WorkflowClass, run)`). A failure is a `SchemaValidationError`
+(`FatalError`, code `invalid_input` / `invalid_output` — `schema_validation` for a
+`searchAttributes` write — issues in `details.issues`) — never
+retried.
+
+```ts
+import * as v from 'valibot'
+
+export const charge = defineStep('billing:charge', async (input) => input.amount * 100, {
+  input: v.object({ amount: v.number() }), // input: { amount: number }, inferred
+})
+```
+
 ### Pattern 2 — pausing durably: sleeps, signals, events
 
 All three suspend the run with zero compute and survive restarts.
@@ -151,15 +181,12 @@ events with `{ onEvent: ['payment.settled'] }`; coalesce chatty sources with
 `{ mode: 'batch', maxSize: 50, windowMs: 60_000 }` (receives `{ events: [...] }`).
 
 ```ts title="app/controllers/approvals_controller.ts"
-import { inject } from '@adonisjs/core'
-import { WorkflowEngine } from '@adonis-agora/durable'
+import type { HttpContext } from '@adonisjs/core/http'
+import engine from '@adonis-agora/durable/services/main'
 
-@inject()
 export default class ApprovalsController {
-  constructor(private engine: WorkflowEngine) {}
-
   async approve({ params, request, response }: HttpContext) {
-    await this.engine.signal(`approve:${params.id}`, request.body())
+    await engine.signal(`approve:${params.id}`, request.body())
     return response.noContent()
   }
 }
@@ -181,9 +208,9 @@ use `ctx.allSettled(Workflow, inputs, { concurrency })` instead: the same fan,
 resolving with `{ status: 'fulfilled', value } | { status: 'rejected', id, error, code?, details? }`
 per input and never throwing for a child. Child ids derive from the call position and
 are replay-stable; never pass a random one. `{ concurrency: n }` caps how many
-children run at once (waves of `floor(n / 2)`, two at a time), and the wave goes into each child
-id), so changing it for an in-flight run is non-deterministic, like changing
-`inputs`.
+children run at once (waves of `floor(n / 2)`, two at a time). The wave is part of
+each child id (`<runId>.all.<firstSeq>.w<k>.<i>`), so changing it for an in-flight
+run is non-deterministic, like changing `inputs`.
 
 For the same step over many inputs, pass the step reference (`defineStep` ref
 or `@Step` method) to `ctx.all`. It
@@ -203,8 +230,10 @@ Prefer it to `Promise.all` past a handful of steps (every result of a
 child-workflow `ctx.all` when each item is just a step. A `queue` or a
 `timeoutMs` is rejected on a step fan.
 
-Serialize runs per subject with a singleton — runs sharing a key queue FIFO and
-are race-free across instances:
+Serialize runs per subject with a singleton — runs sharing a key queue FIFO.
+Admission is an atomic store claim (`StateStore.tryAdmitSingleton`, built into the
+Lucid and in-memory stores), and a holder keeps its slot until it settles
+terminally, so replicas never over-admit and a later start never displaces it:
 
 ```ts
 export default class SyncShopWorkflow extends BaseWorkflow {
@@ -264,6 +293,16 @@ export default class DailyReportWorkflow extends BaseWorkflow {
 }
 ```
 
+In `config/durable.ts`, name the workflow with a lazy import, as Adonis does for
+controllers — never a direct class import, which can hang the boot (a service the
+workflow imports awaits `app.booted` while the config is still loading):
+
+```ts
+schedules: [
+  { key: 'daily-report', workflow: () => import('#workflows/daily_report_workflow'), cron: '0 2 * * *' },
+]
+```
+
 Each schedule window starts exactly once across a fleet via an idempotent
 time-bucket run id.
 
@@ -354,7 +393,7 @@ each replay breaks that dedup and re-fires the child. Source:
 Wrong:
 
 ```ts
-const result = await CheckoutWorkflow.start(order) // request held open until settle
+const result = await CheckoutWorkflow.start(order) // request held open until the run is terminal
 ```
 
 Correct:
@@ -364,8 +403,9 @@ const { runId } = await CheckoutWorkflow.dispatch(order, { runId: `checkout:${or
 return response.accepted({ runId })
 ```
 
-Mechanism: `.start` resolves only when the run reaches a terminal state or
-`suspends` — hours later for a long workflow. Source: `docs/getting-started.mdx`
+Mechanism: `.start` resolves only when the run reaches a terminal state — it
+keeps waiting through every sleep/signal suspension, hours later for a long
+workflow. Source: `docs/getting-started.mdx`
 (warning callout).
 
 ### MEDIUM awaiting a webhook with no timeout
@@ -419,5 +459,30 @@ caller suspended on `ctx.callEntity`; model expected rejections as returned
 discriminated results and reserve throws for genuine programmer errors.
 Source: `docs/authoring/entities.mdx` (warning callout).
 
+### MEDIUM the same workflow or step name in two discovery directories
+
+Wrong:
+
+```ts
+// config/durable.ts — both contexts declare a workflow named 'charge' (version 1)
+export default defineConfig({
+  workflowsPath: ['app/orders/flows', 'app/billing/flows'],
+})
+```
+
+Correct:
+
+```ts
+// app/billing/flows/charge_workflow.ts
+static workflow = { name: 'billing.charge', version: '1' }
+```
+
+Mechanism: names route runs and steps, so with several `workflowsPath` /
+`stepsPath` directories boot throws `DuplicateDiscoveryError` naming both files
+when two directories define the same workflow name + version or step name with
+different classes. Prefix names by context. The hooks' `source` list must name
+the same directories, otherwise the barrel and the runtime scan disagree.
+Source: `docs/authoring/app-workflows.mdx` ("Several directories").
+
 See also: `durable-determinism` — why the body must stay deterministic and how
 to change it safely once live.