@hotmeshio/hotmesh 0.25.4 → 0.25.6

package/ISSUE.md

@@ -0,0 +1,64 @@
+## EIGHTH ITEM — a claimed row's post-signal settle can fail, stranding pending state that contradicts the workflow (dashboard-visible)
+
+Two order trails, same signature (hike-mono, 0.25.5/0.7.5):
+- run A order `a0e68dba`: every station row resolved EXCEPT quality-control —
+  yet the QC stationWorker delivered (shipping minted 5s later, order
+  COMPLETED). The QC row sits `pending` + claimed forever.
+- run B order `89942462`: identical, at post-print-qa (claimed 07:34:44,
+  gluing resolved 07:34:54, order COMPLETED 07:35:11; the qa row still
+  pending+claimed 20+ min later).
+
+Trigger context (our side, fixed by us): two associates raced a shallow pond
+and BOTH booked the same order in the same second (audit rows at 07:34:44) —
+so a concurrent claim write was interleaved with the resolve.
+
+The library-side gap: the resolve path delivered the SIGNAL (workflow
+advanced, next station minted) but the row's own transition to `resolved`
+was lost against the concurrent claim — no error surfaced anywhere. The
+result is dashboard-visible corruption: a live, claimable, countdown-bearing
+work item asserting QA is owed on a COMPLETED order (real associates would
+work it). Whatever branch handled this (atomic signal_key resolve or the
+legacy handle.signal → ltResolveEscalation), signal delivery and row settle
+must be atomic — or the settle must retry until the row cannot lie.
+
+Repro shape: claim a signal_key escalation from principal A, and within the
+same second re-claim/book from principal B while principal C resolves by
+metadata. Expect: signal delivered AND row resolved, always.
+
+# RE-FILED — two reports lost when this file was replaced with your 0.25.6 reply (both found the same evening, before it)
+
+## NINTH ITEM (re-filed seventh) — a timeout wake KILLS the collator of a collated wait set; parent hangs forever
+
+`Promise.all(21 × condition(signalId, {timeout:'1320s'}))` in one parent
+(hike-mono simulationDirector, 0.25.5). Signal-path delivery through the
+collation worked (seat 18 resolved cleanly at 04:55:05, its collator_waiter
+slot consumed). At 05:07:25 UTC the FIRST batch of SLA expirations landed
+(vm0..vm4 rows flipped 'expired' in the same second) and the collator job
+(`HfSVaQSgUaYu3XGsCCjyQWy$C`) went **status -1 in that same second**
+(context={}, is_live=t). Every subsequent expiry wake then hit
+`Inactive job …$C` at CollatorService.assertJobActive (collator/index.js:13)
+via Hook.processTimeHookEvent (hook.js:650) and was dropped. End state,
+stable 10+ min: 20 live collator_waiter registry rows, all rows 'expired',
+zero pending engine_streams for the collator, parent wedged in Promise.all
+forever with no surfaced error.
+
+Repro: `await Promise.all([condition('k1',{timeout:'30s'}), condition('k2',{timeout:'30s'})])`,
+no resolver — let both expire. Expect collator -1 on first expiry, parent wedged.
+
+Our standing mitigation: each timeout-bearing wait moved into its own child
+workflow (inline wait), parent collates ExecChild completions — works, but
+collated waits + SLA should compose.
+
+## TENTH ITEM (re-filed follow-up) — second of two near-simultaneous sibling wakes lost (no crash, no restart)
+
+Clean floor, 10 orders: per order, TWO sibling printJobWorkers under one
+dispatcher park signal_key waits; a machine resolves BOTH escalations
+seconds apart. 3-for-3 wedged orders: LEFT worker woke and completed; RIGHT
+worker (resolved moments later) never woke — job row untouched since
+creation, 10+ min, stored-pending-signal path did not recover it. 14/14
+wakes delivered for the 7 orders whose resolves were spread out. Process up
+the whole time (distinct from the reclaim case). Plausibly fixed by commit
+3's atomic wake INSERT ("no interleave window") — flagging so it gets a
+sibling-burst regression test: resolve two waits of one parent within the
+same tick, both must wake. Our mitigation: 0–4s jitter between sibling
+resolves (also more realistic for print hardware).

package/RESPONSE-0.25.6.md

@@ -0,0 +1,77 @@
+# hotmesh 0.25.6 — fixes for the wake-loss and reservation reports (branch: fix/atomic-cancel-wake, three commits)
+
+Everything below is implemented and fail-first tested (each fix was watched failing
+before its change); the branch is in final full-suite verification before merge.
+
+## Commit 1 — cancellation wakes are durable
+
+**Your second addendum (3 of 6 pills slept after the 400ms cancel wave).** Your
+watchers settle waits via `cancel()`, and cancel's wake was a post-commit,
+best-effort publish whose failure was silently discarded. Now the cancellation
+wake commits inside the cancel transaction: a cancel that is durable always
+resumes its `condition()` with `null`. A native six-pill burst test pins your
+wave shape.
+
+Also adds a permanent regression test for your converge topology
+(`Promise.all([executeChild x4, sleep branch])`): the sleep's timehook mints and
+the converge settles.
+
+## Commit 2 — dead reservations reclaim; resolved waits disarm their timers
+
+**Your fifth data point (reservations never reclaimed after process death) — the
+highest-leverage fix.** The fallback poller's discovery query filtered
+`reserved_at IS NULL`, so a dead holder's reservation was invisible forever.
+It now surfaces stale reservations (threshold tracks the configured window;
+heartbeating work never matches), through dedicated index-backed branches, and
+the fixed function re-deploys to existing databases on boot. A crash or deploy
+now costs ≤ ~90 seconds of redelivery latency instead of a permanent wedge —
+and this most plausibly explains your third data point too (wake messages
+claimed by churning per-principal clients that died were never reclaimed).
+
+**Your sixth item (armed-timeout drizzle).** When the signal wins an SLA-gated
+wait, the armed timeout is now disarmed with the settled wait — root and cycled
+dimensions both covered. New timers are jid-stamped so job-scoped purges
+(terminate) remove them as well. Timers armed before the upgrade drain as the
+one-shot benign events you observed.
+
+Also hardens the pending-signal handoff and cancel wake into single atomic
+statements (safe on pooled connections, no interleave window; an in-window
+wake refreshed by a newer signal delivers the newest payload).
+
+## Commit 3 — resolve paths made single-statement; bulk resolve protects waiters
+
+- `resolve()` and `resolveByMetadata()` now use the same single-statement
+  atomic idiom as cancel: row lock, status guard, and the wake INSERT commit
+  as one unit on any connection type.
+- `resolveMany()` now excludes rows backing a live `condition()` waiter — bulk
+  resolution carries no wake, so those rows stay pending for a targeted
+  `resolve()`/`cancel()` instead of silently stranding the workflow.
+- Secured-mode worker responses stamp `jid`, so job-scoped purges address them.
+
+## Status of the remaining items
+
+- **Lost `sleep()` timer (your timer forensics):** still not reproduced — your
+  exact topology mints and converges cleanly under test, and every registration
+  path checks out atomic. The reclaim fix (commit 2) may retire this too if the
+  collator's driving message was claimed by a dying client. The decisive
+  forensic if it recurs: the wedged director's collator job
+  (`SELECT * FROM durable.jobs WHERE key LIKE '%$C'`) plus that collator jid's
+  `engine_streams` rows, no filters — it records exactly which item-cycle
+  stopped.
+- **`createClient` construction churn (your fourth data point):** your
+  per-principal memoization is the right mitigation; an upstream answer
+  (internal per-auth memoization or an explicit `client.close()`) is under
+  design as its own change.
+- **Crash windows on interrupt's parent notification** (found by our internal
+  audit, same family as your wake fixes): being engineered in a dedicated
+  follow-up PR — the straightforward implementation measurably slowed interrupt
+  dispatch, and interrupts race against live completions by design, so it is
+  getting the unhurried version.
+
+## After upgrading
+
+- Restarting workers drains any orphaned-reservation backlog automatically.
+- The settlement guard can be retired again — cancel and resolve wakes are both
+  transactional now, and dead reservations self-heal.
+- Watch for `hook-signal-pending-redelivered` (in-window wake handled) and the
+  reclaim redeliveries after any crash: both are the system working as intended.

package/build/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@hotmeshio/hotmesh",
-    "version": "0.25.4",
+    "version": "0.25.6",
     "description": "Durable Workflow",
     "main": "./build/index.js",
     "types": "./build/index.d.ts",

package/build/services/activities/hook.js

@@ -575,6 +575,9 @@ class Hook extends activity_1.Activity {
             this.context.metadata.jid = jobId;
             this.context.metadata.gid = gId;
             this.context.metadata.dad = dad;
+            //captured for the timeout disarm below: processEvent's state
+            //hydration replaces context.metadata before the disarm runs
+            const waiterJid = jobId;
             // Inline retry for FORBIDDEN: Leg2 arrived in the window between
             // setHookSignal (standalone) and Leg1 transaction.exec(). The 100B
             // ledger digit is not yet visible. Leg1 needs only milliseconds to
@@ -589,6 +592,25 @@ class Hook extends activity_1.Activity {
                     await this.processEvent(status, code, 'hook');
                     if (code === 200) {
                         await taskService.deleteWebHookSignal(this.config.hook.topic, data);
+                        //signal won the race: disarm the SLA timeout leg (the mirror
+                        //of the timeout-won path, which deletes the signal). The
+                        //armed timehook may not survive to fire against the settled
+                        //wait. Addressed by the SIGNAL's jid + dimensional address —
+                        //processEvent has since rehydrated this.context.
+                        if (this.config.sleep) {
+                            try {
+                                await this.store.expireTimeHook?.(waiterJid, this.metadata.aid);
+                            }
+                            catch (error) {
+                                //best-effort: a surviving timer fires once and is
+                                //consumed as an inactive-job event
+                                this.logger.warn('hook-timeout-disarm-error', {
+                                    jid: waiterJid,
+                                    aid: this.metadata.aid,
+                                    error,
+                                });
+                            }
+                        }
                         //clean up orphan pending on the sibling signal topic
                         //  wfs.wait delivered → remove wfs.signal pending
                         //  wfs.signal delivered → remove wfs.wait pending

package/build/services/escalations/client.d.ts

@@ -72,6 +72,16 @@ export declare class EscalationClientService {
     private _makeEngineFactory;
     private _hashConnection;
     private _deliverEscalationSignal;
+    /**
+     * Builds the wake as a webhook message so the store can commit it
+     * INSIDE the resolve/cancel transaction — the wake becomes durable
+     * with the status change, closing the crash window between commit and
+     * post-commit signal delivery. Mirrors `_deliverEscalationSignal`'s
+     * topic fallback chain; returns null when no hook rule is deployed
+     * for any candidate topic (the caller then keeps post-commit
+     * delivery as the only path, preserving prior behavior).
+     */
+    private _buildWakeCommand;
     /**
      * Returns all escalation rows matching the given filters. Each row includes
      * a computed `available` field (true = claimable). Supports `sortBy`,
@@ -122,10 +132,10 @@ export declare class EscalationClientService {
     /**
      * Cancels a pending escalation and delivers a cancellation signal to the
      * waiting workflow so that `condition()` returns `null`. Terminal rows
-     * return `already-terminal`. Signal delivery is best-effort post-commit —
-     * the committed cancelled row is the durable record; any missed delivery
-     * can be detected via a sweep of rows with `status = 'cancelled'` and a
-     * non-null `signal_key`.
+     * return `already-terminal`. The cancellation wake commits inside the
+     * cancel transaction (same durability contract as `resolve()`); when no
+     * hook rule is deployed for any candidate topic, delivery falls back to
+     * a best-effort post-commit publish.
      */
     cancel(id: string, namespace?: string): Promise<CancelEscalationResult>;
     /**
@@ -197,6 +207,12 @@ export declare class EscalationClientService {
      * (still-pending) row's GIN-indexed `metadata` in the single atomic UPDATE.
      * See {@link resolve}.
      */
+    /**
+     * Bulk-resolves standalone escalations (rows with `signal_key = null`).
+     * Rows backing a live `condition()` waiter are excluded — they stay
+     * pending so a targeted `resolve()`/`cancel()` can deliver their wake;
+     * bulk resolution carries no wake and would strand the workflow.
+     */
     resolveMany(params: ResolveManyParams): Promise<EscalationEntry[]>;
     /**
      * Returns dashboard-ready escalation counts. `period` controls the window

package/build/services/escalations/client.js

@@ -5,6 +5,7 @@ const enums_1 = require("../../modules/enums");
 const utils_1 = require("../../modules/utils");
 const hotmesh_1 = require("../hotmesh");
 const types_1 = require("../../types");
+const stream_1 = require("../../types/stream");
 const factory_1 = require("../durable/schemas/factory");
 /**
  * Standalone client for the `public.hmsh_escalations` signal-pause surface.
@@ -145,6 +146,43 @@ class EscalationClientService {
         catch { }
         return delivered;
     }
+    /**
+     * Builds the wake as a webhook message so the store can commit it
+     * INSIDE the resolve/cancel transaction — the wake becomes durable
+     * with the status change, closing the crash window between commit and
+     * post-commit signal delivery. Mirrors `_deliverEscalationSignal`'s
+     * topic fallback chain; returns null when no hook rule is deployed
+     * for any candidate topic (the caller then keeps post-commit
+     * delivery as the only path, preserving prior behavior).
+     */
+    async _buildWakeCommand(ns, topic, signalKey, data) {
+        const hm = await this._engine(null, ns);
+        const engine = hm.engine;
+        const candidates = [topic, `${ns}.wfs.signal`, `${ns}.wfs.wait`].filter(Boolean);
+        for (const candidate of candidates) {
+            try {
+                const hookRule = await engine.taskService.getHookRule(candidate);
+                if (!hookRule)
+                    continue;
+                const [aid] = await engine.getSchema(`.${hookRule.to}`);
+                const streamData = {
+                    type: stream_1.StreamDataType.WEBHOOK,
+                    status: types_1.StreamStatus.SUCCESS,
+                    code: 200,
+                    metadata: { guid: (0, utils_1.guid)(), aid, topic: candidate },
+                    data: { id: signalKey, data },
+                };
+                return {
+                    forSignalKey: signalKey,
+                    message: JSON.stringify(streamData),
+                };
+            }
+            catch {
+                /* candidate not deployed — try the next topic */
+            }
+        }
+        return null;
+    }
     // ─── Public API ─────────────────────────────────────────────────────────────
     /**
      * Returns all escalation rows matching the given filters. Each row includes
@@ -244,18 +282,27 @@ class EscalationClientService {
     /**
      * Cancels a pending escalation and delivers a cancellation signal to the
      * waiting workflow so that `condition()` returns `null`. Terminal rows
-     * return `already-terminal`. Signal delivery is best-effort post-commit —
-     * the committed cancelled row is the durable record; any missed delivery
-     * can be detected via a sweep of rows with `status = 'cancelled'` and a
-     * non-null `signal_key`.
+     * return `already-terminal`. The cancellation wake commits inside the
+     * cancel transaction (same durability contract as `resolve()`); when no
+     * hook rule is deployed for any candidate topic, delivery falls back to
+     * a best-effort post-commit publish.
      */
     async cancel(id, namespace) {
         const ns = namespace ?? factory_1.APP_ID;
         const hm = await this._engine(null, ns);
-        const result = await hm.engine.store.cancelEscalation(id, namespace);
+        const store = hm.engine.store;
+        //pre-build the cancellation wake so it commits INSIDE the cancel
+        //transaction — condition() resumes with null even if the process
+        //dies the instant the cancel commits
+        let wakeCommand = null;
+        const preview = await store.getEscalation(id, namespace);
+        if (preview?.signal_key) {
+            wakeCommand = await this._buildWakeCommand(ns, preview.topic, preview.signal_key, { __escalation_cancelled: true });
+        }
+        const result = await store.cancelEscalation(id, namespace, wakeCommand ?? undefined);
         if (result.ok === true) {
             this._emit('cancelled', result.entry);
-            if (result.entry.signal_key) {
+            if (result.entry.signal_key && !result.wakeEnqueued) {
                 await this._deliverEscalationSignal(ns, result.entry.topic, {
                     id: result.entry.signal_key,
                     data: { __escalation_cancelled: true },
@@ -292,10 +339,19 @@ class EscalationClientService {
         const ns = (params.namespace ?? namespace) ?? factory_1.APP_ID;
         const hm = await this._engine(null, ns);
         const store = hm.engine.store;
-        const dbResult = await store.resolveEscalation({ id: params.id, resolverPayload: params.resolverPayload, metadata: params.metadata });
+        //pre-build the wake so it commits INSIDE the resolve transaction; the
+        //row's signal routing (signal_key, topic) is immutable after creation
+        let wakeCommand = null;
+        const preview = await store.getEscalation(params.id, params.namespace);
+        if (preview?.signal_key) {
+            wakeCommand = await this._buildWakeCommand(ns, preview.topic, preview.signal_key, params.resolverPayload ?? {});
+        }
+        const dbResult = await store.resolveEscalation({ id: params.id, resolverPayload: params.resolverPayload, metadata: params.metadata }, wakeCommand ?? undefined);
         if (!dbResult.ok)
             return dbResult;
-        if (dbResult.signalKey) {
+        if (dbResult.signalKey && !dbResult.wakeEnqueued) {
+            //the wake was not part of the commit (no hook rule found, or the
+            //enqueue was rolled back to its savepoint) — deliver post-commit
             await this._deliverEscalationSignal(ns, dbResult.topic, {
                 id: dbResult.signalKey,
                 data: params.resolverPayload ?? {},
@@ -316,10 +372,22 @@ class EscalationClientService {
         const ns = (params.namespace ?? namespace) ?? factory_1.APP_ID;
         const hm = await this._engine(null, ns);
         const store = hm.engine.store;
-        const dbResult = await store.resolveEscalationByMetadata({ key: params.key, value: params.value, resolverPayload: params.resolverPayload, roles: params.roles, metadata: params.metadata });
+        //peek the row the resolve is expected to lock and pre-build its wake;
+        //the forSignalKey guard covers the race where a different row wins
+        let wakeCommand = null;
+        const preview = await store.peekEscalationByMetadata({
+            key: params.key,
+            value: params.value,
+            roles: params.roles,
+            namespace: params.namespace,
+        });
+        if (preview?.signalKey) {
+            wakeCommand = await this._buildWakeCommand(ns, preview.topic, preview.signalKey, params.resolverPayload ?? {});
+        }
+        const dbResult = await store.resolveEscalationByMetadata({ key: params.key, value: params.value, resolverPayload: params.resolverPayload, roles: params.roles, metadata: params.metadata }, wakeCommand ?? undefined);
         if (!dbResult.ok)
             return dbResult;
-        if (dbResult.signalKey) {
+        if (dbResult.signalKey && !dbResult.wakeEnqueued) {
             await this._deliverEscalationSignal(ns, dbResult.topic, {
                 id: dbResult.signalKey,
                 data: params.resolverPayload ?? {},
@@ -381,6 +449,12 @@ class EscalationClientService {
      * (still-pending) row's GIN-indexed `metadata` in the single atomic UPDATE.
      * See {@link resolve}.
      */
+    /**
+     * Bulk-resolves standalone escalations (rows with `signal_key = null`).
+     * Rows backing a live `condition()` waiter are excluded — they stay
+     * pending so a targeted `resolve()`/`cancel()` can deliver their wake;
+     * bulk resolution carries no wake and would strand the workflow.
+     */
     async resolveMany(params) {
         const hm = await this._engine(null, params.namespace);
         const entries = await hm.engine.store.resolveManyEscalations(params);

package/build/services/store/index.d.ts

@@ -68,8 +68,17 @@ declare abstract class StoreService<Provider extends ProviderClient, Transaction
      * Leg1: Attempts to set the hook signal. If a pending signal occupies
      * the key (race condition), overwrites it and returns the pending data.
      * When called with a transaction, queues the setnxex (no pending detection).
+     *
+     * When `redelivery` is provided (the webhook routing for this hook),
+     * a consumed pending signal is republished as an engine stream message
+     * in the SAME transaction that overwrites the marker — the wake
+     * survives a crash at any instant. `pendingData` is then not returned,
+     * since the store already owns the redelivery.
      */
-    abstract setHookSignal(hook: HookSignal, transaction?: TransactionProvider): Promise<{
+    abstract setHookSignal(hook: HookSignal, transaction?: TransactionProvider, redelivery?: {
+        aid: string;
+        topic: string;
+    }): Promise<{
         success: boolean;
         pendingData?: string;
     }>;

package/build/services/store/providers/postgres/postgres.d.ts

@@ -128,7 +128,10 @@ declare class PostgresStoreService extends StoreService<ProviderClient, Provider
      *
      * In a transaction: queues the setnxex; pending detection deferred.
      */
-    setHookSignal(hook: HookSignal, transaction?: ProviderTransaction): Promise<{
+    setHookSignal(hook: HookSignal, transaction?: ProviderTransaction, redelivery?: {
+        aid: string;
+        topic: string;
+    }): Promise<{
         success: boolean;
         pendingData?: string;
     }>;
@@ -159,6 +162,18 @@ declare class PostgresStoreService extends StoreService<ProviderClient, Provider
      * picks it up — no intermediate table, no polling, fully transactional.
      */
     registerTimeHook(jobId: string, gId: string, activityId: string, _type: WorkListTaskType, deletionTime: number, dad: string, transaction?: ProviderTransaction): Promise<void>;
+    /**
+     * Disarms a scheduled timehook (soft delete) for one activity of a
+     * job — the mirror of registerTimeHook, called when the SIGNAL wins
+     * an SLA-gated wait so the armed timeout cannot fire against the
+     * settled workflow. Scoped by the jid index, then narrowed to the
+     * activity via the message metadata. Dimensional addressing is
+     * deliberately NOT matched: a job's waits on one activity are
+     * sequential (cycle N settles before cycle N+1 arms), so at most one
+     * timer per (jid, aid) is armed at a time, and the signal composite's
+     * address can differ from the stored one at cycle offsets.
+     */
+    expireTimeHook(jobId: string, activityId: string): Promise<number>;
     getNextTask(_listKey?: string): Promise<[
         listKey: string,
         jobId: string,
@@ -270,15 +285,39 @@ declare class PostgresStoreService extends StoreService<ProviderClient, Provider
     claimEscalation(params: import('../../../../types/hmsh_escalations').ClaimEscalationParams): Promise<import('../../../../types/hmsh_escalations').ClaimEscalationResult>;
     claimEscalationByMetadata(params: import('../../../../types/hmsh_escalations').ClaimByMetadataParams): Promise<import('../../../../types/hmsh_escalations').ClaimByMetadataResult>;
     releaseEscalation(params: import('../../../../types/hmsh_escalations').ReleaseEscalationParams): Promise<import('../../../../types/hmsh_escalations').ReleaseEscalationResult>;
-    resolveEscalation(params: import('../../../../types/hmsh_escalations').ResolveEscalationParams): Promise<import('../../../../types/hmsh_escalations').ResolveEscalationResult & {
+    /**
+     * Composes the wake as a data-modifying CTE appended to a settle
+     * statement (resolve/cancel). The INSERT fires iff the settle CTE
+     * (`fromCTE`) produced a row whose signal_key matches the wake — one
+     * atomic statement, safe on any connection type (Client or Pool),
+     * with no transaction window for unrelated queries to interleave
+     * into. Appends the wake params to `params` in place.
+     */
+    private composeEscalationWakeCTE;
+    resolveEscalation(params: import('../../../../types/hmsh_escalations').ResolveEscalationParams, wakeCommand?: import('../../../../types/hmsh_escalations').EscalationWakeCommand): Promise<import('../../../../types/hmsh_escalations').ResolveEscalationResult & {
         signalKey?: string | null;
         topic?: string | null;
+        wakeEnqueued?: boolean;
     }>;
-    resolveEscalationByMetadata(params: import('../../../../types/hmsh_escalations').ResolveByMetadataParams): Promise<import('../../../../types/hmsh_escalations').ResolveEscalationResult & {
+    /**
+     * Non-locking preview of the row `resolveEscalationByMetadata` would
+     * select — used to pre-build the wake command before the resolve
+     * transaction opens. The `forSignalKey` guard on the wake command
+     * handles the race where a different row wins the lock.
+     */
+    peekEscalationByMetadata(params: import('../../../../types/hmsh_escalations').ResolveByMetadataParams): Promise<{
+        id: string;
+        signalKey: string | null;
+        topic: string | null;
+    } | null>;
+    resolveEscalationByMetadata(params: import('../../../../types/hmsh_escalations').ResolveByMetadataParams, wakeCommand?: import('../../../../types/hmsh_escalations').EscalationWakeCommand): Promise<import('../../../../types/hmsh_escalations').ResolveEscalationResult & {
         signalKey?: string | null;
         topic?: string | null;
+        wakeEnqueued?: boolean;
+    }>;
+    cancelEscalation(id: string, namespace?: string, wakeCommand?: import('../../../../types/hmsh_escalations').EscalationWakeCommand): Promise<import('../../../../types/hmsh_escalations').CancelEscalationResult & {
+        wakeEnqueued?: boolean;
     }>;
-    cancelEscalation(id: string, namespace?: string): Promise<import('../../../../types/hmsh_escalations').CancelEscalationResult>;
     escalateEscalationToRole(params: import('../../../../types/hmsh_escalations').EscalateToRoleParams): Promise<import('../../../../types/hmsh_escalations').EscalationEntry | null>;
     updateEscalation(params: import('../../../../types/hmsh_escalations').UpdateEscalationParams): Promise<import('../../../../types/hmsh_escalations').EscalationEntry | null>;
     appendEscalationMilestones(params: import('../../../../types/hmsh_escalations').AppendMilestonesParams): Promise<import('../../../../types/hmsh_escalations').EscalationEntry | null>;

package/build/services/store/providers/postgres/postgres.js

@@ -777,7 +777,7 @@ class PostgresStoreService extends __1.StoreService {
      *
      * In a transaction: queues the setnxex; pending detection deferred.
      */
-    async setHookSignal(hook, transaction) {
+    async setHookSignal(hook, transaction, redelivery) {
         const key = this.mintKey(key_1.KeyType.SIGNALS, { appId: this.appId });
         const { topic, resolved, jobId } = hook;
         const signalKey = `${topic}:${resolved}`;
@@ -813,6 +813,54 @@ class PostgresStoreService extends __1.StoreService {
             const captured = lockResult.rows[0]?.value;
             const wasInsert = lockResult.rows[0]?.inserted;
             const isPending = captured?.startsWith('$pending::');
+            if (isPending && redelivery) {
+                //consume the marker and commit its redelivery as ONE statement:
+                //the pending wake becomes a durable engine stream message in
+                //the same atomic unit that destroys the marker, so the wake
+                //survives a crash at any instant. Single-statement composition
+                //is safe on any connection type (Client or Pool) and cannot
+                //interleave with unrelated queries. The consume is guarded on
+                //the value still being a pending marker, and the redelivered
+                //payload is read from the row itself — a marker refreshed by a
+                //newer signal between capture and consume delivers the NEWER
+                //payload rather than a stale copy.
+                const schemaName = this.kvsql().safeName(this.appId);
+                const res = await this.pgClient.query(`WITH prev AS (
+             SELECT value FROM ${tableName}
+             WHERE key = $1 FOR UPDATE
+           ),
+           consumed AS (
+             UPDATE ${tableName} t
+             SET value = $2, expiry = NOW() + INTERVAL '${delay} seconds'
+             FROM prev
+             WHERE t.key = $1 AND prev.value LIKE '$pending::%'
+             RETURNING prev.value AS pending_value
+           )
+           INSERT INTO ${schemaName}.engine_streams
+             (stream_name, message, priority)
+           SELECT $3,
+             json_build_object(
+               'type', 'webhook',
+               'status', 'success',
+               'code', 200,
+               'metadata', json_build_object(
+                 'guid', $4::text, 'aid', $5::text, 'topic', $6::text),
+               'data', substr(consumed.pending_value, 11)::json
+             )::text,
+             5
+           FROM consumed
+           RETURNING id`, [storedKey, jobId, this.appId, (0, utils_1.guid)(), redelivery.aid, redelivery.topic]);
+                if ((res.rowCount ?? 0) > 0) {
+                    this.logger.warn('hook-signal-pending-redelivered', {
+                        key: signalKey,
+                        topic: redelivery.topic,
+                    });
+                    return { success: true };
+                }
+                //the marker vanished between capture and consume (a concurrent
+                //duplicate Leg1 consumed it) — idempotent no-op
+                return { success: false };
+            }
             if (!wasInsert) {
                 //step 2: row existed — overwrite with hook value
                 await this.pgClient.query(`UPDATE ${tableName}
@@ -983,10 +1031,12 @@ class PostgresStoreService extends __1.StoreService {
             metadata: { guid: (0, utils_1.guid)(), jid: jobId, gid: gId, dad: msgDad, aid },
             data: { timestamp: Date.now() },
         });
+        //jid column stamped so job-scoped expiry (interrupt purge, timeout
+        //disarm) can address armed timers through the partial jid index
         const sql = `INSERT INTO ${schemaName}.engine_streams
-      (stream_name, message, priority, visible_at)
-      VALUES ($1, $2, 5, NOW() + INTERVAL '${Math.max(delayMs, 0)} milliseconds')`;
-        const params = [this.appId, message];
+      (stream_name, jid, message, priority, visible_at)
+      VALUES ($1, $2, $3, 5, NOW() + INTERVAL '${Math.max(delayMs, 0)} milliseconds')`;
+        const params = [this.appId, jobId, message];
         if (transaction && typeof transaction.addCommand === 'function') {
             transaction.addCommand(sql, params);
         }
@@ -994,6 +1044,26 @@ class PostgresStoreService extends __1.StoreService {
             await this.pgClient.query(sql, params);
         }
     }
+    /**
+     * Disarms a scheduled timehook (soft delete) for one activity of a
+     * job — the mirror of registerTimeHook, called when the SIGNAL wins
+     * an SLA-gated wait so the armed timeout cannot fire against the
+     * settled workflow. Scoped by the jid index, then narrowed to the
+     * activity via the message metadata. Dimensional addressing is
+     * deliberately NOT matched: a job's waits on one activity are
+     * sequential (cycle N settles before cycle N+1 arms), so at most one
+     * timer per (jid, aid) is armed at a time, and the signal composite's
+     * address can differ from the stored one at cycle offsets.
+     */
+    async expireTimeHook(jobId, activityId) {
+        const schemaName = this.kvsql().safeName(this.appId);
+        const res = await this.pgClient.query(`UPDATE ${schemaName}.engine_streams
+       SET expired_at = NOW()
+       WHERE jid = $1 AND expired_at IS NULL AND visible_at > NOW()
+         AND (message::jsonb ->> 'type') = 'timehook'
+         AND (message::jsonb -> 'metadata' ->> 'aid') = $2`, [jobId, activityId]);
+        return res.rowCount ?? 0;
+    }
     async getNextTask(_listKey) {
         //no-op: time hooks are now engine_streams messages with future visible_at.
         //the engine's normal dequeue handles them when they become visible.
@@ -1792,101 +1862,183 @@ class PostgresStoreService extends __1.StoreService {
             return { ok: false, reason: 'wrong-assignee' };
         return { ok: true, entry: row.entry_json };
     }
-    async resolveEscalation(params) {
+    /**
+     * Composes the wake as a data-modifying CTE appended to a settle
+     * statement (resolve/cancel). The INSERT fires iff the settle CTE
+     * (`fromCTE`) produced a row whose signal_key matches the wake — one
+     * atomic statement, safe on any connection type (Client or Pool),
+     * with no transaction window for unrelated queries to interleave
+     * into. Appends the wake params to `params` in place.
+     */
+    composeEscalationWakeCTE(wakeCommand, params, fromCTE) {
+        if (!wakeCommand) {
+            return { wakeCTE: '', wakeCount: '0::int AS wake_count' };
+        }
+        const schemaName = this.kvsql().safeName(this.appId);
+        const base = params.length;
+        params.push(this.appId, wakeCommand.message, wakeCommand.forSignalKey);
+        return {
+            wakeCTE: `,
+        wake AS (
+          INSERT INTO ${schemaName}.engine_streams (stream_name, message, priority)
+          SELECT $${base + 1}, $${base + 2}, 5 FROM ${fromCTE}
+          WHERE ${fromCTE}.signal_key = $${base + 3}
+          RETURNING id
+        )`,
+            wakeCount: '(SELECT COUNT(*) FROM wake)::int AS wake_count',
+        };
+    }
+    async resolveEscalation(params, wakeCommand) {
         const { id, namespace, resolverPayload, metadata } = params;
         const payloadJson = resolverPayload ? JSON.stringify(resolverPayload) : null;
-        // metaJson is bound at a fixed index; the CASE no-ops when null so resolution
-        // can merge into the GIN-indexed metadata without disturbing the param layout.
         const metaJson = metadata ? JSON.stringify(metadata) : null;
-        // Explicit transaction: FOR UPDATE locks the row; WHERE guard on UPDATE is the TOCTOU
-        // barrier — a concurrent caller whose UPDATE matches 0 rows sees 'already-resolved'.
-        // On crash before COMMIT neither write lands; on crash after COMMIT both are durable.
-        // The resolved row's signal_key is the durable proof for recovery sweeps.
-        await this.pgClient.query('BEGIN');
-        try {
-            const lockResult = await this.pgClient.query(`SELECT id, signal_key, topic, status FROM public.hmsh_escalations
-         WHERE id = $1 ${namespace ? 'AND namespace = $2' : ''} FOR UPDATE`, namespace ? [id, namespace] : [id]);
-            if (!lockResult.rows[0]) {
-                await this.pgClient.query('ROLLBACK');
-                return { ok: false, reason: 'not-found' };
-            }
-            const { signal_key, topic, status } = lockResult.rows[0];
-            if (status === 'cancelled') {
-                await this.pgClient.query('ROLLBACK');
+        //single statement: FOR UPDATE lock, TOCTOU-guarded UPDATE, and the
+        //wake INSERT are one atomic unit. The wake commits WITH the
+        //resolve — a crash after the statement leaves both the resolved row
+        //and its wake message durable; before, neither. The engine_streams
+        //INSERT trigger emits the delivery NOTIFY on commit. Safe on any
+        //connection type; no window for unrelated queries to interleave.
+        const sqlParams = namespace
+            ? [id, payloadJson, metaJson, namespace]
+            : [id, payloadJson, metaJson];
+        const { wakeCTE, wakeCount } = this.composeEscalationWakeCTE(wakeCommand, sqlParams, 'resolved');
+        const result = await this.pgClient.query(`
+      WITH target AS MATERIALIZED (
+        SELECT id, signal_key, topic, status FROM public.hmsh_escalations
+        WHERE id = $1 ${namespace ? 'AND namespace = $4' : ''}
+        LIMIT 1 FOR UPDATE
+      ),
+      resolved AS (
+        UPDATE public.hmsh_escalations e
+        SET status = 'resolved', resolved_at = NOW(), resolver_payload = $2,
+            metadata = CASE WHEN $3::jsonb IS NOT NULL
+                            THEN COALESCE(e.metadata, '{}'::jsonb) || $3::jsonb
+                            ELSE e.metadata END,
+            updated_at = NOW()
+        FROM target
+        WHERE e.id = target.id AND target.status = 'pending'
+        RETURNING e.*
+      )${wakeCTE}
+      SELECT t.id, t.status AS prior_status, t.signal_key, t.topic,
+        CASE
+          WHEN r.id IS NOT NULL THEN 'resolved'
+          WHEN t.id IS NULL     THEN 'not-found'
+          ELSE 'blocked'
+        END AS outcome,
+        row_to_json(r.*) AS entry_json,
+        ${wakeCount}
+      FROM (SELECT * FROM target) t
+      FULL OUTER JOIN (SELECT * FROM resolved) r ON r.id = t.id
+    `, sqlParams);
+        const row = result.rows[0];
+        if (!row || row.outcome === 'not-found')
+            return { ok: false, reason: 'not-found' };
+        if (row.outcome === 'blocked') {
+            if (row.prior_status === 'cancelled')
                 return { ok: false, reason: 'already-cancelled' };
-            }
-            if (status === 'expired') {
+            if (row.prior_status === 'expired') {
                 // The wait's SLA timer fired first and the workflow resumed with
                 // false — the resolver payload has nowhere to go. Name it, so the
                 // operator learns the deadline passed rather than believing the
                 // resolution landed.
-                await this.pgClient.query('ROLLBACK');
                 return { ok: false, reason: 'already-expired' };
             }
-            const updateResult = await this.pgClient.query(`UPDATE public.hmsh_escalations
-         SET status = 'resolved', resolved_at = NOW(), resolver_payload = $2,
-             metadata = CASE WHEN $3::jsonb IS NOT NULL
-                             THEN COALESCE(metadata, '{}'::jsonb) || $3::jsonb
-                             ELSE metadata END,
-             updated_at = NOW()
-         WHERE id = $1 ${namespace ? 'AND namespace = $4' : ''} AND status = 'pending'
-         RETURNING *`, namespace ? [id, payloadJson, metaJson, namespace] : [id, payloadJson, metaJson]);
-            if (!updateResult.rows[0]) {
-                await this.pgClient.query('ROLLBACK');
-                return { ok: false, reason: 'already-resolved' };
-            }
-            await this.pgClient.query('COMMIT');
-            return { ok: true, entry: updateResult.rows[0], signalKey: signal_key, topic };
-        }
-        catch (e) {
-            await this.pgClient.query('ROLLBACK');
-            throw e;
+            return { ok: false, reason: 'already-resolved' };
         }
+        return {
+            ok: true,
+            entry: row.entry_json,
+            signalKey: row.signal_key,
+            topic: row.topic,
+            wakeEnqueued: row.wake_count > 0,
+        };
+    }
+    /**
+     * Non-locking preview of the row `resolveEscalationByMetadata` would
+     * select — used to pre-build the wake command before the resolve
+     * transaction opens. The `forSignalKey` guard on the wake command
+     * handles the race where a different row wins the lock.
+     */
+    async peekEscalationByMetadata(params) {
+        const { key, value, namespace, roles } = params;
+        const filter = JSON.stringify({ [key]: value });
+        const result = await this.pgClient.query(`SELECT id, signal_key, topic FROM public.hmsh_escalations
+       WHERE ${namespace ? 'namespace = $3 AND' : ''}
+             metadata @> $1::jsonb
+         AND ($2::text[] IS NULL OR role = ANY($2::text[]))
+         AND status IN ('pending', 'cancelled')
+       ORDER BY priority ASC, created_at ASC
+       LIMIT 1`, namespace ? [filter, roles ?? null, namespace] : [filter, roles ?? null]);
+        const row = result.rows[0];
+        if (!row)
+            return null;
+        return { id: row.id, signalKey: row.signal_key, topic: row.topic };
     }
-    async resolveEscalationByMetadata(params) {
+    async resolveEscalationByMetadata(params, wakeCommand) {
         const { key, value, namespace, resolverPayload, roles, metadata } = params;
         const filter = JSON.stringify({ [key]: value });
         const payloadJson = resolverPayload ? JSON.stringify(resolverPayload) : null;
         const metaJson = metadata ? JSON.stringify(metadata) : null;
-        await this.pgClient.query('BEGIN');
-        try {
-            const lockResult = await this.pgClient.query(`SELECT id, signal_key, topic, status FROM public.hmsh_escalations
-         WHERE ${namespace ? 'namespace = $3 AND' : ''}
-               metadata @> $1::jsonb
-           AND ($2::text[] IS NULL OR role = ANY($2::text[]))
-           AND status IN ('pending', 'cancelled')
-         ORDER BY priority ASC, created_at ASC
-         LIMIT 1 FOR UPDATE`, namespace ? [filter, roles ?? null, namespace] : [filter, roles ?? null]);
-            if (!lockResult.rows[0]) {
-                await this.pgClient.query('ROLLBACK');
-                return { ok: false, reason: 'not-found' };
-            }
-            const { id, signal_key, topic, status } = lockResult.rows[0];
-            if (status === 'cancelled') {
-                await this.pgClient.query('ROLLBACK');
+        //single statement — see resolveEscalation for the atomicity contract
+        const sqlParams = namespace
+            ? [filter, roles ?? null, payloadJson, metaJson, namespace]
+            : [filter, roles ?? null, payloadJson, metaJson];
+        const { wakeCTE, wakeCount } = this.composeEscalationWakeCTE(wakeCommand, sqlParams, 'resolved');
+        const result = await this.pgClient.query(`
+      WITH target AS MATERIALIZED (
+        SELECT id, signal_key, topic, status FROM public.hmsh_escalations
+        WHERE ${namespace ? 'namespace = $5 AND' : ''}
+              metadata @> $1::jsonb
+          AND ($2::text[] IS NULL OR role = ANY($2::text[]))
+          AND status IN ('pending', 'cancelled')
+        ORDER BY priority ASC, created_at ASC
+        LIMIT 1 FOR UPDATE
+      ),
+      resolved AS (
+        UPDATE public.hmsh_escalations e
+        SET status = 'resolved', resolved_at = NOW(), resolver_payload = $3,
+            metadata = CASE WHEN $4::jsonb IS NOT NULL
+                            THEN COALESCE(e.metadata, '{}'::jsonb) || $4::jsonb
+                            ELSE e.metadata END,
+            updated_at = NOW()
+        FROM target
+        WHERE e.id = target.id AND target.status = 'pending'
+        RETURNING e.*
+      )${wakeCTE}
+      SELECT t.id, t.status AS prior_status, t.signal_key, t.topic,
+        CASE
+          WHEN r.id IS NOT NULL THEN 'resolved'
+          WHEN t.id IS NULL     THEN 'not-found'
+          ELSE 'blocked'
+        END AS outcome,
+        row_to_json(r.*) AS entry_json,
+        ${wakeCount}
+      FROM (SELECT * FROM target) t
+      FULL OUTER JOIN (SELECT * FROM resolved) r ON r.id = t.id
+    `, sqlParams);
+        const row = result.rows[0];
+        if (!row || row.outcome === 'not-found')
+            return { ok: false, reason: 'not-found' };
+        if (row.outcome === 'blocked') {
+            if (row.prior_status === 'cancelled')
                 return { ok: false, reason: 'already-cancelled' };
-            }
-            const updateResult = await this.pgClient.query(`UPDATE public.hmsh_escalations
-         SET status = 'resolved', resolved_at = NOW(), resolver_payload = $2,
-             metadata = CASE WHEN $3::jsonb IS NOT NULL
-                             THEN COALESCE(metadata, '{}'::jsonb) || $3::jsonb
-                             ELSE metadata END,
-             updated_at = NOW()
-         WHERE id = $1 AND status = 'pending'
-         RETURNING *`, [id, payloadJson, metaJson]);
-            if (!updateResult.rows[0]) {
-                await this.pgClient.query('ROLLBACK');
-                return { ok: false, reason: 'already-resolved' };
-            }
-            await this.pgClient.query('COMMIT');
-            return { ok: true, entry: updateResult.rows[0], signalKey: signal_key, topic };
-        }
-        catch (e) {
-            await this.pgClient.query('ROLLBACK');
-            throw e;
+            return { ok: false, reason: 'already-resolved' };
         }
+        return {
+            ok: true,
+            entry: row.entry_json,
+            signalKey: row.signal_key,
+            topic: row.topic,
+            wakeEnqueued: row.wake_count > 0,
+        };
     }
-    async cancelEscalation(id, namespace) {
+    async cancelEscalation(id, namespace, wakeCommand) {
+        //single statement: the cancellation wake is a data-modifying CTE
+        //that fires iff the cancel lands AND the row's signal_key matches
+        //the wake — atomic on any connection type (Client or Pool), with
+        //no BEGIN/COMMIT window for unrelated queries to interleave into
+        const params = namespace ? [id, namespace] : [id];
+        const { wakeCTE, wakeCount } = this.composeEscalationWakeCTE(wakeCommand, params, 'cancelled');
         const result = await this.pgClient.query(`
       WITH target AS MATERIALIZED (
         SELECT id, status FROM public.hmsh_escalations
@@ -1900,23 +2052,24 @@ class PostgresStoreService extends __1.StoreService {
         WHERE public.hmsh_escalations.id = target.id
           AND target.status = 'pending'
         RETURNING public.hmsh_escalations.*
-      )
+      )${wakeCTE}
       SELECT t.id, t.status AS prior_status,
         CASE
           WHEN c.id IS NOT NULL THEN 'cancelled'
           WHEN t.id IS NULL     THEN 'not-found'
           ELSE 'already-terminal'
         END AS outcome,
-        row_to_json(c.*) AS entry_json
+        row_to_json(c.*) AS entry_json,
+        ${wakeCount}
       FROM (SELECT * FROM target) t
       FULL OUTER JOIN (SELECT * FROM cancelled) c ON c.id = t.id
-    `, namespace ? [id, namespace] : [id]);
+    `, params);
         if (!result.rows[0] || result.rows[0].outcome === 'not-found')
             return { ok: false, reason: 'not-found' };
         if (result.rows[0].outcome === 'already-terminal')
             return { ok: false, reason: 'already-terminal' };
         const entry = result.rows[0].entry_json;
-        return { ok: true, entry };
+        return { ok: true, entry, wakeEnqueued: result.rows[0].wake_count > 0 };
     }
     async escalateEscalationToRole(params) {
         const { id, targetRole, namespace } = params;
@@ -2065,6 +2218,7 @@ class PostgresStoreService extends __1.StoreService {
        WHERE id = ANY($2::uuid[])
          ${namespace ? 'AND namespace = $4' : ''}
          AND status = 'pending'
+         AND signal_key IS NULL
        RETURNING *`, namespace ? [payloadJson, ids, metaJson, namespace] : [payloadJson, ids, metaJson]);
         return result.rows;
     }

package/build/services/stream/providers/postgres/kvtables.js

@@ -35,6 +35,10 @@ async function deploySchema(streamClient, appId, logger) {
                 await ensureIndexes(client, schemaName);
                 await ensureProcedures(client, schemaName);
                 await ensureStatementLevelTriggers(client, schemaName);
+                // Re-deploy the fallback poller's discovery function so existing
+                // databases receive predicate changes (v0.25.6: stale-reservation
+                // reclaim — see getNotifyVisibleMessagesSQL)
+                await client.query(getNotifyVisibleMessagesSQL(schemaName));
             }
             finally {
                 await client.query('SELECT pg_advisory_unlock($1)', [lockId]);
@@ -177,6 +181,19 @@ async function ensureIndexes(client, schemaName) {
     CREATE INDEX IF NOT EXISTS idx_worker_streams_message_fetch
     ON ${workerTable} (stream_name, priority DESC, id)
     WHERE expired_at IS NULL;
+  `);
+    // v0.25.6: in-flight partial indexes — bounded to currently reserved
+    // rows (≈ concurrent executions) — serve the fallback poller's
+    // stale-reservation reclaim branch (see getNotifyVisibleMessagesSQL)
+    await client.query(`
+    CREATE INDEX IF NOT EXISTS idx_engine_streams_inflight
+    ON ${engineTable} (stream_name, reserved_at)
+    WHERE expired_at IS NULL AND reserved_at IS NOT NULL;
+  `);
+    await client.query(`
+    CREATE INDEX IF NOT EXISTS idx_worker_streams_inflight
+    ON ${workerTable} (stream_name, reserved_at)
+    WHERE expired_at IS NULL AND reserved_at IS NOT NULL;
   `);
     // v0.18.0: add jid column to engine_streams for job tracing
     await client.query(`ALTER TABLE ${engineTable} ADD COLUMN IF NOT EXISTS jid TEXT NOT NULL DEFAULT ''`);
@@ -461,7 +478,34 @@ async function createNotificationTriggers(client, schemaName) {
       EXECUTE FUNCTION ${schemaName}.notify_new_worker_stream_message();
   `);
     // ---- Visibility timeout notification function (queries both tables) ----
-    await client.query(`
+    await client.query(getNotifyVisibleMessagesSQL(schemaName));
+}
+/**
+ * The fallback poller's discovery function. Surfaces streams that have
+ * deliverable work: unreserved visible messages AND stale reservations
+ * whose holder died (heartbeats refresh reserved_at every ~15s while an
+ * activity runs, so any reservation older than 60s has a dead holder or
+ * is already claimable). Stale-reservation discovery is the ONLY path
+ * that reclaims orphaned in-flight work on a quiet stream — without it,
+ * a process death wedges its reserved messages forever. A premature
+ * surface (reservation window configured above 60s on a provider with
+ * static leases) costs one empty fetch; the claim query enforces the
+ * real window.
+ *
+ * Deployed on fresh schemas AND re-deployed on every boot (see
+ * deploySchema) so existing databases receive predicate changes.
+ */
+function getNotifyVisibleMessagesSQL(schemaName) {
+    const engineTable = `${schemaName}.engine_streams`;
+    const workerTable = `${schemaName}.worker_streams`;
+    //stale threshold tracks the configured base window: heartbeats refresh
+    //reserved_at every base/2 (15s default), so live holders never look
+    //stale; 2x base keeps static-lease holders (secured workers, one
+    //adaptive doubling) from surfacing as false positives. Baked at
+    //deploy time from the deploying node's env — a premature surface
+    //costs one empty fetch (the claim query enforces the real window).
+    const staleSeconds = Math.max(60, enums_1.HMSH_RESERVATION_TIMEOUT_S * 2);
+    return `
     CREATE OR REPLACE FUNCTION ${schemaName}.notify_visible_messages()
     RETURNS INTEGER AS $$
     DECLARE
@@ -470,13 +514,23 @@ async function createNotificationTriggers(client, schemaName) {
       payload JSON;
       notification_count INTEGER := 0;
     BEGIN
-      -- Engine streams
+      -- Engine streams: visible unreserved work (active_messages
+      -- partial index) UNION stale reservations whose holder died
+      -- (inflight partial index) — each branch index-matched
       FOR msg IN
-        SELECT DISTINCT stream_name
-        FROM ${engineTable}
-        WHERE visible_at <= NOW()
-          AND reserved_at IS NULL
-          AND expired_at IS NULL
+        SELECT DISTINCT u.stream_name FROM (
+          (SELECT stream_name FROM ${engineTable}
+           WHERE visible_at <= NOW()
+             AND reserved_at IS NULL
+             AND expired_at IS NULL
+           LIMIT 50)
+          UNION ALL
+          (SELECT stream_name FROM ${engineTable}
+           WHERE expired_at IS NULL
+             AND reserved_at IS NOT NULL
+             AND reserved_at < NOW() - INTERVAL '${staleSeconds} seconds'
+           LIMIT 50)
+        ) u
         LIMIT 50
       LOOP
         channel_name := 'eng_' || msg.stream_name;
@@ -493,13 +547,21 @@ async function createNotificationTriggers(client, schemaName) {
         notification_count := notification_count + 1;
       END LOOP;
 
-      -- Worker streams
+      -- Worker streams: same two index-matched branches
       FOR msg IN
-        SELECT DISTINCT stream_name
-        FROM ${workerTable}
-        WHERE visible_at <= NOW()
-          AND reserved_at IS NULL
-          AND expired_at IS NULL
+        SELECT DISTINCT u.stream_name FROM (
+          (SELECT stream_name FROM ${workerTable}
+           WHERE visible_at <= NOW()
+             AND reserved_at IS NULL
+             AND expired_at IS NULL
+           LIMIT 50)
+          UNION ALL
+          (SELECT stream_name FROM ${workerTable}
+           WHERE expired_at IS NULL
+             AND reserved_at IS NOT NULL
+             AND reserved_at < NOW() - INTERVAL '${staleSeconds} seconds'
+           LIMIT 50)
+        ) u
         LIMIT 50
       LOOP
         channel_name := 'wrk_' || msg.stream_name;
@@ -519,7 +581,7 @@ async function createNotificationTriggers(client, schemaName) {
       RETURN notification_count;
     END;
     $$ LANGUAGE plpgsql;
-  `);
+  `;
 }
 function getNotificationChannelName(streamName, isEngine) {
     const prefix = isEngine ? 'eng_' : 'wrk_';

package/build/services/stream/providers/postgres/messages.js

@@ -296,12 +296,15 @@ async function dropDeadJobMessages(client, tableName, streamName, rows, liveness
         }
     }
     catch (error) {
-        if (error?.code === '42P01') {
-            //jobs table is not visible from this connection; the guard cannot
-            //run here — interrupt-time purging (expireJobMessages) still applies
+        if (error?.code === '42P01' || error?.code === '42501') {
+            //jobs table is not visible (42P01) or not readable (42501) from
+            //this connection; the guard cannot run here — interrupt-time
+            //purging (expireJobMessages) still applies. Self-disable so the
+            //hot path stops issuing a failing cross-table query per fetch.
             liveness.enabled = false;
             logger.info('postgres-stream-liveness-guard-disabled', {
                 jobsTable: liveness.jobsTable,
+                code: error.code,
             });
         }
         else {

package/build/services/stream/providers/postgres/procedures.js

@@ -150,17 +150,19 @@ function getCreateProceduresSQL(schemaName) {
       -- Engine stream_name is the schema/appId name
       engine_stream_name := '${schemaName}';
 
+      -- jid stamped from the message metadata (parity with the raw
+      -- publish path) so job-scoped expiry can address these rows
       IF p_max_retry_attempts IS NOT NULL THEN
         INSERT INTO ${engineTable}
-          (stream_name, message, priority, max_retry_attempts, backoff_coefficient, maximum_interval_seconds, visible_at, retry_attempt)
+          (stream_name, jid, message, priority, max_retry_attempts, backoff_coefficient, maximum_interval_seconds, visible_at, retry_attempt)
         VALUES
-          (engine_stream_name, p_message, p_priority, p_max_retry_attempts, p_backoff_coefficient, p_maximum_interval_seconds, p_visible_at, p_retry_attempt)
+          (engine_stream_name, COALESCE(p_message::jsonb -> 'metadata' ->> 'jid', ''), p_message, p_priority, p_max_retry_attempts, p_backoff_coefficient, p_maximum_interval_seconds, p_visible_at, p_retry_attempt)
         RETURNING id INTO new_id;
       ELSE
         INSERT INTO ${engineTable}
-          (stream_name, message, priority, visible_at, retry_attempt)
+          (stream_name, jid, message, priority, visible_at, retry_attempt)
         VALUES
-          (engine_stream_name, p_message, p_priority, p_visible_at, p_retry_attempt)
+          (engine_stream_name, COALESCE(p_message::jsonb -> 'metadata' ->> 'jid', ''), p_message, p_priority, p_visible_at, p_retry_attempt)
         RETURNING id INTO new_id;
       END IF;
 

package/build/services/task/index.js

@@ -73,8 +73,13 @@ class TaskService {
                 expire,
             };
             //called standalone (no transaction) so the single CTE query can
-            //atomically detect and return pending signal data on collision
-            const result = await this.store.setHookSignal(hook);
+            //atomically detect and return pending signal data on collision.
+            //the redelivery routing lets the store republish a consumed
+            //pending signal durably, in the same transaction that consumes it
+            const result = await this.store.setHookSignal(hook, undefined, {
+                aid: hookRule.to,
+                topic,
+            });
             if (result.pendingData) {
                 this.logger.warn('task-signal-race-pending-consumed', {
                     topic,

package/build/types/hmsh_escalations.d.ts

@@ -105,6 +105,18 @@ export type ResolveEscalationResult = {
     ok: false;
     reason: 'not-found' | 'already-resolved' | 'already-cancelled' | 'already-expired';
 };
+/**
+ * A pre-built wake message, committed INSIDE the resolve/cancel transaction
+ * so the awaiting workflow's wake is durable with the status change. The
+ * client composes the webhook message; the store owns the stream INSERT.
+ * `forSignalKey` pins the wake to the row it was built for — it is written
+ * only when the affected row's `signal_key` matches (a mismatched row falls
+ * back to post-commit delivery).
+ */
+export interface EscalationWakeCommand {
+    forSignalKey: string;
+    message: string;
+}
 export type ReleaseEscalationResult = {
     ok: true;
     entry: EscalationEntry;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hotmeshio/hotmesh",
-  "version": "0.25.4",
+  "version": "0.25.6",
   "description": "Durable Workflow",
   "main": "./build/index.js",
   "types": "./build/index.d.ts",

package/wedge-capture-20260706.csv

@@ -0,0 +1,19 @@
+jid,reserved_at,retry_attempt,created_at,message
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm3,2026-07-06 20:19:07.953447+00,0,2026-07-06 20:18:07.945876+00,"{""metadata"":{""guid"":""HxfFFVmLVwtk37194ysP1qQ"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm3"",""gid"":""HAXgFtL8ZicUsOywHCKUauE"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""virtual-farm-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""printerBatch"",""arguments"":[{""machineId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm3"",""operatorId"":""0dfa0000-0000-4000-8000-000000000004"",""printSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""runsSoFar"":6,""plan"":[],""advertId"":""0a4af7d1-2dc4-4e06-9d43-791acf3c49e5""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm3-$printerBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm3-$printerBatch$1-5"",""workflowTopic"":""virtual-farm-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm4,2026-07-06 20:19:10.889217+00,0,2026-07-06 20:18:10.880925+00,"{""metadata"":{""guid"":""HXZnTUGkTix4uF7q14PuP0f"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm4"",""gid"":""HO9SAZEoZGiycaugLuwuUXv"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""virtual-farm-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""printerBatch"",""arguments"":[{""machineId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm4"",""operatorId"":""0dfa0000-0000-4000-8000-000000000005"",""printSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""runsSoFar"":6,""plan"":[],""advertId"":""a1d6d90a-a2cf-4621-a74f-e0ed1013cbc6""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm4-$printerBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm4-$printerBatch$1-5"",""workflowTopic"":""virtual-farm-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm5,2026-07-06 20:19:16.082786+00,0,2026-07-06 20:18:16.072927+00,"{""metadata"":{""guid"":""HE7nFSVKoh2oFe6-lYtCJWV"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm5"",""gid"":""H697gK8YvdS-_waP4NLiL-L"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""virtual-farm-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""printerBatch"",""arguments"":[{""machineId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm5"",""operatorId"":""0dfa0000-0000-4000-8000-000000000006"",""printSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""runsSoFar"":4,""plan"":[],""advertId"":""df441cd6-1811-41f0-b646-3c794ebf71d4""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm5-$printerBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm5-$printerBatch$1-5"",""workflowTopic"":""virtual-farm-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm1,2026-07-06 20:19:05.227375+00,0,2026-07-06 20:18:20.216145+00,"{""metadata"":{""guid"":""HTFIOwV5q0HYi8z04GwX9qj"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm1"",""gid"":""HSR3wD1ryGQ6f5vzzGcO8eV"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""virtual-farm-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""printerBatch"",""arguments"":[{""machineId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm1"",""operatorId"":""0dfa0000-0000-4000-8000-000000000002"",""printSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""runsSoFar"":13,""plan"":[],""advertId"":""e8ee662b-6de6-4a57-b8d6-ee9ed76e1f99""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm1-$printerBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm1-$printerBatch$1-5"",""workflowTopic"":""virtual-farm-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm0,2026-07-06 20:19:09.068044+00,0,2026-07-06 20:18:39.059327+00,"{""metadata"":{""guid"":""HcFhO_I7Fqn_5ftBd4uUTuZ"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm0"",""gid"":""HyVUxkj0OkvEO2r0sdgtTan"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""virtual-farm-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""printerBatch"",""arguments"":[{""machineId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm0"",""operatorId"":""0dfa0000-0000-4000-8000-000000000001"",""printSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""runsSoFar"":12,""plan"":[],""advertId"":""53bd1b3a-8ec3-4c35-a839-da93d9e23bc0""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm0-$printerBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm0-$printerBatch$1-5"",""workflowTopic"":""virtual-farm-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm2,2026-07-06 20:19:18.195319+00,0,2026-07-06 20:19:18.191286+00,"{""metadata"":{""guid"":""HWpXdbtZ-Bx5iqINxvAa6wf"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm2"",""gid"":""HWzNcP3ToTg27Qc5u5MljTT"",""dad"":"",0,16,0,0"",""aid"":""proxyer"",""topic"":""virtual-farm-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""printerBatch"",""arguments"":[{""machineId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm2"",""operatorId"":""0dfa0000-0000-4000-8000-000000000003"",""printSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""runsSoFar"":8,""plan"":[],""advertId"":""a0a84553-5031-456d-ae02-5d9746b606fc""}],""workflowDimension"":""$1"",""index"":6,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm2-$printerBatch$1-6"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-farm-vm2-$printerBatch$1-6"",""workflowTopic"":""virtual-farm-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-manager,2026-07-06 20:19:14.76557+00,0,2026-07-06 20:18:29.759824+00,"{""metadata"":{""guid"":""HzkL-DZTtXyIyDV8oZNZJxI"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-manager"",""gid"":""HCGavoqirvD8FwdwQOWgcov"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""managerBatch"",""arguments"":[{""laneId"":""9a93b820-dde3-425b-a320-260b91c08ec2"",""reviewSeconds"":3,""moveSeconds"":3,""idleTickSeconds"":3,""maxIterations"":25,""actorEmail"":""sim.manager.marco@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-manager-$managerBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-manager-$managerBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_quality_control-0,2026-07-06 20:19:12.094467+00,0,2026-07-06 20:18:42.089308+00,"{""metadata"":{""guid"":""H2MwGXqBeP8-8nPX-T6ndnU"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_quality_control-0"",""gid"":""H7nf5o8VvELz6NHWGWGNsXe"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""NEEDS_QUALITY_CONTROL"",""role"":""quality-inspector"",""toStatus"":""NEEDS_SHIPPING"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":7,""unparkedRejects"":[],""troubledOrders"":[""931a1e99-2db6-4280-b034-57e3c42a9aba""],""actorEmail"":""sim.inspector.ines@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_quality_control-0-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_quality_control-0-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_quality_control-1,2026-07-06 20:19:12.096894+00,0,2026-07-06 20:18:42.093836+00,"{""metadata"":{""guid"":""H4pQ4lScxaW8wUEiP8BQh8E"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_quality_control-1"",""gid"":""HQUqSMYqzsvODyjMvRNPtvl"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""NEEDS_QUALITY_CONTROL"",""role"":""quality-inspector"",""toStatus"":""NEEDS_SHIPPING"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":7,""unparkedRejects"":[],""troubledOrders"":[""931a1e99-2db6-4280-b034-57e3c42a9aba""],""actorEmail"":""sim.inspector.tomas@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_quality_control-1-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_quality_control-1-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-draft,2026-07-06 20:19:16.150935+00,0,2026-07-06 20:18:46.146153+00,"{""metadata"":{""guid"":""HgtJMxOb1oYv6jJb2uErF0_"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-draft"",""gid"":""HN6b23imRu-wgDHA-DpxzLV"",""dad"":"",0,14,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""DRAFT"",""role"":""insole-designer"",""toStatus"":""NEEDS_MANUFACTURING"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":24,""unparkedRejects"":[],""troubledOrders"":[""2b294913-f8ca-4639-9928-3abd53ea9616""],""actorEmail"":""sim.designer.dana@sim.hike.local"",""worklistStation"":""design-approval""}],""workflowDimension"":""$1"",""index"":4,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-draft-$associateBatch$1-4"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-draft-$associateBatch$1-4"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_finishing-1,2026-07-06 20:19:04.938838+00,0,2026-07-06 20:18:49.932504+00,"{""metadata"":{""guid"":""HQD-nuRyxBmH431uJ8noMBS"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_finishing-1"",""gid"":""HWM3gVxjxz9_K-EnQpCeFjY"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""NEEDS_FINISHING"",""role"":""finisher"",""toStatus"":""NEEDS_QUALITY_CONTROL"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":9,""unparkedRejects"":[],""troubledOrders"":[],""actorEmail"":""sim.finisher.kofi@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_finishing-1-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_finishing-1-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_finishing-0,2026-07-06 20:19:14.016307+00,0,2026-07-06 20:18:59.008759+00,"{""metadata"":{""guid"":""HHBHdHpX2Wear_cVJim70Ps"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_finishing-0"",""gid"":""HkZyt5-JhLoCNX26Q4cyqbq"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""NEEDS_FINISHING"",""role"":""finisher"",""toStatus"":""NEEDS_QUALITY_CONTROL"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":11,""unparkedRejects"":[],""troubledOrders"":[],""actorEmail"":""sim.finisher.ada@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_finishing-0-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_finishing-0-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_shipping,2026-07-06 20:19:15.654293+00,0,2026-07-06 20:19:00.646061+00,"{""metadata"":{""guid"":""HffnP49JTwmHAwk3BNRn22l"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_shipping"",""gid"":""HuK-iSfDdYXXBJWo2WwhCDF"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""NEEDS_SHIPPING"",""role"":""shipping-associate"",""toStatus"":""COMPLETED"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":12,""unparkedRejects"":[],""troubledOrders"":[],""actorEmail"":""sim.shipper.june@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_shipping-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_shipping-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_gluing-1,2026-07-06 20:19:05.381804+00,0,2026-07-06 20:19:05.3792+00,"{""metadata"":{""guid"":""Hx82JQ7g0Y4XT7nqPFK4km9"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_gluing-1"",""gid"":""HoT3fZZg04_z7BKimsTs-Ei"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""NEEDS_GLUING"",""role"":""gluer"",""toStatus"":""NEEDS_FINISHING"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":13,""unparkedRejects"":[],""troubledOrders"":[],""actorEmail"":""sim.gluer.dev@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_gluing-1-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_gluing-1-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_gluing-0,2026-07-06 20:19:05.386589+00,0,2026-07-06 20:19:05.384446+00,"{""metadata"":{""guid"":""HvNFncRHNbqR_OSOOiQ0M12"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_gluing-0"",""gid"":""Hk7Wy8HO98wh45Clgp5-sUo"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""NEEDS_GLUING"",""role"":""gluer"",""toStatus"":""NEEDS_FINISHING"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":13,""unparkedRejects"":[],""troubledOrders"":[],""actorEmail"":""sim.gluer.rosa@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_gluing-0-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-needs_gluing-0-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-post_print_qa-1,2026-07-06 20:19:06.832852+00,0,2026-07-06 20:19:06.829652+00,"{""metadata"":{""guid"":""Hg1xKYket-ULnS-HdHaNuiD"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-post_print_qa-1"",""gid"":""HLjxCrhPCUyqBaWbwSE0pxp"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""POST_PRINT_QA"",""role"":""post-print-qa"",""toStatus"":""NEEDS_GLUING"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":14,""rejectRate"":{""pct"":15,""targetStation"":""PRINTING"",""reasons"":[""Layer separation on the arch — reprint"",""Surface finish below spec"",""Edge delamination at the heel cup"",""Warping on the medial side""],""leftQuantity"":1,""rightQuantity"":0},""unparkedRejects"":[],""troubledOrders"":[],""actorEmail"":""sim.qa.elena@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-post_print_qa-1-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-post_print_qa-1-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-post_print_qa-0,2026-07-06 20:19:12.746444+00,0,2026-07-06 20:19:12.744474+00,"{""metadata"":{""guid"":""HTl4tkkOMhgso2QMlQlKYvn"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-post_print_qa-0"",""gid"":""HY9mpKnlDbC3a1o3B4R-53h"",""dad"":"",0,15,0,0"",""aid"":""proxyer"",""topic"":""actor-crew-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""associateBatch"",""arguments"":[{""station"":""POST_PRINT_QA"",""role"":""post-print-qa"",""toStatus"":""NEEDS_GLUING"",""dwellSeconds"":4,""betweenSeconds"":2,""idleTickSeconds"":3,""maxIterations"":25,""touchesSoFar"":16,""rejectRate"":{""pct"":15,""targetStation"":""PRINTING"",""reasons"":[""Layer separation on the arch — reprint"",""Surface finish below spec"",""Edge delamination at the heel cup"",""Warping on the medial side""],""leftQuantity"":1,""rightQuantity"":0},""unparkedRejects"":[],""troubledOrders"":[],""actorEmail"":""sim.qa.marcus@sim.hike.local""}],""workflowDimension"":""$1"",""index"":5,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-post_print_qa-0-$associateBatch$1-5"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-crew-post_print_qa-0-$associateBatch$1-5"",""workflowTopic"":""actor-crew-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"
+simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-design,2026-07-06 20:19:10.594054+00,0,2026-07-06 20:18:55.585455+00,"{""metadata"":{""guid"":""HeSdgdXIms_1Lhh2nNgc4EP"",""jid"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-design"",""gid"":""HU01R82t5rnplNLKWryLJPz"",""dad"":"",0,40,0,0"",""aid"":""proxyer"",""topic"":""simulation-activity"",""wfn"":""durable.execute""},""data"":{""activityName"":""designTwinBatch"",""arguments"":[{""user"":""sim.taika.iris@sim.hike.local"",""designSeconds"":2,""idleTickSeconds"":3,""maxIterations"":10,""concurrency"":4,""designedOrders"":[""0f596ab9-f449-485d-b06d-e74864750ae7"",""a5cedbc2-f726-4411-9ac3-10fd113532d2"",""c1a09bd7-bd4b-4eb1-bd3e-91d6b45b479f"",""94d74e0e-e4ce-4db7-91f3-02c1c081c2ba"",""0ed99be7-83b6-4a91-9aa5-560d5c110c6b"",""33b987bb-9253-483f-baa4-37d4ffd0903c"",""2c4bc240-1335-423d-96af-e5855b6ccc0f"",""0e7f24bb-3fba-40e5-a5a8-666f5c91fd49"",""ad0c05b6-4528-4f2a-9569-d06c76589b54"",""de2a02e2-33ca-4c0c-843c-5ad422c7c517"",""c008abd0-8b3b-44f7-9f1c-7d314f72f587"",""04a28a34-20f3-4d10-a636-3cb8f504f56e"",""52295fa8-452b-48ca-862b-64cf3ff8cae6"",""cb1dd956-639d-4ff8-9f56-c12f9c5278e8"",""fca4a3f0-fc49-47cd-8285-bf7e11c60dbe"",""bcd42ec0-f047-4acc-8e70-33575ae7535a"",""3b73fb95-2366-4498-ac9a-3f2d9eb4d1c0"",""3c9d174f-7928-42e7-9edb-2f3b38ba0ddc"",""1e61d445-77ba-4013-af89-1a6de07ff407"",""0b066160-06ab-4681-af20-8c36866c9987"",""fa9f996c-2327-4f30-8fdc-e63239483f23"",""ff31ec18-c7e3-4907-85f0-5c4eda42841f"",""f17a825c-8905-4166-b687-54015ddeae6c"",""931a1e99-2db6-4280-b034-57e3c42a9aba"",""2b294913-f8ca-4639-9928-3abd53ea9616"",""04caa323-662a-4c28-8de0-3655d435f206"",""b2eac688-d074-45d9-8a60-e87719b18c81"",""a11092fe-31e5-4cfe-815c-230a09f2165e"",""176e3d03-0813-4816-83fd-a785798a83bb"",""e1d887bd-1ad4-4436-b728-d75e7df2c39a"",""c8a0ffdc-91b0-443a-a8f9-c01c2e3935c0"",""83cbe6eb-ea9b-492e-8657-4f138e3df05e"",""e8e56685-82a3-4f88-a804-b6a8c21f0a4c"",""89d231bf-600b-40ed-8161-e58dbcc7157c"",""6904b329-7a63-465f-aa6a-82a1095638fb"",""9ca2abe9-ae5f-459a-a003-9ea4ad5cafc0"",""cae10b57-439a-4e98-b0d0-1bb172a9d79f"",""f90af7cb-401b-4e68-9f22-a45d34581d64"",""86a1d1b2-7287-48ab-94b3-544a743fda41"",""76793c77-67a0-4148-b83b-fa50f900111c""]}],""workflowDimension"":""$3"",""index"":8,""originJobId"":""simulationDirector-H5zQndzkcfIGHhUM-QXFzcG"",""parentWorkflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-design-$designTwinBatch$3-8"",""workflowId"":""-simulationDirector-H5zQndzkcfIGHhUM-QXFzcG-design-$designTwinBatch$3-8"",""workflowTopic"":""simulation-activity"",""expire"":2592000,""backoffCoefficient"":5,""initialInterval"":1,""maximumAttempts"":3,""maximumInterval"":120}}"