@hotmeshio/hotmesh 0.25.4 → 0.25.5

package/ISSUE.md

@@ -0,0 +1,53 @@
+# ADDENDUM — lost condition wakes are NOT restart-dependent (registration-window race)
+
+Follow-up to the earlier "Durable wakes lost across process restart" report filed
+tonight (sleep timers + condition signals). New evidence sharpens the condition case.
+
+## Reproduced with no restart involved
+
+Same environment (hotmesh 0.25.4, Postgres, long-tail 0.7.3), process running
+continuously the whole time:
+
+- `13:59:36.204` — pill escalation created (conditionLT atomic park, child
+  `...-winddown-gluer-0`).
+- `13:59:36.237` — row **resolved**, 33 ms after creation (a hot consumer was
+  already scanning the pond and grabbed it instantly).
+- 15+ minutes later — the awaiting child is still `status > 0`, never woke;
+  its parent's `Promise.all` wedged with it.
+
+## The pattern across all three observed condition cases
+
+Every lost condition wake we have seen shares one property: **the resolve landed
+sub-second after the conditionLT park** (33 ms, 232 ms, ~600 ms). Slow resolves
+(seconds or minutes after park) have never lost a wake across hundreds of
+station-worker resolutions tonight.
+
+Hypothesis: a resolve that commits **inside the awaiting workflow's
+wake-registration window** — after the escalation row is visible to consumers
+but before the awaiter's subscription is fully registered — delivers to nobody,
+and nothing ever re-checks the persisted row against sleeping awaiters. The
+restart in the earlier case was likely incidental; the window is the bug.
+
+## Suggested repro (tighter than the original)
+
+1. Workflow A parks via `conditionLT` (the escalation-creating overload).
+2. A hot loop polls for the row and resolves it the instant it appears
+   (sub-100 ms). No process kill needed.
+3. Observe whether A ever wakes. In our runs, a same-process consumer beating
+   the registration window wedges A reliably enough to hit twice in one evening.
+
+## Suggested direction (unchanged, reinforced)
+
+Boot-time AND periodic reconciliation of resolved-rows-with-sleeping-awaiters
+would heal both the restart loss and this race — the row carries the awaiting
+`workflow_id`; the join is cheap. Making the wake itself a stream message
+(liveness-redelivered since 0.25.4) also closes the window, since the message
+persists until reserved and acked.
+
+## Our workaround (in production use as of tonight)
+
+Settlement guard in the watcher activity: after the row's outcome is known,
+poll `durable.jobs` for the awaiting child's status; if still running ~10s
+after its row resolved, terminate the child by id (terminate purges cleanly
+since 0.25.4) and treat the row's persisted resolver payload as authoritative.
+The parent catches the child's terminate rejection. Ugly but wedge-proof.

package/build/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@hotmeshio/hotmesh",
-    "version": "0.25.4",
+    "version": "0.25.5",
     "description": "Durable Workflow",
     "main": "./build/index.js",
     "types": "./build/index.d.ts",

package/build/services/escalations/client.d.ts

@@ -72,6 +72,16 @@ export declare class EscalationClientService {
     private _makeEngineFactory;
     private _hashConnection;
     private _deliverEscalationSignal;
+    /**
+     * Builds the wake publish as a SQL command so the store can commit it
+     * INSIDE the resolve transaction — the wake becomes durable with the
+     * resolved row, closing the crash window between resolve commit and
+     * post-commit signal delivery. Mirrors `_deliverEscalationSignal`'s
+     * topic fallback chain; returns null when no hook rule is deployed
+     * for any candidate topic (the caller then keeps post-commit
+     * delivery as the only path, preserving prior behavior).
+     */
+    private _buildWakeCommand;
     /**
      * Returns all escalation rows matching the given filters. Each row includes
      * a computed `available` field (true = claimable). Supports `sortBy`,

package/build/services/escalations/client.js

@@ -3,8 +3,10 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.EscalationClientService = void 0;
 const enums_1 = require("../../modules/enums");
 const utils_1 = require("../../modules/utils");
+const key_1 = require("../../modules/key");
 const hotmesh_1 = require("../hotmesh");
 const types_1 = require("../../types");
+const stream_1 = require("../../types/stream");
 const factory_1 = require("../durable/schemas/factory");
 /**
  * Standalone client for the `public.hmsh_escalations` signal-pause surface.
@@ -145,6 +147,46 @@ class EscalationClientService {
         catch { }
         return delivered;
     }
+    /**
+     * Builds the wake publish as a SQL command so the store can commit it
+     * INSIDE the resolve transaction — the wake becomes durable with the
+     * resolved row, closing the crash window between resolve commit and
+     * post-commit signal delivery. Mirrors `_deliverEscalationSignal`'s
+     * topic fallback chain; returns null when no hook rule is deployed
+     * for any candidate topic (the caller then keeps post-commit
+     * delivery as the only path, preserving prior behavior).
+     */
+    async _buildWakeCommand(ns, topic, signalKey, data) {
+        const hm = await this._engine(null, ns);
+        const engine = hm.engine;
+        const candidates = [topic, `${ns}.wfs.signal`, `${ns}.wfs.wait`].filter(Boolean);
+        for (const candidate of candidates) {
+            try {
+                const hookRule = await engine.taskService.getHookRule(candidate);
+                if (!hookRule)
+                    continue;
+                const [aid] = await engine.getSchema(`.${hookRule.to}`);
+                const streamData = {
+                    type: stream_1.StreamDataType.WEBHOOK,
+                    status: types_1.StreamStatus.SUCCESS,
+                    code: 200,
+                    metadata: { guid: (0, utils_1.guid)(), aid, topic: candidate },
+                    data: { id: signalKey, data },
+                };
+                const streamKey = engine.stream.mintKey(key_1.KeyType.STREAMS, {
+                    topic: null,
+                });
+                const { sql, params } = engine.stream._publishMessages(streamKey, [
+                    JSON.stringify(streamData),
+                ]);
+                return { forSignalKey: signalKey, sql, params };
+            }
+            catch {
+                /* candidate not deployed — try the next topic */
+            }
+        }
+        return null;
+    }
     // ─── Public API ─────────────────────────────────────────────────────────────
     /**
      * Returns all escalation rows matching the given filters. Each row includes
@@ -292,10 +334,19 @@ class EscalationClientService {
         const ns = (params.namespace ?? namespace) ?? factory_1.APP_ID;
         const hm = await this._engine(null, ns);
         const store = hm.engine.store;
-        const dbResult = await store.resolveEscalation({ id: params.id, resolverPayload: params.resolverPayload, metadata: params.metadata });
+        //pre-build the wake so it commits INSIDE the resolve transaction; the
+        //row's signal routing (signal_key, topic) is immutable after creation
+        let wakeCommand = null;
+        const preview = await store.getEscalation(params.id, params.namespace);
+        if (preview?.signal_key) {
+            wakeCommand = await this._buildWakeCommand(ns, preview.topic, preview.signal_key, params.resolverPayload ?? {});
+        }
+        const dbResult = await store.resolveEscalation({ id: params.id, resolverPayload: params.resolverPayload, metadata: params.metadata }, wakeCommand ?? undefined);
         if (!dbResult.ok)
             return dbResult;
-        if (dbResult.signalKey) {
+        if (dbResult.signalKey && !dbResult.wakeEnqueued) {
+            //the wake was not part of the commit (no hook rule found, or the
+            //enqueue was rolled back to its savepoint) — deliver post-commit
             await this._deliverEscalationSignal(ns, dbResult.topic, {
                 id: dbResult.signalKey,
                 data: params.resolverPayload ?? {},
@@ -316,10 +367,22 @@ class EscalationClientService {
         const ns = (params.namespace ?? namespace) ?? factory_1.APP_ID;
         const hm = await this._engine(null, ns);
         const store = hm.engine.store;
-        const dbResult = await store.resolveEscalationByMetadata({ key: params.key, value: params.value, resolverPayload: params.resolverPayload, roles: params.roles, metadata: params.metadata });
+        //peek the row the resolve is expected to lock and pre-build its wake;
+        //the forSignalKey guard covers the race where a different row wins
+        let wakeCommand = null;
+        const preview = await store.peekEscalationByMetadata({
+            key: params.key,
+            value: params.value,
+            roles: params.roles,
+            namespace: params.namespace,
+        });
+        if (preview?.signalKey) {
+            wakeCommand = await this._buildWakeCommand(ns, preview.topic, preview.signalKey, params.resolverPayload ?? {});
+        }
+        const dbResult = await store.resolveEscalationByMetadata({ key: params.key, value: params.value, resolverPayload: params.resolverPayload, roles: params.roles, metadata: params.metadata }, wakeCommand ?? undefined);
         if (!dbResult.ok)
             return dbResult;
-        if (dbResult.signalKey) {
+        if (dbResult.signalKey && !dbResult.wakeEnqueued) {
             await this._deliverEscalationSignal(ns, dbResult.topic, {
                 id: dbResult.signalKey,
                 data: params.resolverPayload ?? {},

package/build/services/store/index.d.ts

@@ -68,8 +68,17 @@ declare abstract class StoreService<Provider extends ProviderClient, Transaction
      * Leg1: Attempts to set the hook signal. If a pending signal occupies
      * the key (race condition), overwrites it and returns the pending data.
      * When called with a transaction, queues the setnxex (no pending detection).
+     *
+     * When `redelivery` is provided (the webhook routing for this hook),
+     * a consumed pending signal is republished as an engine stream message
+     * in the SAME transaction that overwrites the marker — the wake
+     * survives a crash at any instant. `pendingData` is then not returned,
+     * since the store already owns the redelivery.
      */
-    abstract setHookSignal(hook: HookSignal, transaction?: TransactionProvider): Promise<{
+    abstract setHookSignal(hook: HookSignal, transaction?: TransactionProvider, redelivery?: {
+        aid: string;
+        topic: string;
+    }): Promise<{
         success: boolean;
         pendingData?: string;
     }>;

package/build/services/store/providers/postgres/postgres.d.ts

@@ -128,7 +128,10 @@ declare class PostgresStoreService extends StoreService<ProviderClient, Provider
      *
      * In a transaction: queues the setnxex; pending detection deferred.
      */
-    setHookSignal(hook: HookSignal, transaction?: ProviderTransaction): Promise<{
+    setHookSignal(hook: HookSignal, transaction?: ProviderTransaction, redelivery?: {
+        aid: string;
+        topic: string;
+    }): Promise<{
         success: boolean;
         pendingData?: string;
     }>;
@@ -270,13 +273,34 @@ declare class PostgresStoreService extends StoreService<ProviderClient, Provider
     claimEscalation(params: import('../../../../types/hmsh_escalations').ClaimEscalationParams): Promise<import('../../../../types/hmsh_escalations').ClaimEscalationResult>;
     claimEscalationByMetadata(params: import('../../../../types/hmsh_escalations').ClaimByMetadataParams): Promise<import('../../../../types/hmsh_escalations').ClaimByMetadataResult>;
     releaseEscalation(params: import('../../../../types/hmsh_escalations').ReleaseEscalationParams): Promise<import('../../../../types/hmsh_escalations').ReleaseEscalationResult>;
-    resolveEscalation(params: import('../../../../types/hmsh_escalations').ResolveEscalationParams): Promise<import('../../../../types/hmsh_escalations').ResolveEscalationResult & {
+    /**
+     * Executes a pre-built wake publish inside the currently open resolve
+     * transaction, guarded by a SAVEPOINT so a wake failure can never roll
+     * back the resolve itself. Returns true when the wake row committed
+     * with the transaction; false means the caller should fall back to
+     * post-commit delivery.
+     */
+    private enqueueEscalationWake;
+    resolveEscalation(params: import('../../../../types/hmsh_escalations').ResolveEscalationParams, wakeCommand?: import('../../../../types/hmsh_escalations').EscalationWakeCommand): Promise<import('../../../../types/hmsh_escalations').ResolveEscalationResult & {
         signalKey?: string | null;
         topic?: string | null;
+        wakeEnqueued?: boolean;
     }>;
-    resolveEscalationByMetadata(params: import('../../../../types/hmsh_escalations').ResolveByMetadataParams): Promise<import('../../../../types/hmsh_escalations').ResolveEscalationResult & {
+    /**
+     * Non-locking preview of the row `resolveEscalationByMetadata` would
+     * select — used to pre-build the wake command before the resolve
+     * transaction opens. The `forSignalKey` guard on the wake command
+     * handles the race where a different row wins the lock.
+     */
+    peekEscalationByMetadata(params: import('../../../../types/hmsh_escalations').ResolveByMetadataParams): Promise<{
+        id: string;
+        signalKey: string | null;
+        topic: string | null;
+    } | null>;
+    resolveEscalationByMetadata(params: import('../../../../types/hmsh_escalations').ResolveByMetadataParams, wakeCommand?: import('../../../../types/hmsh_escalations').EscalationWakeCommand): Promise<import('../../../../types/hmsh_escalations').ResolveEscalationResult & {
         signalKey?: string | null;
         topic?: string | null;
+        wakeEnqueued?: boolean;
     }>;
     cancelEscalation(id: string, namespace?: string): Promise<import('../../../../types/hmsh_escalations').CancelEscalationResult>;
     escalateEscalationToRole(params: import('../../../../types/hmsh_escalations').EscalateToRoleParams): Promise<import('../../../../types/hmsh_escalations').EscalationEntry | null>;

package/build/services/store/providers/postgres/postgres.js

@@ -777,7 +777,7 @@ class PostgresStoreService extends __1.StoreService {
      *
      * In a transaction: queues the setnxex; pending detection deferred.
      */
-    async setHookSignal(hook, transaction) {
+    async setHookSignal(hook, transaction, redelivery) {
         const key = this.mintKey(key_1.KeyType.SIGNALS, { appId: this.appId });
         const { topic, resolved, jobId } = hook;
         const signalKey = `${topic}:${resolved}`;
@@ -813,6 +813,45 @@ class PostgresStoreService extends __1.StoreService {
             const captured = lockResult.rows[0]?.value;
             const wasInsert = lockResult.rows[0]?.inserted;
             const isPending = captured?.startsWith('$pending::');
+            if (isPending && redelivery) {
+                //consume the marker and commit its redelivery as ONE unit: the
+                //pending wake becomes a durable engine stream message in the
+                //same transaction that destroys the marker, so the wake
+                //survives a crash at any instant. A crash before COMMIT leaves
+                //the marker intact for the resume path to consume again.
+                const pendingData = captured.slice('$pending::'.length);
+                const message = JSON.stringify({
+                    type: 'webhook',
+                    status: 'success',
+                    code: 200,
+                    metadata: {
+                        guid: (0, utils_1.guid)(),
+                        aid: redelivery.aid,
+                        topic: redelivery.topic,
+                    },
+                    data: JSON.parse(pendingData),
+                });
+                const schemaName = this.kvsql().safeName(this.appId);
+                await this.pgClient.query('BEGIN');
+                try {
+                    await this.pgClient.query(`UPDATE ${tableName}
+             SET value = $1, expiry = NOW() + INTERVAL '${delay} seconds'
+             WHERE key = $2`, [jobId, storedKey]);
+                    await this.pgClient.query(`INSERT INTO ${schemaName}.engine_streams
+             (stream_name, message, priority)
+             VALUES ($1, $2, 5)`, [this.appId, message]);
+                    await this.pgClient.query('COMMIT');
+                }
+                catch (redeliveryError) {
+                    await this.pgClient.query('ROLLBACK');
+                    throw redeliveryError;
+                }
+                this.logger.warn('hook-signal-pending-redelivered', {
+                    key: signalKey,
+                    topic: redelivery.topic,
+                });
+                return { success: true };
+            }
             if (!wasInsert) {
                 //step 2: row existed — overwrite with hook value
                 await this.pgClient.query(`UPDATE ${tableName}
@@ -1792,7 +1831,32 @@ class PostgresStoreService extends __1.StoreService {
             return { ok: false, reason: 'wrong-assignee' };
         return { ok: true, entry: row.entry_json };
     }
-    async resolveEscalation(params) {
+    /**
+     * Executes a pre-built wake publish inside the currently open resolve
+     * transaction, guarded by a SAVEPOINT so a wake failure can never roll
+     * back the resolve itself. Returns true when the wake row committed
+     * with the transaction; false means the caller should fall back to
+     * post-commit delivery.
+     */
+    async enqueueEscalationWake(wakeCommand, signalKey, escalationId) {
+        if (!wakeCommand || !signalKey || wakeCommand.forSignalKey !== signalKey) {
+            return false;
+        }
+        await this.pgClient.query('SAVEPOINT escalation_wake');
+        try {
+            await this.pgClient.query(wakeCommand.sql, wakeCommand.params);
+            return true;
+        }
+        catch (error) {
+            await this.pgClient.query('ROLLBACK TO SAVEPOINT escalation_wake');
+            this.logger.warn('escalation-wake-enqueue-error', {
+                escalationId,
+                error: error.message,
+            });
+            return false;
+        }
+    }
+    async resolveEscalation(params, wakeCommand) {
         const { id, namespace, resolverPayload, metadata } = params;
         const payloadJson = resolverPayload ? JSON.stringify(resolverPayload) : null;
         // metaJson is bound at a fixed index; the CASE no-ops when null so resolution
@@ -1835,15 +1899,41 @@ class PostgresStoreService extends __1.StoreService {
                 await this.pgClient.query('ROLLBACK');
                 return { ok: false, reason: 'already-resolved' };
             }
+            //the wake commits WITH the resolve: a crash after COMMIT leaves both
+            //the resolved row and its wake message durable; a crash before
+            //leaves neither. The engine_streams INSERT trigger emits the
+            //delivery NOTIFY on commit.
+            const wakeEnqueued = await this.enqueueEscalationWake(wakeCommand, signal_key, id);
             await this.pgClient.query('COMMIT');
-            return { ok: true, entry: updateResult.rows[0], signalKey: signal_key, topic };
+            return { ok: true, entry: updateResult.rows[0], signalKey: signal_key, topic, wakeEnqueued };
         }
         catch (e) {
             await this.pgClient.query('ROLLBACK');
             throw e;
         }
     }
-    async resolveEscalationByMetadata(params) {
+    /**
+     * Non-locking preview of the row `resolveEscalationByMetadata` would
+     * select — used to pre-build the wake command before the resolve
+     * transaction opens. The `forSignalKey` guard on the wake command
+     * handles the race where a different row wins the lock.
+     */
+    async peekEscalationByMetadata(params) {
+        const { key, value, namespace, roles } = params;
+        const filter = JSON.stringify({ [key]: value });
+        const result = await this.pgClient.query(`SELECT id, signal_key, topic FROM public.hmsh_escalations
+       WHERE ${namespace ? 'namespace = $3 AND' : ''}
+             metadata @> $1::jsonb
+         AND ($2::text[] IS NULL OR role = ANY($2::text[]))
+         AND status IN ('pending', 'cancelled')
+       ORDER BY priority ASC, created_at ASC
+       LIMIT 1`, namespace ? [filter, roles ?? null, namespace] : [filter, roles ?? null]);
+        const row = result.rows[0];
+        if (!row)
+            return null;
+        return { id: row.id, signalKey: row.signal_key, topic: row.topic };
+    }
+    async resolveEscalationByMetadata(params, wakeCommand) {
         const { key, value, namespace, resolverPayload, roles, metadata } = params;
         const filter = JSON.stringify({ [key]: value });
         const payloadJson = resolverPayload ? JSON.stringify(resolverPayload) : null;
@@ -1878,8 +1968,9 @@ class PostgresStoreService extends __1.StoreService {
                 await this.pgClient.query('ROLLBACK');
                 return { ok: false, reason: 'already-resolved' };
             }
+            const wakeEnqueued = await this.enqueueEscalationWake(wakeCommand, signal_key, id);
             await this.pgClient.query('COMMIT');
-            return { ok: true, entry: updateResult.rows[0], signalKey: signal_key, topic };
+            return { ok: true, entry: updateResult.rows[0], signalKey: signal_key, topic, wakeEnqueued };
         }
         catch (e) {
             await this.pgClient.query('ROLLBACK');

package/build/services/task/index.js

@@ -73,8 +73,13 @@ class TaskService {
                 expire,
             };
             //called standalone (no transaction) so the single CTE query can
-            //atomically detect and return pending signal data on collision
-            const result = await this.store.setHookSignal(hook);
+            //atomically detect and return pending signal data on collision.
+            //the redelivery routing lets the store republish a consumed
+            //pending signal durably, in the same transaction that consumes it
+            const result = await this.store.setHookSignal(hook, undefined, {
+                aid: hookRule.to,
+                topic,
+            });
             if (result.pendingData) {
                 this.logger.warn('task-signal-race-pending-consumed', {
                     topic,

package/build/types/hmsh_escalations.d.ts

@@ -105,6 +105,19 @@ export type ResolveEscalationResult = {
     ok: false;
     reason: 'not-found' | 'already-resolved' | 'already-cancelled' | 'already-expired';
 };
+/**
+ * A pre-built wake publish, executed INSIDE the resolve transaction so the
+ * awaiting workflow's wake commits atomically with `status='resolved'`. The
+ * SQL is a stream INSERT produced by the stream provider; `forSignalKey`
+ * pins the command to the row it was built for — the store executes it only
+ * when the locked row's `signal_key` matches (a mismatched row falls back
+ * to post-commit delivery).
+ */
+export interface EscalationWakeCommand {
+    forSignalKey: string;
+    sql: string;
+    params: unknown[];
+}
 export type ReleaseEscalationResult = {
     ok: true;
     entry: EscalationEntry;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hotmeshio/hotmesh",
-  "version": "0.25.4",
+  "version": "0.25.5",
   "description": "Durable Workflow",
   "main": "./build/index.js",
   "types": "./build/index.d.ts",