@hotmeshio/hotmesh 0.25.5 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/OPTIMIZATIONS.md +128 -0
- package/RELEASE-0.26.0.md +111 -0
- package/build/package.json +1 -1
- package/build/services/activities/hook.js +22 -0
- package/build/services/durable/workflow/condition.d.ts +20 -3
- package/build/services/durable/workflow/condition.js +20 -3
- package/build/services/durable/workflow/proxyActivities.d.ts +14 -0
- package/build/services/durable/workflow/proxyActivities.js +14 -0
- package/build/services/escalations/client.d.ts +26 -8
- package/build/services/escalations/client.js +43 -17
- package/build/services/store/providers/postgres/postgres.d.ts +35 -7
- package/build/services/store/providers/postgres/postgres.js +244 -142
- package/build/services/stream/providers/postgres/kvtables.js +76 -14
- package/build/services/stream/providers/postgres/messages.js +6 -3
- package/build/services/stream/providers/postgres/procedures.js +6 -4
- package/build/types/hmsh_escalations.d.ts +30 -8
- package/package.json +1 -1
- package/ISSUE.md +0 -53
package/OPTIMIZATIONS.md
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# HotMesh needs — from long-tail
|
|
2
|
+
|
|
3
|
+
A needs document from the long-tail project: the usage patterns we run at
|
|
4
|
+
production scale and the engine support that would let us run them well. Each item
|
|
5
|
+
states what we do, the concrete access pattern, and the engine capability or
|
|
6
|
+
guarantee we need. Ordered by leverage.
|
|
7
|
+
|
|
8
|
+
Context: long-tail is a durable HITL escalation platform on Postgres. Escalations
|
|
9
|
+
live in `public.hmsh_escalations` (engine-created); we layer a `public.lt_escalations`
|
|
10
|
+
view and a few app-owned indexes on top. Our dominant runtime shape is
|
|
11
|
+
**single-role** work-queue traffic — claim / available / count / settle scoped to
|
|
12
|
+
one `role` — plus a per-role escalation form the resolve UI renders.
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## 1. Safe retention for terminal escalation rows
|
|
17
|
+
|
|
18
|
+
**Usage.** An escalation is written once, resolved once, then read for audit. The
|
|
19
|
+
pending working set stays small — a role-leading pending partial index keeps
|
|
20
|
+
single-role claim/available queries `O(pending)` regardless of table size:
|
|
21
|
+
|
|
22
|
+
```sql
|
|
23
|
+
WHERE role = $1 AND status = 'pending'
|
|
24
|
+
AND (assigned_to IS NULL OR assigned_until <= NOW())
|
|
25
|
+
AND metadata @> $facets
|
|
26
|
+
ORDER BY priority ASC, created_at ASC
|
|
27
|
+
FOR UPDATE SKIP LOCKED
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
The terminal set (resolved / expired / cancelled) grows for the life of the
|
|
31
|
+
deployment — the source of long-run table bloat, autovacuum load, and index growth.
|
|
32
|
+
|
|
33
|
+
**Need.** A safe, engine-blessed way to age out terminal rows. Tell us what state
|
|
34
|
+
the engine still reads once an escalation is terminal (replay, completion, audit),
|
|
35
|
+
so we know what is inert, then one of:
|
|
36
|
+
- a supported retention/archive call the engine owns; or
|
|
37
|
+
- a documented contract that a row is safe to delete once its status is terminal
|
|
38
|
+
and older than a horizon; or
|
|
39
|
+
- time sub-partitioning of the terminal set the engine can `DETACH`/`DROP`.
|
|
40
|
+
|
|
41
|
+
`hmsh_escalations` stays a single physical table keyed by `id`/`signal_key` for
|
|
42
|
+
the engine's lookups; retention touches only rows the engine confirms are inert. We
|
|
43
|
+
can supply the horizon and exact DDL.
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## 2. Stable ownership of app-added indexes on `hmsh_escalations`
|
|
48
|
+
|
|
49
|
+
**Usage.** We add indexes to the engine table via advisory-locked
|
|
50
|
+
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`:
|
|
51
|
+
- a role-leading pending index `(role, priority, created_at) WHERE status='pending'`
|
|
52
|
+
for the claim/available order;
|
|
53
|
+
- a **pending-scoped GIN** on `metadata` (`jsonb_path_ops`) `WHERE status='pending'`
|
|
54
|
+
— keeps the GIN sized to the working set and cuts write amplification on the
|
|
55
|
+
high-churn insert→resolve path;
|
|
56
|
+
- covering partial indexes for resolved-row reads.
|
|
57
|
+
|
|
58
|
+
**Need.** A stable contract that app-added indexes on `hmsh_escalations` survive
|
|
59
|
+
engine migrations — the engine keeps the table in place across upgrades — or a
|
|
60
|
+
documented post-migration hook we can register to re-ensure them. Shipping these
|
|
61
|
+
index shapes in the engine itself also works; the requirement is that they persist.
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## 3. Atomic metadata write in the `condition()` Leg1 checkpoint
|
|
66
|
+
|
|
67
|
+
**Usage.** A workflow surfaces a HITL wait with `condition(signalId, config)`. We
|
|
68
|
+
pass the target `role`, claim/routing facets, and `metadata.schema_version` (the
|
|
69
|
+
form version the resolve UI renders) in `config.metadata`. Claim/resolve-by-metadata
|
|
70
|
+
and the dashboard read those facets via GIN `@>`; the resolve UI reads
|
|
71
|
+
`schema_version` to render the exact form.
|
|
72
|
+
|
|
73
|
+
**Need.** Guarantee — as a documented public contract — that `config.metadata` is
|
|
74
|
+
written into the escalation row **atomically in the same Leg1 checkpoint** that
|
|
75
|
+
creates the row (one commit, crash-safe). This is the foundation our JIT-versioned
|
|
76
|
+
form pinning and metadata-driven routing stand on.
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## 4. Honored long-activity timeout + reclaim safety
|
|
81
|
+
|
|
82
|
+
**Usage.** We run a batch pattern: a single proxyActivity loops internally for
|
|
83
|
+
~60s (poll → reconcile → act), returning one durable checkpoint per call instead of
|
|
84
|
+
one per step, so replay stays compact.
|
|
85
|
+
|
|
86
|
+
**Need.** A documented guarantee that:
|
|
87
|
+
- `proxyActivities({ startToCloseTimeout })` is honored for long (30–120s) activities;
|
|
88
|
+
- the stream reclaim window (`HMSH_XCLAIM_DELAY_MS`) holds an in-flight activity for
|
|
89
|
+
the duration of its `startToCloseTimeout`, so a long activity is not reclaimed and
|
|
90
|
+
re-dispatched (duplicate execution) before it completes. Deriving the reclaim
|
|
91
|
+
window from `startToCloseTimeout` automatically makes a 60s+ activity safe by
|
|
92
|
+
default.
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## 5. Documented concurrent-condition bound for harvest fan-out
|
|
97
|
+
|
|
98
|
+
**Usage.** A market-maker workflow opens N `condition()` waits and harvests them
|
|
99
|
+
with `Promise.all`, chunked to stay within the safe concurrent-condition bound (a
|
|
100
|
+
fast signaler can arrive before a later condition registers).
|
|
101
|
+
|
|
102
|
+
**Need.** Publish the safe concurrent-condition bound as a stable number with the
|
|
103
|
+
recommended chunking pattern, so we size harvest fan-out to spec rather than
|
|
104
|
+
empirically — and raise it where the engine can.
|
|
105
|
+
|
|
106
|
+
---
|
|
107
|
+
|
|
108
|
+
## 6. Signal buffering + startChild leader-keeps-seat as public contract
|
|
109
|
+
|
|
110
|
+
**Usage.** Two behaviors our crash-safe patterns build on:
|
|
111
|
+
- a signal delivered to a workflow **not yet parked** on that signalId is buffered
|
|
112
|
+
and delivered on the next `condition()` await;
|
|
113
|
+
- `startChild` with a **duplicate workflowId** lets the sitting execution keep its
|
|
114
|
+
seat (leader-keeps-seat), enabling self-continuing link chains (`x-l1`, `x-l2`, …)
|
|
115
|
+
as an alternative to `continueAsNew`.
|
|
116
|
+
|
|
117
|
+
**Need.** Elevate both to documented, stable public guarantees (including the signal
|
|
118
|
+
buffering TTL), versioned as contract.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
### How we consume these
|
|
123
|
+
|
|
124
|
+
Items 1–2 are the scale needs — retention for terminal rows, and durable app-owned
|
|
125
|
+
indexes that keep the hot path `O(pending)`. Item 3 is the atomicity guarantee our
|
|
126
|
+
versioned forms and metadata routing depend on. Items 4–6 are behaviors we already
|
|
127
|
+
build on and want made explicit. We can provide `EXPLAIN` plans, row-count/latency
|
|
128
|
+
profiles, and the exact DDL for any of these on request.
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# HotMesh v0.26.0 — for the long-tail project
|
|
2
|
+
|
|
3
|
+
This release answers the needs document. Items 1–5 and the signal-buffering
|
|
4
|
+
half of item 6 ship as published contracts with proving tests; the startChild
|
|
5
|
+
contract (item 6b) is scheduled for its own follow-up release. Numbering below
|
|
6
|
+
matches your document.
|
|
7
|
+
|
|
8
|
+
## 1. Retention for terminal escalation rows — new API
|
|
9
|
+
|
|
10
|
+
Terminal rows (`resolved`, `cancelled`, `expired`) are inert: every engine
|
|
11
|
+
state transition guards on `status = 'pending'`, and the engine reads terminal
|
|
12
|
+
rows only through `list()` / `get()` / `stats()`. Age them out with the new
|
|
13
|
+
engine-owned call:
|
|
14
|
+
|
|
15
|
+
```typescript
|
|
16
|
+
let deleted: number;
|
|
17
|
+
do {
|
|
18
|
+
({ deleted } = await client.escalations.prune({ olderThan: '90 days' }));
|
|
19
|
+
} while (deleted > 0);
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
- `olderThan` is a Postgres interval; rows qualify when
|
|
23
|
+
`updated_at < NOW() - olderThan`.
|
|
24
|
+
- `statuses` narrows which terminal states are pruned (defaults to all three);
|
|
25
|
+
`namespace` scopes to one app; `limit` bounds rows per call (default 10,000,
|
|
26
|
+
cap 100,000) so vacuum pressure stays flat.
|
|
27
|
+
- Each call is one atomic statement — candidate selection
|
|
28
|
+
(`FOR UPDATE SKIP LOCKED`) and the DELETE commit together, so concurrent
|
|
29
|
+
pruners cooperate and live waiters, claims, and signal delivery proceed
|
|
30
|
+
untouched. The proving test backdates a parked waiter 120 days, prunes, and
|
|
31
|
+
resolves it to completion.
|
|
32
|
+
- If you run retention with your own tooling instead, the documented
|
|
33
|
+
safe-to-delete predicate is exactly the one `prune()` uses:
|
|
34
|
+
`status IN ('resolved','cancelled','expired') AND updated_at` older than
|
|
35
|
+
your horizon.
|
|
36
|
+
- Choose a horizon at least as long as your reporting window: `stats()`
|
|
37
|
+
created/resolved counts and `get()` by id reflect the rows retained.
|
|
38
|
+
|
|
39
|
+
## 2. App-added indexes survive engine migrations — published contract
|
|
40
|
+
|
|
41
|
+
Engine migrations on `public.hmsh_escalations` are strictly additive
|
|
42
|
+
(`CREATE TABLE / INDEX IF NOT EXISTS`, `ADD COLUMN IF NOT EXISTS`, advisory-
|
|
43
|
+
locked), and the table persists in place across upgrades. The engine manages
|
|
44
|
+
only indexes with the `idx_hmsh_esc_*` prefix; indexes under any other name
|
|
45
|
+
are yours and persist. Your advisory-locked
|
|
46
|
+
`CREATE INDEX CONCURRENTLY IF NOT EXISTS` pattern is the recommended one.
|
|
47
|
+
|
|
48
|
+
Two notes on your shapes: the engine's `idx_hmsh_esc_available`
|
|
49
|
+
(`(namespace, app_id, role, priority ASC, created_at ASC) WHERE
|
|
50
|
+
status = 'pending'`) already serves the single-role claim/available order at
|
|
51
|
+
`O(pending)`. Your pending-scoped GIN stays app-owned: the engine keeps its
|
|
52
|
+
full-table GIN because resolver-merged metadata (v0.24.0) is read on resolved
|
|
53
|
+
rows.
|
|
54
|
+
|
|
55
|
+
## 3. Atomic metadata in the Leg1 checkpoint — published contract
|
|
56
|
+
|
|
57
|
+
Every field of the `condition()` config — `metadata` facets included — is one
|
|
58
|
+
INSERT committed inside the Leg1 checkpoint transaction: one commit,
|
|
59
|
+
crash-safe. From the row's first visible moment it carries its complete
|
|
60
|
+
routing context and metadata, so claim-by-metadata routing and
|
|
61
|
+
`schema_version`-pinned forms can trust every row they read. The proving test
|
|
62
|
+
tight-polls from workflow start and asserts the first sighting is complete.
|
|
63
|
+
|
|
64
|
+
## 4. Long activities execute exactly once — published contract
|
|
65
|
+
|
|
66
|
+
- `startToCloseTimeout` is honored for long activities (deadline enforcement
|
|
67
|
+
is covered by positive and negative tests).
|
|
68
|
+
- While an activity runs, the consumer heartbeats its stream reservation at
|
|
69
|
+
half the base window, so the message stays leased for the full run —
|
|
70
|
+
your 60s batch loop (poll → reconcile → act, one checkpoint per call) is a
|
|
71
|
+
supported shape at default settings, for any duration. Redelivery happens
|
|
72
|
+
when a consumer crashes and stops heartbeating, and the collation ledger
|
|
73
|
+
settles a redelivered message as a duplicate before the activity function
|
|
74
|
+
re-executes.
|
|
75
|
+
- The proving test runs a 45s activity against a 30s reservation window with
|
|
76
|
+
a second worker actively polling the same queue: exactly one execution.
|
|
77
|
+
- Deployment note: standard Postgres connections have this lease extension.
|
|
78
|
+
A `securedWorker` connection (SECURITY DEFINER stored-proc mode) enforces
|
|
79
|
+
the adaptive reservation window instead — 30s base, scaling to 30 minutes
|
|
80
|
+
under load — so size activities to that window in secured mode.
|
|
81
|
+
|
|
82
|
+
## 5. Concurrent-condition bound — the published number is a TTL
|
|
83
|
+
|
|
84
|
+
Fan-out is sized by time, and by count it scales freely. A signal that races
|
|
85
|
+
ahead of its `condition()` registration is buffered as a pending signal and
|
|
86
|
+
delivered — crash-safe, in a single transaction — when the wait registers.
|
|
87
|
+
The buffer holds a signal for **10 minutes** by default; pass `expire` to
|
|
88
|
+
`signal()` (e.g. `'1h'`, `'30d'`) when signaling early on purpose.
|
|
89
|
+
|
|
90
|
+
Chunk your harvest so every condition in a chunk registers within the TTL of
|
|
91
|
+
its fastest signaler. The proving test opens 25 waits in one `Promise.all`
|
|
92
|
+
and fires all 25 signals before any wait has registered: every signal is
|
|
93
|
+
delivered to its own wait, in order.
|
|
94
|
+
|
|
95
|
+
## 6. Contracts
|
|
96
|
+
|
|
97
|
+
- **Signal buffering** is now a documented public guarantee (item 5 above,
|
|
98
|
+
TTL included).
|
|
99
|
+
- **startChild leader-keeps-seat**: the sitting execution keeps its seat
|
|
100
|
+
today, and fire-and-forget `startChild` link chains (`x-l1`, `x-l2`, …)
|
|
101
|
+
work on that behavior. The *awaited* duplicate path (`executeChild` on an
|
|
102
|
+
id that already exists) gets defined semantics — a typed result rather
|
|
103
|
+
than an open wait — in a dedicated follow-up release; until then, treat
|
|
104
|
+
awaited duplicates as unspecified and build chains on the fire-and-forget
|
|
105
|
+
form.
|
|
106
|
+
|
|
107
|
+
## Upgrading
|
|
108
|
+
|
|
109
|
+
`npm install @hotmeshio/hotmesh@0.26.0`. Migrations are additive and run
|
|
110
|
+
automatically under the existing advisory lock; your indexes and views
|
|
111
|
+
persist. Full durable suite: 433 tests green.
|
package/build/package.json
CHANGED
|
@@ -575,6 +575,9 @@ class Hook extends activity_1.Activity {
|
|
|
575
575
|
this.context.metadata.jid = jobId;
|
|
576
576
|
this.context.metadata.gid = gId;
|
|
577
577
|
this.context.metadata.dad = dad;
|
|
578
|
+
//captured for the timeout disarm below: processEvent's state
|
|
579
|
+
//hydration replaces context.metadata before the disarm runs
|
|
580
|
+
const waiterJid = jobId;
|
|
578
581
|
// Inline retry for FORBIDDEN: Leg2 arrived in the window between
|
|
579
582
|
// setHookSignal (standalone) and Leg1 transaction.exec(). The 100B
|
|
580
583
|
// ledger digit is not yet visible. Leg1 needs only milliseconds to
|
|
@@ -589,6 +592,25 @@ class Hook extends activity_1.Activity {
|
|
|
589
592
|
await this.processEvent(status, code, 'hook');
|
|
590
593
|
if (code === 200) {
|
|
591
594
|
await taskService.deleteWebHookSignal(this.config.hook.topic, data);
|
|
595
|
+
//signal won the race: disarm the SLA timeout leg (the mirror
|
|
596
|
+
//of the timeout-won path, which deletes the signal). The
|
|
597
|
+
//armed timehook may not survive to fire against the settled
|
|
598
|
+
//wait. Addressed by the SIGNAL's jid + dimensional address —
|
|
599
|
+
//processEvent has since rehydrated this.context.
|
|
600
|
+
if (this.config.sleep) {
|
|
601
|
+
try {
|
|
602
|
+
await this.store.expireTimeHook?.(waiterJid, this.metadata.aid);
|
|
603
|
+
}
|
|
604
|
+
catch (error) {
|
|
605
|
+
//best-effort: a surviving timer fires once and is
|
|
606
|
+
//consumed as an inactive-job event
|
|
607
|
+
this.logger.warn('hook-timeout-disarm-error', {
|
|
608
|
+
jid: waiterJid,
|
|
609
|
+
aid: this.metadata.aid,
|
|
610
|
+
error,
|
|
611
|
+
});
|
|
612
|
+
}
|
|
613
|
+
}
|
|
592
614
|
//clean up orphan pending on the sibling signal topic
|
|
593
615
|
// wfs.wait delivered → remove wfs.signal pending
|
|
594
616
|
// wfs.signal delivered → remove wfs.wait pending
|
|
@@ -46,9 +46,12 @@ import { ConditionQueueConfig } from '../../../types/hmsh_escalations';
|
|
|
46
46
|
* ## With escalation queue config
|
|
47
47
|
*
|
|
48
48
|
* Pass a {@link ConditionQueueConfig} as the second argument to surface the
|
|
49
|
-
* pause as a claimable row in `public.hmsh_escalations`. The INSERT
|
|
50
|
-
*
|
|
51
|
-
*
|
|
49
|
+
* pause as a claimable row in `public.hmsh_escalations`. The INSERT — every
|
|
50
|
+
* field of the config, including `metadata` facets — is committed atomically
|
|
51
|
+
* with the workflow checkpoint: one write, one commit, crash-safe. From the
|
|
52
|
+
* row's first visible moment it carries its complete routing context and
|
|
53
|
+
* metadata, so claim-by-metadata routing and version-pinned facets (e.g. a
|
|
54
|
+
* `schema_version` the resolver UI renders) can trust every row they read.
|
|
52
55
|
*
|
|
53
56
|
* ```typescript
|
|
54
57
|
* const decision = await Durable.workflow.condition<{ approved: boolean }>(
|
|
@@ -81,6 +84,14 @@ import { ConditionQueueConfig } from '../../../types/hmsh_escalations';
|
|
|
81
84
|
* SLA-gated human waits in the workflow body and let hook functions report
|
|
82
85
|
* back via `signal()`.
|
|
83
86
|
*
|
|
87
|
+
* ## Early signals are buffered
|
|
88
|
+
*
|
|
89
|
+
* A signal delivered before its `condition()` registers — a fast signaler,
|
|
90
|
+
* or a payload deposited before the workflow starts — is buffered as a
|
|
91
|
+
* pending signal and delivered when the wait registers. The buffer holds a
|
|
92
|
+
* signal for 10 minutes by default; pass `expire` to `signal()` (e.g.
|
|
93
|
+
* `'1h'`, `'30d'`) to hold it longer when signaling early on purpose.
|
|
94
|
+
*
|
|
84
95
|
* ## Fan-in: wait for multiple signals in parallel
|
|
85
96
|
*
|
|
86
97
|
* ```typescript
|
|
@@ -90,6 +101,12 @@ import { ConditionQueueConfig } from '../../../types/hmsh_escalations';
|
|
|
90
101
|
* ]);
|
|
91
102
|
* ```
|
|
92
103
|
*
|
|
104
|
+
* Harvest fan-out scales the same way: open N waits with `Promise.all` and
|
|
105
|
+
* signal all of them at once. Buffering covers every signal that races
|
|
106
|
+
* ahead of registration, so size fan-out by the pending-signal TTL — how
|
|
107
|
+
* long a racing signal may wait for its condition to register — with no
|
|
108
|
+
* separate bound on the number of concurrent waits.
|
|
109
|
+
*
|
|
93
110
|
* ## Paired with hook: spawn work, wait for its signal
|
|
94
111
|
*
|
|
95
112
|
* ```typescript
|
|
@@ -51,9 +51,12 @@ const didRun_1 = require("./didRun");
|
|
|
51
51
|
* ## With escalation queue config
|
|
52
52
|
*
|
|
53
53
|
* Pass a {@link ConditionQueueConfig} as the second argument to surface the
|
|
54
|
-
* pause as a claimable row in `public.hmsh_escalations`. The INSERT
|
|
55
|
-
*
|
|
56
|
-
*
|
|
54
|
+
* pause as a claimable row in `public.hmsh_escalations`. The INSERT — every
|
|
55
|
+
* field of the config, including `metadata` facets — is committed atomically
|
|
56
|
+
* with the workflow checkpoint: one write, one commit, crash-safe. From the
|
|
57
|
+
* row's first visible moment it carries its complete routing context and
|
|
58
|
+
* metadata, so claim-by-metadata routing and version-pinned facets (e.g. a
|
|
59
|
+
* `schema_version` the resolver UI renders) can trust every row they read.
|
|
57
60
|
*
|
|
58
61
|
* ```typescript
|
|
59
62
|
* const decision = await Durable.workflow.condition<{ approved: boolean }>(
|
|
@@ -86,6 +89,14 @@ const didRun_1 = require("./didRun");
|
|
|
86
89
|
* SLA-gated human waits in the workflow body and let hook functions report
|
|
87
90
|
* back via `signal()`.
|
|
88
91
|
*
|
|
92
|
+
* ## Early signals are buffered
|
|
93
|
+
*
|
|
94
|
+
* A signal delivered before its `condition()` registers — a fast signaler,
|
|
95
|
+
* or a payload deposited before the workflow starts — is buffered as a
|
|
96
|
+
* pending signal and delivered when the wait registers. The buffer holds a
|
|
97
|
+
* signal for 10 minutes by default; pass `expire` to `signal()` (e.g.
|
|
98
|
+
* `'1h'`, `'30d'`) to hold it longer when signaling early on purpose.
|
|
99
|
+
*
|
|
89
100
|
* ## Fan-in: wait for multiple signals in parallel
|
|
90
101
|
*
|
|
91
102
|
* ```typescript
|
|
@@ -95,6 +106,12 @@ const didRun_1 = require("./didRun");
|
|
|
95
106
|
* ]);
|
|
96
107
|
* ```
|
|
97
108
|
*
|
|
109
|
+
* Harvest fan-out scales the same way: open N waits with `Promise.all` and
|
|
110
|
+
* signal all of them at once. Buffering covers every signal that races
|
|
111
|
+
* ahead of registration, so size fan-out by the pending-signal TTL — how
|
|
112
|
+
* long a racing signal may wait for its condition to register — with no
|
|
113
|
+
* separate bound on the number of concurrent waits.
|
|
114
|
+
*
|
|
98
115
|
* ## Paired with hook: spawn work, wait for its signal
|
|
99
116
|
*
|
|
100
117
|
* ```typescript
|
|
@@ -111,6 +111,20 @@ declare function wrapActivity<T>(activityName: string, options?: ActivityConfig)
|
|
|
111
111
|
* }
|
|
112
112
|
* ```
|
|
113
113
|
*
|
|
114
|
+
* ## Long-running activities execute exactly once
|
|
115
|
+
*
|
|
116
|
+
* `startToCloseTimeout` bounds an activity's run and is honored for long
|
|
117
|
+
* work (a 30–120s batch loop is a supported shape: poll → reconcile → act,
|
|
118
|
+
* one durable checkpoint per call). While the activity runs, the consumer
|
|
119
|
+
* heartbeats its stream reservation at half the base window, so the message
|
|
120
|
+
* stays leased for the full run — however long — and is redelivered to
|
|
121
|
+
* another worker only when the owning consumer crashes and stops
|
|
122
|
+
* heartbeating. The collation ledger then guarantees any redelivered
|
|
123
|
+
* message settles as a duplicate before re-executing the activity. With a
|
|
124
|
+
* `securedWorker` connection (SECURITY DEFINER stored-proc mode), lease
|
|
125
|
+
* extension is unavailable and an activity must finish within the adaptive
|
|
126
|
+
* reservation window instead.
|
|
127
|
+
*
|
|
114
128
|
* @template ACT - The activity type map (use `typeof activities` for inline registration).
|
|
115
129
|
* @param {ActivityConfig} [options] - Activity configuration including retry policy and routing.
|
|
116
130
|
* @returns {ProxyType<ACT>} A typed proxy object mapping activity names to their durable wrappers.
|
|
@@ -230,6 +230,20 @@ exports.wrapActivity = wrapActivity;
|
|
|
230
230
|
* }
|
|
231
231
|
* ```
|
|
232
232
|
*
|
|
233
|
+
* ## Long-running activities execute exactly once
|
|
234
|
+
*
|
|
235
|
+
* `startToCloseTimeout` bounds an activity's run and is honored for long
|
|
236
|
+
* work (a 30–120s batch loop is a supported shape: poll → reconcile → act,
|
|
237
|
+
* one durable checkpoint per call). While the activity runs, the consumer
|
|
238
|
+
* heartbeats its stream reservation at half the base window, so the message
|
|
239
|
+
* stays leased for the full run — however long — and is redelivered to
|
|
240
|
+
* another worker only when the owning consumer crashes and stops
|
|
241
|
+
* heartbeating. The collation ledger then guarantees any redelivered
|
|
242
|
+
* message settles as a duplicate before re-executing the activity. With a
|
|
243
|
+
* `securedWorker` connection (SECURITY DEFINER stored-proc mode), lease
|
|
244
|
+
* extension is unavailable and an activity must finish within the adaptive
|
|
245
|
+
* reservation window instead.
|
|
246
|
+
*
|
|
233
247
|
* @template ACT - The activity type map (use `typeof activities` for inline registration).
|
|
234
248
|
* @param {ActivityConfig} [options] - Activity configuration including retry policy and routing.
|
|
235
249
|
* @returns {ProxyType<ACT>} A typed proxy object mapping activity names to their durable wrappers.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import { HotMesh } from '../hotmesh';
|
|
2
2
|
import { EventsConfig } from '../../types/system_events';
|
|
3
3
|
import { Connection } from '../../types/durable';
|
|
4
|
-
import { EscalationEntry, ClaimEscalationResult, ClaimByMetadataResult, ReleaseEscalationResult, ResolveEscalationResult, CancelEscalationResult, ListEscalationsParams, StatsEscalationsParams, EscalationStats, CreateEscalationParams, UpdateEscalationParams, AppendMilestonesParams, ClaimEscalationParams, ClaimByMetadataParams, ReleaseEscalationParams, ResolveEscalationParams, ResolveByMetadataParams, EscalateToRoleParams, MigrateEscalationParams, ClaimManyParams, EscalateManyToRoleParams, UpdateManyPriorityParams, ResolveManyParams } from '../../types/hmsh_escalations';
|
|
4
|
+
import { EscalationEntry, ClaimEscalationResult, ClaimByMetadataResult, ReleaseEscalationResult, ResolveEscalationResult, CancelEscalationResult, ListEscalationsParams, StatsEscalationsParams, EscalationStats, CreateEscalationParams, UpdateEscalationParams, AppendMilestonesParams, ClaimEscalationParams, ClaimByMetadataParams, ReleaseEscalationParams, ResolveEscalationParams, ResolveByMetadataParams, EscalateToRoleParams, MigrateEscalationParams, ClaimManyParams, EscalateManyToRoleParams, UpdateManyPriorityParams, ResolveManyParams, PruneEscalationsParams, PruneEscalationsResult } from '../../types/hmsh_escalations';
|
|
5
5
|
export type GetHotMeshFn = (topic: string | null, namespace?: string) => Promise<HotMesh>;
|
|
6
6
|
export interface EscalationClientConfig {
|
|
7
7
|
/** Postgres connection options — used when creating a standalone EscalationClient. */
|
|
@@ -73,9 +73,9 @@ export declare class EscalationClientService {
|
|
|
73
73
|
private _hashConnection;
|
|
74
74
|
private _deliverEscalationSignal;
|
|
75
75
|
/**
|
|
76
|
-
* Builds the wake
|
|
77
|
-
* INSIDE the resolve transaction — the wake becomes durable
|
|
78
|
-
*
|
|
76
|
+
* Builds the wake as a webhook message so the store can commit it
|
|
77
|
+
* INSIDE the resolve/cancel transaction — the wake becomes durable
|
|
78
|
+
* with the status change, closing the crash window between commit and
|
|
79
79
|
* post-commit signal delivery. Mirrors `_deliverEscalationSignal`'s
|
|
80
80
|
* topic fallback chain; returns null when no hook rule is deployed
|
|
81
81
|
* for any candidate topic (the caller then keeps post-commit
|
|
@@ -93,6 +93,18 @@ export declare class EscalationClientService {
|
|
|
93
93
|
* Uses the same filter parameters as `list()`.
|
|
94
94
|
*/
|
|
95
95
|
count(params?: ListEscalationsParams): Promise<number>;
|
|
96
|
+
/**
|
|
97
|
+
* Retention: deletes terminal escalation rows (`resolved`/`cancelled`/`expired`)
|
|
98
|
+
* whose `updated_at` is older than the horizon (a Postgres interval string,
|
|
99
|
+
* e.g. `'90 days'`). Terminal rows are inert — every engine state transition
|
|
100
|
+
* guards on `status = 'pending'` — so pruning them is safe for live waiters,
|
|
101
|
+
* claims, and signal delivery; this call is the engine-blessed way to age
|
|
102
|
+
* out the audit backlog. Each call deletes at most `limit` rows (default
|
|
103
|
+
* 10,000) in one atomic statement; loop until `deleted` is 0 to drain a
|
|
104
|
+
* large backlog. `list()`/`get()`/`stats()` reads over windows older than
|
|
105
|
+
* the pruning horizon reflect only the rows retained.
|
|
106
|
+
*/
|
|
107
|
+
prune(params: PruneEscalationsParams): Promise<PruneEscalationsResult>;
|
|
96
108
|
/** Returns a single escalation row by UUID. Returns `null` if not found. */
|
|
97
109
|
get(id: string, namespace?: string): Promise<EscalationEntry | null>;
|
|
98
110
|
/** Looks up an escalation by `signal_key` — the value passed to `condition()`. */
|
|
@@ -132,10 +144,10 @@ export declare class EscalationClientService {
|
|
|
132
144
|
/**
|
|
133
145
|
* Cancels a pending escalation and delivers a cancellation signal to the
|
|
134
146
|
* waiting workflow so that `condition()` returns `null`. Terminal rows
|
|
135
|
-
* return `already-terminal`.
|
|
136
|
-
*
|
|
137
|
-
*
|
|
138
|
-
*
|
|
147
|
+
* return `already-terminal`. The cancellation wake commits inside the
|
|
148
|
+
* cancel transaction (same durability contract as `resolve()`); when no
|
|
149
|
+
* hook rule is deployed for any candidate topic, delivery falls back to
|
|
150
|
+
* a best-effort post-commit publish.
|
|
139
151
|
*/
|
|
140
152
|
cancel(id: string, namespace?: string): Promise<CancelEscalationResult>;
|
|
141
153
|
/**
|
|
@@ -207,6 +219,12 @@ export declare class EscalationClientService {
|
|
|
207
219
|
* (still-pending) row's GIN-indexed `metadata` in the single atomic UPDATE.
|
|
208
220
|
* See {@link resolve}.
|
|
209
221
|
*/
|
|
222
|
+
/**
|
|
223
|
+
* Bulk-resolves standalone escalations (rows with `signal_key = null`).
|
|
224
|
+
* Rows backing a live `condition()` waiter are excluded — they stay
|
|
225
|
+
* pending so a targeted `resolve()`/`cancel()` can deliver their wake;
|
|
226
|
+
* bulk resolution carries no wake and would strand the workflow.
|
|
227
|
+
*/
|
|
210
228
|
resolveMany(params: ResolveManyParams): Promise<EscalationEntry[]>;
|
|
211
229
|
/**
|
|
212
230
|
* Returns dashboard-ready escalation counts. `period` controls the window
|
|
@@ -3,7 +3,6 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
3
3
|
exports.EscalationClientService = void 0;
|
|
4
4
|
const enums_1 = require("../../modules/enums");
|
|
5
5
|
const utils_1 = require("../../modules/utils");
|
|
6
|
-
const key_1 = require("../../modules/key");
|
|
7
6
|
const hotmesh_1 = require("../hotmesh");
|
|
8
7
|
const types_1 = require("../../types");
|
|
9
8
|
const stream_1 = require("../../types/stream");
|
|
@@ -148,9 +147,9 @@ class EscalationClientService {
|
|
|
148
147
|
return delivered;
|
|
149
148
|
}
|
|
150
149
|
/**
|
|
151
|
-
* Builds the wake
|
|
152
|
-
* INSIDE the resolve transaction — the wake becomes durable
|
|
153
|
-
*
|
|
150
|
+
* Builds the wake as a webhook message so the store can commit it
|
|
151
|
+
* INSIDE the resolve/cancel transaction — the wake becomes durable
|
|
152
|
+
* with the status change, closing the crash window between commit and
|
|
154
153
|
* post-commit signal delivery. Mirrors `_deliverEscalationSignal`'s
|
|
155
154
|
* topic fallback chain; returns null when no hook rule is deployed
|
|
156
155
|
* for any candidate topic (the caller then keeps post-commit
|
|
@@ -173,13 +172,10 @@ class EscalationClientService {
|
|
|
173
172
|
metadata: { guid: (0, utils_1.guid)(), aid, topic: candidate },
|
|
174
173
|
data: { id: signalKey, data },
|
|
175
174
|
};
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
JSON.stringify(streamData),
|
|
181
|
-
]);
|
|
182
|
-
return { forSignalKey: signalKey, sql, params };
|
|
175
|
+
return {
|
|
176
|
+
forSignalKey: signalKey,
|
|
177
|
+
message: JSON.stringify(streamData),
|
|
178
|
+
};
|
|
183
179
|
}
|
|
184
180
|
catch {
|
|
185
181
|
/* candidate not deployed — try the next topic */
|
|
@@ -205,6 +201,21 @@ class EscalationClientService {
|
|
|
205
201
|
const hm = await this._engine(null, params?.namespace);
|
|
206
202
|
return hm.engine.store.countEscalations(params ?? {});
|
|
207
203
|
}
|
|
204
|
+
/**
|
|
205
|
+
* Retention: deletes terminal escalation rows (`resolved`/`cancelled`/`expired`)
|
|
206
|
+
* whose `updated_at` is older than the horizon (a Postgres interval string,
|
|
207
|
+
* e.g. `'90 days'`). Terminal rows are inert — every engine state transition
|
|
208
|
+
* guards on `status = 'pending'` — so pruning them is safe for live waiters,
|
|
209
|
+
* claims, and signal delivery; this call is the engine-blessed way to age
|
|
210
|
+
* out the audit backlog. Each call deletes at most `limit` rows (default
|
|
211
|
+
* 10,000) in one atomic statement; loop until `deleted` is 0 to drain a
|
|
212
|
+
* large backlog. `list()`/`get()`/`stats()` reads over windows older than
|
|
213
|
+
* the pruning horizon reflect only the rows retained.
|
|
214
|
+
*/
|
|
215
|
+
async prune(params) {
|
|
216
|
+
const hm = await this._engine(null, params.namespace);
|
|
217
|
+
return hm.engine.store.pruneEscalations(params);
|
|
218
|
+
}
|
|
208
219
|
/** Returns a single escalation row by UUID. Returns `null` if not found. */
|
|
209
220
|
async get(id, namespace) {
|
|
210
221
|
const hm = await this._engine(null, namespace);
|
|
@@ -286,18 +297,27 @@ class EscalationClientService {
|
|
|
286
297
|
/**
|
|
287
298
|
* Cancels a pending escalation and delivers a cancellation signal to the
|
|
288
299
|
* waiting workflow so that `condition()` returns `null`. Terminal rows
|
|
289
|
-
* return `already-terminal`.
|
|
290
|
-
*
|
|
291
|
-
*
|
|
292
|
-
*
|
|
300
|
+
* return `already-terminal`. The cancellation wake commits inside the
|
|
301
|
+
* cancel transaction (same durability contract as `resolve()`); when no
|
|
302
|
+
* hook rule is deployed for any candidate topic, delivery falls back to
|
|
303
|
+
* a best-effort post-commit publish.
|
|
293
304
|
*/
|
|
294
305
|
async cancel(id, namespace) {
|
|
295
306
|
const ns = namespace ?? factory_1.APP_ID;
|
|
296
307
|
const hm = await this._engine(null, ns);
|
|
297
|
-
const
|
|
308
|
+
const store = hm.engine.store;
|
|
309
|
+
//pre-build the cancellation wake so it commits INSIDE the cancel
|
|
310
|
+
//transaction — condition() resumes with null even if the process
|
|
311
|
+
//dies the instant the cancel commits
|
|
312
|
+
let wakeCommand = null;
|
|
313
|
+
const preview = await store.getEscalation(id, namespace);
|
|
314
|
+
if (preview?.signal_key) {
|
|
315
|
+
wakeCommand = await this._buildWakeCommand(ns, preview.topic, preview.signal_key, { __escalation_cancelled: true });
|
|
316
|
+
}
|
|
317
|
+
const result = await store.cancelEscalation(id, namespace, wakeCommand ?? undefined);
|
|
298
318
|
if (result.ok === true) {
|
|
299
319
|
this._emit('cancelled', result.entry);
|
|
300
|
-
if (result.entry.signal_key) {
|
|
320
|
+
if (result.entry.signal_key && !result.wakeEnqueued) {
|
|
301
321
|
await this._deliverEscalationSignal(ns, result.entry.topic, {
|
|
302
322
|
id: result.entry.signal_key,
|
|
303
323
|
data: { __escalation_cancelled: true },
|
|
@@ -444,6 +464,12 @@ class EscalationClientService {
|
|
|
444
464
|
* (still-pending) row's GIN-indexed `metadata` in the single atomic UPDATE.
|
|
445
465
|
* See {@link resolve}.
|
|
446
466
|
*/
|
|
467
|
+
/**
|
|
468
|
+
* Bulk-resolves standalone escalations (rows with `signal_key = null`).
|
|
469
|
+
* Rows backing a live `condition()` waiter are excluded — they stay
|
|
470
|
+
* pending so a targeted `resolve()`/`cancel()` can deliver their wake;
|
|
471
|
+
* bulk resolution carries no wake and would strand the workflow.
|
|
472
|
+
*/
|
|
447
473
|
async resolveMany(params) {
|
|
448
474
|
const hm = await this._engine(null, params.namespace);
|
|
449
475
|
const entries = await hm.engine.store.resolveManyEscalations(params);
|