@adonis-agora/durable 0.17.1 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -0
- package/dist/src/commands/runs.d.ts +7 -1
- package/dist/src/commands/runs.d.ts.map +1 -1
- package/dist/src/commands/runs.js +12 -3
- package/dist/src/commands/runs.js.map +1 -1
- package/dist/src/dashboard/index.d.ts +1 -1
- package/dist/src/dashboard/index.js +1 -1
- package/dist/src/engine.d.ts +14 -0
- package/dist/src/engine.d.ts.map +1 -1
- package/dist/src/engine.js +66 -5
- package/dist/src/engine.js.map +1 -1
- package/dist/src/index.d.ts +1 -1
- package/dist/src/index.js +1 -1
- package/dist/src/interfaces.d.ts +51 -0
- package/dist/src/interfaces.d.ts.map +1 -1
- package/dist/src/protocol.d.ts +2 -2
- package/dist/src/protocol.d.ts.map +1 -1
- package/dist/src/protocol.js +22 -2
- package/dist/src/protocol.js.map +1 -1
- package/dist/src/singleton-gate.d.ts.map +1 -1
- package/dist/src/singleton-gate.js +10 -7
- package/dist/src/singleton-gate.js.map +1 -1
- package/dist/src/step-logger.d.ts +1 -7
- package/dist/src/step-logger.d.ts.map +1 -1
- package/dist/src/step-logger.js +17 -1
- package/dist/src/step-logger.js.map +1 -1
- package/dist/src/step-name-symbol.d.ts +3 -0
- package/dist/src/step-name-symbol.d.ts.map +1 -1
- package/dist/src/step-name-symbol.js.map +1 -1
- package/dist/src/step-ref.d.ts +5 -2
- package/dist/src/step-ref.d.ts.map +1 -1
- package/dist/src/step-ref.js +2 -0
- package/dist/src/step-ref.js.map +1 -1
- package/dist/src/stores/lucid-mappers.d.ts +4 -0
- package/dist/src/stores/lucid-mappers.d.ts.map +1 -1
- package/dist/src/stores/lucid-mappers.js +6 -0
- package/dist/src/stores/lucid-mappers.js.map +1 -1
- package/dist/src/stores/lucid-schema.d.ts.map +1 -1
- package/dist/src/stores/lucid-schema.js +18 -6
- package/dist/src/stores/lucid-schema.js.map +1 -1
- package/dist/src/stores/lucid.d.ts +4 -0
- package/dist/src/stores/lucid.d.ts.map +1 -1
- package/dist/src/stores/lucid.js +13 -0
- package/dist/src/stores/lucid.js.map +1 -1
- package/dist/src/testing/in-memory-state-store.d.ts +2 -0
- package/dist/src/testing/in-memory-state-store.d.ts.map +1 -1
- package/dist/src/testing/in-memory-state-store.js +8 -0
- package/dist/src/testing/in-memory-state-store.js.map +1 -1
- package/dist/src/transports/db.d.ts.map +1 -1
- package/dist/src/transports/db.js +1 -1
- package/dist/src/transports/db.js.map +1 -1
- package/dist/src/transports/event-emitter.js +1 -1
- package/dist/src/transports/event-emitter.js.map +1 -1
- package/dist/src/transports/queue.d.ts.map +1 -1
- package/dist/src/transports/queue.js +1 -1
- package/dist/src/transports/queue.js.map +1 -1
- package/dist/src/workflow-ctx.d.ts.map +1 -1
- package/dist/src/workflow-ctx.js +3 -0
- package/dist/src/workflow-ctx.js.map +1 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,29 @@
|
|
|
1
1
|
# @adonis-agora/durable
|
|
2
2
|
|
|
3
|
+
## 0.19.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- [#37](https://github.com/DavideCarvalho/adonis-durable/pull/37) [`9cff848`](https://github.com/DavideCarvalho/adonis-durable/commit/9cff84804bd3b8b3f63a3b7fd4b841b6346deb47) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Liveness follow-ups — both halves paid for in production:
|
|
8
|
+
|
|
9
|
+
**`pickupTimeoutMs` + automatic pickup beat: queue wait is not silence.** The `timeoutMs` window is anchored at dispatch, but the stretch a job spends QUEUED produces no heartbeats by definition — a single-concurrency worker whose batches run ~15min made the next batch wait silently the whole time, and a dispatch-anchored window false-failed a healthy fleet. Now every worker emits an automatic pickup beat the moment it claims a task (via `runStepHandler` — this also stamps `lastHeartbeatAt` for handlers that never beat manually, so "queued" vs "executing" is visible on every remote checkpoint), and the new `pickupTimeoutMs` step option (def-level or per-call; defaults to `timeoutMs` — the historical behavior) governs the pre-first-beat stretch, handing over to the tighter `timeoutMs` once execution starts: `pickupTimeoutMs` = "how long may it stay queued", `timeoutMs` = "max silence while running". The heartbeat persist throttle lets a progress-carrying beat punch through when the last persisted beat had no payload, so the pickup beat never shadows the handler's first real progress.
|
|
10
|
+
|
|
11
|
+
**Singleton release no longer orphans gated runs under a no-op run dispatcher.** `wakeNext` used to CLEAR the gated run's retry `wakeAt` before handing it to the run dispatcher — but the dispatcher may legitimately be a no-op (poll-only deployments where `durable:work` owns every pickup), which left the run `suspended` with no wake time, unreachable by every poll path, forever. It now stamps a due-now `wakeAt` instead: a real dispatcher still runs it immediately, and the timer poller is the guaranteed fallback (the run lease makes a double-drive a cheap no-op).
|
|
12
|
+
|
|
13
|
+
## 0.18.0
|
|
14
|
+
|
|
15
|
+
### Minor Changes
|
|
16
|
+
|
|
17
|
+
- [#35](https://github.com/DavideCarvalho/adonis-durable/pull/35) [`7668030`](https://github.com/DavideCarvalho/adonis-durable/commit/7668030d377bd879237c97efd98e4d553a995658) Thanks [@DavideCarvalho](https://github.com/DavideCarvalho)! - Step-level liveness, end to end: `log.heartbeat(progress?)` for long step handlers.
|
|
18
|
+
|
|
19
|
+
A long remote step (a 15-minute browser batch) was indistinguishable from a hung one: logs and sub-events only ship WITH the result, the heartbeat lane existed in every transport but nothing could emit into it, and the in-memory `timeoutMs` path persisted no checkpoint at all mid-flight — so status surfaces had to infer liveness from domain tables.
|
|
20
|
+
|
|
21
|
+
Now the step handler's `StepLogger` carries `heartbeat(progress?)`: it travels immediately over the transport's heartbeat lane (throttled, ≥5s between emissions), rearms the step's `timeoutMs` window on the engine (a beating step never falsely times out — so `timeoutMs` can be tightened to "max tolerated silence" instead of "max total duration"), and the engine persists the latest beat on the step's checkpoint (throttled ≥10s, best-effort) as `StepCheckpoint.lastHeartbeatAt` / `heartbeatProgress` — visible cross-process via `listCheckpoints`, and in `durable:runs` (the PENDING column shows `hb <age>`; `--stale` no longer flags a step whose worker beat within the threshold).
|
|
22
|
+
|
|
23
|
+
The in-memory `timeoutMs` path now also persists its `pending` checkpoint at dispatch (per attempt): the step becomes visible outside its process, beats have somewhere to land, and a result consumed by a different engine instance completes the checkpoint instead of being dropped. Replay semantics are unchanged (`timeoutMs` steps route to the in-memory path regardless of an existing pending row; `completed` still short-circuits).
|
|
24
|
+
|
|
25
|
+
New optional store capability: `StateStore.recordStepHeartbeat(runId, seq, at, progress?)` (implemented by the Lucid and in-memory stores; the checkpoints table gains nullable `last_heartbeat_at`/`heartbeat_progress`, auto-migrated in place by `ensureSchema`).
|
|
26
|
+
|
|
3
27
|
## 0.17.1
|
|
4
28
|
|
|
5
29
|
### Patch Changes
|
|
@@ -36,6 +36,10 @@ export interface StalePendingStep {
|
|
|
36
36
|
attempts: number;
|
|
37
37
|
/** Milliseconds since the step was enqueued to the transport. */
|
|
38
38
|
ageMs: number;
|
|
39
|
+
/** Milliseconds since the worker's last persisted heartbeat for this step, or `null` when the
|
|
40
|
+
* handler never beat (see `StepCheckpoint.lastHeartbeatAt`). THE disambiguator for a long pending
|
|
41
|
+
* step: old + beating = alive mid-flight; old + silent = hung or lost. */
|
|
42
|
+
heartbeatAgeMs: number | null;
|
|
39
43
|
}
|
|
40
44
|
/** A run plus the liveness signals that tell "working" apart from "stranded" (see module doc above). */
|
|
41
45
|
export interface RunLiveness {
|
|
@@ -57,7 +61,9 @@ export declare const DEFAULT_STALE_MS: number;
|
|
|
57
61
|
*/
|
|
58
62
|
export declare function attachLiveness(source: RunLister, runs: WorkflowRun[], now?: number): Promise<RunLiveness[]>;
|
|
59
63
|
/** Keep only runs whose oldest pending remote step has been unresolved for at least `thresholdMs` —
|
|
60
|
-
* the "these are probably stranded" view behind `--stale`.
|
|
64
|
+
* the "these are probably stranded" view behind `--stale`. A step whose worker heartbeat is
|
|
65
|
+
* YOUNGER than the threshold is excluded however old the step is: a beating worker is mid-flight
|
|
66
|
+
* (a 40-minute browser batch is working, not stranded), and re-driving it would double-run it. */
|
|
61
67
|
export declare function filterStale(liveRuns: RunLiveness[], thresholdMs?: number): RunLiveness[];
|
|
62
68
|
/** Parse a compact single-unit duration (`90s`, `15m`, `4h`, `2d`) into milliseconds. Returns
|
|
63
69
|
* `undefined` for anything else so a fat-fingered `--stale=15mins` in an ops one-liner falls back to
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runs.d.ts","sourceRoot":"","sources":["../../../src/commands/runs.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACV,QAAQ,EACR,SAAS,EACT,SAAS,EACT,cAAc,EACd,cAAc,EACd,WAAW,EACZ,MAAM,aAAa,CAAC;AAErB,MAAM,WAAW,eAAe;IAC9B,oGAAoG;IACpG,MAAM,CAAC,EAAE,SAAS,GAAG,SAAS,CAAC;IAC/B;;;;;OAKG;IACH,QAAQ,CAAC,EAAE,SAAS,EAAE,GAAG,SAAS,CAAC;IACnC,+BAA+B;IAC/B,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B,4BAA4B;IAC5B,KAAK,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAC5B;AAED;+BAC+B;AAC/B,MAAM,WAAW,SAAS;IACxB,QAAQ,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC,CAAC;IAClD,eAAe,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC;CAC3D;AAED,mFAAmF;AACnF,wBAAsB,QAAQ,CAAC,MAAM,EAAE,SAAS,EAAE,IAAI,EAAE,eAAe,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC,CAM/F;AAED;;;;GAIG;AACH,wBAAsB,QAAQ,CAAC,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC,CAE/F;AAeD,wGAAwG;AACxG,MAAM,WAAW,gBAAgB;IAC/B,GAAG,EAAE,MAAM,CAAC;IACZ,IAAI,EAAE,MAAM,CAAC;IACb,sGAAsG;IACtG,QAAQ,EAAE,MAAM,CAAC;IACjB,iEAAiE;IACjE,KAAK,EAAE,MAAM,CAAC;
|
|
1
|
+
{"version":3,"file":"runs.d.ts","sourceRoot":"","sources":["../../../src/commands/runs.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EACV,QAAQ,EACR,SAAS,EACT,SAAS,EACT,cAAc,EACd,cAAc,EACd,WAAW,EACZ,MAAM,aAAa,CAAC;AAErB,MAAM,WAAW,eAAe;IAC9B,oGAAoG;IACpG,MAAM,CAAC,EAAE,SAAS,GAAG,SAAS,CAAC;IAC/B;;;;;OAKG;IACH,QAAQ,CAAC,EAAE,SAAS,EAAE,GAAG,SAAS,CAAC;IACnC,+BAA+B;IAC/B,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B,4BAA4B;IAC5B,KAAK,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAC5B;AAED;+BAC+B;AAC/B,MAAM,WAAW,SAAS;IACxB,QAAQ,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC,CAAC;IAClD,eAAe,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC,CAAC;CAC3D;AAED,mFAAmF;AACnF,wBAAsB,QAAQ,CAAC,MAAM,EAAE,SAAS,EAAE,IAAI,EAAE,eAAe,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC,CAM/F;AAED;;;;GAIG;AACH,wBAAsB,QAAQ,CAAC,MAAM,EAAE,cAAc,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC,CAE/F;AAeD,wGAAwG;AACxG,MAAM,WAAW,gBAAgB;IAC/B,GAAG,EAAE,MAAM,CAAC;IACZ,IAAI,EAAE,MAAM,CAAC;IACb,sGAAsG;IACtG,QAAQ,EAAE,MAAM,CAAC;IACjB,iEAAiE;IACjE,KAAK,EAAE,MAAM,CAAC;IACd;;+EAE2E;IAC3E,cAAc,EAAE,MAAM,GAAG,IAAI,CAAC;CAC/B;AAED,wGAAwG;AACxG,MAAM,WAAW,WAAW;IAC1B,GAAG,EAAE,WAAW,CAAC;IACjB,0CAA0C;IAC1C,KAAK,EAAE,MAAM,CAAC;IACd,sEAAsE;IACtE,gBAAgB,EAAE,MAAM,CAAC;IACzB;sFACkF;IAClF,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC;CACvC;AAED,sGAAsG;AACtG,eAAO,MAAM,gBAAgB,QAAc,CAAC;AAyB5C;;;;GAIG;AACH,wBAAsB,cAAc,CAClC,MAAM,EAAE,SAAS,EACjB,IAAI,EAAE,WAAW,EAAE,EACnB,GAAG,GAAE,MAAmB,GACvB,OAAO,CAAC,WAAW,EAAE,CAAC,CAexB;AAED;;;mGAGmG;AACnG,wBAAgB,WAAW,CACzB,QAAQ,EAAE,WAAW,EAAE,EACvB,WAAW,GAAE,MAAyB,GACrC,WAAW,EAAE,CAOf;AAED;;iDAEiD;AACjD,wBAAgB,eAAe,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,GAAG,SAAS,CAOjE;AAED;;sBAEsB;AACtB,wBAAgB,SAAS,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAK/C;AAED;;;+EAG+E;AAC/E,wBAAgB,eAAe,CAAC,QAAQ,EAAE,WAAW,EAAE,GAAG,MAAM,CAiB/D"}
|
|
@@ -34,6 +34,7 @@ async function oldestPendingRemoteStep(source, runId, now) {
|
|
|
34
34
|
name: oldest.name,
|
|
35
35
|
attempts: oldest.attempts,
|
|
36
36
|
ageMs: now - oldest.enqueuedAt.getTime(),
|
|
37
|
+
heartbeatAgeMs: oldest.lastHeartbeatAt ? now - oldest.lastHeartbeatAt.getTime() : null,
|
|
37
38
|
};
|
|
38
39
|
}
|
|
39
40
|
/**
|
|
@@ -55,9 +56,13 @@ export async function attachLiveness(source, runs, now = Date.now()) {
|
|
|
55
56
|
}));
|
|
56
57
|
}
|
|
57
58
|
/** Keep only runs whose oldest pending remote step has been unresolved for at least `thresholdMs` —
|
|
58
|
-
* the "these are probably stranded" view behind `--stale`.
|
|
59
|
+
* the "these are probably stranded" view behind `--stale`. A step whose worker heartbeat is
|
|
60
|
+
* YOUNGER than the threshold is excluded however old the step is: a beating worker is mid-flight
|
|
61
|
+
* (a 40-minute browser batch is working, not stranded), and re-driving it would double-run it. */
|
|
59
62
|
export function filterStale(liveRuns, thresholdMs = DEFAULT_STALE_MS) {
|
|
60
|
-
return liveRuns.filter((rl) => rl.stalePending !== null &&
|
|
63
|
+
return liveRuns.filter((rl) => rl.stalePending !== null &&
|
|
64
|
+
rl.stalePending.ageMs >= thresholdMs &&
|
|
65
|
+
(rl.stalePending.heartbeatAgeMs === null || rl.stalePending.heartbeatAgeMs >= thresholdMs));
|
|
61
66
|
}
|
|
62
67
|
/** Parse a compact single-unit duration (`90s`, `15m`, `4h`, `2d`) into milliseconds. Returns
|
|
63
68
|
* `undefined` for anything else so a fat-fingered `--stale=15mins` in an ops one-liner falls back to
|
|
@@ -91,7 +96,11 @@ export function renderRunsTable(liveRuns) {
|
|
|
91
96
|
run.status,
|
|
92
97
|
compactAge(ageMs),
|
|
93
98
|
recoveryAttempts > 0 ? String(recoveryAttempts) : '-',
|
|
94
|
-
stalePending
|
|
99
|
+
stalePending
|
|
100
|
+
? `${compactAge(stalePending.ageMs)} (attempt ${stalePending.attempts}${stalePending.heartbeatAgeMs !== null
|
|
101
|
+
? `, hb ${compactAge(stalePending.heartbeatAgeMs)}`
|
|
102
|
+
: ''})`
|
|
103
|
+
: '-',
|
|
95
104
|
]);
|
|
96
105
|
return table(['RUN', 'WORKFLOW', 'STATUS', 'UPDATED', 'RECOVERY', 'PENDING'], rows);
|
|
97
106
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"runs.js","sourceRoot":"","sources":["../../../src/commands/runs.ts"],"names":[],"mappings":"AAgCA,mFAAmF;AACnF,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,MAAiB,EAAE,IAAqB;IACrE,OAAO,MAAM,CAAC,QAAQ,CAAC;QACrB,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC7F,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACrD,KAAK,EAAE,IAAI,CAAC,KAAK,IAAI,EAAE;KACxB,CAAC,CAAC;AACL,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,MAAsB,EAAE,KAAa;IAClE,OAAO,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC/B,CAAC;
|
|
1
|
+
{"version":3,"file":"runs.js","sourceRoot":"","sources":["../../../src/commands/runs.ts"],"names":[],"mappings":"AAgCA,mFAAmF;AACnF,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,MAAiB,EAAE,IAAqB;IACrE,OAAO,MAAM,CAAC,QAAQ,CAAC;QACrB,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QAC7F,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,QAAQ,EAAE,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;QACrD,KAAK,EAAE,IAAI,CAAC,KAAK,IAAI,EAAE;KACxB,CAAC,CAAC;AACL,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,MAAsB,EAAE,KAAa;IAClE,OAAO,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAC/B,CAAC;AAyCD,sGAAsG;AACtG,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,GAAG,MAAM,CAAC;AAE5C;uDACuD;AACvD,KAAK,UAAU,uBAAuB,CACpC,MAAiB,EACjB,KAAa,EACb,GAAW;IAEX,MAAM,WAAW,GAAG,MAAM,MAAM,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC;IACxD,IAAI,MAAkC,CAAC;IACvC,KAAK,MAAM,EAAE,IAAI,WAAW,EAAE,CAAC;QAC7B,IAAI,EAAE,CAAC,IAAI,KAAK,QAAQ,IAAI,EAAE,CAAC,MAAM,KAAK,SAAS;YAAE,SAAS;QAC9D,IAAI,CAAC,MAAM,IAAI,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,GAAG,MAAM,CAAC,UAAU,CAAC,OAAO,EAAE;YAAE,MAAM,GAAG,EAAE,CAAC;IACpF,CAAC;IACD,IAAI,CAAC,MAAM;QAAE,OAAO,IAAI,CAAC;IACzB,OAAO;QACL,GAAG,EAAE,MAAM,CAAC,GAAG;QACf,IAAI,EAAE,MAAM,CAAC,IAAI;QACjB,QAAQ,EAAE,MAAM,CAAC,QAAQ;QACzB,KAAK,EAAE,GAAG,GAAG,MAAM,CAAC,UAAU,CAAC,OAAO,EAAE;QACxC,cAAc,EAAE,MAAM,CAAC,eAAe,CAAC,CAAC,CAAC,GAAG,GAAG,MAAM,CAAC,eAAe,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,IAAI;KACvF,CAAC;AACJ,CAAC;AAED;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,MAAiB,EACjB,IAAmB,EACnB,MAAc,IAAI,CAAC,GAAG,EAAE;IAExB,OAAO,OAAO,CAAC,GAAG,CAChB,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,GAAG,EAAE,EAAE;QACrB,MAAM,YAAY,GAChB,GAAG,CAAC,MAAM,KAAK,SAAS,IAAI,GAAG,CAAC,MAAM,KAAK,WAAW;YACpD,CAAC,CAAC,MAAM,uBAAuB,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,EAAE,GAAG,CAAC;YACpD,CAAC,CAAC,IAAI,CAAC;QACX,OAAO;YACL,GAAG;YACH,KAAK,EAAE,GAAG,GAAG,GAAG,CAAC,SAAS,CAAC,OAAO,EAAE;YACpC,gBAAgB,EAAE,GAAG,CAAC,gBAAgB,IAAI,CAAC;YAC3C,YAAY;SACb,CAAC;IACJ,CAAC,CAAC,CACH,CAAC;AACJ,CAAC;AAED;;;mGAGmG;AACnG,MAAM,UAAU,WAAW,CACzB,QAAuB,EACvB,cAAsB,gBAAgB;IAEtC,OAAO,QAAQ,CAAC,MAAM,CACpB,CAAC,EAAE,EAAE,EAAE,CACL,EAAE,CAAC,YAAY,KAAK,IAAI;QACxB,EAAE,CAAC,YAAY,CAAC,KAAK,IAAI,WAAW;QACpC,CAAC,EAAE,CAAC,YAAY,CAAC,cAAc,KAAK,IAAI,IAAI,EAAE,CAAC,YAAY,CAAC,cAAc,IAAI,WAAW,CAAC,CAC7F,CAAC;AACJ,CAAC;AAED;;iDAEiD;AACjD,MAAM,UAAU,eAAe,CAAC,KAAa;IAC3C,MAAM,KAAK,GAAG,qBAAqB,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC;IACvD,IAAI,CAAC,KAAK;QAAE,OAAO,SAAS,CAAC;IAC7B,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IAC/B,MAAM,IAAI,GAAG,KAAK,CAAC,CAAC,CAAiC,CAAC;IACtD,MAAM,UAAU,GAAG,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,UAAU,EAAE,CAAC,IAAI,CAAC,CAAC;IACpF,OAAO,KAAK,GAAG,UAAU,CAAC;AAC5B,CAAC;AAED;;sBAEsB;AACtB,MAAM,UAAU,SAAS,CAAC,KAAa;IACrC,OAAO,CACL,OAAO,KAAK,+EAA+E;QAC3F,+BAA+B,KAAK,oCAAoC,KAAK,KAAK,CACnF,CAAC;AACJ,CAAC;AAED;;;+EAG+E;AAC/E,MAAM,UAAU,eAAe,CAAC,QAAuB;IACrD,IAAI,QAAQ,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,UAAU,CAAC;IAC7C,MAAM,IAAI,GAAG,QAAQ,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,KAAK,EAAE,gBAAgB,EAAE,YAAY,EAAE,EAAE,EAAE,CAAC;QAC5E,GAAG,CAAC,EAAE;QACN,GAAG,CAAC,QAAQ;QACZ,GAAG,CAAC,MAAM;QACV,UAAU,CAAC,KAAK,CAAC;QACjB,gBAAgB,GAAG,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC,GAAG;QACrD,YAAY;YACV,CAAC,CAAC,GAAG,UAAU,CAAC,YAAY,CAAC,KAAK,CAAC,aAAa,YAAY,CAAC,QAAQ,GACjE,YAAY,CAAC,cAAc,KAAK,IAAI;gBAClC,CAAC,CAAC,QAAQ,UAAU,CAAC,YAAY,CAAC,cAAc,CAAC,EAAE;gBACnD,CAAC,CAAC,EACN,GAAG;YACL,CAAC,CAAC,GAAG;KACR,CAAC,CAAC;IACH,OAAO,KAAK,CAAC,CAAC,KAAK,EAAE,UAAU,EAAE,QAAQ,EAAE,SAAS,EAAE,UAAU,EAAE,SAAS,CAAC,EAAE,IAAI,CAAC,CAAC;AACtF,CAAC;AAED,SAAS,KAAK,CAAC,OAAiB,EAAE,IAAgB;IAChD,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;IAClG,MAAM,GAAG,GAAG,CAAC,KAAe,EAAE,EAAE,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC1F,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACrD,CAAC;AAED;mFACmF;AACnF,SAAS,UAAU,CAAC,EAAU;IAC5B,MAAM,YAAY,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,EAAE,GAAG,IAAI,CAAC,CAAC,CAAC;IACxD,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,GAAG,MAAM,CAAC,CAAC;IAC/C,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,YAAY,GAAG,MAAM,CAAC,GAAG,IAAI,CAAC,CAAC;IACzD,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,YAAY,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;IACvD,MAAM,OAAO,GAAG,YAAY,GAAG,EAAE,CAAC;IAClC,IAAI,IAAI,GAAG,CAAC;QAAE,OAAO,GAAG,IAAI,IAAI,KAAK,GAAG,CAAC;IACzC,IAAI,KAAK,GAAG,CAAC;QAAE,OAAO,GAAG,KAAK,IAAI,OAAO,GAAG,CAAC;IAC7C,IAAI,OAAO,GAAG,CAAC;QAAE,OAAO,GAAG,OAAO,IAAI,OAAO,GAAG,CAAC;IACjD,OAAO,GAAG,OAAO,GAAG,CAAC;AACvB,CAAC"}
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
/** Keep in sync with this package's `version` in package.json. */
|
|
2
|
-
export declare const VERSION = "0.
|
|
2
|
+
export declare const VERSION = "0.19.0";
|
|
3
3
|
export { defineConfig, defaultAuthorize, resolveConfig } from './define_config.js';
|
|
4
4
|
export type { AuthorizeHook, DurableDashboardConfig, ResolvedDurableDashboardConfig, } from './define_config.js';
|
|
5
5
|
export { listRuns, getRun, retryRun, redispatchPendingRun, cancelRun, health, ok, } from './handlers.js';
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
/** Keep in sync with this package's `version` in package.json. */
|
|
2
|
-
export const VERSION = '0.
|
|
2
|
+
export const VERSION = '0.19.0';
|
|
3
3
|
export { defineConfig, defaultAuthorize, resolveConfig } from './define_config.js';
|
|
4
4
|
export { listRuns, getRun, retryRun, redispatchPendingRun, cancelRun, health, ok, } from './handlers.js';
|
|
5
5
|
export { renderDashboard } from './html.js';
|
package/dist/src/engine.d.ts
CHANGED
|
@@ -375,6 +375,20 @@ export declare class WorkflowEngine {
|
|
|
375
375
|
* durable. Per engine instance (see {@link QueueConfig}). Registering the same name replaces it.
|
|
376
376
|
*/
|
|
377
377
|
registerQueue(config: QueueConfig): void;
|
|
378
|
+
/** Last persisted beat per stepId (time + whether it carried progress) — the write throttle for
|
|
379
|
+
* {@link persistHeartbeat}. */
|
|
380
|
+
private readonly heartbeatPersists;
|
|
381
|
+
/**
|
|
382
|
+
* Persist a worker heartbeat on its step's checkpoint (see {@link StepCheckpoint.lastHeartbeatAt}),
|
|
383
|
+
* best-effort and THROTTLED per step (≥10s between writes) — beats may arrive every few seconds
|
|
384
|
+
* from a hot batch loop, and liveness observability must not turn into a per-item UPDATE storm.
|
|
385
|
+
* One exemption: a beat CARRYING progress punches through when the last persisted one had none —
|
|
386
|
+
* the automatic pickup beat lands first (no payload) and must not shadow the handler's first real
|
|
387
|
+
* progress for a whole window. Optional store capability: no `recordStepHeartbeat` → no-op.
|
|
388
|
+
* Failures are swallowed (a lost liveness update never harms the run). The throttle map is cleared
|
|
389
|
+
* wholesale past 1024 entries — a rare extra write, never a leak.
|
|
390
|
+
*/
|
|
391
|
+
private persistHeartbeat;
|
|
378
392
|
/**
|
|
379
393
|
* Declare this process a worker: flush consumer subscriptions parked by a transport whose
|
|
380
394
|
* consumption was deferred (see {@link Transport.deferConsumers}). The `durable:work` loop calls
|
package/dist/src/engine.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"engine.d.ts","sourceRoot":"","sources":["../../src/engine.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,gBAAgB,EAA4B,MAAM,gBAAgB,CAAC;AAYjF,OAAO,EAAY,KAAK,YAAY,EAAE,MAAM,eAAe,CAAC;AAS5D,OAAO,EAAqB,KAAK,gBAAgB,EAAE,MAAM,yBAAyB,CAAC;AAGnF,OAAO,KAAK,EACV,YAAY,EAEZ,cAAc,EACd,WAAW,EACX,cAAc,EAEd,aAAa,EACb,QAAQ,EACR,SAAS,EAET,gBAAgB,EAEhB,UAAU,EACV,cAAc,EAEd,SAAS,EAET,eAAe,EAIf,SAAS,EACT,YAAY,EACZ,eAAe,EAEf,WAAW,EAEX,gBAAgB,EAChB,WAAW,EAEZ,MAAM,iBAAiB,CAAC;AAGzB,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AAE9C,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,gBAAgB,CAAC;AAWxD,OAAO,EACL,KAAK,aAAa,EAClB,KAAK,eAAe,EACpB,KAAK,WAAW,EAEjB,MAAM,mBAAmB,CAAC;AAE3B,KAAK,UAAU,GAAG,CAAC,GAAG,EAAE,WAAW,EAAE,KAAK,EAAE,OAAO,KAAK,OAAO,CAAC,OAAO,CAAC,CAAC;AAEzE,gDAAgD;AAChD,MAAM,WAAW,YAAY;IAC3B,gGAAgG;IAChG,IAAI,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;IAC5B,0FAA0F;IAC1F,gBAAgB,CAAC,EAAE,gBAAgB,GAAG,SAAS,CAAC;IAChD;;;OAGG;IACH,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B;;;;;OAKG;IACH,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAChC;AAED;;;;;GAKG;AACH,MAAM,WAAW,eAAe;IAC9B,4DAA4D;IAC5D,GAAG,EAAE,CAAC,KAAK,EAAE,OAAO,KAAK,MAAM,CAAC;IAChC,gEAAgE;IAChE,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;;;;;OAKG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;CACxB;AAyED,MAAM,WAAW,kBAAkB;IACjC,KAAK,EAAE,UAAU,CAAC;IAClB,2FAA2F;IAC3F,SAAS,CAAC,EAAE,SAAS,GAAG,SAAS,CAAC;IAClC;;;;OAIG;IACH,UAAU,CAAC,EAAE,cAAc,EAAE,GAAG,SAAS,CAAC;IAC1C;;;;OAIG;IACH,YAAY,CAAC,EAAE,YAAY,GAAG,SAAS,CAAC;IACxC,oEAAoE;IACpE,KAAK,CAAC,EAAE,CAAC,MAAM,MAAM,CAAC,GAAG,SAAS,CAAC;IACnC,6FAA6F;IAC7F,UAAU,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAChC;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC/B,mGAAmG;IACnG,OAAO,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC7B;;;;OAIG;IACH,SAAS,CAAC,EAAE,gBAAgB,GAAG,SAAS,CAAC;IACzC;;;;OAIG;IACH,mBAAmB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACzC;;;;;;;;;;;OAWG;IACH,WAAW,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACjC;;;;;;;;;;;;;OAaG;IACH,kBAAkB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACxC;;;;OAIG;IACH,mBAAmB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACzC;;;;OAIG;IACH,UAAU,CAAC,EAAE,CAAC,CAAC,KAAK,EAAE,MAAM,KAAK,MAAM,CAAC,GAAG,SAAS,CAAC;IACrD;;;;OAIG;IACH,WAAW,CAAC,EAAE,CAAC,MAAM,MAAM,GAAG,SAAS,CAAC,GAAG,SAAS,CAAC;IACrD;;;;;;;;;;;OAWG;IACH,OAAO,CAAC,EAAE,CAAC,MAAM,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,SAAS,CAAC,GAAG,SAAS,CAAC;IAClE;;;;;;;;;OASG;IACH,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,SAAS,EAAE,EAAE,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,GAAG,SAAS,CAAC;IAC9F;;;OAGG;IACH,mBAAmB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACzC;;;;;;;OAOG;IACH,cAAc,CAAC,EAAE,OAAO,GAAG,SAAS,CAAC;IACrC;;;;;OAKG;IACH,aAAa,CAAC,EAAE,aAAa,GAAG,SAAS,CAAC;IAC1C;;;;OAIG;IACH,wBAAwB,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;IAChD;;;;;OAKG;IACH,oBAAoB,CAAC,EAAE;QAAE,OAAO,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,CAAA;KAAE,GAAG,SAAS,CAAC;IAChF;;;;;OAKG;IACH,aAAa,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CACpC;AAED,6FAA6F;AAC7F,qBAAa,iBAAkB,SAAQ,KAAK;;CAK3C;AAED;;;;;GAKG;AACH,qBAAa,cAAc;;IACzB,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAa;IACnC,6EAA6E;IAC7E,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAgB;IACrC;kFAC8E;IAC9E,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAwB;IAC1D,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAC,CAA2B;IACzD,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAe;IACrC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAS;IACpC,6FAA6F;IAC7F,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAS;IACjC,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAC,CAAqB;IAC1D,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAqB;IACjD,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAqB;IACxD,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAS;IAC7C,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAA0C;IACtE,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAyC;IACtE,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,CAA0D;IACnF,4IAA4I;IAC5I,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAsE;IAChG,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAS;IAC7C,wHAAwH;IACxH,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAU;IACzC,gGAAgG;IAChG,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAgB;IAC9C;;;;;;OAMG;IACH,OAAO,CAAC,QAAQ,CAAC,qBAAqB,CAAU;IAChD;0GACsG;IACtG,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAmB;IAChD,sGAAsG;IACtG,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAS;IAEvC,mGAAmG;IACnG,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAgB;IAC3C,0FAA0F;IAC1F,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAyC;IACnE,kFAAkF;IAClF,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAyC;IAChE,wFAAwF;IACxF,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAkC;IAChE,wFAAwF;IACxF,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAW;IAGpC,oGAAoG;IACpG,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAoB;IACjD,wEAAwE;IACxE,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAoC;IAC5D,sFAAsF;IACtF,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAiC;IACjE,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA6B;IACvD,gGAAgG;IAChG,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAyB;IACtD,mGAAmG;IACnG,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAsC;IACtE,gGAAgG;IAChG,OAAO,CAAC,QAAQ,CAAC,iBAAiB,CAAsC;IACxE,yFAAyF;IACzF,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAyC;IACvE,6EAA6E;IAC7E,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAsC;IACvE,wFAAwF;IACxF,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAqB;IACrD,mFAAmF;IACnF,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAmB;IAC7C,iGAAiG;IACjG,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAkC;IAC/D,gGAAgG;IAChG,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA6B;IACvD,0FAA0F;IAC1F,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAiC;IAC1D;;;;;;;;;OASG;IACH,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA+B;IAC1D,OAAO,CAAC,QAAQ,CAAS;gBAEb,IAAI,EAAE,kBAAkB;IAwHpC;;;;OAIG;IACH,cAAc,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,YAAY,CAAC,CAAC,CAAC,GAAG,IAAI;IAI9D;;;;;;OAMG;IACH,iBAAiB,CAAC,SAAS,EAAE,SAAS,iBAAiB,EAAE,GAAG,IAAI;IAchE,uGAAuG;IACvG,IAAI,mBAAmB,IAAI,SAAS,iBAAiB,EAAE,CAEtD;IAED,wFAAwF;IACxF,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,EAAE,GAAG,CAAC,EAAE,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC;IAIjF,yGAAyG;IACzG,cAAc,CAAC,CAAC,GAAG,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,CAAC,GAAG,SAAS,CAAC;IAI9E;;;;OAIG;IACH,UAAU,CAAC,QAAQ,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,IAAI,GAAG,MAAM,IAAI;IAKzD,mGAAmG;IACnG,OAAO,CAAC,eAAe;IAUvB;;;;OAIG;IACH,QAAQ,CACN,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,MAAM,EACf,EAAE,EAAE,UAAU,EACd,IAAI,CAAC,EAAE;QACL,IAAI,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;QAC5B,SAAS,CAAC,EAAE,eAAe,GAAG,SAAS,CAAC;QACxC,gBAAgB,CAAC,EAAE,MAAM,GAAG,MAAM,GAAG,SAAS,CAAC;QAC/C,aAAa,CAAC,EAAE,CAAC,CAAC,KAAK,EAAE,OAAO,KAAK,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,GAAG,SAAS,CAAC;QACvE,OAAO,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;QAC/B,UAAU,CAAC,EAAE,gBAAgB,GAAG,SAAS,CAAC;QAC1C,QAAQ,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;KACjC,GACA,IAAI;IAwBP;;;;;OAKG;IACH,cAAc,CACZ,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,MAAM,EACf,IAAI,EAAE;QACJ,KAAK,EAAE,MAAM,CAAC;QACd,QAAQ,EAAE,gBAAgB,CAAC;QAC3B,IAAI,CAAC,EAAE,MAAM,EAAE,CAAC;QAChB,SAAS,CAAC,EAAE,eAAe,CAAC;QAC5B,gBAAgB,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;QACnC,aAAa,CAAC,EAAE,CAAC,KAAK,EAAE,OAAO,KAAK,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;QACzD,QAAQ,CAAC,EAAE,MAAM,EAAE,CAAC;KACrB,GACA,IAAI;IAqBP;;;;;;;;OAQG;YACW,wBAAwB;IAYtC;;;;OAIG;IACH,aAAa,CAAC,MAAM,EAAE,WAAW,GAAG,IAAI;IAIxC;;;;;;OAMG;IACH,cAAc,IAAI,IAAI;IAItB,sEAAsE;IACtE,SAAS,CAAC,QAAQ,EAAE,cAAc,GAAG,MAAM,IAAI;IAK/C;;;;;OAKG;IACH,GAAG,CAAC,WAAW,EAAE,eAAe,GAAG,MAAM,IAAI;IAQ7C;;;;;;OAMG;IACH,OAAO,CAAC,aAAa;IAarB;;;;OAIG;IACH,QAAQ,CAAC,QAAQ,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,IAAI,GAAG,MAAM,IAAI;IAKvD;;;;;;OAMG;IACH,MAAM,CAAC,QAAQ,EAAE,CAAC,GAAG,EAAE,WAAW,KAAK,IAAI,GAAG,MAAM,IAAI;IAKxD,OAAO,CAAC,UAAU;IAUlB;0FACsF;IACtF,OAAO,CAAC,IAAI;IAYZ;wGACoG;IACpG,OAAO,CAAC,OAAO;IAUT,KAAK,CAAC,CAAC,SAAS,aAAa,EACjC,QAAQ,EAAE,CAAC,EACX,KAAK,EAAE,eAAe,CAAC,CAAC,CAAC,EACzB,KAAK,EAAE,MAAM,EACb,IAAI,CAAC,EAAE,YAAY,GAClB,OAAO,CAAC,SAAS,CAAC;IACf,KAAK,CAAC,MAAM,EAChB,QAAQ,EAAE,MAAM,EAChB,KAAK,EAAE,MAAM,EACb,KAAK,EAAE,MAAM,EACb,IAAI,CAAC,EAAE,YAAY,GAClB,OAAO,CAAC,SAAS,CAAC;IAarB;;;;;;OAMG;YACW,SAAS;IAoEvB;;;;;;;;;;;;;;;;;;OAkBG;IACH,OAAO,CAAC,UAAU;IAmClB,mGAAmG;IACnG,MAAM,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,WAAW,GAAG,IAAI,CAAC;IAIlD;;;;OAIG;IACH,QAAQ,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC;IAIjD,yGAAyG;IACzG,eAAe,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC;IAIzD;;;;;OAKG;IACG,SAAS,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;IAYzC,MAAM,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,CAAC;IAmC/C,qEAAqE;IACrE,OAAO,CAAC,KAAK;IAMb;;;;;;OAMG;IACH,OAAO,CAAC,WAAW;IAMnB;;;;;;;;;;;;OAYG;IACG,KAAK,CAAC,SAAS,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAmB9C;;;;OAIG;IACG,aAAa,CAAC,GAAG,GAAE,MAAqB,GAAG,OAAO,CAAC,IAAI,CAAC;IA+B9D;;;OAGG;IACG,iBAAiB,CAAC,KAAK,GAAE,MAAqB,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IA6B3E;;;;OAIG;YACW,aAAa;IAwB3B;;;;;;OAMG;IACG,eAAe,CAAC,KAAK,GAAE,MAAqB,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAMzE;;;;;OAKG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAKtD;;;;;;OAMG;YACW,cAAc;IAiB5B;;;;;;OAMG;IACG,OAAO,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IA+E5E;;;;;;;;;OASG;IACG,iBAAiB,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,CAAC,SAAS,GAAG;QAAE,YAAY,EAAE,MAAM,CAAA;KAAE,CAAC,GAAG,IAAI,CAAC;IAgC9F;;;;OAIG;IACG,cAAc,CAClB,KAAK,EAAE,MAAM,EACb,KAAK,EAAE,OAAO,EACd,QAAQ,CAAC,EAAE,MAAM,GAChB,OAAO,CAAC;QAAE,KAAK,EAAE,MAAM,CAAA;KAAE,GAAG,IAAI,CAAC;IAUpC;;;;;OAKG;IACG,UAAU,CAAC,KAAK,GAAE,MAAqB,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAMpE;;;;;;;;;;OAUG;IACH,UAAU,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;QAAE,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,OAAO,CAAA;KAAE,GAAG,OAAO,CAAC,SAAS,CAAC;IAiDhG;;;OAGG;YACW,YAAY;IAsB1B;;;OAGG;IACH;;;;;;;;;;;;;;;;;;OAkBG;IACG,YAAY,CAChB,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,OAAO,EAChB,IAAI,CAAC,EAAE;QAAE,EAAE,CAAC,EAAE,MAAM,CAAC;QAAC,MAAM,CAAC,EAAE,OAAO,CAAA;KAAE,GACvC,OAAO,CAAC,MAAM,CAAC;IAwDlB;;;;OAIG;YACW,aAAa;IAkB3B;;;;;;;;;;;OAWG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAqBxE;;;;;;;OAOG;IACG,eAAe,CACnB,QAAQ,EAAE,WAAW,EACrB,KAAK,EAAE,OAAO,EACd,KAAK,EAAE,MAAM,EACb,MAAM,EAAE;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,CAAA;KAAE,EAC5C,IAAI,CAAC,EAAE,YAAY,GAClB,OAAO,CAAC;QAAE,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAQ7B;;;;OAIG;IACG,YAAY,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAO3F,6FAA6F;IACvF,QAAQ,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAIrF;;;;OAIG;IACH,OAAO,CAAC,YAAY;IAepB;;;;;;;;;;;OAWG;IACH,OAAO,CAAC,kBAAkB;IAkB1B;;;;;OAKG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;QAAE,UAAU,CAAC,EAAE,OAAO,CAAA;KAAE,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IA4DvF;;;;;;;;OAQG;IACG,WAAW,CACf,MAAM,EAAE,IAAI,CAAC,QAAQ,EAAE,OAAO,GAAG,QAAQ,CAAC,EAC1C,IAAI,CAAC,EAAE;QAAE,UAAU,CAAC,EAAE,OAAO,CAAA;KAAE,GAC9B,OAAO,CAAC,SAAS,EAAE,CAAC;IAUvB;;;;;OAKG;YACW,cAAc;IAS5B;;;;;;;;;;;OAWG;IACG,cAAc,CAAC,WAAW,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC;IAoB5D;;;yGAGqG;IACrG,WAAW,CAAC,KAAK,GAAE,MAAM,EAAO,GAAG,MAAM,EAAE;IAQ3C;;;;gEAI4D;IACtD,YAAY,CAAC,KAAK,GAAE,MAAM,EAAO,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC;IAUhE;;;;;;;;OAQG;YACW,gBAAgB;IAsD9B;;;;;;OAMG;YACW,SAAS;IA4BvB,oGAAoG;YACtF,YAAY;IA8B1B,oGAAoG;YACtF,QAAQ;YA8BR,OAAO;IAiDrB;;;;;OAKG;IACH;;;;;;OAMG;YACW,SAAS;IAuDvB;;;;;OAKG;IACH,OAAO,CAAC,eAAe;YAKT,kBAAkB;IA2FhC,gGAAgG;YAClF,aAAa;IA0C3B;oGACgG;YAClF,aAAa;IA6K3B,6DAA6D;YAC/C,YAAY;IA2H1B;;;;;;OAMG;YACW,eAAe;IAmC7B;;;;;;OAMG;YACW,oBAAoB;IAkClC;;wDAEoD;IACpD,OAAO,CAAC,UAAU;IAgDlB;;;;OAIG;IACG,QAAQ,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAOxD;;;;OAIG;IACG,QAAQ,CAAC,MAAM,GAAG,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,GAAG,SAAS,CAAC;IAgBzF;;;;OAIG;IACH,uBAAuB,CAAC,IAAI,EAC1B,QAAQ,EAAE,MAAM,EAChB,IAAI,EAAE,MAAM,EACZ,QAAQ,EAAE,eAAe,CAAC,IAAI,CAAC,GAC9B,IAAI;IAIP;;;;;OAKG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,OAAO,GAAG,OAAO,CAAC,YAAY,CAAC;IAoB9E;;;0FAGsF;YACxE,kBAAkB;IAUhC;;;;;;OAMG;YACW,SAAS;IASvB;;;6EAGyE;YAC3D,cAAc;IAK5B;;;;;;OAMG;YACW,WAAW;IA4BzB;;qGAEiG;YACnF,cAAc;YAUd,UAAU;IA+JxB;;;;OAIG;IACH,OAAO,CAAC,kBAAkB;IA4B1B;;;OAGG;YACW,oBAAoB;IA8ClC;;;;OAIG;IACH,OAAO,CAAC,gBAAgB;IAQxB,oFAAoF;YACtE,gBAAgB;IAO9B,mGAAmG;YACrF,kBAAkB;IAyEhC;;;;OAIG;IACH,OAAO,CAAC,kBAAkB;CAoC3B;AAED,8EAA8E;AAC9E,qBAAa,eAAgB,SAAQ,KAAK;IACxC,QAAQ,CAAC,SAAS,CAAC,EAAE,SAAS,GAAG,SAAS,CAAC;gBAC/B,SAAS,CAAC,EAAE,SAAS;CAKlC"}
|
|
1
|
+
{"version":3,"file":"engine.d.ts","sourceRoot":"","sources":["../../src/engine.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,gBAAgB,EAA4B,MAAM,gBAAgB,CAAC;AAYjF,OAAO,EAAY,KAAK,YAAY,EAAE,MAAM,eAAe,CAAC;AAS5D,OAAO,EAAqB,KAAK,gBAAgB,EAAE,MAAM,yBAAyB,CAAC;AAGnF,OAAO,KAAK,EACV,YAAY,EAEZ,cAAc,EACd,WAAW,EAEX,cAAc,EAEd,aAAa,EACb,QAAQ,EACR,SAAS,EAET,gBAAgB,EAEhB,UAAU,EACV,cAAc,EAEd,SAAS,EAET,eAAe,EAIf,SAAS,EACT,YAAY,EACZ,eAAe,EAEf,WAAW,EAEX,gBAAgB,EAChB,WAAW,EAEZ,MAAM,iBAAiB,CAAC;AAGzB,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AAE9C,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,gBAAgB,CAAC;AAWxD,OAAO,EACL,KAAK,aAAa,EAClB,KAAK,eAAe,EACpB,KAAK,WAAW,EAEjB,MAAM,mBAAmB,CAAC;AAE3B,KAAK,UAAU,GAAG,CAAC,GAAG,EAAE,WAAW,EAAE,KAAK,EAAE,OAAO,KAAK,OAAO,CAAC,OAAO,CAAC,CAAC;AAEzE,gDAAgD;AAChD,MAAM,WAAW,YAAY;IAC3B,gGAAgG;IAChG,IAAI,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;IAC5B,0FAA0F;IAC1F,gBAAgB,CAAC,EAAE,gBAAgB,GAAG,SAAS,CAAC;IAChD;;;OAGG;IACH,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B;;;;;OAKG;IACH,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAChC;AAED;;;;;GAKG;AACH,MAAM,WAAW,eAAe;IAC9B,4DAA4D;IAC5D,GAAG,EAAE,CAAC,KAAK,EAAE,OAAO,KAAK,MAAM,CAAC;IAChC,gEAAgE;IAChE,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;;;;;OAKG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;CACxB;AAyED,MAAM,WAAW,kBAAkB;IACjC,KAAK,EAAE,UAAU,CAAC;IAClB,2FAA2F;IAC3F,SAAS,CAAC,EAAE,SAAS,GAAG,SAAS,CAAC;IAClC;;;;OAIG;IACH,UAAU,CAAC,EAAE,cAAc,EAAE,GAAG,SAAS,CAAC;IAC1C;;;;OAIG;IACH,YAAY,CAAC,EAAE,YAAY,GAAG,SAAS,CAAC;IACxC,oEAAoE;IACpE,KAAK,CAAC,EAAE,CAAC,MAAM,MAAM,CAAC,GAAG,SAAS,CAAC;IACnC,6FAA6F;IAC7F,UAAU,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAChC;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC/B,mGAAmG;IACnG,OAAO,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC7B;;;;OAIG;IACH,SAAS,CAAC,EAAE,gBAAgB,GAAG,SAAS,CAAC;IACzC;;;;OAIG;IACH,mBAAmB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACzC;;;;;;;;;;;OAWG;IACH,WAAW,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACjC;;;;;;;;;;;;;OAaG;IACH,kBAAkB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACxC;;;;OAIG;IACH,mBAAmB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACzC;;;;OAIG;IACH,UAAU,CAAC,EAAE,CAAC,CAAC,KAAK,EAAE,MAAM,KAAK,MAAM,CAAC,GAAG,SAAS,CAAC;IACrD;;;;OAIG;IACH,WAAW,CAAC,EAAE,CAAC,MAAM,MAAM,GAAG,SAAS,CAAC,GAAG,SAAS,CAAC;IACrD;;;;;;;;;;;OAWG;IACH,OAAO,CAAC,EAAE,CAAC,MAAM,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,SAAS,CAAC,GAAG,SAAS,CAAC;IAClE;;;;;;;;;OASG;IACH,SAAS,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,SAAS,EAAE,EAAE,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC,GAAG,SAAS,CAAC;IAC9F;;;OAGG;IACH,mBAAmB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACzC;;;;;;;OAOG;IACH,cAAc,CAAC,EAAE,OAAO,GAAG,SAAS,CAAC;IACrC;;;;;OAKG;IACH,aAAa,CAAC,EAAE,aAAa,GAAG,SAAS,CAAC;IAC1C;;;;OAIG;IACH,wBAAwB,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;IAChD;;;;;OAKG;IACH,oBAAoB,CAAC,EAAE;QAAE,OAAO,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,CAAA;KAAE,GAAG,SAAS,CAAC;IAChF;;;;;OAKG;IACH,aAAa,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CACpC;AAED,6FAA6F;AAC7F,qBAAa,iBAAkB,SAAQ,KAAK;;CAK3C;AAED;;;;;GAKG;AACH,qBAAa,cAAc;;IACzB,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAa;IACnC,6EAA6E;IAC7E,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAgB;IACrC;kFAC8E;IAC9E,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAwB;IAC1D,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAC,CAA2B;IACzD,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAe;IACrC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAS;IACpC,6FAA6F;IAC7F,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAS;IACjC,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAC,CAAqB;IAC1D,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAqB;IACjD,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAqB;IACxD,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAS;IAC7C,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAA0C;IACtE,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAyC;IACtE,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,CAA0D;IACnF,4IAA4I;IAC5I,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAsE;IAChG,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAS;IAC7C,wHAAwH;IACxH,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAU;IACzC,gGAAgG;IAChG,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAgB;IAC9C;;;;;;OAMG;IACH,OAAO,CAAC,QAAQ,CAAC,qBAAqB,CAAU;IAChD;0GACsG;IACtG,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAmB;IAChD,sGAAsG;IACtG,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAS;IAEvC,mGAAmG;IACnG,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAgB;IAC3C,0FAA0F;IAC1F,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAyC;IACnE,kFAAkF;IAClF,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAyC;IAChE,wFAAwF;IACxF,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAkC;IAChE,wFAAwF;IACxF,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAW;IAGpC,oGAAoG;IACpG,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAoB;IACjD,wEAAwE;IACxE,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAoC;IAC5D,sFAAsF;IACtF,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAiC;IACjE,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA6B;IACvD,gGAAgG;IAChG,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAyB;IACtD,mGAAmG;IACnG,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAsC;IACtE,gGAAgG;IAChG,OAAO,CAAC,QAAQ,CAAC,iBAAiB,CAAsC;IACxE,yFAAyF;IACzF,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAyC;IACvE,6EAA6E;IAC7E,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAsC;IACvE,wFAAwF;IACxF,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAqB;IACrD,mFAAmF;IACnF,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAmB;IAC7C,iGAAiG;IACjG,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAkC;IAC/D,gGAAgG;IAChG,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA6B;IACvD,0FAA0F;IAC1F,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAiC;IAC1D;;;;;;;;;OASG;IACH,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA+B;IAC1D,OAAO,CAAC,QAAQ,CAAS;gBAEb,IAAI,EAAE,kBAAkB;IA0HpC;;;;OAIG;IACH,cAAc,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,YAAY,CAAC,CAAC,CAAC,GAAG,IAAI;IAI9D;;;;;;OAMG;IACH,iBAAiB,CAAC,SAAS,EAAE,SAAS,iBAAiB,EAAE,GAAG,IAAI;IAchE,uGAAuG;IACvG,IAAI,mBAAmB,IAAI,SAAS,iBAAiB,EAAE,CAEtD;IAED,wFAAwF;IACxF,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,EAAE,GAAG,CAAC,EAAE,OAAO,GAAG,OAAO,CAAC,IAAI,CAAC;IAIjF,yGAAyG;IACzG,cAAc,CAAC,CAAC,GAAG,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,CAAC,GAAG,SAAS,CAAC;IAI9E;;;;OAIG;IACH,UAAU,CAAC,QAAQ,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,IAAI,GAAG,MAAM,IAAI;IAKzD,mGAAmG;IACnG,OAAO,CAAC,eAAe;IAUvB;;;;OAIG;IACH,QAAQ,CACN,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,MAAM,EACf,EAAE,EAAE,UAAU,EACd,IAAI,CAAC,EAAE;QACL,IAAI,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;QAC5B,SAAS,CAAC,EAAE,eAAe,GAAG,SAAS,CAAC;QACxC,gBAAgB,CAAC,EAAE,MAAM,GAAG,MAAM,GAAG,SAAS,CAAC;QAC/C,aAAa,CAAC,EAAE,CAAC,CAAC,KAAK,EAAE,OAAO,KAAK,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,GAAG,SAAS,CAAC;QACvE,OAAO,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;QAC/B,UAAU,CAAC,EAAE,gBAAgB,GAAG,SAAS,CAAC;QAC1C,QAAQ,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,CAAC;KACjC,GACA,IAAI;IAwBP;;;;;OAKG;IACH,cAAc,CACZ,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,MAAM,EACf,IAAI,EAAE;QACJ,KAAK,EAAE,MAAM,CAAC;QACd,QAAQ,EAAE,gBAAgB,CAAC;QAC3B,IAAI,CAAC,EAAE,MAAM,EAAE,CAAC;QAChB,SAAS,CAAC,EAAE,eAAe,CAAC;QAC5B,gBAAgB,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;QACnC,aAAa,CAAC,EAAE,CAAC,KAAK,EAAE,OAAO,KAAK,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;QACzD,QAAQ,CAAC,EAAE,MAAM,EAAE,CAAC;KACrB,GACA,IAAI;IAqBP;;;;;;;;OAQG;YACW,wBAAwB;IAYtC;;;;OAIG;IACH,aAAa,CAAC,MAAM,EAAE,WAAW,GAAG,IAAI;IAIxC;oCACgC;IAChC,OAAO,CAAC,QAAQ,CAAC,iBAAiB,CAA2D;IAE7F;;;;;;;;;OASG;IACH,OAAO,CAAC,gBAAgB;IAcxB;;;;;;OAMG;IACH,cAAc,IAAI,IAAI;IAItB,sEAAsE;IACtE,SAAS,CAAC,QAAQ,EAAE,cAAc,GAAG,MAAM,IAAI;IAK/C;;;;;OAKG;IACH,GAAG,CAAC,WAAW,EAAE,eAAe,GAAG,MAAM,IAAI;IAQ7C;;;;;;OAMG;IACH,OAAO,CAAC,aAAa;IAarB;;;;OAIG;IACH,QAAQ,CAAC,QAAQ,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,IAAI,GAAG,MAAM,IAAI;IAKvD;;;;;;OAMG;IACH,MAAM,CAAC,QAAQ,EAAE,CAAC,GAAG,EAAE,WAAW,KAAK,IAAI,GAAG,MAAM,IAAI;IAKxD,OAAO,CAAC,UAAU;IAUlB;0FACsF;IACtF,OAAO,CAAC,IAAI;IAYZ;wGACoG;IACpG,OAAO,CAAC,OAAO;IAUT,KAAK,CAAC,CAAC,SAAS,aAAa,EACjC,QAAQ,EAAE,CAAC,EACX,KAAK,EAAE,eAAe,CAAC,CAAC,CAAC,EACzB,KAAK,EAAE,MAAM,EACb,IAAI,CAAC,EAAE,YAAY,GAClB,OAAO,CAAC,SAAS,CAAC;IACf,KAAK,CAAC,MAAM,EAChB,QAAQ,EAAE,MAAM,EAChB,KAAK,EAAE,MAAM,EACb,KAAK,EAAE,MAAM,EACb,IAAI,CAAC,EAAE,YAAY,GAClB,OAAO,CAAC,SAAS,CAAC;IAarB;;;;;;OAMG;YACW,SAAS;IAoEvB;;;;;;;;;;;;;;;;;;OAkBG;IACH,OAAO,CAAC,UAAU;IAmClB,mGAAmG;IACnG,MAAM,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,WAAW,GAAG,IAAI,CAAC;IAIlD;;;;OAIG;IACH,QAAQ,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC;IAIjD,yGAAyG;IACzG,eAAe,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC;IAIzD;;;;;OAKG;IACG,SAAS,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;IAYzC,MAAM,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,CAAC;IAmC/C,qEAAqE;IACrE,OAAO,CAAC,KAAK;IAMb;;;;;;OAMG;IACH,OAAO,CAAC,WAAW;IAMnB;;;;;;;;;;;;OAYG;IACG,KAAK,CAAC,SAAS,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;IAmB9C;;;;OAIG;IACG,aAAa,CAAC,GAAG,GAAE,MAAqB,GAAG,OAAO,CAAC,IAAI,CAAC;IA+B9D;;;OAGG;IACG,iBAAiB,CAAC,KAAK,GAAE,MAAqB,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IA6B3E;;;;OAIG;YACW,aAAa;IAwB3B;;;;;;OAMG;IACG,eAAe,CAAC,KAAK,GAAE,MAAqB,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAMzE;;;;;OAKG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAKtD;;;;;;OAMG;YACW,cAAc;IAiB5B;;;;;;OAMG;IACG,OAAO,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IA+E5E;;;;;;;;;OASG;IACG,iBAAiB,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,CAAC,SAAS,GAAG;QAAE,YAAY,EAAE,MAAM,CAAA;KAAE,CAAC,GAAG,IAAI,CAAC;IAgC9F;;;;OAIG;IACG,cAAc,CAClB,KAAK,EAAE,MAAM,EACb,KAAK,EAAE,OAAO,EACd,QAAQ,CAAC,EAAE,MAAM,GAChB,OAAO,CAAC;QAAE,KAAK,EAAE,MAAM,CAAA;KAAE,GAAG,IAAI,CAAC;IAUpC;;;;;OAKG;IACG,UAAU,CAAC,KAAK,GAAE,MAAqB,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC;IAMpE;;;;;;;;;;OAUG;IACH,UAAU,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;QAAE,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,OAAO,CAAA;KAAE,GAAG,OAAO,CAAC,SAAS,CAAC;IAiDhG;;;OAGG;YACW,YAAY;IAsB1B;;;OAGG;IACH;;;;;;;;;;;;;;;;;;OAkBG;IACG,YAAY,CAChB,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,OAAO,EAChB,IAAI,CAAC,EAAE;QAAE,EAAE,CAAC,EAAE,MAAM,CAAC;QAAC,MAAM,CAAC,EAAE,OAAO,CAAA;KAAE,GACvC,OAAO,CAAC,MAAM,CAAC;IAwDlB;;;;OAIG;YACW,aAAa;IAkB3B;;;;;;;;;;;OAWG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAqBxE;;;;;;;OAOG;IACG,eAAe,CACnB,QAAQ,EAAE,WAAW,EACrB,KAAK,EAAE,OAAO,EACd,KAAK,EAAE,MAAM,EACb,MAAM,EAAE;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,CAAA;KAAE,EAC5C,IAAI,CAAC,EAAE,YAAY,GAClB,OAAO,CAAC;QAAE,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAQ7B;;;;OAIG;IACG,YAAY,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAO3F,6FAA6F;IACvF,QAAQ,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAIrF;;;;OAIG;IACH,OAAO,CAAC,YAAY;IAepB;;;;;;;;;;;OAWG;IACH,OAAO,CAAC,kBAAkB;IAkB1B;;;;;OAKG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;QAAE,UAAU,CAAC,EAAE,OAAO,CAAA;KAAE,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IA4DvF;;;;;;;;OAQG;IACG,WAAW,CACf,MAAM,EAAE,IAAI,CAAC,QAAQ,EAAE,OAAO,GAAG,QAAQ,CAAC,EAC1C,IAAI,CAAC,EAAE;QAAE,UAAU,CAAC,EAAE,OAAO,CAAA;KAAE,GAC9B,OAAO,CAAC,SAAS,EAAE,CAAC;IAUvB;;;;;OAKG;YACW,cAAc;IAS5B;;;;;;;;;;;OAWG;IACG,cAAc,CAAC,WAAW,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC;IAoB5D;;;yGAGqG;IACrG,WAAW,CAAC,KAAK,GAAE,MAAM,EAAO,GAAG,MAAM,EAAE;IAQ3C;;;;gEAI4D;IACtD,YAAY,CAAC,KAAK,GAAE,MAAM,EAAO,GAAG,OAAO,CAAC,WAAW,EAAE,CAAC;IAUhE;;;;;;;;OAQG;YACW,gBAAgB;IAsD9B;;;;;;OAMG;YACW,SAAS;IA4BvB,oGAAoG;YACtF,YAAY;IA8B1B,oGAAoG;YACtF,QAAQ;YA8BR,OAAO;IAiDrB;;;;;OAKG;IACH;;;;;;OAMG;YACW,SAAS;IAuDvB;;;;;OAKG;IACH,OAAO,CAAC,eAAe;YAKT,kBAAkB;IA2FhC,gGAAgG;YAClF,aAAa;IA0C3B;oGACgG;YAClF,aAAa;IA6K3B,6DAA6D;YAC/C,YAAY;IA2H1B;;;;;;OAMG;YACW,eAAe;IAmC7B;;;;;;OAMG;YACW,oBAAoB;IAkClC;;wDAEoD;IACpD,OAAO,CAAC,UAAU;IAgDlB;;;;OAIG;IACG,QAAQ,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,GAAG,IAAI,CAAC;IAOxD;;;;OAIG;IACG,QAAQ,CAAC,MAAM,GAAG,OAAO,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,GAAG,SAAS,CAAC;IAgBzF;;;;OAIG;IACH,uBAAuB,CAAC,IAAI,EAC1B,QAAQ,EAAE,MAAM,EAChB,IAAI,EAAE,MAAM,EACZ,QAAQ,EAAE,eAAe,CAAC,IAAI,CAAC,GAC9B,IAAI;IAIP;;;;;OAKG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,OAAO,GAAG,OAAO,CAAC,YAAY,CAAC;IAoB9E;;;0FAGsF;YACxE,kBAAkB;IAUhC;;;;;;OAMG;YACW,SAAS;IASvB;;;6EAGyE;YAC3D,cAAc;IAK5B;;;;;;OAMG;YACW,WAAW;IA4BzB;;qGAEiG;YACnF,cAAc;YAUd,UAAU;IA+JxB;;;;OAIG;IACH,OAAO,CAAC,kBAAkB;IA4B1B;;;OAGG;YACW,oBAAoB;IA8ClC;;;;OAIG;IACH,OAAO,CAAC,gBAAgB;IAQxB,oFAAoF;YACtE,gBAAgB;IAO9B,mGAAmG;YACrF,kBAAkB;IA8FhC;;;;OAIG;IACH,OAAO,CAAC,kBAAkB;CA+C3B;AAED,8EAA8E;AAC9E,qBAAa,eAAgB,SAAQ,KAAK;IACxC,QAAQ,CAAC,SAAS,CAAC,EAAE,SAAS,GAAG,SAAS,CAAC;gBAC/B,SAAS,CAAC,EAAE,SAAS;CAKlC"}
|
package/dist/src/engine.js
CHANGED
|
@@ -222,9 +222,11 @@ export class WorkflowEngine {
|
|
|
222
222
|
// instance) → complete the checkpoint and resume the run here.
|
|
223
223
|
await this.completeRemoteResult(result);
|
|
224
224
|
},
|
|
225
|
-
// A heartbeat for an in-flight long step resets its liveness window (see callRemote)
|
|
225
|
+
// A heartbeat for an in-flight long step resets its liveness window (see callRemote) and is
|
|
226
|
+
// persisted (throttled) on the step's checkpoint so liveness is visible cross-process.
|
|
226
227
|
async (beat) => {
|
|
227
228
|
this.heartbeatResets.get(beat.stepId)?.();
|
|
229
|
+
this.persistHeartbeat(beat);
|
|
228
230
|
},
|
|
229
231
|
// A remote workflow worker streams each local step's lifecycle (running → completed/failed) so
|
|
230
232
|
// it's checkpointed live, not all-at-once when the long turn ends.
|
|
@@ -403,6 +405,34 @@ export class WorkflowEngine {
|
|
|
403
405
|
registerQueue(config) {
|
|
404
406
|
this.admission.register(config);
|
|
405
407
|
}
|
|
408
|
+
/** Last persisted beat per stepId (time + whether it carried progress) — the write throttle for
|
|
409
|
+
* {@link persistHeartbeat}. */
|
|
410
|
+
heartbeatPersists = new Map();
|
|
411
|
+
/**
|
|
412
|
+
* Persist a worker heartbeat on its step's checkpoint (see {@link StepCheckpoint.lastHeartbeatAt}),
|
|
413
|
+
* best-effort and THROTTLED per step (≥10s between writes) — beats may arrive every few seconds
|
|
414
|
+
* from a hot batch loop, and liveness observability must not turn into a per-item UPDATE storm.
|
|
415
|
+
* One exemption: a beat CARRYING progress punches through when the last persisted one had none —
|
|
416
|
+
* the automatic pickup beat lands first (no payload) and must not shadow the handler's first real
|
|
417
|
+
* progress for a whole window. Optional store capability: no `recordStepHeartbeat` → no-op.
|
|
418
|
+
* Failures are swallowed (a lost liveness update never harms the run). The throttle map is cleared
|
|
419
|
+
* wholesale past 1024 entries — a rare extra write, never a leak.
|
|
420
|
+
*/
|
|
421
|
+
persistHeartbeat(beat) {
|
|
422
|
+
if (!this.store.recordStepHeartbeat)
|
|
423
|
+
return;
|
|
424
|
+
const now = this.clock();
|
|
425
|
+
const last = this.heartbeatPersists.get(beat.stepId);
|
|
426
|
+
const upgradesToProgress = beat.progress !== undefined && last !== undefined && !last.hadProgress;
|
|
427
|
+
if (last !== undefined && now - last.at < 10_000 && !upgradesToProgress)
|
|
428
|
+
return;
|
|
429
|
+
if (this.heartbeatPersists.size > 1024)
|
|
430
|
+
this.heartbeatPersists.clear();
|
|
431
|
+
this.heartbeatPersists.set(beat.stepId, { at: now, hadProgress: beat.progress !== undefined });
|
|
432
|
+
void this.store
|
|
433
|
+
.recordStepHeartbeat(beat.runId, beat.seq, new Date(beat.at ?? now), beat.progress)
|
|
434
|
+
.catch(() => undefined);
|
|
435
|
+
}
|
|
406
436
|
/**
|
|
407
437
|
* Declare this process a worker: flush consumer subscriptions parked by a transport whose
|
|
408
438
|
* consumption was deferred (see {@link Transport.deferConsumers}). The `durable:work` loop calls
|
|
@@ -2695,6 +2725,27 @@ export class WorkflowEngine {
|
|
|
2695
2725
|
const resultPromise = new Promise((resolve, reject) => {
|
|
2696
2726
|
this.pending.set(id, { resolve, reject });
|
|
2697
2727
|
});
|
|
2728
|
+
// Persist the in-flight `pending` checkpoint BEFORE dispatching. The in-memory await needs no
|
|
2729
|
+
// row to resume — but without one this step was INVISIBLE outside this process (no liveness
|
|
2730
|
+
// surface, nowhere for worker heartbeats to land) and a result consumed by another engine
|
|
2731
|
+
// instance (shared results queue) found no checkpoint and was dropped. Replay-safe: a replayed
|
|
2732
|
+
// `timeoutMs` step routes back here regardless of a pending row (see callRemote's routing),
|
|
2733
|
+
// and `completeStep` below overwrites this row when the result lands.
|
|
2734
|
+
const attemptAt = attempt === 1 ? enqueuedAt : new Date();
|
|
2735
|
+
await this.store.saveCheckpoint({
|
|
2736
|
+
runId,
|
|
2737
|
+
seq,
|
|
2738
|
+
name: step.name,
|
|
2739
|
+
kind: 'remote',
|
|
2740
|
+
stepId: id,
|
|
2741
|
+
status: 'pending',
|
|
2742
|
+
input: validInput,
|
|
2743
|
+
attempts: attempt,
|
|
2744
|
+
workerGroup: token,
|
|
2745
|
+
enqueuedAt: attemptAt,
|
|
2746
|
+
startedAt: attemptAt,
|
|
2747
|
+
finishedAt: attemptAt,
|
|
2748
|
+
});
|
|
2698
2749
|
await this.pool.dispatch({
|
|
2699
2750
|
runId,
|
|
2700
2751
|
seq,
|
|
@@ -2708,7 +2759,7 @@ export class WorkflowEngine {
|
|
|
2708
2759
|
}, transport);
|
|
2709
2760
|
try {
|
|
2710
2761
|
const resolution = step.timeoutMs
|
|
2711
|
-
? await this.awaitWithHeartbeat(id, resultPromise, step.timeoutMs)
|
|
2762
|
+
? await this.awaitWithHeartbeat(id, resultPromise, step.timeoutMs, step.pickupTimeoutMs)
|
|
2712
2763
|
: await resultPromise;
|
|
2713
2764
|
const output = (step.output ? step.output.parse(resolution.output) : resolution.output);
|
|
2714
2765
|
// The worker reports when it actually picked the task up; fall back to dispatch time if a
|
|
@@ -2742,22 +2793,32 @@ export class WorkflowEngine {
|
|
|
2742
2793
|
* arrives within `timeoutMs`. Each heartbeat (delivered via `transport.onHeartbeat`) rearms the
|
|
2743
2794
|
* window, so a worker that keeps beating stays alive past `timeoutMs`.
|
|
2744
2795
|
*/
|
|
2745
|
-
awaitWithHeartbeat(id, resultPromise, timeoutMs) {
|
|
2796
|
+
awaitWithHeartbeat(id, resultPromise, timeoutMs, pickupTimeoutMs) {
|
|
2746
2797
|
return new Promise((resolve, reject) => {
|
|
2747
2798
|
let timer;
|
|
2799
|
+
// Two windows, one timer: until the FIRST beat arrives the job may be sitting in the broker
|
|
2800
|
+
// queue behind long work — a stretch that produces no heartbeats by definition, so it gets
|
|
2801
|
+
// `pickupTimeoutMs` (default: `timeoutMs`, the historical behavior). The worker's automatic
|
|
2802
|
+
// pickup beat flips to the (typically tighter) `timeoutMs`, which then means "max silence
|
|
2803
|
+
// while RUNNING". Without the split, a dispatch-anchored window false-failed a healthy
|
|
2804
|
+
// single-concurrency fleet whose next batch queued ~15min behind the current one.
|
|
2805
|
+
let seenBeat = false;
|
|
2806
|
+
const window = () => (seenBeat ? timeoutMs : (pickupTimeoutMs ?? timeoutMs));
|
|
2748
2807
|
const cleanup = () => {
|
|
2749
2808
|
clearTimeout(timer);
|
|
2750
2809
|
this.heartbeatResets.delete(id);
|
|
2751
2810
|
};
|
|
2752
2811
|
const arm = () => {
|
|
2812
|
+
const ms = window();
|
|
2753
2813
|
timer = setTimeout(() => {
|
|
2754
2814
|
cleanup();
|
|
2755
2815
|
this.pending.delete(id);
|
|
2756
|
-
reject(new RemoteStepTimeout(id,
|
|
2757
|
-
},
|
|
2816
|
+
reject(new RemoteStepTimeout(id, ms));
|
|
2817
|
+
}, ms);
|
|
2758
2818
|
timer.unref?.();
|
|
2759
2819
|
};
|
|
2760
2820
|
this.heartbeatResets.set(id, () => {
|
|
2821
|
+
seenBeat = true;
|
|
2761
2822
|
clearTimeout(timer);
|
|
2762
2823
|
arm();
|
|
2763
2824
|
});
|