velocious 1.0.524 → 1.0.526
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -2
- package/build/background-jobs/main.js +48 -5
- package/build/background-jobs/store.js +28 -7
- package/build/background-jobs/worker.js +126 -7
- package/build/configuration-types.js +9 -0
- package/build/configuration.js +9 -1
- package/build/src/background-jobs/main.d.ts +12 -0
- package/build/src/background-jobs/main.d.ts.map +1 -1
- package/build/src/background-jobs/main.js +48 -6
- package/build/src/background-jobs/store.d.ts +2 -2
- package/build/src/background-jobs/store.d.ts.map +1 -1
- package/build/src/background-jobs/store.js +30 -9
- package/build/src/background-jobs/worker.d.ts +85 -2
- package/build/src/background-jobs/worker.d.ts.map +1 -1
- package/build/src/background-jobs/worker.js +116 -8
- package/build/src/configuration-types.d.ts +21 -0
- package/build/src/configuration-types.d.ts.map +1 -1
- package/build/src/configuration-types.js +10 -1
- package/build/src/configuration.d.ts.map +1 -1
- package/build/src/configuration.js +10 -2
- package/package.json +1 -1
- package/src/background-jobs/main.js +48 -5
- package/src/background-jobs/store.js +28 -7
- package/src/background-jobs/worker.js +126 -7
- package/src/configuration-types.js +9 -0
- package/src/configuration.js +9 -1
package/README.md
CHANGED
|
@@ -1940,7 +1940,7 @@ Velocious includes a simple background jobs system inspired by Sidekiq.
|
|
|
1940
1940
|
|
|
1941
1941
|
Jobs can opt into cross-worker durable concurrency limits by pairing a non-empty `concurrencyKey` with a positive-integer `maxConcurrency` in their background-job options. The first cap registered for a key is stable; conflicting caps are rejected. See [durable concurrency limits](docs/background-jobs.md#durable-concurrency-limits).
|
|
1942
1942
|
|
|
1943
|
-
Production apps can listen for `background-job-failed` or its `all-error` mirror to report accepted failed attempts, including retry and terminal
|
|
1943
|
+
Production apps can listen for `background-job-failed` (or its `all-error` mirror) to report accepted failed attempts, including retry and terminal-state metadata, and for `background-job-orphaned` to react to a specific job the main process reclaimed after its worker died mid-run — e.g. enqueue a targeted recovery for the work it left behind, instead of only polling for the aftermath. See [docs/background-jobs.md](docs/background-jobs.md#failure-events).
|
|
1944
1944
|
|
|
1945
1945
|
## Setup
|
|
1946
1946
|
|
|
@@ -1975,7 +1975,8 @@ export default new Configuration({
|
|
|
1975
1975
|
databaseIdentifier: "default",
|
|
1976
1976
|
maxConcurrentForkedJobs: 4,
|
|
1977
1977
|
maxConcurrentInlineJobs: 4,
|
|
1978
|
-
dispatchStrategy: "beacon"
|
|
1978
|
+
dispatchStrategy: "beacon",
|
|
1979
|
+
jobTimeoutMs: null
|
|
1979
1980
|
}
|
|
1980
1981
|
})
|
|
1981
1982
|
```
|
|
@@ -1991,6 +1992,7 @@ VELOCIOUS_BACKGROUND_JOBS_MAX_CONCURRENT_INLINE_JOBS=4
|
|
|
1991
1992
|
VELOCIOUS_BACKGROUND_JOBS_DISPATCH_STRATEGY=beacon
|
|
1992
1993
|
VELOCIOUS_BACKGROUND_JOBS_POLL_INTERVAL_MS=1000
|
|
1993
1994
|
VELOCIOUS_BACKGROUND_JOBS_WORKER_SHUTDOWN_TIMEOUT_MS=indefinite
|
|
1995
|
+
VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS=5400000
|
|
1994
1996
|
```
|
|
1995
1997
|
|
|
1996
1998
|
`VELOCIOUS_BACKGROUND_JOBS_WORKER_SHUTDOWN_TIMEOUT_MS` (default: `indefinite`) bounds how long a `background-jobs-worker` waits for in-flight jobs on `SIGTERM`/`SIGINT` before terminating any forked or spawned child runners still running, so they are not orphaned across a deploy. The default waits for jobs to finish and never interrupts a running job; set a positive number of milliseconds for a finite cap (keep it shorter than your process supervisor's graceful-stop window so the worker reaps its own runners first). See [docs/background-jobs.md](docs/background-jobs.md#worker-shutdown-and-process-job-draining).
|
|
@@ -1999,6 +2001,8 @@ VELOCIOUS_BACKGROUND_JOBS_WORKER_SHUTDOWN_TIMEOUT_MS=indefinite
|
|
|
1999
2001
|
|
|
2000
2002
|
`maxConcurrentForkedJobs` (default: `4`) caps how many out-of-process `executionMode: "forked"` or `executionMode: "spawned"` jobs one worker may keep in flight. Forked jobs use `child_process.fork()` with an attached IPC channel. After the main process acknowledges their durable status report, forked and spawned one-shot runners exit without waiting for graceful Beacon/database teardown; the OS closes their process-owned resources. A missing or rejected status acknowledgement makes the runner exit as failed instead of reporting clean success. Spawned jobs use the legacy `background-jobs-runner` CLI process via `child_process.spawn()` and are only for callers that intentionally want that spawned behavior.
|
|
2001
2003
|
|
|
2004
|
+
`jobTimeoutMs` (or `VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS`, milliseconds; default: disabled) is a wall-clock backstop for a single `"forked"` runner. A forked job still running after the timeout is terminated (`SIGTERM`, then `SIGKILL` after the reaping grace) and reported `failed`, so a genuinely-hung runner can't pin a worker — and its whole-app boot and DB connections — indefinitely (notably a retired-release worker draining after a deploy). It's a coarse safety net, not per-job tuning: it applies to every forked runner, so set it well above the longest legitimate forked job. Omit it, or set `null`/`<= 0`, to disable. `"inline"` jobs are not covered — they share the worker's process and can't be killed without killing the worker. See [docs/background-jobs.md](docs/background-jobs.md#forked-job-timeout-hung-runner-backstop).
|
|
2005
|
+
|
|
2002
2006
|
### Dispatch strategy
|
|
2003
2007
|
|
|
2004
2008
|
`dispatchStrategy` controls how `background-jobs-main` detects new work.
|
|
@@ -784,6 +784,37 @@ export default class BackgroundJobsMain {
|
|
|
784
784
|
errorEvents.emit("all-error", {...payload, errorType: "background-job-failed"})
|
|
785
785
|
}
|
|
786
786
|
|
|
787
|
+
/**
|
|
788
|
+
* Emits `background-job-orphaned` (mirrored to `all-error`) for a job the time-based orphan sweep
|
|
789
|
+
* reclaimed after its worker died mid-run. Unlike `background-job-failed`, which fires on a
|
|
790
|
+
* worker's failure report, this fires from the main process's sweep, so applications can react to
|
|
791
|
+
* a dead worker's specific job — recover the work it left behind — without polling. `willRetry`
|
|
792
|
+
* reflects whether the reclaim returned the job to the queue for another attempt.
|
|
793
|
+
* @param {{job: import("./types.js").BackgroundJobRow}} args - The orphaned job.
|
|
794
|
+
* @returns {void}
|
|
795
|
+
*/
|
|
796
|
+
_emitBackgroundJobOrphaned({job}) {
|
|
797
|
+
const normalizedError = this._normalizeFailureError(job.lastError ?? "Job orphaned after timeout")
|
|
798
|
+
const payload = {
|
|
799
|
+
context: {
|
|
800
|
+
attempts: job.attempts,
|
|
801
|
+
jobArgs: job.args,
|
|
802
|
+
jobId: job.id,
|
|
803
|
+
jobName: job.jobName,
|
|
804
|
+
maxRetries: job.maxRetries,
|
|
805
|
+
stage: "background-job-orphaned",
|
|
806
|
+
status: job.status,
|
|
807
|
+
terminal: job.status === "failed" || job.status === "orphaned",
|
|
808
|
+
willRetry: job.status === "queued"
|
|
809
|
+
},
|
|
810
|
+
error: normalizedError
|
|
811
|
+
}
|
|
812
|
+
const errorEvents = this.configuration.getErrorEvents()
|
|
813
|
+
|
|
814
|
+
errorEvents.emit("background-job-orphaned", payload)
|
|
815
|
+
errorEvents.emit("all-error", {...payload, errorType: "background-job-orphaned"})
|
|
816
|
+
}
|
|
817
|
+
|
|
787
818
|
/**
|
|
788
819
|
* Runs normalize failure error.
|
|
789
820
|
* @param {?} error - Reported failure value.
|
|
@@ -1170,14 +1201,26 @@ export default class BackgroundJobsMain {
|
|
|
1170
1201
|
|
|
1171
1202
|
async _sweepOrphans() {
|
|
1172
1203
|
try {
|
|
1173
|
-
const
|
|
1204
|
+
const orphanedJobs = await this.store.markOrphanedJobs()
|
|
1174
1205
|
|
|
1175
|
-
if (
|
|
1176
|
-
this.logger.warn(() => ["Marked orphaned background jobs",
|
|
1177
|
-
// Reclaimed orphans become `queued` again — wake the dispatcher
|
|
1178
|
-
//
|
|
1206
|
+
if (orphanedJobs.length > 0) {
|
|
1207
|
+
this.logger.warn(() => ["Marked orphaned background jobs", orphanedJobs.length])
|
|
1208
|
+
// Reclaimed orphans become `queued` again — wake the dispatcher first so
|
|
1209
|
+
// an application event handler that throws below cannot strand them
|
|
1210
|
+
// queued until the next external enqueue/reconnect.
|
|
1179
1211
|
this._notifyEnqueued()
|
|
1180
1212
|
await this._drain()
|
|
1213
|
+
// Emit an event per orphaned job so applications can react to a dead
|
|
1214
|
+
// worker's specific job (e.g. targeted recovery) instead of only polling
|
|
1215
|
+
// for its aftermath. Isolate each so one throwing handler can't suppress
|
|
1216
|
+
// the events for the rest.
|
|
1217
|
+
for (const job of orphanedJobs) {
|
|
1218
|
+
try {
|
|
1219
|
+
this._emitBackgroundJobOrphaned({job})
|
|
1220
|
+
} catch (error) {
|
|
1221
|
+
this.logger.error(() => ["A background-job-orphaned event handler threw:", error])
|
|
1222
|
+
}
|
|
1223
|
+
}
|
|
1181
1224
|
}
|
|
1182
1225
|
} catch (error) {
|
|
1183
1226
|
this.logger.error(() => ["Failed to mark orphaned jobs:", error])
|
|
@@ -547,7 +547,7 @@ export default class BackgroundJobsStore {
|
|
|
547
547
|
* Runs mark orphaned jobs.
|
|
548
548
|
* @param {object} [args] - Options.
|
|
549
549
|
* @param {number} [args.orphanedAfterMs] - Mark jobs orphaned after this duration.
|
|
550
|
-
* @returns {Promise<
|
|
550
|
+
* @returns {Promise<import("./types.js").BackgroundJobRow[]>} - The jobs this sweep marked orphaned.
|
|
551
551
|
*/
|
|
552
552
|
async markOrphanedJobs({orphanedAfterMs = ORPHANED_AFTER_MS} = {}) {
|
|
553
553
|
await this.ensureReady()
|
|
@@ -562,7 +562,8 @@ export default class BackgroundJobsStore {
|
|
|
562
562
|
|
|
563
563
|
const rows = await query.results()
|
|
564
564
|
|
|
565
|
-
|
|
565
|
+
/** @type {import("./types.js").BackgroundJobRow[]} */
|
|
566
|
+
const orphanedJobs = []
|
|
566
567
|
|
|
567
568
|
for (const row of rows) {
|
|
568
569
|
const job = this._normalizeJobRow(row)
|
|
@@ -588,10 +589,10 @@ export default class BackgroundJobsStore {
|
|
|
588
589
|
conditions: {id: job.id, status: "handed_off", handed_off_at_ms: job.handedOffAtMs}
|
|
589
590
|
})
|
|
590
591
|
|
|
591
|
-
if (orphanedJob)
|
|
592
|
+
if (orphanedJob) orphanedJobs.push(orphanedJob)
|
|
592
593
|
}
|
|
593
594
|
|
|
594
|
-
return
|
|
595
|
+
return orphanedJobs
|
|
595
596
|
})
|
|
596
597
|
}
|
|
597
598
|
|
|
@@ -1075,10 +1076,30 @@ export default class BackgroundJobsStore {
|
|
|
1075
1076
|
if (affectedRows !== 1) return null
|
|
1076
1077
|
await this._releaseConcurrency(db, job.concurrencyKey)
|
|
1077
1078
|
|
|
1078
|
-
|
|
1079
|
+
// Return a snapshot of the transition this update just applied rather than re-reading the row.
|
|
1080
|
+
// We won the conditional update (affectedRows === 1), so this state is authoritative; re-reading
|
|
1081
|
+
// could instead observe a newer state if another dispatcher reclaims a requeued job between the
|
|
1082
|
+
// update and the read (overlapping mains / polling dispatch), which would misreport the
|
|
1083
|
+
// status/terminal/willRetry of this transition to failure/orphan event listeners.
|
|
1084
|
+
const status = shouldRetry ? "queued" : (markOrphaned ? "orphaned" : "failed")
|
|
1085
|
+
/** @type {import("./types.js").BackgroundJobRow} */
|
|
1086
|
+
const transitionedJob = {
|
|
1087
|
+
...job,
|
|
1088
|
+
attempts: nextAttempt,
|
|
1089
|
+
handedOffAtMs: null,
|
|
1090
|
+
lastError: failureMessage,
|
|
1091
|
+
status,
|
|
1092
|
+
workerId: null
|
|
1093
|
+
}
|
|
1094
|
+
|
|
1095
|
+
if (markOrphaned) transitionedJob.orphanedAtMs = now
|
|
1096
|
+
if (shouldRetry) {
|
|
1097
|
+
transitionedJob.scheduledAtMs = scheduledAt
|
|
1098
|
+
} else if (!markOrphaned) {
|
|
1099
|
+
transitionedJob.failedAtMs = now
|
|
1100
|
+
}
|
|
1079
1101
|
|
|
1080
|
-
|
|
1081
|
-
return updatedJob
|
|
1102
|
+
return transitionedJob
|
|
1082
1103
|
}
|
|
1083
1104
|
|
|
1084
1105
|
/**
|
|
@@ -11,6 +11,13 @@ import {fileURLToPath} from "node:url"
|
|
|
11
11
|
|
|
12
12
|
/** Grace period after SIGTERM before a lingering process runner is SIGKILLed. */
|
|
13
13
|
const FORKED_CHILD_SIGKILL_GRACE_MS = 5000
|
|
14
|
+
/**
|
|
15
|
+
* Largest delay Node's `setTimeout` accepts without overflowing to a 1ms delay
|
|
16
|
+
* (a 32-bit signed int of ms, ~24.8 days). A `jobTimeoutMs` above this — or a
|
|
17
|
+
* non-finite one like `Infinity` — is clamped/disabled rather than coerced to
|
|
18
|
+
* ~1ms, which would otherwise terminate every forked job almost immediately.
|
|
19
|
+
*/
|
|
20
|
+
const MAX_FORKED_JOB_TIMEOUT_MS = 2_147_483_647
|
|
14
21
|
const FORKED_RUNNER_ENTRY_PATH = fileURLToPath(new URL("./forked-runner-child.js", import.meta.url))
|
|
15
22
|
/** How often the worker sends a liveness heartbeat to the main. */
|
|
16
23
|
const HEARTBEAT_INTERVAL_MS = 15000
|
|
@@ -21,6 +28,15 @@ const SOCKET_KEEPALIVE_MS = 10000
|
|
|
21
28
|
* @type {import("./types.js").BackgroundJobExecutionMode[]} */
|
|
22
29
|
const EXECUTION_MODES = ["inline", "forked", "spawned"]
|
|
23
30
|
|
|
31
|
+
/**
|
|
32
|
+
* Per-forked-child timeout bookkeeping.
|
|
33
|
+
* @typedef {object} ForkedJobTimeoutState
|
|
34
|
+
* @property {boolean} timedOut - Whether the timeout fired and the child was terminated.
|
|
35
|
+
* @property {number | null} timeoutMs - The armed timeout in ms, or null when disabled.
|
|
36
|
+
* @property {ReturnType<typeof setTimeout> | null} timer - The pending timeout timer, cleared on exit.
|
|
37
|
+
* @property {ReturnType<typeof setTimeout> | null} sigkillTimer - The pending SIGKILL grace timer, cleared on exit.
|
|
38
|
+
*/
|
|
39
|
+
|
|
24
40
|
export default class BackgroundJobsWorker {
|
|
25
41
|
/**
|
|
26
42
|
* Runs constructor.
|
|
@@ -32,8 +48,9 @@ export default class BackgroundJobsWorker {
|
|
|
32
48
|
* @param {number} [args.maxConcurrentInlineJobs] - Override the inline-job concurrency cap from `configuration.getBackgroundJobsConfig()`.
|
|
33
49
|
* @param {number} [args.forkedChildSigkillGraceMs] - Override the grace period between SIGTERM and SIGKILL when reaping lingering process runners on stop.
|
|
34
50
|
* @param {number} [args.heartbeatIntervalMs] - Override the liveness heartbeat interval (default 15000ms).
|
|
51
|
+
* @param {number} [args.jobTimeoutMs] - Override the forked-job wall-clock timeout from `configuration.getBackgroundJobsConfig()`. `0` disables it.
|
|
35
52
|
*/
|
|
36
|
-
constructor({configuration, host, port, maxConcurrentForkedJobs, maxConcurrentInlineJobs, forkedChildSigkillGraceMs, heartbeatIntervalMs} = {}) {
|
|
53
|
+
constructor({configuration, host, port, maxConcurrentForkedJobs, maxConcurrentInlineJobs, forkedChildSigkillGraceMs, heartbeatIntervalMs, jobTimeoutMs} = {}) {
|
|
37
54
|
/**
|
|
38
55
|
* Narrows the runtime value to the documented type.
|
|
39
56
|
* @type {Promise<import("../configuration.js").default>} */
|
|
@@ -77,6 +94,13 @@ export default class BackgroundJobsWorker {
|
|
|
77
94
|
this.forkedChildSigkillGraceMs = typeof forkedChildSigkillGraceMs === "number" && forkedChildSigkillGraceMs >= 0
|
|
78
95
|
? forkedChildSigkillGraceMs
|
|
79
96
|
: FORKED_CHILD_SIGKILL_GRACE_MS
|
|
97
|
+
/**
|
|
98
|
+
* Constructor override for the forked-job wall-clock timeout. When unset the
|
|
99
|
+
* timeout is read from `configuration.getBackgroundJobsConfig().jobTimeoutMs`
|
|
100
|
+
* at fork time (default: disabled).
|
|
101
|
+
* @type {number | undefined}
|
|
102
|
+
*/
|
|
103
|
+
this.jobTimeoutMsOverride = typeof jobTimeoutMs === "number" ? jobTimeoutMs : undefined
|
|
80
104
|
this.shouldStop = false
|
|
81
105
|
this.workerId = randomUUID()
|
|
82
106
|
this.heartbeatIntervalMs = typeof heartbeatIntervalMs === "number" && heartbeatIntervalMs >= 1
|
|
@@ -590,16 +614,109 @@ export default class BackgroundJobsWorker {
|
|
|
590
614
|
* @returns {Promise<void>} - Resolves when the child exits.
|
|
591
615
|
*/
|
|
592
616
|
_waitForForkedChild({child, payload}) {
|
|
617
|
+
const timeoutState = this._armForkedJobTimeout({child})
|
|
618
|
+
|
|
593
619
|
return new Promise((resolve) => {
|
|
594
620
|
child.once("exit", (code, signal) => {
|
|
595
|
-
this.
|
|
621
|
+
this._clearForkedJobTimeout(timeoutState)
|
|
622
|
+
this._handleForkedChildExit({child, code, signal, payload, resolve, timeoutState})
|
|
596
623
|
})
|
|
597
624
|
child.once("error", (error) => {
|
|
625
|
+
this._clearForkedJobTimeout(timeoutState)
|
|
598
626
|
this._handleForkedChildError({child, error, payload, resolve})
|
|
599
627
|
})
|
|
600
628
|
})
|
|
601
629
|
}
|
|
602
630
|
|
|
631
|
+
/**
|
|
632
|
+
* Arms a wall-clock backstop for a forked job runner. A forked job still
|
|
633
|
+
* running after `jobTimeoutMs` is terminated (SIGTERM, then SIGKILL after the
|
|
634
|
+
* grace) so a single genuinely-hung runner can't pin a draining worker — and
|
|
635
|
+
* its full-app boot and database connections — indefinitely. Returns a state
|
|
636
|
+
* object the exit/error handlers use to cancel the timer and to report a
|
|
637
|
+
* timeout-specific failure. When no timeout is configured the timer is null
|
|
638
|
+
* and behavior is unchanged.
|
|
639
|
+
* @param {object} args - Options.
|
|
640
|
+
* @param {import("node:child_process").ChildProcess} args.child - Forked child process.
|
|
641
|
+
* @returns {ForkedJobTimeoutState} - Timeout state.
|
|
642
|
+
*/
|
|
643
|
+
_armForkedJobTimeout({child}) {
|
|
644
|
+
const timeoutMs = this._resolveForkedJobTimeoutMs()
|
|
645
|
+
/** @type {ForkedJobTimeoutState} */
|
|
646
|
+
const state = {timedOut: false, timeoutMs, timer: null, sigkillTimer: null}
|
|
647
|
+
|
|
648
|
+
if (!(typeof timeoutMs === "number" && timeoutMs > 0)) return state
|
|
649
|
+
|
|
650
|
+
state.timer = setTimeout(() => this._onForkedJobTimeout({child, state}), timeoutMs)
|
|
651
|
+
|
|
652
|
+
return state
|
|
653
|
+
}
|
|
654
|
+
|
|
655
|
+
/**
|
|
656
|
+
* Resolves the effective forked-job timeout in ms, or null when disabled. The
|
|
657
|
+
* constructor override wins; otherwise the value comes from the background-jobs
|
|
658
|
+
* configuration. A non-positive value disables the backstop.
|
|
659
|
+
* @returns {number | null} - Timeout in ms, or null when disabled.
|
|
660
|
+
*/
|
|
661
|
+
_resolveForkedJobTimeoutMs() {
|
|
662
|
+
const raw = typeof this.jobTimeoutMsOverride === "number"
|
|
663
|
+
? this.jobTimeoutMsOverride
|
|
664
|
+
: (this.configuration ? this.configuration.getBackgroundJobsConfig().jobTimeoutMs : null)
|
|
665
|
+
|
|
666
|
+
// A non-finite (e.g. Infinity) or non-positive value disables the backstop;
|
|
667
|
+
// a finite value beyond Node's timer range is clamped to the max rather than
|
|
668
|
+
// silently coerced to ~1ms (which would kill every forked job immediately).
|
|
669
|
+
if (typeof raw !== "number" || !Number.isFinite(raw) || raw <= 0) return null
|
|
670
|
+
|
|
671
|
+
return Math.min(raw, MAX_FORKED_JOB_TIMEOUT_MS)
|
|
672
|
+
}
|
|
673
|
+
|
|
674
|
+
/**
|
|
675
|
+
* Fired when a forked runner overruns its timeout. Sends SIGTERM for a clean
|
|
676
|
+
* shutdown, then SIGKILL after the grace for a runner that ignores it. The
|
|
677
|
+
* resulting non-clean exit flows through `_handleForkedChildExit`, which frees
|
|
678
|
+
* the slot and reports the job failed.
|
|
679
|
+
* @param {object} args - Options.
|
|
680
|
+
* @param {import("node:child_process").ChildProcess} args.child - Forked child process.
|
|
681
|
+
* @param {ForkedJobTimeoutState} args.state - Timeout state.
|
|
682
|
+
* @returns {void}
|
|
683
|
+
*/
|
|
684
|
+
_onForkedJobTimeout({child, state}) {
|
|
685
|
+
state.timedOut = true
|
|
686
|
+
|
|
687
|
+
try {
|
|
688
|
+
child.kill("SIGTERM")
|
|
689
|
+
} catch {
|
|
690
|
+
// Child already exited; nothing to do.
|
|
691
|
+
}
|
|
692
|
+
|
|
693
|
+
state.sigkillTimer = setTimeout(() => {
|
|
694
|
+
try {
|
|
695
|
+
child.kill("SIGKILL")
|
|
696
|
+
} catch {
|
|
697
|
+
// Child already exited; nothing to do.
|
|
698
|
+
}
|
|
699
|
+
}, this.forkedChildSigkillGraceMs)
|
|
700
|
+
}
|
|
701
|
+
|
|
702
|
+
/**
|
|
703
|
+
* Cancels any pending timeout/SIGKILL timers for a forked runner that has
|
|
704
|
+
* exited (or errored) so they never fire against a gone or reused child.
|
|
705
|
+
* @param {ForkedJobTimeoutState} state - Timeout state.
|
|
706
|
+
* @returns {void}
|
|
707
|
+
*/
|
|
708
|
+
_clearForkedJobTimeout(state) {
|
|
709
|
+
if (state.timer) {
|
|
710
|
+
clearTimeout(state.timer)
|
|
711
|
+
state.timer = null
|
|
712
|
+
}
|
|
713
|
+
|
|
714
|
+
if (state.sigkillTimer) {
|
|
715
|
+
clearTimeout(state.sigkillTimer)
|
|
716
|
+
state.sigkillTimer = null
|
|
717
|
+
}
|
|
718
|
+
}
|
|
719
|
+
|
|
603
720
|
/**
|
|
604
721
|
* Runs handle forked child exit.
|
|
605
722
|
* @param {object} args - Options.
|
|
@@ -608,9 +725,10 @@ export default class BackgroundJobsWorker {
|
|
|
608
725
|
* @param {keyof typeof import("node:os").constants.signals | null} args.signal - Exit signal.
|
|
609
726
|
* @param {import("./types.js").BackgroundJobPayload & {id: string}} args.payload - Payload.
|
|
610
727
|
* @param {(value: void) => void} args.resolve - Promise resolver.
|
|
728
|
+
* @param {ForkedJobTimeoutState} [args.timeoutState] - Timeout state, when the runner had a wall-clock backstop.
|
|
611
729
|
* @returns {void}
|
|
612
730
|
*/
|
|
613
|
-
_handleForkedChildExit({child, code, signal, payload, resolve}) {
|
|
731
|
+
_handleForkedChildExit({child, code, signal, payload, resolve, timeoutState}) {
|
|
614
732
|
this.inflightProcessChildren.delete(child)
|
|
615
733
|
|
|
616
734
|
// Free the worker slot as soon as the child is gone — never gate it on the
|
|
@@ -620,10 +738,11 @@ export default class BackgroundJobsWorker {
|
|
|
620
738
|
|
|
621
739
|
if (this._forkedChildExitedCleanly({code, signal})) return
|
|
622
740
|
|
|
623
|
-
|
|
624
|
-
|
|
625
|
-
|
|
626
|
-
|
|
741
|
+
const error = timeoutState?.timedOut
|
|
742
|
+
? new Error(`Forked background job runner timed out after ${timeoutState.timeoutMs}ms and was terminated: code=${code} signal=${signal || "none"}`)
|
|
743
|
+
: new Error(`Forked background job runner exited before reporting: code=${code} signal=${signal || "none"}`)
|
|
744
|
+
|
|
745
|
+
this._reportForkedChildFailure({payload, error})
|
|
627
746
|
}
|
|
628
747
|
|
|
629
748
|
/**
|
|
@@ -192,6 +192,15 @@
|
|
|
192
192
|
* to restore the legacy fixed-interval poll.
|
|
193
193
|
* @property {number} [pollIntervalMs] - Poll interval in milliseconds. Only used
|
|
194
194
|
* when `dispatchStrategy === "polling"`. Default: `1000`.
|
|
195
|
+
* @property {number | null} [jobTimeoutMs] - Wall-clock backstop, in ms, for a
|
|
196
|
+
* `"forked"` job runner. A forked job still running after this is terminated
|
|
197
|
+
* (SIGTERM, then SIGKILL after the reaping grace) and reported failed, so a
|
|
198
|
+
* single genuinely-hung runner can't pin a draining worker — and its full-app
|
|
199
|
+
* boot and DB connections — indefinitely (e.g. across a deploy where a retired
|
|
200
|
+
* worker drains in-flight jobs). This is a coarse safety net, not per-job
|
|
201
|
+
* tuning: set it well above the longest legitimate forked job (build runners,
|
|
202
|
+
* long imports) so only genuinely-stuck runners are killed. Omit, `null`, or
|
|
203
|
+
* `<= 0` to disable (default), which preserves the prior unbounded behavior.
|
|
195
204
|
* @property {BackgroundJobsRetentionConfiguration} [retention] - Retention/pruning
|
|
196
205
|
* of terminal job rows. Without pruning the jobs table grows unbounded
|
|
197
206
|
* (completed rows accumulate forever), which bloats storage and indexes and
|
package/build/configuration.js
CHANGED
|
@@ -1270,10 +1270,12 @@ export default class VelociousConfiguration {
|
|
|
1270
1270
|
const envMaxConcurrentRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_MAX_CONCURRENT_INLINE_JOBS
|
|
1271
1271
|
const envDispatchStrategy = process.env.VELOCIOUS_BACKGROUND_JOBS_DISPATCH_STRATEGY
|
|
1272
1272
|
const envPollIntervalRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_POLL_INTERVAL_MS
|
|
1273
|
+
const envJobTimeoutRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS
|
|
1273
1274
|
const envPort = envPortRaw ? Number(envPortRaw) : undefined
|
|
1274
1275
|
const envMaxConcurrentForked = envMaxConcurrentForkedRaw ? Number(envMaxConcurrentForkedRaw) : undefined
|
|
1275
1276
|
const envMaxConcurrent = envMaxConcurrentRaw ? Number(envMaxConcurrentRaw) : undefined
|
|
1276
1277
|
const envPollInterval = envPollIntervalRaw ? Number(envPollIntervalRaw) : undefined
|
|
1278
|
+
const envJobTimeout = envJobTimeoutRaw ? Number(envJobTimeoutRaw) : undefined
|
|
1277
1279
|
const configured = this._backgroundJobs || {}
|
|
1278
1280
|
const host = configured.host || envHost || "127.0.0.1"
|
|
1279
1281
|
const port = typeof configured.port === "number"
|
|
@@ -1292,6 +1294,12 @@ export default class VelociousConfiguration {
|
|
|
1292
1294
|
? configured.pollIntervalMs
|
|
1293
1295
|
: (typeof envPollInterval === "number" && Number.isFinite(envPollInterval) && envPollInterval >= 1 ? envPollInterval : 1000)
|
|
1294
1296
|
const queues = configured.queues && typeof configured.queues === "object" ? configured.queues : {}
|
|
1297
|
+
// An explicit config value wins over the env var — including `null`/`0`,
|
|
1298
|
+
// which disable the backstop even when the environment sets a default.
|
|
1299
|
+
// Only fall through to the env var when config omits `jobTimeoutMs` entirely.
|
|
1300
|
+
const jobTimeoutMs = "jobTimeoutMs" in configured
|
|
1301
|
+
? (typeof configured.jobTimeoutMs === "number" && configured.jobTimeoutMs > 0 ? configured.jobTimeoutMs : null)
|
|
1302
|
+
: (typeof envJobTimeout === "number" && Number.isFinite(envJobTimeout) && envJobTimeout > 0 ? envJobTimeout : null)
|
|
1295
1303
|
const configuredRetention = configured.retention && typeof configured.retention === "object" ? configured.retention : {}
|
|
1296
1304
|
const retention = {
|
|
1297
1305
|
completedTtlMs: typeof configuredRetention.completedTtlMs === "number" || configuredRetention.completedTtlMs === null
|
|
@@ -1308,7 +1316,7 @@ export default class VelociousConfiguration {
|
|
|
1308
1316
|
: 60 * 60 * 1000
|
|
1309
1317
|
}
|
|
1310
1318
|
|
|
1311
|
-
return {host, port, databaseIdentifier, maxConcurrentForkedJobs, maxConcurrentInlineJobs, dispatchStrategy, pollIntervalMs, queues, retention}
|
|
1319
|
+
return {host, port, databaseIdentifier, maxConcurrentForkedJobs, maxConcurrentInlineJobs, dispatchStrategy, pollIntervalMs, queues, jobTimeoutMs, retention}
|
|
1312
1320
|
}
|
|
1313
1321
|
|
|
1314
1322
|
/**
|
|
@@ -332,6 +332,18 @@ export default class BackgroundJobsMain {
|
|
|
332
332
|
job: import("./types.js").BackgroundJobRow;
|
|
333
333
|
workerId?: string;
|
|
334
334
|
}): void;
|
|
335
|
+
/**
|
|
336
|
+
* Emits `background-job-orphaned` (mirrored to `all-error`) for a job the time-based orphan sweep
|
|
337
|
+
* reclaimed after its worker died mid-run. Unlike `background-job-failed`, which fires on a
|
|
338
|
+
* worker's failure report, this fires from the main process's sweep, so applications can react to
|
|
339
|
+
* a dead worker's specific job — recover the work it left behind — without polling. `willRetry`
|
|
340
|
+
* reflects whether the reclaim returned the job to the queue for another attempt.
|
|
341
|
+
* @param {{job: import("./types.js").BackgroundJobRow}} args - The orphaned job.
|
|
342
|
+
* @returns {void}
|
|
343
|
+
*/
|
|
344
|
+
_emitBackgroundJobOrphaned({ job }: {
|
|
345
|
+
job: import("./types.js").BackgroundJobRow;
|
|
346
|
+
}): void;
|
|
335
347
|
/**
|
|
336
348
|
* Runs normalize failure error.
|
|
337
349
|
* @param {?} error - Reported failure value.
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"main.d.ts","sourceRoot":"","sources":["../../../src/background-jobs/main.js"],"names":[],"mappings":"AA8CA;IACE;;;;;;;;OAQG;IACH,wFANG;QAAoD,aAAa,EAAzD,OAAO,qBAAqB,EAAE,OAAO;QACvB,IAAI;QACJ,IAAI;QACJ,oBAAoB;QACpB,qBAAqB;KAC7C,EAsEA;IApEC,qDAAkC;IAElC,aAA+B;IAC/B,aAAyD;IACzD,qFAA+C;IAC/C,uBAA2C;IAC3C,uKAAiC;IAGjC,6BAAkJ;IAClJ,8BAAuJ;IACvJ,2BAAoG;IACpG,eAA8B;IAC9B;;iCAE6B;IAC7B,SADU,GAAG,CAAC,UAAU,CAAC,CACD;IACxB;;iCAE6B;IAC7B,cADU,GAAG,CAAC,UAAU,CAAC,CACI;IAC7B;;sDAEkD;IAClD,gBADU,GAAG,CAAC,UAAU,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC,CACf;IAC/B;;wCAEoC;IACpC,QADU,GAAG,CAAC,MAAM,GAAG,SAAS,CACT;IACvB;;2DAEuD;IACvD,YADU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,SAAS,CACxB;IAC3B;;2DAEuD;IACvD,iBADU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,SAAS,CACnB;IAChC;;2DAEuD;IACvD,kBADU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,SAAS,CAClB;IACjC;;2DAEuD;IACvD,cADU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,SAAS,CACtB;IAC7B;;4DAEwD;IACxD,mBADU,UAAU,CAAC,OAAO,WAAW,CAAC,GAAG,SAAS,CAClB;IAClC;;qDAEiD;IACjD,WADU,uBAAuB,GAAG,SAAS,CACnB;IAC1B,mBAAsB;IACtB,wBAA2B;IAC3B,kBAAqB;IACrB;;0CAEsC;IACtC,oBADU,CAAC,MAAM,IAAI,CAAC,GAAG,SAAS,CACC;IACnC;;2DAEuD;IACvD,uBADU,CAAC,CAAC,GAAG,IAAI,EAAE,KAAK,CAAC,OAAC,CAAC,KAAK,IAAI,CAAC,GAAG,SAAS,CACb;IACtC;;sHAEkH;IAClH,eADU,OAAO,qBAAqB,EAAE,OAAO,GAAG,OAAO,gCAAgC,EAAE,OAAO,GAAG,SAAS,CAChF;IAGhC;;;OAGG;IACH,SAFa,OAAO,CAAC,IAAI,CAAC,CAqEzB;IAED;;;OAGG;IACH,QAFa,OAAO,CAAC,IAAI,CAAC,CAWzB;IAED;;yBAEqB;IACrB,iBADa,IAAI,CAKhB;IAED;;yBAEqB;IACrB,gBADa,IAAI,CAYhB;IAED;;yBAEqB;IACrB,6BADa,IAAI,CAYhB;IAED;;kCAE8B;IAC9B,wBADa,OAAO,CAAC,IAAI,CAAC,CAOzB;IAED;;kCAE8B;IAC9B,sCADa,OAAO,CAAC,IAAI,CAAC,CAOzB;IAED;;kCAE8B;IAC9B,gBADa,OAAO,CAAC,IAAI,CAAC,CAOzB;IAED;;;OAGG;IACH,WAFa,MAAM,CAIlB;IAED;;;;;;;;;;OAUG;IACH,0BAFa,IAAI,CA2BhB;IAED;;;;;;OAMG;IACH,mBAFa,IAAI,CAiBhB;IAED;;;;OAIG;IACH,0BAHW,OAAO,KAAK,EAAE,MAAM,GAClB,IAAI,CA0BhB;IAED;;;;;;;OAOG;IACH,oDALG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;QACW,IAAI,EAA9D,OAAO,YAAY,EAAE,uBAAuB,GAAG,IAAI;KAC3D,GAAU,OAAO,YAAY,EAAE,uBAAuB,GAAG,IAAI,CAS/D;IAED;;;;;;OAMG;IACH,sDAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,OAAO,YAAY,EAAE,uBAAuB,GAAG,IAAI,CAgB/D;IAED;;;;;;;;;;;;;OAaG;IACH,iCAHW,UAAU,GACR,OAAO,CAAC,IAAI,CAAC,CAqBzB;IAED;;;;;;OAMG;IACH,oDAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,IAAI,CAMhB;IAED;;;;;;OAMG;IACH,oDAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,IAAI,CAsBhB;IAED;;;;;;OAMG;IACH,sDAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,IAAI,CAWhB;IAED;;;;;;OAMG;IACH,4CAJG;QAAyB,UAAU,EAA3B,UAAU;QAC2C,OAAO,EAA5D,OAAO,YAAY,EAAE,yBAAyB;KACtD,GAAU,IAAI,CAYhB;IAED;;;;;OAKG;IACH,sCAHG;QAAyB,UAAU,EAA3B,UAAU;KAClB,GAAU,IAAI,CAOhB;IAED;;;;OAIG;IACH,kCAHW,UAAU,GACR,OAAO,CAAC,IAAI,CAAC,CAiBzB;IAED;;;;OAIG;IACH,+BAHW,UAAU,GACR,OAAO,CAAC,IAAI,CAAC,CAiBzB;IAED;;;;;;;OAOG;IACH,8CALG;QAAqB,SAAS,EAAtB,MAAM;QACO,KAAK,EAAlB,MAAM;QACW,MAAM,EAAvB,UAAU;KAClB,GAAU,OAAO,CAAC,IAAI,CAAC,CAQzB;IAED;;;;;;OAMG;IACH,qCAJG;QAAqB,SAAS,EAAtB,MAAM;QACO,KAAK,EAAlB,MAAM;KACd,GAAU,IAAI,CAUhB;IAED;;;;OAIG;IACH,kCAHW,OAAC,GACC,IAAI,CAUhB;IAED;;;;;;OAMG;IACH,gCAHW,OAAC,GACC,IAAI,CAUhB;IAED;;;;;;OAMG;IACH,wCAJG;QAAyB,UAAU,EAA3B,UAAU;QAC6C,OAAO,EAA9D,OAAO,YAAY,EAAE,2BAA2B;KACxD,GAAU,OAAO,CAAC,IAAI,CAAC,CAiBzB;IAED;;;;;;OAMG;IACH,4CAJG;QAAyB,UAAU,EAA3B,UAAU;QAC8C,OAAO,EAA/D,OAAO,YAAY,EAAE,4BAA4B;KACzD,GAAU,OAAO,CAAC,IAAI,CAAC,CAkBzB;IAED;;;;;;OAMG;IACH,0CAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,OAAO,CAAC,IAAI,CAAC,CAkCzB;IAED;;;;OAIG;IACH,6EAHW;QAAC,KAAK,EAAE,OAAC,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,OAAO,YAAY,EAAE,gBAAgB,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;KAAC,GACnH,IAAI,CAyBhB;IAED;;;;OAIG;IACH,8BAHW,OAAC,GACC,KAAK,CAMjB;IAED;;;;OAIG;IACH,gCAHW,OAAC,GACC,KAAK,CASjB;IAED;;;;OAIG;IACH,kCAHW,OAAC,GACC,MAAM,CAMlB;IAED;;;;OAIG;IACH,yBAHW,OAAC,GACC,KAAK,IAAI,MAAM,CAI3B;IAED;;;;;;OAMG;IACH,oDAJG;QAAgB,KAAK,EAAb,OAAC;QACW,eAAe,EAA3B,KAAK;KACb,GAAU,IAAI,CAIhB;IAED;;;;;;;;;;;;;;;;OAgBG;IACH,UAFa,OAAO,CAAC,IAAI,CAAC,CAQzB;IAED;;;OAGG;IACH,eAFa,OAAO,CAQnB;IAED;;;;;OAKG;IACH,0BAHG;QAAsB,OAAO,EAArB,OAAO;KACf,GAAU,OAAO,CAAC,IAAI,CAAC,CAOzB;IAED;;;OAGG;IACH,6BAFa,OAAO,CAAC,IAAI,CAAC,CAYzB;IAED;;yBAEqB;IACrB,yBADa,IAAI,CAUhB;IAED;;;OAGG;IACH,+BAFa,OAAO,CAOnB;IAED;;;OAGG;IACH,mBAFa,OAAO,CAAC,OAAO,CAAC,CAQ5B;IAED;;;OAGG;IACH,iBAFa,OAAO,CAAC,OAAO,CAAC,CAW5B;IAED;;;OAGG;IACH,6BAFa,OAAO,CAAC,OAAO,CAAC,CAU5B;IAED;;;;;;OAMG;IACH,uBAFa,IAAI,CAWhB;IAED;;;OAGG;IACH,oBAFa,OAAO,CAAC,IAAI,CAAC,CAgBzB;IAED;;;;OAIG;IACH,cAFa,OAAO,CAAC,IAAI,CAAC,CAwDzB;IAED;;;OAGG;IACH,mCAFa,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CASjE;IAED;;;OAGG;IACH,6BAFa,OAAO,YAAY,EAAE,0BAA0B,EAAE,CAU7D;IAED;;;;;;OAMG;IACH,uDAJG;QAAmE,cAAc,EAAzE,GAAG,CAAC,OAAO,YAAY,EAAE,0BAA0B,CAAC;QACnC,MAAM,EAAvB,UAAU;KAClB,GAAU,IAAI,CAQhB;IAED;;;;OAIG;IACH,uBAHW,OAAO,YAAY,EAAE,gBAAgB,GACnC,UAAU,GAAG,SAAS,CAMlC;IAED;;;;;;OAMG;IACH,mCAJG;QAAoD,GAAG,EAA/C,OAAO,YAAY,EAAE,gBAAgB;QACpB,MAAM,EAAvB,UAAU;KAClB,GAAU,OAAO,CAUnB;IAED;;;;;;OAMG;IACH,sBAFa,OAAO,CAAC,IAAI,CAAC,CAoBzB;IAED,+
|
|
1
|
+
{"version":3,"file":"main.d.ts","sourceRoot":"","sources":["../../../src/background-jobs/main.js"],"names":[],"mappings":"AA8CA;IACE;;;;;;;;OAQG;IACH,wFANG;QAAoD,aAAa,EAAzD,OAAO,qBAAqB,EAAE,OAAO;QACvB,IAAI;QACJ,IAAI;QACJ,oBAAoB;QACpB,qBAAqB;KAC7C,EAsEA;IApEC,qDAAkC;IAElC,aAA+B;IAC/B,aAAyD;IACzD,qFAA+C;IAC/C,uBAA2C;IAC3C,uKAAiC;IAGjC,6BAAkJ;IAClJ,8BAAuJ;IACvJ,2BAAoG;IACpG,eAA8B;IAC9B;;iCAE6B;IAC7B,SADU,GAAG,CAAC,UAAU,CAAC,CACD;IACxB;;iCAE6B;IAC7B,cADU,GAAG,CAAC,UAAU,CAAC,CACI;IAC7B;;sDAEkD;IAClD,gBADU,GAAG,CAAC,UAAU,EAAE,GAAG,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC,CACf;IAC/B;;wCAEoC;IACpC,QADU,GAAG,CAAC,MAAM,GAAG,SAAS,CACT;IACvB;;2DAEuD;IACvD,YADU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,SAAS,CACxB;IAC3B;;2DAEuD;IACvD,iBADU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,SAAS,CACnB;IAChC;;2DAEuD;IACvD,kBADU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,SAAS,CAClB;IACjC;;2DAEuD;IACvD,cADU,UAAU,CAAC,OAAO,UAAU,CAAC,GAAG,SAAS,CACtB;IAC7B;;4DAEwD;IACxD,mBADU,UAAU,CAAC,OAAO,WAAW,CAAC,GAAG,SAAS,CAClB;IAClC;;qDAEiD;IACjD,WADU,uBAAuB,GAAG,SAAS,CACnB;IAC1B,mBAAsB;IACtB,wBAA2B;IAC3B,kBAAqB;IACrB;;0CAEsC;IACtC,oBADU,CAAC,MAAM,IAAI,CAAC,GAAG,SAAS,CACC;IACnC;;2DAEuD;IACvD,uBADU,CAAC,CAAC,GAAG,IAAI,EAAE,KAAK,CAAC,OAAC,CAAC,KAAK,IAAI,CAAC,GAAG,SAAS,CACb;IACtC;;sHAEkH;IAClH,eADU,OAAO,qBAAqB,EAAE,OAAO,GAAG,OAAO,gCAAgC,EAAE,OAAO,GAAG,SAAS,CAChF;IAGhC;;;OAGG;IACH,SAFa,OAAO,CAAC,IAAI,CAAC,CAqEzB;IAED;;;OAGG;IACH,QAFa,OAAO,CAAC,IAAI,CAAC,CAWzB;IAED;;yBAEqB;IACrB,iBADa,IAAI,CAKhB;IAED;;yBAEqB;IACrB,gBADa,IAAI,CAYhB;IAED;;yBAEqB;IACrB,6BADa,IAAI,CAYhB;IAED;;kCAE8B;IAC9B,wBADa,OAAO,CAAC,IAAI,CAAC,CAOzB;IAED;;kCAE8B;IAC9B,sCADa,OAAO,CAAC,IAAI,CAAC,CAOzB;IAED;;kCAE8B;IAC9B,gBADa,OAAO,CAAC,IAAI,CAAC,CAOzB;IAED;;;OAGG;IACH,WAFa,MAAM,CAIlB;IAED;;;;;;;;;;OAUG;IACH,0BAFa,IAAI,CA2BhB;IAED;;;;;;OAMG;IACH,mBAFa,IAAI,CAiBhB;IAED;;;;OAIG;IACH,0BAHW,OAAO,KAAK,EAAE,MAAM,GAClB,IAAI,CA0BhB;IAED;;;;;;;OAOG;IACH,oDALG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;QACW,IAAI,EAA9D,OAAO,YAAY,EAAE,uBAAuB,GAAG,IAAI;KAC3D,GAAU,OAAO,YAAY,EAAE,uBAAuB,GAAG,IAAI,CAS/D;IAED;;;;;;OAMG;IACH,sDAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,OAAO,YAAY,EAAE,uBAAuB,GAAG,IAAI,CAgB/D;IAED;;;;;;;;;;;;;OAaG;IACH,iCAHW,UAAU,GACR,OAAO,CAAC,IAAI,CAAC,CAqBzB;IAED;;;;;;OAMG;IACH,oDAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,IAAI,CAMhB;IAED;;;;;;OAMG;IACH,oDAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,IAAI,CAsBhB;IAED;;;;;;OAMG;IACH,sDAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,IAAI,CAWhB;IAED;;;;;;OAMG;IACH,4CAJG;QAAyB,UAAU,EAA3B,UAAU;QAC2C,OAAO,EAA5D,OAAO,YAAY,EAAE,yBAAyB;KACtD,GAAU,IAAI,CAYhB;IAED;;;;;OAKG;IACH,sCAHG;QAAyB,UAAU,EAA3B,UAAU;KAClB,GAAU,IAAI,CAOhB;IAED;;;;OAIG;IACH,kCAHW,UAAU,GACR,OAAO,CAAC,IAAI,CAAC,CAiBzB;IAED;;;;OAIG;IACH,+BAHW,UAAU,GACR,OAAO,CAAC,IAAI,CAAC,CAiBzB;IAED;;;;;;;OAOG;IACH,8CALG;QAAqB,SAAS,EAAtB,MAAM;QACO,KAAK,EAAlB,MAAM;QACW,MAAM,EAAvB,UAAU;KAClB,GAAU,OAAO,CAAC,IAAI,CAAC,CAQzB;IAED;;;;;;OAMG;IACH,qCAJG;QAAqB,SAAS,EAAtB,MAAM;QACO,KAAK,EAAlB,MAAM;KACd,GAAU,IAAI,CAUhB;IAED;;;;OAIG;IACH,kCAHW,OAAC,GACC,IAAI,CAUhB;IAED;;;;;;OAMG;IACH,gCAHW,OAAC,GACC,IAAI,CAUhB;IAED;;;;;;OAMG;IACH,wCAJG;QAAyB,UAAU,EAA3B,UAAU;QAC6C,OAAO,EAA9D,OAAO,YAAY,EAAE,2BAA2B;KACxD,GAAU,OAAO,CAAC,IAAI,CAAC,CAiBzB;IAED;;;;;;OAMG;IACH,4CAJG;QAAyB,UAAU,EAA3B,UAAU;QAC8C,OAAO,EAA/D,OAAO,YAAY,EAAE,4BAA4B;KACzD,GAAU,OAAO,CAAC,IAAI,CAAC,CAkBzB;IAED;;;;;;OAMG;IACH,0CAJG;QAAyB,UAAU,EAA3B,UAAU;QAC4C,OAAO,EAA7D,OAAO,YAAY,EAAE,0BAA0B;KACvD,GAAU,OAAO,CAAC,IAAI,CAAC,CAkCzB;IAED;;;;OAIG;IACH,6EAHW;QAAC,KAAK,EAAE,OAAC,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,OAAO,YAAY,EAAE,gBAAgB,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;KAAC,GACnH,IAAI,CAyBhB;IAED;;;;;;;;OAQG;IACH,oCAHW;QAAC,GAAG,EAAE,OAAO,YAAY,EAAE,gBAAgB,CAAA;KAAC,GAC1C,IAAI,CAsBhB;IAED;;;;OAIG;IACH,8BAHW,OAAC,GACC,KAAK,CAMjB;IAED;;;;OAIG;IACH,gCAHW,OAAC,GACC,KAAK,CASjB;IAED;;;;OAIG;IACH,kCAHW,OAAC,GACC,MAAM,CAMlB;IAED;;;;OAIG;IACH,yBAHW,OAAC,GACC,KAAK,IAAI,MAAM,CAI3B;IAED;;;;;;OAMG;IACH,oDAJG;QAAgB,KAAK,EAAb,OAAC;QACW,eAAe,EAA3B,KAAK;KACb,GAAU,IAAI,CAIhB;IAED;;;;;;;;;;;;;;;;OAgBG;IACH,UAFa,OAAO,CAAC,IAAI,CAAC,CAQzB;IAED;;;OAGG;IACH,eAFa,OAAO,CAQnB;IAED;;;;;OAKG;IACH,0BAHG;QAAsB,OAAO,EAArB,OAAO;KACf,GAAU,OAAO,CAAC,IAAI,CAAC,CAOzB;IAED;;;OAGG;IACH,6BAFa,OAAO,CAAC,IAAI,CAAC,CAYzB;IAED;;yBAEqB;IACrB,yBADa,IAAI,CAUhB;IAED;;;OAGG;IACH,+BAFa,OAAO,CAOnB;IAED;;;OAGG;IACH,mBAFa,OAAO,CAAC,OAAO,CAAC,CAQ5B;IAED;;;OAGG;IACH,iBAFa,OAAO,CAAC,OAAO,CAAC,CAW5B;IAED;;;OAGG;IACH,6BAFa,OAAO,CAAC,OAAO,CAAC,CAU5B;IAED;;;;;;OAMG;IACH,uBAFa,IAAI,CAWhB;IAED;;;OAGG;IACH,oBAFa,OAAO,CAAC,IAAI,CAAC,CAgBzB;IAED;;;;OAIG;IACH,cAFa,OAAO,CAAC,IAAI,CAAC,CAwDzB;IAED;;;OAGG;IACH,mCAFa,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CASjE;IAED;;;OAGG;IACH,6BAFa,OAAO,YAAY,EAAE,0BAA0B,EAAE,CAU7D;IAED;;;;;;OAMG;IACH,uDAJG;QAAmE,cAAc,EAAzE,GAAG,CAAC,OAAO,YAAY,EAAE,0BAA0B,CAAC;QACnC,MAAM,EAAvB,UAAU;KAClB,GAAU,IAAI,CAQhB;IAED;;;;OAIG;IACH,uBAHW,OAAO,YAAY,EAAE,gBAAgB,GACnC,UAAU,GAAG,SAAS,CAMlC;IAED;;;;;;OAMG;IACH,mCAJG;QAAoD,GAAG,EAA/C,OAAO,YAAY,EAAE,gBAAgB;QACpB,MAAM,EAAvB,UAAU;KAClB,GAAU,OAAO,CAUnB;IAED;;;;;;OAMG;IACH,sBAFa,OAAO,CAAC,IAAI,CAAC,CAoBzB;IAED,+BA0BC;IAED;;;;;;;;OAQG;IACH,sBAFa,OAAO,CAAC,IAAI,CAAC,CAgCzB;CACF;;;;;;;;mBAttCa,OAAO,YAAY,EAAE,0BAA0B;;;;aAC/C,CAAC,MAAM,EAAE,UAAU,KAAK,OAAO;;gCA3Bb,YAAY;mBACzB,cAAc;uBAHV,kBAAkB;gBADzB,KAAK;oCAEe,gBAAgB"}
|