velocious 1.0.519 → 1.0.521
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -0
- package/build/background-jobs/json-socket.js +12 -0
- package/build/background-jobs/main.js +78 -23
- package/build/background-jobs/store.js +127 -5
- package/build/background-jobs/types.js +4 -2
- package/build/background-jobs/worker.js +113 -23
- package/build/configuration-types.js +29 -6
- package/build/configuration.js +1 -1
- package/build/environment-handlers/node/cli/commands/generate/base-models.js +1 -1
- package/build/jobs/prune-terminal-background-jobs.js +67 -0
- package/build/src/background-jobs/json-socket.d.ts +12 -0
- package/build/src/background-jobs/json-socket.d.ts.map +1 -1
- package/build/src/background-jobs/json-socket.js +13 -1
- package/build/src/background-jobs/main.d.ts +18 -9
- package/build/src/background-jobs/main.d.ts.map +1 -1
- package/build/src/background-jobs/main.js +73 -26
- package/build/src/background-jobs/store.d.ts +30 -0
- package/build/src/background-jobs/store.d.ts.map +1 -1
- package/build/src/background-jobs/store.js +110 -5
- package/build/src/background-jobs/types.d.ts +14 -3
- package/build/src/background-jobs/types.d.ts.map +1 -1
- package/build/src/background-jobs/types.js +5 -3
- package/build/src/background-jobs/worker.d.ts +55 -7
- package/build/src/background-jobs/worker.d.ts.map +1 -1
- package/build/src/background-jobs/worker.js +108 -22
- package/build/src/configuration-types.d.ts +71 -11
- package/build/src/configuration-types.d.ts.map +1 -1
- package/build/src/configuration-types.js +30 -7
- package/build/src/configuration.d.ts +4 -2
- package/build/src/configuration.d.ts.map +1 -1
- package/build/src/configuration.js +2 -2
- package/build/src/environment-handlers/node/cli/commands/generate/base-models.js +2 -2
- package/build/src/jobs/prune-terminal-background-jobs.d.ts +23 -0
- package/build/src/jobs/prune-terminal-background-jobs.d.ts.map +1 -0
- package/build/src/jobs/prune-terminal-background-jobs.js +61 -0
- package/build/tsconfig.tsbuildinfo +1 -1
- package/package.json +1 -1
- package/src/background-jobs/json-socket.js +12 -0
- package/src/background-jobs/main.js +78 -23
- package/src/background-jobs/store.js +127 -5
- package/src/background-jobs/types.js +4 -2
- package/src/background-jobs/worker.js +113 -23
- package/src/configuration-types.js +29 -6
- package/src/configuration.js +1 -1
- package/src/environment-handlers/node/cli/commands/generate/base-models.js +1 -1
- package/src/jobs/prune-terminal-background-jobs.js +67 -0
package/README.md
CHANGED
|
@@ -2131,6 +2131,42 @@ await MyJob.performLaterWithOptions({
|
|
|
2131
2131
|
|
|
2132
2132
|
If a handed-off job does not report back within 2 hours, it is marked orphaned and re-queued if retries remain.
|
|
2133
2133
|
|
|
2134
|
+
Workers also send periodic heartbeats (and use TCP keepalive) so the main can drop a wedged or half-open worker that never fires a socket `close` and release its leases, and job slots are freed independently of durable, background result reporting so a transient outage can't wedge a worker or lose a completion. See [docs/background-jobs.md](docs/background-jobs.md#worker-liveness).
|
|
2135
|
+
|
|
2136
|
+
## Queues
|
|
2137
|
+
|
|
2138
|
+
Give a job a queue with `static queue = "..."` (or a `{queue}` job option; the option wins) and cap how many of that queue's jobs run in flight across the whole cluster under `backgroundJobs.queues`:
|
|
2139
|
+
|
|
2140
|
+
```js
|
|
2141
|
+
backgroundJobs: {
|
|
2142
|
+
queues: {
|
|
2143
|
+
builds: {maxConcurrent: 100}, // I/O-bound: can run well above the core count
|
|
2144
|
+
default: {maxConcurrent: 8} // CPU-bound: keep near the core count
|
|
2145
|
+
}
|
|
2146
|
+
}
|
|
2147
|
+
```
|
|
2148
|
+
|
|
2149
|
+
A job with no queue runs on `"default"`; a queue with no cap is unlimited. Caps are enforced through the durable per-key concurrency mechanism (the reserved `queue:<name>` key), hold regardless of how many worker processes run, and are reconciled against the existing backlog on startup when you change them. Scheduled jobs honor a job's `static queue` too.
|
|
2150
|
+
|
|
2151
|
+
Set `priority` (default `0`) to dispatch a queue ahead of lower-priority ones regardless of enqueue order, so a small time-critical queue is never starved by a flood of low-priority work sharing a worker pool. Unlike Sidekiq's strict queue ordering, priority composes with the caps: a higher-priority queue already at its `maxConcurrent` is skipped and dispatch falls through to the next eligible job. See [docs/background-jobs.md](docs/background-jobs.md#queues-per-queue-concurrency-caps).
|
|
2152
|
+
|
|
2153
|
+
## Retention
|
|
2154
|
+
|
|
2155
|
+
Terminal `background_jobs` rows are not deleted automatically unless you configure retention, so a busy app otherwise grows the table indefinitely. Set `backgroundJobs.retention` to prune old terminal rows:
|
|
2156
|
+
|
|
2157
|
+
```js
|
|
2158
|
+
backgroundJobs: {
|
|
2159
|
+
retention: {
|
|
2160
|
+
completedTtlMs: 7 * 24 * 60 * 60 * 1000, // default: 7 days (null/0 disables)
|
|
2161
|
+
failedTtlMs: 30 * 24 * 60 * 60 * 1000, // default: 30 days for failed/orphaned (null/0 disables)
|
|
2162
|
+
batchSize: 1000, // default: 1000 rows per delete batch
|
|
2163
|
+
sweepIntervalMs: 60 * 60 * 1000 // default: 1 hour
|
|
2164
|
+
}
|
|
2165
|
+
}
|
|
2166
|
+
```
|
|
2167
|
+
|
|
2168
|
+
`background-jobs-main` registers a built-in `velocious:prune-terminal-background-jobs` job on the scheduler when retention is enabled, so pruning runs as an ordinary scheduled/queued job (it needs a worker, appears in the job tables, and is bounded to one non-overlapping run). See [docs/background-jobs.md](docs/background-jobs.md#retention-pruning-old-job-rows).
|
|
2169
|
+
|
|
2134
2170
|
## Dashboard
|
|
2135
2171
|
|
|
2136
2172
|
Velocious ships a mountable read-only HTTP API for inspecting jobs (queued, running, completed, failed, orphaned and scheduled), similar in spirit to `sidekiq-web`. Mount it in your routes file the way `Sidekiq::Web` is mounted in Rails:
|
|
@@ -30,6 +30,18 @@ export default class JsonSocket extends EventEmitter {
|
|
|
30
30
|
* Narrows the runtime value to the documented type.
|
|
31
31
|
* @type {boolean} */
|
|
32
32
|
this.acceptsInlineJobs = true
|
|
33
|
+
/**
|
|
34
|
+
* Whether this worker advertised heartbeat support in its hello. Only
|
|
35
|
+
* heartbeat-capable workers are subject to the main's stale-liveness
|
|
36
|
+
* eviction; a legacy worker (e.g. mid rolling deploy) is exempt so its
|
|
37
|
+
* active leases are not released while it is still running them.
|
|
38
|
+
* @type {boolean} */
|
|
39
|
+
this.supportsHeartbeat = false
|
|
40
|
+
/**
|
|
41
|
+
* Last time (ms) the main saw any message from this worker socket; used by
|
|
42
|
+
* the main's liveness sweep to drop a wedged/silent worker.
|
|
43
|
+
* @type {number | undefined} */
|
|
44
|
+
this.lastSeenAt = undefined
|
|
33
45
|
this.buffer = ""
|
|
34
46
|
this.socket.setEncoding("utf8")
|
|
35
47
|
this.socket.on("data", (chunk) => this._onData(String(chunk)))
|
|
@@ -5,6 +5,7 @@ import JsonSocket from "./json-socket.js"
|
|
|
5
5
|
import BackgroundJobsScheduler from "./scheduler.js"
|
|
6
6
|
import BackgroundJobsStore from "./store.js"
|
|
7
7
|
import Logger from "../logger.js"
|
|
8
|
+
import PruneTerminalBackgroundJobsJob from "../jobs/prune-terminal-background-jobs.js"
|
|
8
9
|
|
|
9
10
|
/**
|
|
10
11
|
* Channel used by `background-jobs-main` to coordinate dispatch wake-ups
|
|
@@ -21,6 +22,10 @@ const DISPATCH_CHANNEL = "velocious-background-jobs-dispatch"
|
|
|
21
22
|
* scheduled-job timer here and re-arm when it expires.
|
|
22
23
|
*/
|
|
23
24
|
const MAX_TIMER_MS = 2_147_483_647 // ~24.8 days
|
|
25
|
+
/** A worker silent (no heartbeat/ready/report) longer than this is dropped. */
|
|
26
|
+
const WORKER_STALE_TIMEOUT_MS = 60000
|
|
27
|
+
/** How often the main scans workers for staleness. */
|
|
28
|
+
const WORKER_LIVENESS_SWEEP_MS = 15000
|
|
24
29
|
/**
|
|
25
30
|
* WorkerExecutionModeCapability type.
|
|
26
31
|
* @typedef {object} WorkerExecutionModeCapability
|
|
@@ -46,8 +51,10 @@ export default class BackgroundJobsMain {
|
|
|
46
51
|
* @param {import("../configuration.js").default} args.configuration - Configuration.
|
|
47
52
|
* @param {string} [args.host] - Hostname.
|
|
48
53
|
* @param {number} [args.port] - Port.
|
|
54
|
+
* @param {number} [args.workerStaleTimeoutMs] - Override how long a silent worker may go before being dropped (default 60000ms).
|
|
55
|
+
* @param {number} [args.workerLivenessSweepMs] - Override how often stale workers are swept for (default 15000ms).
|
|
49
56
|
*/
|
|
50
|
-
constructor({configuration, host, port}) {
|
|
57
|
+
constructor({configuration, host, port, workerStaleTimeoutMs, workerLivenessSweepMs}) {
|
|
51
58
|
this.configuration = configuration
|
|
52
59
|
const config = configuration.getBackgroundJobsConfig()
|
|
53
60
|
this.host = host || config.host
|
|
@@ -55,6 +62,10 @@ export default class BackgroundJobsMain {
|
|
|
55
62
|
this.dispatchStrategy = config.dispatchStrategy
|
|
56
63
|
this.pollIntervalMs = config.pollIntervalMs
|
|
57
64
|
this.retention = config.retention
|
|
65
|
+
// A worker that stops sending anything (heartbeat/ready/report) for this
|
|
66
|
+
// long is treated as wedged/dead: its leases are released and it is dropped.
|
|
67
|
+
this.workerStaleTimeoutMs = typeof workerStaleTimeoutMs === "number" && workerStaleTimeoutMs >= 1 ? workerStaleTimeoutMs : WORKER_STALE_TIMEOUT_MS
|
|
68
|
+
this.workerLivenessSweepMs = typeof workerLivenessSweepMs === "number" && workerLivenessSweepMs >= 1 ? workerLivenessSweepMs : WORKER_LIVENESS_SWEEP_MS
|
|
58
69
|
this.store = new BackgroundJobsStore({configuration, databaseIdentifier: config.databaseIdentifier})
|
|
59
70
|
this.logger = new Logger(this)
|
|
60
71
|
/**
|
|
@@ -91,8 +102,8 @@ export default class BackgroundJobsMain {
|
|
|
91
102
|
this._orphanTimer = undefined
|
|
92
103
|
/**
|
|
93
104
|
* Narrows the runtime value to the documented type.
|
|
94
|
-
* @type {ReturnType<typeof
|
|
95
|
-
this.
|
|
105
|
+
* @type {ReturnType<typeof setInterval> | undefined} */
|
|
106
|
+
this._workerStaleTimer = undefined
|
|
96
107
|
/**
|
|
97
108
|
* Narrows the runtime value to the documented type.
|
|
98
109
|
* @type {BackgroundJobsScheduler | undefined} */
|
|
@@ -144,9 +155,9 @@ export default class BackgroundJobsMain {
|
|
|
144
155
|
void this._sweepOrphans()
|
|
145
156
|
}, 60000)
|
|
146
157
|
|
|
147
|
-
this.
|
|
148
|
-
void this.
|
|
149
|
-
}, this.
|
|
158
|
+
this._workerStaleTimer = setInterval(() => {
|
|
159
|
+
void this._sweepStaleWorkers()
|
|
160
|
+
}, this.workerLivenessSweepMs)
|
|
150
161
|
|
|
151
162
|
this.scheduler = new BackgroundJobsScheduler({
|
|
152
163
|
configuration: this.configuration,
|
|
@@ -154,7 +165,10 @@ export default class BackgroundJobsMain {
|
|
|
154
165
|
await this.store.enqueue({
|
|
155
166
|
jobName: jobClass.jobName(),
|
|
156
167
|
args,
|
|
157
|
-
|
|
168
|
+
// Fold in the job class's static `queue` (as performLater* do) so a
|
|
169
|
+
// scheduled job with `static queue = "..."` lands on its queue and
|
|
170
|
+
// honors the configured cap without every schedule repeating it.
|
|
171
|
+
options: jobClass._withQueue(options)
|
|
158
172
|
})
|
|
159
173
|
this._notifyEnqueued()
|
|
160
174
|
await this._drain()
|
|
@@ -162,6 +176,16 @@ export default class BackgroundJobsMain {
|
|
|
162
176
|
})
|
|
163
177
|
await this.scheduler.start()
|
|
164
178
|
|
|
179
|
+
// Retention pruning runs as an ordinary scheduled job on the normal
|
|
180
|
+
// scheduler (so it is visible in the job tables and dispatched to a
|
|
181
|
+
// worker), rather than a hidden in-process timer. Skipped when retention
|
|
182
|
+
// is disabled. The scheduler owns the timer, so scheduler.stop() clears it.
|
|
183
|
+
const retentionSchedule = PruneTerminalBackgroundJobsJob.scheduleConfiguration(this.retention)
|
|
184
|
+
|
|
185
|
+
if (retentionSchedule) {
|
|
186
|
+
this.scheduler.scheduleJob({jobConfiguration: retentionSchedule, jobKey: "velociousPruneTerminalBackgroundJobs"})
|
|
187
|
+
}
|
|
188
|
+
|
|
165
189
|
// Startup catch-up: drain anything that was waiting before this
|
|
166
190
|
// process came up. In beacon mode this is also the safety net for
|
|
167
191
|
// races between attaching the connect listener and the initial
|
|
@@ -206,12 +230,12 @@ export default class BackgroundJobsMain {
|
|
|
206
230
|
if (this._scheduledTimer) clearTimeout(this._scheduledTimer)
|
|
207
231
|
if (this._errorRetryTimer) clearTimeout(this._errorRetryTimer)
|
|
208
232
|
if (this._orphanTimer) clearInterval(this._orphanTimer)
|
|
209
|
-
if (this.
|
|
233
|
+
if (this._workerStaleTimer) clearInterval(this._workerStaleTimer)
|
|
210
234
|
this._pollTimer = undefined
|
|
211
235
|
this._scheduledTimer = undefined
|
|
212
236
|
this._errorRetryTimer = undefined
|
|
213
237
|
this._orphanTimer = undefined
|
|
214
|
-
this.
|
|
238
|
+
this._workerStaleTimer = undefined
|
|
215
239
|
}
|
|
216
240
|
|
|
217
241
|
/**
|
|
@@ -394,6 +418,8 @@ export default class BackgroundJobsMain {
|
|
|
394
418
|
if (message.role === "worker") {
|
|
395
419
|
jsonSocket.workerId = message.workerId
|
|
396
420
|
jsonSocket.supportsHandoffIdReporting = message.supportsHandoffIdReporting === true
|
|
421
|
+
jsonSocket.supportsHeartbeat = message.supportsHeartbeat === true
|
|
422
|
+
jsonSocket.lastSeenAt = Date.now()
|
|
397
423
|
this.workers.add(jsonSocket)
|
|
398
424
|
this.workerHandoffs.set(jsonSocket, new Map())
|
|
399
425
|
}
|
|
@@ -422,6 +448,14 @@ export default class BackgroundJobsMain {
|
|
|
422
448
|
* @returns {void}
|
|
423
449
|
*/
|
|
424
450
|
_handleWorkerSocketMessage({jsonSocket, message}) {
|
|
451
|
+
// Any message from the worker proves it is alive; the liveness sweep uses
|
|
452
|
+
// this to detect a wedged/silent worker.
|
|
453
|
+
jsonSocket.lastSeenAt = Date.now()
|
|
454
|
+
|
|
455
|
+
if (message?.type === "heartbeat") {
|
|
456
|
+
return
|
|
457
|
+
}
|
|
458
|
+
|
|
425
459
|
if (message?.type === "ready") {
|
|
426
460
|
this._handleWorkerReady({jsonSocket, message})
|
|
427
461
|
return
|
|
@@ -1098,22 +1132,43 @@ export default class BackgroundJobsMain {
|
|
|
1098
1132
|
}
|
|
1099
1133
|
|
|
1100
1134
|
/**
|
|
1101
|
-
*
|
|
1102
|
-
*
|
|
1103
|
-
*
|
|
1104
|
-
*
|
|
1135
|
+
* Drops workers that have gone silent past `workerStaleTimeoutMs` (no
|
|
1136
|
+
* heartbeat, ready, or report). A wedged worker keeps its socket open, so the
|
|
1137
|
+
* `close`-based cleanup never fires and its in-flight leases — and the whole
|
|
1138
|
+
* queue — stay stuck until a human notices. Releasing the lost worker's
|
|
1139
|
+
* leases lets its jobs run elsewhere and stops dispatch to it; the worker's
|
|
1140
|
+
* own process lifecycle is the supervisor's concern.
|
|
1141
|
+
* @returns {Promise<void>} - Resolves after the sweep.
|
|
1105
1142
|
*/
|
|
1106
|
-
async
|
|
1107
|
-
|
|
1108
|
-
const deleted = await this.store.pruneTerminalJobs({
|
|
1109
|
-
completedTtlMs: this.retention.completedTtlMs,
|
|
1110
|
-
failedTtlMs: this.retention.failedTtlMs,
|
|
1111
|
-
batchSize: this.retention.batchSize
|
|
1112
|
-
})
|
|
1143
|
+
async _sweepStaleWorkers() {
|
|
1144
|
+
if (this._stopped) return
|
|
1113
1145
|
|
|
1114
|
-
|
|
1115
|
-
|
|
1116
|
-
|
|
1146
|
+
const cutoff = Date.now() - this.workerStaleTimeoutMs
|
|
1147
|
+
/** @type {JsonSocket[]} */
|
|
1148
|
+
const stale = []
|
|
1149
|
+
|
|
1150
|
+
for (const worker of this.workers) {
|
|
1151
|
+
// Only evict heartbeat-capable workers. A legacy worker (e.g. one from the
|
|
1152
|
+
// previous release during a rolling deploy) never heartbeats, so evicting
|
|
1153
|
+
// it on silence would wrongly release the leases of a job it is still
|
|
1154
|
+
// running. Its disconnect is still handled by the socket `close` path.
|
|
1155
|
+
if (!worker.supportsHeartbeat) continue
|
|
1156
|
+
|
|
1157
|
+
const lastSeenAt = typeof worker.lastSeenAt === "number" ? worker.lastSeenAt : 0
|
|
1158
|
+
|
|
1159
|
+
if (lastSeenAt <= cutoff) stale.push(worker)
|
|
1160
|
+
}
|
|
1161
|
+
|
|
1162
|
+
for (const worker of stale) {
|
|
1163
|
+
this.logger.warn(() => ["Dropping stale background jobs worker", {workerId: worker.workerId, lastSeenAt: worker.lastSeenAt}])
|
|
1164
|
+
|
|
1165
|
+
try {
|
|
1166
|
+
worker.close()
|
|
1167
|
+
} catch {
|
|
1168
|
+
// Already closing; the lease release below is what matters.
|
|
1169
|
+
}
|
|
1170
|
+
|
|
1171
|
+
await this._handleWorkerSocketClosed(worker)
|
|
1117
1172
|
}
|
|
1118
1173
|
}
|
|
1119
1174
|
}
|
|
@@ -51,6 +51,7 @@ export default class BackgroundJobsStore {
|
|
|
51
51
|
this.databaseIdentifier = databaseIdentifier
|
|
52
52
|
this.logger = new Logger(this)
|
|
53
53
|
this._readyPromise = null
|
|
54
|
+
this._queueConcurrencyReconciled = false
|
|
54
55
|
}
|
|
55
56
|
|
|
56
57
|
/**
|
|
@@ -101,8 +102,26 @@ export default class BackgroundJobsStore {
|
|
|
101
102
|
const argsJson = JSON.stringify(args || [])
|
|
102
103
|
const queue = this._normalizeQueue(options)
|
|
103
104
|
const concurrency = this._resolveConcurrency(options, queue)
|
|
105
|
+
/** @type {string} */
|
|
106
|
+
let resultJobId = jobId
|
|
104
107
|
|
|
105
108
|
await this._withDb(async (db) => {
|
|
109
|
+
if (options?.deduplicateWhileQueued && concurrency?.concurrencyKey) {
|
|
110
|
+
const existing = await db
|
|
111
|
+
.newQuery()
|
|
112
|
+
.from(JOBS_TABLE)
|
|
113
|
+
.select("id")
|
|
114
|
+
.where({status: "queued", concurrency_key: concurrency.concurrencyKey})
|
|
115
|
+
.limit(1)
|
|
116
|
+
.results()
|
|
117
|
+
|
|
118
|
+
if (existing[0]) {
|
|
119
|
+
resultJobId = String(/** @type {Record<string, ?>} */ (existing[0]).id)
|
|
120
|
+
|
|
121
|
+
return
|
|
122
|
+
}
|
|
123
|
+
}
|
|
124
|
+
|
|
106
125
|
if (concurrency) {
|
|
107
126
|
if (concurrency.queueDerived) {
|
|
108
127
|
await this._ensureQueueConcurrencyKey(db, concurrency)
|
|
@@ -130,7 +149,7 @@ export default class BackgroundJobsStore {
|
|
|
130
149
|
})
|
|
131
150
|
})
|
|
132
151
|
|
|
133
|
-
return
|
|
152
|
+
return resultJobId
|
|
134
153
|
}
|
|
135
154
|
|
|
136
155
|
/**
|
|
@@ -206,6 +225,12 @@ export default class BackgroundJobsStore {
|
|
|
206
225
|
query = query.where({execution_mode: executionModes})
|
|
207
226
|
}
|
|
208
227
|
|
|
228
|
+
if (scheduledAtOperator === "<=") {
|
|
229
|
+
const priorityOrder = this._queuePriorityOrderSql(db)
|
|
230
|
+
|
|
231
|
+
if (priorityOrder) query = query.order(`${priorityOrder} DESC`)
|
|
232
|
+
}
|
|
233
|
+
|
|
209
234
|
query = query
|
|
210
235
|
.order("scheduled_at_ms ASC")
|
|
211
236
|
.order("created_at_ms ASC")
|
|
@@ -219,6 +244,40 @@ export default class BackgroundJobsStore {
|
|
|
219
244
|
return this._normalizeJobRow(row)
|
|
220
245
|
}
|
|
221
246
|
|
|
247
|
+
/**
|
|
248
|
+
* Builds a raw SQL ORDER BY expression ranking queued jobs by their queue's
|
|
249
|
+
* configured priority (`backgroundJobs.queues[queue].priority`, default `0`),
|
|
250
|
+
* so the dispatcher picks higher-priority queues first regardless of enqueue
|
|
251
|
+
* order. Only applied to the dispatch path (`scheduledAtOperator === "<="`);
|
|
252
|
+
* the future-scheduled lookup must stay strictly time-ordered. Composes with
|
|
253
|
+
* the concurrency EXISTS filter: a higher-priority queue already at its cap is
|
|
254
|
+
* filtered out, so dispatch falls through to the next eligible lower-priority
|
|
255
|
+
* job. Returns null when no queue configures a non-zero priority so the plain
|
|
256
|
+
* FIFO ordering is left untouched (and no needless filesort is introduced).
|
|
257
|
+
* @param {import("../database/drivers/base.js").default} db - Database connection.
|
|
258
|
+
* @returns {string | null} - Raw SQL CASE expression, or null when no queue is prioritized.
|
|
259
|
+
*/
|
|
260
|
+
_queuePriorityOrderSql(db) {
|
|
261
|
+
const queues = this.configuration.getBackgroundJobsConfig().queues || {}
|
|
262
|
+
/** @type {Array<[string, number]>} */
|
|
263
|
+
const prioritized = []
|
|
264
|
+
|
|
265
|
+
for (const [queue, queueConfig] of Object.entries(queues)) {
|
|
266
|
+
const priority = queueConfig?.priority
|
|
267
|
+
|
|
268
|
+
if (Number.isFinite(priority) && Number(priority) !== 0) prioritized.push([queue, Number(priority)])
|
|
269
|
+
}
|
|
270
|
+
|
|
271
|
+
if (prioritized.length === 0) return null
|
|
272
|
+
|
|
273
|
+
const queueColumn = db.quoteColumn("queue")
|
|
274
|
+
const whens = prioritized
|
|
275
|
+
.map(([queue, priority]) => `WHEN ${db.quote(queue)} THEN ${priority}`)
|
|
276
|
+
.join(" ")
|
|
277
|
+
|
|
278
|
+
return `CASE COALESCE(${queueColumn}, ${db.quote(DEFAULT_QUEUE)}) ${whens} ELSE 0 END`
|
|
279
|
+
}
|
|
280
|
+
|
|
222
281
|
/**
|
|
223
282
|
* Runs get job.
|
|
224
283
|
* @param {string} jobId - Job id.
|
|
@@ -648,6 +707,7 @@ export default class BackgroundJobsStore {
|
|
|
648
707
|
if (alreadyApplied && await db.tableExists(JOBS_TABLE)) {
|
|
649
708
|
await this._ensureJobsTableColumns(db)
|
|
650
709
|
await this._ensureConcurrencyTable(db)
|
|
710
|
+
await this._reconcileQueueConcurrency(db)
|
|
651
711
|
await this._reconcileConcurrency(db)
|
|
652
712
|
return
|
|
653
713
|
}
|
|
@@ -655,6 +715,7 @@ export default class BackgroundJobsStore {
|
|
|
655
715
|
await this._applyMigrations(db)
|
|
656
716
|
await this._ensureJobsTableColumns(db)
|
|
657
717
|
await this._ensureConcurrencyTable(db)
|
|
718
|
+
await this._reconcileQueueConcurrency(db)
|
|
658
719
|
await this._reconcileConcurrency(db)
|
|
659
720
|
|
|
660
721
|
if (alreadyApplied) return
|
|
@@ -834,16 +895,15 @@ export default class BackgroundJobsStore {
|
|
|
834
895
|
* @returns {Promise<void>} - Resolves when ensured.
|
|
835
896
|
*/
|
|
836
897
|
async _ensureQueueColumn(db) {
|
|
837
|
-
const table = await db.getTableByNameOrFail(JOBS_TABLE)
|
|
838
|
-
|
|
839
|
-
if (await table.getColumnByName("queue")) return
|
|
840
|
-
|
|
841
898
|
const lockName = `${MIGRATION_SCOPE}:queue_column`
|
|
842
899
|
const acquired = await db.acquireAdvisoryLock(lockName)
|
|
843
900
|
|
|
844
901
|
if (!acquired) throw new Error("Failed to acquire background jobs queue schema lock")
|
|
845
902
|
|
|
846
903
|
try {
|
|
904
|
+
// SQL Server schema reads can deadlock with a concurrent ALTER TABLE, so
|
|
905
|
+
// acquire the lock before inspecting the column rather than only
|
|
906
|
+
// protecting the mutation (mirrors the concurrency-column migration).
|
|
847
907
|
db.clearSchemaCache()
|
|
848
908
|
const lockedTable = await db.getTableByNameOrFail(JOBS_TABLE)
|
|
849
909
|
|
|
@@ -1101,6 +1161,9 @@ export default class BackgroundJobsStore {
|
|
|
1101
1161
|
if (typeof key !== "string" || key.length === 0 || !Number.isInteger(cap) || Number(cap) <= 0) {
|
|
1102
1162
|
throw new Error("background job concurrencyKey and maxConcurrency must be paired; concurrencyKey must be non-empty and maxConcurrency must be a positive integer")
|
|
1103
1163
|
}
|
|
1164
|
+
if (key.startsWith(QUEUE_CONCURRENCY_KEY_PREFIX)) {
|
|
1165
|
+
throw new Error(`background job concurrencyKey must not start with the reserved "${QUEUE_CONCURRENCY_KEY_PREFIX}" prefix, which is reserved for queue-derived concurrency caps`)
|
|
1166
|
+
}
|
|
1104
1167
|
return {concurrencyKey: key, maxConcurrency: Number(cap)}
|
|
1105
1168
|
}
|
|
1106
1169
|
|
|
@@ -1277,6 +1340,65 @@ export default class BackgroundJobsStore {
|
|
|
1277
1340
|
)
|
|
1278
1341
|
}
|
|
1279
1342
|
|
|
1343
|
+
/**
|
|
1344
|
+
* Reconciles queue-derived concurrency with the current configuration, once
|
|
1345
|
+
* per process. Enqueue only consults config for new jobs, so a cap added,
|
|
1346
|
+
* removed, or changed while a backlog exists otherwise leaves persisted rows
|
|
1347
|
+
* stale: pre-cap jobs keep a null key and bypass the cap, post-removal jobs
|
|
1348
|
+
* stay capped under a now-unconfigured key, and a changed numeric cap stays
|
|
1349
|
+
* stale until the next enqueue. Bring the durable state in line with config:
|
|
1350
|
+
* sync each configured queue's stored cap, adopt not-yet-keyed non-terminal
|
|
1351
|
+
* jobs onto their queue key, and release non-terminal jobs from queue keys
|
|
1352
|
+
* whose queue is no longer capped. Runs before {@link _reconcileConcurrency}
|
|
1353
|
+
* so the rebuilt active counts reflect the adopted/released keys.
|
|
1354
|
+
* @param {import("../database/drivers/base.js").default} db - Database connection.
|
|
1355
|
+
* @returns {Promise<void>} - Resolves when reconciled.
|
|
1356
|
+
*/
|
|
1357
|
+
async _reconcileQueueConcurrency(db) {
|
|
1358
|
+
if (this._queueConcurrencyReconciled) return
|
|
1359
|
+
if (!(await db.tableExists(CONCURRENCY_TABLE))) return
|
|
1360
|
+
|
|
1361
|
+
const queuesConfig = this.configuration.getBackgroundJobsConfig().queues || {}
|
|
1362
|
+
const jobsTable = db.quoteTable(JOBS_TABLE)
|
|
1363
|
+
const keyColumn = db.quoteColumn("concurrency_key")
|
|
1364
|
+
const capColumn = db.quoteColumn("max_concurrency")
|
|
1365
|
+
const queueColumn = db.quoteColumn("queue")
|
|
1366
|
+
const nonTerminal = `${db.quoteColumn("status")} IN (${db.quote("queued")}, ${db.quote("handed_off")})`
|
|
1367
|
+
/** @type {Set<string>} */
|
|
1368
|
+
const cappedQueues = new Set()
|
|
1369
|
+
|
|
1370
|
+
for (const queue of Object.keys(queuesConfig)) {
|
|
1371
|
+
const cap = this._queueMaxConcurrency(queue)
|
|
1372
|
+
|
|
1373
|
+
if (cap === null) continue
|
|
1374
|
+
|
|
1375
|
+
cappedQueues.add(queue)
|
|
1376
|
+
const concurrencyKey = `${QUEUE_CONCURRENCY_KEY_PREFIX}${queue}`
|
|
1377
|
+
|
|
1378
|
+
await this._ensureQueueConcurrencyKey(db, {concurrencyKey, maxConcurrency: cap})
|
|
1379
|
+
await db.query(
|
|
1380
|
+
`UPDATE ${jobsTable} SET ${keyColumn} = ${db.quote(concurrencyKey)}, ${capColumn} = ${Number(cap)} ` +
|
|
1381
|
+
`WHERE ${queueColumn} = ${db.quote(queue)} AND ${keyColumn} IS NULL AND ${nonTerminal}`
|
|
1382
|
+
)
|
|
1383
|
+
}
|
|
1384
|
+
|
|
1385
|
+
const concurrencyRows = await db.newQuery().from(CONCURRENCY_TABLE).select("concurrency_key").results()
|
|
1386
|
+
|
|
1387
|
+
for (const row of concurrencyRows) {
|
|
1388
|
+
const concurrencyKey = String(/** @type {Record<string, ?>} */ (row).concurrency_key)
|
|
1389
|
+
|
|
1390
|
+
if (!concurrencyKey.startsWith(QUEUE_CONCURRENCY_KEY_PREFIX)) continue
|
|
1391
|
+
if (cappedQueues.has(concurrencyKey.slice(QUEUE_CONCURRENCY_KEY_PREFIX.length))) continue
|
|
1392
|
+
|
|
1393
|
+
await db.query(
|
|
1394
|
+
`UPDATE ${jobsTable} SET ${keyColumn} = NULL, ${capColumn} = NULL ` +
|
|
1395
|
+
`WHERE ${keyColumn} = ${db.quote(concurrencyKey)} AND ${nonTerminal}`
|
|
1396
|
+
)
|
|
1397
|
+
}
|
|
1398
|
+
|
|
1399
|
+
this._queueConcurrencyReconciled = true
|
|
1400
|
+
}
|
|
1401
|
+
|
|
1280
1402
|
/**
|
|
1281
1403
|
* Runs normalize number.
|
|
1282
1404
|
* @param {?} value - Input value.
|
|
@@ -16,6 +16,7 @@
|
|
|
16
16
|
* @property {string} [queue] - Queue name. Defaults to `"default"`. When the queue has a configured cap in `backgroundJobs.queues`, that cap is enforced cluster-wide.
|
|
17
17
|
* @property {string} [concurrencyKey] - Opaque non-empty key used to share a concurrency cap. Overrides any queue-derived cap.
|
|
18
18
|
* @property {number} [maxConcurrency] - Positive integer cap; must be paired with `concurrencyKey`.
|
|
19
|
+
* @property {boolean} [deduplicateWhileQueued] - When true (requires `concurrencyKey`), skip the enqueue if a still-queued job with the same `concurrencyKey` already exists, returning that job's id. Keeps an interval-scheduled recurring job (e.g. retention pruning) from piling up redundant queued rows when it runs slower than its interval or no worker is free.
|
|
19
20
|
*/
|
|
20
21
|
/**
|
|
21
22
|
* @typedef {object} BackgroundJobPayload
|
|
@@ -65,9 +66,10 @@
|
|
|
65
66
|
* @typedef {"worker" | "client" | "reporter"} BackgroundJobSocketRole
|
|
66
67
|
*/
|
|
67
68
|
/**
|
|
68
|
-
* @typedef {{type: "hello", role: BackgroundJobSocketRole, supportsHandoffIdReporting?: boolean, workerId?: string}} BackgroundJobHelloMessage
|
|
69
|
+
* @typedef {{type: "hello", role: BackgroundJobSocketRole, supportsHandoffIdReporting?: boolean, supportsHeartbeat?: boolean, workerId?: string}} BackgroundJobHelloMessage
|
|
69
70
|
* @typedef {{type: "ready", acceptsForked?: boolean, acceptsInline?: boolean, acceptsSpawned?: boolean}} BackgroundJobReadyMessage
|
|
70
71
|
* @typedef {{type: "draining"}} BackgroundJobDrainingMessage
|
|
72
|
+
* @typedef {{type: "heartbeat", workerId?: string}} BackgroundJobHeartbeatMessage
|
|
71
73
|
* @typedef {{type: "enqueue", jobName: string, args?: Array<?>, options?: BackgroundJobOptions}} BackgroundJobEnqueueMessage
|
|
72
74
|
* @typedef {{type: "enqueued", jobId: string}} BackgroundJobEnqueuedMessage
|
|
73
75
|
* @typedef {{type: "enqueue-error", error?: string}} BackgroundJobEnqueueErrorMessage
|
|
@@ -78,7 +80,7 @@
|
|
|
78
80
|
* @typedef {{type: "job-update-error", jobId: string, error?: string}} BackgroundJobUpdateErrorMessage
|
|
79
81
|
*/
|
|
80
82
|
/**
|
|
81
|
-
* @typedef {BackgroundJobHelloMessage | BackgroundJobReadyMessage | BackgroundJobDrainingMessage | BackgroundJobEnqueueMessage | BackgroundJobEnqueuedMessage | BackgroundJobEnqueueErrorMessage | BackgroundJobJobMessage | BackgroundJobCompleteMessage | BackgroundJobFailedMessage | BackgroundJobUpdatedMessage | BackgroundJobUpdateErrorMessage} BackgroundJobSocketMessage
|
|
83
|
+
* @typedef {BackgroundJobHelloMessage | BackgroundJobReadyMessage | BackgroundJobDrainingMessage | BackgroundJobHeartbeatMessage | BackgroundJobEnqueueMessage | BackgroundJobEnqueuedMessage | BackgroundJobEnqueueErrorMessage | BackgroundJobJobMessage | BackgroundJobCompleteMessage | BackgroundJobFailedMessage | BackgroundJobUpdatedMessage | BackgroundJobUpdateErrorMessage} BackgroundJobSocketMessage
|
|
82
84
|
*/
|
|
83
85
|
|
|
84
86
|
export const nothing = {}
|