@bymax-one/nest-queue 1.2.14 → 1.2.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -1
- package/README.md +41 -36
- package/dist/server/index.cjs +133 -42
- package/dist/server/index.d.cts +39 -17
- package/dist/server/index.d.ts +39 -17
- package/dist/server/index.mjs +133 -42
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -11,6 +11,29 @@ versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
11
11
|
|
|
12
12
|
---
|
|
13
13
|
|
|
14
|
+
## [1.2.15] — 2026-11-14
|
|
15
|
+
|
|
16
|
+
Against 1.2.14 the built declarations gain `ConnectionResolver.awaitReady()`; nothing else in the public API changes.
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
|
|
20
|
+
- **A call made while the connection is down now reports the documented error.** With the owned connection's offline queue off, a call on a queue already in use during a refused-connection outage rejected at once with ioredis's plain `Stream isn't writeable and enableOfflineQueue options is false`, so a host retrying on `CONNECTION_TIMEOUT` as documented never retried. It is now `CONNECTION_TIMEOUT` with `details.operation` and `details.reason: 'offline'`, and the ioredis error as `cause`.
|
|
21
|
+
- **The first call on a queue no longer waits for Redis without a deadline.** `getMetrics`, `getJob`, `getJobs`, the scheduler calls, `pauseQueue`, `resumeQueue` and `cleanQueue` on a queue name not yet used waited in BullMQ for the client to be ready, with no bound, for as long as the outage lasted. Every `QueueService` call is now bounded by `enqueueTimeoutMs` and rejects with `CONNECTION_TIMEOUT`.
|
|
22
|
+
- **Shutdown no longer leaves a worker running after its drain budget.** A handler that never settled kept the worker's lock-renewal timer alive after `app.close()`: the process did not exit by itself and logged `Connection is closed.` repeatedly. BullMQ ignores a forced close once a graceful one has started, so the worker is now paused first and closed with `close(true)` as its first close when the budget runs out. What the abandoned handler produces after the budget, a result or a failure, is dropped, so BullMQ cannot resume the stopped worker and reopen its connection. The handler's own promise still keeps running, since user code cannot be cancelled. A sandboxed worker is the exception: its job runs in a child process the registry cannot wrap, and in a measured run the host process did not exit by itself after the budget ended with such a job still running.
|
|
23
|
+
|
|
24
|
+
### Documentation
|
|
25
|
+
|
|
26
|
+
- `enqueueTimeoutMs` is documented as the per-call deadline of every `QueueService` call, and `queue.connection_timeout` as covering a connection that is down.
|
|
27
|
+
|
|
28
|
+
#### Apply to a derived backend
|
|
29
|
+
|
|
30
|
+
- If you match the ioredis error `Stream isn't writeable and enableOfflineQueue options is false` on a call, match `CONNECTION_TIMEOUT` instead; `details.reason` is `'offline'` and the original error is its `cause`.
|
|
31
|
+
- If a `getMetrics`, `getJob`, `getJobs`, scheduler, pause, resume or clean call relied on waiting out an outage on a queue's first use, it now rejects with `CONNECTION_TIMEOUT` after `enqueueTimeoutMs`; raise the option to wait longer.
|
|
32
|
+
- If you replace `ConnectionResolver` with your own implementation, add `awaitReady(): Promise<void>`: `QueueService` calls it before the first call on each queue name.
|
|
33
|
+
- If you parse the shutdown log lines, the wording changed: `forcing close` is gone and the summary reads `N worker(s) stopped without draining`.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
14
37
|
## [1.2.14] — 2026-11-14
|
|
15
38
|
|
|
16
39
|
Several defaults change in this release, so read the Apply note below. A boot that succeeds, an
|
|
@@ -1013,7 +1036,8 @@ v6 peer range.
|
|
|
1013
1036
|
|
|
1014
1037
|
---
|
|
1015
1038
|
|
|
1016
|
-
[Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.
|
|
1039
|
+
[Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.15...HEAD
|
|
1040
|
+
[1.2.15]: https://github.com/bymaxone/nest-queue/compare/v1.2.14...v1.2.15
|
|
1017
1041
|
[1.2.14]: https://github.com/bymaxone/nest-queue/compare/v1.2.13...v1.2.14
|
|
1018
1042
|
[1.2.13]: https://github.com/bymaxone/nest-queue/compare/v1.2.12...v1.2.13
|
|
1019
1043
|
[1.2.12]: https://github.com/bymaxone/nest-queue/compare/v1.2.11...v1.2.12
|
package/README.md
CHANGED
|
@@ -66,7 +66,7 @@ peer dependencies, so you control exact versions and the supply-chain surface st
|
|
|
66
66
|
- ✅ **Dual-mode connection** — bring your own `ioredis` client, or let the library open its own from a URL or an options object
|
|
67
67
|
- ✅ **Per-role retry policy** — `maxRetriesPerRequest: null` reaches only the duplicated Worker and QueueEvents connections; the producer keeps a finite retry limit, a 5 s command deadline and no offline queue on the connection the library opens, and `enqueue` has its own 10 s deadline, so a Redis outage rejects a call in seconds instead of holding the request; a shared client with `maxRetriesPerRequest: null` is the exception, and the library warns about it at boot
|
|
68
68
|
- ✅ **One namespace, four objects** — the configured `prefix` reaches `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer`, so producers and consumers can never end up on separate keyspaces
|
|
69
|
-
- ✅ **Bounded graceful shutdown** — a single ordered sequence drains every worker under
|
|
69
|
+
- ✅ **Bounded graceful shutdown** — a single ordered sequence drains every worker under one deadline and then stops any that are still busy, whose jobs stall and are retried; delivery is at-least-once by design, so handlers must be idempotent
|
|
70
70
|
- ✅ **No unlistened emitter** — every `Queue`, `Worker`, `QueueEvents` and `FlowProducer` the library creates carries a fallback `error` listener, so a transient Redis fault cannot become an uncaught exception in your process; your own `@OnWorkerEvent('error')` or `@OnQueueEvent('error')` still runs alongside it
|
|
71
71
|
|
|
72
72
|
### 🧩 Developer Experience
|
|
@@ -243,7 +243,7 @@ Every option passed to `forRoot` / `forRootAsync`. Only `connection` is required
|
|
|
243
243
|
| `shutdown.drainOnShutdown` | `boolean` | `false` | Development and test only: once the HTTP server has stopped, call `drain()` on every cached queue, which **removes** its waiting jobs (delayed and active jobs are left alone). Leave it `false` in production. |
|
|
244
244
|
| `connectionReadyTimeoutMs` | `number` | `10_000` | Mode B only: the budget for Redis to reach `ready`. A connection error while the client is still reconnecting does not end the wait; when the budget ends, startup fails with `CONNECTION_TIMEOUT` carrying `lastErrorCode` (a code such as `ECONNREFUSED`, never the message) and the client the library opened is closed. |
|
|
245
245
|
| `commandTimeoutMs` | `number` | `5_000` | Mode B only: the deadline for one Redis command on the connection the library opened (ioredis `commandTimeout`); an `options.commandTimeout` wins. A call that hits it rejects with `CONNECTION_TIMEOUT` (`details` carries `operation` and `timeoutMs`), as when `enqueueTimeoutMs` passes. Worker and QueueEvents connections have none. |
|
|
246
|
-
| `enqueueTimeoutMs` | `number` | `10_000` | The deadline for one `
|
|
246
|
+
| `enqueueTimeoutMs` | `number` | `10_000` | The deadline for one `QueueService` call that reaches Redis, in either mode, including the wait for the client to be ready on a queue's first call; it rejects with `CONNECTION_TIMEOUT`, and an `enqueue` may still complete in Redis, so retry with a stable `jobId` or `deduplication`. |
|
|
247
247
|
| `isGlobal` | `boolean` | `true` | Register the module globally, so `QueueService` is injectable without re-importing. |
|
|
248
248
|
|
|
249
249
|
> [!WARNING]
|
|
@@ -440,7 +440,7 @@ When a custom `jobId` already exists, or a `deduplication` rule keeps the job th
|
|
|
440
440
|
|
|
441
441
|
### A timed-out `enqueue` may still have been added
|
|
442
442
|
|
|
443
|
-
When Redis does not answer in time, whether within `commandTimeoutMs` on the connection the library opened or within `enqueueTimeoutMs`, `enqueue` and `enqueueBulk` reject with `CONNECTION_TIMEOUT`, but the library cannot cancel the command: if Redis applies it late, the job exists anyway. Retrying a call that timed out can therefore add the job twice. For anything that must not run twice, pass a deterministic `jobId` (for example one derived from the business key): BullMQ ignores a second add with the same `jobId` for as long as the first job is still stored, so a retry inside that window is safe. A completed job is removed by `removeOnComplete`, and from then on the same `jobId` adds a new job, so keep the retry inside that window or keep the handler idempotent. A `deduplication` rule gives the same guarantee for work that is allowed to repeat after a window.
|
|
443
|
+
When Redis does not answer in time, whether within `commandTimeoutMs` on the connection the library opened or within `enqueueTimeoutMs`, or the connection is down, `enqueue` and `enqueueBulk` reject with `CONNECTION_TIMEOUT`, but the library cannot cancel the command: if Redis applies it late, the job exists anyway. Retrying a call that timed out can therefore add the job twice. For anything that must not run twice, pass a deterministic `jobId` (for example one derived from the business key): BullMQ ignores a second add with the same `jobId` for as long as the first job is still stored, so a retry inside that window is safe. A completed job is removed by `removeOnComplete`, and from then on the same `jobId` adds a new job, so keep the retry inside that window or keep the handler idempotent. A `deduplication` rule gives the same guarantee for work that is allowed to repeat after a window.
|
|
444
444
|
|
|
445
445
|
### OpenTelemetry
|
|
446
446
|
|
|
@@ -465,10 +465,15 @@ It implements a **bounded drain protocol** with at-least-once semantics:
|
|
|
465
465
|
1. All workers are closed **in parallel**, within one budget (`drainTimeoutMs`, default 30 s) that
|
|
466
466
|
covers steps 1 to 4 together: however many workers are stuck, consumption stops after one budget,
|
|
467
467
|
not one per worker.
|
|
468
|
-
2.
|
|
469
|
-
first.
|
|
470
|
-
3. On timeout, the worker is
|
|
471
|
-
|
|
468
|
+
2. Each worker is paused (it stops fetching and waits for its active jobs) and then closed, or the
|
|
469
|
+
budget runs out — whichever comes first.
|
|
470
|
+
3. On timeout, the worker is stopped with `worker.close(true)` as its first close, so it does not
|
|
471
|
+
wait for the job and its lock renewal ends; the lock expires and the job stalls and is retried
|
|
472
|
+
by the next instance. What the abandoned handler produces afterwards, a result or a failure, is
|
|
473
|
+
dropped, so the stopped worker cannot restart its connection. The handler's own promise keeps
|
|
474
|
+
running, because user code cannot be cancelled. A sandboxed (file-based) worker is the
|
|
475
|
+
exception: its job runs in a child process the registry cannot wrap, and in a measured run the
|
|
476
|
+
host process did not exit by itself after the budget ended with such a job still running.
|
|
472
477
|
4. Queue-events and the connections the library opened for workers and queue-events are closed,
|
|
473
478
|
each within what is left of the budget; a connection that has not quit by then is disconnected.
|
|
474
479
|
5. Once NestJS has stopped the HTTP server (`onApplicationShutdown`), the flow producer, the
|
|
@@ -478,7 +483,7 @@ It implements a **bounded drain protocol** with at-least-once semantics:
|
|
|
478
483
|
|
|
479
484
|
**Handlers must be idempotent.** Use `jobId`, `deduplication`, or application-level
|
|
480
485
|
idempotency keys to guard against reprocessing a job that was partially handled before
|
|
481
|
-
a crash or
|
|
486
|
+
a crash or a shutdown that outlasts the drain budget.
|
|
482
487
|
|
|
483
488
|
**DLQ pattern.** When a job exhausts its `attempts`, move it to a dead-letter queue:
|
|
484
489
|
|
|
@@ -506,14 +511,14 @@ BymaxQueueModule.forRoot({
|
|
|
506
511
|
|
|
507
512
|
## 🤔 Why over `@nestjs/bullmq`
|
|
508
513
|
|
|
509
|
-
| Concern | `@nestjs/bullmq` (official) | `@bymax-one/nest-queue`
|
|
510
|
-
| ------------- | ---------------------------------------------------- |
|
|
511
|
-
| Module setup | `BullModule.forRoot()` + `registerQueue()` per queue | Single `forRoot`/`forRootAsync`; queues created on demand by name
|
|
512
|
-
| Connection | Manual ioredis + `maxRetriesPerRequest` wiring | Dual mode (BYO / lib-owned); correct per-role retry policy applied automatically
|
|
513
|
-
| Job defaults | Per `registerQueue` | Centralized opinionated defaults with per-queue/per-job overrides
|
|
514
|
-
| Producer API | `@InjectQueue` + raw `queue.add` | Typed `enqueue<TData, TResult>()`, `enqueueBulk`, `upsertJobScheduler`
|
|
515
|
-
| Shutdown | Framework closes workers | Explicit bounded-drain protocol;
|
|
516
|
-
| Observability | Manual | `telemetry` passthrough (OpenTelemetry) + `MetricsService`
|
|
514
|
+
| Concern | `@nestjs/bullmq` (official) | `@bymax-one/nest-queue` |
|
|
515
|
+
| ------------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
|
516
|
+
| Module setup | `BullModule.forRoot()` + `registerQueue()` per queue | Single `forRoot`/`forRootAsync`; queues created on demand by name |
|
|
517
|
+
| Connection | Manual ioredis + `maxRetriesPerRequest` wiring | Dual mode (BYO / lib-owned); correct per-role retry policy applied automatically |
|
|
518
|
+
| Job defaults | Per `registerQueue` | Centralized opinionated defaults with per-queue/per-job overrides |
|
|
519
|
+
| Producer API | `@InjectQueue` + raw `queue.add` | Typed `enqueue<TData, TResult>()`, `enqueueBulk`, `upsertJobScheduler` |
|
|
520
|
+
| Shutdown | Framework closes workers | Explicit bounded-drain protocol; stops a busy worker at the deadline; stalled accounting |
|
|
521
|
+
| Observability | Manual | `telemetry` passthrough (OpenTelemetry) + `MetricsService` |
|
|
517
522
|
|
|
518
523
|
**Choose this library** when you want a single opinionated setup, the `@bymax-one/nest-cache`
|
|
519
524
|
connection-sharing story, enforced defaults, and a tested bounded-shutdown protocol.
|
|
@@ -552,7 +557,7 @@ wire connection and defaults yourself.
|
|
|
552
557
|
(@Processor / @Process / @OnWorkerEvent)
|
|
553
558
|
│
|
|
554
559
|
QueueLifecycle
|
|
555
|
-
(bounded drain →
|
|
560
|
+
(bounded drain → stop → disconnect)
|
|
556
561
|
```
|
|
557
562
|
|
|
558
563
|
### Design Principles
|
|
@@ -850,18 +855,18 @@ try {
|
|
|
850
855
|
}
|
|
851
856
|
```
|
|
852
857
|
|
|
853
|
-
| Code | HTTP | When thrown
|
|
854
|
-
| ---------------------------------------- | :--: |
|
|
855
|
-
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs`
|
|
856
|
-
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, Mode A's injected client is in an unusable `status`, or a Mode B client was refused by the server (`WRONGPASS`, `NOAUTH`) or stopped reconnecting during startup
|
|
857
|
-
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null`
|
|
858
|
-
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` (`details` carries `timeoutMs` and the last error's `lastErrorCode`), or a `QueueService`/`FlowService` call got no answer
|
|
859
|
-
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every`
|
|
860
|
-
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name
|
|
861
|
-
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled`
|
|
862
|
-
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled`
|
|
863
|
-
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed
|
|
864
|
-
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed
|
|
858
|
+
| Code | HTTP | When thrown |
|
|
859
|
+
| ---------------------------------------- | :--: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
860
|
+
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
|
|
861
|
+
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, Mode A's injected client is in an unusable `status`, or a Mode B client was refused by the server (`WRONGPASS`, `NOAUTH`) or stopped reconnecting during startup |
|
|
862
|
+
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
|
|
863
|
+
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` (`details` carries `timeoutMs` and the last error's `lastErrorCode`), or a `QueueService`/`FlowService` call got no answer from Redis: it timed out (`commandTimeoutMs` on the owned connection, or `enqueueTimeoutMs` for a `QueueService` call; `details` carries `operation` and `timeoutMs`) or the connection is down (`details` carries `operation` and `reason: 'offline'`); an enqueue may still have been added |
|
|
864
|
+
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
|
|
865
|
+
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
|
|
866
|
+
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
|
|
867
|
+
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
|
|
868
|
+
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
|
|
869
|
+
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
|
|
865
870
|
|
|
866
871
|
### Reserved codes
|
|
867
872
|
|
|
@@ -869,12 +874,12 @@ These are part of the published catalog and stable, but nothing currently throws
|
|
|
869
874
|
They are listed so a consumer writing an exhaustive `switch` over `QueueErrorCode`
|
|
870
875
|
compiles today and keeps compiling when a future release starts emitting them.
|
|
871
876
|
|
|
872
|
-
| Code | Why it is not thrown today
|
|
873
|
-
| --------------------------------- |
|
|
874
|
-
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses
|
|
875
|
-
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing
|
|
876
|
-
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them
|
|
877
|
-
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline
|
|
877
|
+
| Code | Why it is not thrown today |
|
|
878
|
+
| --------------------------------- | ----------------------------------------------------------------------------- |
|
|
879
|
+
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
|
|
880
|
+
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
|
|
881
|
+
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
|
|
882
|
+
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline stops the workers still busy rather than raising |
|
|
878
883
|
|
|
879
884
|
The same catalog is exported from `@bymax-one/nest-queue/shared` with no dependencies,
|
|
880
885
|
so a frontend can map codes to messages without importing the server surface.
|
|
@@ -908,7 +913,7 @@ If you see it, the client you passed is not a plain ioredis client (a wrapper or
|
|
|
908
913
|
double whose `duplicate()` ignores its options). Pass the ioredis client itself, or let the
|
|
909
914
|
library open its own connection with Mode B. The producer keeps the client's own retry
|
|
910
915
|
policy; with a finite `maxRetriesPerRequest` that is what lets it fail fast during an outage, and
|
|
911
|
-
`
|
|
916
|
+
every `QueueService` call is bounded by `enqueueTimeoutMs` whatever the client does.
|
|
912
917
|
|
|
913
918
|
### A handler never settles (the queue stops moving)
|
|
914
919
|
|
package/dist/server/index.cjs
CHANGED
|
@@ -250,13 +250,14 @@ function duplicateConnection(client) {
|
|
|
250
250
|
}
|
|
251
251
|
|
|
252
252
|
// src/server/services/connection-resolver.service.ts
|
|
253
|
-
var _logger, _client, _options;
|
|
253
|
+
var _logger, _client, _options, _pendingReady;
|
|
254
254
|
exports.ConnectionResolver = class ConnectionResolver {
|
|
255
255
|
constructor(options) {
|
|
256
256
|
__privateAdd(this, _logger, new common.Logger(exports.ConnectionResolver.name));
|
|
257
257
|
__privateAdd(this, _client);
|
|
258
258
|
__privateAdd(this, _options);
|
|
259
259
|
__publicField(this, "mode");
|
|
260
|
+
__privateAdd(this, _pendingReady);
|
|
260
261
|
__privateSet(this, _options, options);
|
|
261
262
|
}
|
|
262
263
|
async init() {
|
|
@@ -312,6 +313,17 @@ exports.ConnectionResolver = class ConnectionResolver {
|
|
|
312
313
|
isOwned() {
|
|
313
314
|
return this.mode === "mode-b-owned";
|
|
314
315
|
}
|
|
316
|
+
awaitReady() {
|
|
317
|
+
const client = this.getClient();
|
|
318
|
+
if (!WAITING_STATUSES.has(client.status)) return Promise.resolve();
|
|
319
|
+
__privateGet(this, _pendingReady) ?? __privateSet(this, _pendingReady, new Promise((resolve) => {
|
|
320
|
+
client.once("ready", () => {
|
|
321
|
+
__privateSet(this, _pendingReady, void 0);
|
|
322
|
+
resolve();
|
|
323
|
+
});
|
|
324
|
+
}));
|
|
325
|
+
return __privateGet(this, _pendingReady);
|
|
326
|
+
}
|
|
315
327
|
async teardown() {
|
|
316
328
|
if (this.isOwned() && __privateGet(this, _client)) {
|
|
317
329
|
const client = __privateGet(this, _client);
|
|
@@ -332,7 +344,7 @@ exports.ConnectionResolver = class ConnectionResolver {
|
|
|
332
344
|
}
|
|
333
345
|
if (client.options.maxRetriesPerRequest === null) {
|
|
334
346
|
__privateGet(this, _logger).warn(
|
|
335
|
-
"The shared client has maxRetriesPerRequest: null, so a
|
|
347
|
+
"The shared client has maxRetriesPerRequest: null, so a command issued while Redis is unreachable stays queued in the client for as long as the outage lasts. QueueService calls are bounded by enqueueTimeoutMs and the caller is released, but the command is not cancelled. Give the client a finite limit to fail fast."
|
|
336
348
|
);
|
|
337
349
|
}
|
|
338
350
|
const probe = duplicateConnection(client);
|
|
@@ -382,10 +394,17 @@ exports.ConnectionResolver = class ConnectionResolver {
|
|
|
382
394
|
_logger = new WeakMap();
|
|
383
395
|
_client = new WeakMap();
|
|
384
396
|
_options = new WeakMap();
|
|
397
|
+
_pendingReady = new WeakMap();
|
|
385
398
|
exports.ConnectionResolver = __decorateClass([
|
|
386
399
|
common.Injectable(),
|
|
387
400
|
__decorateParam(0, common.Inject(BYMAX_QUEUE_OPTIONS))
|
|
388
401
|
], exports.ConnectionResolver);
|
|
402
|
+
var WAITING_STATUSES = /* @__PURE__ */ new Set([
|
|
403
|
+
"connecting",
|
|
404
|
+
"connect",
|
|
405
|
+
"reconnecting",
|
|
406
|
+
"close"
|
|
407
|
+
]);
|
|
389
408
|
function isReplyError(err) {
|
|
390
409
|
return typeof err === "object" && err !== null && err.name === "ReplyError";
|
|
391
410
|
}
|
|
@@ -456,6 +475,18 @@ function timeoutException(operation, timeoutMs, cause) {
|
|
|
456
475
|
if (cause !== void 0) exception.cause = cause;
|
|
457
476
|
return exception;
|
|
458
477
|
}
|
|
478
|
+
function unavailableException(operation, cause) {
|
|
479
|
+
const exception = new QueueException(
|
|
480
|
+
QUEUE_ERROR_CODES.CONNECTION_TIMEOUT,
|
|
481
|
+
common.HttpStatus.INTERNAL_SERVER_ERROR,
|
|
482
|
+
{ operation, reason: "offline" }
|
|
483
|
+
);
|
|
484
|
+
exception.cause = cause;
|
|
485
|
+
return exception;
|
|
486
|
+
}
|
|
487
|
+
function isOfflineRejection(err) {
|
|
488
|
+
return err instanceof Error && err.message === "Stream isn't writeable and enableOfflineQueue options is false";
|
|
489
|
+
}
|
|
459
490
|
function isCommandTimeout(err) {
|
|
460
491
|
return err instanceof Error && err.message === "Command timed out";
|
|
461
492
|
}
|
|
@@ -463,7 +494,8 @@ async function withCommandTimeoutCode(operation, commandTimeoutMs, work) {
|
|
|
463
494
|
try {
|
|
464
495
|
return await work;
|
|
465
496
|
} catch (err) {
|
|
466
|
-
|
|
497
|
+
if (isCommandTimeout(err)) throw timeoutException(operation, commandTimeoutMs(), err);
|
|
498
|
+
throw isOfflineRejection(err) ? unavailableException(operation, err) : err;
|
|
467
499
|
}
|
|
468
500
|
}
|
|
469
501
|
|
|
@@ -479,13 +511,14 @@ function assertValidCron(repeat, queueOptions) {
|
|
|
479
511
|
});
|
|
480
512
|
}
|
|
481
513
|
}
|
|
482
|
-
var _queues;
|
|
514
|
+
var _queues, _warm;
|
|
483
515
|
exports.QueueService = class QueueService {
|
|
484
516
|
constructor(connection, options) {
|
|
485
517
|
this.connection = connection;
|
|
486
518
|
this.options = options;
|
|
487
519
|
__publicField(this, "logger", new common.Logger(exports.QueueService.name));
|
|
488
520
|
__privateAdd(this, _queues, /* @__PURE__ */ new Map());
|
|
521
|
+
__privateAdd(this, _warm, /* @__PURE__ */ new Set());
|
|
489
522
|
}
|
|
490
523
|
connection;
|
|
491
524
|
options;
|
|
@@ -504,27 +537,38 @@ exports.QueueService = class QueueService {
|
|
|
504
537
|
__privateGet(this, _queues).set(queueName, queue);
|
|
505
538
|
return queue;
|
|
506
539
|
}
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
540
|
+
call(operation, queueName, run) {
|
|
541
|
+
let expired = false;
|
|
542
|
+
const guarded = withDeadline(
|
|
543
|
+
this.afterReady(queueName).then(() => {
|
|
544
|
+
if (expired) throw new Error("the call had already timed out");
|
|
545
|
+
return run();
|
|
546
|
+
}),
|
|
547
|
+
this.options.enqueueTimeoutMs,
|
|
548
|
+
() => {
|
|
549
|
+
expired = true;
|
|
550
|
+
return timeoutException(operation, this.options.enqueueTimeoutMs);
|
|
551
|
+
}
|
|
552
|
+
);
|
|
511
553
|
return withCommandTimeoutCode(
|
|
512
554
|
operation,
|
|
513
555
|
() => this.connection.getClient().options.commandTimeout ?? this.options.commandTimeoutMs,
|
|
514
|
-
|
|
556
|
+
guarded
|
|
515
557
|
);
|
|
516
558
|
}
|
|
559
|
+
async afterReady(queueName) {
|
|
560
|
+
if (__privateGet(this, _warm).has(queueName)) return;
|
|
561
|
+
await this.connection.awaitReady();
|
|
562
|
+
__privateGet(this, _warm).add(queueName);
|
|
563
|
+
}
|
|
517
564
|
queueFor(queueName) {
|
|
518
565
|
return this.getOrCreateQueue(queueName);
|
|
519
566
|
}
|
|
520
567
|
async enqueue(queueName, jobName, data, options) {
|
|
521
568
|
const job = await this.call(
|
|
522
569
|
"enqueue",
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
this.options.enqueueTimeoutMs,
|
|
526
|
-
() => this.enqueueTimeout("enqueue")
|
|
527
|
-
)
|
|
570
|
+
queueName,
|
|
571
|
+
() => this.queueFor(queueName).add(jobName, data, options)
|
|
528
572
|
);
|
|
529
573
|
return job;
|
|
530
574
|
}
|
|
@@ -543,16 +587,13 @@ exports.QueueService = class QueueService {
|
|
|
543
587
|
try {
|
|
544
588
|
const created = await this.call(
|
|
545
589
|
"enqueueBulk",
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
),
|
|
554
|
-
this.options.enqueueTimeoutMs,
|
|
555
|
-
() => this.enqueueTimeout("enqueueBulk")
|
|
590
|
+
queueName,
|
|
591
|
+
() => this.queueFor(queueName).addBulk(
|
|
592
|
+
jobs.map((job) => ({
|
|
593
|
+
name: job.name,
|
|
594
|
+
data: job.data,
|
|
595
|
+
...job.options === void 0 ? {} : { opts: job.options }
|
|
596
|
+
}))
|
|
556
597
|
)
|
|
557
598
|
);
|
|
558
599
|
return created;
|
|
@@ -570,7 +611,7 @@ exports.QueueService = class QueueService {
|
|
|
570
611
|
}
|
|
571
612
|
}
|
|
572
613
|
async getJob(queueName, jobId) {
|
|
573
|
-
const job = await this.call("getJob", this.queueFor(queueName).getJob(jobId));
|
|
614
|
+
const job = await this.call("getJob", queueName, () => this.queueFor(queueName).getJob(jobId));
|
|
574
615
|
return job ?? null;
|
|
575
616
|
}
|
|
576
617
|
async getJobs(queueName, status, start = 0, end = 50) {
|
|
@@ -579,13 +620,18 @@ exports.QueueService = class QueueService {
|
|
|
579
620
|
reason: `getJobs needs integers with 0 <= start <= end and at most ${String(MAX_JOBS_PAGE_SIZE)} jobs per call`
|
|
580
621
|
});
|
|
581
622
|
}
|
|
582
|
-
const jobs = await this.call(
|
|
623
|
+
const jobs = await this.call(
|
|
624
|
+
"getJobs",
|
|
625
|
+
queueName,
|
|
626
|
+
() => this.queueFor(queueName).getJobs([status], start, end)
|
|
627
|
+
);
|
|
583
628
|
return jobs;
|
|
584
629
|
}
|
|
585
630
|
async getMetrics(queueName) {
|
|
586
631
|
const counts = await this.call(
|
|
587
632
|
"getMetrics",
|
|
588
|
-
|
|
633
|
+
queueName,
|
|
634
|
+
() => this.queueFor(queueName).getJobCounts("waiting", "active", "completed", "failed", "delayed")
|
|
589
635
|
);
|
|
590
636
|
return {
|
|
591
637
|
queue: queueName,
|
|
@@ -604,24 +650,37 @@ exports.QueueService = class QueueService {
|
|
|
604
650
|
if ("pattern" in repeat) assertValidCron(repeat, this.options.queueOptions);
|
|
605
651
|
const job = await this.call(
|
|
606
652
|
"upsertJobScheduler",
|
|
607
|
-
|
|
653
|
+
queueName,
|
|
654
|
+
() => queue.upsertJobScheduler(schedulerId, repeat, jobTemplate)
|
|
608
655
|
);
|
|
609
656
|
return job;
|
|
610
657
|
}
|
|
611
658
|
async removeJobScheduler(queueName, schedulerId) {
|
|
612
|
-
return this.call(
|
|
659
|
+
return this.call(
|
|
660
|
+
"removeJobScheduler",
|
|
661
|
+
queueName,
|
|
662
|
+
() => this.queueFor(queueName).removeJobScheduler(schedulerId)
|
|
663
|
+
);
|
|
613
664
|
}
|
|
614
665
|
async getJobSchedulers(queueName, start = 0, end = 50, asc = true) {
|
|
615
|
-
return this.call(
|
|
666
|
+
return this.call(
|
|
667
|
+
"getJobSchedulers",
|
|
668
|
+
queueName,
|
|
669
|
+
() => this.queueFor(queueName).getJobSchedulers(start, end, asc)
|
|
670
|
+
);
|
|
616
671
|
}
|
|
617
672
|
async pauseQueue(queueName) {
|
|
618
|
-
await this.call("pauseQueue", this.queueFor(queueName).pause());
|
|
673
|
+
await this.call("pauseQueue", queueName, () => this.queueFor(queueName).pause());
|
|
619
674
|
}
|
|
620
675
|
async resumeQueue(queueName) {
|
|
621
|
-
await this.call("resumeQueue", this.queueFor(queueName).resume());
|
|
676
|
+
await this.call("resumeQueue", queueName, () => this.queueFor(queueName).resume());
|
|
622
677
|
}
|
|
623
678
|
async cleanQueue(queueName, gracePeriodMs, limit, status) {
|
|
624
|
-
return this.call(
|
|
679
|
+
return this.call(
|
|
680
|
+
"cleanQueue",
|
|
681
|
+
queueName,
|
|
682
|
+
() => this.queueFor(queueName).clean(gracePeriodMs, limit, status)
|
|
683
|
+
);
|
|
625
684
|
}
|
|
626
685
|
getCachedQueues() {
|
|
627
686
|
return __privateGet(this, _queues);
|
|
@@ -631,9 +690,11 @@ exports.QueueService = class QueueService {
|
|
|
631
690
|
await queue.close().catch(() => void 0);
|
|
632
691
|
}
|
|
633
692
|
__privateGet(this, _queues).clear();
|
|
693
|
+
__privateGet(this, _warm).clear();
|
|
634
694
|
}
|
|
635
695
|
};
|
|
636
696
|
_queues = new WeakMap();
|
|
697
|
+
_warm = new WeakMap();
|
|
637
698
|
exports.QueueService = __decorateClass([
|
|
638
699
|
common.Injectable(),
|
|
639
700
|
__decorateParam(0, common.Inject(exports.ConnectionResolver)),
|
|
@@ -736,6 +797,17 @@ exports.MetricsService = class MetricsService {
|
|
|
736
797
|
exports.MetricsService = __decorateClass([
|
|
737
798
|
common.Injectable()
|
|
738
799
|
], exports.MetricsService);
|
|
800
|
+
|
|
801
|
+
// src/server/utils/abandoned-workers.ts
|
|
802
|
+
var abandoned = /* @__PURE__ */ new WeakSet();
|
|
803
|
+
function abandonWorker(worker) {
|
|
804
|
+
abandoned.add(worker);
|
|
805
|
+
}
|
|
806
|
+
function isAbandoned(worker) {
|
|
807
|
+
return abandoned.has(worker);
|
|
808
|
+
}
|
|
809
|
+
|
|
810
|
+
// src/server/services/worker-registry.service.ts
|
|
739
811
|
var _entries;
|
|
740
812
|
exports.WorkerRegistry = class WorkerRegistry {
|
|
741
813
|
constructor(connection, options) {
|
|
@@ -756,7 +828,7 @@ exports.WorkerRegistry = class WorkerRegistry {
|
|
|
756
828
|
try {
|
|
757
829
|
worker = new bullmq.Worker(
|
|
758
830
|
queueName,
|
|
759
|
-
toProcessor(handler, options?.jobTimeoutMs),
|
|
831
|
+
toProcessor(handler, options?.jobTimeoutMs, () => isAbandoned(worker)),
|
|
760
832
|
bullOpts
|
|
761
833
|
);
|
|
762
834
|
} catch (cause) {
|
|
@@ -970,7 +1042,7 @@ var JobTimeoutError = class extends Error {
|
|
|
970
1042
|
this.name = "JobTimeoutError";
|
|
971
1043
|
}
|
|
972
1044
|
};
|
|
973
|
-
function
|
|
1045
|
+
function adaptHandler(handler, timeoutMs) {
|
|
974
1046
|
if (timeoutMs === void 0) return (job, _token, signal) => handler(job, signal);
|
|
975
1047
|
return (job, _token, signal) => {
|
|
976
1048
|
const own = new AbortController();
|
|
@@ -992,6 +1064,16 @@ function toProcessor(handler, timeoutMs) {
|
|
|
992
1064
|
});
|
|
993
1065
|
};
|
|
994
1066
|
}
|
|
1067
|
+
function toProcessor(handler, timeoutMs, abandoned2) {
|
|
1068
|
+
const run = adaptHandler(handler, timeoutMs);
|
|
1069
|
+
return (job, token, signal) => run(job, token, signal).then(
|
|
1070
|
+
(value) => abandoned2() ? new Promise(() => void 0) : value,
|
|
1071
|
+
(error) => {
|
|
1072
|
+
if (abandoned2()) return new Promise(() => void 0);
|
|
1073
|
+
throw error;
|
|
1074
|
+
}
|
|
1075
|
+
);
|
|
1076
|
+
}
|
|
995
1077
|
var _events, _connections;
|
|
996
1078
|
exports.QueueEventsRegistry = class QueueEventsRegistry {
|
|
997
1079
|
constructor(connection, options) {
|
|
@@ -1170,6 +1252,7 @@ ProcessorDiscoveryService = __decorateClass([
|
|
|
1170
1252
|
__decorateParam(1, common.Inject(exports.WorkerRegistry)),
|
|
1171
1253
|
__decorateParam(2, common.Inject(exports.QueueEventsRegistry))
|
|
1172
1254
|
], ProcessorDiscoveryService);
|
|
1255
|
+
var FORCE_CLOSE_GRACE_MS = 1e3;
|
|
1173
1256
|
var DrainTimeoutError = class extends Error {
|
|
1174
1257
|
};
|
|
1175
1258
|
var QueueLifecycle = class {
|
|
@@ -1203,25 +1286,33 @@ var QueueLifecycle = class {
|
|
|
1203
1286
|
await this.closeQueues();
|
|
1204
1287
|
await this.teardownConnection();
|
|
1205
1288
|
this.logger.log(
|
|
1206
|
-
`Queue shutdown complete in ${String(Date.now() - (this.startedAt ?? Date.now()))}ms (
|
|
1289
|
+
`Queue shutdown complete in ${String(Date.now() - (this.startedAt ?? Date.now()))}ms (${String(this.forcedWorkers)} worker(s) stopped without draining)`
|
|
1207
1290
|
);
|
|
1208
1291
|
}
|
|
1209
1292
|
async closeWorkers(deadline) {
|
|
1210
|
-
let
|
|
1293
|
+
let undrained = 0;
|
|
1211
1294
|
await Promise.all(
|
|
1212
1295
|
Array.from(this.workers.getAll(), async ([name, worker]) => {
|
|
1213
1296
|
try {
|
|
1214
|
-
await bounded(
|
|
1297
|
+
await bounded(
|
|
1298
|
+
worker.pause().then(() => worker.close()),
|
|
1299
|
+
deadline
|
|
1300
|
+
);
|
|
1215
1301
|
} catch (cause) {
|
|
1216
|
-
|
|
1302
|
+
undrained++;
|
|
1303
|
+
abandonWorker(worker);
|
|
1304
|
+
const stopped = "it is stopped without waiting for its in-flight job, which becomes stalled and is retried";
|
|
1217
1305
|
this.logger.warn(
|
|
1218
|
-
cause instanceof DrainTimeoutError ? `Worker "${name}" exceeded the ${String(this.resolved.shutdown.drainTimeoutMs)}ms drain budget
|
|
1306
|
+
cause instanceof DrainTimeoutError ? `Worker "${name}" exceeded the ${String(this.resolved.shutdown.drainTimeoutMs)}ms drain budget (${QUEUE_ERROR_CODES.SHUTDOWN_TIMEOUT_EXCEEDED}); ${stopped}` : `Worker "${name}" failed to close gracefully (${describeError(cause)}); ${stopped}`
|
|
1307
|
+
);
|
|
1308
|
+
const deadlineError = () => new DrainTimeoutError();
|
|
1309
|
+
await withDeadline(worker.close(true), FORCE_CLOSE_GRACE_MS, deadlineError).catch(
|
|
1310
|
+
() => void 0
|
|
1219
1311
|
);
|
|
1220
|
-
void worker.close(true).catch(() => void 0);
|
|
1221
1312
|
}
|
|
1222
1313
|
})
|
|
1223
1314
|
);
|
|
1224
|
-
return
|
|
1315
|
+
return undrained;
|
|
1225
1316
|
}
|
|
1226
1317
|
async closeQueueEvents(deadline) {
|
|
1227
1318
|
await Promise.all(
|
package/dist/server/index.d.cts
CHANGED
|
@@ -105,9 +105,11 @@ interface BymaxQueueModuleOptions {
|
|
|
105
105
|
*/
|
|
106
106
|
commandTimeoutMs?: number;
|
|
107
107
|
/**
|
|
108
|
-
* The deadline, in milliseconds, for one `
|
|
109
|
-
*
|
|
110
|
-
*
|
|
108
|
+
* The deadline, in milliseconds, for one `QueueService` call that reaches Redis (`enqueue`,
|
|
109
|
+
* `getJob`, `getMetrics`, the scheduler, pause and clean calls, and the rest) in either connection
|
|
110
|
+
* mode, including the wait for the client to be ready on a queue's first call. When it passes, the
|
|
111
|
+
* call rejects with `CONNECTION_TIMEOUT`; an `enqueue` may still complete in Redis afterwards, so
|
|
112
|
+
* retry with a stable `jobId` or `deduplication`. Default: 10_000.
|
|
111
113
|
*/
|
|
112
114
|
enqueueTimeoutMs?: number;
|
|
113
115
|
}
|
|
@@ -401,6 +403,14 @@ declare class ConnectionResolver {
|
|
|
401
403
|
getMode(): QueueConnectionMode;
|
|
402
404
|
/** Whether the library owns (and must close) the connection. */
|
|
403
405
|
isOwned(): boolean;
|
|
406
|
+
/**
|
|
407
|
+
* Settle when the client can take a command. While it is still connecting or reconnecting, every
|
|
408
|
+
* caller shares one wait on its `ready` event, so a health check that polls through a long outage
|
|
409
|
+
* adds one listener set to the client, not one per call, and gives it back when Redis returns. A
|
|
410
|
+
* lazy client (`wait`) connects on its first command and a finished one (`end`) never becomes
|
|
411
|
+
* ready, so both are left for the call itself.
|
|
412
|
+
*/
|
|
413
|
+
awaitReady(): Promise<void>;
|
|
404
414
|
/** Close the library-owned connection on shutdown; never touch a BYO client. */
|
|
405
415
|
teardown(): Promise<void>;
|
|
406
416
|
/** Validate and adopt a bring-your-own client for the Queue role. */
|
|
@@ -446,20 +456,27 @@ declare class QueueService {
|
|
|
446
456
|
* @param overrides - Optional per-queue option overrides.
|
|
447
457
|
*/
|
|
448
458
|
getOrCreateQueue<TData = unknown, TResult = unknown>(queueName: string, overrides?: Partial<Omit<QueueOptions, 'connection' | 'prefix'>>): Queue<TData, TResult>;
|
|
449
|
-
/** The error a producer call receives when Redis does not answer within `enqueueTimeoutMs`. */
|
|
450
|
-
private enqueueTimeout;
|
|
451
459
|
/**
|
|
452
|
-
*
|
|
453
|
-
*
|
|
460
|
+
* Run one Redis call under the per-call deadline (`enqueueTimeoutMs`), reporting every way Redis
|
|
461
|
+
* can fail to answer as the documented `CONNECTION_TIMEOUT`: the deadline passing, an ioredis
|
|
462
|
+
* command timeout, or a call refused because the connection is down.
|
|
463
|
+
*
|
|
464
|
+
* The first call on a queue name waits for the client to be ready through ONE wait shared by all
|
|
465
|
+
* callers, instead of letting BullMQ start its own unbounded wait per call, so a caller that
|
|
466
|
+
* polls through an outage does not pile up listeners. The call runs only if it is still within
|
|
467
|
+
* its deadline when the client is ready; a caller that was told it timed out never has its call
|
|
468
|
+
* run later, which would turn its retry into a duplicate.
|
|
454
469
|
*/
|
|
455
470
|
private call;
|
|
471
|
+
/** Settle when a call on `queueName` may start: at once for a queue that has been used before. */
|
|
472
|
+
private afterReady;
|
|
456
473
|
/** Resolve the raw cached queue (default generics) for internal delegation. */
|
|
457
474
|
private queueFor;
|
|
458
475
|
/**
|
|
459
476
|
* Add a single job to a queue. The queue is created lazily if needed.
|
|
460
477
|
*
|
|
461
478
|
* Delivery is **at-least-once** — a handler may run more than once for the same
|
|
462
|
-
* job (worker crash, lock expiry, or a shutdown
|
|
479
|
+
* job (worker crash, lock expiry, or a shutdown that outlasts the drain budget). Make handlers
|
|
463
480
|
* idempotent: use an idempotency key on writes, prefer upserts over inserts, or
|
|
464
481
|
* keep an "already-processed" marker keyed by `job.id`.
|
|
465
482
|
*
|
|
@@ -1087,9 +1104,9 @@ declare class QueueEventsRegistry {
|
|
|
1087
1104
|
* HTTP server has stopped, it releases what a request can still be using: the
|
|
1088
1105
|
* flow producer, the cached queues, and finally the Queue-role connection.
|
|
1089
1106
|
*
|
|
1090
|
-
* Delivery is at-least-once: when a worker exceeds the drain budget it is
|
|
1091
|
-
*
|
|
1092
|
-
* worker. Handlers must therefore be idempotent, and `lockDuration` should
|
|
1107
|
+
* Delivery is at-least-once: when a worker exceeds the drain budget it is stopped
|
|
1108
|
+
* without waiting for its job, whose lock then expires so the job becomes `stalled`
|
|
1109
|
+
* and is retried by another worker. Handlers must therefore be idempotent, and `lockDuration` should
|
|
1093
1110
|
* comfortably exceed the worst-case handler runtime so healthy long jobs are
|
|
1094
1111
|
* never treated as stalled.
|
|
1095
1112
|
* @layer server/lifecycle
|
|
@@ -1138,11 +1155,16 @@ declare class QueueLifecycle implements OnModuleDestroy, OnApplicationShutdown {
|
|
|
1138
1155
|
*/
|
|
1139
1156
|
onApplicationShutdown(): Promise<void>;
|
|
1140
1157
|
/**
|
|
1141
|
-
*
|
|
1142
|
-
*
|
|
1158
|
+
* Stop every worker in parallel, each bounded by what is left of the shutdown budget. A worker
|
|
1159
|
+
* is paused first (it stops fetching and waits for its active jobs) and then closed. One that is
|
|
1160
|
+
* still busy when the budget ends is closed with `close(true)`: because `pause` does not start a
|
|
1161
|
+
* close, that is the worker's FIRST close, so it does not wait for the job and stops the lock
|
|
1162
|
+
* renewal that would otherwise keep running and keep the process alive. The job's lock then
|
|
1163
|
+
* expires, and it stalls and is retried elsewhere. Returns how many workers were stopped
|
|
1164
|
+
* without draining.
|
|
1143
1165
|
*
|
|
1144
1166
|
* @param deadline - The epoch millisecond at which the shutdown budget ends.
|
|
1145
|
-
* @returns The count of
|
|
1167
|
+
* @returns The count of workers stopped without draining.
|
|
1146
1168
|
*/
|
|
1147
1169
|
private closeWorkers;
|
|
1148
1170
|
/** Close every cached `QueueEvents` in parallel within the budget, swallowing failures. */
|
|
@@ -1221,7 +1243,7 @@ declare function Processor(queueName: string, options?: WorkerOptions): ClassDec
|
|
|
1221
1243
|
* an entry to the accumulated metadata array.
|
|
1222
1244
|
*
|
|
1223
1245
|
* Delivery is at-least-once: a handler may run more than once for the same job
|
|
1224
|
-
* (worker crash, lock expiry, or a shutdown
|
|
1246
|
+
* (worker crash, lock expiry, or a shutdown that outlasts the drain budget). Handlers MUST be
|
|
1225
1247
|
* idempotent — use an idempotency key on writes, prefer upserts over inserts, or
|
|
1226
1248
|
* keep an "already-processed" marker keyed by `job.id`.
|
|
1227
1249
|
*
|
|
@@ -1273,7 +1295,7 @@ type WorkerEventName = 'completed' | 'failed' | 'progress' | 'active' | 'stalled
|
|
|
1273
1295
|
* overwriting each other.
|
|
1274
1296
|
*
|
|
1275
1297
|
* Delivery is at-least-once. A `'stalled'` job — for example after a shutdown
|
|
1276
|
-
*
|
|
1298
|
+
* that outlasts the drain budget, or a lock expiry — is re-run by another worker: this is the
|
|
1277
1299
|
* visible at-least-once path, so handlers must be idempotent. Set `lockDuration`
|
|
1278
1300
|
* comfortably above the worst-case handler runtime to avoid false stalls.
|
|
1279
1301
|
*
|
|
@@ -1490,7 +1512,7 @@ declare const DEFAULT_JOB_OPTIONS: {
|
|
|
1490
1512
|
};
|
|
1491
1513
|
/** Default TTL (ms) for the metrics cache. */
|
|
1492
1514
|
declare const DEFAULT_METRICS_CACHE_TTL_MS: 5000;
|
|
1493
|
-
/** Default window (ms) for in-flight jobs to drain before
|
|
1515
|
+
/** Default window (ms) for in-flight jobs to drain before the worker is stopped on shutdown. */
|
|
1494
1516
|
declare const DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS: 30000;
|
|
1495
1517
|
|
|
1496
1518
|
/**
|
package/dist/server/index.d.ts
CHANGED
|
@@ -105,9 +105,11 @@ interface BymaxQueueModuleOptions {
|
|
|
105
105
|
*/
|
|
106
106
|
commandTimeoutMs?: number;
|
|
107
107
|
/**
|
|
108
|
-
* The deadline, in milliseconds, for one `
|
|
109
|
-
*
|
|
110
|
-
*
|
|
108
|
+
* The deadline, in milliseconds, for one `QueueService` call that reaches Redis (`enqueue`,
|
|
109
|
+
* `getJob`, `getMetrics`, the scheduler, pause and clean calls, and the rest) in either connection
|
|
110
|
+
* mode, including the wait for the client to be ready on a queue's first call. When it passes, the
|
|
111
|
+
* call rejects with `CONNECTION_TIMEOUT`; an `enqueue` may still complete in Redis afterwards, so
|
|
112
|
+
* retry with a stable `jobId` or `deduplication`. Default: 10_000.
|
|
111
113
|
*/
|
|
112
114
|
enqueueTimeoutMs?: number;
|
|
113
115
|
}
|
|
@@ -401,6 +403,14 @@ declare class ConnectionResolver {
|
|
|
401
403
|
getMode(): QueueConnectionMode;
|
|
402
404
|
/** Whether the library owns (and must close) the connection. */
|
|
403
405
|
isOwned(): boolean;
|
|
406
|
+
/**
|
|
407
|
+
* Settle when the client can take a command. While it is still connecting or reconnecting, every
|
|
408
|
+
* caller shares one wait on its `ready` event, so a health check that polls through a long outage
|
|
409
|
+
* adds one listener set to the client, not one per call, and gives it back when Redis returns. A
|
|
410
|
+
* lazy client (`wait`) connects on its first command and a finished one (`end`) never becomes
|
|
411
|
+
* ready, so both are left for the call itself.
|
|
412
|
+
*/
|
|
413
|
+
awaitReady(): Promise<void>;
|
|
404
414
|
/** Close the library-owned connection on shutdown; never touch a BYO client. */
|
|
405
415
|
teardown(): Promise<void>;
|
|
406
416
|
/** Validate and adopt a bring-your-own client for the Queue role. */
|
|
@@ -446,20 +456,27 @@ declare class QueueService {
|
|
|
446
456
|
* @param overrides - Optional per-queue option overrides.
|
|
447
457
|
*/
|
|
448
458
|
getOrCreateQueue<TData = unknown, TResult = unknown>(queueName: string, overrides?: Partial<Omit<QueueOptions, 'connection' | 'prefix'>>): Queue<TData, TResult>;
|
|
449
|
-
/** The error a producer call receives when Redis does not answer within `enqueueTimeoutMs`. */
|
|
450
|
-
private enqueueTimeout;
|
|
451
459
|
/**
|
|
452
|
-
*
|
|
453
|
-
*
|
|
460
|
+
* Run one Redis call under the per-call deadline (`enqueueTimeoutMs`), reporting every way Redis
|
|
461
|
+
* can fail to answer as the documented `CONNECTION_TIMEOUT`: the deadline passing, an ioredis
|
|
462
|
+
* command timeout, or a call refused because the connection is down.
|
|
463
|
+
*
|
|
464
|
+
* The first call on a queue name waits for the client to be ready through ONE wait shared by all
|
|
465
|
+
* callers, instead of letting BullMQ start its own unbounded wait per call, so a caller that
|
|
466
|
+
* polls through an outage does not pile up listeners. The call runs only if it is still within
|
|
467
|
+
* its deadline when the client is ready; a caller that was told it timed out never has its call
|
|
468
|
+
* run later, which would turn its retry into a duplicate.
|
|
454
469
|
*/
|
|
455
470
|
private call;
|
|
471
|
+
/** Settle when a call on `queueName` may start: at once for a queue that has been used before. */
|
|
472
|
+
private afterReady;
|
|
456
473
|
/** Resolve the raw cached queue (default generics) for internal delegation. */
|
|
457
474
|
private queueFor;
|
|
458
475
|
/**
|
|
459
476
|
* Add a single job to a queue. The queue is created lazily if needed.
|
|
460
477
|
*
|
|
461
478
|
* Delivery is **at-least-once** — a handler may run more than once for the same
|
|
462
|
-
* job (worker crash, lock expiry, or a shutdown
|
|
479
|
+
* job (worker crash, lock expiry, or a shutdown that outlasts the drain budget). Make handlers
|
|
463
480
|
* idempotent: use an idempotency key on writes, prefer upserts over inserts, or
|
|
464
481
|
* keep an "already-processed" marker keyed by `job.id`.
|
|
465
482
|
*
|
|
@@ -1087,9 +1104,9 @@ declare class QueueEventsRegistry {
|
|
|
1087
1104
|
* HTTP server has stopped, it releases what a request can still be using: the
|
|
1088
1105
|
* flow producer, the cached queues, and finally the Queue-role connection.
|
|
1089
1106
|
*
|
|
1090
|
-
* Delivery is at-least-once: when a worker exceeds the drain budget it is
|
|
1091
|
-
*
|
|
1092
|
-
* worker. Handlers must therefore be idempotent, and `lockDuration` should
|
|
1107
|
+
* Delivery is at-least-once: when a worker exceeds the drain budget it is stopped
|
|
1108
|
+
* without waiting for its job, whose lock then expires so the job becomes `stalled`
|
|
1109
|
+
* and is retried by another worker. Handlers must therefore be idempotent, and `lockDuration` should
|
|
1093
1110
|
* comfortably exceed the worst-case handler runtime so healthy long jobs are
|
|
1094
1111
|
* never treated as stalled.
|
|
1095
1112
|
* @layer server/lifecycle
|
|
@@ -1138,11 +1155,16 @@ declare class QueueLifecycle implements OnModuleDestroy, OnApplicationShutdown {
|
|
|
1138
1155
|
*/
|
|
1139
1156
|
onApplicationShutdown(): Promise<void>;
|
|
1140
1157
|
/**
|
|
1141
|
-
*
|
|
1142
|
-
*
|
|
1158
|
+
* Stop every worker in parallel, each bounded by what is left of the shutdown budget. A worker
|
|
1159
|
+
* is paused first (it stops fetching and waits for its active jobs) and then closed. One that is
|
|
1160
|
+
* still busy when the budget ends is closed with `close(true)`: because `pause` does not start a
|
|
1161
|
+
* close, that is the worker's FIRST close, so it does not wait for the job and stops the lock
|
|
1162
|
+
* renewal that would otherwise keep running and keep the process alive. The job's lock then
|
|
1163
|
+
* expires, and it stalls and is retried elsewhere. Returns how many workers were stopped
|
|
1164
|
+
* without draining.
|
|
1143
1165
|
*
|
|
1144
1166
|
* @param deadline - The epoch millisecond at which the shutdown budget ends.
|
|
1145
|
-
* @returns The count of
|
|
1167
|
+
* @returns The count of workers stopped without draining.
|
|
1146
1168
|
*/
|
|
1147
1169
|
private closeWorkers;
|
|
1148
1170
|
/** Close every cached `QueueEvents` in parallel within the budget, swallowing failures. */
|
|
@@ -1221,7 +1243,7 @@ declare function Processor(queueName: string, options?: WorkerOptions): ClassDec
|
|
|
1221
1243
|
* an entry to the accumulated metadata array.
|
|
1222
1244
|
*
|
|
1223
1245
|
* Delivery is at-least-once: a handler may run more than once for the same job
|
|
1224
|
-
* (worker crash, lock expiry, or a shutdown
|
|
1246
|
+
* (worker crash, lock expiry, or a shutdown that outlasts the drain budget). Handlers MUST be
|
|
1225
1247
|
* idempotent — use an idempotency key on writes, prefer upserts over inserts, or
|
|
1226
1248
|
* keep an "already-processed" marker keyed by `job.id`.
|
|
1227
1249
|
*
|
|
@@ -1273,7 +1295,7 @@ type WorkerEventName = 'completed' | 'failed' | 'progress' | 'active' | 'stalled
|
|
|
1273
1295
|
* overwriting each other.
|
|
1274
1296
|
*
|
|
1275
1297
|
* Delivery is at-least-once. A `'stalled'` job — for example after a shutdown
|
|
1276
|
-
*
|
|
1298
|
+
* that outlasts the drain budget, or a lock expiry — is re-run by another worker: this is the
|
|
1277
1299
|
* visible at-least-once path, so handlers must be idempotent. Set `lockDuration`
|
|
1278
1300
|
* comfortably above the worst-case handler runtime to avoid false stalls.
|
|
1279
1301
|
*
|
|
@@ -1490,7 +1512,7 @@ declare const DEFAULT_JOB_OPTIONS: {
|
|
|
1490
1512
|
};
|
|
1491
1513
|
/** Default TTL (ms) for the metrics cache. */
|
|
1492
1514
|
declare const DEFAULT_METRICS_CACHE_TTL_MS: 5000;
|
|
1493
|
-
/** Default window (ms) for in-flight jobs to drain before
|
|
1515
|
+
/** Default window (ms) for in-flight jobs to drain before the worker is stopped on shutdown. */
|
|
1494
1516
|
declare const DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS: 30000;
|
|
1495
1517
|
|
|
1496
1518
|
/**
|
package/dist/server/index.mjs
CHANGED
|
@@ -248,13 +248,14 @@ function duplicateConnection(client) {
|
|
|
248
248
|
}
|
|
249
249
|
|
|
250
250
|
// src/server/services/connection-resolver.service.ts
|
|
251
|
-
var _logger, _client, _options;
|
|
251
|
+
var _logger, _client, _options, _pendingReady;
|
|
252
252
|
var ConnectionResolver = class {
|
|
253
253
|
constructor(options) {
|
|
254
254
|
__privateAdd(this, _logger, new Logger(ConnectionResolver.name));
|
|
255
255
|
__privateAdd(this, _client);
|
|
256
256
|
__privateAdd(this, _options);
|
|
257
257
|
__publicField(this, "mode");
|
|
258
|
+
__privateAdd(this, _pendingReady);
|
|
258
259
|
__privateSet(this, _options, options);
|
|
259
260
|
}
|
|
260
261
|
async init() {
|
|
@@ -310,6 +311,17 @@ var ConnectionResolver = class {
|
|
|
310
311
|
isOwned() {
|
|
311
312
|
return this.mode === "mode-b-owned";
|
|
312
313
|
}
|
|
314
|
+
awaitReady() {
|
|
315
|
+
const client = this.getClient();
|
|
316
|
+
if (!WAITING_STATUSES.has(client.status)) return Promise.resolve();
|
|
317
|
+
__privateGet(this, _pendingReady) ?? __privateSet(this, _pendingReady, new Promise((resolve) => {
|
|
318
|
+
client.once("ready", () => {
|
|
319
|
+
__privateSet(this, _pendingReady, void 0);
|
|
320
|
+
resolve();
|
|
321
|
+
});
|
|
322
|
+
}));
|
|
323
|
+
return __privateGet(this, _pendingReady);
|
|
324
|
+
}
|
|
313
325
|
async teardown() {
|
|
314
326
|
if (this.isOwned() && __privateGet(this, _client)) {
|
|
315
327
|
const client = __privateGet(this, _client);
|
|
@@ -330,7 +342,7 @@ var ConnectionResolver = class {
|
|
|
330
342
|
}
|
|
331
343
|
if (client.options.maxRetriesPerRequest === null) {
|
|
332
344
|
__privateGet(this, _logger).warn(
|
|
333
|
-
"The shared client has maxRetriesPerRequest: null, so a
|
|
345
|
+
"The shared client has maxRetriesPerRequest: null, so a command issued while Redis is unreachable stays queued in the client for as long as the outage lasts. QueueService calls are bounded by enqueueTimeoutMs and the caller is released, but the command is not cancelled. Give the client a finite limit to fail fast."
|
|
334
346
|
);
|
|
335
347
|
}
|
|
336
348
|
const probe = duplicateConnection(client);
|
|
@@ -380,10 +392,17 @@ var ConnectionResolver = class {
|
|
|
380
392
|
_logger = new WeakMap();
|
|
381
393
|
_client = new WeakMap();
|
|
382
394
|
_options = new WeakMap();
|
|
395
|
+
_pendingReady = new WeakMap();
|
|
383
396
|
ConnectionResolver = __decorateClass([
|
|
384
397
|
Injectable(),
|
|
385
398
|
__decorateParam(0, Inject(BYMAX_QUEUE_OPTIONS))
|
|
386
399
|
], ConnectionResolver);
|
|
400
|
+
var WAITING_STATUSES = /* @__PURE__ */ new Set([
|
|
401
|
+
"connecting",
|
|
402
|
+
"connect",
|
|
403
|
+
"reconnecting",
|
|
404
|
+
"close"
|
|
405
|
+
]);
|
|
387
406
|
function isReplyError(err) {
|
|
388
407
|
return typeof err === "object" && err !== null && err.name === "ReplyError";
|
|
389
408
|
}
|
|
@@ -454,6 +473,18 @@ function timeoutException(operation, timeoutMs, cause) {
|
|
|
454
473
|
if (cause !== void 0) exception.cause = cause;
|
|
455
474
|
return exception;
|
|
456
475
|
}
|
|
476
|
+
function unavailableException(operation, cause) {
|
|
477
|
+
const exception = new QueueException(
|
|
478
|
+
QUEUE_ERROR_CODES.CONNECTION_TIMEOUT,
|
|
479
|
+
HttpStatus.INTERNAL_SERVER_ERROR,
|
|
480
|
+
{ operation, reason: "offline" }
|
|
481
|
+
);
|
|
482
|
+
exception.cause = cause;
|
|
483
|
+
return exception;
|
|
484
|
+
}
|
|
485
|
+
function isOfflineRejection(err) {
|
|
486
|
+
return err instanceof Error && err.message === "Stream isn't writeable and enableOfflineQueue options is false";
|
|
487
|
+
}
|
|
457
488
|
function isCommandTimeout(err) {
|
|
458
489
|
return err instanceof Error && err.message === "Command timed out";
|
|
459
490
|
}
|
|
@@ -461,7 +492,8 @@ async function withCommandTimeoutCode(operation, commandTimeoutMs, work) {
|
|
|
461
492
|
try {
|
|
462
493
|
return await work;
|
|
463
494
|
} catch (err) {
|
|
464
|
-
|
|
495
|
+
if (isCommandTimeout(err)) throw timeoutException(operation, commandTimeoutMs(), err);
|
|
496
|
+
throw isOfflineRejection(err) ? unavailableException(operation, err) : err;
|
|
465
497
|
}
|
|
466
498
|
}
|
|
467
499
|
|
|
@@ -477,13 +509,14 @@ function assertValidCron(repeat, queueOptions) {
|
|
|
477
509
|
});
|
|
478
510
|
}
|
|
479
511
|
}
|
|
480
|
-
var _queues;
|
|
512
|
+
var _queues, _warm;
|
|
481
513
|
var QueueService = class {
|
|
482
514
|
constructor(connection, options) {
|
|
483
515
|
this.connection = connection;
|
|
484
516
|
this.options = options;
|
|
485
517
|
__publicField(this, "logger", new Logger(QueueService.name));
|
|
486
518
|
__privateAdd(this, _queues, /* @__PURE__ */ new Map());
|
|
519
|
+
__privateAdd(this, _warm, /* @__PURE__ */ new Set());
|
|
487
520
|
}
|
|
488
521
|
connection;
|
|
489
522
|
options;
|
|
@@ -502,27 +535,38 @@ var QueueService = class {
|
|
|
502
535
|
__privateGet(this, _queues).set(queueName, queue);
|
|
503
536
|
return queue;
|
|
504
537
|
}
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
538
|
+
call(operation, queueName, run) {
|
|
539
|
+
let expired = false;
|
|
540
|
+
const guarded = withDeadline(
|
|
541
|
+
this.afterReady(queueName).then(() => {
|
|
542
|
+
if (expired) throw new Error("the call had already timed out");
|
|
543
|
+
return run();
|
|
544
|
+
}),
|
|
545
|
+
this.options.enqueueTimeoutMs,
|
|
546
|
+
() => {
|
|
547
|
+
expired = true;
|
|
548
|
+
return timeoutException(operation, this.options.enqueueTimeoutMs);
|
|
549
|
+
}
|
|
550
|
+
);
|
|
509
551
|
return withCommandTimeoutCode(
|
|
510
552
|
operation,
|
|
511
553
|
() => this.connection.getClient().options.commandTimeout ?? this.options.commandTimeoutMs,
|
|
512
|
-
|
|
554
|
+
guarded
|
|
513
555
|
);
|
|
514
556
|
}
|
|
557
|
+
async afterReady(queueName) {
|
|
558
|
+
if (__privateGet(this, _warm).has(queueName)) return;
|
|
559
|
+
await this.connection.awaitReady();
|
|
560
|
+
__privateGet(this, _warm).add(queueName);
|
|
561
|
+
}
|
|
515
562
|
queueFor(queueName) {
|
|
516
563
|
return this.getOrCreateQueue(queueName);
|
|
517
564
|
}
|
|
518
565
|
async enqueue(queueName, jobName, data, options) {
|
|
519
566
|
const job = await this.call(
|
|
520
567
|
"enqueue",
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
this.options.enqueueTimeoutMs,
|
|
524
|
-
() => this.enqueueTimeout("enqueue")
|
|
525
|
-
)
|
|
568
|
+
queueName,
|
|
569
|
+
() => this.queueFor(queueName).add(jobName, data, options)
|
|
526
570
|
);
|
|
527
571
|
return job;
|
|
528
572
|
}
|
|
@@ -541,16 +585,13 @@ var QueueService = class {
|
|
|
541
585
|
try {
|
|
542
586
|
const created = await this.call(
|
|
543
587
|
"enqueueBulk",
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
),
|
|
552
|
-
this.options.enqueueTimeoutMs,
|
|
553
|
-
() => this.enqueueTimeout("enqueueBulk")
|
|
588
|
+
queueName,
|
|
589
|
+
() => this.queueFor(queueName).addBulk(
|
|
590
|
+
jobs.map((job) => ({
|
|
591
|
+
name: job.name,
|
|
592
|
+
data: job.data,
|
|
593
|
+
...job.options === void 0 ? {} : { opts: job.options }
|
|
594
|
+
}))
|
|
554
595
|
)
|
|
555
596
|
);
|
|
556
597
|
return created;
|
|
@@ -568,7 +609,7 @@ var QueueService = class {
|
|
|
568
609
|
}
|
|
569
610
|
}
|
|
570
611
|
async getJob(queueName, jobId) {
|
|
571
|
-
const job = await this.call("getJob", this.queueFor(queueName).getJob(jobId));
|
|
612
|
+
const job = await this.call("getJob", queueName, () => this.queueFor(queueName).getJob(jobId));
|
|
572
613
|
return job ?? null;
|
|
573
614
|
}
|
|
574
615
|
async getJobs(queueName, status, start = 0, end = 50) {
|
|
@@ -577,13 +618,18 @@ var QueueService = class {
|
|
|
577
618
|
reason: `getJobs needs integers with 0 <= start <= end and at most ${String(MAX_JOBS_PAGE_SIZE)} jobs per call`
|
|
578
619
|
});
|
|
579
620
|
}
|
|
580
|
-
const jobs = await this.call(
|
|
621
|
+
const jobs = await this.call(
|
|
622
|
+
"getJobs",
|
|
623
|
+
queueName,
|
|
624
|
+
() => this.queueFor(queueName).getJobs([status], start, end)
|
|
625
|
+
);
|
|
581
626
|
return jobs;
|
|
582
627
|
}
|
|
583
628
|
async getMetrics(queueName) {
|
|
584
629
|
const counts = await this.call(
|
|
585
630
|
"getMetrics",
|
|
586
|
-
|
|
631
|
+
queueName,
|
|
632
|
+
() => this.queueFor(queueName).getJobCounts("waiting", "active", "completed", "failed", "delayed")
|
|
587
633
|
);
|
|
588
634
|
return {
|
|
589
635
|
queue: queueName,
|
|
@@ -602,24 +648,37 @@ var QueueService = class {
|
|
|
602
648
|
if ("pattern" in repeat) assertValidCron(repeat, this.options.queueOptions);
|
|
603
649
|
const job = await this.call(
|
|
604
650
|
"upsertJobScheduler",
|
|
605
|
-
|
|
651
|
+
queueName,
|
|
652
|
+
() => queue.upsertJobScheduler(schedulerId, repeat, jobTemplate)
|
|
606
653
|
);
|
|
607
654
|
return job;
|
|
608
655
|
}
|
|
609
656
|
async removeJobScheduler(queueName, schedulerId) {
|
|
610
|
-
return this.call(
|
|
657
|
+
return this.call(
|
|
658
|
+
"removeJobScheduler",
|
|
659
|
+
queueName,
|
|
660
|
+
() => this.queueFor(queueName).removeJobScheduler(schedulerId)
|
|
661
|
+
);
|
|
611
662
|
}
|
|
612
663
|
async getJobSchedulers(queueName, start = 0, end = 50, asc = true) {
|
|
613
|
-
return this.call(
|
|
664
|
+
return this.call(
|
|
665
|
+
"getJobSchedulers",
|
|
666
|
+
queueName,
|
|
667
|
+
() => this.queueFor(queueName).getJobSchedulers(start, end, asc)
|
|
668
|
+
);
|
|
614
669
|
}
|
|
615
670
|
async pauseQueue(queueName) {
|
|
616
|
-
await this.call("pauseQueue", this.queueFor(queueName).pause());
|
|
671
|
+
await this.call("pauseQueue", queueName, () => this.queueFor(queueName).pause());
|
|
617
672
|
}
|
|
618
673
|
async resumeQueue(queueName) {
|
|
619
|
-
await this.call("resumeQueue", this.queueFor(queueName).resume());
|
|
674
|
+
await this.call("resumeQueue", queueName, () => this.queueFor(queueName).resume());
|
|
620
675
|
}
|
|
621
676
|
async cleanQueue(queueName, gracePeriodMs, limit, status) {
|
|
622
|
-
return this.call(
|
|
677
|
+
return this.call(
|
|
678
|
+
"cleanQueue",
|
|
679
|
+
queueName,
|
|
680
|
+
() => this.queueFor(queueName).clean(gracePeriodMs, limit, status)
|
|
681
|
+
);
|
|
623
682
|
}
|
|
624
683
|
getCachedQueues() {
|
|
625
684
|
return __privateGet(this, _queues);
|
|
@@ -629,9 +688,11 @@ var QueueService = class {
|
|
|
629
688
|
await queue.close().catch(() => void 0);
|
|
630
689
|
}
|
|
631
690
|
__privateGet(this, _queues).clear();
|
|
691
|
+
__privateGet(this, _warm).clear();
|
|
632
692
|
}
|
|
633
693
|
};
|
|
634
694
|
_queues = new WeakMap();
|
|
695
|
+
_warm = new WeakMap();
|
|
635
696
|
QueueService = __decorateClass([
|
|
636
697
|
Injectable(),
|
|
637
698
|
__decorateParam(0, Inject(ConnectionResolver)),
|
|
@@ -734,6 +795,17 @@ var MetricsService = class {
|
|
|
734
795
|
MetricsService = __decorateClass([
|
|
735
796
|
Injectable()
|
|
736
797
|
], MetricsService);
|
|
798
|
+
|
|
799
|
+
// src/server/utils/abandoned-workers.ts
|
|
800
|
+
var abandoned = /* @__PURE__ */ new WeakSet();
|
|
801
|
+
function abandonWorker(worker) {
|
|
802
|
+
abandoned.add(worker);
|
|
803
|
+
}
|
|
804
|
+
function isAbandoned(worker) {
|
|
805
|
+
return abandoned.has(worker);
|
|
806
|
+
}
|
|
807
|
+
|
|
808
|
+
// src/server/services/worker-registry.service.ts
|
|
737
809
|
var _entries;
|
|
738
810
|
var WorkerRegistry = class {
|
|
739
811
|
constructor(connection, options) {
|
|
@@ -754,7 +826,7 @@ var WorkerRegistry = class {
|
|
|
754
826
|
try {
|
|
755
827
|
worker = new Worker(
|
|
756
828
|
queueName,
|
|
757
|
-
toProcessor(handler, options?.jobTimeoutMs),
|
|
829
|
+
toProcessor(handler, options?.jobTimeoutMs, () => isAbandoned(worker)),
|
|
758
830
|
bullOpts
|
|
759
831
|
);
|
|
760
832
|
} catch (cause) {
|
|
@@ -968,7 +1040,7 @@ var JobTimeoutError = class extends Error {
|
|
|
968
1040
|
this.name = "JobTimeoutError";
|
|
969
1041
|
}
|
|
970
1042
|
};
|
|
971
|
-
function
|
|
1043
|
+
function adaptHandler(handler, timeoutMs) {
|
|
972
1044
|
if (timeoutMs === void 0) return (job, _token, signal) => handler(job, signal);
|
|
973
1045
|
return (job, _token, signal) => {
|
|
974
1046
|
const own = new AbortController();
|
|
@@ -990,6 +1062,16 @@ function toProcessor(handler, timeoutMs) {
|
|
|
990
1062
|
});
|
|
991
1063
|
};
|
|
992
1064
|
}
|
|
1065
|
+
function toProcessor(handler, timeoutMs, abandoned2) {
|
|
1066
|
+
const run = adaptHandler(handler, timeoutMs);
|
|
1067
|
+
return (job, token, signal) => run(job, token, signal).then(
|
|
1068
|
+
(value) => abandoned2() ? new Promise(() => void 0) : value,
|
|
1069
|
+
(error) => {
|
|
1070
|
+
if (abandoned2()) return new Promise(() => void 0);
|
|
1071
|
+
throw error;
|
|
1072
|
+
}
|
|
1073
|
+
);
|
|
1074
|
+
}
|
|
993
1075
|
var _events, _connections;
|
|
994
1076
|
var QueueEventsRegistry = class {
|
|
995
1077
|
constructor(connection, options) {
|
|
@@ -1168,6 +1250,7 @@ ProcessorDiscoveryService = __decorateClass([
|
|
|
1168
1250
|
__decorateParam(1, Inject(WorkerRegistry)),
|
|
1169
1251
|
__decorateParam(2, Inject(QueueEventsRegistry))
|
|
1170
1252
|
], ProcessorDiscoveryService);
|
|
1253
|
+
var FORCE_CLOSE_GRACE_MS = 1e3;
|
|
1171
1254
|
var DrainTimeoutError = class extends Error {
|
|
1172
1255
|
};
|
|
1173
1256
|
var QueueLifecycle = class {
|
|
@@ -1201,25 +1284,33 @@ var QueueLifecycle = class {
|
|
|
1201
1284
|
await this.closeQueues();
|
|
1202
1285
|
await this.teardownConnection();
|
|
1203
1286
|
this.logger.log(
|
|
1204
|
-
`Queue shutdown complete in ${String(Date.now() - (this.startedAt ?? Date.now()))}ms (
|
|
1287
|
+
`Queue shutdown complete in ${String(Date.now() - (this.startedAt ?? Date.now()))}ms (${String(this.forcedWorkers)} worker(s) stopped without draining)`
|
|
1205
1288
|
);
|
|
1206
1289
|
}
|
|
1207
1290
|
async closeWorkers(deadline) {
|
|
1208
|
-
let
|
|
1291
|
+
let undrained = 0;
|
|
1209
1292
|
await Promise.all(
|
|
1210
1293
|
Array.from(this.workers.getAll(), async ([name, worker]) => {
|
|
1211
1294
|
try {
|
|
1212
|
-
await bounded(
|
|
1295
|
+
await bounded(
|
|
1296
|
+
worker.pause().then(() => worker.close()),
|
|
1297
|
+
deadline
|
|
1298
|
+
);
|
|
1213
1299
|
} catch (cause) {
|
|
1214
|
-
|
|
1300
|
+
undrained++;
|
|
1301
|
+
abandonWorker(worker);
|
|
1302
|
+
const stopped = "it is stopped without waiting for its in-flight job, which becomes stalled and is retried";
|
|
1215
1303
|
this.logger.warn(
|
|
1216
|
-
cause instanceof DrainTimeoutError ? `Worker "${name}" exceeded the ${String(this.resolved.shutdown.drainTimeoutMs)}ms drain budget
|
|
1304
|
+
cause instanceof DrainTimeoutError ? `Worker "${name}" exceeded the ${String(this.resolved.shutdown.drainTimeoutMs)}ms drain budget (${QUEUE_ERROR_CODES.SHUTDOWN_TIMEOUT_EXCEEDED}); ${stopped}` : `Worker "${name}" failed to close gracefully (${describeError(cause)}); ${stopped}`
|
|
1305
|
+
);
|
|
1306
|
+
const deadlineError = () => new DrainTimeoutError();
|
|
1307
|
+
await withDeadline(worker.close(true), FORCE_CLOSE_GRACE_MS, deadlineError).catch(
|
|
1308
|
+
() => void 0
|
|
1217
1309
|
);
|
|
1218
|
-
void worker.close(true).catch(() => void 0);
|
|
1219
1310
|
}
|
|
1220
1311
|
})
|
|
1221
1312
|
);
|
|
1222
|
-
return
|
|
1313
|
+
return undrained;
|
|
1223
1314
|
}
|
|
1224
1315
|
async closeQueueEvents(deadline) {
|
|
1225
1316
|
await Promise.all(
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@bymax-one/nest-queue",
|
|
3
|
-
"version": "1.2.
|
|
3
|
+
"version": "1.2.15",
|
|
4
4
|
"description": "NestJS dynamic module wrapping BullMQ — typed jobs, flows, job schedulers, deduplication, OpenTelemetry, graceful shutdown",
|
|
5
5
|
"author": "Bymax One <support@bymax.one>",
|
|
6
6
|
"license": "MIT",
|