@bymax-one/nest-queue 1.2.13 → 1.2.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +117 -1
- package/README.md +106 -53
- package/dist/server/index.cjs +369 -100
- package/dist/server/index.d.cts +117 -32
- package/dist/server/index.d.ts +117 -32
- package/dist/server/index.mjs +369 -100
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -11,6 +11,120 @@ versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
11
11
|
|
|
12
12
|
---
|
|
13
13
|
|
|
14
|
+
## [1.2.15] — 2026-11-14
|
|
15
|
+
|
|
16
|
+
Against 1.2.14 the built declarations gain `ConnectionResolver.awaitReady()`; nothing else in the public API changes.
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
|
|
20
|
+
- **A call made while the connection is down now reports the documented error.** With the owned connection's offline queue off, a call on a queue already in use during a refused-connection outage rejected at once with ioredis's plain `Stream isn't writeable and enableOfflineQueue options is false`, so a host retrying on `CONNECTION_TIMEOUT` as documented never retried. It is now `CONNECTION_TIMEOUT` with `details.operation` and `details.reason: 'offline'`, and the ioredis error as `cause`.
|
|
21
|
+
- **The first call on a queue no longer waits for Redis without a deadline.** `getMetrics`, `getJob`, `getJobs`, the scheduler calls, `pauseQueue`, `resumeQueue` and `cleanQueue` on a queue name not yet used waited in BullMQ for the client to be ready, with no bound, for as long as the outage lasted. Every `QueueService` call is now bounded by `enqueueTimeoutMs` and rejects with `CONNECTION_TIMEOUT`.
|
|
22
|
+
- **Shutdown no longer leaves a worker running after its drain budget.** A handler that never settled kept the worker's lock-renewal timer alive after `app.close()`: the process did not exit by itself and logged `Connection is closed.` repeatedly. BullMQ ignores a forced close once a graceful one has started, so the worker is now paused first and closed with `close(true)` as its first close when the budget runs out. What the abandoned handler produces after the budget, a result or a failure, is dropped, so BullMQ cannot resume the stopped worker and reopen its connection. The handler's own promise still keeps running, since user code cannot be cancelled. A sandboxed worker is the exception: its job runs in a child process the registry cannot wrap, and in a measured run the host process did not exit by itself after the budget ended with such a job still running.
|
|
23
|
+
|
|
24
|
+
### Documentation
|
|
25
|
+
|
|
26
|
+
- `enqueueTimeoutMs` is documented as the per-call deadline of every `QueueService` call, and `queue.connection_timeout` as covering a connection that is down.
|
|
27
|
+
|
|
28
|
+
#### Apply to a derived backend
|
|
29
|
+
|
|
30
|
+
- If you match the ioredis error `Stream isn't writeable and enableOfflineQueue options is false` on a call, match `CONNECTION_TIMEOUT` instead; `details.reason` is `'offline'` and the original error is its `cause`.
|
|
31
|
+
- If a `getMetrics`, `getJob`, `getJobs`, scheduler, pause, resume or clean call relied on waiting out an outage on a queue's first use, it now rejects with `CONNECTION_TIMEOUT` after `enqueueTimeoutMs`; raise the option to wait longer.
|
|
32
|
+
- If you replace `ConnectionResolver` with your own implementation, add `awaitReady(): Promise<void>`: `QueueService` calls it before the first call on each queue name.
|
|
33
|
+
- If you parse the shutdown log lines, the wording changed: `forcing close` is gone and the summary reads `N worker(s) stopped without draining`.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## [1.2.14] — 2026-11-14
|
|
38
|
+
|
|
39
|
+
Several defaults change in this release, so read the Apply note below. A boot that succeeds, an
|
|
40
|
+
enqueue that Redis answers and a job that completes behave as before.
|
|
41
|
+
|
|
42
|
+
### Fixed
|
|
43
|
+
|
|
44
|
+
- **A failed Mode B startup no longer leaves a client reconnecting.** When Redis was not ready in
|
|
45
|
+
time the library rejected the boot but kept the `ioredis` client it had opened, which kept
|
|
46
|
+
reconnecting and held the process open. The
|
|
47
|
+
client is now closed on every startup failure.
|
|
48
|
+
- **`connectionReadyTimeoutMs` is a budget, not a first-error limit.** The first connection error
|
|
49
|
+
used to reject the boot at once with the raw `ioredis` error, so a
|
|
50
|
+
Redis restarting during a deploy failed the application. Errors while the client is
|
|
51
|
+
reconnecting are now waited through until the budget ends; the boot then rejects with
|
|
52
|
+
`queue.connection_timeout` whose `details` carry `timeoutMs` and `lastErrorCode` (for example
|
|
53
|
+
`ECONNREFUSED`), never the message, which can name the host. A refused credential
|
|
54
|
+
(`WRONGPASS`, `NOAUTH`) or a client that stopped reconnecting rejects at once with
|
|
55
|
+
`queue.connection_invalid`. Code that matched the raw `ioredis` error of a failed boot should
|
|
56
|
+
match the `QueueException` code instead.
|
|
57
|
+
|
|
58
|
+
### Changed
|
|
59
|
+
|
|
60
|
+
- **A producer call can no longer hang on a Redis that does not answer.** On the connection the
|
|
61
|
+
library opens, every command now carries a 5 s deadline (`commandTimeoutMs`, or the ioredis
|
|
62
|
+
`commandTimeout` you set), and `enqueue` and `enqueueBulk` reject with `CONNECTION_TIMEOUT` after
|
|
63
|
+
`enqueueTimeoutMs` (10 s) in either connection mode. A command that hits the connection's deadline
|
|
64
|
+
is reported the same way, as `CONNECTION_TIMEOUT` with `details.operation` and `timeoutMs` and the
|
|
65
|
+
original error as `cause`.
|
|
66
|
+
- **The owned connection no longer buffers commands during an outage.** Its offline queue is off, so
|
|
67
|
+
a call made while Redis is down fails at once instead of waiting in memory. Set `options.enableOfflineQueue: true` to keep the old
|
|
68
|
+
behaviour. Worker and QueueEvents connections keep their offline queue and have no command deadline.
|
|
69
|
+
- **A shared client with `maxRetriesPerRequest: null` is reported at boot.** Calls other than `enqueue` and
|
|
70
|
+
`enqueueBulk` then wait for as long as an outage lasts; the library warns and does not refuse.
|
|
71
|
+
- **Shutdown is bounded by one budget, not one per worker.** Workers are closed in parallel and the
|
|
72
|
+
queue-events and connection steps share what is left of `drainTimeoutMs`; a connection that has not
|
|
73
|
+
quit by then is disconnected. With several stuck workers the previous sequence took the budget once
|
|
74
|
+
for each.
|
|
75
|
+
- **`unregister` releases the connection even when the worker's `close()` throws.** It then rethrows
|
|
76
|
+
that error; the registry entry is dropped and the socket closed either way.
|
|
77
|
+
- **`jobTimeoutMs` bounds a handler that never settles.** It is a new worker option with no default,
|
|
78
|
+
a whole number of milliseconds from 1 to 2 147 483 647, as are `commandTimeoutMs` and `enqueueTimeoutMs`.
|
|
79
|
+
When it passes, the job fails with a `queue.job_timeout` error, retries under its `attempts` and
|
|
80
|
+
`backoff`, and its concurrency slot is free again. Handlers receive an `AbortSignal` as a second
|
|
81
|
+
argument, which aborts at that moment or when BullMQ cancels the job (`worker.cancelJob()`,
|
|
82
|
+
`worker.cancelAllJobs()`); closing the worker does not abort it. Before, a hung handler held its
|
|
83
|
+
slot for good and the rest of the queue waited.
|
|
84
|
+
- **Retries are spread by default.** The default backoff now carries 50% jitter: a retry waits between
|
|
85
|
+
half of its exponential delay and the whole of it, so jobs that failed together no longer retry in
|
|
86
|
+
the same instant after an outage. A `backoff` you set yourself is used as given.
|
|
87
|
+
- **`getJobs` reads a bounded window.** At most 1 000 jobs per call, with integer bounds and
|
|
88
|
+
`0 <= start <= end`; an `end` of `-1`, which BullMQ reads as the whole list, is refused with
|
|
89
|
+
`INVALID_OPTIONS`.
|
|
90
|
+
|
|
91
|
+
### Documentation
|
|
92
|
+
|
|
93
|
+
- **Queue names are documented as keys, not user input**, because each distinct name keeps a queue and
|
|
94
|
+
its Redis resources for the life of the process.
|
|
95
|
+
- **The `prefix` option's JSDoc says it is a namespace, not an access control**.
|
|
96
|
+
- **A timed-out `enqueue` may still have been added**, and the README says to pass a deterministic
|
|
97
|
+
`jobId` for anything that must not run twice.
|
|
98
|
+
- **The README no longer says a Redis outage fails fast regardless of configuration.** It now states
|
|
99
|
+
what bounds a producer call and when the bound does not apply.
|
|
100
|
+
|
|
101
|
+
#### Apply to a derived backend
|
|
102
|
+
|
|
103
|
+
- If you matched ioredis's `Command timed out` error on a producer or inspection call, match
|
|
104
|
+
`CONNECTION_TIMEOUT` instead; the original error is its `cause`.
|
|
105
|
+
- If you rely on an `enqueue` waiting out a Redis outage, it now rejects with `CONNECTION_TIMEOUT`
|
|
106
|
+
after `enqueueTimeoutMs` (the add may still complete in Redis afterwards, so retry with a stable
|
|
107
|
+
`jobId` or `deduplication`); raise the option to keep waiting longer.
|
|
108
|
+
- If you assert exact retry delays, expect them between 50% and 100% of the computed value; set
|
|
109
|
+
`defaultJobOptions.backoff` without `jitter` to keep exact delays.
|
|
110
|
+
- If you call `getJobs` with an `end` of `-1` or a window over 1 000 jobs, those calls now throw
|
|
111
|
+
`INVALID_OPTIONS`. Page through the list with several calls of at most 1 000 jobs, for example
|
|
112
|
+
`getJobs(queue, status, start, start + 499)` in a loop until a page comes back short; the README
|
|
113
|
+
has the loop.
|
|
114
|
+
- Nothing changes until you set `jobTimeoutMs`. If you do, a handler that runs longer than it is
|
|
115
|
+
failed and retried while its own promise keeps running unless it honours the `AbortSignal`, so keep
|
|
116
|
+
handlers idempotent. If you wrap `ProgrammaticWorkerConfig.handler`, accept the new optional second
|
|
117
|
+
parameter.
|
|
118
|
+
- If you relied on shutdown taking up to one `drainTimeoutMs` per stuck worker, it now takes at most
|
|
119
|
+
one in total; raise `drainTimeoutMs` if jobs need longer.
|
|
120
|
+
- If your Mode B code calls a `QueueService` method during a Redis outage and expected it to wait for
|
|
121
|
+
the reconnect, it now rejects at once, since the owned connection has no offline queue; set
|
|
122
|
+
`connection.options.enableOfflineQueue: true` to restore that.
|
|
123
|
+
- A command that takes longer than 5 s on the owned connection (a very large `getJobs` range, for
|
|
124
|
+
example) now rejects; raise `commandTimeoutMs`.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
14
128
|
## [1.2.13] — 2026-11-12
|
|
15
129
|
|
|
16
130
|
No action required. Measured with `diff -r` of the unpacked 1.2.12 tarball against a fresh
|
|
@@ -922,7 +1036,9 @@ v6 peer range.
|
|
|
922
1036
|
|
|
923
1037
|
---
|
|
924
1038
|
|
|
925
|
-
[Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.
|
|
1039
|
+
[Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.15...HEAD
|
|
1040
|
+
[1.2.15]: https://github.com/bymaxone/nest-queue/compare/v1.2.14...v1.2.15
|
|
1041
|
+
[1.2.14]: https://github.com/bymaxone/nest-queue/compare/v1.2.13...v1.2.14
|
|
926
1042
|
[1.2.13]: https://github.com/bymaxone/nest-queue/compare/v1.2.12...v1.2.13
|
|
927
1043
|
[1.2.12]: https://github.com/bymaxone/nest-queue/compare/v1.2.11...v1.2.12
|
|
928
1044
|
[1.2.11]: https://github.com/bymaxone/nest-queue/compare/v1.2.10...v1.2.11
|
package/README.md
CHANGED
|
@@ -64,9 +64,9 @@ peer dependencies, so you control exact versions and the supply-chain surface st
|
|
|
64
64
|
### 🔌 Connections & Shutdown
|
|
65
65
|
|
|
66
66
|
- ✅ **Dual-mode connection** — bring your own `ioredis` client, or let the library open its own from a URL or an options object
|
|
67
|
-
- ✅ **Per-role retry policy** — `maxRetriesPerRequest: null` reaches only the duplicated Worker and QueueEvents connections; the producer keeps
|
|
67
|
+
- ✅ **Per-role retry policy** — `maxRetriesPerRequest: null` reaches only the duplicated Worker and QueueEvents connections; the producer keeps a finite retry limit, a 5 s command deadline and no offline queue on the connection the library opens, and `enqueue` has its own 10 s deadline, so a Redis outage rejects a call in seconds instead of holding the request; a shared client with `maxRetriesPerRequest: null` is the exception, and the library warns about it at boot
|
|
68
68
|
- ✅ **One namespace, four objects** — the configured `prefix` reaches `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer`, so producers and consumers can never end up on separate keyspaces
|
|
69
|
-
- ✅ **Bounded graceful shutdown** — a single ordered sequence drains every worker under
|
|
69
|
+
- ✅ **Bounded graceful shutdown** — a single ordered sequence drains every worker under one deadline and then stops any that are still busy, whose jobs stall and are retried; delivery is at-least-once by design, so handlers must be idempotent
|
|
70
70
|
- ✅ **No unlistened emitter** — every `Queue`, `Worker`, `QueueEvents` and `FlowProducer` the library creates carries a fallback `error` listener, so a transient Redis fault cannot become an uncaught exception in your process; your own `@OnWorkerEvent('error')` or `@OnQueueEvent('error')` still runs alongside it
|
|
71
71
|
|
|
72
72
|
### 🧩 Developer Experience
|
|
@@ -229,20 +229,22 @@ export class EmailModule {}
|
|
|
229
229
|
|
|
230
230
|
Every option passed to `forRoot` / `forRootAsync`. Only `connection` is required.
|
|
231
231
|
|
|
232
|
-
| Option | Type | Default
|
|
233
|
-
| -------------------------- | ------------------------------------------------ |
|
|
234
|
-
| `connection` | `{ client } \| { url, options? } \| { options }` | —
|
|
235
|
-
| `prefix` | `string` | `'bull'`
|
|
236
|
-
| `defaultJobOptions` | `JobsOptions` | `attempts: 3` · exponential backoff `2000` ms · `removeOnComplete: { age: 24 h, count: 1000 }` · `removeOnFail: { age: 7 d, count: 5000 }` | Merged into every job enqueued through the service; per-job options win. The bounded `removeOn*` caps are what keep Redis from growing without limit under a failure storm.
|
|
237
|
-
| `queueOptions` | `Partial<QueueOptions>` | `{}`
|
|
238
|
-
| `flows.enabled` | `boolean` | `false`
|
|
239
|
-
| `metrics.enabled` | `boolean` | `false`
|
|
240
|
-
| `metrics.cacheTtlMs` | `number` | `5_000`
|
|
241
|
-
| `telemetry` | `Telemetry` | —
|
|
242
|
-
| `shutdown.drainTimeoutMs` | `number` | `30_000`
|
|
243
|
-
| `shutdown.drainOnShutdown` | `boolean` | `false`
|
|
244
|
-
| `connectionReadyTimeoutMs` | `number` | `10_000`
|
|
245
|
-
| `
|
|
232
|
+
| Option | Type | Default | Description |
|
|
233
|
+
| -------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
234
|
+
| `connection` | `{ client } \| { url, options? } \| { options }` | — | **Required.** Mode A supplies an existing `ioredis` client; Mode B lets the library open its own. The arms are mutually exclusive. |
|
|
235
|
+
| `prefix` | `string` | `'bull'` | Redis key namespace. Applied to the `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer` — see [Architecture](#-architecture). |
|
|
236
|
+
| `defaultJobOptions` | `JobsOptions` | `attempts: 3` · exponential backoff `2000` ms with 50% jitter · `removeOnComplete: { age: 24 h, count: 1000 }` · `removeOnFail: { age: 7 d, count: 5000 }` | Merged into every job enqueued through the service; per-job options win. The bounded `removeOn*` caps are what keep Redis from growing without limit under a failure storm. |
|
|
237
|
+
| `queueOptions` | `Partial<QueueOptions>` | `{}` | Extra BullMQ `Queue` options. `connection`, `defaultJobOptions` and `prefix` are owned by the library and cannot be set here. |
|
|
238
|
+
| `flows.enabled` | `boolean` | `false` | `FlowService` is always provided; while disabled, every method except `close()` throws `FLOW_DISABLED`. |
|
|
239
|
+
| `metrics.enabled` | `boolean` | `false` | `MetricsService` is always provided; while disabled, every method throws `METRICS_DISABLED`. |
|
|
240
|
+
| `metrics.cacheTtlMs` | `number` | `5_000` | TTL of the metrics snapshot cache. `0` disables caching. Must be `>= 0`. |
|
|
241
|
+
| `telemetry` | `Telemetry` | — | A BullMQ `Telemetry` instance (typically `new BullMQOtel(...)` from the optional `bullmq-otel` peer), attached to every `Queue` and `Worker`. |
|
|
242
|
+
| `shutdown.drainTimeoutMs` | `number` | `30_000` | Upper bound on the graceful drain before workers are force-closed. Must be `> 0`. |
|
|
243
|
+
| `shutdown.drainOnShutdown` | `boolean` | `false` | Development and test only: once the HTTP server has stopped, call `drain()` on every cached queue, which **removes** its waiting jobs (delayed and active jobs are left alone). Leave it `false` in production. |
|
|
244
|
+
| `connectionReadyTimeoutMs` | `number` | `10_000` | Mode B only: the budget for Redis to reach `ready`. A connection error while the client is still reconnecting does not end the wait; when the budget ends, startup fails with `CONNECTION_TIMEOUT` carrying `lastErrorCode` (a code such as `ECONNREFUSED`, never the message) and the client the library opened is closed. |
|
|
245
|
+
| `commandTimeoutMs` | `number` | `5_000` | Mode B only: the deadline for one Redis command on the connection the library opened (ioredis `commandTimeout`); an `options.commandTimeout` wins. A call that hits it rejects with `CONNECTION_TIMEOUT` (`details` carries `operation` and `timeoutMs`), as when `enqueueTimeoutMs` passes. Worker and QueueEvents connections have none. |
|
|
246
|
+
| `enqueueTimeoutMs` | `number` | `10_000` | The deadline for one `QueueService` call that reaches Redis, in either mode, including the wait for the client to be ready on a queue's first call; it rejects with `CONNECTION_TIMEOUT`, and an `enqueue` may still complete in Redis, so retry with a stable `jobId` or `deduplication`. |
|
|
247
|
+
| `isGlobal` | `boolean` | `true` | Register the module globally, so `QueueService` is injectable without re-importing. |
|
|
246
248
|
|
|
247
249
|
> [!WARNING]
|
|
248
250
|
> **`shutdown.drainOnShutdown` deletes jobs.** When it is `true`, shutdown calls `drain()` on every
|
|
@@ -436,6 +438,10 @@ await this.queue.enqueue(
|
|
|
436
438
|
|
|
437
439
|
When a custom `jobId` already exists, or a `deduplication` rule keeps the job that is already there (a rule with `replace` swaps it for the new one instead), `enqueue`, `enqueueBulk` and the flow methods add nothing: BullMQ answers with the id of the **existing** job together with the data and options of **this** call, so the returned object does not describe what is stored. Call `QueueService.getJob` when you need the stored job.
|
|
438
440
|
|
|
441
|
+
### A timed-out `enqueue` may still have been added
|
|
442
|
+
|
|
443
|
+
When Redis does not answer in time, whether within `commandTimeoutMs` on the connection the library opened or within `enqueueTimeoutMs`, or the connection is down, `enqueue` and `enqueueBulk` reject with `CONNECTION_TIMEOUT`, but the library cannot cancel the command: if Redis applies it late, the job exists anyway. Retrying a call that timed out can therefore add the job twice. For anything that must not run twice, pass a deterministic `jobId` (for example one derived from the business key): BullMQ ignores a second add with the same `jobId` for as long as the first job is still stored, so a retry inside that window is safe. A completed job is removed by `removeOnComplete`, and from then on the same `jobId` adds a new job, so keep the retry inside that window or keep the handler idempotent. A `deduplication` rule gives the same guarantee for work that is allowed to repeat after a window.
|
|
444
|
+
|
|
439
445
|
### OpenTelemetry
|
|
440
446
|
|
|
441
447
|
Pass a `bullmq-otel` telemetry instance (optional peer dep) to trace spans from `enqueue` through the handler:
|
|
@@ -456,12 +462,20 @@ BymaxQueueModule.forRoot({
|
|
|
456
462
|
`QueueLifecycle` runs automatically when NestJS closes the application context, through its `onModuleDestroy` and `onApplicationShutdown` hooks (call `app.enableShutdownHooks()` so that a termination signal closes the context).
|
|
457
463
|
It implements a **bounded drain protocol** with at-least-once semantics:
|
|
458
464
|
|
|
459
|
-
1. All workers are closed
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
+
1. All workers are closed **in parallel**, within one budget (`drainTimeoutMs`, default 30 s) that
|
|
466
|
+
covers steps 1 to 4 together: however many workers are stuck, consumption stops after one budget,
|
|
467
|
+
not one per worker.
|
|
468
|
+
2. Each worker is paused (it stops fetching and waits for its active jobs) and then closed, or the
|
|
469
|
+
budget runs out — whichever comes first.
|
|
470
|
+
3. On timeout, the worker is stopped with `worker.close(true)` as its first close, so it does not
|
|
471
|
+
wait for the job and its lock renewal ends; the lock expires and the job stalls and is retried
|
|
472
|
+
by the next instance. What the abandoned handler produces afterwards, a result or a failure, is
|
|
473
|
+
dropped, so the stopped worker cannot restart its connection. The handler's own promise keeps
|
|
474
|
+
running, because user code cannot be cancelled. A sandboxed (file-based) worker is the
|
|
475
|
+
exception: its job runs in a child process the registry cannot wrap, and in a measured run the
|
|
476
|
+
host process did not exit by itself after the budget ended with such a job still running.
|
|
477
|
+
4. Queue-events and the connections the library opened for workers and queue-events are closed,
|
|
478
|
+
each within what is left of the budget; a connection that has not quit by then is disconnected.
|
|
465
479
|
5. Once NestJS has stopped the HTTP server (`onApplicationShutdown`), the flow producer, the
|
|
466
480
|
cached queues and, in Mode B, the library's own Redis connection are closed. NestJS runs
|
|
467
481
|
`onModuleDestroy` before it stops the HTTP server, so a request that is still being served can
|
|
@@ -469,7 +483,7 @@ It implements a **bounded drain protocol** with at-least-once semantics:
|
|
|
469
483
|
|
|
470
484
|
**Handlers must be idempotent.** Use `jobId`, `deduplication`, or application-level
|
|
471
485
|
idempotency keys to guard against reprocessing a job that was partially handled before
|
|
472
|
-
a crash or
|
|
486
|
+
a crash or a shutdown that outlasts the drain budget.
|
|
473
487
|
|
|
474
488
|
**DLQ pattern.** When a job exhausts its `attempts`, move it to a dead-letter queue:
|
|
475
489
|
|
|
@@ -497,14 +511,14 @@ BymaxQueueModule.forRoot({
|
|
|
497
511
|
|
|
498
512
|
## 🤔 Why over `@nestjs/bullmq`
|
|
499
513
|
|
|
500
|
-
| Concern | `@nestjs/bullmq` (official) | `@bymax-one/nest-queue`
|
|
501
|
-
| ------------- | ---------------------------------------------------- |
|
|
502
|
-
| Module setup | `BullModule.forRoot()` + `registerQueue()` per queue | Single `forRoot`/`forRootAsync`; queues created on demand by name
|
|
503
|
-
| Connection | Manual ioredis + `maxRetriesPerRequest` wiring | Dual mode (BYO / lib-owned); correct per-role retry policy applied automatically
|
|
504
|
-
| Job defaults | Per `registerQueue` | Centralized opinionated defaults with per-queue/per-job overrides
|
|
505
|
-
| Producer API | `@InjectQueue` + raw `queue.add` | Typed `enqueue<TData, TResult>()`, `enqueueBulk`, `upsertJobScheduler`
|
|
506
|
-
| Shutdown | Framework closes workers | Explicit bounded-drain protocol;
|
|
507
|
-
| Observability | Manual | `telemetry` passthrough (OpenTelemetry) + `MetricsService`
|
|
514
|
+
| Concern | `@nestjs/bullmq` (official) | `@bymax-one/nest-queue` |
|
|
515
|
+
| ------------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
|
516
|
+
| Module setup | `BullModule.forRoot()` + `registerQueue()` per queue | Single `forRoot`/`forRootAsync`; queues created on demand by name |
|
|
517
|
+
| Connection | Manual ioredis + `maxRetriesPerRequest` wiring | Dual mode (BYO / lib-owned); correct per-role retry policy applied automatically |
|
|
518
|
+
| Job defaults | Per `registerQueue` | Centralized opinionated defaults with per-queue/per-job overrides |
|
|
519
|
+
| Producer API | `@InjectQueue` + raw `queue.add` | Typed `enqueue<TData, TResult>()`, `enqueueBulk`, `upsertJobScheduler` |
|
|
520
|
+
| Shutdown | Framework closes workers | Explicit bounded-drain protocol; stops a busy worker at the deadline; stalled accounting |
|
|
521
|
+
| Observability | Manual | `telemetry` passthrough (OpenTelemetry) + `MetricsService` |
|
|
508
522
|
|
|
509
523
|
**Choose this library** when you want a single opinionated setup, the `@bymax-one/nest-cache`
|
|
510
524
|
connection-sharing story, enforced defaults, and a tested bounded-shutdown protocol.
|
|
@@ -543,7 +557,7 @@ wire connection and defaults yourself.
|
|
|
543
557
|
(@Processor / @Process / @OnWorkerEvent)
|
|
544
558
|
│
|
|
545
559
|
QueueLifecycle
|
|
546
|
-
(bounded drain →
|
|
560
|
+
(bounded drain → stop → disconnect)
|
|
547
561
|
```
|
|
548
562
|
|
|
549
563
|
### Design Principles
|
|
@@ -598,6 +612,30 @@ consumed from another. It is **not** an access control: anything that can reach
|
|
|
598
612
|
instance can read every namespace on it. Cross-tenant isolation that must survive a
|
|
599
613
|
compromised client belongs in Redis ACLs or separate instances.
|
|
600
614
|
|
|
615
|
+
### Reading more than 1 000 jobs
|
|
616
|
+
|
|
617
|
+
`getJobs` reads at most 1 000 jobs per call, and refuses BullMQ's `-1` ("everything") with `INVALID_OPTIONS`, because that reads a status list of any size into memory in one reply. Page through it instead:
|
|
618
|
+
|
|
619
|
+
```typescript
|
|
620
|
+
import type { QueueService } from '@bymax-one/nest-queue'
|
|
621
|
+
|
|
622
|
+
async function inspectFailed(
|
|
623
|
+
queueService: QueueService,
|
|
624
|
+
inspect: (job: unknown) => Promise<void>,
|
|
625
|
+
): Promise<void> {
|
|
626
|
+
const PAGE = 500
|
|
627
|
+
for (let start = 0; ; start += PAGE) {
|
|
628
|
+
const page = await queueService.getJobs('reports', 'failed', start, start + PAGE - 1)
|
|
629
|
+
for (const job of page) await inspect(job)
|
|
630
|
+
if (page.length < PAGE) break
|
|
631
|
+
}
|
|
632
|
+
}
|
|
633
|
+
```
|
|
634
|
+
|
|
635
|
+
### Queue names are keys, not user input
|
|
636
|
+
|
|
637
|
+
Every queue name used with `QueueService` keeps a `Queue` object and its Redis resources for the life of the process, and names are never evicted. A name built from request data (a tenant id, a URL segment) lets whoever controls it grow memory without limit and create keys in your Redis. Take queue names from a fixed set in code or validate them against an allowlist before use.
|
|
638
|
+
|
|
601
639
|
### Shutdown is bounded and at-least-once
|
|
602
640
|
|
|
603
641
|
A job still running when the drain deadline expires is **not** acknowledged, so BullMQ's stalled-job
|
|
@@ -629,7 +667,7 @@ check re-queues it and it runs again. **Every handler must be idempotent** — s
|
|
|
629
667
|
| Error payloads | `QueueException.details` restricted to scalar values — no `job.data`, no connection object |
|
|
630
668
|
| Job payloads | Treated as opaque; never deep-merged (prototype-pollution guard), never logged |
|
|
631
669
|
| Namespacing | `prefix` applied uniformly to `Queue`, `Worker`, `QueueEvents` and `FlowProducer` |
|
|
632
|
-
| Connection policy | `maxRetriesPerRequest: null` confined to duplicated consumer connections; producers keep
|
|
670
|
+
| Connection policy | `maxRetriesPerRequest: null` confined to duplicated consumer connections; producers keep finite retries, a command deadline and no offline queue (Mode B) |
|
|
633
671
|
| Delivery | At-least-once with a bounded drain; unacknowledged work is re-queued by BullMQ's stalled check |
|
|
634
672
|
| Supply chain | SHA-pinned third-party Actions, OSV-Scanner, TruffleHog, OpenSSF Scorecard; npm publish over OIDC |
|
|
635
673
|
| Input validation | Options validated at bootstrap; cron patterns validated by BullMQ's own parser, never a hand-rolled regex |
|
|
@@ -692,7 +730,7 @@ import type {
|
|
|
692
730
|
| `enqueue` | `<TData, TResult>(queue, name, data, opts?) → Promise<Job<TData, TResult>>` | Add a single job |
|
|
693
731
|
| `enqueueBulk` | `<TData, TResult>(queue, jobs) → Promise<Job<TData, TResult>[]>` | Add up to 1 000 jobs in one Redis roundtrip |
|
|
694
732
|
| `getJob` | `<TData, TResult>(queue, jobId) → Promise<Job<TData, TResult> \| null>` | Get job by ID; `null` when it does not exist |
|
|
695
|
-
| `getJobs` | `<TData, TResult>(queue, status, start?, end?) → Promise<Job<TData, TResult>[]>` | Page jobs in one status (default `0`–`50`)
|
|
733
|
+
| `getJobs` | `<TData, TResult>(queue, status, start?, end?) → Promise<Job<TData, TResult>[]>` | Page jobs in one status (default `0`–`50`, at most 1 000 per call; `-1` is refused) |
|
|
696
734
|
| `getMetrics` | `(queue) → Promise<QueueMetrics>` | Uncached snapshot of job counts per status, with a collection timestamp |
|
|
697
735
|
| `pauseQueue` | `(queue) → Promise<void>` | Pause a queue |
|
|
698
736
|
| `resumeQueue` | `(queue) → Promise<void>` | Resume a paused queue |
|
|
@@ -817,18 +855,18 @@ try {
|
|
|
817
855
|
}
|
|
818
856
|
```
|
|
819
857
|
|
|
820
|
-
| Code | HTTP | When thrown
|
|
821
|
-
| ---------------------------------------- | :--: |
|
|
822
|
-
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs`
|
|
823
|
-
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized,
|
|
824
|
-
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null`
|
|
825
|
-
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs`
|
|
826
|
-
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every`
|
|
827
|
-
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name
|
|
828
|
-
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled`
|
|
829
|
-
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled`
|
|
830
|
-
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed
|
|
831
|
-
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed
|
|
858
|
+
| Code | HTTP | When thrown |
|
|
859
|
+
| ---------------------------------------- | :--: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
860
|
+
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
|
|
861
|
+
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, Mode A's injected client is in an unusable `status`, or a Mode B client was refused by the server (`WRONGPASS`, `NOAUTH`) or stopped reconnecting during startup |
|
|
862
|
+
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
|
|
863
|
+
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` (`details` carries `timeoutMs` and the last error's `lastErrorCode`), or a `QueueService`/`FlowService` call got no answer from Redis: it timed out (`commandTimeoutMs` on the owned connection, or `enqueueTimeoutMs` for a `QueueService` call; `details` carries `operation` and `timeoutMs`) or the connection is down (`details` carries `operation` and `reason: 'offline'`); an enqueue may still have been added |
|
|
864
|
+
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
|
|
865
|
+
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
|
|
866
|
+
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
|
|
867
|
+
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
|
|
868
|
+
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
|
|
869
|
+
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
|
|
832
870
|
|
|
833
871
|
### Reserved codes
|
|
834
872
|
|
|
@@ -836,12 +874,12 @@ These are part of the published catalog and stable, but nothing currently throws
|
|
|
836
874
|
They are listed so a consumer writing an exhaustive `switch` over `QueueErrorCode`
|
|
837
875
|
compiles today and keeps compiling when a future release starts emitting them.
|
|
838
876
|
|
|
839
|
-
| Code | Why it is not thrown today
|
|
840
|
-
| --------------------------------- |
|
|
841
|
-
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses
|
|
842
|
-
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing
|
|
843
|
-
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them
|
|
844
|
-
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline
|
|
877
|
+
| Code | Why it is not thrown today |
|
|
878
|
+
| --------------------------------- | ----------------------------------------------------------------------------- |
|
|
879
|
+
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
|
|
880
|
+
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
|
|
881
|
+
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
|
|
882
|
+
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline stops the workers still busy rather than raising |
|
|
845
883
|
|
|
846
884
|
The same catalog is exported from `@bymax-one/nest-queue/shared` with no dependencies,
|
|
847
885
|
so a frontend can map codes to messages without importing the server surface.
|
|
@@ -874,7 +912,22 @@ that way. This error means one did not — measured with ioredis, a client creat
|
|
|
874
912
|
If you see it, the client you passed is not a plain ioredis client (a wrapper or a test
|
|
875
913
|
double whose `duplicate()` ignores its options). Pass the ioredis client itself, or let the
|
|
876
914
|
library open its own connection with Mode B. The producer keeps the client's own retry
|
|
877
|
-
policy
|
|
915
|
+
policy; with a finite `maxRetriesPerRequest` that is what lets it fail fast during an outage, and
|
|
916
|
+
every `QueueService` call is bounded by `enqueueTimeoutMs` whatever the client does.
|
|
917
|
+
|
|
918
|
+
### A handler never settles (the queue stops moving)
|
|
919
|
+
|
|
920
|
+
A handler whose promise never settles holds its concurrency slot for as long as the process lives, and BullMQ keeps renewing its lock, so the stalled-job check never reclaims it: with every slot held, the rest of the queue waits. Set `jobTimeoutMs` in the worker `options` to bound it. When the time passes, the job fails with a `queue.job_timeout` error and retries under its `attempts` and `backoff`, and the slot is free again. The handler receives the job's `AbortSignal` as its second argument; it aborts at that moment, or when BullMQ cancels the job (`worker.cancelJob()`, `worker.cancelAllJobs()`), and closing the worker does not abort it; pass it to whatever the handler awaits so the work stops with the job. A handler that ignores the signal keeps running in the background, and its retry can overlap it, so it must stay idempotent. There is no default, and sandboxed workers do not accept the option.
|
|
921
|
+
|
|
922
|
+
```typescript
|
|
923
|
+
@Processor('reports', { jobTimeoutMs: 60_000 })
|
|
924
|
+
export class ReportsProcessor {
|
|
925
|
+
@Process()
|
|
926
|
+
async build(job: Job, signal?: AbortSignal): Promise<void> {
|
|
927
|
+
await fetch('https://reports.internal/build', { method: 'POST', signal: signal ?? null })
|
|
928
|
+
}
|
|
929
|
+
}
|
|
930
|
+
```
|
|
878
931
|
|
|
879
932
|
### Jobs get stuck (not processed after restart)
|
|
880
933
|
|