@bymax-one/nest-queue 1.2.11 → 1.2.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +58 -1
- package/README.md +254 -324
- package/dist/server/index.cjs +36 -13
- package/dist/server/index.mjs +37 -14
- package/package.json +4 -1
package/CHANGELOG.md
CHANGED
|
@@ -11,6 +11,61 @@ versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
11
11
|
|
|
12
12
|
---
|
|
13
13
|
|
|
14
|
+
## [1.2.13] — 2026-11-12
|
|
15
|
+
|
|
16
|
+
No action required. Measured with `diff -r` of the unpacked 1.2.12 tarball against a fresh
|
|
17
|
+
`npm pack`, the files that differ are `README.md`, `CHANGELOG.md` and the `version` field of
|
|
18
|
+
`package.json`; `dist/` is byte-identical, so the public API, the declaration files and the peer
|
|
19
|
+
ranges are those of 1.2.12.
|
|
20
|
+
|
|
21
|
+
### Documentation
|
|
22
|
+
|
|
23
|
+
- **The bundle budgets are no longer misquoted.** The Testing section gave "server ≤ 18 KiB brotli,
|
|
24
|
+
shared ≤ 2 500 B"; the gate sets 9 200 B and 440 B. The README now points at `pnpm size` instead of
|
|
25
|
+
restating the numbers.
|
|
26
|
+
- **The tech stack names the real end-to-end dependency.** It listed `@testcontainers/redis`, which
|
|
27
|
+
is not a dependency; the suite uses `testcontainers`.
|
|
28
|
+
- **The Quick Start no longer implies `isGlobal: true` is needed.** It is the default, so the option is
|
|
29
|
+
gone from the snippet and the step says `QueueService` is injectable without it.
|
|
30
|
+
- **The connection-release claim covers only what the library opens.** In Mode A the shared client is
|
|
31
|
+
yours and the library never closes it.
|
|
32
|
+
- **Mode A, Mode B and the connection budget are one Connections section**, the deduplication examples
|
|
33
|
+
are one block, and the msgpackr entry is a short paragraph. API Reference and the error catalog now
|
|
34
|
+
follow Testing, the tech stack is a badge row, and the Quick Start is numbered.
|
|
35
|
+
- **Removed what a reader of the package does not need:** a note about an internal test fixture, a
|
|
36
|
+
history line about a BullMQ state, the pnpm 10 versus 11 narration, and the pointers to the technical
|
|
37
|
+
specification.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## [1.2.12] — 2026-11-09
|
|
42
|
+
|
|
43
|
+
No action required. The public API, the declaration files and the peer ranges are unchanged
|
|
44
|
+
(`check-api-surface` reports no difference against 1.2.11); the runtime bundles (`index.mjs`,
|
|
45
|
+
`index.cjs`) differ in code, measured with `diff -r` of a fresh build against the unpacked 1.2.11
|
|
46
|
+
tarball. The package also ships a corrected README.
|
|
47
|
+
|
|
48
|
+
### Fixed
|
|
49
|
+
|
|
50
|
+
- **An outage on a connection the library opens no longer prints `[ioredis] Unhandled error event`
|
|
51
|
+
to stderr.** In Mode B the owned connection listened for `error` only while it waited for
|
|
52
|
+
`ready`, so until BullMQ built something on it (BullMQ adds its own listener to a client it is
|
|
53
|
+
handed) a later outage reached ioredis with no listener, and ioredis wrote one line per
|
|
54
|
+
reconnect attempt to raw stderr, outside the structured logger. The owned connection now keeps a
|
|
55
|
+
listener for its whole life that logs through the module's logger. The same holds for the
|
|
56
|
+
duplicate opened for each `QueueEvents` (BullMQ listens on its own copy, not on this one) and for
|
|
57
|
+
the probe the Mode A check duplicates. Every error is logged as it arrives, with no throttling.
|
|
58
|
+
A client you pass in Mode A is yours and the library adds no listener to it.
|
|
59
|
+
|
|
60
|
+
### Documentation
|
|
61
|
+
|
|
62
|
+
- **The README's Mode A example now compiles and runs.** It imported `CACHE_REDIS_CLIENT` from
|
|
63
|
+
`@bymax-one/nest-cache`, a name that package does not export. The example now injects
|
|
64
|
+
`ConnectionManager` and passes `cache.getClient()`, and says that `getClient()` is typed
|
|
65
|
+
`Redis | Cluster` while Mode A takes a standalone `Redis`.
|
|
66
|
+
|
|
67
|
+
---
|
|
68
|
+
|
|
14
69
|
## [1.2.11] — 2026-11-06
|
|
15
70
|
|
|
16
71
|
Action required only if you extend or replace `QueueLifecycle`, or rely on the order in which the
|
|
@@ -867,7 +922,9 @@ v6 peer range.
|
|
|
867
922
|
|
|
868
923
|
---
|
|
869
924
|
|
|
870
|
-
[Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.
|
|
925
|
+
[Unreleased]: https://github.com/bymaxone/nest-queue/compare/v1.2.13...HEAD
|
|
926
|
+
[1.2.13]: https://github.com/bymaxone/nest-queue/compare/v1.2.12...v1.2.13
|
|
927
|
+
[1.2.12]: https://github.com/bymaxone/nest-queue/compare/v1.2.11...v1.2.12
|
|
871
928
|
[1.2.11]: https://github.com/bymaxone/nest-queue/compare/v1.2.10...v1.2.11
|
|
872
929
|
[1.2.10]: https://github.com/bymaxone/nest-queue/compare/v1.2.9...v1.2.10
|
|
873
930
|
[1.2.9]: https://github.com/bymaxone/nest-queue/compare/v1.2.8...v1.2.9
|
package/README.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
<p align="center">
|
|
8
8
|
<strong>Typed BullMQ queues for NestJS</strong><br />
|
|
9
|
-
<sub>BullMQ 6 · Workers · Flows · Job Schedulers · Deduplication · OpenTelemetry · Graceful Shutdown
|
|
9
|
+
<sub>BullMQ 6 · Workers · Flows · Job Schedulers · Deduplication · OpenTelemetry · Graceful Shutdown</sub>
|
|
10
10
|
</p>
|
|
11
11
|
|
|
12
12
|
<p align="center">
|
|
@@ -42,24 +42,13 @@ OpenTelemetry passthrough, and a tested bounded graceful-shutdown protocol.
|
|
|
42
42
|
The library has **zero direct dependencies** — `bullmq`, `ioredis`, and `@nestjs/*` arrive as
|
|
43
43
|
peer dependencies, so you control exact versions and the supply-chain surface stays minimal.
|
|
44
44
|
|
|
45
|
-
### Why nest-queue?
|
|
46
|
-
|
|
47
|
-
- **One setup, every queue.** A single `forRoot`/`forRootAsync`; queues are created on demand by
|
|
48
|
-
name — no `registerQueue()` per queue.
|
|
49
|
-
- **Correct connections by default.** Dual-mode (bring-your-own client or lib-owned) with the
|
|
50
|
-
per-role `maxRetriesPerRequest` policy applied automatically — the Queue connection keeps
|
|
51
|
-
default retries (fail fast), only the duplicated Worker/QueueEvents connections force `null`.
|
|
52
|
-
- **Current BullMQ API only.** Recurring jobs use Job Schedulers (`upsertJobScheduler`), never the
|
|
53
|
-
removed `addRepeatable`. Cron is validated by BullMQ's own parser, never a hand-rolled regex.
|
|
54
|
-
- **Safe shutdown.** A bounded-drain protocol with at-least-once semantics, force-close, and
|
|
55
|
-
stalled-job accounting — verified against a real Redis (Testcontainers) E2E suite.
|
|
56
|
-
|
|
57
45
|
---
|
|
58
46
|
|
|
59
47
|
## 🔥 Features
|
|
60
48
|
|
|
61
49
|
### 📤 Producing
|
|
62
50
|
|
|
51
|
+
- ✅ **One setup, every queue** — a single `forRoot` / `forRootAsync`; queues are created on demand by name, with no `registerQueue()` per queue
|
|
63
52
|
- ✅ **Typed producer API** — `enqueue<TData, TResult>()` threads both generics through to the returned `Job`, so `job.data` and `job.returnvalue` need no cast
|
|
64
53
|
- ✅ **Bounded bulk insert** — `enqueueBulk` adds up to 1 000 jobs in a single Redis roundtrip and rejects a larger batch outright, rather than letting one call inflate the heap
|
|
65
54
|
- ✅ **Native deduplication** — simple, throttle, debounce and keep-last-if-active, passed straight through to BullMQ; the library writes no deduplication logic of its own
|
|
@@ -82,8 +71,6 @@ peer dependencies, so you control exact versions and the supply-chain surface st
|
|
|
82
71
|
|
|
83
72
|
### 🧩 Developer Experience
|
|
84
73
|
|
|
85
|
-
- ✅ **Zero runtime dependencies** — BullMQ, ioredis and `@nestjs/*` all arrive as peer dependencies, so you pin the versions and the supply-chain surface stays yours
|
|
86
|
-
- ✅ **Two subpaths** — the server surface, and `./shared` carrying types and constants that pull in nothing at all
|
|
87
74
|
- ✅ **Stable error catalog** — namespaced codes exported from both subpaths, so a frontend maps a `code` without importing the server
|
|
88
75
|
- ✅ **Dual-format output** — ESM + CJS with declarations for each format, verified against the packed tarball on every PR
|
|
89
76
|
- ✅ **Opt-in observability** — OpenTelemetry passthrough via `bullmq-otel`, plus a `MetricsService` with a configurable TTL cache
|
|
@@ -132,20 +119,19 @@ Both subpaths ship ESM **and** CommonJS with declarations for each format, so a
|
|
|
132
119
|
> A full reference application lives in
|
|
133
120
|
> [`bymaxone/nest-queue-example`](https://github.com/bymaxone/nest-queue-example) —
|
|
134
121
|
> an API and a web console exercising every public export against a real Redis.
|
|
135
|
-
>
|
|
136
|
-
> The `test/consumer-app` directory in this repository is **not** that. It is a
|
|
137
|
-
> typecheck fixture: a few hundred lines of application-shaped code that CI
|
|
138
|
-
> compiles against the freshly built library on every pull request, to catch an
|
|
139
|
-
> ergonomic break in the decorator API that a signature test cannot see.
|
|
140
122
|
|
|
141
123
|
---
|
|
142
124
|
|
|
143
125
|
## 🚀 Quick Start
|
|
144
126
|
|
|
127
|
+
### 1. Install
|
|
128
|
+
|
|
145
129
|
```bash
|
|
146
130
|
pnpm add @bymax-one/nest-queue bullmq ioredis reflect-metadata
|
|
147
131
|
```
|
|
148
132
|
|
|
133
|
+
### 2. Load `reflect-metadata`
|
|
134
|
+
|
|
149
135
|
`reflect-metadata` is loaded **once by your application**, before anything else — this
|
|
150
136
|
is the standard NestJS entry point and the decorators in this library depend on it:
|
|
151
137
|
|
|
@@ -163,6 +149,8 @@ async function bootstrap(): Promise<void> {
|
|
|
163
149
|
void bootstrap()
|
|
164
150
|
```
|
|
165
151
|
|
|
152
|
+
### 3. Register the module
|
|
153
|
+
|
|
166
154
|
Wire the module in your root `AppModule`:
|
|
167
155
|
|
|
168
156
|
```typescript
|
|
@@ -174,7 +162,6 @@ import { EmailModule } from './email.module'
|
|
|
174
162
|
imports: [
|
|
175
163
|
BymaxQueueModule.forRoot({
|
|
176
164
|
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
177
|
-
isGlobal: true,
|
|
178
165
|
}),
|
|
179
166
|
EmailModule,
|
|
180
167
|
],
|
|
@@ -182,7 +169,9 @@ import { EmailModule } from './email.module'
|
|
|
182
169
|
export class AppModule {}
|
|
183
170
|
```
|
|
184
171
|
|
|
185
|
-
Enqueue a job
|
|
172
|
+
### 4. Enqueue a job
|
|
173
|
+
|
|
174
|
+
From any service:
|
|
186
175
|
|
|
187
176
|
```typescript
|
|
188
177
|
import { Injectable } from '@nestjs/common'
|
|
@@ -198,7 +187,9 @@ export class EmailService {
|
|
|
198
187
|
}
|
|
199
188
|
```
|
|
200
189
|
|
|
201
|
-
Process jobs
|
|
190
|
+
### 5. Process jobs
|
|
191
|
+
|
|
192
|
+
With a decorated handler:
|
|
202
193
|
|
|
203
194
|
```typescript
|
|
204
195
|
import { Injectable } from '@nestjs/common'
|
|
@@ -215,8 +206,11 @@ export class EmailProcessor {
|
|
|
215
206
|
}
|
|
216
207
|
```
|
|
217
208
|
|
|
218
|
-
|
|
219
|
-
|
|
209
|
+
### 6. Register the providers
|
|
210
|
+
|
|
211
|
+
Put the service and the processor in a feature module that `AppModule` imports, like the `EmailModule`
|
|
212
|
+
above. The module is global by default (`isGlobal` defaults to `true`), so `QueueService` is injectable
|
|
213
|
+
there without importing `BymaxQueueModule` again:
|
|
220
214
|
|
|
221
215
|
```typescript
|
|
222
216
|
import { Module } from '@nestjs/common'
|
|
@@ -262,41 +256,39 @@ first job is enqueued.
|
|
|
262
256
|
|
|
263
257
|
---
|
|
264
258
|
|
|
265
|
-
## 🔌
|
|
259
|
+
## 🔌 Connections
|
|
260
|
+
|
|
261
|
+
`connection` takes a client (**Mode A**) or a URL or ioredis options (**Mode B**); the arms are mutually
|
|
262
|
+
exclusive. In both modes the Queue connection keeps the ioredis defaults, and the Worker and
|
|
263
|
+
`QueueEvents` connections are `duplicate()`s of it with `maxRetriesPerRequest: null`.
|
|
264
|
+
|
|
265
|
+
### Mode A — your own client
|
|
266
266
|
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
`maxRetriesPerRequest: null` applied automatically; the Queue connection uses ioredis defaults.
|
|
267
|
+
Recommended when the application already runs `@bymax-one/nest-cache`: one client is shared and
|
|
268
|
+
duplicated per role, and your cache connection is left alone.
|
|
270
269
|
|
|
271
270
|
```typescript
|
|
272
271
|
import { BymaxQueueModule } from '@bymax-one/nest-queue'
|
|
273
|
-
import {
|
|
272
|
+
import { ConnectionManager } from '@bymax-one/nest-cache'
|
|
274
273
|
import type { Redis } from 'ioredis'
|
|
275
274
|
|
|
276
275
|
BymaxQueueModule.forRootAsync({
|
|
277
|
-
inject: [
|
|
278
|
-
useFactory: (
|
|
279
|
-
connection: { client:
|
|
280
|
-
isGlobal: true,
|
|
276
|
+
inject: [ConnectionManager],
|
|
277
|
+
useFactory: (cache: ConnectionManager) => ({
|
|
278
|
+
connection: { client: cache.getClient() as Redis },
|
|
281
279
|
}),
|
|
282
280
|
})
|
|
283
281
|
```
|
|
284
282
|
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
without touching your cache connection.
|
|
288
|
-
|
|
289
|
-
---
|
|
290
|
-
|
|
291
|
-
## 🔌 Mode B — Lib-Owned Connection
|
|
283
|
+
`BymaxCacheModule` is global by default and provides `ConnectionManager`. `getClient()` is typed
|
|
284
|
+
`Redis | Cluster` and Mode A takes a standalone `Redis`, so run `@bymax-one/nest-cache` in standalone mode.
|
|
292
285
|
|
|
293
|
-
Mode B
|
|
286
|
+
### Mode B — a connection the library owns
|
|
294
287
|
|
|
295
288
|
```typescript
|
|
296
289
|
// URL form
|
|
297
290
|
BymaxQueueModule.forRoot({
|
|
298
291
|
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
299
|
-
isGlobal: true,
|
|
300
292
|
})
|
|
301
293
|
|
|
302
294
|
// Options form
|
|
@@ -309,222 +301,30 @@ BymaxQueueModule.forRoot({
|
|
|
309
301
|
tls: {},
|
|
310
302
|
},
|
|
311
303
|
},
|
|
312
|
-
isGlobal: true,
|
|
313
|
-
})
|
|
314
|
-
```
|
|
315
|
-
|
|
316
|
-
The library opens one connection for all queues and flows, and two for each worker and for each
|
|
317
|
-
queue that has an `@OnQueueEvent` listener ([the budget is below](#-redis-connections)). The worker
|
|
318
|
-
and `QueueEvents` connections carry `maxRetriesPerRequest: null`.
|
|
319
|
-
|
|
320
|
-
---
|
|
321
|
-
|
|
322
|
-
## 🔌 Redis connections
|
|
323
|
-
|
|
324
|
-
How many connections the library holds is linear in what you register, so you can budget it before
|
|
325
|
-
you deploy. Counts were read from `INFO clients` against Redis 8 with BullMQ 6 and `ioredis` 6, and
|
|
326
|
-
every connection is released when the application context closes.
|
|
327
|
-
|
|
328
|
-
| What you register | Connections held |
|
|
329
|
-
| ---------------------------------------------- | ------------------------------------------------------------------------------ |
|
|
330
|
-
| The module (Mode B, library-owned connection) | 1, shared by every producer and every flow |
|
|
331
|
-
| The module (Mode A, your client) | 0 of its own; your client is yours and is never counted |
|
|
332
|
-
| A queue you enqueue to, with `QueueService` | 0 more: queues share the module's connection |
|
|
333
|
-
| `FlowService` (flows enabled) | 0 more |
|
|
334
|
-
| Each worker (`@Processor` or `WorkerRegistry`) | 2: a blocking connection that waits for jobs, and a command connection |
|
|
335
|
-
| Each queue observed with `@OnQueueEvent` | 2: a blocking connection that reads the event stream, and a command connection |
|
|
336
|
-
|
|
337
|
-
Producers and flows add nothing, so the cost comes from consumers. The blocking connection of a
|
|
338
|
-
worker cannot be shared: it is parked inside a blocking command until a job arrives.
|
|
339
|
-
|
|
340
|
-
**Worked example.** One application instance in Mode B with 20 workers and 5 queues observed with
|
|
341
|
-
`@OnQueueEvent` holds 1 + (2 × 20) + (2 × 5) = **51** connections; in Mode A the same instance holds
|
|
342
|
-
50, plus whatever your own client uses. Each instance of the application opens its own set, so three
|
|
343
|
-
instances hold 153.
|
|
344
|
-
|
|
345
|
-
**Managed Redis.** Hosted Redis plans cap the number of concurrent connections, and a plan sized for
|
|
346
|
-
a handful of clients is exhausted by one instance with a few dozen workers. Multiply the table by
|
|
347
|
-
the number of instances you run (include rolling deploys, where old and new instances overlap) and
|
|
348
|
-
compare it with the limit of your plan; self-managed Redis allows 10 000 by default (`maxclients`).
|
|
349
|
-
In `CLIENT LIST`, the blocking connection of a worker has a name that starts with
|
|
350
|
-
`<prefix>:<base64 of the queue name>`, which identifies it; the command connection is unnamed.
|
|
351
|
-
|
|
352
|
-
---
|
|
353
|
-
|
|
354
|
-
## 📖 API Reference
|
|
355
|
-
|
|
356
|
-
### `QueueService`
|
|
357
|
-
|
|
358
|
-
The central interaction point with BullMQ queues.
|
|
359
|
-
|
|
360
|
-
```typescript
|
|
361
|
-
import { QueueService } from '@bymax-one/nest-queue'
|
|
362
|
-
import type {
|
|
363
|
-
Job,
|
|
364
|
-
JobsOptions,
|
|
365
|
-
JobSchedulerRepeatOptions,
|
|
366
|
-
QueueMetrics,
|
|
367
|
-
} from '@bymax-one/nest-queue'
|
|
368
|
-
```
|
|
369
|
-
|
|
370
|
-
| Method | Signature | Description |
|
|
371
|
-
| -------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
|
372
|
-
| `enqueue` | `<TData, TResult>(queue, name, data, opts?) → Promise<Job<TData, TResult>>` | Add a single job |
|
|
373
|
-
| `enqueueBulk` | `<TData, TResult>(queue, jobs) → Promise<Job<TData, TResult>[]>` | Add up to 1 000 jobs in one Redis roundtrip |
|
|
374
|
-
| `getJob` | `<TData, TResult>(queue, jobId) → Promise<Job<TData, TResult> \| null>` | Get job by ID; `null` when it does not exist |
|
|
375
|
-
| `getJobs` | `<TData, TResult>(queue, status, start?, end?) → Promise<Job<TData, TResult>[]>` | Page jobs in one status (default `0`–`50`) |
|
|
376
|
-
| `getMetrics` | `(queue) → Promise<QueueMetrics>` | Uncached snapshot of job counts per status, with a collection timestamp |
|
|
377
|
-
| `pauseQueue` | `(queue) → Promise<void>` | Pause a queue |
|
|
378
|
-
| `resumeQueue` | `(queue) → Promise<void>` | Resume a paused queue |
|
|
379
|
-
| `cleanQueue` | `(queue, gracePeriodMs, limit, status?) → Promise<string[]>` | Remove up to `limit` jobs older than `gracePeriodMs` (`0` = no limit); returns removed ids |
|
|
380
|
-
| `upsertJobScheduler` | `<TData, TResult>(queue, id, repeat, template?) → Promise<Job<TData, TResult> \| undefined>` | Create or update a recurring Job Scheduler; returns the first scheduled job |
|
|
381
|
-
| `removeJobScheduler` | `(queue, id) → Promise<boolean>` | Remove a Job Scheduler by ID |
|
|
382
|
-
| `getJobSchedulers` | `(queue, start?, end?, asc?) → Promise<JobSchedulerJson[]>` | Page the schedulers registered on a queue |
|
|
383
|
-
| `getOrCreateQueue` | `<TData, TResult>(queue, overrides?) → Queue<TData, TResult>` | Return the cached raw `Queue` instance |
|
|
384
|
-
| `getCachedQueues` | `() → ReadonlyMap<string, Queue>` | Read-only view of every queue created so far |
|
|
385
|
-
|
|
386
|
-
### `WorkerRegistry`
|
|
387
|
-
|
|
388
|
-
Programmatic API to create and manage BullMQ workers at runtime.
|
|
389
|
-
|
|
390
|
-
```typescript
|
|
391
|
-
import path from 'node:path'
|
|
392
|
-
import { WorkerRegistry } from '@bymax-one/nest-queue'
|
|
393
|
-
import type { ProgrammaticWorkerConfig, SandboxedWorkerConfig } from '@bymax-one/nest-queue'
|
|
394
|
-
|
|
395
|
-
// Register an in-process worker
|
|
396
|
-
registry.register({
|
|
397
|
-
queueName: 'email',
|
|
398
|
-
handler: async (job) => {
|
|
399
|
-
/* … */
|
|
400
|
-
},
|
|
401
|
-
options: { concurrency: 4 },
|
|
402
|
-
})
|
|
403
|
-
|
|
404
|
-
// Register a sandboxed (out-of-process) processor
|
|
405
|
-
registry.registerSandboxed({
|
|
406
|
-
queueName: 'heavy-compute',
|
|
407
|
-
processorFile: path.resolve(__dirname, 'compute.processor.js'),
|
|
408
|
-
options: { concurrency: 2 },
|
|
409
|
-
})
|
|
410
|
-
```
|
|
411
|
-
|
|
412
|
-
### `FlowService` (opt-in)
|
|
413
|
-
|
|
414
|
-
Enable flows in the module options and inject the service:
|
|
415
|
-
|
|
416
|
-
```typescript
|
|
417
|
-
BymaxQueueModule.forRoot({
|
|
418
|
-
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
419
|
-
flows: { enabled: true },
|
|
420
|
-
})
|
|
421
|
-
|
|
422
|
-
// Then inject:
|
|
423
|
-
import { FlowService } from '@bymax-one/nest-queue'
|
|
424
|
-
import type { FlowJob, JobNode } from '@bymax-one/nest-queue'
|
|
425
|
-
|
|
426
|
-
const tree: JobNode = await this.flows.add({
|
|
427
|
-
name: 'process-order',
|
|
428
|
-
queueName: 'orders',
|
|
429
|
-
data: { orderId: '123' },
|
|
430
|
-
children: [
|
|
431
|
-
{ name: 'charge-payment', queueName: 'payments', data: {} },
|
|
432
|
-
{ name: 'send-confirmation', queueName: 'email', data: {} },
|
|
433
|
-
],
|
|
434
304
|
})
|
|
435
305
|
```
|
|
436
306
|
|
|
437
|
-
###
|
|
307
|
+
### Connection budget
|
|
438
308
|
|
|
439
|
-
|
|
309
|
+
The connections held are linear in what you register. Counts were read from `INFO clients` against
|
|
310
|
+
Redis 8 with BullMQ 6 and `ioredis` 6; every connection the library opens is released when the application context closes.
|
|
440
311
|
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
import { MetricsService } from '@bymax-one/nest-queue'
|
|
448
|
-
import type { QueueMetrics } from '@bymax-one/nest-queue'
|
|
449
|
-
|
|
450
|
-
const metrics: QueueMetrics = await this.metricsService.get('email')
|
|
451
|
-
const all: readonly QueueMetrics[] = await this.metricsService.getAll()
|
|
452
|
-
```
|
|
312
|
+
| What you register | Connections held |
|
|
313
|
+
| ---------------------------------------------- | ----------------------------------------------------------------------- |
|
|
314
|
+
| The module, Mode B | 1, shared by every queue and every flow |
|
|
315
|
+
| The module, Mode A | 0 of its own; your client is yours and is never counted |
|
|
316
|
+
| Each worker (`@Processor` or `WorkerRegistry`) | 2: a blocking connection that waits for jobs, and a command connection |
|
|
317
|
+
| Each queue observed with `@OnQueueEvent` | 2: a blocking connection that reads the event stream, and a command one |
|
|
453
318
|
|
|
454
|
-
|
|
319
|
+
Queues and `FlowService` add none, so the cost comes from consumers: a blocking connection is parked
|
|
320
|
+
inside a blocking command until a job arrives and cannot be shared. **Example:** 20 workers and 5 observed
|
|
321
|
+
queues in Mode B hold 1 + (2 × 20) + (2 × 5) = **51** connections (50 in Mode A, plus whatever your own
|
|
322
|
+
client uses). Each instance opens its own set, so three instances hold 153.
|
|
455
323
|
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
| `MetricsService.invalidate(queue?)` | Drop the cached snapshot of one queue, or of all of them |
|
|
461
|
-
| `QueueEventsRegistry` | The shared `QueueEvents` per queue (`getOrCreate`, `list`), used by `@OnQueueEvent` |
|
|
462
|
-
| `BYMAX_QUEUE_OPTIONS`, `BYMAX_QUEUE_RESOLVED_OPTIONS` | Injection tokens for the options as given and with the defaults applied |
|
|
463
|
-
| `BYMAX_QUEUE_REDIS_CLIENT`, `BYMAX_QUEUE_CONNECTION_MODE` | Injection tokens for the Queue-role ioredis client and for `'mode-a-byo'` or `'mode-b-owned'` |
|
|
464
|
-
| `DEFAULT_WORKER_CONCURRENCY`, `MAX_WORKER_CONCURRENCY` | The worker concurrency used when none is given, and its ceiling |
|
|
465
|
-
| `DEFAULT_JOB_OPTIONS`, `DEFAULT_METRICS_CACHE_TTL_MS`, `DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS` | The defaults of the Configuration table |
|
|
466
|
-
| `QUEUE_ERROR_MESSAGES` | The default message of each error code |
|
|
467
|
-
| `BymaxQueueModuleOptions`, `BymaxQueueModuleAsyncOptions`, `BymaxQueueOptionsFactory`, `QueueConnectionConfig`, `QueueConnectionMode` | The option and connection types behind `forRoot` and `forRootAsync` |
|
|
468
|
-
| `ProcessorMetadata`, `ProcessHandlerMetadata`, `QueueEventListenerMetadata` | The reflection metadata the decorators write |
|
|
469
|
-
| `JobStatus`, `JOB_STATUS`, `BulkJob`, `SandboxedJob`, `WorkerEventName`, `QueueEventName`, `WorkerOptions` | Constants and types for the APIs above |
|
|
470
|
-
|
|
471
|
-
---
|
|
472
|
-
|
|
473
|
-
## 🚨 Error Code Catalog
|
|
474
|
-
|
|
475
|
-
Whatever **this library rejects** is a `QueueException` carrying a stable,
|
|
476
|
-
transport-independent `code`. Branch on the code, never on the HTTP status or the
|
|
477
|
-
message text — the code is the contract.
|
|
478
|
-
|
|
479
|
-
Whatever **fails underneath it** is not wrapped at all. A dropped connection, a
|
|
480
|
-
timeout or any other operational fault reaches you as the original error, with its
|
|
481
|
-
own type and message, rather than being relabelled as invalid input. So the catalog
|
|
482
|
-
below is a complete list of the library's own rejections, and deliberately not a
|
|
483
|
-
complete list of everything a call can throw.
|
|
484
|
-
|
|
485
|
-
```typescript
|
|
486
|
-
import { QueueException, QUEUE_ERROR_CODES } from '@bymax-one/nest-queue'
|
|
487
|
-
|
|
488
|
-
try {
|
|
489
|
-
await queue.enqueue('email', 'welcome', { userId })
|
|
490
|
-
} catch (err) {
|
|
491
|
-
if (err instanceof QueueException) {
|
|
492
|
-
const { code } = (err.getResponse() as { error: { code: string } }).error
|
|
493
|
-
if (code === QUEUE_ERROR_CODES.BULK_ENQUEUE_FAILED) {
|
|
494
|
-
/* … */
|
|
495
|
-
}
|
|
496
|
-
}
|
|
497
|
-
}
|
|
498
|
-
```
|
|
499
|
-
|
|
500
|
-
| Code | HTTP | When thrown |
|
|
501
|
-
| ---------------------------------------- | :--: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
502
|
-
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
|
|
503
|
-
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, or Mode A's injected client is in an unusable `status` |
|
|
504
|
-
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
|
|
505
|
-
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` |
|
|
506
|
-
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
|
|
507
|
-
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
|
|
508
|
-
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
|
|
509
|
-
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
|
|
510
|
-
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
|
|
511
|
-
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
|
|
512
|
-
|
|
513
|
-
### Reserved codes
|
|
514
|
-
|
|
515
|
-
These are part of the published catalog and stable, but nothing currently throws them.
|
|
516
|
-
They are listed so a consumer writing an exhaustive `switch` over `QueueErrorCode`
|
|
517
|
-
compiles today and keeps compiling when a future release starts emitting them.
|
|
518
|
-
|
|
519
|
-
| Code | Why it is not thrown today |
|
|
520
|
-
| --------------------------------- | --------------------------------------------------------------------- |
|
|
521
|
-
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
|
|
522
|
-
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
|
|
523
|
-
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
|
|
524
|
-
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline force-closes workers rather than raising |
|
|
525
|
-
|
|
526
|
-
The same catalog is exported from `@bymax-one/nest-queue/shared` with no dependencies,
|
|
527
|
-
so a frontend can map codes to messages without importing the server surface.
|
|
324
|
+
Hosted Redis plans cap concurrent connections: multiply by the instances you run (rolling deploys
|
|
325
|
+
overlap old and new) and compare with your plan's limit; self-managed Redis allows 10 000 (`maxclients`).
|
|
326
|
+
In `CLIENT LIST`, a worker's blocking connection is named `<prefix>:<base64 of the queue name>…`; the
|
|
327
|
+
command connection is unnamed.
|
|
528
328
|
|
|
529
329
|
---
|
|
530
330
|
|
|
@@ -615,34 +415,20 @@ A pattern the parser rejects is `INVALID_REPEAT_OPTIONS` (400) before Redis is a
|
|
|
615
415
|
Pass the `deduplication` option directly on `enqueue` — the library delegates the behavior entirely to BullMQ:
|
|
616
416
|
|
|
617
417
|
```typescript
|
|
418
|
+
const id = `welcome:${userId}`
|
|
419
|
+
|
|
618
420
|
// Simple: collapse duplicates until the first in-flight job completes
|
|
619
|
-
await this.queue.enqueue(
|
|
620
|
-
'email',
|
|
621
|
-
'welcome',
|
|
622
|
-
{ userId },
|
|
623
|
-
{
|
|
624
|
-
deduplication: { id: `welcome:${userId}` },
|
|
625
|
-
},
|
|
626
|
-
)
|
|
421
|
+
await this.queue.enqueue('email', 'welcome', { userId }, { deduplication: { id } })
|
|
627
422
|
|
|
628
423
|
// Throttle: ignore duplicates for 30 seconds
|
|
629
|
-
await this.queue.enqueue(
|
|
630
|
-
'email',
|
|
631
|
-
'welcome',
|
|
632
|
-
{ userId },
|
|
633
|
-
{
|
|
634
|
-
deduplication: { id: `welcome:${userId}`, ttl: 30_000 },
|
|
635
|
-
},
|
|
636
|
-
)
|
|
424
|
+
await this.queue.enqueue('email', 'welcome', { userId }, { deduplication: { id, ttl: 30_000 } })
|
|
637
425
|
|
|
638
426
|
// Debounce: keep only the latest; each duplicate resets the TTL
|
|
639
427
|
await this.queue.enqueue(
|
|
640
428
|
'email',
|
|
641
429
|
'welcome',
|
|
642
430
|
{ userId },
|
|
643
|
-
{
|
|
644
|
-
deduplication: { id: `welcome:${userId}`, ttl: 5_000, extend: true, replace: true },
|
|
645
|
-
},
|
|
431
|
+
{ deduplication: { id, ttl: 5_000, extend: true, replace: true } },
|
|
646
432
|
)
|
|
647
433
|
```
|
|
648
434
|
|
|
@@ -762,14 +548,13 @@ wire connection and defaults yourself.
|
|
|
762
548
|
|
|
763
549
|
### Design Principles
|
|
764
550
|
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
|
|
|
769
|
-
|
|
|
770
|
-
|
|
|
771
|
-
|
|
|
772
|
-
| 🧊 **Zero runtime dependencies** | `dependencies` is `{}`. You choose the exact `bullmq` and `ioredis` versions, and the supply-chain surface stays yours. |
|
|
551
|
+
The role-shaped connections, the shared `prefix` and the Job Scheduler API are described under
|
|
552
|
+
[Features](#-features) and [Connections](#-connections). Two principles are not:
|
|
553
|
+
|
|
554
|
+
| Principle | Description |
|
|
555
|
+
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
556
|
+
| 💥 **Fail at boot, not at runtime** | Options are validated before any connection is opened, and every rejection carries a distinct reason. |
|
|
557
|
+
| 🧬 **Explicit DI tokens** | Tokens are `Symbol()` and every injectable constructor parameter carries an explicit `@Inject`. The package ships bundled, where design-time metadata is not emitted — implicit resolution would break inside your build, not ours. |
|
|
773
558
|
|
|
774
559
|
---
|
|
775
560
|
|
|
@@ -815,11 +600,9 @@ compromised client belongs in Redis ACLs or separate instances.
|
|
|
815
600
|
|
|
816
601
|
### Shutdown is bounded and at-least-once
|
|
817
602
|
|
|
818
|
-
|
|
819
|
-
|
|
820
|
-
|
|
821
|
-
be idempotent.** A handler that charges a card or sends an email without an idempotency
|
|
822
|
-
key will do it twice at some point.
|
|
603
|
+
A job still running when the drain deadline expires is **not** acknowledged, so BullMQ's stalled-job
|
|
604
|
+
check re-queues it and it runs again. **Every handler must be idempotent** — see
|
|
605
|
+
[Graceful Shutdown](#-graceful-shutdown).
|
|
823
606
|
|
|
824
607
|
### Security Checklist
|
|
825
608
|
|
|
@@ -848,25 +631,23 @@ key will do it twice at some point.
|
|
|
848
631
|
| Namespacing | `prefix` applied uniformly to `Queue`, `Worker`, `QueueEvents` and `FlowProducer` |
|
|
849
632
|
| Connection policy | `maxRetriesPerRequest: null` confined to duplicated consumer connections; producers keep fail-fast retries |
|
|
850
633
|
| Delivery | At-least-once with a bounded drain; unacknowledged work is re-queued by BullMQ's stalled check |
|
|
851
|
-
| Supply chain |
|
|
634
|
+
| Supply chain | SHA-pinned third-party Actions, OSV-Scanner, TruffleHog, OpenSSF Scorecard; npm publish over OIDC |
|
|
852
635
|
| Input validation | Options validated at bootstrap; cron patterns validated by BullMQ's own parser, never a hand-rolled regex |
|
|
853
636
|
|
|
854
|
-
> [!IMPORTANT]
|
|
855
|
-
> **Delivery is at-least-once, and `prefix` is not an access boundary.** Handlers must be
|
|
856
|
-
> idempotent, and anything with credentials for the Redis instance can read every
|
|
857
|
-
> namespace on it.
|
|
858
|
-
|
|
859
637
|
---
|
|
860
638
|
|
|
861
639
|
## 🧱 Tech Stack
|
|
862
640
|
|
|
863
|
-
|
|
864
|
-
-
|
|
865
|
-
-
|
|
866
|
-
-
|
|
867
|
-
-
|
|
868
|
-
|
|
869
|
-
-
|
|
641
|
+
[](https://nodejs.org)
|
|
642
|
+
[](https://nestjs.com)
|
|
643
|
+
[](https://bullmq.io)
|
|
644
|
+
[](https://github.com/redis/ioredis)
|
|
645
|
+
[](https://www.typescriptlang.org)
|
|
646
|
+
[](https://jestjs.io)
|
|
647
|
+
[](https://testcontainers.com)
|
|
648
|
+
[](https://stryker-mutator.io)
|
|
649
|
+
[](https://pnpm.io)
|
|
650
|
+
[](https://tsup.egoist.dev)
|
|
870
651
|
|
|
871
652
|
---
|
|
872
653
|
|
|
@@ -881,9 +662,7 @@ key will do it twice at some point.
|
|
|
881
662
|
- **Published-shape gates**: `pnpm check:exports` resolves every entrypoint the way each
|
|
882
663
|
module-resolution mode would (all four `attw` quadrants green, `node10` included), and
|
|
883
664
|
`pnpm smoke` resolves the packed tarball from a real consumer in ESM _and_ CommonJS.
|
|
884
|
-
- **Bundle budgets** enforced
|
|
885
|
-
- **Supply chain:** SHA-pinned third-party GitHub Actions, OSV-Scanner, TruffleHog, OpenSSF Scorecard,
|
|
886
|
-
npm publish with provenance (OIDC).
|
|
665
|
+
- **Bundle budgets** enforced by `pnpm size`.
|
|
887
666
|
|
|
888
667
|
```bash
|
|
889
668
|
pnpm typecheck && pnpm test:types && pnpm lint && pnpm test:cov:all && \
|
|
@@ -892,40 +671,196 @@ pnpm typecheck && pnpm test:types && pnpm lint && pnpm test:cov:all && \
|
|
|
892
671
|
|
|
893
672
|
---
|
|
894
673
|
|
|
895
|
-
##
|
|
674
|
+
## 📖 API Reference
|
|
896
675
|
|
|
897
|
-
### `
|
|
676
|
+
### `QueueService`
|
|
898
677
|
|
|
899
|
-
|
|
678
|
+
The central interaction point with BullMQ queues.
|
|
900
679
|
|
|
680
|
+
```typescript
|
|
681
|
+
import { QueueService } from '@bymax-one/nest-queue'
|
|
682
|
+
import type {
|
|
683
|
+
Job,
|
|
684
|
+
JobsOptions,
|
|
685
|
+
JobSchedulerRepeatOptions,
|
|
686
|
+
QueueMetrics,
|
|
687
|
+
} from '@bymax-one/nest-queue'
|
|
901
688
|
```
|
|
902
|
-
|
|
903
|
-
|
|
689
|
+
|
|
690
|
+
| Method | Signature | Description |
|
|
691
|
+
| -------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
|
692
|
+
| `enqueue` | `<TData, TResult>(queue, name, data, opts?) → Promise<Job<TData, TResult>>` | Add a single job |
|
|
693
|
+
| `enqueueBulk` | `<TData, TResult>(queue, jobs) → Promise<Job<TData, TResult>[]>` | Add up to 1 000 jobs in one Redis roundtrip |
|
|
694
|
+
| `getJob` | `<TData, TResult>(queue, jobId) → Promise<Job<TData, TResult> \| null>` | Get job by ID; `null` when it does not exist |
|
|
695
|
+
| `getJobs` | `<TData, TResult>(queue, status, start?, end?) → Promise<Job<TData, TResult>[]>` | Page jobs in one status (default `0`–`50`) |
|
|
696
|
+
| `getMetrics` | `(queue) → Promise<QueueMetrics>` | Uncached snapshot of job counts per status, with a collection timestamp |
|
|
697
|
+
| `pauseQueue` | `(queue) → Promise<void>` | Pause a queue |
|
|
698
|
+
| `resumeQueue` | `(queue) → Promise<void>` | Resume a paused queue |
|
|
699
|
+
| `cleanQueue` | `(queue, gracePeriodMs, limit, status?) → Promise<string[]>` | Remove up to `limit` jobs older than `gracePeriodMs` (`0` = no limit); returns removed ids |
|
|
700
|
+
| `upsertJobScheduler` | `<TData, TResult>(queue, id, repeat, template?) → Promise<Job<TData, TResult> \| undefined>` | Create or update a recurring Job Scheduler; returns the first scheduled job |
|
|
701
|
+
| `removeJobScheduler` | `(queue, id) → Promise<boolean>` | Remove a Job Scheduler by ID |
|
|
702
|
+
| `getJobSchedulers` | `(queue, start?, end?, asc?) → Promise<JobSchedulerJson[]>` | Page the schedulers registered on a queue |
|
|
703
|
+
| `getOrCreateQueue` | `<TData, TResult>(queue, overrides?) → Queue<TData, TResult>` | Return the cached raw `Queue` instance |
|
|
704
|
+
| `getCachedQueues` | `() → ReadonlyMap<string, Queue>` | Read-only view of every queue created so far |
|
|
705
|
+
|
|
706
|
+
### `WorkerRegistry`
|
|
707
|
+
|
|
708
|
+
Programmatic API to create and manage BullMQ workers at runtime.
|
|
709
|
+
|
|
710
|
+
```typescript
|
|
711
|
+
import path from 'node:path'
|
|
712
|
+
import { WorkerRegistry } from '@bymax-one/nest-queue'
|
|
713
|
+
import type { ProgrammaticWorkerConfig, SandboxedWorkerConfig } from '@bymax-one/nest-queue'
|
|
714
|
+
|
|
715
|
+
// Register an in-process worker
|
|
716
|
+
registry.register({
|
|
717
|
+
queueName: 'email',
|
|
718
|
+
handler: async (job) => {
|
|
719
|
+
/* … */
|
|
720
|
+
},
|
|
721
|
+
options: { concurrency: 4 },
|
|
722
|
+
})
|
|
723
|
+
|
|
724
|
+
// Register a sandboxed (out-of-process) processor
|
|
725
|
+
registry.registerSandboxed({
|
|
726
|
+
queueName: 'heavy-compute',
|
|
727
|
+
processorFile: path.resolve(__dirname, 'compute.processor.js'),
|
|
728
|
+
options: { concurrency: 2 },
|
|
729
|
+
})
|
|
904
730
|
```
|
|
905
731
|
|
|
906
|
-
|
|
907
|
-
|
|
908
|
-
|
|
909
|
-
|
|
910
|
-
|
|
732
|
+
### `FlowService` (opt-in)
|
|
733
|
+
|
|
734
|
+
Enable flows in the module options and inject the service:
|
|
735
|
+
|
|
736
|
+
```typescript
|
|
737
|
+
BymaxQueueModule.forRoot({
|
|
738
|
+
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
739
|
+
flows: { enabled: true },
|
|
740
|
+
})
|
|
911
741
|
|
|
912
|
-
|
|
742
|
+
// Then inject:
|
|
743
|
+
import { FlowService } from '@bymax-one/nest-queue'
|
|
744
|
+
import type { FlowJob, JobNode } from '@bymax-one/nest-queue'
|
|
913
745
|
|
|
914
|
-
|
|
915
|
-
|
|
746
|
+
const tree: JobNode = await this.flows.add({
|
|
747
|
+
name: 'process-order',
|
|
748
|
+
queueName: 'orders',
|
|
749
|
+
data: { orderId: '123' },
|
|
750
|
+
children: [
|
|
751
|
+
{ name: 'charge-payment', queueName: 'payments', data: {} },
|
|
752
|
+
{ name: 'send-confirmation', queueName: 'email', data: {} },
|
|
753
|
+
],
|
|
754
|
+
})
|
|
755
|
+
```
|
|
756
|
+
|
|
757
|
+
### `MetricsService` (opt-in)
|
|
758
|
+
|
|
759
|
+
Provide a per-queue snapshot with a configurable TTL cache:
|
|
760
|
+
|
|
761
|
+
```typescript
|
|
762
|
+
BymaxQueueModule.forRoot({
|
|
763
|
+
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
764
|
+
metrics: { enabled: true, cacheTtlMs: 5_000 },
|
|
765
|
+
})
|
|
766
|
+
|
|
767
|
+
import { MetricsService } from '@bymax-one/nest-queue'
|
|
768
|
+
import type { QueueMetrics } from '@bymax-one/nest-queue'
|
|
769
|
+
|
|
770
|
+
const metrics: QueueMetrics = await this.metricsService.get('email')
|
|
771
|
+
const all: readonly QueueMetrics[] = await this.metricsService.getAll()
|
|
916
772
|
```
|
|
917
773
|
|
|
918
|
-
|
|
774
|
+
### Other exports
|
|
775
|
+
|
|
776
|
+
| Export | What it is |
|
|
777
|
+
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
778
|
+
| `WorkerRegistry.unregister(queue)`, `list()` | Stop and remove a worker and its connection; list the queue names that have a worker |
|
|
779
|
+
| `FlowService.addBulk(flows)`, `getProducer()`, `close()` | Add several flows at once; the underlying BullMQ `FlowProducer`; close it (a no-op while flows are disabled) |
|
|
780
|
+
| `MetricsService.invalidate(queue?)` | Drop the cached snapshot of one queue, or of all of them |
|
|
781
|
+
| `QueueEventsRegistry` | The shared `QueueEvents` per queue (`getOrCreate`, `list`), used by `@OnQueueEvent` |
|
|
782
|
+
| `BYMAX_QUEUE_OPTIONS`, `BYMAX_QUEUE_RESOLVED_OPTIONS` | Injection tokens for the options as given and with the defaults applied |
|
|
783
|
+
| `BYMAX_QUEUE_REDIS_CLIENT`, `BYMAX_QUEUE_CONNECTION_MODE` | Injection tokens for the Queue-role ioredis client and for `'mode-a-byo'` or `'mode-b-owned'` |
|
|
784
|
+
| `DEFAULT_WORKER_CONCURRENCY`, `MAX_WORKER_CONCURRENCY` | The worker concurrency used when none is given, and its ceiling |
|
|
785
|
+
| `DEFAULT_JOB_OPTIONS`, `DEFAULT_METRICS_CACHE_TTL_MS`, `DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS` | The defaults of the Configuration table |
|
|
786
|
+
| `QUEUE_ERROR_MESSAGES` | The default message of each error code |
|
|
787
|
+
| `BymaxQueueModuleOptions`, `BymaxQueueModuleAsyncOptions`, `BymaxQueueOptionsFactory`, `QueueConnectionConfig`, `QueueConnectionMode` | The option and connection types behind `forRoot` and `forRootAsync` |
|
|
788
|
+
| `ProcessorMetadata`, `ProcessHandlerMetadata`, `QueueEventListenerMetadata` | The reflection metadata the decorators write |
|
|
789
|
+
| `JobStatus`, `JOB_STATUS`, `BulkJob`, `SandboxedJob`, `WorkerEventName`, `QueueEventName`, `WorkerOptions` | Constants and types for the APIs above |
|
|
790
|
+
|
|
791
|
+
---
|
|
792
|
+
|
|
793
|
+
## 🚨 Error Code Catalog
|
|
794
|
+
|
|
795
|
+
Whatever **this library rejects** is a `QueueException` carrying a stable,
|
|
796
|
+
transport-independent `code`. Branch on the code, never on the HTTP status or the
|
|
797
|
+
message text — the code is the contract.
|
|
798
|
+
|
|
799
|
+
Whatever **fails underneath it** is not wrapped at all. A dropped connection, a
|
|
800
|
+
timeout or any other operational fault reaches you as the original error, with its
|
|
801
|
+
own type and message, rather than being relabelled as invalid input. So the catalog
|
|
802
|
+
below is a complete list of the library's own rejections, and deliberately not a
|
|
803
|
+
complete list of everything a call can throw.
|
|
804
|
+
|
|
805
|
+
```typescript
|
|
806
|
+
import { QueueException, QUEUE_ERROR_CODES } from '@bymax-one/nest-queue'
|
|
807
|
+
|
|
808
|
+
try {
|
|
809
|
+
await queue.enqueue('email', 'welcome', { userId })
|
|
810
|
+
} catch (err) {
|
|
811
|
+
if (err instanceof QueueException) {
|
|
812
|
+
const { code } = (err.getResponse() as { error: { code: string } }).error
|
|
813
|
+
if (code === QUEUE_ERROR_CODES.BULK_ENQUEUE_FAILED) {
|
|
814
|
+
/* … */
|
|
815
|
+
}
|
|
816
|
+
}
|
|
817
|
+
}
|
|
818
|
+
```
|
|
819
|
+
|
|
820
|
+
| Code | HTTP | When thrown |
|
|
821
|
+
| ---------------------------------------- | :--: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
822
|
+
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
|
|
823
|
+
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, or Mode A's injected client is in an unusable `status` |
|
|
824
|
+
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
|
|
825
|
+
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` |
|
|
826
|
+
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
|
|
827
|
+
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
|
|
828
|
+
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
|
|
829
|
+
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
|
|
830
|
+
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
|
|
831
|
+
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
|
|
832
|
+
|
|
833
|
+
### Reserved codes
|
|
834
|
+
|
|
835
|
+
These are part of the published catalog and stable, but nothing currently throws them.
|
|
836
|
+
They are listed so a consumer writing an exhaustive `switch` over `QueueErrorCode`
|
|
837
|
+
compiles today and keeps compiling when a future release starts emitting them.
|
|
838
|
+
|
|
839
|
+
| Code | Why it is not thrown today |
|
|
840
|
+
| --------------------------------- | --------------------------------------------------------------------- |
|
|
841
|
+
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
|
|
842
|
+
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
|
|
843
|
+
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
|
|
844
|
+
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline force-closes workers rather than raising |
|
|
845
|
+
|
|
846
|
+
The same catalog is exported from `@bymax-one/nest-queue/shared` with no dependencies,
|
|
847
|
+
so a frontend can map codes to messages without importing the server surface.
|
|
848
|
+
|
|
849
|
+
---
|
|
850
|
+
|
|
851
|
+
## 🩺 Troubleshooting
|
|
852
|
+
|
|
853
|
+
### `ERR_PNPM_IGNORED_BUILDS: msgpackr-extract` on install (pnpm 11)
|
|
854
|
+
|
|
855
|
+
pnpm 11 blocks the build script of `msgpackr-extract`, a native optional dependency of BullMQ, and fails the
|
|
856
|
+
install. Approve it once with `pnpm approve-builds`, or in `pnpm-workspace.yaml`:
|
|
919
857
|
|
|
920
858
|
```yaml
|
|
921
859
|
onlyBuiltDependencies:
|
|
922
860
|
- msgpackr-extract
|
|
923
861
|
```
|
|
924
862
|
|
|
925
|
-
Skipping the build is also fine
|
|
926
|
-
implementation, so BullMQ still works, just without the native accelerator.
|
|
927
|
-
|
|
928
|
-
---
|
|
863
|
+
Skipping the build is also fine: `msgpackr` falls back to pure JavaScript, only without the native accelerator.
|
|
929
864
|
|
|
930
865
|
### `CONNECTION_REQUIRES_NULL_RETRIES`
|
|
931
866
|
|
|
@@ -960,23 +895,18 @@ do not pick them up:
|
|
|
960
895
|
- **NestJS only** — does not work with plain Express, standalone Fastify, or other frameworks.
|
|
961
896
|
- **Node.js only** — Deno and Bun are not supported.
|
|
962
897
|
- **BullMQ `^6.0` only** — the library targets BullMQ 6 (with `bullmq-otel ^2` for telemetry)
|
|
963
|
-
and does not support BullMQ 5
|
|
964
|
-
See the BullMQ version policy in [`CHANGELOG.md`](CHANGELOG.md).
|
|
898
|
+
and does not support BullMQ 5. See the BullMQ version policy in [`CHANGELOG.md`](CHANGELOG.md).
|
|
965
899
|
- **No Redis Cluster sharding** — BullMQ's cluster support is not exercised by this library.
|
|
966
900
|
- **No built-in DLQ helper** — use the `@OnWorkerEvent('failed')` pattern above.
|
|
967
901
|
- **No Prometheus metrics export** — `getMetrics()` returns snapshots; publish them
|
|
968
902
|
via `prom-client` in your own health endpoint.
|
|
969
903
|
|
|
970
|
-
For the full list of limitations and their alternatives, see
|
|
971
|
-
[`docs/technical_specification.md § 16`](docs/technical_specification.md).
|
|
972
|
-
|
|
973
904
|
---
|
|
974
905
|
|
|
975
906
|
## 🤝 Contributing
|
|
976
907
|
|
|
977
908
|
Pull requests are welcome. Please open an issue first for significant changes.
|
|
978
909
|
|
|
979
|
-
- Read [`docs/technical_specification.md`](docs/technical_specification.md) for architecture decisions.
|
|
980
910
|
- Run the full gate listed in [`CONTRIBUTING.md`](CONTRIBUTING.md) before opening a PR.
|
|
981
911
|
- Conventional Commits are enforced by `commitlint.config.cjs`.
|
|
982
912
|
|
package/dist/server/index.cjs
CHANGED
|
@@ -192,15 +192,46 @@ function isClientUsable(client) {
|
|
|
192
192
|
return client.status === "ready" || client.status === "connecting";
|
|
193
193
|
}
|
|
194
194
|
|
|
195
|
+
// src/server/utils/describe-error.ts
|
|
196
|
+
function describeError(cause) {
|
|
197
|
+
try {
|
|
198
|
+
if (typeof cause === "object" && cause !== null && "message" in cause) {
|
|
199
|
+
if (typeof cause.message === "string") return cause.message;
|
|
200
|
+
}
|
|
201
|
+
return String(cause);
|
|
202
|
+
} catch {
|
|
203
|
+
return "unknown error";
|
|
204
|
+
}
|
|
205
|
+
}
|
|
206
|
+
|
|
207
|
+
// src/server/utils/attach-client-error-listener.ts
|
|
208
|
+
function describeClientError(err) {
|
|
209
|
+
const message = describeError(err);
|
|
210
|
+
if (message !== "") return message;
|
|
211
|
+
try {
|
|
212
|
+
const { name, code } = err;
|
|
213
|
+
const label = typeof name === "string" && name !== "" ? name : "unknown error";
|
|
214
|
+
return typeof code === "string" ? `${label} [${code}]` : label;
|
|
215
|
+
} catch {
|
|
216
|
+
return "unknown error";
|
|
217
|
+
}
|
|
218
|
+
}
|
|
219
|
+
function attachClientErrorListener(client, logger, role) {
|
|
220
|
+
client.on("error", (err) => {
|
|
221
|
+
logger.error(`Redis connection (${role}) emitted an error: ${describeClientError(err)}`);
|
|
222
|
+
});
|
|
223
|
+
}
|
|
224
|
+
|
|
195
225
|
// src/server/utils/duplicate-connection.ts
|
|
196
226
|
function duplicateConnection(client) {
|
|
197
227
|
return client.duplicate({ maxRetriesPerRequest: null });
|
|
198
228
|
}
|
|
199
229
|
|
|
200
230
|
// src/server/services/connection-resolver.service.ts
|
|
201
|
-
var _client, _options;
|
|
231
|
+
var _logger, _client, _options;
|
|
202
232
|
exports.ConnectionResolver = class ConnectionResolver {
|
|
203
233
|
constructor(options) {
|
|
234
|
+
__privateAdd(this, _logger, new common.Logger(exports.ConnectionResolver.name));
|
|
204
235
|
__privateAdd(this, _client);
|
|
205
236
|
__privateAdd(this, _options);
|
|
206
237
|
__publicField(this, "mode");
|
|
@@ -214,6 +245,7 @@ exports.ConnectionResolver = class ConnectionResolver {
|
|
|
214
245
|
}
|
|
215
246
|
this.mode = "mode-b-owned";
|
|
216
247
|
__privateSet(this, _client, "url" in cfg ? new ioredis.Redis(cfg.url, this.ownedOptions(cfg.options)) : new ioredis.Redis(this.ownedOptions(cfg.options)));
|
|
248
|
+
attachClientErrorListener(__privateGet(this, _client), __privateGet(this, _logger), "owned");
|
|
217
249
|
await this.waitReady(
|
|
218
250
|
__privateGet(this, _options).connectionReadyTimeoutMs ?? DEFAULT_CONNECTION_READY_TIMEOUT_MS
|
|
219
251
|
);
|
|
@@ -267,6 +299,7 @@ exports.ConnectionResolver = class ConnectionResolver {
|
|
|
267
299
|
);
|
|
268
300
|
}
|
|
269
301
|
const probe = duplicateConnection(client);
|
|
302
|
+
attachClientErrorListener(probe, __privateGet(this, _logger), "probe");
|
|
270
303
|
try {
|
|
271
304
|
assertBlockingConnection(probe);
|
|
272
305
|
} finally {
|
|
@@ -305,6 +338,7 @@ exports.ConnectionResolver = class ConnectionResolver {
|
|
|
305
338
|
});
|
|
306
339
|
}
|
|
307
340
|
};
|
|
341
|
+
_logger = new WeakMap();
|
|
308
342
|
_client = new WeakMap();
|
|
309
343
|
_options = new WeakMap();
|
|
310
344
|
exports.ConnectionResolver = __decorateClass([
|
|
@@ -343,18 +377,6 @@ function validateJobSchedulerOptions(repeat) {
|
|
|
343
377
|
}
|
|
344
378
|
}
|
|
345
379
|
|
|
346
|
-
// src/server/utils/describe-error.ts
|
|
347
|
-
function describeError(cause) {
|
|
348
|
-
try {
|
|
349
|
-
if (typeof cause === "object" && cause !== null && "message" in cause) {
|
|
350
|
-
if (typeof cause.message === "string") return cause.message;
|
|
351
|
-
}
|
|
352
|
-
return String(cause);
|
|
353
|
-
} catch {
|
|
354
|
-
return "unknown error";
|
|
355
|
-
}
|
|
356
|
-
}
|
|
357
|
-
|
|
358
380
|
// src/server/utils/attach-error-listener.ts
|
|
359
381
|
function attachDefaultErrorListener(emitter, logger, kind, queueName) {
|
|
360
382
|
emitter.on("error", (err) => {
|
|
@@ -797,6 +819,7 @@ exports.QueueEventsRegistry = class QueueEventsRegistry {
|
|
|
797
819
|
const existing = __privateGet(this, _events).get(queueName);
|
|
798
820
|
if (existing) return existing;
|
|
799
821
|
const conn = duplicateConnection(this.connection.getClient());
|
|
822
|
+
attachClientErrorListener(conn, this.logger, `queue events: ${queueName}`);
|
|
800
823
|
let qe;
|
|
801
824
|
try {
|
|
802
825
|
qe = new bullmq.QueueEvents(queueName, { connection: conn, prefix: this.options.prefix });
|
package/dist/server/index.mjs
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { Injectable, Inject, ConfigurableModuleBuilder, Module,
|
|
1
|
+
import { Injectable, Inject, ConfigurableModuleBuilder, Module, Logger, HttpStatus, HttpException } from '@nestjs/common';
|
|
2
2
|
import { DiscoveryService, DiscoveryModule } from '@nestjs/core';
|
|
3
3
|
import { Redis } from 'ioredis';
|
|
4
4
|
import { Worker, QueueEvents, Queue, FlowProducer, defaultRepeatStrategy } from 'bullmq';
|
|
@@ -190,15 +190,46 @@ function isClientUsable(client) {
|
|
|
190
190
|
return client.status === "ready" || client.status === "connecting";
|
|
191
191
|
}
|
|
192
192
|
|
|
193
|
+
// src/server/utils/describe-error.ts
|
|
194
|
+
function describeError(cause) {
|
|
195
|
+
try {
|
|
196
|
+
if (typeof cause === "object" && cause !== null && "message" in cause) {
|
|
197
|
+
if (typeof cause.message === "string") return cause.message;
|
|
198
|
+
}
|
|
199
|
+
return String(cause);
|
|
200
|
+
} catch {
|
|
201
|
+
return "unknown error";
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
// src/server/utils/attach-client-error-listener.ts
|
|
206
|
+
function describeClientError(err) {
|
|
207
|
+
const message = describeError(err);
|
|
208
|
+
if (message !== "") return message;
|
|
209
|
+
try {
|
|
210
|
+
const { name, code } = err;
|
|
211
|
+
const label = typeof name === "string" && name !== "" ? name : "unknown error";
|
|
212
|
+
return typeof code === "string" ? `${label} [${code}]` : label;
|
|
213
|
+
} catch {
|
|
214
|
+
return "unknown error";
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
function attachClientErrorListener(client, logger, role) {
|
|
218
|
+
client.on("error", (err) => {
|
|
219
|
+
logger.error(`Redis connection (${role}) emitted an error: ${describeClientError(err)}`);
|
|
220
|
+
});
|
|
221
|
+
}
|
|
222
|
+
|
|
193
223
|
// src/server/utils/duplicate-connection.ts
|
|
194
224
|
function duplicateConnection(client) {
|
|
195
225
|
return client.duplicate({ maxRetriesPerRequest: null });
|
|
196
226
|
}
|
|
197
227
|
|
|
198
228
|
// src/server/services/connection-resolver.service.ts
|
|
199
|
-
var _client, _options;
|
|
229
|
+
var _logger, _client, _options;
|
|
200
230
|
var ConnectionResolver = class {
|
|
201
231
|
constructor(options) {
|
|
232
|
+
__privateAdd(this, _logger, new Logger(ConnectionResolver.name));
|
|
202
233
|
__privateAdd(this, _client);
|
|
203
234
|
__privateAdd(this, _options);
|
|
204
235
|
__publicField(this, "mode");
|
|
@@ -212,6 +243,7 @@ var ConnectionResolver = class {
|
|
|
212
243
|
}
|
|
213
244
|
this.mode = "mode-b-owned";
|
|
214
245
|
__privateSet(this, _client, "url" in cfg ? new Redis(cfg.url, this.ownedOptions(cfg.options)) : new Redis(this.ownedOptions(cfg.options)));
|
|
246
|
+
attachClientErrorListener(__privateGet(this, _client), __privateGet(this, _logger), "owned");
|
|
215
247
|
await this.waitReady(
|
|
216
248
|
__privateGet(this, _options).connectionReadyTimeoutMs ?? DEFAULT_CONNECTION_READY_TIMEOUT_MS
|
|
217
249
|
);
|
|
@@ -265,6 +297,7 @@ var ConnectionResolver = class {
|
|
|
265
297
|
);
|
|
266
298
|
}
|
|
267
299
|
const probe = duplicateConnection(client);
|
|
300
|
+
attachClientErrorListener(probe, __privateGet(this, _logger), "probe");
|
|
268
301
|
try {
|
|
269
302
|
assertBlockingConnection(probe);
|
|
270
303
|
} finally {
|
|
@@ -303,6 +336,7 @@ var ConnectionResolver = class {
|
|
|
303
336
|
});
|
|
304
337
|
}
|
|
305
338
|
};
|
|
339
|
+
_logger = new WeakMap();
|
|
306
340
|
_client = new WeakMap();
|
|
307
341
|
_options = new WeakMap();
|
|
308
342
|
ConnectionResolver = __decorateClass([
|
|
@@ -341,18 +375,6 @@ function validateJobSchedulerOptions(repeat) {
|
|
|
341
375
|
}
|
|
342
376
|
}
|
|
343
377
|
|
|
344
|
-
// src/server/utils/describe-error.ts
|
|
345
|
-
function describeError(cause) {
|
|
346
|
-
try {
|
|
347
|
-
if (typeof cause === "object" && cause !== null && "message" in cause) {
|
|
348
|
-
if (typeof cause.message === "string") return cause.message;
|
|
349
|
-
}
|
|
350
|
-
return String(cause);
|
|
351
|
-
} catch {
|
|
352
|
-
return "unknown error";
|
|
353
|
-
}
|
|
354
|
-
}
|
|
355
|
-
|
|
356
378
|
// src/server/utils/attach-error-listener.ts
|
|
357
379
|
function attachDefaultErrorListener(emitter, logger, kind, queueName) {
|
|
358
380
|
emitter.on("error", (err) => {
|
|
@@ -795,6 +817,7 @@ var QueueEventsRegistry = class {
|
|
|
795
817
|
const existing = __privateGet(this, _events).get(queueName);
|
|
796
818
|
if (existing) return existing;
|
|
797
819
|
const conn = duplicateConnection(this.connection.getClient());
|
|
820
|
+
attachClientErrorListener(conn, this.logger, `queue events: ${queueName}`);
|
|
798
821
|
let qe;
|
|
799
822
|
try {
|
|
800
823
|
qe = new QueueEvents(queueName, { connection: conn, prefix: this.options.prefix });
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@bymax-one/nest-queue",
|
|
3
|
-
"version": "1.2.
|
|
3
|
+
"version": "1.2.13",
|
|
4
4
|
"description": "NestJS dynamic module wrapping BullMQ — typed jobs, flows, job schedulers, deduplication, OpenTelemetry, graceful shutdown",
|
|
5
5
|
"author": "Bymax One <support@bymax.one>",
|
|
6
6
|
"license": "MIT",
|
|
@@ -58,10 +58,12 @@
|
|
|
58
58
|
"check:api": "node scripts/check-api-surface.mjs",
|
|
59
59
|
"check:exports": "attw --pack .",
|
|
60
60
|
"check:mutants": "node scripts/check-mutation-directives.mjs",
|
|
61
|
+
"check:file-length": "node scripts/check-file-length.mjs",
|
|
61
62
|
"check:published": "node scripts/check-published-surface.mjs",
|
|
62
63
|
"clean": "rm -rf dist coverage",
|
|
63
64
|
"lint": "eslint src scripts test",
|
|
64
65
|
"lint:fix": "eslint src scripts test --fix",
|
|
66
|
+
"mutation:const-tables": "node scripts/mutate-const-tables.mjs -- pnpm exec jest --config jest.mutants.config.ts --findRelatedTests {file} --coverage=false --silent",
|
|
65
67
|
"mutation": "stryker run",
|
|
66
68
|
"mutation:full": "node -e \"require('node:fs').rmSync('reports/stryker-incremental.json',{force:true,recursive:true})\" && stryker run",
|
|
67
69
|
"mutation:dry-run": "stryker run --dryRunOnly",
|
|
@@ -95,6 +97,7 @@
|
|
|
95
97
|
},
|
|
96
98
|
"devDependencies": {
|
|
97
99
|
"@arethetypeswrong/cli": "^0.18.5",
|
|
100
|
+
"@bymax-one/nest-cache": "1.2.8",
|
|
98
101
|
"@commitlint/cli": "^21.2.1",
|
|
99
102
|
"@commitlint/config-conventional": "^21.2.0",
|
|
100
103
|
"@eslint/js": "^10.0.1",
|