@bymax-one/nest-queue 1.2.12 → 1.2.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +121 -1
- package/README.md +319 -345
- package/dist/server/index.cjs +273 -95
- package/dist/server/index.d.cts +87 -24
- package/dist/server/index.d.ts +87 -24
- package/dist/server/index.mjs +273 -95
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
<p align="center">
|
|
8
8
|
<strong>Typed BullMQ queues for NestJS</strong><br />
|
|
9
|
-
<sub>BullMQ 6 · Workers · Flows · Job Schedulers · Deduplication · OpenTelemetry · Graceful Shutdown
|
|
9
|
+
<sub>BullMQ 6 · Workers · Flows · Job Schedulers · Deduplication · OpenTelemetry · Graceful Shutdown</sub>
|
|
10
10
|
</p>
|
|
11
11
|
|
|
12
12
|
<p align="center">
|
|
@@ -42,24 +42,13 @@ OpenTelemetry passthrough, and a tested bounded graceful-shutdown protocol.
|
|
|
42
42
|
The library has **zero direct dependencies** — `bullmq`, `ioredis`, and `@nestjs/*` arrive as
|
|
43
43
|
peer dependencies, so you control exact versions and the supply-chain surface stays minimal.
|
|
44
44
|
|
|
45
|
-
### Why nest-queue?
|
|
46
|
-
|
|
47
|
-
- **One setup, every queue.** A single `forRoot`/`forRootAsync`; queues are created on demand by
|
|
48
|
-
name — no `registerQueue()` per queue.
|
|
49
|
-
- **Correct connections by default.** Dual-mode (bring-your-own client or lib-owned) with the
|
|
50
|
-
per-role `maxRetriesPerRequest` policy applied automatically — the Queue connection keeps
|
|
51
|
-
default retries (fail fast), only the duplicated Worker/QueueEvents connections force `null`.
|
|
52
|
-
- **Current BullMQ API only.** Recurring jobs use Job Schedulers (`upsertJobScheduler`), never the
|
|
53
|
-
removed `addRepeatable`. Cron is validated by BullMQ's own parser, never a hand-rolled regex.
|
|
54
|
-
- **Safe shutdown.** A bounded-drain protocol with at-least-once semantics, force-close, and
|
|
55
|
-
stalled-job accounting — verified against a real Redis (Testcontainers) E2E suite.
|
|
56
|
-
|
|
57
45
|
---
|
|
58
46
|
|
|
59
47
|
## 🔥 Features
|
|
60
48
|
|
|
61
49
|
### 📤 Producing
|
|
62
50
|
|
|
51
|
+
- ✅ **One setup, every queue** — a single `forRoot` / `forRootAsync`; queues are created on demand by name, with no `registerQueue()` per queue
|
|
63
52
|
- ✅ **Typed producer API** — `enqueue<TData, TResult>()` threads both generics through to the returned `Job`, so `job.data` and `job.returnvalue` need no cast
|
|
64
53
|
- ✅ **Bounded bulk insert** — `enqueueBulk` adds up to 1 000 jobs in a single Redis roundtrip and rejects a larger batch outright, rather than letting one call inflate the heap
|
|
65
54
|
- ✅ **Native deduplication** — simple, throttle, debounce and keep-last-if-active, passed straight through to BullMQ; the library writes no deduplication logic of its own
|
|
@@ -75,15 +64,13 @@ peer dependencies, so you control exact versions and the supply-chain surface st
|
|
|
75
64
|
### 🔌 Connections & Shutdown
|
|
76
65
|
|
|
77
66
|
- ✅ **Dual-mode connection** — bring your own `ioredis` client, or let the library open its own from a URL or an options object
|
|
78
|
-
- ✅ **Per-role retry policy** — `maxRetriesPerRequest: null` reaches only the duplicated Worker and QueueEvents connections; the producer keeps
|
|
67
|
+
- ✅ **Per-role retry policy** — `maxRetriesPerRequest: null` reaches only the duplicated Worker and QueueEvents connections; the producer keeps a finite retry limit, a 5 s command deadline and no offline queue on the connection the library opens, and `enqueue` has its own 10 s deadline, so a Redis outage rejects a call in seconds instead of holding the request; a shared client with `maxRetriesPerRequest: null` is the exception, and the library warns about it at boot
|
|
79
68
|
- ✅ **One namespace, four objects** — the configured `prefix` reaches `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer`, so producers and consumers can never end up on separate keyspaces
|
|
80
69
|
- ✅ **Bounded graceful shutdown** — a single ordered sequence drains every worker under a deadline and then force-closes; delivery is at-least-once by design, so handlers must be idempotent
|
|
81
70
|
- ✅ **No unlistened emitter** — every `Queue`, `Worker`, `QueueEvents` and `FlowProducer` the library creates carries a fallback `error` listener, so a transient Redis fault cannot become an uncaught exception in your process; your own `@OnWorkerEvent('error')` or `@OnQueueEvent('error')` still runs alongside it
|
|
82
71
|
|
|
83
72
|
### 🧩 Developer Experience
|
|
84
73
|
|
|
85
|
-
- ✅ **Zero runtime dependencies** — BullMQ, ioredis and `@nestjs/*` all arrive as peer dependencies, so you pin the versions and the supply-chain surface stays yours
|
|
86
|
-
- ✅ **Two subpaths** — the server surface, and `./shared` carrying types and constants that pull in nothing at all
|
|
87
74
|
- ✅ **Stable error catalog** — namespaced codes exported from both subpaths, so a frontend maps a `code` without importing the server
|
|
88
75
|
- ✅ **Dual-format output** — ESM + CJS with declarations for each format, verified against the packed tarball on every PR
|
|
89
76
|
- ✅ **Opt-in observability** — OpenTelemetry passthrough via `bullmq-otel`, plus a `MetricsService` with a configurable TTL cache
|
|
@@ -132,20 +119,19 @@ Both subpaths ship ESM **and** CommonJS with declarations for each format, so a
|
|
|
132
119
|
> A full reference application lives in
|
|
133
120
|
> [`bymaxone/nest-queue-example`](https://github.com/bymaxone/nest-queue-example) —
|
|
134
121
|
> an API and a web console exercising every public export against a real Redis.
|
|
135
|
-
>
|
|
136
|
-
> The `test/consumer-app` directory in this repository is **not** that. It is a
|
|
137
|
-
> typecheck fixture: a few hundred lines of application-shaped code that CI
|
|
138
|
-
> compiles against the freshly built library on every pull request, to catch an
|
|
139
|
-
> ergonomic break in the decorator API that a signature test cannot see.
|
|
140
122
|
|
|
141
123
|
---
|
|
142
124
|
|
|
143
125
|
## 🚀 Quick Start
|
|
144
126
|
|
|
127
|
+
### 1. Install
|
|
128
|
+
|
|
145
129
|
```bash
|
|
146
130
|
pnpm add @bymax-one/nest-queue bullmq ioredis reflect-metadata
|
|
147
131
|
```
|
|
148
132
|
|
|
133
|
+
### 2. Load `reflect-metadata`
|
|
134
|
+
|
|
149
135
|
`reflect-metadata` is loaded **once by your application**, before anything else — this
|
|
150
136
|
is the standard NestJS entry point and the decorators in this library depend on it:
|
|
151
137
|
|
|
@@ -163,6 +149,8 @@ async function bootstrap(): Promise<void> {
|
|
|
163
149
|
void bootstrap()
|
|
164
150
|
```
|
|
165
151
|
|
|
152
|
+
### 3. Register the module
|
|
153
|
+
|
|
166
154
|
Wire the module in your root `AppModule`:
|
|
167
155
|
|
|
168
156
|
```typescript
|
|
@@ -174,7 +162,6 @@ import { EmailModule } from './email.module'
|
|
|
174
162
|
imports: [
|
|
175
163
|
BymaxQueueModule.forRoot({
|
|
176
164
|
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
177
|
-
isGlobal: true,
|
|
178
165
|
}),
|
|
179
166
|
EmailModule,
|
|
180
167
|
],
|
|
@@ -182,7 +169,9 @@ import { EmailModule } from './email.module'
|
|
|
182
169
|
export class AppModule {}
|
|
183
170
|
```
|
|
184
171
|
|
|
185
|
-
Enqueue a job
|
|
172
|
+
### 4. Enqueue a job
|
|
173
|
+
|
|
174
|
+
From any service:
|
|
186
175
|
|
|
187
176
|
```typescript
|
|
188
177
|
import { Injectable } from '@nestjs/common'
|
|
@@ -198,7 +187,9 @@ export class EmailService {
|
|
|
198
187
|
}
|
|
199
188
|
```
|
|
200
189
|
|
|
201
|
-
Process jobs
|
|
190
|
+
### 5. Process jobs
|
|
191
|
+
|
|
192
|
+
With a decorated handler:
|
|
202
193
|
|
|
203
194
|
```typescript
|
|
204
195
|
import { Injectable } from '@nestjs/common'
|
|
@@ -215,8 +206,11 @@ export class EmailProcessor {
|
|
|
215
206
|
}
|
|
216
207
|
```
|
|
217
208
|
|
|
218
|
-
|
|
219
|
-
|
|
209
|
+
### 6. Register the providers
|
|
210
|
+
|
|
211
|
+
Put the service and the processor in a feature module that `AppModule` imports, like the `EmailModule`
|
|
212
|
+
above. The module is global by default (`isGlobal` defaults to `true`), so `QueueService` is injectable
|
|
213
|
+
there without importing `BymaxQueueModule` again:
|
|
220
214
|
|
|
221
215
|
```typescript
|
|
222
216
|
import { Module } from '@nestjs/common'
|
|
@@ -235,20 +229,22 @@ export class EmailModule {}
|
|
|
235
229
|
|
|
236
230
|
Every option passed to `forRoot` / `forRootAsync`. Only `connection` is required.
|
|
237
231
|
|
|
238
|
-
| Option | Type | Default
|
|
239
|
-
| -------------------------- | ------------------------------------------------ |
|
|
240
|
-
| `connection` | `{ client } \| { url, options? } \| { options }` | —
|
|
241
|
-
| `prefix` | `string` | `'bull'`
|
|
242
|
-
| `defaultJobOptions` | `JobsOptions` | `attempts: 3` · exponential backoff `2000` ms · `removeOnComplete: { age: 24 h, count: 1000 }` · `removeOnFail: { age: 7 d, count: 5000 }` | Merged into every job enqueued through the service; per-job options win. The bounded `removeOn*` caps are what keep Redis from growing without limit under a failure storm.
|
|
243
|
-
| `queueOptions` | `Partial<QueueOptions>` | `{}`
|
|
244
|
-
| `flows.enabled` | `boolean` | `false`
|
|
245
|
-
| `metrics.enabled` | `boolean` | `false`
|
|
246
|
-
| `metrics.cacheTtlMs` | `number` | `5_000`
|
|
247
|
-
| `telemetry` | `Telemetry` | —
|
|
248
|
-
| `shutdown.drainTimeoutMs` | `number` | `30_000`
|
|
249
|
-
| `shutdown.drainOnShutdown` | `boolean` | `false`
|
|
250
|
-
| `connectionReadyTimeoutMs` | `number` | `10_000`
|
|
251
|
-
| `
|
|
232
|
+
| Option | Type | Default | Description |
|
|
233
|
+
| -------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
234
|
+
| `connection` | `{ client } \| { url, options? } \| { options }` | — | **Required.** Mode A supplies an existing `ioredis` client; Mode B lets the library open its own. The arms are mutually exclusive. |
|
|
235
|
+
| `prefix` | `string` | `'bull'` | Redis key namespace. Applied to the `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer` — see [Architecture](#-architecture). |
|
|
236
|
+
| `defaultJobOptions` | `JobsOptions` | `attempts: 3` · exponential backoff `2000` ms with 50% jitter · `removeOnComplete: { age: 24 h, count: 1000 }` · `removeOnFail: { age: 7 d, count: 5000 }` | Merged into every job enqueued through the service; per-job options win. The bounded `removeOn*` caps are what keep Redis from growing without limit under a failure storm. |
|
|
237
|
+
| `queueOptions` | `Partial<QueueOptions>` | `{}` | Extra BullMQ `Queue` options. `connection`, `defaultJobOptions` and `prefix` are owned by the library and cannot be set here. |
|
|
238
|
+
| `flows.enabled` | `boolean` | `false` | `FlowService` is always provided; while disabled, every method except `close()` throws `FLOW_DISABLED`. |
|
|
239
|
+
| `metrics.enabled` | `boolean` | `false` | `MetricsService` is always provided; while disabled, every method throws `METRICS_DISABLED`. |
|
|
240
|
+
| `metrics.cacheTtlMs` | `number` | `5_000` | TTL of the metrics snapshot cache. `0` disables caching. Must be `>= 0`. |
|
|
241
|
+
| `telemetry` | `Telemetry` | — | A BullMQ `Telemetry` instance (typically `new BullMQOtel(...)` from the optional `bullmq-otel` peer), attached to every `Queue` and `Worker`. |
|
|
242
|
+
| `shutdown.drainTimeoutMs` | `number` | `30_000` | Upper bound on the graceful drain before workers are force-closed. Must be `> 0`. |
|
|
243
|
+
| `shutdown.drainOnShutdown` | `boolean` | `false` | Development and test only: once the HTTP server has stopped, call `drain()` on every cached queue, which **removes** its waiting jobs (delayed and active jobs are left alone). Leave it `false` in production. |
|
|
244
|
+
| `connectionReadyTimeoutMs` | `number` | `10_000` | Mode B only: the budget for Redis to reach `ready`. A connection error while the client is still reconnecting does not end the wait; when the budget ends, startup fails with `CONNECTION_TIMEOUT` carrying `lastErrorCode` (a code such as `ECONNREFUSED`, never the message) and the client the library opened is closed. |
|
|
245
|
+
| `commandTimeoutMs` | `number` | `5_000` | Mode B only: the deadline for one Redis command on the connection the library opened (ioredis `commandTimeout`); an `options.commandTimeout` wins. A call that hits it rejects with `CONNECTION_TIMEOUT` (`details` carries `operation` and `timeoutMs`), as when `enqueueTimeoutMs` passes. Worker and QueueEvents connections have none. |
|
|
246
|
+
| `enqueueTimeoutMs` | `number` | `10_000` | The deadline for one `enqueue` or `enqueueBulk`, in either mode; it rejects with `CONNECTION_TIMEOUT`, and the add may still complete in Redis, so retry with a stable `jobId` or `deduplication`. |
|
|
247
|
+
| `isGlobal` | `boolean` | `true` | Register the module globally, so `QueueService` is injectable without re-importing. |
|
|
252
248
|
|
|
253
249
|
> [!WARNING]
|
|
254
250
|
> **`shutdown.drainOnShutdown` deletes jobs.** When it is `true`, shutdown calls `drain()` on every
|
|
@@ -262,11 +258,16 @@ first job is enqueued.
|
|
|
262
258
|
|
|
263
259
|
---
|
|
264
260
|
|
|
265
|
-
## 🔌
|
|
261
|
+
## 🔌 Connections
|
|
262
|
+
|
|
263
|
+
`connection` takes a client (**Mode A**) or a URL or ioredis options (**Mode B**); the arms are mutually
|
|
264
|
+
exclusive. In both modes the Queue connection keeps the ioredis defaults, and the Worker and
|
|
265
|
+
`QueueEvents` connections are `duplicate()`s of it with `maxRetriesPerRequest: null`.
|
|
266
266
|
|
|
267
|
-
Mode A
|
|
268
|
-
|
|
269
|
-
|
|
267
|
+
### Mode A — your own client
|
|
268
|
+
|
|
269
|
+
Recommended when the application already runs `@bymax-one/nest-cache`: one client is shared and
|
|
270
|
+
duplicated per role, and your cache connection is left alone.
|
|
270
271
|
|
|
271
272
|
```typescript
|
|
272
273
|
import { BymaxQueueModule } from '@bymax-one/nest-queue'
|
|
@@ -277,30 +278,19 @@ BymaxQueueModule.forRootAsync({
|
|
|
277
278
|
inject: [ConnectionManager],
|
|
278
279
|
useFactory: (cache: ConnectionManager) => ({
|
|
279
280
|
connection: { client: cache.getClient() as Redis },
|
|
280
|
-
isGlobal: true,
|
|
281
281
|
}),
|
|
282
282
|
})
|
|
283
283
|
```
|
|
284
284
|
|
|
285
|
-
`BymaxCacheModule` is global by default and provides `ConnectionManager
|
|
286
|
-
|
|
287
|
-
`@bymax-one/nest-cache` in standalone mode.
|
|
288
|
-
|
|
289
|
-
Mode A is the recommended setup when you already have `@bymax-one/nest-cache` in the app.
|
|
290
|
-
One client is shared; the lib duplicates it per role and applies the correct retry policy
|
|
291
|
-
without touching your cache connection.
|
|
292
|
-
|
|
293
|
-
---
|
|
294
|
-
|
|
295
|
-
## 🔌 Mode B — Lib-Owned Connection
|
|
285
|
+
`BymaxCacheModule` is global by default and provides `ConnectionManager`. `getClient()` is typed
|
|
286
|
+
`Redis | Cluster` and Mode A takes a standalone `Redis`, so run `@bymax-one/nest-cache` in standalone mode.
|
|
296
287
|
|
|
297
|
-
Mode B
|
|
288
|
+
### Mode B — a connection the library owns
|
|
298
289
|
|
|
299
290
|
```typescript
|
|
300
291
|
// URL form
|
|
301
292
|
BymaxQueueModule.forRoot({
|
|
302
293
|
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
303
|
-
isGlobal: true,
|
|
304
294
|
})
|
|
305
295
|
|
|
306
296
|
// Options form
|
|
@@ -313,222 +303,30 @@ BymaxQueueModule.forRoot({
|
|
|
313
303
|
tls: {},
|
|
314
304
|
},
|
|
315
305
|
},
|
|
316
|
-
isGlobal: true,
|
|
317
306
|
})
|
|
318
307
|
```
|
|
319
308
|
|
|
320
|
-
|
|
321
|
-
queue that has an `@OnQueueEvent` listener ([the budget is below](#-redis-connections)). The worker
|
|
322
|
-
and `QueueEvents` connections carry `maxRetriesPerRequest: null`.
|
|
323
|
-
|
|
324
|
-
---
|
|
325
|
-
|
|
326
|
-
## 🔌 Redis connections
|
|
327
|
-
|
|
328
|
-
How many connections the library holds is linear in what you register, so you can budget it before
|
|
329
|
-
you deploy. Counts were read from `INFO clients` against Redis 8 with BullMQ 6 and `ioredis` 6, and
|
|
330
|
-
every connection is released when the application context closes.
|
|
309
|
+
### Connection budget
|
|
331
310
|
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
| The module (Mode B, library-owned connection) | 1, shared by every producer and every flow |
|
|
335
|
-
| The module (Mode A, your client) | 0 of its own; your client is yours and is never counted |
|
|
336
|
-
| A queue you enqueue to, with `QueueService` | 0 more: queues share the module's connection |
|
|
337
|
-
| `FlowService` (flows enabled) | 0 more |
|
|
338
|
-
| Each worker (`@Processor` or `WorkerRegistry`) | 2: a blocking connection that waits for jobs, and a command connection |
|
|
339
|
-
| Each queue observed with `@OnQueueEvent` | 2: a blocking connection that reads the event stream, and a command connection |
|
|
311
|
+
The connections held are linear in what you register. Counts were read from `INFO clients` against
|
|
312
|
+
Redis 8 with BullMQ 6 and `ioredis` 6; every connection the library opens is released when the application context closes.
|
|
340
313
|
|
|
341
|
-
|
|
342
|
-
|
|
314
|
+
| What you register | Connections held |
|
|
315
|
+
| ---------------------------------------------- | ----------------------------------------------------------------------- |
|
|
316
|
+
| The module, Mode B | 1, shared by every queue and every flow |
|
|
317
|
+
| The module, Mode A | 0 of its own; your client is yours and is never counted |
|
|
318
|
+
| Each worker (`@Processor` or `WorkerRegistry`) | 2: a blocking connection that waits for jobs, and a command connection |
|
|
319
|
+
| Each queue observed with `@OnQueueEvent` | 2: a blocking connection that reads the event stream, and a command one |
|
|
343
320
|
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
instances hold 153.
|
|
321
|
+
Queues and `FlowService` add none, so the cost comes from consumers: a blocking connection is parked
|
|
322
|
+
inside a blocking command until a job arrives and cannot be shared. **Example:** 20 workers and 5 observed
|
|
323
|
+
queues in Mode B hold 1 + (2 × 20) + (2 × 5) = **51** connections (50 in Mode A, plus whatever your own
|
|
324
|
+
client uses). Each instance opens its own set, so three instances hold 153.
|
|
348
325
|
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
In `CLIENT LIST`, the blocking connection of a worker has a name that starts with
|
|
354
|
-
`<prefix>:<base64 of the queue name>`, which identifies it; the command connection is unnamed.
|
|
355
|
-
|
|
356
|
-
---
|
|
357
|
-
|
|
358
|
-
## 📖 API Reference
|
|
359
|
-
|
|
360
|
-
### `QueueService`
|
|
361
|
-
|
|
362
|
-
The central interaction point with BullMQ queues.
|
|
363
|
-
|
|
364
|
-
```typescript
|
|
365
|
-
import { QueueService } from '@bymax-one/nest-queue'
|
|
366
|
-
import type {
|
|
367
|
-
Job,
|
|
368
|
-
JobsOptions,
|
|
369
|
-
JobSchedulerRepeatOptions,
|
|
370
|
-
QueueMetrics,
|
|
371
|
-
} from '@bymax-one/nest-queue'
|
|
372
|
-
```
|
|
373
|
-
|
|
374
|
-
| Method | Signature | Description |
|
|
375
|
-
| -------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
|
376
|
-
| `enqueue` | `<TData, TResult>(queue, name, data, opts?) → Promise<Job<TData, TResult>>` | Add a single job |
|
|
377
|
-
| `enqueueBulk` | `<TData, TResult>(queue, jobs) → Promise<Job<TData, TResult>[]>` | Add up to 1 000 jobs in one Redis roundtrip |
|
|
378
|
-
| `getJob` | `<TData, TResult>(queue, jobId) → Promise<Job<TData, TResult> \| null>` | Get job by ID; `null` when it does not exist |
|
|
379
|
-
| `getJobs` | `<TData, TResult>(queue, status, start?, end?) → Promise<Job<TData, TResult>[]>` | Page jobs in one status (default `0`–`50`) |
|
|
380
|
-
| `getMetrics` | `(queue) → Promise<QueueMetrics>` | Uncached snapshot of job counts per status, with a collection timestamp |
|
|
381
|
-
| `pauseQueue` | `(queue) → Promise<void>` | Pause a queue |
|
|
382
|
-
| `resumeQueue` | `(queue) → Promise<void>` | Resume a paused queue |
|
|
383
|
-
| `cleanQueue` | `(queue, gracePeriodMs, limit, status?) → Promise<string[]>` | Remove up to `limit` jobs older than `gracePeriodMs` (`0` = no limit); returns removed ids |
|
|
384
|
-
| `upsertJobScheduler` | `<TData, TResult>(queue, id, repeat, template?) → Promise<Job<TData, TResult> \| undefined>` | Create or update a recurring Job Scheduler; returns the first scheduled job |
|
|
385
|
-
| `removeJobScheduler` | `(queue, id) → Promise<boolean>` | Remove a Job Scheduler by ID |
|
|
386
|
-
| `getJobSchedulers` | `(queue, start?, end?, asc?) → Promise<JobSchedulerJson[]>` | Page the schedulers registered on a queue |
|
|
387
|
-
| `getOrCreateQueue` | `<TData, TResult>(queue, overrides?) → Queue<TData, TResult>` | Return the cached raw `Queue` instance |
|
|
388
|
-
| `getCachedQueues` | `() → ReadonlyMap<string, Queue>` | Read-only view of every queue created so far |
|
|
389
|
-
|
|
390
|
-
### `WorkerRegistry`
|
|
391
|
-
|
|
392
|
-
Programmatic API to create and manage BullMQ workers at runtime.
|
|
393
|
-
|
|
394
|
-
```typescript
|
|
395
|
-
import path from 'node:path'
|
|
396
|
-
import { WorkerRegistry } from '@bymax-one/nest-queue'
|
|
397
|
-
import type { ProgrammaticWorkerConfig, SandboxedWorkerConfig } from '@bymax-one/nest-queue'
|
|
398
|
-
|
|
399
|
-
// Register an in-process worker
|
|
400
|
-
registry.register({
|
|
401
|
-
queueName: 'email',
|
|
402
|
-
handler: async (job) => {
|
|
403
|
-
/* … */
|
|
404
|
-
},
|
|
405
|
-
options: { concurrency: 4 },
|
|
406
|
-
})
|
|
407
|
-
|
|
408
|
-
// Register a sandboxed (out-of-process) processor
|
|
409
|
-
registry.registerSandboxed({
|
|
410
|
-
queueName: 'heavy-compute',
|
|
411
|
-
processorFile: path.resolve(__dirname, 'compute.processor.js'),
|
|
412
|
-
options: { concurrency: 2 },
|
|
413
|
-
})
|
|
414
|
-
```
|
|
415
|
-
|
|
416
|
-
### `FlowService` (opt-in)
|
|
417
|
-
|
|
418
|
-
Enable flows in the module options and inject the service:
|
|
419
|
-
|
|
420
|
-
```typescript
|
|
421
|
-
BymaxQueueModule.forRoot({
|
|
422
|
-
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
423
|
-
flows: { enabled: true },
|
|
424
|
-
})
|
|
425
|
-
|
|
426
|
-
// Then inject:
|
|
427
|
-
import { FlowService } from '@bymax-one/nest-queue'
|
|
428
|
-
import type { FlowJob, JobNode } from '@bymax-one/nest-queue'
|
|
429
|
-
|
|
430
|
-
const tree: JobNode = await this.flows.add({
|
|
431
|
-
name: 'process-order',
|
|
432
|
-
queueName: 'orders',
|
|
433
|
-
data: { orderId: '123' },
|
|
434
|
-
children: [
|
|
435
|
-
{ name: 'charge-payment', queueName: 'payments', data: {} },
|
|
436
|
-
{ name: 'send-confirmation', queueName: 'email', data: {} },
|
|
437
|
-
],
|
|
438
|
-
})
|
|
439
|
-
```
|
|
440
|
-
|
|
441
|
-
### `MetricsService` (opt-in)
|
|
442
|
-
|
|
443
|
-
Provide a per-queue snapshot with a configurable TTL cache:
|
|
444
|
-
|
|
445
|
-
```typescript
|
|
446
|
-
BymaxQueueModule.forRoot({
|
|
447
|
-
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
448
|
-
metrics: { enabled: true, cacheTtlMs: 5_000 },
|
|
449
|
-
})
|
|
450
|
-
|
|
451
|
-
import { MetricsService } from '@bymax-one/nest-queue'
|
|
452
|
-
import type { QueueMetrics } from '@bymax-one/nest-queue'
|
|
453
|
-
|
|
454
|
-
const metrics: QueueMetrics = await this.metricsService.get('email')
|
|
455
|
-
const all: readonly QueueMetrics[] = await this.metricsService.getAll()
|
|
456
|
-
```
|
|
457
|
-
|
|
458
|
-
### Other exports
|
|
459
|
-
|
|
460
|
-
| Export | What it is |
|
|
461
|
-
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
462
|
-
| `WorkerRegistry.unregister(queue)`, `list()` | Stop and remove a worker and its connection; list the queue names that have a worker |
|
|
463
|
-
| `FlowService.addBulk(flows)`, `getProducer()`, `close()` | Add several flows at once; the underlying BullMQ `FlowProducer`; close it (a no-op while flows are disabled) |
|
|
464
|
-
| `MetricsService.invalidate(queue?)` | Drop the cached snapshot of one queue, or of all of them |
|
|
465
|
-
| `QueueEventsRegistry` | The shared `QueueEvents` per queue (`getOrCreate`, `list`), used by `@OnQueueEvent` |
|
|
466
|
-
| `BYMAX_QUEUE_OPTIONS`, `BYMAX_QUEUE_RESOLVED_OPTIONS` | Injection tokens for the options as given and with the defaults applied |
|
|
467
|
-
| `BYMAX_QUEUE_REDIS_CLIENT`, `BYMAX_QUEUE_CONNECTION_MODE` | Injection tokens for the Queue-role ioredis client and for `'mode-a-byo'` or `'mode-b-owned'` |
|
|
468
|
-
| `DEFAULT_WORKER_CONCURRENCY`, `MAX_WORKER_CONCURRENCY` | The worker concurrency used when none is given, and its ceiling |
|
|
469
|
-
| `DEFAULT_JOB_OPTIONS`, `DEFAULT_METRICS_CACHE_TTL_MS`, `DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS` | The defaults of the Configuration table |
|
|
470
|
-
| `QUEUE_ERROR_MESSAGES` | The default message of each error code |
|
|
471
|
-
| `BymaxQueueModuleOptions`, `BymaxQueueModuleAsyncOptions`, `BymaxQueueOptionsFactory`, `QueueConnectionConfig`, `QueueConnectionMode` | The option and connection types behind `forRoot` and `forRootAsync` |
|
|
472
|
-
| `ProcessorMetadata`, `ProcessHandlerMetadata`, `QueueEventListenerMetadata` | The reflection metadata the decorators write |
|
|
473
|
-
| `JobStatus`, `JOB_STATUS`, `BulkJob`, `SandboxedJob`, `WorkerEventName`, `QueueEventName`, `WorkerOptions` | Constants and types for the APIs above |
|
|
474
|
-
|
|
475
|
-
---
|
|
476
|
-
|
|
477
|
-
## 🚨 Error Code Catalog
|
|
478
|
-
|
|
479
|
-
Whatever **this library rejects** is a `QueueException` carrying a stable,
|
|
480
|
-
transport-independent `code`. Branch on the code, never on the HTTP status or the
|
|
481
|
-
message text — the code is the contract.
|
|
482
|
-
|
|
483
|
-
Whatever **fails underneath it** is not wrapped at all. A dropped connection, a
|
|
484
|
-
timeout or any other operational fault reaches you as the original error, with its
|
|
485
|
-
own type and message, rather than being relabelled as invalid input. So the catalog
|
|
486
|
-
below is a complete list of the library's own rejections, and deliberately not a
|
|
487
|
-
complete list of everything a call can throw.
|
|
488
|
-
|
|
489
|
-
```typescript
|
|
490
|
-
import { QueueException, QUEUE_ERROR_CODES } from '@bymax-one/nest-queue'
|
|
491
|
-
|
|
492
|
-
try {
|
|
493
|
-
await queue.enqueue('email', 'welcome', { userId })
|
|
494
|
-
} catch (err) {
|
|
495
|
-
if (err instanceof QueueException) {
|
|
496
|
-
const { code } = (err.getResponse() as { error: { code: string } }).error
|
|
497
|
-
if (code === QUEUE_ERROR_CODES.BULK_ENQUEUE_FAILED) {
|
|
498
|
-
/* … */
|
|
499
|
-
}
|
|
500
|
-
}
|
|
501
|
-
}
|
|
502
|
-
```
|
|
503
|
-
|
|
504
|
-
| Code | HTTP | When thrown |
|
|
505
|
-
| ---------------------------------------- | :--: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
506
|
-
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
|
|
507
|
-
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, or Mode A's injected client is in an unusable `status` |
|
|
508
|
-
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
|
|
509
|
-
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` |
|
|
510
|
-
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
|
|
511
|
-
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
|
|
512
|
-
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
|
|
513
|
-
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
|
|
514
|
-
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
|
|
515
|
-
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
|
|
516
|
-
|
|
517
|
-
### Reserved codes
|
|
518
|
-
|
|
519
|
-
These are part of the published catalog and stable, but nothing currently throws them.
|
|
520
|
-
They are listed so a consumer writing an exhaustive `switch` over `QueueErrorCode`
|
|
521
|
-
compiles today and keeps compiling when a future release starts emitting them.
|
|
522
|
-
|
|
523
|
-
| Code | Why it is not thrown today |
|
|
524
|
-
| --------------------------------- | --------------------------------------------------------------------- |
|
|
525
|
-
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
|
|
526
|
-
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
|
|
527
|
-
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
|
|
528
|
-
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline force-closes workers rather than raising |
|
|
529
|
-
|
|
530
|
-
The same catalog is exported from `@bymax-one/nest-queue/shared` with no dependencies,
|
|
531
|
-
so a frontend can map codes to messages without importing the server surface.
|
|
326
|
+
Hosted Redis plans cap concurrent connections: multiply by the instances you run (rolling deploys
|
|
327
|
+
overlap old and new) and compare with your plan's limit; self-managed Redis allows 10 000 (`maxclients`).
|
|
328
|
+
In `CLIENT LIST`, a worker's blocking connection is named `<prefix>:<base64 of the queue name>…`; the
|
|
329
|
+
command connection is unnamed.
|
|
532
330
|
|
|
533
331
|
---
|
|
534
332
|
|
|
@@ -619,34 +417,20 @@ A pattern the parser rejects is `INVALID_REPEAT_OPTIONS` (400) before Redis is a
|
|
|
619
417
|
Pass the `deduplication` option directly on `enqueue` — the library delegates the behavior entirely to BullMQ:
|
|
620
418
|
|
|
621
419
|
```typescript
|
|
420
|
+
const id = `welcome:${userId}`
|
|
421
|
+
|
|
622
422
|
// Simple: collapse duplicates until the first in-flight job completes
|
|
623
|
-
await this.queue.enqueue(
|
|
624
|
-
'email',
|
|
625
|
-
'welcome',
|
|
626
|
-
{ userId },
|
|
627
|
-
{
|
|
628
|
-
deduplication: { id: `welcome:${userId}` },
|
|
629
|
-
},
|
|
630
|
-
)
|
|
423
|
+
await this.queue.enqueue('email', 'welcome', { userId }, { deduplication: { id } })
|
|
631
424
|
|
|
632
425
|
// Throttle: ignore duplicates for 30 seconds
|
|
633
|
-
await this.queue.enqueue(
|
|
634
|
-
'email',
|
|
635
|
-
'welcome',
|
|
636
|
-
{ userId },
|
|
637
|
-
{
|
|
638
|
-
deduplication: { id: `welcome:${userId}`, ttl: 30_000 },
|
|
639
|
-
},
|
|
640
|
-
)
|
|
426
|
+
await this.queue.enqueue('email', 'welcome', { userId }, { deduplication: { id, ttl: 30_000 } })
|
|
641
427
|
|
|
642
428
|
// Debounce: keep only the latest; each duplicate resets the TTL
|
|
643
429
|
await this.queue.enqueue(
|
|
644
430
|
'email',
|
|
645
431
|
'welcome',
|
|
646
432
|
{ userId },
|
|
647
|
-
{
|
|
648
|
-
deduplication: { id: `welcome:${userId}`, ttl: 5_000, extend: true, replace: true },
|
|
649
|
-
},
|
|
433
|
+
{ deduplication: { id, ttl: 5_000, extend: true, replace: true } },
|
|
650
434
|
)
|
|
651
435
|
```
|
|
652
436
|
|
|
@@ -654,6 +438,10 @@ await this.queue.enqueue(
|
|
|
654
438
|
|
|
655
439
|
When a custom `jobId` already exists, or a `deduplication` rule keeps the job that is already there (a rule with `replace` swaps it for the new one instead), `enqueue`, `enqueueBulk` and the flow methods add nothing: BullMQ answers with the id of the **existing** job together with the data and options of **this** call, so the returned object does not describe what is stored. Call `QueueService.getJob` when you need the stored job.
|
|
656
440
|
|
|
441
|
+
### A timed-out `enqueue` may still have been added
|
|
442
|
+
|
|
443
|
+
When Redis does not answer in time, whether within `commandTimeoutMs` on the connection the library opened or within `enqueueTimeoutMs`, `enqueue` and `enqueueBulk` reject with `CONNECTION_TIMEOUT`, but the library cannot cancel the command: if Redis applies it late, the job exists anyway. Retrying a call that timed out can therefore add the job twice. For anything that must not run twice, pass a deterministic `jobId` (for example one derived from the business key): BullMQ ignores a second add with the same `jobId` for as long as the first job is still stored, so a retry inside that window is safe. A completed job is removed by `removeOnComplete`, and from then on the same `jobId` adds a new job, so keep the retry inside that window or keep the handler idempotent. A `deduplication` rule gives the same guarantee for work that is allowed to repeat after a window.
|
|
444
|
+
|
|
657
445
|
### OpenTelemetry
|
|
658
446
|
|
|
659
447
|
Pass a `bullmq-otel` telemetry instance (optional peer dep) to trace spans from `enqueue` through the handler:
|
|
@@ -674,12 +462,15 @@ BymaxQueueModule.forRoot({
|
|
|
674
462
|
`QueueLifecycle` runs automatically when NestJS closes the application context, through its `onModuleDestroy` and `onApplicationShutdown` hooks (call `app.enableShutdownHooks()` so that a termination signal closes the context).
|
|
675
463
|
It implements a **bounded drain protocol** with at-least-once semantics:
|
|
676
464
|
|
|
677
|
-
1. All workers are closed
|
|
678
|
-
|
|
679
|
-
|
|
465
|
+
1. All workers are closed **in parallel**, within one budget (`drainTimeoutMs`, default 30 s) that
|
|
466
|
+
covers steps 1 to 4 together: however many workers are stuck, consumption stops after one budget,
|
|
467
|
+
not one per worker.
|
|
468
|
+
2. A worker resolves either when it drains normally or when the budget runs out — whichever comes
|
|
469
|
+
first.
|
|
680
470
|
3. On timeout, the worker is force-closed (`worker.close(true)`), preserving stalled
|
|
681
471
|
jobs for reprocessing by the next instance.
|
|
682
|
-
4. Queue-events and the connections the library opened for workers and queue-events are closed
|
|
472
|
+
4. Queue-events and the connections the library opened for workers and queue-events are closed,
|
|
473
|
+
each within what is left of the budget; a connection that has not quit by then is disconnected.
|
|
683
474
|
5. Once NestJS has stopped the HTTP server (`onApplicationShutdown`), the flow producer, the
|
|
684
475
|
cached queues and, in Mode B, the library's own Redis connection are closed. NestJS runs
|
|
685
476
|
`onModuleDestroy` before it stops the HTTP server, so a request that is still being served can
|
|
@@ -766,14 +557,13 @@ wire connection and defaults yourself.
|
|
|
766
557
|
|
|
767
558
|
### Design Principles
|
|
768
559
|
|
|
769
|
-
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
|
773
|
-
|
|
|
774
|
-
|
|
|
775
|
-
|
|
|
776
|
-
| 🧊 **Zero runtime dependencies** | `dependencies` is `{}`. You choose the exact `bullmq` and `ioredis` versions, and the supply-chain surface stays yours. |
|
|
560
|
+
The role-shaped connections, the shared `prefix` and the Job Scheduler API are described under
|
|
561
|
+
[Features](#-features) and [Connections](#-connections). Two principles are not:
|
|
562
|
+
|
|
563
|
+
| Principle | Description |
|
|
564
|
+
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
565
|
+
| 💥 **Fail at boot, not at runtime** | Options are validated before any connection is opened, and every rejection carries a distinct reason. |
|
|
566
|
+
| 🧬 **Explicit DI tokens** | Tokens are `Symbol()` and every injectable constructor parameter carries an explicit `@Inject`. The package ships bundled, where design-time metadata is not emitted — implicit resolution would break inside your build, not ours. |
|
|
777
567
|
|
|
778
568
|
---
|
|
779
569
|
|
|
@@ -817,13 +607,35 @@ consumed from another. It is **not** an access control: anything that can reach
|
|
|
817
607
|
instance can read every namespace on it. Cross-tenant isolation that must survive a
|
|
818
608
|
compromised client belongs in Redis ACLs or separate instances.
|
|
819
609
|
|
|
610
|
+
### Reading more than 1 000 jobs
|
|
611
|
+
|
|
612
|
+
`getJobs` reads at most 1 000 jobs per call, and refuses BullMQ's `-1` ("everything") with `INVALID_OPTIONS`, because that reads a status list of any size into memory in one reply. Page through it instead:
|
|
613
|
+
|
|
614
|
+
```typescript
|
|
615
|
+
import type { QueueService } from '@bymax-one/nest-queue'
|
|
616
|
+
|
|
617
|
+
async function inspectFailed(
|
|
618
|
+
queueService: QueueService,
|
|
619
|
+
inspect: (job: unknown) => Promise<void>,
|
|
620
|
+
): Promise<void> {
|
|
621
|
+
const PAGE = 500
|
|
622
|
+
for (let start = 0; ; start += PAGE) {
|
|
623
|
+
const page = await queueService.getJobs('reports', 'failed', start, start + PAGE - 1)
|
|
624
|
+
for (const job of page) await inspect(job)
|
|
625
|
+
if (page.length < PAGE) break
|
|
626
|
+
}
|
|
627
|
+
}
|
|
628
|
+
```
|
|
629
|
+
|
|
630
|
+
### Queue names are keys, not user input
|
|
631
|
+
|
|
632
|
+
Every queue name used with `QueueService` keeps a `Queue` object and its Redis resources for the life of the process, and names are never evicted. A name built from request data (a tenant id, a URL segment) lets whoever controls it grow memory without limit and create keys in your Redis. Take queue names from a fixed set in code or validate them against an allowlist before use.
|
|
633
|
+
|
|
820
634
|
### Shutdown is bounded and at-least-once
|
|
821
635
|
|
|
822
|
-
|
|
823
|
-
|
|
824
|
-
|
|
825
|
-
be idempotent.** A handler that charges a card or sends an email without an idempotency
|
|
826
|
-
key will do it twice at some point.
|
|
636
|
+
A job still running when the drain deadline expires is **not** acknowledged, so BullMQ's stalled-job
|
|
637
|
+
check re-queues it and it runs again. **Every handler must be idempotent** — see
|
|
638
|
+
[Graceful Shutdown](#-graceful-shutdown).
|
|
827
639
|
|
|
828
640
|
### Security Checklist
|
|
829
641
|
|
|
@@ -850,27 +662,25 @@ key will do it twice at some point.
|
|
|
850
662
|
| Error payloads | `QueueException.details` restricted to scalar values — no `job.data`, no connection object |
|
|
851
663
|
| Job payloads | Treated as opaque; never deep-merged (prototype-pollution guard), never logged |
|
|
852
664
|
| Namespacing | `prefix` applied uniformly to `Queue`, `Worker`, `QueueEvents` and `FlowProducer` |
|
|
853
|
-
| Connection policy | `maxRetriesPerRequest: null` confined to duplicated consumer connections; producers keep
|
|
665
|
+
| Connection policy | `maxRetriesPerRequest: null` confined to duplicated consumer connections; producers keep finite retries, a command deadline and no offline queue (Mode B) |
|
|
854
666
|
| Delivery | At-least-once with a bounded drain; unacknowledged work is re-queued by BullMQ's stalled check |
|
|
855
|
-
| Supply chain |
|
|
667
|
+
| Supply chain | SHA-pinned third-party Actions, OSV-Scanner, TruffleHog, OpenSSF Scorecard; npm publish over OIDC |
|
|
856
668
|
| Input validation | Options validated at bootstrap; cron patterns validated by BullMQ's own parser, never a hand-rolled regex |
|
|
857
669
|
|
|
858
|
-
> [!IMPORTANT]
|
|
859
|
-
> **Delivery is at-least-once, and `prefix` is not an access boundary.** Handlers must be
|
|
860
|
-
> idempotent, and anything with credentials for the Redis instance can read every
|
|
861
|
-
> namespace on it.
|
|
862
|
-
|
|
863
670
|
---
|
|
864
671
|
|
|
865
672
|
## 🧱 Tech Stack
|
|
866
673
|
|
|
867
|
-
|
|
868
|
-
-
|
|
869
|
-
-
|
|
870
|
-
-
|
|
871
|
-
-
|
|
872
|
-
|
|
873
|
-
-
|
|
674
|
+
[](https://nodejs.org)
|
|
675
|
+
[](https://nestjs.com)
|
|
676
|
+
[](https://bullmq.io)
|
|
677
|
+
[](https://github.com/redis/ioredis)
|
|
678
|
+
[](https://www.typescriptlang.org)
|
|
679
|
+
[](https://jestjs.io)
|
|
680
|
+
[](https://testcontainers.com)
|
|
681
|
+
[](https://stryker-mutator.io)
|
|
682
|
+
[](https://pnpm.io)
|
|
683
|
+
[](https://tsup.egoist.dev)
|
|
874
684
|
|
|
875
685
|
---
|
|
876
686
|
|
|
@@ -885,9 +695,7 @@ key will do it twice at some point.
|
|
|
885
695
|
- **Published-shape gates**: `pnpm check:exports` resolves every entrypoint the way each
|
|
886
696
|
module-resolution mode would (all four `attw` quadrants green, `node10` included), and
|
|
887
697
|
`pnpm smoke` resolves the packed tarball from a real consumer in ESM _and_ CommonJS.
|
|
888
|
-
- **Bundle budgets** enforced
|
|
889
|
-
- **Supply chain:** SHA-pinned third-party GitHub Actions, OSV-Scanner, TruffleHog, OpenSSF Scorecard,
|
|
890
|
-
npm publish with provenance (OIDC).
|
|
698
|
+
- **Bundle budgets** enforced by `pnpm size`.
|
|
891
699
|
|
|
892
700
|
```bash
|
|
893
701
|
pnpm typecheck && pnpm test:types && pnpm lint && pnpm test:cov:all && \
|
|
@@ -896,40 +704,196 @@ pnpm typecheck && pnpm test:types && pnpm lint && pnpm test:cov:all && \
|
|
|
896
704
|
|
|
897
705
|
---
|
|
898
706
|
|
|
899
|
-
##
|
|
707
|
+
## 📖 API Reference
|
|
900
708
|
|
|
901
|
-
### `
|
|
709
|
+
### `QueueService`
|
|
902
710
|
|
|
903
|
-
|
|
711
|
+
The central interaction point with BullMQ queues.
|
|
904
712
|
|
|
713
|
+
```typescript
|
|
714
|
+
import { QueueService } from '@bymax-one/nest-queue'
|
|
715
|
+
import type {
|
|
716
|
+
Job,
|
|
717
|
+
JobsOptions,
|
|
718
|
+
JobSchedulerRepeatOptions,
|
|
719
|
+
QueueMetrics,
|
|
720
|
+
} from '@bymax-one/nest-queue'
|
|
905
721
|
```
|
|
906
|
-
|
|
907
|
-
|
|
722
|
+
|
|
723
|
+
| Method | Signature | Description |
|
|
724
|
+
| -------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
|
725
|
+
| `enqueue` | `<TData, TResult>(queue, name, data, opts?) → Promise<Job<TData, TResult>>` | Add a single job |
|
|
726
|
+
| `enqueueBulk` | `<TData, TResult>(queue, jobs) → Promise<Job<TData, TResult>[]>` | Add up to 1 000 jobs in one Redis roundtrip |
|
|
727
|
+
| `getJob` | `<TData, TResult>(queue, jobId) → Promise<Job<TData, TResult> \| null>` | Get job by ID; `null` when it does not exist |
|
|
728
|
+
| `getJobs` | `<TData, TResult>(queue, status, start?, end?) → Promise<Job<TData, TResult>[]>` | Page jobs in one status (default `0`–`50`, at most 1 000 per call; `-1` is refused) |
|
|
729
|
+
| `getMetrics` | `(queue) → Promise<QueueMetrics>` | Uncached snapshot of job counts per status, with a collection timestamp |
|
|
730
|
+
| `pauseQueue` | `(queue) → Promise<void>` | Pause a queue |
|
|
731
|
+
| `resumeQueue` | `(queue) → Promise<void>` | Resume a paused queue |
|
|
732
|
+
| `cleanQueue` | `(queue, gracePeriodMs, limit, status?) → Promise<string[]>` | Remove up to `limit` jobs older than `gracePeriodMs` (`0` = no limit); returns removed ids |
|
|
733
|
+
| `upsertJobScheduler` | `<TData, TResult>(queue, id, repeat, template?) → Promise<Job<TData, TResult> \| undefined>` | Create or update a recurring Job Scheduler; returns the first scheduled job |
|
|
734
|
+
| `removeJobScheduler` | `(queue, id) → Promise<boolean>` | Remove a Job Scheduler by ID |
|
|
735
|
+
| `getJobSchedulers` | `(queue, start?, end?, asc?) → Promise<JobSchedulerJson[]>` | Page the schedulers registered on a queue |
|
|
736
|
+
| `getOrCreateQueue` | `<TData, TResult>(queue, overrides?) → Queue<TData, TResult>` | Return the cached raw `Queue` instance |
|
|
737
|
+
| `getCachedQueues` | `() → ReadonlyMap<string, Queue>` | Read-only view of every queue created so far |
|
|
738
|
+
|
|
739
|
+
### `WorkerRegistry`
|
|
740
|
+
|
|
741
|
+
Programmatic API to create and manage BullMQ workers at runtime.
|
|
742
|
+
|
|
743
|
+
```typescript
|
|
744
|
+
import path from 'node:path'
|
|
745
|
+
import { WorkerRegistry } from '@bymax-one/nest-queue'
|
|
746
|
+
import type { ProgrammaticWorkerConfig, SandboxedWorkerConfig } from '@bymax-one/nest-queue'
|
|
747
|
+
|
|
748
|
+
// Register an in-process worker
|
|
749
|
+
registry.register({
|
|
750
|
+
queueName: 'email',
|
|
751
|
+
handler: async (job) => {
|
|
752
|
+
/* … */
|
|
753
|
+
},
|
|
754
|
+
options: { concurrency: 4 },
|
|
755
|
+
})
|
|
756
|
+
|
|
757
|
+
// Register a sandboxed (out-of-process) processor
|
|
758
|
+
registry.registerSandboxed({
|
|
759
|
+
queueName: 'heavy-compute',
|
|
760
|
+
processorFile: path.resolve(__dirname, 'compute.processor.js'),
|
|
761
|
+
options: { concurrency: 2 },
|
|
762
|
+
})
|
|
908
763
|
```
|
|
909
764
|
|
|
910
|
-
|
|
911
|
-
of BullMQ, and pnpm blocks build scripts by default. **pnpm 10 warns and installs
|
|
912
|
-
anyway; pnpm 11 turns the same block into an error**, which is why this appears on
|
|
913
|
-
an upgrade with no change on your side. It is BullMQ's dependency, not this
|
|
914
|
-
package's: this package ships zero runtime dependencies.
|
|
765
|
+
### `FlowService` (opt-in)
|
|
915
766
|
|
|
916
|
-
|
|
767
|
+
Enable flows in the module options and inject the service:
|
|
917
768
|
|
|
918
|
-
```
|
|
919
|
-
|
|
769
|
+
```typescript
|
|
770
|
+
BymaxQueueModule.forRoot({
|
|
771
|
+
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
772
|
+
flows: { enabled: true },
|
|
773
|
+
})
|
|
774
|
+
|
|
775
|
+
// Then inject:
|
|
776
|
+
import { FlowService } from '@bymax-one/nest-queue'
|
|
777
|
+
import type { FlowJob, JobNode } from '@bymax-one/nest-queue'
|
|
778
|
+
|
|
779
|
+
const tree: JobNode = await this.flows.add({
|
|
780
|
+
name: 'process-order',
|
|
781
|
+
queueName: 'orders',
|
|
782
|
+
data: { orderId: '123' },
|
|
783
|
+
children: [
|
|
784
|
+
{ name: 'charge-payment', queueName: 'payments', data: {} },
|
|
785
|
+
{ name: 'send-confirmation', queueName: 'email', data: {} },
|
|
786
|
+
],
|
|
787
|
+
})
|
|
788
|
+
```
|
|
789
|
+
|
|
790
|
+
### `MetricsService` (opt-in)
|
|
791
|
+
|
|
792
|
+
Provide a per-queue snapshot with a configurable TTL cache:
|
|
793
|
+
|
|
794
|
+
```typescript
|
|
795
|
+
BymaxQueueModule.forRoot({
|
|
796
|
+
connection: { url: process.env.REDIS_URL ?? 'redis://localhost:6379' },
|
|
797
|
+
metrics: { enabled: true, cacheTtlMs: 5_000 },
|
|
798
|
+
})
|
|
799
|
+
|
|
800
|
+
import { MetricsService } from '@bymax-one/nest-queue'
|
|
801
|
+
import type { QueueMetrics } from '@bymax-one/nest-queue'
|
|
802
|
+
|
|
803
|
+
const metrics: QueueMetrics = await this.metricsService.get('email')
|
|
804
|
+
const all: readonly QueueMetrics[] = await this.metricsService.getAll()
|
|
920
805
|
```
|
|
921
806
|
|
|
922
|
-
|
|
807
|
+
### Other exports
|
|
808
|
+
|
|
809
|
+
| Export | What it is |
|
|
810
|
+
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
811
|
+
| `WorkerRegistry.unregister(queue)`, `list()` | Stop and remove a worker and its connection; list the queue names that have a worker |
|
|
812
|
+
| `FlowService.addBulk(flows)`, `getProducer()`, `close()` | Add several flows at once; the underlying BullMQ `FlowProducer`; close it (a no-op while flows are disabled) |
|
|
813
|
+
| `MetricsService.invalidate(queue?)` | Drop the cached snapshot of one queue, or of all of them |
|
|
814
|
+
| `QueueEventsRegistry` | The shared `QueueEvents` per queue (`getOrCreate`, `list`), used by `@OnQueueEvent` |
|
|
815
|
+
| `BYMAX_QUEUE_OPTIONS`, `BYMAX_QUEUE_RESOLVED_OPTIONS` | Injection tokens for the options as given and with the defaults applied |
|
|
816
|
+
| `BYMAX_QUEUE_REDIS_CLIENT`, `BYMAX_QUEUE_CONNECTION_MODE` | Injection tokens for the Queue-role ioredis client and for `'mode-a-byo'` or `'mode-b-owned'` |
|
|
817
|
+
| `DEFAULT_WORKER_CONCURRENCY`, `MAX_WORKER_CONCURRENCY` | The worker concurrency used when none is given, and its ceiling |
|
|
818
|
+
| `DEFAULT_JOB_OPTIONS`, `DEFAULT_METRICS_CACHE_TTL_MS`, `DEFAULT_SHUTDOWN_DRAIN_TIMEOUT_MS` | The defaults of the Configuration table |
|
|
819
|
+
| `QUEUE_ERROR_MESSAGES` | The default message of each error code |
|
|
820
|
+
| `BymaxQueueModuleOptions`, `BymaxQueueModuleAsyncOptions`, `BymaxQueueOptionsFactory`, `QueueConnectionConfig`, `QueueConnectionMode` | The option and connection types behind `forRoot` and `forRootAsync` |
|
|
821
|
+
| `ProcessorMetadata`, `ProcessHandlerMetadata`, `QueueEventListenerMetadata` | The reflection metadata the decorators write |
|
|
822
|
+
| `JobStatus`, `JOB_STATUS`, `BulkJob`, `SandboxedJob`, `WorkerEventName`, `QueueEventName`, `WorkerOptions` | Constants and types for the APIs above |
|
|
823
|
+
|
|
824
|
+
---
|
|
825
|
+
|
|
826
|
+
## 🚨 Error Code Catalog
|
|
827
|
+
|
|
828
|
+
Whatever **this library rejects** is a `QueueException` carrying a stable,
|
|
829
|
+
transport-independent `code`. Branch on the code, never on the HTTP status or the
|
|
830
|
+
message text — the code is the contract.
|
|
831
|
+
|
|
832
|
+
Whatever **fails underneath it** is not wrapped at all. A dropped connection, a
|
|
833
|
+
timeout or any other operational fault reaches you as the original error, with its
|
|
834
|
+
own type and message, rather than being relabelled as invalid input. So the catalog
|
|
835
|
+
below is a complete list of the library's own rejections, and deliberately not a
|
|
836
|
+
complete list of everything a call can throw.
|
|
837
|
+
|
|
838
|
+
```typescript
|
|
839
|
+
import { QueueException, QUEUE_ERROR_CODES } from '@bymax-one/nest-queue'
|
|
840
|
+
|
|
841
|
+
try {
|
|
842
|
+
await queue.enqueue('email', 'welcome', { userId })
|
|
843
|
+
} catch (err) {
|
|
844
|
+
if (err instanceof QueueException) {
|
|
845
|
+
const { code } = (err.getResponse() as { error: { code: string } }).error
|
|
846
|
+
if (code === QUEUE_ERROR_CODES.BULK_ENQUEUE_FAILED) {
|
|
847
|
+
/* … */
|
|
848
|
+
}
|
|
849
|
+
}
|
|
850
|
+
}
|
|
851
|
+
```
|
|
852
|
+
|
|
853
|
+
| Code | HTTP | When thrown |
|
|
854
|
+
| ---------------------------------------- | :--: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
855
|
+
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
|
|
856
|
+
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, Mode A's injected client is in an unusable `status`, or a Mode B client was refused by the server (`WRONGPASS`, `NOAUTH`) or stopped reconnecting during startup |
|
|
857
|
+
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
|
|
858
|
+
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` (`details` carries `timeoutMs` and the last error's `lastErrorCode`), or a `QueueService`/`FlowService` call got no answer within the connection's `commandTimeoutMs`, or `enqueue`/`enqueueBulk` within `enqueueTimeoutMs` (`details` carries `operation` and `timeoutMs`; an enqueue may still have been added) |
|
|
859
|
+
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
|
|
860
|
+
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
|
|
861
|
+
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
|
|
862
|
+
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
|
|
863
|
+
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
|
|
864
|
+
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
|
|
865
|
+
|
|
866
|
+
### Reserved codes
|
|
867
|
+
|
|
868
|
+
These are part of the published catalog and stable, but nothing currently throws them.
|
|
869
|
+
They are listed so a consumer writing an exhaustive `switch` over `QueueErrorCode`
|
|
870
|
+
compiles today and keeps compiling when a future release starts emitting them.
|
|
871
|
+
|
|
872
|
+
| Code | Why it is not thrown today |
|
|
873
|
+
| --------------------------------- | --------------------------------------------------------------------- |
|
|
874
|
+
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
|
|
875
|
+
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
|
|
876
|
+
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
|
|
877
|
+
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline force-closes workers rather than raising |
|
|
878
|
+
|
|
879
|
+
The same catalog is exported from `@bymax-one/nest-queue/shared` with no dependencies,
|
|
880
|
+
so a frontend can map codes to messages without importing the server surface.
|
|
881
|
+
|
|
882
|
+
---
|
|
883
|
+
|
|
884
|
+
## 🩺 Troubleshooting
|
|
885
|
+
|
|
886
|
+
### `ERR_PNPM_IGNORED_BUILDS: msgpackr-extract` on install (pnpm 11)
|
|
887
|
+
|
|
888
|
+
pnpm 11 blocks the build script of `msgpackr-extract`, a native optional dependency of BullMQ, and fails the
|
|
889
|
+
install. Approve it once with `pnpm approve-builds`, or in `pnpm-workspace.yaml`:
|
|
923
890
|
|
|
924
891
|
```yaml
|
|
925
892
|
onlyBuiltDependencies:
|
|
926
893
|
- msgpackr-extract
|
|
927
894
|
```
|
|
928
895
|
|
|
929
|
-
Skipping the build is also fine
|
|
930
|
-
implementation, so BullMQ still works, just without the native accelerator.
|
|
931
|
-
|
|
932
|
-
---
|
|
896
|
+
Skipping the build is also fine: `msgpackr` falls back to pure JavaScript, only without the native accelerator.
|
|
933
897
|
|
|
934
898
|
### `CONNECTION_REQUIRES_NULL_RETRIES`
|
|
935
899
|
|
|
@@ -943,7 +907,22 @@ that way. This error means one did not — measured with ioredis, a client creat
|
|
|
943
907
|
If you see it, the client you passed is not a plain ioredis client (a wrapper or a test
|
|
944
908
|
double whose `duplicate()` ignores its options). Pass the ioredis client itself, or let the
|
|
945
909
|
library open its own connection with Mode B. The producer keeps the client's own retry
|
|
946
|
-
policy
|
|
910
|
+
policy; with a finite `maxRetriesPerRequest` that is what lets it fail fast during an outage, and
|
|
911
|
+
`enqueue` is bounded by `enqueueTimeoutMs` whatever the client does.
|
|
912
|
+
|
|
913
|
+
### A handler never settles (the queue stops moving)
|
|
914
|
+
|
|
915
|
+
A handler whose promise never settles holds its concurrency slot for as long as the process lives, and BullMQ keeps renewing its lock, so the stalled-job check never reclaims it: with every slot held, the rest of the queue waits. Set `jobTimeoutMs` in the worker `options` to bound it. When the time passes, the job fails with a `queue.job_timeout` error and retries under its `attempts` and `backoff`, and the slot is free again. The handler receives the job's `AbortSignal` as its second argument; it aborts at that moment, or when BullMQ cancels the job (`worker.cancelJob()`, `worker.cancelAllJobs()`), and closing the worker does not abort it; pass it to whatever the handler awaits so the work stops with the job. A handler that ignores the signal keeps running in the background, and its retry can overlap it, so it must stay idempotent. There is no default, and sandboxed workers do not accept the option.
|
|
916
|
+
|
|
917
|
+
```typescript
|
|
918
|
+
@Processor('reports', { jobTimeoutMs: 60_000 })
|
|
919
|
+
export class ReportsProcessor {
|
|
920
|
+
@Process()
|
|
921
|
+
async build(job: Job, signal?: AbortSignal): Promise<void> {
|
|
922
|
+
await fetch('https://reports.internal/build', { method: 'POST', signal: signal ?? null })
|
|
923
|
+
}
|
|
924
|
+
}
|
|
925
|
+
```
|
|
947
926
|
|
|
948
927
|
### Jobs get stuck (not processed after restart)
|
|
949
928
|
|
|
@@ -964,23 +943,18 @@ do not pick them up:
|
|
|
964
943
|
- **NestJS only** — does not work with plain Express, standalone Fastify, or other frameworks.
|
|
965
944
|
- **Node.js only** — Deno and Bun are not supported.
|
|
966
945
|
- **BullMQ `^6.0` only** — the library targets BullMQ 6 (with `bullmq-otel ^2` for telemetry)
|
|
967
|
-
and does not support BullMQ 5
|
|
968
|
-
See the BullMQ version policy in [`CHANGELOG.md`](CHANGELOG.md).
|
|
946
|
+
and does not support BullMQ 5. See the BullMQ version policy in [`CHANGELOG.md`](CHANGELOG.md).
|
|
969
947
|
- **No Redis Cluster sharding** — BullMQ's cluster support is not exercised by this library.
|
|
970
948
|
- **No built-in DLQ helper** — use the `@OnWorkerEvent('failed')` pattern above.
|
|
971
949
|
- **No Prometheus metrics export** — `getMetrics()` returns snapshots; publish them
|
|
972
950
|
via `prom-client` in your own health endpoint.
|
|
973
951
|
|
|
974
|
-
For the full list of limitations and their alternatives, see
|
|
975
|
-
[`docs/technical_specification.md § 16`](docs/technical_specification.md).
|
|
976
|
-
|
|
977
952
|
---
|
|
978
953
|
|
|
979
954
|
## 🤝 Contributing
|
|
980
955
|
|
|
981
956
|
Pull requests are welcome. Please open an issue first for significant changes.
|
|
982
957
|
|
|
983
|
-
- Read [`docs/technical_specification.md`](docs/technical_specification.md) for architecture decisions.
|
|
984
958
|
- Run the full gate listed in [`CONTRIBUTING.md`](CONTRIBUTING.md) before opening a PR.
|
|
985
959
|
- Conventional Commits are enforced by `commitlint.config.cjs`.
|
|
986
960
|
|