@bymax-one/nest-queue 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +123 -0
- package/LICENSE +21 -0
- package/README.md +858 -0
- package/dist/server/index.cjs +1518 -0
- package/dist/server/index.d.cts +1386 -0
- package/dist/server/index.d.ts +1386 -0
- package/dist/server/index.mjs +1500 -0
- package/dist/shared/index.cjs +32 -0
- package/dist/shared/index.d.cts +132 -0
- package/dist/shared/index.d.ts +132 -0
- package/dist/shared/index.mjs +29 -0
- package/package.json +153 -0
package/README.md
ADDED
|
@@ -0,0 +1,858 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="https://img.shields.io/badge/%40bymax--one-nest--queue-000000?style=for-the-badge&logo=nestjs&logoColor=E0234E" alt="@bymax-one/nest-queue" />
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<h1 align="center">@bymax-one/nest-queue</h1>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
<strong>Typed BullMQ queues for NestJS</strong><br />
|
|
9
|
+
<sub>BullMQ 5 · Workers · Flows · Job Schedulers · Deduplication · OpenTelemetry · Graceful Shutdown · Zero Runtime Dependencies</sub>
|
|
10
|
+
</p>
|
|
11
|
+
|
|
12
|
+
<p align="center">
|
|
13
|
+
<a href="https://www.npmjs.com/package/@bymax-one/nest-queue"><img src="https://img.shields.io/npm/v/@bymax-one/nest-queue?style=flat-square&colorA=000000&colorB=000000" alt="npm version" /></a>
|
|
14
|
+
<a href="https://www.npmjs.com/package/@bymax-one/nest-queue"><img src="https://img.shields.io/npm/dm/@bymax-one/nest-queue?style=flat-square&colorA=000000&colorB=000000" alt="npm downloads" /></a>
|
|
15
|
+
<a href="https://github.com/bymaxone/nest-queue/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/bymaxone/nest-queue/ci.yml?branch=main&style=flat-square&colorA=000000&label=CI" alt="CI status" /></a>
|
|
16
|
+
<a href="https://github.com/bymaxone/nest-queue/actions/workflows/ci.yml"><img src="https://img.shields.io/badge/coverage-100%25-brightgreen?style=flat-square&colorA=000000" alt="coverage" /></a>
|
|
17
|
+
<a href="https://github.com/bymaxone/nest-queue/blob/main/docs/mutation_testing_results.md"><img src="https://img.shields.io/badge/mutation-99.06%25-brightgreen?style=flat-square&colorA=000000" alt="mutation score" /></a>
|
|
18
|
+
<a href="https://scorecard.dev/viewer/?uri=github.com/bymaxone/nest-queue"><img src="https://api.scorecard.dev/projects/github.com/bymaxone/nest-queue/badge?style=flat-square" alt="OpenSSF Scorecard" /></a>
|
|
19
|
+
<a href="https://github.com/bymaxone/nest-queue/blob/main/LICENSE"><img src="https://img.shields.io/github/license/bymaxone/nest-queue?style=flat-square&colorA=000000&colorB=000000" alt="license" /></a>
|
|
20
|
+
<a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/TypeScript-strict-3178C6?style=flat-square&logo=typescript&logoColor=white" alt="TypeScript" /></a>
|
|
21
|
+
<a href="https://nodejs.org/"><img src="https://img.shields.io/badge/Node.js-24%2B-339933?style=flat-square&logo=node.js&logoColor=white" alt="Node.js" /></a>
|
|
22
|
+
</p>
|
|
23
|
+
|
|
24
|
+
<p align="center">
|
|
25
|
+
<a href="https://github.com/bymaxone/nest-queue">GitHub</a> ·
|
|
26
|
+
<a href="https://github.com/bymaxone/nest-queue/issues">Issues</a> ·
|
|
27
|
+
<a href="#-quick-start">Quick Start</a> ·
|
|
28
|
+
<a href="#-api-reference">API Reference</a> ·
|
|
29
|
+
<a href="https://github.com/bymaxone/nest-queue/tree/main/examples/nest-queue-example">Example App</a>
|
|
30
|
+
</p>
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## ✨ Overview
|
|
35
|
+
|
|
36
|
+
`@bymax-one/nest-queue` is a NestJS dynamic module that wraps **BullMQ 5** behind a typed,
|
|
37
|
+
opinionated surface. Instead of wiring `ioredis` connections, per-queue registration, and
|
|
38
|
+
`maxRetriesPerRequest` policy by hand, you install one library and get typed producers,
|
|
39
|
+
decorator-based workers, flows, recurring **Job Schedulers**, native deduplication,
|
|
40
|
+
OpenTelemetry passthrough, and a tested bounded graceful-shutdown protocol.
|
|
41
|
+
|
|
42
|
+
The library has **zero direct dependencies** — `bullmq`, `ioredis`, and `@nestjs/*` arrive as
|
|
43
|
+
peer dependencies, so you control exact versions and the supply-chain surface stays minimal.
|
|
44
|
+
|
|
45
|
+
### Why nest-queue?
|
|
46
|
+
|
|
47
|
+
- **One setup, every queue.** A single `forRoot`/`forRootAsync`; queues are created on demand by
|
|
48
|
+
name — no `registerQueue()` per queue.
|
|
49
|
+
- **Correct connections by default.** Dual-mode (bring-your-own client or lib-owned) with the
|
|
50
|
+
per-role `maxRetriesPerRequest` policy applied automatically — the Queue connection keeps
|
|
51
|
+
default retries (fail fast), only the duplicated Worker/QueueEvents connections force `null`.
|
|
52
|
+
- **Current BullMQ API only.** Recurring jobs use Job Schedulers (`upsertJobScheduler`), never the
|
|
53
|
+
removed `addRepeatable`. Cron is validated by BullMQ's own parser, never a hand-rolled regex.
|
|
54
|
+
- **Safe shutdown.** A bounded-drain protocol with at-least-once semantics, force-close, and
|
|
55
|
+
stalled-job accounting — verified against a real Redis (Testcontainers) E2E suite.
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## 🔥 Features
|
|
60
|
+
|
|
61
|
+
### 📤 Producing
|
|
62
|
+
|
|
63
|
+
- ✅ **Typed producer API** — `enqueue<TData, TResult>()` threads both generics through to the returned `Job`, so `job.data` and `job.returnvalue` need no cast
|
|
64
|
+
- ✅ **Bounded bulk insert** — `enqueueBulk` adds up to 1 000 jobs in a single Redis roundtrip and rejects a larger batch outright, rather than letting one call inflate the heap
|
|
65
|
+
- ✅ **Native deduplication** — simple, throttle, debounce and keep-last-if-active, passed straight through to BullMQ; the library writes no deduplication logic of its own
|
|
66
|
+
- ✅ **Job Schedulers** — cron (5-field, or 6-field with seconds) and fixed intervals through `upsertJobScheduler`; the `addRepeatable` API removed in BullMQ v6 is never called
|
|
67
|
+
- ✅ **Flows** — opt-in `FlowService` over BullMQ's `FlowProducer`, with fail-parent semantics preserved
|
|
68
|
+
|
|
69
|
+
### ⚙️ Consuming
|
|
70
|
+
|
|
71
|
+
- ✅ **Decorator workers** — `@Processor`, `@Process`, `@OnWorkerEvent` and `@OnQueueEvent`, discovered automatically from your providers
|
|
72
|
+
- ✅ **Programmatic registry** — `WorkerRegistry.register()` for workers built at runtime, and `registerSandboxed()` for file-based out-of-process processors
|
|
73
|
+
- ✅ **Progress reporting** — `job.updateProgress()` with the matching worker event wired through to your handler
|
|
74
|
+
|
|
75
|
+
### 🔌 Connections & Shutdown
|
|
76
|
+
|
|
77
|
+
- ✅ **Dual-mode connection** — bring your own `ioredis` client, or let the library open its own from a URL or an options object
|
|
78
|
+
- ✅ **Per-role retry policy** — `maxRetriesPerRequest: null` reaches only the duplicated Worker and QueueEvents connections; the producer keeps ioredis' defaults, so a Redis outage fails fast instead of hanging a request
|
|
79
|
+
- ✅ **One namespace, four objects** — the configured `prefix` reaches `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer`, so producers and consumers can never end up on separate keyspaces
|
|
80
|
+
- ✅ **Bounded graceful shutdown** — a single ordered sequence drains every worker under a deadline and then force-closes; delivery is at-least-once by design, so handlers must be idempotent
|
|
81
|
+
- ✅ **No unlistened emitter** — every `Queue`, `Worker`, `QueueEvents` and `FlowProducer` the library creates carries a fallback `error` listener, so a transient Redis fault cannot become an uncaught exception in your process; your own `@OnWorkerEvent('error')` or `@OnQueueEvent('error')` still runs alongside it
|
|
82
|
+
|
|
83
|
+
### 🧩 Developer Experience
|
|
84
|
+
|
|
85
|
+
- ✅ **Zero runtime dependencies** — BullMQ, ioredis and `@nestjs/*` all arrive as peer dependencies, so you pin the versions and the supply-chain surface stays yours
|
|
86
|
+
- ✅ **Two subpaths** — the server surface, and `./shared` carrying types and constants that pull in nothing at all
|
|
87
|
+
- ✅ **Stable error catalog** — namespaced codes exported from both subpaths, so a frontend maps a `code` without importing the server
|
|
88
|
+
- ✅ **Dual-format output** — ESM + CJS with declarations for each format, verified against the packed tarball on every PR
|
|
89
|
+
- ✅ **Opt-in observability** — OpenTelemetry passthrough via `bullmq-otel`, plus a `MetricsService` with a configurable TTL cache
|
|
90
|
+
- ✅ **Typed end to end** — TypeScript `strict` with `exactOptionalPropertyTypes` and `noUncheckedIndexedAccess`; zero `any`, no suppression comments
|
|
91
|
+
|
|
92
|
+
---
|
|
93
|
+
|
|
94
|
+
## 📦 Subpath Exports
|
|
95
|
+
|
|
96
|
+
`./shared` exists so a browser or edge bundle can import the job-status union and the
|
|
97
|
+
error-code catalog without pulling in `bullmq`, `ioredis`, or `@nestjs/*`. Nothing under
|
|
98
|
+
that subpath imports a runtime dependency.
|
|
99
|
+
|
|
100
|
+
| Subpath | Import | Purpose | Dependencies |
|
|
101
|
+
| -------------- | ------------------------------ | --------------------------------------------------------- | :-----------------------: |
|
|
102
|
+
| **`.`** | `@bymax-one/nest-queue` | Full server surface: module, services, decorators, types | NestJS · BullMQ · ioredis |
|
|
103
|
+
| **`./shared`** | `@bymax-one/nest-queue/shared` | Job-status union, error codes, metric and scheduler types | none |
|
|
104
|
+
|
|
105
|
+
```
|
|
106
|
+
@bymax-one/nest-queue (server — NestJS + BullMQ + ioredis)
|
|
107
|
+
│
|
|
108
|
+
└── re-exports ──▶ @bymax-one/nest-queue/shared (zero dependencies)
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Both subpaths ship ESM **and** CommonJS with declarations for each format, so a
|
|
112
|
+
`require()` consumer receives CommonJS declarations rather than ESM ones. The
|
|
113
|
+
`pnpm check:exports` gate verifies this against the packed tarball.
|
|
114
|
+
|
|
115
|
+
### Peer dependency matrix
|
|
116
|
+
|
|
117
|
+
| Subpath | Required peers |
|
|
118
|
+
| ------------ | ----------------------------------------------------------------------------------------------- |
|
|
119
|
+
| `.` (server) | `@nestjs/common ^11`, `@nestjs/core ^11`, `bullmq ^5.16`, `ioredis ^5`, `reflect-metadata ^0.2` |
|
|
120
|
+
| `./shared` | None |
|
|
121
|
+
| Optional | `bullmq-otel ^1` (OpenTelemetry) |
|
|
122
|
+
|
|
123
|
+
---
|
|
124
|
+
|
|
125
|
+
> [!TIP]
|
|
126
|
+
> A runnable reference application lives in
|
|
127
|
+
> [`examples/nest-queue-example`](https://github.com/bymaxone/nest-queue/tree/main/examples/nest-queue-example) —
|
|
128
|
+
> a NestJS app that consumes this library and exercises producers, decorated
|
|
129
|
+
> processors, and a health endpoint against a real Redis.
|
|
130
|
+
|
|
131
|
+
---
|
|
132
|
+
|
|
133
|
+
## 🚀 Quick Start
|
|
134
|
+
|
|
135
|
+
```bash
|
|
136
|
+
pnpm add @bymax-one/nest-queue bullmq ioredis
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Wire the module in your root `AppModule`:
|
|
140
|
+
|
|
141
|
+
```typescript
|
|
142
|
+
import { Module } from '@nestjs/common'
|
|
143
|
+
import { BymaxQueueModule } from '@bymax-one/nest-queue'
|
|
144
|
+
|
|
145
|
+
@Module({
|
|
146
|
+
imports: [
|
|
147
|
+
BymaxQueueModule.forRoot({
|
|
148
|
+
connection: { url: process.env.REDIS_URL },
|
|
149
|
+
isGlobal: true,
|
|
150
|
+
}),
|
|
151
|
+
],
|
|
152
|
+
})
|
|
153
|
+
export class AppModule {}
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Enqueue a job from any service:
|
|
157
|
+
|
|
158
|
+
```typescript
|
|
159
|
+
import { Injectable } from '@nestjs/common'
|
|
160
|
+
import { QueueService } from '@bymax-one/nest-queue'
|
|
161
|
+
|
|
162
|
+
@Injectable()
|
|
163
|
+
export class EmailService {
|
|
164
|
+
constructor(private readonly queue: QueueService) {}
|
|
165
|
+
|
|
166
|
+
async sendWelcome(userId: string): Promise<void> {
|
|
167
|
+
await this.queue.enqueue<{ userId: string }, void>('email', 'welcome', { userId })
|
|
168
|
+
}
|
|
169
|
+
}
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Process jobs with a decorated handler:
|
|
173
|
+
|
|
174
|
+
```typescript
|
|
175
|
+
import { Injectable } from '@nestjs/common'
|
|
176
|
+
import { Processor, Process } from '@bymax-one/nest-queue'
|
|
177
|
+
import type { Job } from '@bymax-one/nest-queue'
|
|
178
|
+
|
|
179
|
+
@Injectable()
|
|
180
|
+
@Processor('email')
|
|
181
|
+
export class EmailProcessor {
|
|
182
|
+
@Process('welcome')
|
|
183
|
+
async handleWelcome(job: Job<{ userId: string }, void>): Promise<void> {
|
|
184
|
+
// handle job.data.userId
|
|
185
|
+
}
|
|
186
|
+
}
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Register the processor in your module alongside `BymaxQueueModule`:
|
|
190
|
+
|
|
191
|
+
```typescript
|
|
192
|
+
@Module({
|
|
193
|
+
providers: [EmailProcessor],
|
|
194
|
+
})
|
|
195
|
+
export class EmailModule {}
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
---
|
|
199
|
+
|
|
200
|
+
## ⚙️ Configuration
|
|
201
|
+
|
|
202
|
+
Every option passed to `forRoot` / `forRootAsync`. Only `connection` is required.
|
|
203
|
+
|
|
204
|
+
| Option | Type | Default | Description |
|
|
205
|
+
| -------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
206
|
+
| `connection` | `{ client } \| { url, options? } \| { options }` | — | **Required.** Mode A supplies an existing `ioredis` client; Mode B lets the library open its own. The three arms are mutually exclusive. |
|
|
207
|
+
| `prefix` | `string` | `'bull'` | Redis key namespace. Applied to the `Queue`, `Worker`, `QueueEvents` **and** `FlowProducer` — see [Architecture](#-architecture). |
|
|
208
|
+
| `defaultJobOptions` | `JobsOptions` | `attempts: 3` · exponential backoff `2000` ms · `removeOnComplete: { age: 24 h, count: 1000 }` · `removeOnFail: { age: 7 d, count: 5000 }` | Merged into every job enqueued through the service; per-job options win. The bounded `removeOn*` caps are what keep Redis from growing without limit under a failure storm. |
|
|
209
|
+
| `queueOptions` | `Partial<QueueOptions>` | `{}` | Extra BullMQ `Queue` options. `connection`, `defaultJobOptions` and `prefix` are owned by the library and cannot be set here. |
|
|
210
|
+
| `flows.enabled` | `boolean` | `false` | Registers `FlowService`. Injecting it while disabled throws `FLOW_DISABLED`. |
|
|
211
|
+
| `metrics.enabled` | `boolean` | `false` | Registers `MetricsService`. Injecting it while disabled throws `METRICS_DISABLED`. |
|
|
212
|
+
| `metrics.cacheTtlMs` | `number` | `5_000` | TTL of the metrics snapshot cache. `0` disables caching. Must be `>= 0`. |
|
|
213
|
+
| `telemetry` | `Telemetry` | — | A BullMQ `Telemetry` instance (typically `new BullMQOtel(...)` from the optional `bullmq-otel` peer), attached to every `Queue` and `Worker`. |
|
|
214
|
+
| `shutdown.drainTimeoutMs` | `number` | `30_000` | Upper bound on the graceful drain before workers are force-closed. Must be `> 0`. |
|
|
215
|
+
| `shutdown.drainOnShutdown` | `boolean` | `false` | Wait for the queue to drain, not just for in-flight jobs to finish. |
|
|
216
|
+
| `connectionReadyTimeoutMs` | `number` | `10_000` | Mode B only: how long to wait for Redis `ready` before failing with `CONNECTION_TIMEOUT`. |
|
|
217
|
+
| `isGlobal` | `boolean` | `true` | Register the module globally, so `QueueService` is injectable without re-importing. |
|
|
218
|
+
|
|
219
|
+
Invalid options are rejected at bootstrap with `QueueException(INVALID_OPTIONS)` — never
|
|
220
|
+
silently corrected. A misconfigured queue fails when the application starts, not when the
|
|
221
|
+
first job is enqueued.
|
|
222
|
+
|
|
223
|
+
---
|
|
224
|
+
|
|
225
|
+
## 🔌 Mode A — Bring Your Own Connection (recommended with `@bymax-one/nest-cache`)
|
|
226
|
+
|
|
227
|
+
Mode A accepts an existing `ioredis` client injected from your cache or connection layer.
|
|
228
|
+
Worker and QueueEvents connections are duplicated from this client with
|
|
229
|
+
`maxRetriesPerRequest: null` applied automatically; the Queue connection uses ioredis defaults.
|
|
230
|
+
|
|
231
|
+
```typescript
|
|
232
|
+
import { BymaxQueueModule } from '@bymax-one/nest-queue'
|
|
233
|
+
import { CACHE_REDIS_CLIENT } from '@bymax-one/nest-cache'
|
|
234
|
+
import type { Redis } from 'ioredis'
|
|
235
|
+
|
|
236
|
+
BymaxQueueModule.forRootAsync({
|
|
237
|
+
inject: [CACHE_REDIS_CLIENT],
|
|
238
|
+
useFactory: (queueRedis: Redis) => ({
|
|
239
|
+
connection: { client: queueRedis },
|
|
240
|
+
isGlobal: true,
|
|
241
|
+
}),
|
|
242
|
+
})
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
Mode A is the recommended setup when you already have `@bymax-one/nest-cache` in the app.
|
|
246
|
+
One client is shared; the lib duplicates it per role and applies the correct retry policy
|
|
247
|
+
without touching your cache connection.
|
|
248
|
+
|
|
249
|
+
---
|
|
250
|
+
|
|
251
|
+
## 🔌 Mode B — Lib-Owned Connection
|
|
252
|
+
|
|
253
|
+
Mode B lets the library own the Redis connections. Pass either a URL or raw ioredis options:
|
|
254
|
+
|
|
255
|
+
```typescript
|
|
256
|
+
// URL form
|
|
257
|
+
BymaxQueueModule.forRoot({
|
|
258
|
+
connection: { url: process.env.REDIS_URL },
|
|
259
|
+
isGlobal: true,
|
|
260
|
+
})
|
|
261
|
+
|
|
262
|
+
// Options form
|
|
263
|
+
BymaxQueueModule.forRoot({
|
|
264
|
+
connection: {
|
|
265
|
+
options: {
|
|
266
|
+
host: process.env.REDIS_HOST,
|
|
267
|
+
port: Number(process.env.REDIS_PORT),
|
|
268
|
+
password: process.env.REDIS_PASSWORD,
|
|
269
|
+
tls: {},
|
|
270
|
+
},
|
|
271
|
+
},
|
|
272
|
+
isGlobal: true,
|
|
273
|
+
})
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
The library opens one connection for queues and one per worker/QueueEvents listener, each
|
|
277
|
+
with `maxRetriesPerRequest: null` where BullMQ requires it.
|
|
278
|
+
|
|
279
|
+
---
|
|
280
|
+
|
|
281
|
+
## 📖 API Reference
|
|
282
|
+
|
|
283
|
+
### `QueueService`
|
|
284
|
+
|
|
285
|
+
The central interaction point with BullMQ queues.
|
|
286
|
+
|
|
287
|
+
```typescript
|
|
288
|
+
import { QueueService } from '@bymax-one/nest-queue'
|
|
289
|
+
import type {
|
|
290
|
+
Job,
|
|
291
|
+
JobsOptions,
|
|
292
|
+
JobSchedulerRepeatOptions,
|
|
293
|
+
QueueMetrics,
|
|
294
|
+
} from '@bymax-one/nest-queue'
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
| Method | Signature | Description |
|
|
298
|
+
| -------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
|
299
|
+
| `enqueue` | `<TData, TResult>(queue, name, data, opts?) → Promise<Job<TData, TResult>>` | Add a single job |
|
|
300
|
+
| `enqueueBulk` | `<TData, TResult>(queue, jobs) → Promise<Job<TData, TResult>[]>` | Add up to 1 000 jobs in one Redis roundtrip |
|
|
301
|
+
| `getJob` | `<TData, TResult>(queue, jobId) → Promise<Job<TData, TResult> \| null>` | Get job by ID; `null` when it does not exist |
|
|
302
|
+
| `getJobs` | `<TData, TResult>(queue, status, start?, end?) → Promise<Job<TData, TResult>[]>` | Page jobs in one status (default `0`–`50`) |
|
|
303
|
+
| `getMetrics` | `(queue) → Promise<QueueMetrics>` | Uncached snapshot of job counts per status, with a collection timestamp |
|
|
304
|
+
| `pauseQueue` | `(queue) → Promise<void>` | Pause a queue |
|
|
305
|
+
| `resumeQueue` | `(queue) → Promise<void>` | Resume a paused queue |
|
|
306
|
+
| `cleanQueue` | `(queue, graceMs, limit, status?) → Promise<string[]>` | Remove up to `limit` jobs older than `graceMs` (`0` = no limit); returns removed ids |
|
|
307
|
+
| `upsertJobScheduler` | `<TData, TResult>(queue, id, repeat, template?) → Promise<Job<TData, TResult> \| undefined>` | Create or update a recurring Job Scheduler; returns the first scheduled job |
|
|
308
|
+
| `removeJobScheduler` | `(queue, id) → Promise<boolean>` | Remove a Job Scheduler by ID |
|
|
309
|
+
| `getJobSchedulers` | `(queue, start?, end?, asc?) → Promise<JobSchedulerJson[]>` | Page the schedulers registered on a queue |
|
|
310
|
+
| `getOrCreateQueue` | `<TData, TResult>(queue, overrides?) → Queue<TData, TResult>` | Return the cached raw `Queue` instance |
|
|
311
|
+
| `getCachedQueues` | `() → ReadonlyMap<string, Queue>` | Read-only view of every queue created so far |
|
|
312
|
+
|
|
313
|
+
### `WorkerRegistry`
|
|
314
|
+
|
|
315
|
+
Programmatic API to create and manage BullMQ workers at runtime.
|
|
316
|
+
|
|
317
|
+
```typescript
|
|
318
|
+
import { WorkerRegistry } from '@bymax-one/nest-queue'
|
|
319
|
+
import type { ProgrammaticWorkerConfig, SandboxedWorkerConfig } from '@bymax-one/nest-queue'
|
|
320
|
+
|
|
321
|
+
// Register an in-process worker
|
|
322
|
+
registry.register({
|
|
323
|
+
queueName: 'email',
|
|
324
|
+
processor: async (job) => {
|
|
325
|
+
/* … */
|
|
326
|
+
},
|
|
327
|
+
options: { concurrency: 4 },
|
|
328
|
+
})
|
|
329
|
+
|
|
330
|
+
// Register a sandboxed (out-of-process) processor
|
|
331
|
+
registry.registerSandboxed({
|
|
332
|
+
queueName: 'heavy-compute',
|
|
333
|
+
processorFile: path.resolve(__dirname, 'compute.processor.js'),
|
|
334
|
+
options: { concurrency: 2 },
|
|
335
|
+
})
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
### `FlowService` (opt-in)
|
|
339
|
+
|
|
340
|
+
Enable flows in the module options and inject the service:
|
|
341
|
+
|
|
342
|
+
```typescript
|
|
343
|
+
BymaxQueueModule.forRoot({
|
|
344
|
+
connection: { url: process.env.REDIS_URL },
|
|
345
|
+
flows: { enabled: true },
|
|
346
|
+
})
|
|
347
|
+
|
|
348
|
+
// Then inject:
|
|
349
|
+
import { FlowService } from '@bymax-one/nest-queue'
|
|
350
|
+
import type { FlowJob, JobNode } from '@bymax-one/nest-queue'
|
|
351
|
+
|
|
352
|
+
const tree: JobNode = await this.flows.add({
|
|
353
|
+
name: 'process-order',
|
|
354
|
+
queueName: 'orders',
|
|
355
|
+
data: { orderId: '123' },
|
|
356
|
+
children: [
|
|
357
|
+
{ name: 'charge-payment', queueName: 'payments', data: {} },
|
|
358
|
+
{ name: 'send-confirmation', queueName: 'email', data: {} },
|
|
359
|
+
],
|
|
360
|
+
})
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
### `MetricsService` (opt-in)
|
|
364
|
+
|
|
365
|
+
Provide a per-queue snapshot with a configurable TTL cache:
|
|
366
|
+
|
|
367
|
+
```typescript
|
|
368
|
+
BymaxQueueModule.forRoot({
|
|
369
|
+
connection: { url: process.env.REDIS_URL },
|
|
370
|
+
metrics: { enabled: true, cacheTtlMs: 5_000 },
|
|
371
|
+
})
|
|
372
|
+
|
|
373
|
+
import { MetricsService } from '@bymax-one/nest-queue'
|
|
374
|
+
import type { QueueMetrics } from '@bymax-one/nest-queue'
|
|
375
|
+
|
|
376
|
+
const metrics: QueueMetrics = await this.metricsService.getMetrics('email')
|
|
377
|
+
const all = await this.metricsService.getAll(['email', 'notifications'])
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
---
|
|
381
|
+
|
|
382
|
+
## 🚨 Error Code Catalog
|
|
383
|
+
|
|
384
|
+
Whatever **this library rejects** is a `QueueException` carrying a stable,
|
|
385
|
+
transport-independent `code`. Branch on the code, never on the HTTP status or the
|
|
386
|
+
message text — the code is the contract.
|
|
387
|
+
|
|
388
|
+
Whatever **fails underneath it** is not wrapped at all. A dropped connection, a
|
|
389
|
+
timeout or any other operational fault reaches you as the original error, with its
|
|
390
|
+
own type and message, rather than being relabelled as invalid input. So the catalog
|
|
391
|
+
below is a complete list of the library's own rejections, and deliberately not a
|
|
392
|
+
complete list of everything a call can throw.
|
|
393
|
+
|
|
394
|
+
```typescript
|
|
395
|
+
import { QueueException, QUEUE_ERROR_CODES } from '@bymax-one/nest-queue'
|
|
396
|
+
|
|
397
|
+
try {
|
|
398
|
+
await queue.enqueue('email', 'welcome', { userId })
|
|
399
|
+
} catch (err) {
|
|
400
|
+
if (err instanceof QueueException) {
|
|
401
|
+
const { code } = (err.getResponse() as { error: { code: string } }).error
|
|
402
|
+
if (code === QUEUE_ERROR_CODES.BULK_ENQUEUE_FAILED) {
|
|
403
|
+
/* … */
|
|
404
|
+
}
|
|
405
|
+
}
|
|
406
|
+
}
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
| Code | HTTP | When thrown |
|
|
410
|
+
| ---------------------------------------- | :--: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
411
|
+
| `queue.invalid_options` | 500 | Module options failed bootstrap validation — missing `connection`, mutually exclusive connection arms, non-positive `shutdown.drainTimeoutMs`, or negative `metrics.cacheTtlMs` |
|
|
412
|
+
| `queue.connection_invalid` | 500 | The resolver has no usable client: not yet initialized, or Mode A's injected client is in an unusable `status` |
|
|
413
|
+
| `queue.connection_requires_null_retries` | 500 | A consumer connection could not be coerced to `maxRetriesPerRequest: null` |
|
|
414
|
+
| `queue.connection_timeout` | 500 | Mode B: Redis did not reach `ready` within `connectionReadyTimeoutMs` |
|
|
415
|
+
| `queue.invalid_repeat_options` | 400 | A Job Scheduler schedule was rejected — invalid cron pattern, or neither `pattern` nor `every` |
|
|
416
|
+
| `queue.duplicate_processor` | 500 | Two `@Processor` classes claim the same queue name |
|
|
417
|
+
| `queue.flow_disabled` | 503 | `FlowService` used without `flows.enabled` |
|
|
418
|
+
| `queue.metrics_disabled` | 503 | `MetricsService` used without `metrics.enabled` |
|
|
419
|
+
| `queue.bulk_enqueue_failed` | 500 | `enqueueBulk` exceeded the 1 000-job cap, or the underlying `addBulk` failed |
|
|
420
|
+
| `queue.worker_registration_failed` | 500 | A worker could not be constructed, started, or closed |
|
|
421
|
+
|
|
422
|
+
### Reserved codes
|
|
423
|
+
|
|
424
|
+
These are part of the published catalog and stable, but nothing currently throws them.
|
|
425
|
+
They are listed so a consumer writing an exhaustive `switch` over `QueueErrorCode`
|
|
426
|
+
compiles today and keeps compiling when a future release starts emitting them.
|
|
427
|
+
|
|
428
|
+
| Code | Why it is not thrown today |
|
|
429
|
+
| --------------------------------- | --------------------------------------------------------------------- |
|
|
430
|
+
| `queue.queue_not_found` | Queues are created on demand by name, so a lookup never misses |
|
|
431
|
+
| `queue.job_not_found` | `getJob` resolves to `null` rather than throwing |
|
|
432
|
+
| `queue.invalid_job_data` | Payloads are opaque; the library does not validate them |
|
|
433
|
+
| `queue.shutdown_timeout_exceeded` | Exceeding the drain deadline force-closes workers rather than raising |
|
|
434
|
+
|
|
435
|
+
The same catalog is exported from `@bymax-one/nest-queue/shared` with no dependencies,
|
|
436
|
+
so a frontend can map codes to messages without importing the server surface.
|
|
437
|
+
|
|
438
|
+
---
|
|
439
|
+
|
|
440
|
+
## 🎀 Decorators
|
|
441
|
+
|
|
442
|
+
### `@Processor(queueName)`
|
|
443
|
+
|
|
444
|
+
Marks a class as a worker bound to `queueName`. The class must be an `@Injectable()` registered as a NestJS provider.
|
|
445
|
+
|
|
446
|
+
### `@Process(jobName?)`
|
|
447
|
+
|
|
448
|
+
Method decorator for the handler. If `jobName` is omitted, the method handles all jobs on the queue. If provided, only jobs with that `name` are dispatched here.
|
|
449
|
+
|
|
450
|
+
```typescript
|
|
451
|
+
@Process()
|
|
452
|
+
async handleAll(job: Job): Promise<void> { /* … */ }
|
|
453
|
+
|
|
454
|
+
@Process('send-email')
|
|
455
|
+
async handleSendEmail(job: Job<EmailPayload>): Promise<void> { /* … */ }
|
|
456
|
+
```
|
|
457
|
+
|
|
458
|
+
### `@OnWorkerEvent(eventName)`
|
|
459
|
+
|
|
460
|
+
Listens to BullMQ worker-local events (`'completed'`, `'failed'`, `'progress'`, `'active'`, `'stalled'`, …).
|
|
461
|
+
|
|
462
|
+
```typescript
|
|
463
|
+
@OnWorkerEvent('failed')
|
|
464
|
+
onFailed(job: Job | undefined, err: Error): void {
|
|
465
|
+
console.error(`Job ${job?.id} failed:`, err.message)
|
|
466
|
+
}
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
### `@OnQueueEvent(eventName)`
|
|
470
|
+
|
|
471
|
+
Listens to BullMQ `QueueEvents` (global, cross-process events). Requires `queueEvents: { enabled: true }` in module options.
|
|
472
|
+
|
|
473
|
+
```typescript
|
|
474
|
+
@OnQueueEvent('completed')
|
|
475
|
+
onCompleted(jobId: string, returnValue: string): void {
|
|
476
|
+
console.log(`Job ${jobId} completed`)
|
|
477
|
+
}
|
|
478
|
+
```
|
|
479
|
+
|
|
480
|
+
---
|
|
481
|
+
|
|
482
|
+
## ⏰ Job Schedulers
|
|
483
|
+
|
|
484
|
+
Job Schedulers are the current recurring-jobs API in BullMQ (superseding the removed `addRepeatable` API). Use `upsertJobScheduler` to create or update a scheduler:
|
|
485
|
+
|
|
486
|
+
```typescript
|
|
487
|
+
// Cron-based (every day at 9 AM)
|
|
488
|
+
await this.queue.upsertJobScheduler(
|
|
489
|
+
'notifications',
|
|
490
|
+
'daily-digest',
|
|
491
|
+
{ pattern: '0 9 * * *' },
|
|
492
|
+
{ name: 'send-digest', data: { type: 'daily' } },
|
|
493
|
+
)
|
|
494
|
+
|
|
495
|
+
// Interval-based (every 5 minutes)
|
|
496
|
+
await this.queue.upsertJobScheduler(
|
|
497
|
+
'health-check',
|
|
498
|
+
'ping',
|
|
499
|
+
{ every: 5 * 60 * 1000 },
|
|
500
|
+
{ name: 'ping', data: {} },
|
|
501
|
+
)
|
|
502
|
+
|
|
503
|
+
// Remove
|
|
504
|
+
await this.queue.removeJobScheduler('notifications', 'daily-digest')
|
|
505
|
+
|
|
506
|
+
// List
|
|
507
|
+
const schedulers = await this.queue.getJobSchedulers('notifications')
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
Cron-with-seconds (6-field patterns) is supported. Patterns are validated by BullMQ's
|
|
511
|
+
own `cron-parser`, not a hand-rolled regex.
|
|
512
|
+
|
|
513
|
+
---
|
|
514
|
+
|
|
515
|
+
## 🔁 Deduplication & Telemetry
|
|
516
|
+
|
|
517
|
+
### Deduplication
|
|
518
|
+
|
|
519
|
+
Pass the `deduplication` option directly on `enqueue` — the library delegates the behavior entirely to BullMQ:
|
|
520
|
+
|
|
521
|
+
```typescript
|
|
522
|
+
// Simple: collapse duplicates until the first in-flight job completes
|
|
523
|
+
await this.queue.enqueue(
|
|
524
|
+
'email',
|
|
525
|
+
'welcome',
|
|
526
|
+
{ userId },
|
|
527
|
+
{
|
|
528
|
+
deduplication: { id: `welcome:${userId}` },
|
|
529
|
+
},
|
|
530
|
+
)
|
|
531
|
+
|
|
532
|
+
// Throttle: ignore duplicates for 30 seconds
|
|
533
|
+
await this.queue.enqueue(
|
|
534
|
+
'email',
|
|
535
|
+
'welcome',
|
|
536
|
+
{ userId },
|
|
537
|
+
{
|
|
538
|
+
deduplication: { id: `welcome:${userId}`, ttl: 30_000 },
|
|
539
|
+
},
|
|
540
|
+
)
|
|
541
|
+
|
|
542
|
+
// Debounce: keep only the latest; each duplicate resets the TTL
|
|
543
|
+
await this.queue.enqueue(
|
|
544
|
+
'email',
|
|
545
|
+
'welcome',
|
|
546
|
+
{ userId },
|
|
547
|
+
{
|
|
548
|
+
deduplication: { id: `welcome:${userId}`, ttl: 5_000, extend: true, replace: true },
|
|
549
|
+
},
|
|
550
|
+
)
|
|
551
|
+
```
|
|
552
|
+
|
|
553
|
+
### OpenTelemetry
|
|
554
|
+
|
|
555
|
+
Pass a `bullmq-otel` telemetry instance (optional peer dep) to trace spans from `enqueue` through the handler:
|
|
556
|
+
|
|
557
|
+
```typescript
|
|
558
|
+
import { BullMQOtel } from 'bullmq-otel'
|
|
559
|
+
|
|
560
|
+
BymaxQueueModule.forRoot({
|
|
561
|
+
connection: { url: process.env.REDIS_URL },
|
|
562
|
+
telemetry: new BullMQOtel('my-service', tracer),
|
|
563
|
+
})
|
|
564
|
+
```
|
|
565
|
+
|
|
566
|
+
---
|
|
567
|
+
|
|
568
|
+
## 🛑 Graceful Shutdown
|
|
569
|
+
|
|
570
|
+
`QueueLifecycle` is registered and invoked automatically via NestJS `onApplicationShutdown`.
|
|
571
|
+
It implements a **bounded drain protocol** with at-least-once semantics:
|
|
572
|
+
|
|
573
|
+
1. All workers are closed with a grace period (`drainTimeoutMs`, default 30 s).
|
|
574
|
+
2. A `Promise.race` resolves either when the worker drains normally or when the
|
|
575
|
+
timeout fires — whichever comes first.
|
|
576
|
+
3. On timeout, the worker is force-closed (`worker.close(true)`), preserving stalled
|
|
577
|
+
jobs for reprocessing by the next instance.
|
|
578
|
+
4. In Mode B, Redis connections are disconnected after all workers are closed.
|
|
579
|
+
|
|
580
|
+
**Handlers must be idempotent.** Use `jobId`, `deduplication`, or application-level
|
|
581
|
+
idempotency keys to guard against reprocessing a job that was partially handled before
|
|
582
|
+
a crash or force-close.
|
|
583
|
+
|
|
584
|
+
**DLQ pattern.** When a job exhausts its `attempts`, move it to a dead-letter queue:
|
|
585
|
+
|
|
586
|
+
```typescript
|
|
587
|
+
@OnWorkerEvent('failed')
|
|
588
|
+
async onFailed(job: Job | undefined, err: Error): Promise<void> {
|
|
589
|
+
if (job && job.attemptsMade >= (job.opts.attempts ?? 1)) {
|
|
590
|
+
await this.queue.enqueue(`${job.queueName}-dlq`, job.name, job.data, {
|
|
591
|
+
jobId: `dlq:${job.id}`,
|
|
592
|
+
})
|
|
593
|
+
}
|
|
594
|
+
}
|
|
595
|
+
```
|
|
596
|
+
|
|
597
|
+
Tune `drainTimeoutMs` when jobs may run longer than 30 s:
|
|
598
|
+
|
|
599
|
+
```typescript
|
|
600
|
+
BymaxQueueModule.forRoot({
|
|
601
|
+
connection: { url: process.env.REDIS_URL },
|
|
602
|
+
shutdown: { drainTimeoutMs: 60_000 },
|
|
603
|
+
})
|
|
604
|
+
```
|
|
605
|
+
|
|
606
|
+
---
|
|
607
|
+
|
|
608
|
+
## 🤔 Why over `@nestjs/bullmq`
|
|
609
|
+
|
|
610
|
+
| Concern | `@nestjs/bullmq` (official) | `@bymax-one/nest-queue` |
|
|
611
|
+
| ------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------- |
|
|
612
|
+
| Module setup | `BullModule.forRoot()` + `registerQueue()` per queue | Single `forRoot`/`forRootAsync`; queues created on demand by name |
|
|
613
|
+
| Connection | Manual ioredis + `maxRetriesPerRequest` wiring | Dual mode (BYO / lib-owned); correct per-role retry policy applied automatically |
|
|
614
|
+
| Job defaults | Per `registerQueue` | Centralized opinionated defaults with per-queue/per-job overrides |
|
|
615
|
+
| Producer API | `@InjectQueue` + raw `queue.add` | Typed `enqueue<TData, TResult>()`, `enqueueBulk`, `upsertJobScheduler` |
|
|
616
|
+
| Shutdown | Framework closes workers | Explicit bounded-drain protocol; force-close + stalled accounting |
|
|
617
|
+
| Observability | Manual | `telemetry` passthrough (OpenTelemetry) + `MetricsService` |
|
|
618
|
+
|
|
619
|
+
**Choose this library** when you want a single opinionated setup, the `@bymax-one/nest-cache`
|
|
620
|
+
connection-sharing story, enforced defaults, and a tested bounded-shutdown protocol.
|
|
621
|
+
|
|
622
|
+
**Choose `@nestjs/bullmq`** when you want the thinnest official binding and prefer to
|
|
623
|
+
wire connection and defaults yourself.
|
|
624
|
+
|
|
625
|
+
---
|
|
626
|
+
|
|
627
|
+
## 🏗️ Architecture
|
|
628
|
+
|
|
629
|
+
```
|
|
630
|
+
BymaxQueueModule.forRoot / forRootAsync
|
|
631
|
+
│
|
|
632
|
+
validateOptions (fail fast)
|
|
633
|
+
│
|
|
634
|
+
applyDefaults (frozen)
|
|
635
|
+
│
|
|
636
|
+
┌─────────────┴─────────────┐
|
|
637
|
+
│ ConnectionResolver │
|
|
638
|
+
│ Mode A: your ioredis │
|
|
639
|
+
│ Mode B: lib-owned │
|
|
640
|
+
└─────────────┬─────────────┘
|
|
641
|
+
│
|
|
642
|
+
┌──────────────┬─────────────────┼─────────────────┬──────────────┐
|
|
643
|
+
│ │ │ │ │
|
|
644
|
+
QueueService WorkerRegistry QueueEventsRegistry FlowService MetricsService
|
|
645
|
+
(Queue) (Worker) (QueueEvents) (FlowProducer) (opt-in)
|
|
646
|
+
│ │ │ │
|
|
647
|
+
default duplicate() duplicate() default
|
|
648
|
+
retries maxRetries: null maxRetries: null retries
|
|
649
|
+
│ │ │ │
|
|
650
|
+
└──────────────┴────── same `prefix` ──────────────┘
|
|
651
|
+
│
|
|
652
|
+
ProcessorDiscoveryService
|
|
653
|
+
(@Processor / @Process / @OnWorkerEvent)
|
|
654
|
+
│
|
|
655
|
+
QueueLifecycle
|
|
656
|
+
(bounded drain → force close → disconnect)
|
|
657
|
+
```
|
|
658
|
+
|
|
659
|
+
### Design Principles
|
|
660
|
+
|
|
661
|
+
| Principle | Description |
|
|
662
|
+
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
663
|
+
| 🔌 **Connections are role-shaped** | BullMQ blocks a connection for consumers. Only the duplicated `Worker`/`QueueEvents` connections get `maxRetriesPerRequest: null`; the `Queue` and `FlowProducer` connection keeps ioredis' default retries, so a producer fails fast during a Redis outage instead of hanging a request. |
|
|
664
|
+
| 🏷️ **One namespace, four objects** | The configured `prefix` reaches `Queue`, `Worker`, `QueueEvents` and `FlowProducer`. BullMQ defaults an omitted prefix to `bull`, so a partial application splits producers from consumers silently. |
|
|
665
|
+
| 💥 **Fail at boot, not at runtime** | Options are validated before any connection is opened, and every rejection carries a distinct reason. |
|
|
666
|
+
| 🧬 **Explicit DI tokens** | Tokens are `Symbol()` and every injectable constructor parameter carries an explicit `@Inject`. The package ships bundled, where design-time metadata is not emitted — implicit resolution would break inside your build, not ours. |
|
|
667
|
+
| 📅 **Current BullMQ API only** | Recurring jobs use Job Schedulers; the `addRepeatable` API removed in BullMQ v6 is never called. Cron is validated by BullMQ's own parser. |
|
|
668
|
+
| 🧊 **Zero runtime dependencies** | `dependencies` is `{}`. You choose the exact `bullmq` and `ioredis` versions, and the supply-chain surface stays yours. |
|
|
669
|
+
|
|
670
|
+
---
|
|
671
|
+
|
|
672
|
+
## 🔐 Security Model
|
|
673
|
+
|
|
674
|
+
This library moves your application's data through Redis and runs your code against it.
|
|
675
|
+
Its security contract is about what it does with credentials, what it puts in a payload,
|
|
676
|
+
and which keyspace a job lands in.
|
|
677
|
+
|
|
678
|
+
### Credentials never leave the configuration
|
|
679
|
+
|
|
680
|
+
Connection details arrive through injected options — the library never reads
|
|
681
|
+
`process.env` itself, so credentials live wherever your configuration layer keeps them.
|
|
682
|
+
No connection string, password, or TLS material is written to a log line, an exception
|
|
683
|
+
message, or an exception payload. `QueueException.details` carries only scalar
|
|
684
|
+
configuration values (`{ actualValue, expectedValue, limit, received }`), never a
|
|
685
|
+
connection object and never `job.data`.
|
|
686
|
+
|
|
687
|
+
### Job data is opaque
|
|
688
|
+
|
|
689
|
+
The library treats `job.data` as a payload to transport, never to inspect. It is never
|
|
690
|
+
deep-merged (which would expose a prototype-pollution path from a queued payload), never
|
|
691
|
+
logged, and never copied into an error. Whatever you enqueue is what your handler
|
|
692
|
+
receives — so a payload holding a secret is a decision you make, and Redis is where it
|
|
693
|
+
will sit until the job is removed.
|
|
694
|
+
|
|
695
|
+
### The prefix is a namespace, not a security boundary
|
|
696
|
+
|
|
697
|
+
`prefix` isolates one tenant's or one environment's keys from another's inside a single
|
|
698
|
+
Redis database. It is enforced consistently — `Queue`, `Worker`, `QueueEvents` and
|
|
699
|
+
`FlowProducer` all receive it — so jobs cannot be produced into one namespace and
|
|
700
|
+
consumed from another. It is **not** an access control: anything that can reach the Redis
|
|
701
|
+
instance can read every namespace on it. Cross-tenant isolation that must survive a
|
|
702
|
+
compromised client belongs in Redis ACLs or separate instances.
|
|
703
|
+
|
|
704
|
+
### Shutdown is bounded and at-least-once
|
|
705
|
+
|
|
706
|
+
`QueueLifecycle` drains in-flight work under a deadline and then force-closes. A job still
|
|
707
|
+
running when the deadline expires is **not** acknowledged, so BullMQ's stalled-job check
|
|
708
|
+
re-queues it and it runs again. Delivery is at-least-once by design: **every handler must
|
|
709
|
+
be idempotent.** A handler that charges a card or sends an email without an idempotency
|
|
710
|
+
key will do it twice at some point.
|
|
711
|
+
|
|
712
|
+
### Security Checklist
|
|
713
|
+
|
|
714
|
+
- [ ] Redis reachable only from the application network — never exposed publicly.
|
|
715
|
+
- [ ] Redis requires authentication (`requirepass` or ACL user) and TLS in transit.
|
|
716
|
+
- [ ] A distinct `prefix` per tenant or environment, and a distinct Redis ACL user or
|
|
717
|
+
instance where the isolation must hold against a compromised client.
|
|
718
|
+
- [ ] Every `@Process` handler is idempotent — retries and stalled-job recovery both
|
|
719
|
+
replay work.
|
|
720
|
+
- [ ] Job payloads carry identifiers, not secrets. If a payload must reference a secret,
|
|
721
|
+
store a reference and resolve it inside the handler.
|
|
722
|
+
- [ ] `removeOnComplete` / `removeOnFail` are bounded, so payloads do not accumulate in
|
|
723
|
+
Redis indefinitely.
|
|
724
|
+
- [ ] `shutdown.drainTimeoutMs` exceeds your worst-case job runtime, so a normal deploy
|
|
725
|
+
does not force-close work that was about to finish.
|
|
726
|
+
|
|
727
|
+
---
|
|
728
|
+
|
|
729
|
+
## 🛡️ Security Table
|
|
730
|
+
|
|
731
|
+
| Layer | Implementation |
|
|
732
|
+
| ----------------- | ---------------------------------------------------------------------------------------------------------- |
|
|
733
|
+
| Credentials | Injected options only; never read from `process.env`, never logged, never placed in an exception |
|
|
734
|
+
| Error payloads | `QueueException.details` restricted to scalar configuration values — no `job.data`, no connection object |
|
|
735
|
+
| Job payloads | Treated as opaque; never deep-merged (prototype-pollution guard), never logged |
|
|
736
|
+
| Namespacing | `prefix` applied uniformly to `Queue`, `Worker`, `QueueEvents` and `FlowProducer` |
|
|
737
|
+
| Connection policy | `maxRetriesPerRequest: null` confined to duplicated consumer connections; producers keep fail-fast retries |
|
|
738
|
+
| Delivery | At-least-once with a bounded drain; unacknowledged work is re-queued by BullMQ's stalled check |
|
|
739
|
+
| Supply chain | `dependencies: {}`; SHA-pinned Actions, OSV-Scanner, TruffleHog, OpenSSF Scorecard; npm publish over OIDC |
|
|
740
|
+
| Input validation | Options validated at bootstrap; cron patterns validated by BullMQ's own parser, never a hand-rolled regex |
|
|
741
|
+
|
|
742
|
+
> [!IMPORTANT]
|
|
743
|
+
> **Delivery is at-least-once, and `prefix` is not an access boundary.** Handlers must be
|
|
744
|
+
> idempotent, and anything with credentials for the Redis instance can read every
|
|
745
|
+
> namespace on it.
|
|
746
|
+
|
|
747
|
+
---
|
|
748
|
+
|
|
749
|
+
## 🧱 Tech Stack
|
|
750
|
+
|
|
751
|
+
- **Runtime:** Node.js 24+
|
|
752
|
+
- **Framework:** NestJS 11 (`ConfigurableModuleBuilder` + `setExtras`)
|
|
753
|
+
- **Queue engine:** BullMQ `^5.16` (peer) over `ioredis ^5` (peer)
|
|
754
|
+
- **Telemetry:** `bullmq-otel ^1` (optional peer)
|
|
755
|
+
- **Build:** tsup — ESM + CJS per subpath, with `.d.ts` _and_ `.d.cts` declarations
|
|
756
|
+
- **Tests:** Jest + `@testcontainers/redis` (E2E) + Stryker (mutation)
|
|
757
|
+
- **TypeScript:** 5.x strict (`noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`), zero `any`
|
|
758
|
+
|
|
759
|
+
---
|
|
760
|
+
|
|
761
|
+
## 🧪 Testing & Quality
|
|
762
|
+
|
|
763
|
+
- **100% line/branch coverage** on every implemented file (hard CI gate).
|
|
764
|
+
- **Mutation score 99.06%** (Stryker `break 95`) — see [`docs/mutation_testing_results.md`](docs/mutation_testing_results.md).
|
|
765
|
+
- **Real-Redis E2E** via Testcontainers (workers, flows, schedulers, deduplication, retries,
|
|
766
|
+
DLQ, graceful shutdown).
|
|
767
|
+
- **Type-level API tests** (`pnpm test:types`) pin the published signatures, so a refactor
|
|
768
|
+
that widens or loosens one fails the build instead of reaching consumers.
|
|
769
|
+
- **Published-shape gates**: `pnpm check:exports` resolves every entrypoint the way each
|
|
770
|
+
module-resolution mode would (all four `attw` quadrants green, `node10` included), and
|
|
771
|
+
`pnpm smoke` resolves the packed tarball from a real consumer in ESM _and_ CommonJS.
|
|
772
|
+
- **Bundle budgets** enforced (`pnpm size`): server ≤ 18 KiB brotli, shared ≤ 2.5 KiB.
|
|
773
|
+
- **Supply chain:** SHA-pinned GitHub Actions, OSV-Scanner, TruffleHog, OpenSSF Scorecard,
|
|
774
|
+
npm publish with provenance (OIDC).
|
|
775
|
+
|
|
776
|
+
```bash
|
|
777
|
+
pnpm typecheck && pnpm test:types && pnpm lint && pnpm test:cov:all && \
|
|
778
|
+
pnpm build && pnpm size && pnpm check:exports && pnpm smoke
|
|
779
|
+
```
|
|
780
|
+
|
|
781
|
+
---
|
|
782
|
+
|
|
783
|
+
## 🩺 Troubleshooting
|
|
784
|
+
|
|
785
|
+
### `CONNECTION_REQUIRES_NULL_RETRIES`
|
|
786
|
+
|
|
787
|
+
This error fires in Mode A when the injected client was not configured with
|
|
788
|
+
`maxRetriesPerRequest: null`. BullMQ workers require this setting to block on
|
|
789
|
+
Redis operations rather than giving up after a few retries.
|
|
790
|
+
|
|
791
|
+
If you use `@bymax-one/nest-cache`, pass a queue-dedicated client with `maxRetriesPerRequest: null`:
|
|
792
|
+
|
|
793
|
+
```typescript
|
|
794
|
+
// In your cache module, create a second ioredis client for the queue
|
|
795
|
+
const queueRedis = new Redis({ ..., maxRetriesPerRequest: null })
|
|
796
|
+
```
|
|
797
|
+
|
|
798
|
+
The library duplicates the client you pass and applies the correct setting; ensure
|
|
799
|
+
the **original** client passed to Mode A is usable as a duplication source (ioredis
|
|
800
|
+
allows duplicate on any state).
|
|
801
|
+
|
|
802
|
+
### Jobs get stuck (not processed after restart)
|
|
803
|
+
|
|
804
|
+
Stalled jobs are re-queued automatically by BullMQ's stalled-job check. If workers
|
|
805
|
+
do not pick them up:
|
|
806
|
+
|
|
807
|
+
1. Verify `drainTimeoutMs` is larger than the worst-case job runtime.
|
|
808
|
+
2. Verify `lockDuration` (in `workerOptions`) covers the handler execution time —
|
|
809
|
+
a worker that runs longer than `lockDuration` will have its lock stolen, marking
|
|
810
|
+
the job as stalled.
|
|
811
|
+
3. Check `worker.maxStalledCount` is `> 0` (default `1`) so stalled jobs re-queue.
|
|
812
|
+
|
|
813
|
+
---
|
|
814
|
+
|
|
815
|
+
## 🚫 What This Library Does NOT Do
|
|
816
|
+
|
|
817
|
+
- **NestJS only** — does not work with plain Express, standalone Fastify, or other frameworks.
|
|
818
|
+
- **Node.js only** — Deno and Bun are not supported.
|
|
819
|
+
- **BullMQ `^5.16` (1.x floor)** — v6 promotion is planned once the E2E suite is
|
|
820
|
+
green on the v5 + v6 matrix; see the BullMQ version policy in
|
|
821
|
+
[`CHANGELOG.md`](CHANGELOG.md).
|
|
822
|
+
- **No Redis Cluster sharding** — BullMQ's cluster support is not exercised by this library.
|
|
823
|
+
- **No built-in DLQ helper** — use the `@OnWorkerEvent('failed')` pattern above.
|
|
824
|
+
- **No Prometheus metrics export** — `getMetrics()` returns snapshots; publish them
|
|
825
|
+
via `prom-client` in your own health endpoint.
|
|
826
|
+
|
|
827
|
+
For the full list of limitations and their alternatives, see
|
|
828
|
+
[`docs/technical_specification.md § 16`](docs/technical_specification.md).
|
|
829
|
+
|
|
830
|
+
---
|
|
831
|
+
|
|
832
|
+
## 🤝 Contributing
|
|
833
|
+
|
|
834
|
+
Pull requests are welcome. Please open an issue first for significant changes.
|
|
835
|
+
|
|
836
|
+
- Read [`docs/technical_specification.md`](docs/technical_specification.md) for architecture decisions.
|
|
837
|
+
- Run the full gate listed in [`CONTRIBUTING.md`](CONTRIBUTING.md) before opening a PR.
|
|
838
|
+
- Conventional Commits are enforced by `commitlint.config.cjs`.
|
|
839
|
+
|
|
840
|
+
---
|
|
841
|
+
|
|
842
|
+
## 🔒 Security Policy
|
|
843
|
+
|
|
844
|
+
If you discover a security vulnerability, please **do not** open a public issue. Instead, email us
|
|
845
|
+
at **support@bymax.one** with details. We take security seriously and will respond promptly. See
|
|
846
|
+
[`SECURITY.md`](SECURITY.md) for the full policy.
|
|
847
|
+
|
|
848
|
+
---
|
|
849
|
+
|
|
850
|
+
## 📄 License
|
|
851
|
+
|
|
852
|
+
[MIT](./LICENSE) © [Bymax One](https://github.com/bymaxone)
|
|
853
|
+
|
|
854
|
+
---
|
|
855
|
+
|
|
856
|
+
<p align="center">
|
|
857
|
+
<sub>Built with ❤️ by <a href="https://github.com/bymaxone">Bymax One</a></sub>
|
|
858
|
+
</p>
|