queue-jobs-worker 1.0.2 → 1.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +160 -128
- package/README.md +135 -338
- package/assets/queue-jobs-worker-demo.mp4 +0 -0
- package/assets/queue-jobs-worker-github.png +0 -0
- package/dist/core/client.d.ts +2 -0
- package/dist/core/client.d.ts.map +1 -1
- package/dist/core/worker.d.ts.map +1 -1
- package/dist/index.cjs +327 -93
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +327 -93
- package/dist/index.js.map +1 -1
- package/dist/storage/in-memory.adapter.d.ts +3 -2
- package/dist/storage/in-memory.adapter.d.ts.map +1 -1
- package/dist/storage/mysql.adapter.d.ts +3 -2
- package/dist/storage/mysql.adapter.d.ts.map +1 -1
- package/dist/storage/postgres.adapter.d.ts +3 -2
- package/dist/storage/postgres.adapter.d.ts.map +1 -1
- package/dist/storage/redis.adapter.d.ts +3 -2
- package/dist/storage/redis.adapter.d.ts.map +1 -1
- package/dist/types/storage.types.d.ts +20 -2
- package/dist/types/storage.types.d.ts.map +1 -1
- package/dist/types/worker.types.d.ts +1 -1
- package/dist/types/worker.types.d.ts.map +1 -1
- package/package.json +110 -109
package/README.md
CHANGED
|
@@ -1,51 +1,35 @@
|
|
|
1
|
+

|
|
2
|
+
|
|
1
3
|
# queue-jobs-worker
|
|
2
4
|
|
|
3
|
-
A
|
|
5
|
+
A durable, TypeScript-first job queue for Node.js built for asynchronous work, retries, scheduling, and recovery. It based on multiple storage adapter with postgresql, mysql, redis also in-memory support for dev/testing.
|
|
4
6
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
7
|
+
<p align="center">
|
|
8
|
+
<a href="https://www.npmjs.com/package/queue-jobs-worker">
|
|
9
|
+
<img src="https://img.shields.io/npm/v/queue-jobs-worker.svg" alt="npm version">
|
|
10
|
+
</a>
|
|
11
|
+
<a href="./LICENSE">
|
|
12
|
+
<img src="https://img.shields.io/npm/l/queue-jobs-worker.svg" alt="license">
|
|
13
|
+
</a>
|
|
14
|
+
<a href="https://nodejs.org">
|
|
15
|
+
<img src="https://img.shields.io/node/v/queue-jobs-worker.svg" alt="node">
|
|
16
|
+
</a>
|
|
17
|
+
</p>
|
|
8
18
|
|
|
9
19
|
---
|
|
10
20
|
|
|
11
21
|
## Overview
|
|
12
22
|
|
|
13
|
-
`queue-jobs-worker`
|
|
14
|
-
|
|
15
|
-
For a full breakdown of every feature, see [FEATURES.md](./FEATURES.md).
|
|
23
|
+
`queue-jobs-worker` helps you move background work out of the request lifecycle and into a reliable, persistent queue. Define the processor logic once and let the library handle enqueueing, persistence, retries, schedules, concurrency, rate limiting, and recovery.
|
|
16
24
|
|
|
17
|
-
|
|
18
|
-
- In-memory (dev / testing)
|
|
19
|
-
- Redis (node-redis v4+)
|
|
20
|
-
- PostgreSQL (node-postgres / pg)
|
|
21
|
-
- MySQL (mysql2)
|
|
25
|
+
It supports all major local and production-friendly backends:
|
|
22
26
|
|
|
23
|
-
|
|
27
|
+
- In-memory queue for development and tests
|
|
28
|
+
- Redis via `node-redis` v4+
|
|
29
|
+
- PostgreSQL via `pg`
|
|
30
|
+
- MySQL via `mysql2`
|
|
24
31
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
- [Installation](#installation)
|
|
28
|
-
- [Quick Start](#quick-start)
|
|
29
|
-
- [Dialects](#dialects)
|
|
30
|
-
- [Memory](#memory-no-setup-required)
|
|
31
|
-
- [Redis](#redis)
|
|
32
|
-
- [PostgreSQL](#postgresql)
|
|
33
|
-
- [MySQL](#mysql)
|
|
34
|
-
- [Core Concepts](#core-concepts)
|
|
35
|
-
- [Configuration](#configuration)
|
|
36
|
-
- [Enqueueing Jobs](#enqueueing-jobs)
|
|
37
|
-
- [Processing Jobs](#processing-jobs)
|
|
38
|
-
- [Workers](#workers)
|
|
39
|
-
- [Events](#events)
|
|
40
|
-
- [Querying Jobs](#querying-jobs)
|
|
41
|
-
- [Retry & Backoff](#retry--backoff)
|
|
42
|
-
- [Scheduling](#scheduling)
|
|
43
|
-
- [Priority](#priority)
|
|
44
|
-
- [Rate Limiting](#rate-limiting)
|
|
45
|
-
- [Dead Letter Queue](#dead-letter-queue)
|
|
46
|
-
- [Graceful Shutdown](#graceful-shutdown)
|
|
47
|
-
- [Custom Storage Adapter](#custom-storage-adapter)
|
|
48
|
-
- [API Reference](#api-reference)
|
|
32
|
+
For a detailed feature breakdown, see [FEATURES.md](./FEATURES.md).
|
|
49
33
|
|
|
50
34
|
---
|
|
51
35
|
|
|
@@ -55,7 +39,7 @@ For a full breakdown of every feature, see [FEATURES.md](./FEATURES.md).
|
|
|
55
39
|
npm install queue-jobs-worker
|
|
56
40
|
```
|
|
57
41
|
|
|
58
|
-
Install
|
|
42
|
+
Install the driver you plan to use:
|
|
59
43
|
|
|
60
44
|
```bash
|
|
61
45
|
# Redis
|
|
@@ -72,44 +56,15 @@ npm install mysql2
|
|
|
72
56
|
|
|
73
57
|
## Quick Start
|
|
74
58
|
|
|
75
|
-
**TypeScript**
|
|
76
|
-
```ts
|
|
77
|
-
import { QueueClient } from "queue-jobs-worker";
|
|
78
|
-
|
|
79
|
-
const client = new QueueClient();
|
|
80
|
-
|
|
81
|
-
// Generic type parameter gives you typed job.data
|
|
82
|
-
const emails = client.createQueue<{ to: string; subject: string }>("emails");
|
|
83
|
-
|
|
84
|
-
emails.process("send-email", async (job) => {
|
|
85
|
-
await sendEmail(job.data.to, job.data.subject);
|
|
86
|
-
// Throw to trigger retry; return to mark as completed
|
|
87
|
-
});
|
|
88
|
-
|
|
89
|
-
emails.createWorker({ concurrency: 5 });
|
|
90
|
-
|
|
91
|
-
await emails.enqueue("send-email", {
|
|
92
|
-
to: "user@example.com",
|
|
93
|
-
subject: "Welcome!",
|
|
94
|
-
});
|
|
95
|
-
|
|
96
|
-
process.on("SIGTERM", async () => {
|
|
97
|
-
await client.close();
|
|
98
|
-
process.exit(0);
|
|
99
|
-
});
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
**JavaScript**
|
|
103
59
|
```js
|
|
104
60
|
const { QueueClient } = require("queue-jobs-worker");
|
|
105
61
|
|
|
106
62
|
const client = new QueueClient();
|
|
107
|
-
|
|
108
|
-
// No generic — job.data is untyped
|
|
109
63
|
const emails = client.createQueue("emails");
|
|
110
64
|
|
|
111
65
|
emails.process("send-email", async (job) => {
|
|
112
66
|
await sendEmail(job.data.to, job.data.subject);
|
|
67
|
+
// Return to mark the job complete; throw to trigger retry or DLQ handling
|
|
113
68
|
});
|
|
114
69
|
|
|
115
70
|
emails.createWorker({ concurrency: 5 });
|
|
@@ -125,152 +80,46 @@ process.on("SIGTERM", async () => {
|
|
|
125
80
|
});
|
|
126
81
|
```
|
|
127
82
|
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
## Dialects
|
|
131
|
-
|
|
132
|
-
### Memory (no setup required)
|
|
133
|
-
|
|
134
|
-
Uses an in-process Map. Data is lost on restart. Perfect for development and tests.
|
|
135
|
-
|
|
136
|
-
```js
|
|
137
|
-
const client = new QueueClient();
|
|
138
|
-
// or explicitly:
|
|
139
|
-
const client = new QueueClient({ dialect: "memory" });
|
|
140
|
-
```
|
|
141
|
-
|
|
142
|
-
`init()` is optional for memory — it's a no-op. All other dialects require it.
|
|
83
|
+
If you are using TypeScript, you can optionally make the queue payload type-safe with a generic like `client.createQueue<{ to: string; subject: string }>("emails")`.
|
|
143
84
|
|
|
144
85
|
---
|
|
145
86
|
|
|
146
|
-
|
|
87
|
+
## Supported Backends
|
|
147
88
|
|
|
148
|
-
|
|
89
|
+
Pass `dialect` and `connectionString` to `QueueClient`. Call `await client.init()` to establish database connections and create required schema tables.
|
|
149
90
|
|
|
150
91
|
```js
|
|
151
|
-
//
|
|
152
|
-
const
|
|
153
|
-
|
|
154
|
-
const client = new QueueClient({
|
|
155
|
-
dialect: "redis",
|
|
156
|
-
connectionString: "redis://localhost:6379",
|
|
157
|
-
});
|
|
158
|
-
|
|
159
|
-
await client.init(); // connects + PING — throws if unreachable
|
|
160
|
-
|
|
161
|
-
const jobs = client.createQueue("jobs");
|
|
162
|
-
```
|
|
163
|
-
|
|
164
|
-
**With authentication:**
|
|
165
|
-
```js
|
|
166
|
-
const client = new QueueClient({
|
|
167
|
-
dialect: "redis",
|
|
168
|
-
connectionString: "redis://:yourpassword@redis-host:6379/0",
|
|
169
|
-
});
|
|
170
|
-
await client.init();
|
|
171
|
-
```
|
|
92
|
+
// 1. In-Memory (Default for local development & tests, no persistence)
|
|
93
|
+
const client = new QueueClient({ dialect: "memory" });
|
|
172
94
|
|
|
173
|
-
|
|
174
|
-
```js
|
|
95
|
+
// 2. Redis (Supports standard redis://, TLS rediss://, and authentication)
|
|
175
96
|
const client = new QueueClient({
|
|
176
97
|
dialect: "redis",
|
|
177
|
-
connectionString: "
|
|
98
|
+
connectionString: process.env.REDIS_URL || "redis://localhost:6379",
|
|
178
99
|
});
|
|
179
|
-
await client.init();
|
|
180
|
-
```
|
|
181
|
-
|
|
182
|
-
**What `init()` does for Redis:**
|
|
183
|
-
- Creates the node-redis client
|
|
184
|
-
- Calls `client.connect()`
|
|
185
|
-
- Sends `PING` and asserts the response is `PONG`
|
|
186
|
-
- Throws a descriptive error if the connection fails
|
|
187
|
-
|
|
188
|
-
**Key structure in Redis** (prefix: `qjw:`):
|
|
189
|
-
```
|
|
190
|
-
qjw:job:{id} → Hash (all job fields)
|
|
191
|
-
qjw:queue:{name}:waiting → Sorted Set (score = -priority)
|
|
192
|
-
qjw:queue:{name}:delayed → Sorted Set (score = runAt ms)
|
|
193
|
-
qjw:queue:{name}:active → Set
|
|
194
|
-
qjw:queue:{name}:completed → Set
|
|
195
|
-
qjw:queue:{name}:dead → Set
|
|
196
|
-
qjw:rate:{name} → String (rate-limit counter)
|
|
197
|
-
```
|
|
198
|
-
|
|
199
|
-
Job claiming uses a **Lua script** so it is atomic — two concurrent workers can never claim the same job.
|
|
200
|
-
|
|
201
|
-
---
|
|
202
|
-
|
|
203
|
-
### PostgreSQL
|
|
204
|
-
|
|
205
|
-
Requires `pg` (node-postgres): `npm install pg`
|
|
206
|
-
|
|
207
|
-
```js
|
|
208
|
-
// TypeScript: import { QueueClient } from "queue-jobs-worker";
|
|
209
|
-
const { QueueClient } = require("queue-jobs-worker");
|
|
210
100
|
|
|
101
|
+
// 3. PostgreSQL (Auto-creates required queue tables on init)
|
|
211
102
|
const client = new QueueClient({
|
|
212
103
|
dialect: "postgres",
|
|
213
|
-
connectionString: "postgresql://user:password@localhost:5432/mydb",
|
|
104
|
+
connectionString: process.env.POSTGRES_URL || "postgresql://user:password@localhost:5432/mydb",
|
|
214
105
|
});
|
|
215
106
|
|
|
216
|
-
|
|
217
|
-
```
|
|
218
|
-
|
|
219
|
-
**What `init()` does for PostgreSQL:**
|
|
220
|
-
- Creates a connection pool (`pg.Pool`)
|
|
221
|
-
- Runs `SELECT 1` to verify connectivity
|
|
222
|
-
- Executes `CREATE TABLE IF NOT EXISTS` for `qjw_jobs` and `qjw_rate_limits` — **idempotent, safe to run on every startup**
|
|
223
|
-
- Throws a descriptive error if the connection fails
|
|
224
|
-
|
|
225
|
-
**Tables created automatically** (prefix: `qjw_`):
|
|
226
|
-
```sql
|
|
227
|
-
qjw_jobs -- stores every job and its full lifecycle state
|
|
228
|
-
qjw_rate_limits -- sliding-window rate-limit counters
|
|
229
|
-
```
|
|
230
|
-
|
|
231
|
-
Claiming uses `SELECT ... FOR UPDATE SKIP LOCKED` inside a transaction — safe for any number of concurrent workers.
|
|
232
|
-
|
|
233
|
-
**With SSL (Heroku, Supabase, Neon, etc.):**
|
|
234
|
-
```js
|
|
235
|
-
const client = new QueueClient({
|
|
236
|
-
dialect: "postgres",
|
|
237
|
-
connectionString: process.env.DATABASE_URL,
|
|
238
|
-
// pg respects ?sslmode=require in the connection string
|
|
239
|
-
});
|
|
240
|
-
await client.init();
|
|
241
|
-
```
|
|
242
|
-
|
|
243
|
-
---
|
|
244
|
-
|
|
245
|
-
### MySQL
|
|
246
|
-
|
|
247
|
-
Requires `mysql2`: `npm install mysql2`
|
|
248
|
-
|
|
249
|
-
```js
|
|
250
|
-
// TypeScript: import { QueueClient } from "queue-jobs-worker";
|
|
251
|
-
const { QueueClient } = require("queue-jobs-worker");
|
|
252
|
-
|
|
107
|
+
// 4. MySQL (Auto-creates required queue tables on init)
|
|
253
108
|
const client = new QueueClient({
|
|
254
109
|
dialect: "mysql",
|
|
255
|
-
connectionString: "mysql://user:password@localhost:3306/mydb",
|
|
110
|
+
connectionString: process.env.MYSQL_URL || "mysql://user:password@localhost:3306/mydb",
|
|
256
111
|
});
|
|
257
112
|
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
**What `init()` does for MySQL:**
|
|
262
|
-
- Creates a connection pool (`mysql2.createPool`)
|
|
263
|
-
- Runs `SELECT 1` to verify connectivity
|
|
264
|
-
- Executes `CREATE TABLE IF NOT EXISTS` for `qjw_jobs` and `qjw_rate_limits` — **idempotent**
|
|
265
|
-
- Throws a descriptive error if the connection fails
|
|
266
|
-
|
|
267
|
-
**Tables created automatically** (prefix: `qjw_`):
|
|
268
|
-
```sql
|
|
269
|
-
qjw_jobs -- full job state (InnoDB, utf8mb4)
|
|
270
|
-
qjw_rate_limits -- sliding-window rate-limit counters
|
|
113
|
+
// Initialize backend connection (Required for Redis, PostgreSQL, MySQL)
|
|
114
|
+
await client.init();
|
|
271
115
|
```
|
|
272
116
|
|
|
273
|
-
|
|
117
|
+
| Dialect | Connection Format | `client.init()` Behavior |
|
|
118
|
+
|---|---|---|
|
|
119
|
+
| `memory` | N/A | No-op (transient memory store) |
|
|
120
|
+
| `redis` | `redis://...`, `rediss://...` (TLS), Auth URL | Connects & verifies with `PING` |
|
|
121
|
+
| `postgres` | `postgresql://user:pass@host:5432/dbname` | `SELECT 1` check & creates schema |
|
|
122
|
+
| `mysql` | `mysql://user:pass@host:3306/dbname` | Connection check & creates schema |
|
|
274
123
|
|
|
275
124
|
---
|
|
276
125
|
|
|
@@ -278,26 +127,25 @@ Claiming uses `SELECT ... FOR UPDATE SKIP LOCKED` inside a transaction.
|
|
|
278
127
|
|
|
279
128
|
| Concept | Description |
|
|
280
129
|
|---|---|
|
|
281
|
-
| `QueueClient` | Entry point
|
|
282
|
-
| `Queue` |
|
|
283
|
-
| `Job` | A unit of work
|
|
284
|
-
| `Worker` | Claims and executes jobs
|
|
285
|
-
| `Processor` | Your function
|
|
286
|
-
| `StorageAdapter` |
|
|
287
|
-
| DLQ | Dead Letter Queue
|
|
130
|
+
| `QueueClient` | Entry point that owns configuration, storage, and queues |
|
|
131
|
+
| `Queue` | A separate job stream with its own settings |
|
|
132
|
+
| `Job` | A unit of work passed to your processor |
|
|
133
|
+
| `Worker` | Claims and executes jobs |
|
|
134
|
+
| `Processor` | Your async function, e.g. `async (job) => { ... }` |
|
|
135
|
+
| `StorageAdapter` | A backend abstraction for durable storage |
|
|
136
|
+
| `DLQ` | Dead Letter Queue for permanently failed jobs |
|
|
288
137
|
|
|
289
138
|
---
|
|
290
139
|
|
|
291
140
|
## Configuration
|
|
292
141
|
|
|
293
|
-
|
|
142
|
+
Settings are layered so more specific config overrides broader defaults:
|
|
294
143
|
|
|
295
144
|
```
|
|
296
145
|
Client defaults → Queue options → Worker options → Job options
|
|
297
146
|
```
|
|
298
147
|
|
|
299
148
|
```js
|
|
300
|
-
// TypeScript: import { QueueClient } from "queue-jobs-worker";
|
|
301
149
|
const { QueueClient } = require("queue-jobs-worker");
|
|
302
150
|
|
|
303
151
|
const client = new QueueClient({
|
|
@@ -305,17 +153,17 @@ const client = new QueueClient({
|
|
|
305
153
|
connectionString: process.env.REDIS_URL,
|
|
306
154
|
|
|
307
155
|
defaults: {
|
|
308
|
-
attempts: 3,
|
|
309
|
-
retryDelay: 1000,
|
|
310
|
-
backoff: "exponential",
|
|
311
|
-
timeout: 30_000,
|
|
312
|
-
concurrency: 10,
|
|
313
|
-
pollInterval: 1_000,
|
|
314
|
-
stalledInterval: 30_000
|
|
315
|
-
lockDuration: 60_000,
|
|
156
|
+
attempts: 3,
|
|
157
|
+
retryDelay: 1000,
|
|
158
|
+
backoff: "exponential",
|
|
159
|
+
timeout: 30_000,
|
|
160
|
+
concurrency: 10,
|
|
161
|
+
pollInterval: 1_000,
|
|
162
|
+
stalledInterval: 30_000,
|
|
163
|
+
lockDuration: 60_000,
|
|
316
164
|
rateLimit: {
|
|
317
165
|
max: 100,
|
|
318
|
-
duration: 60_000,
|
|
166
|
+
duration: 60_000,
|
|
319
167
|
},
|
|
320
168
|
},
|
|
321
169
|
});
|
|
@@ -327,24 +175,6 @@ await client.init();
|
|
|
327
175
|
|
|
328
176
|
## Enqueueing Jobs
|
|
329
177
|
|
|
330
|
-
**TypeScript**
|
|
331
|
-
```ts
|
|
332
|
-
// Generic type gives you autocomplete and type-safety on job.data
|
|
333
|
-
const queue = client.createQueue<{ userId: string }>("notifications");
|
|
334
|
-
|
|
335
|
-
await queue.enqueue("send-push", { userId: "u_123" });
|
|
336
|
-
|
|
337
|
-
// With options
|
|
338
|
-
await queue.enqueue("send-push", { userId: "u_123" }, {
|
|
339
|
-
attempts: 5,
|
|
340
|
-
retryDelay: 2000,
|
|
341
|
-
backoff: "linear",
|
|
342
|
-
timeout: 10_000,
|
|
343
|
-
priority: 10, // higher = processed first (default: 0)
|
|
344
|
-
});
|
|
345
|
-
```
|
|
346
|
-
|
|
347
|
-
**JavaScript**
|
|
348
178
|
```js
|
|
349
179
|
const queue = client.createQueue("notifications");
|
|
350
180
|
|
|
@@ -359,24 +189,32 @@ await queue.enqueue("send-push", { userId: "u_123" }, {
|
|
|
359
189
|
});
|
|
360
190
|
```
|
|
361
191
|
|
|
192
|
+
If you want TypeScript type safety for `job.data`, pass a generic when creating the queue, such as `client.createQueue<{ userId: string }>("notifications")`.
|
|
193
|
+
|
|
362
194
|
---
|
|
363
195
|
|
|
364
196
|
## Processing Jobs
|
|
365
197
|
|
|
366
|
-
Register a processor before starting the
|
|
198
|
+
Register a processor before creating or starting a worker. Processors receive the `job` instance as well as an `AbortSignal` for cooperative cancellation when a job attempt times out:
|
|
367
199
|
|
|
368
200
|
```js
|
|
369
|
-
queue.process("send-push", async (job) => {
|
|
201
|
+
queue.process("send-push", async (job, signal) => {
|
|
370
202
|
const { userId } = job.data;
|
|
371
203
|
|
|
372
|
-
|
|
204
|
+
// Pass signal to APIs that support cancellation (e.g. fetch, DB queries):
|
|
205
|
+
await pushService.send(userId, "You have a new message", { signal });
|
|
206
|
+
|
|
207
|
+
// Or check signal.aborted before performing expensive steps:
|
|
208
|
+
if (signal.aborted) return;
|
|
373
209
|
|
|
374
|
-
// Return to mark
|
|
375
|
-
// Throw any error to
|
|
210
|
+
// Return to mark the job complete.
|
|
211
|
+
// Throw any error to trigger retry logic or DLQ handling.
|
|
376
212
|
});
|
|
377
213
|
```
|
|
378
214
|
|
|
379
|
-
|
|
215
|
+
> **Note on Timeout Cancellation**: In Node.js, asynchronous operations cannot be forcibly terminated from the outside. Processors should cooperate with cancellation by checking `signal.aborted` or forwarding `signal` to abortable APIs to ensure timed-out executions do not continue running in the background.
|
|
216
|
+
|
|
217
|
+
See [FEATURES.md](./FEATURES.md) for the full `job` model and helper methods.
|
|
380
218
|
|
|
381
219
|
---
|
|
382
220
|
|
|
@@ -384,42 +222,40 @@ For the full list of `job` fields and helper methods, see [FEATURES.md → Job I
|
|
|
384
222
|
|
|
385
223
|
```js
|
|
386
224
|
const worker = queue.createWorker({
|
|
387
|
-
concurrency: 10,
|
|
388
|
-
shutdownTimeout: 30_000,
|
|
225
|
+
concurrency: 10,
|
|
226
|
+
shutdownTimeout: 30_000,
|
|
389
227
|
});
|
|
390
228
|
|
|
391
|
-
console.log(worker.status);
|
|
392
|
-
console.log(worker.id);
|
|
229
|
+
console.log(worker.status);
|
|
230
|
+
console.log(worker.id);
|
|
393
231
|
|
|
394
232
|
await worker.stop();
|
|
395
233
|
```
|
|
396
234
|
|
|
397
|
-
|
|
235
|
+
Multiple workers can share the same queue and coordinate through the storage layer:
|
|
398
236
|
|
|
399
237
|
```js
|
|
400
238
|
const w1 = queue.createWorker({ concurrency: 5 });
|
|
401
239
|
const w2 = queue.createWorker({ concurrency: 5 });
|
|
402
|
-
//
|
|
240
|
+
// total capacity: 10 concurrent jobs
|
|
403
241
|
```
|
|
404
242
|
|
|
405
243
|
---
|
|
406
244
|
|
|
407
245
|
## Events
|
|
408
246
|
|
|
409
|
-
|
|
247
|
+
The client emits lifecycle events that are useful for monitoring and alerting:
|
|
410
248
|
|
|
411
249
|
```js
|
|
412
250
|
client.on("job:completed", (job) => console.log("Done:", job.id));
|
|
413
|
-
client.on("job:failed",
|
|
414
|
-
client.on("job:dead",
|
|
415
|
-
client.on("worker:error",
|
|
251
|
+
client.on("job:failed", (job, err) => console.error("Failed:", job.id, err.message));
|
|
252
|
+
client.on("job:dead", (job, err) => console.error("DLQ:", job.id, err.message));
|
|
253
|
+
client.on("worker:error", (workerId, err) => console.error("Worker error:", err));
|
|
416
254
|
|
|
417
|
-
client.off("job:completed", myListener);
|
|
418
|
-
client.once("job:dead", (job, err) => alertTeam(job, err));
|
|
255
|
+
client.off("job:completed", myListener);
|
|
256
|
+
client.once("job:dead", (job, err) => alertTeam(job, err));
|
|
419
257
|
```
|
|
420
258
|
|
|
421
|
-
For the full event reference (all job, worker, and system events), see [FEATURES.md → Event System](./FEATURES.md#event-system).
|
|
422
|
-
|
|
423
259
|
---
|
|
424
260
|
|
|
425
261
|
## Querying Jobs
|
|
@@ -430,13 +266,11 @@ if (job) {
|
|
|
430
266
|
console.log(job.status, job.attemptsMade);
|
|
431
267
|
}
|
|
432
268
|
|
|
433
|
-
|
|
434
|
-
const
|
|
435
|
-
const active = await queue.getJobs("active");
|
|
269
|
+
const waiting = await queue.getJobs("waiting", 50, 0);
|
|
270
|
+
const active = await queue.getJobs("active");
|
|
436
271
|
const completed = await queue.getJobs("completed", 100, 0);
|
|
437
|
-
const dead
|
|
272
|
+
const dead = await queue.getJobs("dead");
|
|
438
273
|
|
|
439
|
-
// Counts per status
|
|
440
274
|
const counts = await queue.getJobCounts();
|
|
441
275
|
// {
|
|
442
276
|
// waiting: 12,
|
|
@@ -451,17 +285,15 @@ const counts = await queue.getJobCounts();
|
|
|
451
285
|
|
|
452
286
|
## Retry & Backoff
|
|
453
287
|
|
|
454
|
-
|
|
288
|
+
Retries can be configured at the client, queue, or job level:
|
|
455
289
|
|
|
456
290
|
```js
|
|
457
|
-
// Queue-level
|
|
458
291
|
const queue = client.createQueue("tasks", {
|
|
459
292
|
attempts: 5,
|
|
460
293
|
retryDelay: 2000,
|
|
461
294
|
backoff: "exponential",
|
|
462
295
|
});
|
|
463
296
|
|
|
464
|
-
// Job-level override
|
|
465
297
|
await queue.enqueue("task", payload, {
|
|
466
298
|
attempts: 3,
|
|
467
299
|
retryDelay: 500,
|
|
@@ -469,57 +301,50 @@ await queue.enqueue("task", payload, {
|
|
|
469
301
|
});
|
|
470
302
|
```
|
|
471
303
|
|
|
472
|
-
|
|
304
|
+
Available strategies are `fixed`, `linear`, and `exponential`. Each failure is tracked in `job.attemptHistory` so you can inspect what happened without losing context.
|
|
473
305
|
|
|
474
306
|
---
|
|
475
307
|
|
|
476
308
|
## Scheduling
|
|
477
309
|
|
|
478
310
|
```js
|
|
479
|
-
// Relative delay
|
|
480
311
|
await queue.enqueue("reminder", payload, { schedule: { delay: 30_000 } });
|
|
481
|
-
|
|
482
|
-
// Absolute timestamp
|
|
483
312
|
await queue.enqueue("report", payload, { schedule: { runAt: "2026-09-01T09:00:00Z" } });
|
|
484
|
-
|
|
485
|
-
// Cron expression (stored for recurring jobs)
|
|
486
313
|
await queue.enqueue("cleanup", payload, { schedule: { cron: "0 3 * * *" } });
|
|
487
314
|
```
|
|
488
315
|
|
|
489
|
-
Delayed jobs
|
|
316
|
+
Delayed jobs stay dormant until their scheduled time is reached.
|
|
490
317
|
|
|
491
318
|
---
|
|
492
319
|
|
|
493
320
|
## Priority
|
|
494
321
|
|
|
495
|
-
|
|
322
|
+
Jobs with a higher priority value are processed sooner. The default is `0`.
|
|
496
323
|
|
|
497
324
|
```js
|
|
498
325
|
await queue.enqueue("urgent-task", payload, { priority: 100 });
|
|
499
326
|
await queue.enqueue("normal-task", payload, { priority: 0 });
|
|
500
|
-
await queue.enqueue("low-task",
|
|
501
|
-
//
|
|
327
|
+
await queue.enqueue("low-task", payload, { priority: -10 });
|
|
328
|
+
// order: urgent → normal → low
|
|
502
329
|
```
|
|
503
330
|
|
|
504
|
-
See [FEATURES.md → Priority](./FEATURES.md#priority).
|
|
505
|
-
|
|
506
331
|
---
|
|
507
332
|
|
|
508
333
|
## Rate Limiting
|
|
509
334
|
|
|
510
335
|
```js
|
|
511
336
|
const queue = client.createQueue("webhooks", {
|
|
512
|
-
rateLimit: { max: 50, duration: 60_000 },
|
|
337
|
+
rateLimit: { max: 50, duration: 60_000 },
|
|
513
338
|
});
|
|
514
339
|
```
|
|
515
340
|
|
|
516
|
-
When
|
|
341
|
+
When a queue reaches its limit, workers pause claiming new jobs until the time window resets. Jobs are not discarded.
|
|
517
342
|
|
|
518
343
|
---
|
|
519
344
|
|
|
520
345
|
## Dead Letter Queue
|
|
521
346
|
|
|
522
|
-
When a job
|
|
347
|
+
When a job reaches the end of its retry budget, it is moved to the dead-letter queue with status `"dead"`.
|
|
523
348
|
|
|
524
349
|
```js
|
|
525
350
|
client.on("job:dead", async (job, error) => {
|
|
@@ -529,64 +354,37 @@ client.on("job:dead", async (job, error) => {
|
|
|
529
354
|
const deadJobs = await queue.getJobs("dead");
|
|
530
355
|
```
|
|
531
356
|
|
|
532
|
-
|
|
357
|
+
All failure history remains attached to the job record.
|
|
533
358
|
|
|
534
359
|
---
|
|
535
360
|
|
|
536
361
|
## Graceful Shutdown
|
|
537
362
|
|
|
538
|
-
|
|
363
|
+
Call `client.close()` before your process exits:
|
|
539
364
|
|
|
540
365
|
```js
|
|
541
366
|
process.on("SIGTERM", async () => {
|
|
542
|
-
await client.close();
|
|
367
|
+
await client.close();
|
|
543
368
|
process.exit(0);
|
|
544
369
|
});
|
|
370
|
+
|
|
545
371
|
process.on("SIGINT", async () => {
|
|
546
372
|
await client.close();
|
|
547
373
|
process.exit(0);
|
|
548
374
|
});
|
|
549
375
|
```
|
|
550
376
|
|
|
551
|
-
|
|
377
|
+
This stops workers cleanly, releases locks, and allows stalled-job recovery to continue safely after restarts.
|
|
552
378
|
|
|
553
379
|
---
|
|
554
380
|
|
|
555
381
|
## Custom Storage Adapter
|
|
556
382
|
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
**TypeScript**
|
|
560
|
-
```ts
|
|
561
|
-
import type { StorageAdapter } from "queue-jobs-worker";
|
|
562
|
-
|
|
563
|
-
class MongoStorageAdapter implements StorageAdapter {
|
|
564
|
-
async initialize() { /* connect, create indexes */ }
|
|
565
|
-
async close() { /* disconnect */ }
|
|
566
|
-
async enqueue(input) { /* ... */ }
|
|
567
|
-
async claim(input) { /* atomic claim */ }
|
|
568
|
-
async complete(jobId) { /* ... */ }
|
|
569
|
-
async requeue(input) { /* ... */ }
|
|
570
|
-
async moveToDlq(input) { /* ... */ }
|
|
571
|
-
async releaseLock(jobId) { /* ... */ }
|
|
572
|
-
async recoverStalledJobs(queue, now) { /* ... */ }
|
|
573
|
-
async getJob(jobId) { /* ... */ }
|
|
574
|
-
async getJobs(filter) { /* ... */ }
|
|
575
|
-
async getJobCounts(queue) { /* ... */ }
|
|
576
|
-
async checkAndIncrementRateLimit(queue, max, windowMs, now) { /* ... */ }
|
|
577
|
-
}
|
|
578
|
-
|
|
579
|
-
const client = QueueClient.withAdapter(new MongoStorageAdapter(), {
|
|
580
|
-
defaults: { attempts: 5 },
|
|
581
|
-
});
|
|
582
|
-
await client.init();
|
|
583
|
-
```
|
|
383
|
+
You can provide a custom backend by implementing the `StorageAdapter` interface. The same idea applies in JavaScript or TypeScript; the main difference is whether you add explicit interface typing in TypeScript.
|
|
584
384
|
|
|
585
|
-
**JavaScript**
|
|
586
385
|
```js
|
|
587
386
|
const { QueueClient } = require("queue-jobs-worker");
|
|
588
387
|
|
|
589
|
-
// In JS there's no interface to implement — just match the method signatures
|
|
590
388
|
class MongoStorageAdapter {
|
|
591
389
|
async initialize() { /* connect, create indexes */ }
|
|
592
390
|
async close() { /* disconnect */ }
|
|
@@ -606,6 +404,7 @@ class MongoStorageAdapter {
|
|
|
606
404
|
const client = QueueClient.withAdapter(new MongoStorageAdapter(), {
|
|
607
405
|
defaults: { attempts: 5 },
|
|
608
406
|
});
|
|
407
|
+
|
|
609
408
|
await client.init();
|
|
610
409
|
```
|
|
611
410
|
|
|
@@ -618,66 +417,64 @@ await client.init();
|
|
|
618
417
|
| Option | Type | Default | Description |
|
|
619
418
|
|---|---|---|---|
|
|
620
419
|
| `dialect` | `"memory" \| "redis" \| "postgres" \| "mysql"` | `"memory"` | Storage backend |
|
|
621
|
-
| `connectionString` | `string` | — | Required for
|
|
622
|
-
| `defaults.attempts` | `number` | `3` |
|
|
623
|
-
| `defaults.retryDelay` | `number` | `1000` |
|
|
624
|
-
| `defaults.backoff` | `"fixed" \| "linear" \| "exponential"` | `"exponential"` |
|
|
625
|
-
| `defaults.timeout` | `number` | `30000` |
|
|
420
|
+
| `connectionString` | `string` | — | Required for Redis/PostgreSQL/MySQL |
|
|
421
|
+
| `defaults.attempts` | `number` | `3` | Max retries per job |
|
|
422
|
+
| `defaults.retryDelay` | `number` | `1000` | Base retry delay in ms |
|
|
423
|
+
| `defaults.backoff` | `"fixed" \| "linear" \| "exponential"` | `"exponential"` | Retry strategy |
|
|
424
|
+
| `defaults.timeout` | `number` | `30000` | Per-attempt timeout in ms |
|
|
626
425
|
| `defaults.concurrency` | `number` | `10` | Default worker concurrency |
|
|
627
|
-
| `defaults.pollInterval` | `number` | `1000` |
|
|
628
|
-
| `defaults.stalledInterval` | `number` | `30000` | Stalled-job check interval
|
|
629
|
-
| `defaults.lockDuration` | `number` | `60000` | Lock TTL
|
|
630
|
-
| `defaults.rateLimit` | `{ max, duration }` | — | Optional rate
|
|
426
|
+
| `defaults.pollInterval` | `number` | `1000` | Poll interval in ms |
|
|
427
|
+
| `defaults.stalledInterval` | `number` | `30000` | Stalled-job check interval in ms |
|
|
428
|
+
| `defaults.lockDuration` | `number` | `60000` | Lock TTL in ms |
|
|
429
|
+
| `defaults.rateLimit` | `{ max, duration }` | — | Optional rate limiting |
|
|
631
430
|
|
|
632
431
|
### `client.init()`
|
|
633
432
|
|
|
634
|
-
|
|
433
|
+
Initializes the configured backend. This is required for Redis, PostgreSQL, and MySQL before queue operations. It is safe to call more than once.
|
|
635
434
|
|
|
636
435
|
### `client.createQueue<TPayload>(name, options?)`
|
|
637
436
|
|
|
638
|
-
Creates and returns a
|
|
437
|
+
Creates and returns a queue. Queue-specific options override client defaults.
|
|
639
438
|
|
|
640
439
|
### `client.getQueue<TPayload>(name)` / `client.requireQueue<TPayload>(name)`
|
|
641
440
|
|
|
642
|
-
|
|
441
|
+
Fetches an existing queue by name. `requireQueue()` throws if none exists.
|
|
643
442
|
|
|
644
443
|
### `client.on(event, listener)` / `client.once(...)` / `client.off(...)`
|
|
645
444
|
|
|
646
|
-
|
|
445
|
+
Registers and removes event listeners for queue and worker lifecycle events.
|
|
647
446
|
|
|
648
447
|
### `client.close()`
|
|
649
448
|
|
|
650
|
-
|
|
449
|
+
Stops workers and closes storage connections gracefully.
|
|
651
450
|
|
|
652
451
|
### `QueueClient.withAdapter(adapter, options?)`
|
|
653
452
|
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
---
|
|
453
|
+
Creates a client using a custom storage backend.
|
|
657
454
|
|
|
658
455
|
### `queue.enqueue(type, payload, options?)`
|
|
659
456
|
|
|
660
457
|
| Option | Type | Description |
|
|
661
458
|
|---|---|---|
|
|
662
|
-
| `attempts` | `number` |
|
|
663
|
-
| `retryDelay` | `number` | Base retry delay
|
|
664
|
-
| `backoff` | `string` |
|
|
665
|
-
| `timeout` | `number` | Per-attempt timeout
|
|
666
|
-
| `priority` | `number` | Higher
|
|
667
|
-
| `schedule.delay` | `number` | Delay before eligible
|
|
459
|
+
| `attempts` | `number` | Maximum attempts for this job |
|
|
460
|
+
| `retryDelay` | `number` | Base retry delay in ms |
|
|
461
|
+
| `backoff` | `string` | Retry backoff strategy |
|
|
462
|
+
| `timeout` | `number` | Per-attempt timeout in ms |
|
|
463
|
+
| `priority` | `number` | Higher values are processed first |
|
|
464
|
+
| `schedule.delay` | `number` | Delay before the job becomes eligible |
|
|
668
465
|
| `schedule.runAt` | `string \| number` | Absolute run time |
|
|
669
|
-
| `schedule.cron` | `string` | Cron expression
|
|
466
|
+
| `schedule.cron` | `string` | Cron expression for recurring jobs |
|
|
670
467
|
|
|
671
468
|
### `queue.process(type, processor)`
|
|
672
469
|
|
|
673
|
-
|
|
470
|
+
Registers an async processor for a job type. The processor signature is `async (job, signal) => ...`, where `signal` is an `AbortSignal` aborted when the per-attempt timeout is reached.
|
|
674
471
|
|
|
675
472
|
### `queue.createWorker(options?)`
|
|
676
473
|
|
|
677
474
|
| Option | Type | Default | Description |
|
|
678
475
|
|---|---|---|---|
|
|
679
|
-
| `concurrency` | `number` | queue config |
|
|
680
|
-
| `shutdownTimeout` | `number` | `30000` |
|
|
476
|
+
| `concurrency` | `number` | queue config | Maximum simultaneous job executions |
|
|
477
|
+
| `shutdownTimeout` | `number` | `30000` | Graceful shutdown wait time in ms |
|
|
681
478
|
|
|
682
479
|
### `queue.getJob(id)` / `queue.getJobs(status?, limit?, offset?)`
|
|
683
480
|
### `queue.getJobCounts()`
|
|
@@ -688,7 +485,7 @@ Register an async processor function for a job type.
|
|
|
688
485
|
|
|
689
486
|
| Feature | Memory | Redis | PostgreSQL | MySQL |
|
|
690
487
|
|---|:---:|:---:|:---:|:---:|
|
|
691
|
-
| Persistence | | ✓ | ✓ | ✓ |
|
|
488
|
+
| Persistence | — | ✓ | ✓ | ✓ |
|
|
692
489
|
| Atomic claim | ✓ | ✓ (Lua) | ✓ (SKIP LOCKED) | ✓ (SKIP LOCKED) |
|
|
693
490
|
| Priority ordering | ✓ | ✓ | ✓ | ✓ |
|
|
694
491
|
| Delayed jobs | ✓ | ✓ | ✓ | ✓ |
|
|
@@ -703,9 +500,9 @@ Register an async processor function for a job type.
|
|
|
703
500
|
|
|
704
501
|
## Security
|
|
705
502
|
|
|
706
|
-
- Job payloads are
|
|
707
|
-
- Error messages do not
|
|
708
|
-
- Connection strings should
|
|
503
|
+
- Job payloads are not logged by default.
|
|
504
|
+
- Error messages do not expose payload data.
|
|
505
|
+
- Connection strings should come from environment variables rather than source code.
|
|
709
506
|
- See [SECURITY.md](./SECURITY.md) for the full policy.
|
|
710
507
|
|
|
711
508
|
```js
|
|
@@ -726,7 +523,7 @@ const client = new QueueClient({
|
|
|
726
523
|
|
|
727
524
|
## Contributing
|
|
728
525
|
|
|
729
|
-
|
|
526
|
+
Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for contribution guidelines.
|
|
730
527
|
|
|
731
528
|
---
|
|
732
529
|
|
|
@@ -736,6 +533,6 @@ MIT — [LICENSE](./LICENSE)
|
|
|
736
533
|
|
|
737
534
|
## Donation
|
|
738
535
|
|
|
739
|
-
If
|
|
536
|
+
If this project has been useful to you, consider supporting it with a coffee.
|
|
740
537
|
|
|
741
538
|
**BTC:** `12dxgVQ3sRFhc4g7M6oydsN2tTMMthJJqS`
|