queue-jobs-worker 1.0.3 → 1.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +62 -84
- package/README.md +121 -132
- package/assets/queue-jobs-worker-demo.gif +0 -0
- package/dist/core/client.d.ts +2 -0
- package/dist/core/client.d.ts.map +1 -1
- package/dist/core/queue.d.ts +3 -1
- package/dist/core/queue.d.ts.map +1 -1
- package/dist/index.cjs +194 -140
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +194 -140
- package/dist/index.js.map +1 -1
- package/dist/lib/scripts/claim.lua +27 -0
- package/dist/lib/scripts/index.d.ts +5 -0
- package/dist/lib/scripts/index.d.ts.map +1 -0
- package/dist/lib/scripts/rate-limit.lua +36 -0
- package/dist/lib/scripts/recover-stalled.lua +40 -0
- package/dist/lib/scripts/renew-lock.lua +16 -0
- package/dist/storage/mysql.adapter.d.ts.map +1 -1
- package/dist/storage/postgres.adapter.d.ts.map +1 -1
- package/dist/storage/redis.adapter.d.ts.map +1 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,134 +5,108 @@ All notable changes to **queue-jobs-worker** will be documented in this file.
|
|
|
5
5
|
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
|
6
6
|
This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
-
## [1.0.
|
|
8
|
+
## [1.0.5] — 2026-09-15
|
|
9
9
|
|
|
10
10
|
### Core
|
|
11
11
|
|
|
12
12
|
### Fixed
|
|
13
13
|
|
|
14
|
-
-
|
|
14
|
+
- **Storage Initialization Enforcement prior to Queue Creation & Execution** ([#14](https://github.com/rafidahmed870/queue-jobs-worker/issues/14))
|
|
15
15
|
|
|
16
|
-
Previously,
|
|
16
|
+
Previously, `QueueClient.createQueue()` allowed queues to be created before `await client.init()` was called when using external storage dialects (e.g. Redis, PostgreSQL, MySQL). This caused the created queue to bind to the temporary `InMemoryStorageAdapter` instance. When `client.init()` was subsequently called, the real external storage adapter replaced the internal storage field on `QueueClient`, rendering previously enqueued jobs lost or inaccessible.
|
|
17
17
|
|
|
18
18
|
After the fix:
|
|
19
19
|
|
|
20
|
-
- `
|
|
21
|
-
-
|
|
22
|
-
-
|
|
23
|
-
- User processors can monitor `signal.aborted` or pass `signal` to async operations (e.g. `fetch`, database queries, timers) for cooperative cancellation.
|
|
20
|
+
- `QueueClient.createQueue()` checks `isInitialised` before creating a queue. For external dialects and custom adapters, attempting to create a queue before `await client.init()` throws an explicit error.
|
|
21
|
+
- Queue operations (`enqueue`, `getJob`, `getJobs`, `getJobCounts`, and `createWorker`) enforce initialization status checks before executing, preventing operations on uninitialized storage.
|
|
22
|
+
- In-memory dialect continues to auto-initialize synchronously, preserving convenient single-line setup for tests and local development.
|
|
24
23
|
|
|
25
|
-
|
|
24
|
+
---
|
|
26
25
|
|
|
27
|
-
###
|
|
26
|
+
### Events
|
|
28
27
|
|
|
29
|
-
|
|
28
|
+
### Added
|
|
30
29
|
|
|
31
|
-
|
|
30
|
+
- `QueueEventEmitter` — strongly-typed lifecycle event bus shared across all components.
|
|
31
|
+
- Emits events for the full job lifecycle: enqueued, started, completed, failed, retrying, dead, stalled.
|
|
32
|
+
- All event payloads fully typed via `events.types.ts`.
|
|
32
33
|
|
|
33
34
|
---
|
|
34
35
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
### Core
|
|
38
|
-
|
|
39
|
-
### Fixed
|
|
40
|
-
|
|
41
|
-
- **`Worker` — croner added as a required dependency; invalid expressions no longer fall back to a 1-minute interval** ([#5](https://github.com/rafidahmed870/queue-jobs-worker/issues/5))
|
|
36
|
+
<!-- Links -->
|
|
37
|
+
[1.0.0]: https://github.com/rafidahmed870/queue-jobs-worker/releases/tag/v1.0.0
|
|
42
38
|
|
|
43
|
-
|
|
44
|
-
a try/catch. If the import failed — or if the resolved `Cron` class was not a
|
|
45
|
-
function — the code silently fell back to `Date.now() + 60_000`, scheduling
|
|
46
|
-
the next run 60 seconds later regardless of the configured cron expression.
|
|
47
|
-
The same silent fallback was also triggered for invalid cron expressions that
|
|
48
|
-
caused the `Cron` constructor to throw.
|
|
39
|
+
### Lib
|
|
49
40
|
|
|
50
|
-
|
|
41
|
+
### Added
|
|
51
42
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
-
|
|
59
|
-
occurrences), a `worker:error` is emitted and re-enqueue is skipped. Again,
|
|
60
|
-
the worker keeps running.
|
|
61
|
-
- The 1-minute fallback path has been removed entirely — there is no silent
|
|
62
|
-
fallback under any failure condition.
|
|
63
|
-
|
|
64
|
-
- **`Worker` — rate-limit quota no longer consumed on empty-queue polls** ([#4](https://github.com/rafidahmed870/queue-jobs-worker/issues/4))
|
|
65
|
-
|
|
66
|
-
`claimNext()` previously called `checkAndIncrementRateLimit()` before
|
|
67
|
-
attempting to claim a job. This meant every poll cycle against an empty queue
|
|
68
|
-
burned a quota slot, potentially exhausting the configured window budget
|
|
69
|
-
before any real work was done. After the fix, the storage `claim()` call
|
|
70
|
-
happens first; the rate-limit counter is only incremented when a job is
|
|
71
|
-
actually claimed for processing. If the rate limit is reached at that point
|
|
72
|
-
the lock is immediately released via `releaseLock()` so the job remains
|
|
73
|
-
reclaimable on the next window.
|
|
43
|
+
- **`lib/scripts/` — Redis Lua scripts extracted to standalone `.lua` files**
|
|
44
|
+
- `claim.lua` — atomic job claim with delayed-job promotion.
|
|
45
|
+
- `recover-stalled.lua` — compare-and-swap stalled job recovery.
|
|
46
|
+
- `renew-lock.lua` — atomic lock renewal with ownership guard.
|
|
47
|
+
- `rate-limit.lua` — atomic rate limit decision, reset, and counter increment.
|
|
48
|
+
- `lib/scripts/index.ts` re-exports scripts as named string constants (`CLAIM_LUA`, `RECOVER_STALLED_LUA`, `RENEW_LOCK_LUA`, `RATE_LIMIT_LUA`).
|
|
49
|
+
- Scripts are embedded into the CJS/ESM distribution bundles at build time via `tsup`'s `loader: { ".lua": "text" }`.
|
|
74
50
|
|
|
75
51
|
---
|
|
76
52
|
|
|
77
|
-
###
|
|
53
|
+
### Storage
|
|
78
54
|
|
|
79
55
|
### Added
|
|
80
56
|
|
|
81
|
-
-
|
|
82
|
-
-
|
|
83
|
-
-
|
|
57
|
+
- **Consistent timestamps across Lua scripts**
|
|
58
|
+
- Changed `RedisStorageAdapter` to use `now_iso` from `ARGV[3]` for `updatedAt` in `CLAIM_LUA` and `RECOVER_STALLED_LUA`.
|
|
59
|
+
- Previously, `updatedAt` was sometimes derived from `lockExpiresAt`, which could differ from the actual time of the operation.
|
|
84
60
|
|
|
85
|
-
|
|
61
|
+
### Fixed
|
|
86
62
|
|
|
87
|
-
|
|
88
|
-
|
|
63
|
+
- **Atomic rate limiting across Redis, PostgreSQL, and MySQL adapters**
|
|
64
|
+
- `RedisStorageAdapter`: Implemented `rate-limit.lua` (`RATE_LIMIT_LUA`) script to perform window check, expiry reset, counter evaluation, increment, and TTL renewal atomically inside Redis.
|
|
65
|
+
- `PostgreSQLStorageAdapter`: Wrapped `checkAndIncrementRateLimit` in a pool client transaction (`BEGIN ... COMMIT`) utilizing `INSERT ... ON CONFLICT DO NOTHING` and `SELECT ... FOR UPDATE` row locking.
|
|
66
|
+
- `MySQLStorageAdapter`: Wrapped `checkAndIncrementRateLimit` in a connection transaction (`beginTransaction ... commit`) utilizing `INSERT ... ON DUPLICATE KEY UPDATE` and `SELECT ... FOR UPDATE` row locking.
|
|
89
67
|
|
|
90
|
-
### Storage
|
|
91
68
|
|
|
92
|
-
|
|
69
|
+
---
|
|
93
70
|
|
|
94
|
-
|
|
71
|
+
### Types
|
|
95
72
|
|
|
96
|
-
|
|
73
|
+
### Added
|
|
74
|
+
|
|
75
|
+
- **`lua.d.ts` — ambient module declaration for `.lua` imports**
|
|
76
|
+
- Declares `declare module "*.lua"` so TypeScript recognises `.lua` files as `string`-exporting modules.
|
|
77
|
+
- Required by `src/lib/scripts/index.ts` to import Lua scripts directly without type errors.
|
|
97
78
|
|
|
98
|
-
|
|
99
|
-
2. A separate write pipeline recovered every job whose lock appeared expired.
|
|
79
|
+
---
|
|
100
80
|
|
|
101
|
-
|
|
102
|
-
its lock. The write pipeline had no knowledge of that change and would
|
|
103
|
-
unconditionally overwrite the job back to `"waiting"`, causing duplicate
|
|
104
|
-
processing or data loss.
|
|
81
|
+
### Tests
|
|
105
82
|
|
|
106
|
-
|
|
107
|
-
per-job Lua script (`RECOVER_STALLED_LUA`) that implements a
|
|
108
|
-
**compare-and-swap (CAS)** guard. The script atomically re-reads
|
|
109
|
-
`lockExpiresAt`, `lockId`, and `status` from the hash and aborts if any of
|
|
110
|
-
the three values differ from what the caller observed in the read phase.
|
|
111
|
-
Because Redis executes Lua scripts as a single indivisible command, no
|
|
112
|
-
concurrent write can slip between the re-read and the state update. The
|
|
113
|
-
pre-filter (skip jobs whose lock has not yet expired) is preserved as an
|
|
114
|
-
optimisation to avoid unnecessary Lua round-trips.
|
|
83
|
+
### Added
|
|
115
84
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
85
|
+
- **`lua-scripts.test.ts` — unit tests for Redis Lua scripts**
|
|
86
|
+
- 22 tests covering all four Lua scripts (`CLAIM_LUA`, `RECOVER_STALLED_LUA`, `RENEW_LOCK_LUA`, `RATE_LIMIT_LUA`).
|
|
87
|
+
- Verifies each script loads as a non-empty string from `src/lib/scripts/index.ts`.
|
|
88
|
+
- Asserts presence of critical Redis commands (`ZPOPMIN`, `ZRANGEBYSCORE`, `SADD`, `HSET`, `SREM`, `ZADD`, `INCR`, `EXPIRE`) and CAS/RateLimit guard conditions.
|
|
89
|
+
- **`vitest.config.ts` — `rawLuaPlugin` added**
|
|
90
|
+
- Custom Vite transform plugin that loads `.lua` files as raw text strings during tests, mirroring `tsup`'s `loader: { ".lua": "text" }` used at build time.
|
|
121
91
|
|
|
122
92
|
---
|
|
123
93
|
|
|
124
|
-
## [1.0.
|
|
94
|
+
## [1.0.4] — 2026-09-13
|
|
125
95
|
|
|
126
96
|
### Core
|
|
127
97
|
|
|
128
98
|
### Fixed
|
|
129
99
|
|
|
130
|
-
- **`Worker
|
|
100
|
+
- **`Worker` — Job timeout cooperative cancellation via `AbortSignal`** ([#12](https://github.com/rafidahmed870/queue-jobs-worker/issues/12))
|
|
101
|
+
|
|
102
|
+
Previously, when a job attempt reached its configured `timeout`, the worker rejected the internal execution promise and marked the attempt as failed (or scheduled a retry), but the underlying processor `Promise` continued running in the background. This could lead to duplicate side effects when retries overlapped with timed-out attempts.
|
|
103
|
+
|
|
104
|
+
After the fix:
|
|
131
105
|
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
`
|
|
135
|
-
|
|
106
|
+
- `Processor` type signature is updated: `type Processor<TPayload = unknown> = (job: Job<TPayload>, signal: AbortSignal) => Promise<void>`.
|
|
107
|
+
- An `AbortController` is created for each job attempt.
|
|
108
|
+
- When job execution times out, the worker aborts the `AbortSignal` with a timeout error before rejecting the wrapper promise.
|
|
109
|
+
- User processors can monitor `signal.aborted` or pass `signal` to async operations (e.g. `fetch`, database queries, timers) for cooperative cancellation.
|
|
136
110
|
|
|
137
111
|
---
|
|
138
112
|
|
|
@@ -148,6 +122,10 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
|
|
|
148
122
|
|
|
149
123
|
<!-- Links -->
|
|
150
124
|
|
|
125
|
+
[1.0.5]: https://github.com/rafidahmed870/queue-jobs-worker/compare/v1.0.0...v1.0.5
|
|
126
|
+
[1.0.0]: https://github.com/rafidahmed870/queue-jobs-worker/releases/tag/v1.0.0
|
|
127
|
+
[1.0.4]: https://github.com/rafidahmed870/queue-jobs-worker/compare/v1.0.0...v1.0.4
|
|
128
|
+
[1.0.0]: https://github.com/rafidahmed870/queue-jobs-worker/releases/tag/v1.0.0
|
|
151
129
|
[1.0.3]: https://github.com/rafidahmed870/queue-jobs-worker/compare/v1.0.2...v1.0.3
|
|
152
130
|
[1.0.2]: https://github.com/rafidahmed870/queue-jobs-worker/compare/v1.0.1...v1.0.2
|
|
153
131
|
[1.0.0]: https://github.com/rafidahmed870/queue-jobs-worker/releases/tag/v1.0.0
|
package/README.md
CHANGED
|
@@ -2,17 +2,20 @@
|
|
|
2
2
|
|
|
3
3
|
# queue-jobs-worker
|
|
4
4
|
|
|
5
|
-
A durable, TypeScript-first job queue for Node.js built for asynchronous work, retries, scheduling, and recovery.
|
|
5
|
+
A durable, TypeScript-first job queue for Node.js built for asynchronous work, retries, scheduling, and recovery. It based on multiple storage adapter with postgresql, mysql, redis also in-memory support for dev/testing.
|
|
6
6
|
|
|
7
7
|
<p align="center">
|
|
8
8
|
<a href="https://www.npmjs.com/package/queue-jobs-worker">
|
|
9
|
-
<img src="https://img.shields.io/npm/
|
|
9
|
+
<img src="https://img.shields.io/npm/dm/queue-jobs-worker.svg?label=npm%20downloads" alt="npm downloads">
|
|
10
|
+
</a>
|
|
11
|
+
<a href="https://www.npmjs.com/package/queue-jobs-worker">
|
|
12
|
+
<img src="https://img.shields.io/npm/v/queue-jobs-worker.svg?label=npm%20version" alt="npm version">
|
|
10
13
|
</a>
|
|
11
14
|
<a href="./LICENSE">
|
|
12
15
|
<img src="https://img.shields.io/npm/l/queue-jobs-worker.svg" alt="license">
|
|
13
16
|
</a>
|
|
14
17
|
<a href="https://nodejs.org">
|
|
15
|
-
<img src="https://img.shields.io/node/v/queue-jobs-worker.svg" alt="node">
|
|
18
|
+
<img src="https://img.shields.io/node/v/queue-jobs-worker.svg" alt="node version">
|
|
16
19
|
</a>
|
|
17
20
|
</p>
|
|
18
21
|
|
|
@@ -31,7 +34,7 @@ It supports all major local and production-friendly backends:
|
|
|
31
34
|
|
|
32
35
|
For a detailed feature breakdown, see [FEATURES.md](./FEATURES.md).
|
|
33
36
|
|
|
34
|
-
|
|
37
|
+
<img src="./assets/queue-jobs-worker-demo.gif" alt="demo" />
|
|
35
38
|
|
|
36
39
|
## Installation
|
|
37
40
|
|
|
@@ -86,99 +89,54 @@ If you are using TypeScript, you can optionally make the queue payload type-safe
|
|
|
86
89
|
|
|
87
90
|
## Supported Backends
|
|
88
91
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
Use the in-memory backend for local development and tests. Data is not persisted across restarts.
|
|
92
|
+
Pass `dialect` and `connectionString` to `QueueClient`. Call `await client.init()` to establish database connections and create required schema tables.
|
|
92
93
|
|
|
93
94
|
```js
|
|
94
|
-
|
|
95
|
-
// or explicitly:
|
|
95
|
+
// 1. In-Memory (Default for local development & tests, no persistence)
|
|
96
96
|
const client = new QueueClient({ dialect: "memory" });
|
|
97
|
-
```
|
|
98
|
-
|
|
99
|
-
`init()` is effectively a no-op for this backend.
|
|
100
|
-
|
|
101
|
-
### Redis
|
|
102
|
-
|
|
103
|
-
```js
|
|
104
|
-
const { QueueClient } = require("queue-jobs-worker");
|
|
105
|
-
|
|
106
|
-
const client = new QueueClient({
|
|
107
|
-
dialect: "redis",
|
|
108
|
-
connectionString: "redis://localhost:6379",
|
|
109
|
-
});
|
|
110
|
-
|
|
111
|
-
await client.init();
|
|
112
|
-
const jobs = client.createQueue("jobs");
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
With authentication:
|
|
116
|
-
|
|
117
|
-
```js
|
|
118
|
-
const client = new QueueClient({
|
|
119
|
-
dialect: "redis",
|
|
120
|
-
connectionString: "redis://:yourpassword@redis-host:6379/0",
|
|
121
|
-
});
|
|
122
|
-
|
|
123
|
-
await client.init();
|
|
124
|
-
```
|
|
125
97
|
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
```js
|
|
98
|
+
// 2. Redis (Supports standard redis://, TLS rediss://, and authentication)
|
|
129
99
|
const client = new QueueClient({
|
|
130
100
|
dialect: "redis",
|
|
131
|
-
connectionString: "
|
|
101
|
+
connectionString: process.env.REDIS_URL || "redis://localhost:6379",
|
|
132
102
|
});
|
|
133
103
|
|
|
134
|
-
|
|
135
|
-
```
|
|
136
|
-
|
|
137
|
-
`init()` creates the Redis client, connects to the server, sends `PING`, and verifies the response is `PONG`.
|
|
138
|
-
|
|
139
|
-
### PostgreSQL
|
|
140
|
-
|
|
141
|
-
```js
|
|
142
|
-
const { QueueClient } = require("queue-jobs-worker");
|
|
143
|
-
|
|
104
|
+
// 3. PostgreSQL (Auto-creates required queue tables on init)
|
|
144
105
|
const client = new QueueClient({
|
|
145
106
|
dialect: "postgres",
|
|
146
|
-
connectionString: "postgresql://user:password@localhost:5432/mydb",
|
|
107
|
+
connectionString: process.env.POSTGRES_URL || "postgresql://user:password@localhost:5432/mydb",
|
|
147
108
|
});
|
|
148
109
|
|
|
149
|
-
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
`init()` verifies connectivity with `SELECT 1` and creates the queue tables if they do not already exist.
|
|
153
|
-
|
|
154
|
-
### MySQL
|
|
155
|
-
|
|
156
|
-
```js
|
|
157
|
-
const { QueueClient } = require("queue-jobs-worker");
|
|
158
|
-
|
|
110
|
+
// 4. MySQL (Auto-creates required queue tables on init)
|
|
159
111
|
const client = new QueueClient({
|
|
160
112
|
dialect: "mysql",
|
|
161
|
-
connectionString: "mysql://user:password@localhost:3306/mydb",
|
|
113
|
+
connectionString: process.env.MYSQL_URL || "mysql://user:password@localhost:3306/mydb",
|
|
162
114
|
});
|
|
163
115
|
|
|
116
|
+
// Initialize backend connection (Required for Redis, PostgreSQL, MySQL)
|
|
164
117
|
await client.init();
|
|
165
118
|
```
|
|
166
119
|
|
|
167
|
-
`init()`
|
|
120
|
+
| Dialect | Connection Format | `client.init()` Behavior |
|
|
121
|
+
| ---------- | --------------------------------------------- | --------------------------------- |
|
|
122
|
+
| `memory` | N/A | No-op (transient memory store) |
|
|
123
|
+
| `redis` | `redis://...`, `rediss://...` (TLS), Auth URL | Connects & verifies with `PING` |
|
|
124
|
+
| `postgres` | `postgresql://user:pass@host:5432/dbname` | `SELECT 1` check & creates schema |
|
|
125
|
+
| `mysql` | `mysql://user:pass@host:3306/dbname` | Connection check & creates schema |
|
|
168
126
|
|
|
169
127
|
---
|
|
170
128
|
|
|
171
129
|
## Core Concepts
|
|
172
130
|
|
|
173
|
-
| Concept
|
|
174
|
-
|
|
175
|
-
| `QueueClient`
|
|
176
|
-
| `Queue`
|
|
177
|
-
| `Job`
|
|
178
|
-
| `Worker`
|
|
179
|
-
| `Processor`
|
|
180
|
-
| `StorageAdapter` | A backend abstraction for durable storage
|
|
181
|
-
| `DLQ`
|
|
131
|
+
| Concept | Description |
|
|
132
|
+
| ---------------- | -------------------------------------------------------- |
|
|
133
|
+
| `QueueClient` | Entry point that owns configuration, storage, and queues |
|
|
134
|
+
| `Queue` | A separate job stream with its own settings |
|
|
135
|
+
| `Job` | A unit of work passed to your processor |
|
|
136
|
+
| `Worker` | Claims and executes jobs |
|
|
137
|
+
| `Processor` | Your async function, e.g. `async (job) => { ... }` |
|
|
138
|
+
| `StorageAdapter` | A backend abstraction for durable storage |
|
|
139
|
+
| `DLQ` | Dead Letter Queue for permanently failed jobs |
|
|
182
140
|
|
|
183
141
|
---
|
|
184
142
|
|
|
@@ -225,13 +183,17 @@ const queue = client.createQueue("notifications");
|
|
|
225
183
|
|
|
226
184
|
await queue.enqueue("send-push", { userId: "u_123" });
|
|
227
185
|
|
|
228
|
-
await queue.enqueue(
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
186
|
+
await queue.enqueue(
|
|
187
|
+
"send-push",
|
|
188
|
+
{ userId: "u_123" },
|
|
189
|
+
{
|
|
190
|
+
attempts: 5,
|
|
191
|
+
retryDelay: 2000,
|
|
192
|
+
backoff: "linear",
|
|
193
|
+
timeout: 10_000,
|
|
194
|
+
priority: 10,
|
|
195
|
+
},
|
|
196
|
+
);
|
|
235
197
|
```
|
|
236
198
|
|
|
237
199
|
If you want TypeScript type safety for `job.data`, pass a generic when creating the queue, such as `client.createQueue<{ userId: string }>("notifications")`.
|
|
@@ -431,19 +393,45 @@ You can provide a custom backend by implementing the `StorageAdapter` interface.
|
|
|
431
393
|
const { QueueClient } = require("queue-jobs-worker");
|
|
432
394
|
|
|
433
395
|
class MongoStorageAdapter {
|
|
434
|
-
async initialize() {
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
async
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
async
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
async
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
async
|
|
396
|
+
async initialize() {
|
|
397
|
+
/* connect, create indexes */
|
|
398
|
+
}
|
|
399
|
+
async close() {
|
|
400
|
+
/* disconnect */
|
|
401
|
+
}
|
|
402
|
+
async enqueue(input) {
|
|
403
|
+
/* ... */
|
|
404
|
+
}
|
|
405
|
+
async claim(input) {
|
|
406
|
+
/* atomic claim */
|
|
407
|
+
}
|
|
408
|
+
async complete(jobId) {
|
|
409
|
+
/* ... */
|
|
410
|
+
}
|
|
411
|
+
async requeue(input) {
|
|
412
|
+
/* ... */
|
|
413
|
+
}
|
|
414
|
+
async moveToDlq(input) {
|
|
415
|
+
/* ... */
|
|
416
|
+
}
|
|
417
|
+
async releaseLock(jobId) {
|
|
418
|
+
/* ... */
|
|
419
|
+
}
|
|
420
|
+
async recoverStalledJobs(queue, now) {
|
|
421
|
+
/* ... */
|
|
422
|
+
}
|
|
423
|
+
async getJob(jobId) {
|
|
424
|
+
/* ... */
|
|
425
|
+
}
|
|
426
|
+
async getJobs(filter) {
|
|
427
|
+
/* ... */
|
|
428
|
+
}
|
|
429
|
+
async getJobCounts(queue) {
|
|
430
|
+
/* ... */
|
|
431
|
+
}
|
|
432
|
+
async checkAndIncrementRateLimit(queue, max, windowMs, now) {
|
|
433
|
+
/* ... */
|
|
434
|
+
}
|
|
447
435
|
}
|
|
448
436
|
|
|
449
437
|
const client = QueueClient.withAdapter(new MongoStorageAdapter(), {
|
|
@@ -459,19 +447,19 @@ await client.init();
|
|
|
459
447
|
|
|
460
448
|
### `new QueueClient(options?)`
|
|
461
449
|
|
|
462
|
-
| Option
|
|
463
|
-
|
|
464
|
-
| `dialect`
|
|
465
|
-
| `connectionString`
|
|
466
|
-
| `defaults.attempts`
|
|
467
|
-
| `defaults.retryDelay`
|
|
468
|
-
| `defaults.backoff`
|
|
469
|
-
| `defaults.timeout`
|
|
470
|
-
| `defaults.concurrency`
|
|
471
|
-
| `defaults.pollInterval`
|
|
472
|
-
| `defaults.stalledInterval` | `number`
|
|
473
|
-
| `defaults.lockDuration`
|
|
474
|
-
| `defaults.rateLimit`
|
|
450
|
+
| Option | Type | Default | Description |
|
|
451
|
+
| -------------------------- | ---------------------------------------------- | --------------- | ----------------------------------- |
|
|
452
|
+
| `dialect` | `"memory" \| "redis" \| "postgres" \| "mysql"` | `"memory"` | Storage backend |
|
|
453
|
+
| `connectionString` | `string` | — | Required for Redis/PostgreSQL/MySQL |
|
|
454
|
+
| `defaults.attempts` | `number` | `3` | Max retries per job |
|
|
455
|
+
| `defaults.retryDelay` | `number` | `1000` | Base retry delay in ms |
|
|
456
|
+
| `defaults.backoff` | `"fixed" \| "linear" \| "exponential"` | `"exponential"` | Retry strategy |
|
|
457
|
+
| `defaults.timeout` | `number` | `30000` | Per-attempt timeout in ms |
|
|
458
|
+
| `defaults.concurrency` | `number` | `10` | Default worker concurrency |
|
|
459
|
+
| `defaults.pollInterval` | `number` | `1000` | Poll interval in ms |
|
|
460
|
+
| `defaults.stalledInterval` | `number` | `30000` | Stalled-job check interval in ms |
|
|
461
|
+
| `defaults.lockDuration` | `number` | `60000` | Lock TTL in ms |
|
|
462
|
+
| `defaults.rateLimit` | `{ max, duration }` | — | Optional rate limiting |
|
|
475
463
|
|
|
476
464
|
### `client.init()`
|
|
477
465
|
|
|
@@ -499,16 +487,16 @@ Creates a client using a custom storage backend.
|
|
|
499
487
|
|
|
500
488
|
### `queue.enqueue(type, payload, options?)`
|
|
501
489
|
|
|
502
|
-
| Option
|
|
503
|
-
|
|
504
|
-
| `attempts`
|
|
505
|
-
| `retryDelay`
|
|
506
|
-
| `backoff`
|
|
507
|
-
| `timeout`
|
|
508
|
-
| `priority`
|
|
509
|
-
| `schedule.delay` | `number`
|
|
510
|
-
| `schedule.runAt` | `string \| number` | Absolute run time
|
|
511
|
-
| `schedule.cron`
|
|
490
|
+
| Option | Type | Description |
|
|
491
|
+
| ---------------- | ------------------ | ------------------------------------- |
|
|
492
|
+
| `attempts` | `number` | Maximum attempts for this job |
|
|
493
|
+
| `retryDelay` | `number` | Base retry delay in ms |
|
|
494
|
+
| `backoff` | `string` | Retry backoff strategy |
|
|
495
|
+
| `timeout` | `number` | Per-attempt timeout in ms |
|
|
496
|
+
| `priority` | `number` | Higher values are processed first |
|
|
497
|
+
| `schedule.delay` | `number` | Delay before the job becomes eligible |
|
|
498
|
+
| `schedule.runAt` | `string \| number` | Absolute run time |
|
|
499
|
+
| `schedule.cron` | `string` | Cron expression for recurring jobs |
|
|
512
500
|
|
|
513
501
|
### `queue.process(type, processor)`
|
|
514
502
|
|
|
@@ -516,30 +504,31 @@ Registers an async processor for a job type. The processor signature is `async (
|
|
|
516
504
|
|
|
517
505
|
### `queue.createWorker(options?)`
|
|
518
506
|
|
|
519
|
-
| Option
|
|
520
|
-
|
|
521
|
-
| `concurrency`
|
|
522
|
-
| `shutdownTimeout` | `number` | `30000`
|
|
507
|
+
| Option | Type | Default | Description |
|
|
508
|
+
| ----------------- | -------- | ------------ | ----------------------------------- |
|
|
509
|
+
| `concurrency` | `number` | queue config | Maximum simultaneous job executions |
|
|
510
|
+
| `shutdownTimeout` | `number` | `30000` | Graceful shutdown wait time in ms |
|
|
523
511
|
|
|
524
512
|
### `queue.getJob(id)` / `queue.getJobs(status?, limit?, offset?)`
|
|
513
|
+
|
|
525
514
|
### `queue.getJobCounts()`
|
|
526
515
|
|
|
527
516
|
---
|
|
528
517
|
|
|
529
518
|
## Storage Support Matrix
|
|
530
519
|
|
|
531
|
-
| Feature
|
|
532
|
-
|
|
533
|
-
| Persistence
|
|
534
|
-
| Atomic claim
|
|
535
|
-
| Priority ordering
|
|
536
|
-
| Delayed jobs
|
|
537
|
-
| Retry + backoff
|
|
538
|
-
| DLQ
|
|
539
|
-
| Stalled recovery
|
|
540
|
-
| Rate limiting
|
|
541
|
-
| Connection check on init |
|
|
542
|
-
| Auto-create schema
|
|
520
|
+
| Feature | Memory | Redis | PostgreSQL | MySQL |
|
|
521
|
+
| ------------------------ | :----: | :-----: | :-------------: | :-------------: |
|
|
522
|
+
| Persistence | — | ✓ | ✓ | ✓ |
|
|
523
|
+
| Atomic claim | ✓ | ✓ (Lua) | ✓ (SKIP LOCKED) | ✓ (SKIP LOCKED) |
|
|
524
|
+
| Priority ordering | ✓ | ✓ | ✓ | ✓ |
|
|
525
|
+
| Delayed jobs | ✓ | ✓ | ✓ | ✓ |
|
|
526
|
+
| Retry + backoff | ✓ | ✓ | ✓ | ✓ |
|
|
527
|
+
| DLQ | ✓ | ✓ | ✓ | ✓ |
|
|
528
|
+
| Stalled recovery | ✓ | ✓ | ✓ | ✓ |
|
|
529
|
+
| Rate limiting | ✓ | ✓ | ✓ | ✓ |
|
|
530
|
+
| Connection check on init | — | ✓ PING | ✓ SELECT 1 | ✓ SELECT 1 |
|
|
531
|
+
| Auto-create schema | — | — | ✓ | ✓ |
|
|
543
532
|
|
|
544
533
|
---
|
|
545
534
|
|
|
Binary file
|
package/dist/core/client.d.ts
CHANGED
|
@@ -36,6 +36,8 @@ export declare class QueueClient {
|
|
|
36
36
|
private readonly queues;
|
|
37
37
|
private initialised;
|
|
38
38
|
private closed;
|
|
39
|
+
/** True when the adapter was supplied via {@link withAdapter}; init() skips resolveAdapter() in that case. */
|
|
40
|
+
private _customAdapter;
|
|
39
41
|
constructor(options?: QueueClientOptions);
|
|
40
42
|
private buildMemoryOrEagerAdapter;
|
|
41
43
|
/**
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"client.d.ts","sourceRoot":"","sources":["../../src/core/client.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAEH,OAAO,KAAK,EAAE,kBAAkB,EAAkB,MAAM,0BAA0B,CAAC;AACnF,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AAC5D,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAChE,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,0BAA0B,CAAC;AAC5D,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AAwBnC,qBAAa,WAAW;IACtB,0DAA0D;IAC1D,QAAQ,EAAE,cAAc,CAAC;IAEzB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAoB;IAC5C,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAA2B;IACpD,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAS;IACjC,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAqB;IACtD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAqC;IAC5D,OAAO,CAAC,WAAW,CAAS;IAC5B,OAAO,CAAC,MAAM,CAAS;
|
|
1
|
+
{"version":3,"file":"client.d.ts","sourceRoot":"","sources":["../../src/core/client.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAEH,OAAO,KAAK,EAAE,kBAAkB,EAAkB,MAAM,0BAA0B,CAAC;AACnF,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AAC5D,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAChE,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,0BAA0B,CAAC;AAC5D,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AAwBnC,qBAAa,WAAW;IACtB,0DAA0D;IAC1D,QAAQ,EAAE,cAAc,CAAC;IAEzB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAoB;IAC5C,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAA2B;IACpD,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAS;IACjC,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAqB;IACtD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAqC;IAC5D,OAAO,CAAC,WAAW,CAAS;IAC5B,OAAO,CAAC,MAAM,CAAS;IACvB,8GAA8G;IAC9G,OAAO,CAAC,cAAc,CAAS;gBAEnB,OAAO,GAAE,kBAAuB;IAkB5C,OAAO,CAAC,yBAAyB;IAajC;;;;;;;;;;;;;OAaG;IACG,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;IAY3B,gCAAgC;IAC1B,UAAU,IAAI,OAAO,CAAC,IAAI,CAAC;YAInB,cAAc;IAkC5B;;;;;OAKG;IACH,WAAW,CAAC,QAAQ,GAAG,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,YAAY,GAAG,KAAK,CAAC,QAAQ,CAAC;IAwBtF,6DAA6D;IAC7D,QAAQ,CAAC,QAAQ,GAAG,OAAO,EAAE,IAAI,EAAE,MAAM,GAAG,KAAK,CAAC,QAAQ,CAAC,GAAG,SAAS;IAIvE,wDAAwD;IACxD,YAAY,CAAC,QAAQ,GAAG,OAAO,EAAE,IAAI,EAAE,MAAM,GAAG,KAAK,CAAC,QAAQ,CAAC;IAQ/D,qDAAqD;IACrD,IAAI,UAAU,IAAI,MAAM,EAAE,CAEzB;IAED,wDAAwD;IACxD,IAAI,aAAa,IAAI,OAAO,CAE3B;IAMD,EAAE,CAAC,CAAC,SAAS,MAAM,WAAW,EAAE,KAAK,EAAE,CAAC,EAAE,QAAQ,EAAE,CAAC,GAAG,IAAI,EAAE,WAAW,CAAC,CAAC,CAAC,KAAK,IAAI,GAAG,IAAI;IAK5F,IAAI,CAAC,CAAC,SAAS,MAAM,WAAW,EAAE,KAAK,EAAE,CAAC,EAAE,QAAQ,EAAE,CAAC,GAAG,IAAI,EAAE,WAAW,CAAC,CAAC,CAAC,KAAK,IAAI,GAAG,IAAI;IAK9F,GAAG,CAAC,CAAC,SAAS,MAAM,WAAW,EAAE,KAAK,EAAE,CAAC,EAAE,QAAQ,EAAE,CAAC,GAAG,IAAI,EAAE,WAAW,CAAC,CAAC,CAAC,KAAK,IAAI,GAAG,IAAI;IAS7F;;;OAGG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAa5B;;;;;;;;OAQG;IACH,MAAM,CAAC,WAAW,CAChB,OAAO,EAAE,cAAc,EACvB,OAAO,GAAE,IAAI,CAAC,kBAAkB,EAAE,SAAS,GAAG,kBAAkB,CAAM,GACrE,WAAW;CAOf"}
|
package/dist/core/queue.d.ts
CHANGED
|
@@ -26,11 +26,13 @@ export declare class Queue<TPayload = unknown> {
|
|
|
26
26
|
private readonly emitter;
|
|
27
27
|
private readonly resolvedConfig;
|
|
28
28
|
private readonly clientDefaults;
|
|
29
|
+
private readonly isInitialisedCheck?;
|
|
29
30
|
/** Registered processors keyed by job type. */
|
|
30
31
|
private readonly processors;
|
|
31
32
|
/** Active worker instances created by this queue. */
|
|
32
33
|
private readonly workers;
|
|
33
|
-
constructor(name: string, storage: StorageAdapter, emitter: QueueEventEmitter, options: QueueOptions | undefined, defaults: ResolvedDefaults);
|
|
34
|
+
constructor(name: string, storage: StorageAdapter, emitter: QueueEventEmitter, options: QueueOptions | undefined, defaults: ResolvedDefaults, isInitialisedCheck?: (() => boolean) | undefined);
|
|
35
|
+
private assertInitialised;
|
|
34
36
|
/**
|
|
35
37
|
* Add a new job to the queue.
|
|
36
38
|
*
|
package/dist/core/queue.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"queue.d.ts","sourceRoot":"","sources":["../../src/core/queue.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAChE,OAAO,KAAK,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,uBAAuB,CAAC;AACnE,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AAC5D,OAAO,KAAK,EAAE,aAAa,EAAE,SAAS,EAAE,MAAM,0BAA0B,CAAC;AACzE,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,0BAA0B,CAAC;AAC/D,OAAO,EAAE,GAAG,EAAE,MAAM,UAAU,CAAC;AAC/B,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAErC,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,sBAAsB,CAAC;AAE9D,KAAK,gBAAgB,GAAG,QAAQ,CAAC,cAAc,CAAC,CAAC;AA2BjD,qBAAa,KAAK,CAAC,QAAQ,GAAG,OAAO;IACnC,4DAA4D;IAC5D,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IAEtB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAiB;IACzC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAoB;IAC5C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAyB;IACxD,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAmB;
|
|
1
|
+
{"version":3,"file":"queue.d.ts","sourceRoot":"","sources":["../../src/core/queue.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAEH,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAChE,OAAO,KAAK,EAAE,UAAU,EAAE,SAAS,EAAE,MAAM,uBAAuB,CAAC;AACnE,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,yBAAyB,CAAC;AAC5D,OAAO,KAAK,EAAE,aAAa,EAAE,SAAS,EAAE,MAAM,0BAA0B,CAAC;AACzE,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,0BAA0B,CAAC;AAC/D,OAAO,EAAE,GAAG,EAAE,MAAM,UAAU,CAAC;AAC/B,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAErC,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,sBAAsB,CAAC;AAE9D,KAAK,gBAAgB,GAAG,QAAQ,CAAC,cAAc,CAAC,CAAC;AA2BjD,qBAAa,KAAK,CAAC,QAAQ,GAAG,OAAO;IACnC,4DAA4D;IAC5D,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IAEtB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAiB;IACzC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAoB;IAC5C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAyB;IACxD,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAmB;IAClD,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAC,CAA8B;IAElE,+CAA+C;IAC/C,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAyC;IAEpE,qDAAqD;IACrD,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAgB;gBAGtC,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,cAAc,EACvB,OAAO,EAAE,iBAAiB,EAC1B,OAAO,EAAE,YAAY,GAAG,SAAS,EACjC,QAAQ,EAAE,gBAAgB,EAC1B,kBAAkB,CAAC,EAAE,CAAC,MAAM,OAAO,CAAC,GAAG,SAAS;IAUlD,OAAO,CAAC,iBAAiB;IAYzB;;;;;;OAMG;IACG,OAAO,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,QAAQ,EAAE,OAAO,CAAC,EAAE,UAAU,GAAG,OAAO,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;IAsC5F;;;;;OAKG;IACH,OAAO,CAAC,CAAC,GAAG,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE,SAAS,EAAE,SAAS,CAAC,CAAC,CAAC,GAAG,IAAI;IAQlE;;;;OAIG;IACH,YAAY,CAAC,OAAO,CAAC,EAAE,aAAa,GAAG,MAAM;IAqB7C,8EAA8E;IACxE,MAAM,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,QAAQ,CAAC,GAAG,IAAI,CAAC;IAO1D,iEAAiE;IAC3D,OAAO,CAAC,MAAM,CAAC,EAAE,SAAS,EAAE,KAAK,SAAM,EAAE,MAAM,SAAI,GAAG,OAAO,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;IAWpF,+CAA+C;IACzC,YAAY,IAAI,OAAO,CAAC,MAAM,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC;IASxD;;;OAGG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAI5B,wDAAwD;IACxD,UAAU,IAAI,SAAS,MAAM,EAAE;CAGhC"}
|