groupbullmq 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +65 -0
- package/README.md +186 -0
- package/dist/index.cjs +2520 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +1294 -0
- package/dist/index.d.ts +1294 -0
- package/dist/index.js +2510 -0
- package/dist/index.js.map +1 -0
- package/dist/lua/add-log.lua +25 -0
- package/dist/lua/change-delay.lua +62 -0
- package/dist/lua/check-stalled.lua +86 -0
- package/dist/lua/clean-status.lua +63 -0
- package/dist/lua/cleanup-poisoned-group.lua +46 -0
- package/dist/lua/cleanup.lua +46 -0
- package/dist/lua/complete-and-reserve-next-with-metadata.lua +212 -0
- package/dist/lua/complete-with-metadata.lua +153 -0
- package/dist/lua/complete.lua +51 -0
- package/dist/lua/dead-letter.lua +45 -0
- package/dist/lua/enqueue-batch.lua +155 -0
- package/dist/lua/enqueue.lua +159 -0
- package/dist/lua/get-active-count.lua +6 -0
- package/dist/lua/get-active-jobs.lua +6 -0
- package/dist/lua/get-delayed-count.lua +5 -0
- package/dist/lua/get-delayed-jobs.lua +5 -0
- package/dist/lua/get-unique-groups-count.lua +13 -0
- package/dist/lua/get-unique-groups.lua +15 -0
- package/dist/lua/get-waiting-count.lua +11 -0
- package/dist/lua/get-waiting-jobs.lua +15 -0
- package/dist/lua/heartbeat.lua +22 -0
- package/dist/lua/is-empty.lua +35 -0
- package/dist/lua/promote-delayed-jobs.lua +40 -0
- package/dist/lua/promote-delayed-one.lua +44 -0
- package/dist/lua/promote-staged.lua +70 -0
- package/dist/lua/record-job-result.lua +103 -0
- package/dist/lua/remove.lua +54 -0
- package/dist/lua/reserve-atomic.lua +118 -0
- package/dist/lua/reserve-batch.lua +140 -0
- package/dist/lua/reserve.lua +160 -0
- package/dist/lua/retry.lua +53 -0
- package/package.json +113 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 OpenPanel AB (Carl-Gerhard Lindesvärd)
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Third-Party Code Attribution
|
|
26
|
+
|
|
27
|
+
This software includes code copied from BullMQ (https://github.com/taskforcesh/bullmq), a fantastic library and one of the most popular Redis-based job queue libraries for Node.js. While GroupMQ is inspired by BullMQ's design and concepts, we have also directly copied some code from their implementation.
|
|
28
|
+
|
|
29
|
+
### BullMQ Code Used
|
|
30
|
+
|
|
31
|
+
The following file(s) contain code copied from BullMQ:
|
|
32
|
+
|
|
33
|
+
- `src/async-fifo-queue.ts` - AsyncFifoQueue implementation (copied from BullMQ)
|
|
34
|
+
|
|
35
|
+
### BullMQ License
|
|
36
|
+
|
|
37
|
+
The BullMQ code is licensed under the MIT License:
|
|
38
|
+
|
|
39
|
+
Copyright (c) Taskforce.sh and contributors
|
|
40
|
+
|
|
41
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
42
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
43
|
+
in the Software without restriction, including without limitation the rights
|
|
44
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
45
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
46
|
+
furnished to do so, subject to the following conditions:
|
|
47
|
+
|
|
48
|
+
The above copyright notice and this permission notice shall be included in all
|
|
49
|
+
copies or substantial portions of the Software.
|
|
50
|
+
|
|
51
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
52
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
53
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
54
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
55
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
56
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
57
|
+
SOFTWARE.
|
|
58
|
+
|
|
59
|
+
The original license can be found at: https://github.com/taskforcesh/bullmq/blob/main/LICENSE
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## Fork Maintainer
|
|
64
|
+
|
|
65
|
+
This fork is maintained by [Techspawn Solutions](https://techspawn.com) with additional features including concurrency support for groups, memory leak fixes, and high-performance optimizations.
|
package/README.md
ADDED
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# GroupBullMQ
|
|
2
|
+
|
|
3
|
+
<p align="center">
|
|
4
|
+
<br />
|
|
5
|
+
<strong> High-performance Redis queue with per-group concurrency and strict FIFO ordering</strong>
|
|
6
|
+
<br />
|
|
7
|
+
<br />
|
|
8
|
+
</p>
|
|
9
|
+
|
|
10
|
+
<p align="center">
|
|
11
|
+
<a href="https://www.npmjs.com/package/groupbullmq"><img src="https://img.shields.io/npm/v/groupbullmq.svg?style=flat-square" alt="npm version"></a>
|
|
12
|
+
<a href="https://www.npmjs.com/package/groupbullmq"><img src="https://img.shields.io/npm/dm/groupbullmq.svg?style=flat-square" alt="npm downloads"></a>
|
|
13
|
+
<a href="https://github.com/techspawn/groupbullmq/blob/main/LICENSE"><img src="https://img.shields.io/npm/l/groupbullmq.svg?style=flat-square" alt="license"></a>
|
|
14
|
+
<a href="#"><img src="https://img.shields.io/badge/node-%3E%3D18-brightgreen.svg?style=flat-square" alt="node version"></a>
|
|
15
|
+
</p>
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## 🚀 Why GroupBullMQ?
|
|
20
|
+
|
|
21
|
+
**GroupBullMQ** solves one specific, hard problem: **Processing jobs for specific groups (e.g., customers) in parallel, while maintaining order.**
|
|
22
|
+
|
|
23
|
+
Most queues force a trade-off:
|
|
24
|
+
- **Standard Queues:** Fast parallel processing, but zero ordering guarantees.
|
|
25
|
+
- **FIFO Queues:** Strict ordering, but often locked to single-threaded processing per queue.
|
|
26
|
+
|
|
27
|
+
**GroupBullMQ** gives you the best of both:
|
|
28
|
+
1. **Concurrency per Group**: Process `N` jobs from the *same* group simultaneously.
|
|
29
|
+
2. **Concurrency across Groups**: Process jobs from *different* groups in parallel.
|
|
30
|
+
3. **Ordering**: Jobs generally maintain insertion order within their concurrency window.
|
|
31
|
+
|
|
32
|
+
Perfect for **SaaS**, **E-commerce**, and **High-throughput** systems.
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## ✨ Features
|
|
37
|
+
|
|
38
|
+
- **⚡ Group Concurrency**: Configure how many jobs *per group* run at once.
|
|
39
|
+
- **🔄 Automatic Retries**: Robust retry logic with exponential backoff.
|
|
40
|
+
- **🧹 Auto Cleanup**: Self-managing maintenance of completed and failed jobs.
|
|
41
|
+
- **🚦 Flow Control**: Pause/Resume queues globally.
|
|
42
|
+
- **⏱️ Delayed & Scheduled Jobs**: Run jobs now, later, or on a recurring Cron schedule.
|
|
43
|
+
- **🛡️ Stalled Job Recovery**: Auto-recovers jobs if a worker crashes.
|
|
44
|
+
- **📊 Bull Board Ready**: Visualize your queues with a built-in adapter.
|
|
45
|
+
- **🧩 TypeScript**: First-class type definitions included.
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## 📦 Installation
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
npm install groupbullmq ioredis
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
57
|
+
## 🛠️ Usage
|
|
58
|
+
|
|
59
|
+
### 1. Simple Setup
|
|
60
|
+
|
|
61
|
+
```typescript
|
|
62
|
+
import Redis from 'ioredis';
|
|
63
|
+
import { Queue, Worker } from 'groupbullmq';
|
|
64
|
+
|
|
65
|
+
const redis = new Redis('redis://localhost:6379', { maxRetriesPerRequest: null });
|
|
66
|
+
|
|
67
|
+
// 1. Create a Queue
|
|
68
|
+
const orderQueue = new Queue({
|
|
69
|
+
redis,
|
|
70
|
+
namespace: 'orders',
|
|
71
|
+
groupConcurrency: 5, // 👈 5 orders per customer can be processed at once
|
|
72
|
+
});
|
|
73
|
+
|
|
74
|
+
// 2. Create a Worker
|
|
75
|
+
const orderWorker = new Worker({
|
|
76
|
+
queue: orderQueue,
|
|
77
|
+
concurrency: 20, // 👈 Worker can handle 20 orders total across all customers
|
|
78
|
+
handler: async (job) => {
|
|
79
|
+
console.log(`Processing Order ${job.data.orderId} for Customer ${job.groupId}`);
|
|
80
|
+
await processOrder(job.data);
|
|
81
|
+
},
|
|
82
|
+
});
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### 2. The Power of `groupConcurrency`
|
|
86
|
+
|
|
87
|
+
This is the killer feature. You can control concurrency globally or per-job.
|
|
88
|
+
|
|
89
|
+
**Scenario:** You have a "Premium" customer who needs faster processing than "Free" customers.
|
|
90
|
+
|
|
91
|
+
```typescript
|
|
92
|
+
// Free user: Uses default queue settings (Start 1 job at a time)
|
|
93
|
+
await queue.add({
|
|
94
|
+
groupId: 'free-user-123',
|
|
95
|
+
data: { task: 'slow-processing' },
|
|
96
|
+
// No override, uses default groupConcurrency
|
|
97
|
+
});
|
|
98
|
+
|
|
99
|
+
// Premium user: Overrides to allow 50 concurrent jobs!
|
|
100
|
+
await queue.add({
|
|
101
|
+
groupId: 'premium-corp-999',
|
|
102
|
+
data: { task: 'fast-processing' },
|
|
103
|
+
groupConcurrency: 50, // ⚡ Processing boost for this specific job's group
|
|
104
|
+
});
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
### 3. Advanced Job Types
|
|
108
|
+
|
|
109
|
+
#### Delayed Jobs
|
|
110
|
+
Process a job after a fixed delay.
|
|
111
|
+
```typescript
|
|
112
|
+
await queue.add({
|
|
113
|
+
groupId: 'notifications',
|
|
114
|
+
data: { message: 'Remind me later' },
|
|
115
|
+
delay: 5000, // 5 seconds
|
|
116
|
+
});
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
#### Repeating Jobs (Cron)
|
|
120
|
+
Run a job on a schedule.
|
|
121
|
+
```typescript
|
|
122
|
+
await queue.addRepeating({
|
|
123
|
+
groupId: 'system-maintenance',
|
|
124
|
+
data: { task: 'daily-cleanup' },
|
|
125
|
+
cron: '0 0 * * *', // Every midnight
|
|
126
|
+
});
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
---
|
|
130
|
+
|
|
131
|
+
## ⚙️ Configuration
|
|
132
|
+
|
|
133
|
+
### Queue Options (`new Queue(...)`)
|
|
134
|
+
|
|
135
|
+
| Option | Type | Default | Description |
|
|
136
|
+
| :--- | :--- | :--- | :--- |
|
|
137
|
+
| `redis` | `Redis` | **Required** | ioredis instance. |
|
|
138
|
+
| `namespace` | `string` | **Required** | Unique name for the queue. |
|
|
139
|
+
| `groupConcurrency` | `number` | `1` | **Key Feature.** Max concurrent jobs allowed per group. |
|
|
140
|
+
| `jobTimeoutMs` | `number` | `30000` | How long a job can run before "stalling". |
|
|
141
|
+
| `maxAttemptsDefault` | `number` | `3` | Default retry count for jobs. |
|
|
142
|
+
| `keepCompleted` | `number` | `0` | Number of completed jobs to keep in history. |
|
|
143
|
+
| `keepFailed` | `number` | `0` | Number of failed jobs to keep in history. |
|
|
144
|
+
|
|
145
|
+
### Worker Options (`new Worker(...)`)
|
|
146
|
+
|
|
147
|
+
| Option | Type | Default | Description |
|
|
148
|
+
| :--- | :--- | :--- | :--- |
|
|
149
|
+
| `queue` | `Queue` | **Required** | The queue instance to bind to. |
|
|
150
|
+
| `handler` | `Function` | **Required** | Async function to process jobs. |
|
|
151
|
+
| `concurrency` | `number` | `1` | Total max jobs this worker instance can process in parallel. |
|
|
152
|
+
| `blockingTimeoutSec` | `number` | `2` | IDLE wait time for Redis connection. |
|
|
153
|
+
| `lockDuration` | `number` | `30000` | Lock duration for job processing. |
|
|
154
|
+
|
|
155
|
+
---
|
|
156
|
+
|
|
157
|
+
## 📊 Dashboard (Bull Board)
|
|
158
|
+
|
|
159
|
+
Visualize your queues easily with the official Bull Board.
|
|
160
|
+
|
|
161
|
+
```typescript
|
|
162
|
+
import { createBullBoard } from '@bull-board/api';
|
|
163
|
+
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter'; // Use generic or custom adapter
|
|
164
|
+
import { BullBoardGroupMQAdapter } from 'groupbullmq'; // 👈 Provided adapter
|
|
165
|
+
import { ExpressAdapter } from '@bull-board/express';
|
|
166
|
+
|
|
167
|
+
const serverAdapter = new ExpressAdapter();
|
|
168
|
+
|
|
169
|
+
createBullBoard({
|
|
170
|
+
queues: [new BullBoardGroupMQAdapter(myQueue)],
|
|
171
|
+
serverAdapter,
|
|
172
|
+
});
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## 🤝 Credits & License
|
|
178
|
+
|
|
179
|
+
**Maintainer:** [Techspawn Solutions](https://techspawn.com)
|
|
180
|
+
|
|
181
|
+
This library is built upon the excellent work of the open-source community:
|
|
182
|
+
|
|
183
|
+
* **[GroupMQ](https://github.com/Openpanel-dev/groupmq)** by [OpenPanel.dev](https://openpanel.dev) (Carl-Gerhard Lindesvärd) - *Original architecture and core logic.*
|
|
184
|
+
* **[BullMQ](https://github.com/taskforcesh/bullmq)** by [Taskforce.sh](https://taskforce.sh) - *Inspiration for robust patterns, Lua script structure, and concurrency handling.*
|
|
185
|
+
|
|
186
|
+
License: [MIT](./LICENSE)
|