atomic-queues 1.4.1 → 1.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +279 -286
- package/dist/module/atomic-queues.module.d.ts.map +1 -1
- package/dist/module/atomic-queues.module.js +1 -0
- package/dist/module/atomic-queues.module.js.map +1 -1
- package/dist/services/index.d.ts +1 -0
- package/dist/services/index.d.ts.map +1 -1
- package/dist/services/index.js +1 -0
- package/dist/services/index.js.map +1 -1
- package/dist/services/processor-discovery/processor-discovery.service.d.ts +21 -1
- package/dist/services/processor-discovery/processor-discovery.service.d.ts.map +1 -1
- package/dist/services/processor-discovery/processor-discovery.service.js +97 -3
- package/dist/services/processor-discovery/processor-discovery.service.js.map +1 -1
- package/dist/services/queue-events-manager/queue-events-manager.service.d.ts +23 -2
- package/dist/services/queue-events-manager/queue-events-manager.service.d.ts.map +1 -1
- package/dist/services/queue-events-manager/queue-events-manager.service.js +66 -22
- package/dist/services/queue-events-manager/queue-events-manager.service.js.map +1 -1
- package/dist/services/spawn-queue/index.d.ts +2 -0
- package/dist/services/spawn-queue/index.d.ts.map +1 -0
- package/dist/services/spawn-queue/index.js +18 -0
- package/dist/services/spawn-queue/index.js.map +1 -0
- package/dist/services/spawn-queue/spawn-queue.service.d.ts +119 -0
- package/dist/services/spawn-queue/spawn-queue.service.d.ts.map +1 -0
- package/dist/services/spawn-queue/spawn-queue.service.js +273 -0
- package/dist/services/spawn-queue/spawn-queue.service.js.map +1 -0
- package/dist/services/worker-manager/worker-manager.service.d.ts +18 -3
- package/dist/services/worker-manager/worker-manager.service.d.ts.map +1 -1
- package/dist/services/worker-manager/worker-manager.service.js +44 -20
- package/dist/services/worker-manager/worker-manager.service.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,132 +1,143 @@
|
|
|
1
|
-
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="https://img.shields.io/npm/v/atomic-queues?style=flat-square&color=cb3837" alt="npm version" />
|
|
3
|
+
<img src="https://img.shields.io/badge/NestJS-11-ea2845?style=flat-square&logo=nestjs" alt="NestJS 11" />
|
|
4
|
+
<img src="https://img.shields.io/badge/BullMQ-5-3c873a?style=flat-square" alt="BullMQ 5" />
|
|
5
|
+
<img src="https://img.shields.io/badge/Redis-7-dc382d?style=flat-square&logo=redis&logoColor=white" alt="Redis 7" />
|
|
6
|
+
<img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="MIT License" />
|
|
7
|
+
</p>
|
|
2
8
|
|
|
3
|
-
|
|
9
|
+
<h1 align="center">atomic-queues</h1>
|
|
10
|
+
|
|
11
|
+
<p align="center">
|
|
12
|
+
<strong>Zero-contention, per-entity sequential processing for NestJS.</strong><br/>
|
|
13
|
+
Distributed. Lock-free.
|
|
14
|
+
</p>
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Why atomic-queues?
|
|
19
|
+
|
|
20
|
+
Distributed locks (Redlock, advisory locks, optimistic locking) all share the same fundamental flaw: **contention collapse**. When multiple pods fight for the same lock simultaneously, they spend more time retrying failed acquisitions than doing actual work. The harder you push, the slower they go.
|
|
21
|
+
|
|
22
|
+
**atomic-queues** eliminates contention entirely. Instead of locking, each entity gets its own dedicated BullMQ queue. Operations execute sequentially — back-to-back with zero wasted cycles. There's nothing to contend over.
|
|
23
|
+
|
|
24
|
+
### atomic-queues vs Redlock
|
|
25
|
+
|
|
26
|
+
| | Redlock | atomic-queues |
|
|
27
|
+
|---|---|---|
|
|
28
|
+
| **Architecture** | Distributed mutex (quorum-based) | Per-entity queue (sequential) |
|
|
29
|
+
| **Under contention** | Degrades — retry storms, backoff delays | **Constant** — jobs queue up, execute instantly |
|
|
30
|
+
| **Per-entity throughput** | ~20-50 ops/s (heavy contention) | **~200-300 ops/s** (queue-bound, no contention) |
|
|
31
|
+
| **Failure mode** | Silent double-execution (clock drift) | Job stuck in queue (visible, retryable) |
|
|
32
|
+
| **Split-brain risk** | Yes (timing assumptions) | **Impossible** (serial queue) |
|
|
33
|
+
| **Warm-path overhead** | 5-7ms per op (acquire + release) | **0ms** (in-memory hot cache) |
|
|
34
|
+
| **Cold-start** | None | ~2-3ms one-time per entity |
|
|
35
|
+
| **Multi-pod scaling** | Contention increases with pods | **Throughput increases with pods** |
|
|
4
36
|
|
|
5
37
|
---
|
|
6
38
|
|
|
7
39
|
## Table of Contents
|
|
8
40
|
|
|
9
|
-
- [
|
|
10
|
-
- [
|
|
11
|
-
- [The Per-Entity Queue Architecture](#the-per-entity-queue-architecture)
|
|
41
|
+
- [Why atomic-queues?](#why-atomic-queues)
|
|
42
|
+
- [How It Works](#how-it-works)
|
|
12
43
|
- [Installation](#installation)
|
|
13
44
|
- [Quick Start](#quick-start)
|
|
14
|
-
- [Commands
|
|
45
|
+
- [Commands & Decorators](#commands--decorators)
|
|
15
46
|
- [Configuration](#configuration)
|
|
47
|
+
- [Distributed Worker Lifecycle](#distributed-worker-lifecycle)
|
|
16
48
|
- [Complete Example](#complete-example)
|
|
17
49
|
- [Advanced: Custom Worker Processors](#advanced-custom-worker-processors)
|
|
50
|
+
- [Performance](#performance)
|
|
18
51
|
- [License](#license)
|
|
19
52
|
|
|
20
53
|
---
|
|
21
54
|
|
|
22
|
-
##
|
|
55
|
+
## How It Works
|
|
23
56
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
Rather than relying on distributed locks—which introduce contention, latency degradation, and complex failure modes—this library implements a **per-entity queue architecture** where each entity (user account, order, document) has its own dedicated processing queue and worker.
|
|
27
|
-
|
|
28
|
-
---
|
|
29
|
-
|
|
30
|
-
## The Concurrency Problem
|
|
31
|
-
|
|
32
|
-
Consider a banking system where a user with a $100 balance submits two concurrent $80 withdrawal requests:
|
|
33
|
-
|
|
34
|
-
```
|
|
35
|
-
Time Request A Request B Database State
|
|
36
|
-
─────────────────────────────────────────────────────────────────────────────────
|
|
37
|
-
T₀ SELECT balance → $100 SELECT balance → $100 balance = $100
|
|
38
|
-
T₁ CHECK: $100 >= $80 ✓ CHECK: $100 >= $80 ✓
|
|
39
|
-
T₂ UPDATE: balance = $20 UPDATE: balance = $20 balance = $20
|
|
40
|
-
T₃ UPDATE: balance = -$60 balance = -$60
|
|
41
|
-
─────────────────────────────────────────────────────────────────────────────────
|
|
42
|
-
Result: Both withdrawals succeed. Balance becomes -$60. Integrity violated.
|
|
43
|
-
```
|
|
57
|
+
### The Problem
|
|
44
58
|
|
|
45
|
-
|
|
59
|
+
Every distributed system eventually hits this:
|
|
46
60
|
|
|
47
61
|
```
|
|
48
|
-
Time
|
|
49
|
-
|
|
50
|
-
T₀
|
|
51
|
-
T₁
|
|
52
|
-
T₂
|
|
53
|
-
|
|
54
|
-
|
|
62
|
+
Time Request A Request B Database
|
|
63
|
+
──────────────────────────────────────────────────────────────────────────
|
|
64
|
+
T₀ SELECT balance → $100 SELECT balance → $100 $100
|
|
65
|
+
T₁ CHECK: $100 ≥ $80 ✓ CHECK: $100 ≥ $80 ✓
|
|
66
|
+
T₂ UPDATE: $100 − $80 = $20 $20
|
|
67
|
+
T₃ UPDATE: $100 − $80 = $20 −$60
|
|
68
|
+
──────────────────────────────────────────────────────────────────────────
|
|
69
|
+
Result: Balance is −$60. Both withdrawals succeed. Integrity violated.
|
|
55
70
|
```
|
|
56
71
|
|
|
57
|
-
|
|
72
|
+
### The Solution
|
|
58
73
|
|
|
59
|
-
|
|
74
|
+
atomic-queues routes operations through per-entity queues. Same entity → same queue → sequential execution. Different entities → parallel queues → full throughput.
|
|
60
75
|
|
|
61
76
|
```
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
77
|
+
┌─────────────────────────────────────────────────┐
|
|
78
|
+
Request A ─┐ │ Entity: account-42 │
|
|
79
|
+
│ │ ┌──────┐ ┌──────┐ ┌──────┐ │
|
|
80
|
+
Request B ─┼─► Route ─┼─►│ Op 1 │─►│ Op 2 │─►│ Op 3 │─► [Worker] ──┐ │
|
|
81
|
+
│ │ └──────┘ └──────┘ └──────┘ │ │
|
|
82
|
+
Request C ─┘ │ Sequential ◄─────────────┘ │
|
|
83
|
+
└─────────────────────────────────────────────────┘
|
|
84
|
+
|
|
85
|
+
┌─────────────────────────────────────────────────┐
|
|
86
|
+
Request D ─┐ │ Entity: account-99 │
|
|
87
|
+
│ │ ┌──────┐ ┌──────┐ │
|
|
88
|
+
Request E ─┼─► Route ─┼─►│ Op 1 │─►│ Op 2 │─────────► [Worker] ──┐ │
|
|
89
|
+
│ │ └──────┘ └──────┘ │ │
|
|
90
|
+
Request F ─┘ │ Sequential ◄───────────┘ │
|
|
91
|
+
└─────────────────────────────────────────────────┘
|
|
92
|
+
|
|
93
|
+
▲ These two queues run in PARALLEL across pods ▲
|
|
70
94
|
```
|
|
71
95
|
|
|
72
|
-
**Key
|
|
73
|
-
-
|
|
74
|
-
-
|
|
75
|
-
-
|
|
76
|
-
-
|
|
96
|
+
**Key properties:**
|
|
97
|
+
- **One worker per entity** — enforced via Redis heartbeat TTL. No duplicates, ever.
|
|
98
|
+
- **Auto-spawn** — workers materialize when jobs arrive, on the pod that sees them first.
|
|
99
|
+
- **Auto-terminate** — idle workers shut down after a configurable timeout.
|
|
100
|
+
- **Self-healing** — node failure → heartbeat expires → worker respawns on a healthy pod.
|
|
101
|
+
- **Distributed** — workers spread across all pods via atomic `SET NX` claim. No leader election, no single point of failure.
|
|
77
102
|
|
|
78
103
|
---
|
|
79
104
|
|
|
80
105
|
## Installation
|
|
81
106
|
|
|
82
107
|
```bash
|
|
108
|
+
# npm
|
|
83
109
|
npm install atomic-queues bullmq ioredis
|
|
110
|
+
|
|
111
|
+
# pnpm
|
|
112
|
+
pnpm add atomic-queues bullmq ioredis
|
|
113
|
+
|
|
114
|
+
# yarn
|
|
115
|
+
yarn add atomic-queues bullmq ioredis
|
|
84
116
|
```
|
|
85
117
|
|
|
118
|
+
**Peer dependencies:** NestJS 10+, `@nestjs/cqrs` (optional — for auto-routing commands/queries)
|
|
119
|
+
|
|
86
120
|
---
|
|
87
121
|
|
|
88
122
|
## Quick Start
|
|
89
123
|
|
|
90
124
|
### 1. Configure the Module
|
|
91
125
|
|
|
92
|
-
The `entities` configuration is **optional**. Choose the approach that fits your needs:
|
|
93
|
-
|
|
94
|
-
#### Option A: Minimal Setup (uses default naming)
|
|
95
|
-
|
|
96
126
|
```typescript
|
|
97
127
|
import { Module } from '@nestjs/common';
|
|
128
|
+
import { CqrsModule } from '@nestjs/cqrs';
|
|
98
129
|
import { AtomicQueuesModule } from 'atomic-queues';
|
|
99
130
|
|
|
100
131
|
@Module({
|
|
101
132
|
imports: [
|
|
133
|
+
CqrsModule,
|
|
102
134
|
AtomicQueuesModule.forRoot({
|
|
103
135
|
redis: { host: 'localhost', port: 6379 },
|
|
104
136
|
keyPrefix: 'myapp',
|
|
105
|
-
enableCronManager: true,
|
|
106
|
-
// No entities config needed! Uses default naming:
|
|
107
|
-
// Queue: {keyPrefix}:{entityType}:{entityId}:queue
|
|
108
|
-
// Worker: {keyPrefix}:{entityType}:{entityId}:worker
|
|
109
|
-
}),
|
|
110
|
-
],
|
|
111
|
-
})
|
|
112
|
-
export class AppModule {}
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
#### Option B: Custom Queue/Worker Naming (via entities config)
|
|
116
|
-
|
|
117
|
-
```typescript
|
|
118
|
-
@Module({
|
|
119
|
-
imports: [
|
|
120
|
-
AtomicQueuesModule.forRoot({
|
|
121
|
-
redis: { host: 'localhost', port: 6379 },
|
|
122
|
-
keyPrefix: 'myapp',
|
|
123
|
-
enableCronManager: true,
|
|
124
|
-
|
|
125
|
-
// Optional: Define custom naming and settings per entity type
|
|
126
137
|
entities: {
|
|
127
138
|
account: {
|
|
128
|
-
queueName: (id) =>
|
|
129
|
-
workerName: (id) =>
|
|
139
|
+
queueName: (id) => `account-${id}-queue`,
|
|
140
|
+
workerName: (id) => `account-${id}-worker`,
|
|
130
141
|
maxWorkersPerEntity: 1,
|
|
131
142
|
idleTimeoutSeconds: 15,
|
|
132
143
|
},
|
|
@@ -137,28 +148,9 @@ export class AppModule {}
|
|
|
137
148
|
export class AppModule {}
|
|
138
149
|
```
|
|
139
150
|
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
For advanced use cases, define a processor class instead of entities config:
|
|
143
|
-
|
|
144
|
-
```typescript
|
|
145
|
-
@WorkerProcessor({
|
|
146
|
-
entityType: 'account',
|
|
147
|
-
queueName: (id) => `${id}-queue`,
|
|
148
|
-
workerName: (id) => `${id}-worker`,
|
|
149
|
-
maxWorkersPerEntity: 1,
|
|
150
|
-
idleTimeoutSeconds: 15,
|
|
151
|
-
})
|
|
152
|
-
@Injectable()
|
|
153
|
-
export class AccountProcessor {}
|
|
154
|
-
```
|
|
155
|
-
|
|
156
|
-
> **When to use each:**
|
|
157
|
-
> - **Option A**: Default naming works for you
|
|
158
|
-
> - **Option B**: Need custom naming but no custom job handling logic
|
|
159
|
-
> - **Option C**: Need custom naming AND custom `@JobHandler` methods
|
|
151
|
+
> **Tip:** The `entities` config is optional. Without it, default naming applies: `{keyPrefix}:{entityType}:{entityId}:queue`.
|
|
160
152
|
|
|
161
|
-
### 2.
|
|
153
|
+
### 2. Define Commands
|
|
162
154
|
|
|
163
155
|
```typescript
|
|
164
156
|
import { QueueEntity, QueueEntityId } from 'atomic-queues';
|
|
@@ -168,7 +160,6 @@ export class WithdrawCommand {
|
|
|
168
160
|
constructor(
|
|
169
161
|
@QueueEntityId() public readonly accountId: string,
|
|
170
162
|
public readonly amount: number,
|
|
171
|
-
public readonly transactionId: string,
|
|
172
163
|
) {}
|
|
173
164
|
}
|
|
174
165
|
|
|
@@ -177,35 +168,29 @@ export class DepositCommand {
|
|
|
177
168
|
constructor(
|
|
178
169
|
@QueueEntityId() public readonly accountId: string,
|
|
179
170
|
public readonly amount: number,
|
|
180
|
-
public readonly source: string,
|
|
181
171
|
) {}
|
|
182
172
|
}
|
|
183
173
|
```
|
|
184
174
|
|
|
185
|
-
### 3.
|
|
175
|
+
### 3. Write Handlers (standard @nestjs/cqrs)
|
|
186
176
|
|
|
187
177
|
```typescript
|
|
188
178
|
import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
|
|
189
|
-
import { WithdrawCommand } from './commands';
|
|
190
179
|
|
|
191
180
|
@CommandHandler(WithdrawCommand)
|
|
192
181
|
export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
|
|
193
|
-
constructor(private readonly
|
|
194
|
-
|
|
195
|
-
async execute(
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
const account = await this.accountRepo.findById(accountId);
|
|
200
|
-
|
|
182
|
+
constructor(private readonly repo: AccountRepository) {}
|
|
183
|
+
|
|
184
|
+
async execute({ accountId, amount }: WithdrawCommand) {
|
|
185
|
+
// SAFE: No race conditions. Sequential execution per account.
|
|
186
|
+
const account = await this.repo.findById(accountId);
|
|
187
|
+
|
|
201
188
|
if (account.balance < amount) {
|
|
202
189
|
throw new InsufficientFundsError(accountId, account.balance, amount);
|
|
203
190
|
}
|
|
204
|
-
|
|
191
|
+
|
|
205
192
|
account.balance -= amount;
|
|
206
|
-
await this.
|
|
207
|
-
|
|
208
|
-
return { success: true, newBalance: account.balance };
|
|
193
|
+
await this.repo.save(account);
|
|
209
194
|
}
|
|
210
195
|
}
|
|
211
196
|
```
|
|
@@ -215,137 +200,173 @@ export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
|
|
|
215
200
|
```typescript
|
|
216
201
|
import { Injectable } from '@nestjs/common';
|
|
217
202
|
import { QueueBus } from 'atomic-queues';
|
|
218
|
-
import { WithdrawCommand, DepositCommand } from './commands';
|
|
219
203
|
|
|
220
204
|
@Injectable()
|
|
221
205
|
export class AccountService {
|
|
222
206
|
constructor(private readonly queueBus: QueueBus) {}
|
|
223
207
|
|
|
224
|
-
async withdraw(accountId: string, amount: number
|
|
225
|
-
|
|
226
|
-
await this.queueBus.enqueue(new WithdrawCommand(accountId, amount, transactionId));
|
|
227
|
-
}
|
|
228
|
-
|
|
229
|
-
async deposit(accountId: string, amount: number, source: string) {
|
|
230
|
-
await this.queueBus.enqueue(new DepositCommand(accountId, amount, source));
|
|
208
|
+
async withdraw(accountId: string, amount: number) {
|
|
209
|
+
await this.queueBus.enqueue(new WithdrawCommand(accountId, amount));
|
|
231
210
|
}
|
|
232
211
|
}
|
|
233
212
|
```
|
|
234
213
|
|
|
235
|
-
**That's it
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
214
|
+
**That's it.** The library automatically:
|
|
215
|
+
1. Creates a queue for each `accountId` when jobs arrive
|
|
216
|
+
2. Spawns a worker (spread across pods) to process jobs sequentially
|
|
217
|
+
3. Routes jobs to the correct `@CommandHandler` via CQRS
|
|
218
|
+
4. Terminates idle workers after the configured timeout
|
|
219
|
+
5. Self-heals if a pod dies (heartbeat expires → respawn elsewhere)
|
|
240
220
|
|
|
241
221
|
---
|
|
242
222
|
|
|
243
|
-
## Commands
|
|
223
|
+
## Commands & Decorators
|
|
244
224
|
|
|
245
|
-
###
|
|
225
|
+
### `@QueueEntity(entityType)`
|
|
246
226
|
|
|
247
|
-
Marks a command class for queue routing.
|
|
227
|
+
Marks a command/query class for queue routing.
|
|
248
228
|
|
|
249
229
|
```typescript
|
|
250
230
|
@QueueEntity('account')
|
|
251
231
|
export class TransferCommand { ... }
|
|
252
232
|
```
|
|
253
233
|
|
|
254
|
-
###
|
|
234
|
+
### `@QueueEntityId()`
|
|
255
235
|
|
|
256
|
-
Marks
|
|
236
|
+
Marks the property that contains the entity ID. One per class.
|
|
257
237
|
|
|
258
238
|
```typescript
|
|
259
239
|
@QueueEntity('account')
|
|
260
240
|
export class TransferCommand {
|
|
261
241
|
constructor(
|
|
262
|
-
@QueueEntityId() public readonly
|
|
242
|
+
@QueueEntityId() public readonly accountId: string, // Routes to this account's queue
|
|
263
243
|
public readonly targetAccountId: string,
|
|
264
244
|
public readonly amount: number,
|
|
265
245
|
) {}
|
|
266
246
|
}
|
|
267
247
|
```
|
|
268
248
|
|
|
269
|
-
###
|
|
249
|
+
### `@WorkerProcessor(options)`
|
|
270
250
|
|
|
271
|
-
|
|
251
|
+
Optional. Define a processor class for custom job handling on top of CQRS auto-routing.
|
|
272
252
|
|
|
273
253
|
```typescript
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
}
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
@
|
|
284
|
-
|
|
285
|
-
constructor(
|
|
286
|
-
public readonly accountId: string, // Automatically used
|
|
287
|
-
public readonly amount: number,
|
|
288
|
-
) {}
|
|
254
|
+
@WorkerProcessor({
|
|
255
|
+
entityType: 'account',
|
|
256
|
+
queueName: (id) => `account-${id}-queue`,
|
|
257
|
+
workerName: (id) => `account-${id}-worker`,
|
|
258
|
+
maxWorkersPerEntity: 1,
|
|
259
|
+
idleTimeoutSeconds: 15,
|
|
260
|
+
})
|
|
261
|
+
@Injectable()
|
|
262
|
+
export class AccountProcessor {
|
|
263
|
+
@JobHandler('special-audit')
|
|
264
|
+
async handleAudit(job: Job, entityId: string) { ... }
|
|
289
265
|
}
|
|
290
266
|
```
|
|
291
267
|
|
|
268
|
+
### `@JobHandler(jobName)` / `@JobHandler('*')`
|
|
269
|
+
|
|
270
|
+
Custom job handlers on a `@WorkerProcessor`. The wildcard `'*'` catches anything not matched by a specific handler.
|
|
271
|
+
|
|
292
272
|
---
|
|
293
273
|
|
|
294
274
|
## Configuration
|
|
295
275
|
|
|
296
276
|
```typescript
|
|
297
277
|
AtomicQueuesModule.forRoot({
|
|
278
|
+
// ── Redis connection ──────────────────────────────────────
|
|
298
279
|
redis: {
|
|
299
|
-
host: '
|
|
280
|
+
host: 'redis',
|
|
300
281
|
port: 6379,
|
|
301
|
-
password: 'secret',
|
|
282
|
+
password: 'secret', // optional
|
|
302
283
|
},
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
284
|
+
|
|
285
|
+
// ── Global settings ───────────────────────────────────────
|
|
286
|
+
keyPrefix: 'myapp', // Redis key namespace (default: 'aq')
|
|
287
|
+
enableCronManager: true, // Legacy cron-based scaling (optional)
|
|
288
|
+
cronInterval: 5000, // Cron tick interval in ms
|
|
289
|
+
|
|
290
|
+
// ── Worker defaults ───────────────────────────────────────
|
|
308
291
|
workerDefaults: {
|
|
309
|
-
concurrency: 1,
|
|
310
|
-
stalledInterval: 1000,
|
|
311
|
-
lockDuration: 30000,
|
|
312
|
-
heartbeatTTL: 3,
|
|
292
|
+
concurrency: 1, // Jobs processed concurrently per worker
|
|
293
|
+
stalledInterval: 1000, // ms between stalled-job checks
|
|
294
|
+
lockDuration: 30000, // ms a job is locked during processing
|
|
295
|
+
heartbeatTTL: 3, // Heartbeat key TTL in seconds
|
|
313
296
|
},
|
|
314
|
-
|
|
315
|
-
//
|
|
316
|
-
// If omitted, uses default naming: {keyPrefix}:{entityType}:{entityId}:queue/worker
|
|
297
|
+
|
|
298
|
+
// ── Per-entity configuration (optional) ───────────────────
|
|
317
299
|
entities: {
|
|
318
300
|
account: {
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
workerName: (id) => `${id}-worker`,
|
|
301
|
+
queueName: (id) => `account-${id}-queue`,
|
|
302
|
+
workerName: (id) => `account-${id}-worker`,
|
|
322
303
|
maxWorkersPerEntity: 1,
|
|
323
304
|
idleTimeoutSeconds: 15,
|
|
324
|
-
|
|
325
|
-
workerConfig: {
|
|
305
|
+
defaultEntityId: 'accountId',
|
|
306
|
+
workerConfig: { // Override workerDefaults per entity
|
|
326
307
|
concurrency: 1,
|
|
327
308
|
lockDuration: 60000,
|
|
328
309
|
},
|
|
329
310
|
},
|
|
330
|
-
order: {
|
|
331
|
-
defaultEntityId: 'orderId',
|
|
332
|
-
queueName: (id) => `order-${id}-queue`,
|
|
333
|
-
idleTimeoutSeconds: 30,
|
|
334
|
-
},
|
|
335
311
|
},
|
|
336
312
|
});
|
|
337
313
|
```
|
|
338
314
|
|
|
339
315
|
---
|
|
340
316
|
|
|
317
|
+
## Distributed Worker Lifecycle
|
|
318
|
+
|
|
319
|
+
Workers in atomic-queues have a fully automated lifecycle, distributed across all pods with no leader election:
|
|
320
|
+
|
|
321
|
+
```
|
|
322
|
+
Job arrives SET NX claim
|
|
323
|
+
on any pod ──────► ┌──────────────────────┐
|
|
324
|
+
│ Pod claims worker? │
|
|
325
|
+
└──────┬───────┬───────┘
|
|
326
|
+
YES │ │ NO (another pod won)
|
|
327
|
+
▼ ▼
|
|
328
|
+
┌────────┐ ┌──────────────┐
|
|
329
|
+
│ Spawn │ │ Wait — other │
|
|
330
|
+
│ worker │ │ pod handles │
|
|
331
|
+
│ locally│ └──────────────┘
|
|
332
|
+
└───┬────┘
|
|
333
|
+
▼
|
|
334
|
+
┌──────────────┐
|
|
335
|
+
│ Processing │◄──── Heartbeat refresh (pipeline)
|
|
336
|
+
│ jobs back- │ every 1s (1 Redis round-trip)
|
|
337
|
+
│ to-back │
|
|
338
|
+
└──────┬───────┘
|
|
339
|
+
│ No jobs for idleTimeoutSeconds
|
|
340
|
+
▼
|
|
341
|
+
┌──────────────┐
|
|
342
|
+
│ Idle sweep │──── Hot cache eviction
|
|
343
|
+
│ closes │ Heartbeat keys cleaned up
|
|
344
|
+
│ worker │
|
|
345
|
+
└──────────────┘
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
### Hot Cache (v1.5.0+)
|
|
349
|
+
|
|
350
|
+
After a worker is confirmed alive, subsequent job arrivals for that entity hit an **in-memory cache** — zero Redis calls on the warm path. This eliminates the per-job Redis overhead that plagues lock-based approaches.
|
|
351
|
+
|
|
352
|
+
| Path | Redis calls | When |
|
|
353
|
+
|---|---|---|
|
|
354
|
+
| **Hot** (cache hit) | 0 | Worker known alive |
|
|
355
|
+
| **Warm** (cache miss) | 1 (`EXISTS`) | First time seeing entity |
|
|
356
|
+
| **Cold** (no worker) | 1 (`SET NX`) | Worker needs creation |
|
|
357
|
+
|
|
358
|
+
### SpawnQueueService (v1.4.2+)
|
|
359
|
+
|
|
360
|
+
For multi-pod deployments, the `SpawnQueueService` distributes worker creation across all pods via a shared BullMQ spawn queue. In v1.5.0, the **direct local spawn** path bypasses this queue entirely — the pod that first sees a job for a new entity claims it with an atomic `SET NX` and spawns the worker locally, saving hundreds of milliseconds.
|
|
361
|
+
|
|
362
|
+
---
|
|
363
|
+
|
|
341
364
|
## Complete Example
|
|
342
365
|
|
|
343
|
-
A banking service
|
|
366
|
+
A banking service with withdrawals, deposits, and cross-account transfers:
|
|
344
367
|
|
|
345
368
|
```typescript
|
|
346
|
-
//
|
|
347
|
-
// app.module.ts
|
|
348
|
-
// ─────────────────────────────────────────────────────────────────
|
|
369
|
+
// ── Module ──────────────────────────────────────────────
|
|
349
370
|
import { Module } from '@nestjs/common';
|
|
350
371
|
import { CqrsModule } from '@nestjs/cqrs';
|
|
351
372
|
import { AtomicQueuesModule } from 'atomic-queues';
|
|
@@ -354,19 +375,14 @@ import { AtomicQueuesModule } from 'atomic-queues';
|
|
|
354
375
|
imports: [
|
|
355
376
|
CqrsModule,
|
|
356
377
|
AtomicQueuesModule.forRoot({
|
|
357
|
-
redis: { host: '
|
|
378
|
+
redis: { host: 'redis', port: 6379 },
|
|
358
379
|
keyPrefix: 'banking',
|
|
359
|
-
enableCronManager: true,
|
|
360
380
|
entities: {
|
|
361
381
|
account: {
|
|
362
|
-
queueName: (id) =>
|
|
363
|
-
workerName: (id) =>
|
|
382
|
+
queueName: (id) => `account-${id}-queue`,
|
|
383
|
+
workerName: (id) => `account-${id}-worker`,
|
|
364
384
|
maxWorkersPerEntity: 1,
|
|
365
385
|
idleTimeoutSeconds: 15,
|
|
366
|
-
workerConfig: {
|
|
367
|
-
concurrency: 1,
|
|
368
|
-
lockDuration: 60000,
|
|
369
|
-
},
|
|
370
386
|
},
|
|
371
387
|
},
|
|
372
388
|
}),
|
|
@@ -377,13 +393,10 @@ import { AtomicQueuesModule } from 'atomic-queues';
|
|
|
377
393
|
DepositHandler,
|
|
378
394
|
TransferHandler,
|
|
379
395
|
],
|
|
380
|
-
controllers: [AccountController],
|
|
381
396
|
})
|
|
382
|
-
export class
|
|
397
|
+
export class BankingModule {}
|
|
383
398
|
|
|
384
|
-
//
|
|
385
|
-
// commands/withdraw.command.ts
|
|
386
|
-
// ─────────────────────────────────────────────────────────────────
|
|
399
|
+
// ── Commands ────────────────────────────────────────────
|
|
387
400
|
import { QueueEntity, QueueEntityId } from 'atomic-queues';
|
|
388
401
|
|
|
389
402
|
@QueueEntity('account')
|
|
@@ -395,11 +408,6 @@ export class WithdrawCommand {
|
|
|
395
408
|
) {}
|
|
396
409
|
}
|
|
397
410
|
|
|
398
|
-
// ─────────────────────────────────────────────────────────────────
|
|
399
|
-
// commands/deposit.command.ts
|
|
400
|
-
// ─────────────────────────────────────────────────────────────────
|
|
401
|
-
import { QueueEntity, QueueEntityId } from 'atomic-queues';
|
|
402
|
-
|
|
403
411
|
@QueueEntity('account')
|
|
404
412
|
export class DepositCommand {
|
|
405
413
|
constructor(
|
|
@@ -409,123 +417,72 @@ export class DepositCommand {
|
|
|
409
417
|
) {}
|
|
410
418
|
}
|
|
411
419
|
|
|
412
|
-
// ─────────────────────────────────────────────────────────────────
|
|
413
|
-
// commands/transfer.command.ts
|
|
414
|
-
// ─────────────────────────────────────────────────────────────────
|
|
415
|
-
import { QueueEntity, QueueEntityId } from 'atomic-queues';
|
|
416
|
-
|
|
417
420
|
@QueueEntity('account')
|
|
418
421
|
export class TransferCommand {
|
|
419
422
|
constructor(
|
|
420
|
-
@QueueEntityId() public readonly accountId: string,
|
|
423
|
+
@QueueEntityId() public readonly accountId: string,
|
|
421
424
|
public readonly toAccountId: string,
|
|
422
425
|
public readonly amount: number,
|
|
423
|
-
public readonly transactionId: string,
|
|
424
426
|
) {}
|
|
425
427
|
}
|
|
426
428
|
|
|
427
|
-
//
|
|
428
|
-
// handlers/withdraw.handler.ts
|
|
429
|
-
// ─────────────────────────────────────────────────────────────────
|
|
429
|
+
// ── Handlers ────────────────────────────────────────────
|
|
430
430
|
import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
|
|
431
|
-
import { WithdrawCommand } from '../commands';
|
|
432
431
|
|
|
433
432
|
@CommandHandler(WithdrawCommand)
|
|
434
433
|
export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
|
|
435
|
-
constructor(private readonly
|
|
436
|
-
|
|
437
|
-
async execute(
|
|
438
|
-
const
|
|
439
|
-
|
|
440
|
-
// SAFE: Sequential execution per account
|
|
441
|
-
const account = await this.accountRepo.findById(accountId);
|
|
442
|
-
|
|
443
|
-
if (account.balance < amount) {
|
|
444
|
-
throw new InsufficientFundsError(accountId, account.balance, amount);
|
|
445
|
-
}
|
|
446
|
-
|
|
434
|
+
constructor(private readonly repo: AccountRepository) {}
|
|
435
|
+
|
|
436
|
+
async execute({ accountId, amount }: WithdrawCommand) {
|
|
437
|
+
const account = await this.repo.findById(accountId);
|
|
438
|
+
if (account.balance < amount) throw new InsufficientFundsError();
|
|
447
439
|
account.balance -= amount;
|
|
448
|
-
await this.
|
|
449
|
-
|
|
450
|
-
return { success: true, newBalance: account.balance };
|
|
440
|
+
await this.repo.save(account);
|
|
451
441
|
}
|
|
452
442
|
}
|
|
453
443
|
|
|
454
|
-
// ─────────────────────────────────────────────────────────────────
|
|
455
|
-
// handlers/transfer.handler.ts
|
|
456
|
-
// ─────────────────────────────────────────────────────────────────
|
|
457
|
-
import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
|
|
458
|
-
import { TransferCommand, DepositCommand } from '../commands';
|
|
459
|
-
import { QueueBus } from 'atomic-queues';
|
|
460
|
-
|
|
461
444
|
@CommandHandler(TransferCommand)
|
|
462
445
|
export class TransferHandler implements ICommandHandler<TransferCommand> {
|
|
463
446
|
constructor(
|
|
464
|
-
private readonly
|
|
447
|
+
private readonly repo: AccountRepository,
|
|
465
448
|
private readonly queueBus: QueueBus,
|
|
466
449
|
) {}
|
|
467
450
|
|
|
468
|
-
async execute(
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
const source = await this.accountRepo.findById(accountId);
|
|
473
|
-
if (source.balance < amount) {
|
|
474
|
-
throw new InsufficientFundsError(accountId, source.balance, amount);
|
|
475
|
-
}
|
|
476
|
-
|
|
451
|
+
async execute({ accountId, toAccountId, amount }: TransferCommand) {
|
|
452
|
+
// Debit source (we're in source account's queue — safe)
|
|
453
|
+
const source = await this.repo.findById(accountId);
|
|
454
|
+
if (source.balance < amount) throw new InsufficientFundsError();
|
|
477
455
|
source.balance -= amount;
|
|
478
|
-
await this.
|
|
479
|
-
|
|
480
|
-
// Credit destination (enqueued to destination's queue)
|
|
481
|
-
await this.queueBus.enqueue(
|
|
482
|
-
toAccountId,
|
|
483
|
-
|
|
484
|
-
`transfer:${accountId}`,
|
|
485
|
-
));
|
|
486
|
-
|
|
487
|
-
return { success: true };
|
|
456
|
+
await this.repo.save(source);
|
|
457
|
+
|
|
458
|
+
// Credit destination (enqueued to destination's queue — also safe)
|
|
459
|
+
await this.queueBus.enqueue(
|
|
460
|
+
new DepositCommand(toAccountId, amount, `transfer:${accountId}`),
|
|
461
|
+
);
|
|
488
462
|
}
|
|
489
463
|
}
|
|
490
464
|
|
|
491
|
-
//
|
|
492
|
-
// account.controller.ts
|
|
493
|
-
// ─────────────────────────────────────────────────────────────────
|
|
465
|
+
// ── Controller ──────────────────────────────────────────
|
|
494
466
|
import { Controller, Post, Body, Param } from '@nestjs/common';
|
|
495
467
|
import { QueueBus } from 'atomic-queues';
|
|
496
|
-
import { WithdrawCommand, TransferCommand } from './commands';
|
|
497
|
-
import { v4 as uuid } from 'uuid';
|
|
498
468
|
|
|
499
469
|
@Controller('accounts')
|
|
500
470
|
export class AccountController {
|
|
501
471
|
constructor(private readonly queueBus: QueueBus) {}
|
|
502
472
|
|
|
503
|
-
@Post(':
|
|
504
|
-
async withdraw(
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
) {
|
|
508
|
-
const transactionId = uuid();
|
|
509
|
-
|
|
510
|
-
await this.queueBus.enqueue(
|
|
511
|
-
new WithdrawCommand(accountId, body.amount, transactionId)
|
|
512
|
-
);
|
|
513
|
-
|
|
514
|
-
return { queued: true, transactionId };
|
|
473
|
+
@Post(':id/withdraw')
|
|
474
|
+
async withdraw(@Param('id') id: string, @Body() body: { amount: number }) {
|
|
475
|
+
await this.queueBus.enqueue(new WithdrawCommand(id, body.amount, uuid()));
|
|
476
|
+
return { queued: true };
|
|
515
477
|
}
|
|
516
478
|
|
|
517
|
-
@Post(':
|
|
479
|
+
@Post(':id/transfer')
|
|
518
480
|
async transfer(
|
|
519
|
-
@Param('
|
|
520
|
-
@Body() body: {
|
|
481
|
+
@Param('id') id: string,
|
|
482
|
+
@Body() body: { to: string; amount: number },
|
|
521
483
|
) {
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
await this.queueBus.enqueue(
|
|
525
|
-
new TransferCommand(accountId, body.toAccountId, body.amount, transactionId)
|
|
526
|
-
);
|
|
527
|
-
|
|
528
|
-
return { queued: true, transactionId };
|
|
484
|
+
await this.queueBus.enqueue(new TransferCommand(id, body.to, body.amount));
|
|
485
|
+
return { queued: true };
|
|
529
486
|
}
|
|
530
487
|
}
|
|
531
488
|
```
|
|
@@ -534,7 +491,7 @@ export class AccountController {
|
|
|
534
491
|
|
|
535
492
|
## Advanced: Custom Worker Processors
|
|
536
493
|
|
|
537
|
-
For
|
|
494
|
+
For cases where CQRS auto-routing isn't enough, define a `@WorkerProcessor` with explicit `@JobHandler` methods:
|
|
538
495
|
|
|
539
496
|
```typescript
|
|
540
497
|
import { Injectable } from '@nestjs/common';
|
|
@@ -543,28 +500,64 @@ import { Job } from 'bullmq';
|
|
|
543
500
|
|
|
544
501
|
@WorkerProcessor({
|
|
545
502
|
entityType: 'account',
|
|
546
|
-
queueName: (id) =>
|
|
547
|
-
workerName: (id) =>
|
|
503
|
+
queueName: (id) => `account-${id}-queue`,
|
|
504
|
+
workerName: (id) => `account-${id}-worker`,
|
|
548
505
|
maxWorkersPerEntity: 1,
|
|
549
506
|
idleTimeoutSeconds: 15,
|
|
550
507
|
})
|
|
551
508
|
@Injectable()
|
|
552
509
|
export class AccountProcessor {
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
// Custom logic here
|
|
510
|
+
@JobHandler('high-priority-audit')
|
|
511
|
+
async handleAudit(job: Job, entityId: string) {
|
|
512
|
+
// Specific handler for this job type
|
|
557
513
|
}
|
|
558
514
|
|
|
559
|
-
// Wildcard handler for everything else
|
|
560
515
|
@JobHandler('*')
|
|
561
516
|
async handleAll(job: Job, entityId: string) {
|
|
562
|
-
//
|
|
517
|
+
// Wildcard — catches everything not explicitly handled
|
|
518
|
+
// Falls back to CQRS routing automatically when not defined
|
|
563
519
|
}
|
|
564
520
|
}
|
|
565
521
|
```
|
|
566
522
|
|
|
567
|
-
**
|
|
523
|
+
> **Priority order:** Explicit `@JobHandler` → CQRS auto-routing (`@JobCommand`/`@JobQuery`) → Wildcard handler
|
|
524
|
+
|
|
525
|
+
---
|
|
526
|
+
|
|
527
|
+
## Performance
|
|
528
|
+
|
|
529
|
+
### Throughput (measured — not estimated)
|
|
530
|
+
|
|
531
|
+
Tested on a 5-pod Kubernetes cluster (OrbStack), 20 concurrent entities, 12,300 orders:
|
|
532
|
+
|
|
533
|
+
| Metric | Result |
|
|
534
|
+
|---|---|
|
|
535
|
+
| **Phase 1** — 10,000 orders (50 waves × 200 concurrent) | 167 orders/sec |
|
|
536
|
+
| **Phase 2** — 1,000 orders (workers still draining) | 140 orders/sec |
|
|
537
|
+
| **Phase 4** — 1,000 orders (cold start from zero workers) | 176 orders/sec |
|
|
538
|
+
| **Total deductions processed** | 104,004 |
|
|
539
|
+
| **Stock drift** | **0** (all 20 entities) |
|
|
540
|
+
| **Pod distribution** | 5/5 pods actively creating workers |
|
|
541
|
+
| **Worker creates** | 120 |
|
|
542
|
+
| **Idle closures** | 180 |
|
|
543
|
+
|
|
544
|
+
### Why it's fast
|
|
545
|
+
|
|
546
|
+
1. **Zero contention** — no locks, no retries, no backoff. Jobs queue and execute.
|
|
547
|
+
2. **Hot cache** — after first check, subsequent job arrivals for an entity incur 0 Redis calls.
|
|
548
|
+
3. **Direct local spawn** — atomic `SET NX` claim, local worker creation. No queue round-trip.
|
|
549
|
+
4. **Pipelined heartbeats** — heartbeat refresh uses a single Redis pipeline (1 round-trip for 2 keys).
|
|
550
|
+
5. **O(1) worker existence check** — global alive key replaces `KEYS` pattern scan.
|
|
551
|
+
|
|
552
|
+
### When to use what
|
|
553
|
+
|
|
554
|
+
| Use case | Recommendation |
|
|
555
|
+
|---|---|
|
|
556
|
+
| High-throughput entity operations (payments, inventory, game state) | **atomic-queues** |
|
|
557
|
+
| Rare, low-frequency mutual exclusion (config updates, migrations) | Redlock / advisory locks |
|
|
558
|
+
| Exactly-once semantics with audit trail | **atomic-queues** (BullMQ job IDs) |
|
|
559
|
+
| Sub-millisecond synchronous response required | Redlock (synchronous acquire) |
|
|
560
|
+
| Multi-pod, many entities, sustained load | **atomic-queues** (contention-free scaling) |
|
|
568
561
|
|
|
569
562
|
---
|
|
570
563
|
|