atomic-queues 1.2.7 → 1.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/README.md +350 -440
  2. package/dist/decorators/decorators.d.ts +123 -4
  3. package/dist/decorators/decorators.d.ts.map +1 -1
  4. package/dist/decorators/decorators.js +218 -10
  5. package/dist/decorators/decorators.js.map +1 -1
  6. package/dist/domain/interfaces.d.ts +127 -1
  7. package/dist/domain/interfaces.d.ts.map +1 -1
  8. package/dist/module/atomic-queues.module.d.ts.map +1 -1
  9. package/dist/module/atomic-queues.module.js +1 -0
  10. package/dist/module/atomic-queues.module.js.map +1 -1
  11. package/dist/services/cron-manager/cron-manager.service.d.ts +23 -1
  12. package/dist/services/cron-manager/cron-manager.service.d.ts.map +1 -1
  13. package/dist/services/cron-manager/cron-manager.service.js +150 -41
  14. package/dist/services/cron-manager/cron-manager.service.js.map +1 -1
  15. package/dist/services/index.d.ts +1 -0
  16. package/dist/services/index.d.ts.map +1 -1
  17. package/dist/services/index.js +1 -0
  18. package/dist/services/index.js.map +1 -1
  19. package/dist/services/processor-discovery/processor-discovery.service.d.ts +29 -4
  20. package/dist/services/processor-discovery/processor-discovery.service.d.ts.map +1 -1
  21. package/dist/services/processor-discovery/processor-discovery.service.js +168 -14
  22. package/dist/services/processor-discovery/processor-discovery.service.js.map +1 -1
  23. package/dist/services/queue-bus/queue-bus.service.d.ts +126 -12
  24. package/dist/services/queue-bus/queue-bus.service.d.ts.map +1 -1
  25. package/dist/services/queue-bus/queue-bus.service.js +274 -22
  26. package/dist/services/queue-bus/queue-bus.service.js.map +1 -1
  27. package/dist/services/queue-events-manager/index.d.ts +2 -0
  28. package/dist/services/queue-events-manager/index.d.ts.map +1 -0
  29. package/dist/services/queue-events-manager/index.js +18 -0
  30. package/dist/services/queue-events-manager/index.js.map +1 -0
  31. package/dist/services/queue-events-manager/queue-events-manager.service.d.ts +103 -0
  32. package/dist/services/queue-events-manager/queue-events-manager.service.d.ts.map +1 -0
  33. package/dist/services/queue-events-manager/queue-events-manager.service.js +311 -0
  34. package/dist/services/queue-events-manager/queue-events-manager.service.js.map +1 -0
  35. package/dist/services/queue-manager/queue-manager.service.d.ts +15 -1
  36. package/dist/services/queue-manager/queue-manager.service.d.ts.map +1 -1
  37. package/dist/services/queue-manager/queue-manager.service.js +17 -3
  38. package/dist/services/queue-manager/queue-manager.service.js.map +1 -1
  39. package/dist/services/worker-manager/worker-manager.service.d.ts +43 -0
  40. package/dist/services/worker-manager/worker-manager.service.d.ts.map +1 -1
  41. package/dist/services/worker-manager/worker-manager.service.js +107 -1
  42. package/dist/services/worker-manager/worker-manager.service.js.map +1 -1
  43. package/package.json +1 -1
package/README.md CHANGED
@@ -1,56 +1,98 @@
1
1
  # atomic-queues
2
2
 
3
- A NestJS library for atomic, sequential job processing per entity with BullMQ and Redis.
3
+ A NestJS library for atomic, sequential job processing per entity using BullMQ and Redis.
4
4
 
5
- ## What It Does
5
+ ---
6
+
7
+ ## Table of Contents
8
+
9
+ - [Overview](#overview)
10
+ - [The Concurrency Problem](#the-concurrency-problem)
11
+ - [The Per-Entity Queue Architecture](#the-per-entity-queue-architecture)
12
+ - [Installation](#installation)
13
+ - [Quick Start](#quick-start)
14
+ - [Commands and Decorators](#commands-and-decorators)
15
+ - [Configuration](#configuration)
16
+ - [Complete Example](#complete-example)
17
+ - [Advanced: Custom Worker Processors](#advanced-custom-worker-processors)
18
+ - [License](#license)
19
+
20
+ ---
21
+
22
+ ## Overview
23
+
24
+ **atomic-queues** solves the fundamental concurrency problem in distributed systems: ensuring that operations on the same logical entity execute sequentially, even when requests arrive simultaneously across multiple service instances.
25
+
26
+ Rather than relying on distributed locks—which introduce contention, latency degradation, and complex failure modes—this library implements a **per-entity queue architecture** where each entity (user account, order, document) has its own dedicated processing queue and worker.
27
+
28
+ ---
29
+
30
+ ## The Concurrency Problem
31
+
32
+ Consider a banking system where a user with a $100 balance submits two concurrent $80 withdrawal requests:
33
+
34
+ ```
35
+ Time Request A Request B Database State
36
+ ─────────────────────────────────────────────────────────────────────────────────
37
+ T₀ SELECT balance → $100 SELECT balance → $100 balance = $100
38
+ T₁ CHECK: $100 >= $80 ✓ CHECK: $100 >= $80 ✓
39
+ T₂ UPDATE: balance = $20 UPDATE: balance = $20 balance = $20
40
+ T₃ UPDATE: balance = -$60 balance = -$60
41
+ ─────────────────────────────────────────────────────────────────────────────────
42
+ Result: Both withdrawals succeed. Balance becomes -$60. Integrity violated.
43
+ ```
44
+
45
+ With atomic-queues, operations are queued and processed sequentially:
6
46
 
7
47
  ```
8
- ╔═══════════════════════════════════════════════════════════════════════════════╗
9
- ║ THE PROBLEM ║
10
- ╠═══════════════════════════════════════════════════════════════════════════════╣
11
- ║ ║
12
- ║ User has $100 balance. Two $80 withdrawals arrive at the same time: ║
13
- ║ ║
14
- ║ Withdraw $80 ──┐ ║
15
- ║ (API 1) │ ┌────────────────────┐ ║
16
- ║ ├───▶│ Balance: $100 │ ║
17
- ║ Withdraw $80 ──┘ │ Both read $100 │ ║
18
- ║ (API 2) │ Both approve │ ║
19
- ║ │ Final: -$60 💥 │ ║
20
- ║ └────────────────────┘ ║
21
- ║ ║
22
- ║ Race condition: Both transactions see $100, both succeed, balance goes -$60 ║
23
- ║ ║
24
- ╚═══════════════════════════════════════════════════════════════════════════════╝
25
-
26
- ╔═══════════════════════════════════════════════════════════════════════════════╗
27
- ║ THE SOLUTION ║
28
- ╠═══════════════════════════════════════════════════════════════════════════════╣
29
- ║ ║
30
- ║ atomic-queues processes one transaction at a time per account: ║
31
- ║ ║
32
- ║ Withdraw $80 ──┐ ┌─────────────┐ ┌─────────────────────────────┐ ║
33
- ║ (API 1) │ │ │ │ Worker processes queue: │ ║
34
- ║ ├────▶│ Redis Queue │────▶│ │ ║
35
- ║ Withdraw $80 ──┘ │ [W1] [W2] │ │ W1: $100 - $80 = $20 ✓ │ ║
36
- ║ (API 2) │ │ │ W2: $20 < $80 = REJECTED ✓ │ ║
37
- ║ └─────────────┘ └─────────────────────────────┘ ║
38
- ║ ║
39
- ║ Sequential processing: W1 completes first, W2 sees updated balance ║
40
- ║ ║
41
- ╚═══════════════════════════════════════════════════════════════════════════════╝
48
+ Time Queue State Worker Execution Database State
49
+ ───────────────────────────────────────────────────────────────────────────────────
50
+ T₀ [Withdraw $80, Withdraw $80] balance = $100
51
+ T₁ [Withdraw $80] Process Op₁: $100 - $80 balance = $20
52
+ T₂ [] Process Op₂: $20 < $80 REJECT balance = $20
53
+ ───────────────────────────────────────────────────────────────────────────────────
54
+ Result: First withdrawal succeeds. Second is rejected. Integrity preserved.
42
55
  ```
43
56
 
57
+ ---
58
+
59
+ ## The Per-Entity Queue Architecture
60
+
61
+ ```
62
+ ┌─────────────────────────────────────────┐
63
+ Request A ─┐ │ Per-Entity Queue │
64
+ │ │ ┌─────┐ ┌─────┐ ┌─────┐ │
65
+ Request B ─┼──▶ [Entity Router] ─┼─▶│ Op₁ │→│ Op₂ │→│ Op₃ │→ [Worker] ─┐ │
66
+ │ │ └─────┘ └─────┘ └─────┘ │ │
67
+ Request C ─┘ │ │ │
68
+ │ Sequential Processing ◄─────────┘ │
69
+ └─────────────────────────────────────────┘
70
+ ```
71
+
72
+ **Key features:**
73
+ - Each entity has exactly one active worker (enforced via Redis heartbeat)
74
+ - Workers spawn automatically when jobs arrive
75
+ - Workers terminate after configurable idle period
76
+ - Node failure → heartbeat expires → worker respawns on healthy node
77
+
78
+ ---
79
+
44
80
  ## Installation
45
81
 
46
82
  ```bash
47
83
  npm install atomic-queues bullmq ioredis
48
84
  ```
49
85
 
86
+ ---
87
+
50
88
  ## Quick Start
51
89
 
52
90
  ### 1. Configure the Module
53
91
 
92
+ The `entities` configuration is **optional**. Choose the approach that fits your needs:
93
+
94
+ #### Option A: Minimal Setup (uses default naming)
95
+
54
96
  ```typescript
55
97
  import { Module } from '@nestjs/common';
56
98
  import { AtomicQueuesModule } from 'atomic-queues';
@@ -60,289 +102,309 @@ import { AtomicQueuesModule } from 'atomic-queues';
60
102
  AtomicQueuesModule.forRoot({
61
103
  redis: { host: 'localhost', port: 6379 },
62
104
  keyPrefix: 'myapp',
105
+ enableCronManager: true,
106
+ // No entities config needed! Uses default naming:
107
+ // Queue: {keyPrefix}:{entityType}:{entityId}:queue
108
+ // Worker: {keyPrefix}:{entityType}:{entityId}:worker
63
109
  }),
64
110
  ],
65
111
  })
66
112
  export class AppModule {}
67
113
  ```
68
114
 
69
- ### 2. Create Your Commands
115
+ #### Option B: Custom Queue/Worker Naming (via entities config)
70
116
 
71
- Plain classes - no decorators needed:
117
+ ```typescript
118
+ @Module({
119
+ imports: [
120
+ AtomicQueuesModule.forRoot({
121
+ redis: { host: 'localhost', port: 6379 },
122
+ keyPrefix: 'myapp',
123
+ enableCronManager: true,
124
+
125
+ // Optional: Define custom naming and settings per entity type
126
+ entities: {
127
+ account: {
128
+ queueName: (id) => `${id}-queue`, // Custom queue naming
129
+ workerName: (id) => `${id}-worker`, // Custom worker naming
130
+ maxWorkersPerEntity: 1,
131
+ idleTimeoutSeconds: 15,
132
+ },
133
+ },
134
+ }),
135
+ ],
136
+ })
137
+ export class AppModule {}
138
+ ```
139
+
140
+ #### Option C: Custom Naming via @WorkerProcessor
141
+
142
+ For advanced use cases, define a processor class instead of entities config:
72
143
 
73
144
  ```typescript
74
- // commands/process-order.command.ts
75
- export class ProcessOrderCommand {
145
+ @WorkerProcessor({
146
+ entityType: 'account',
147
+ queueName: (id) => `${id}-queue`,
148
+ workerName: (id) => `${id}-worker`,
149
+ maxWorkersPerEntity: 1,
150
+ idleTimeoutSeconds: 15,
151
+ })
152
+ @Injectable()
153
+ export class AccountProcessor {}
154
+ ```
155
+
156
+ > **When to use each:**
157
+ > - **Option A**: Default naming works for you
158
+ > - **Option B**: Need custom naming but no custom job handling logic
159
+ > - **Option C**: Need custom naming AND custom `@JobHandler` methods
160
+
161
+ ### 2. Create Commands with Decorators
162
+
163
+ ```typescript
164
+ import { QueueEntity, QueueEntityId } from 'atomic-queues';
165
+
166
+ @QueueEntity('account')
167
+ export class WithdrawCommand {
76
168
  constructor(
77
- public readonly orderId: string,
78
- public readonly items: string[],
169
+ @QueueEntityId() public readonly accountId: string,
79
170
  public readonly amount: number,
171
+ public readonly transactionId: string,
80
172
  ) {}
81
173
  }
82
174
 
83
- // commands/ship-order.command.ts
84
- export class ShipOrderCommand {
175
+ @QueueEntity('account')
176
+ export class DepositCommand {
85
177
  constructor(
86
- public readonly orderId: string,
87
- public readonly address: string,
178
+ @QueueEntityId() public readonly accountId: string,
179
+ public readonly amount: number,
180
+ public readonly source: string,
88
181
  ) {}
89
182
  }
90
183
  ```
91
184
 
92
- ### 3. Create a Worker Processor
185
+ ### 3. Create Command Handlers (standard @nestjs/cqrs)
93
186
 
94
187
  ```typescript
95
- import { Injectable } from '@nestjs/common';
96
- import { WorkerProcessor } from 'atomic-queues';
188
+ import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
189
+ import { WithdrawCommand } from './commands';
97
190
 
98
- @WorkerProcessor({
99
- entityType: 'order',
100
- queueName: (orderId) => `order-${orderId}-queue`,
101
- workerName: (orderId) => `order-${orderId}-worker`,
102
- })
103
- @Injectable()
104
- export class OrderProcessor {}
105
- ```
191
+ @CommandHandler(WithdrawCommand)
192
+ export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
193
+ constructor(private readonly accountRepo: AccountRepository) {}
106
194
 
107
- ### 4. Queue Jobs with the Fluent API
195
+ async execute(command: WithdrawCommand) {
196
+ const { accountId, amount, transactionId } = command;
197
+
198
+ // SAFE: No race conditions! Processed sequentially per account.
199
+ const account = await this.accountRepo.findById(accountId);
200
+
201
+ if (account.balance < amount) {
202
+ throw new InsufficientFundsError(accountId, account.balance, amount);
203
+ }
204
+
205
+ account.balance -= amount;
206
+ await this.accountRepo.save(account);
207
+
208
+ return { success: true, newBalance: account.balance };
209
+ }
210
+ }
211
+ ```
108
212
 
109
- Commands are **automatically registered** from your `@CommandHandler` classes - no manual registration needed!
213
+ ### 4. Enqueue Jobs
110
214
 
111
215
  ```typescript
112
216
  import { Injectable } from '@nestjs/common';
113
217
  import { QueueBus } from 'atomic-queues';
114
- import { OrderProcessor } from './order.processor';
115
- import { ProcessOrderCommand, ShipOrderCommand } from './commands';
218
+ import { WithdrawCommand, DepositCommand } from './commands';
116
219
 
117
220
  @Injectable()
118
- export class OrderService {
221
+ export class AccountService {
119
222
  constructor(private readonly queueBus: QueueBus) {}
120
223
 
121
- async createOrder(orderId: string, items: string[], amount: number) {
122
- // Jobs are queued and processed sequentially per orderId
123
- await this.queueBus
124
- .forProcessor(OrderProcessor)
125
- .enqueue(new ProcessOrderCommand(orderId, items, amount));
224
+ async withdraw(accountId: string, amount: number, transactionId: string) {
225
+ // Command is automatically routed to the account's queue
226
+ await this.queueBus.enqueue(new WithdrawCommand(accountId, amount, transactionId));
227
+ }
126
228
 
127
- await this.queueBus
128
- .forProcessor(OrderProcessor)
129
- .enqueue(new ShipOrderCommand(orderId, '123 Main St'));
229
+ async deposit(accountId: string, amount: number, source: string) {
230
+ await this.queueBus.enqueue(new DepositCommand(accountId, amount, source));
130
231
  }
131
232
  }
132
233
  ```
133
234
 
134
- That's it! The library automatically:
135
- - Discovers commands from `@CommandHandler` decorators
136
- - Creates a queue for each `orderId`
235
+ **That's it!** The library automatically:
236
+ - Creates a queue for each `accountId` when jobs arrive
137
237
  - Spawns a worker to process jobs sequentially
138
- - Routes jobs to the correct command handlers
139
-
140
- ---
141
-
142
- ## How It Works
143
-
144
- ```
145
- ╔═══════════════════════════════════════════════════════════════════════════════╗
146
- ║ ARCHITECTURE ║
147
- ╚═══════════════════════════════════════════════════════════════════════════════╝
148
-
149
- ┌──────────────────┐
150
- │ API Request │ POST /accounts/ACC-123/withdraw { amount: 80 }
151
- └────────┬─────────┘
152
-
153
-
154
- ┌──────────────────────────────────────────────────────────────────────────────┐
155
- │ QueueBus.forProcessor(AccountProcessor).enqueue(new WithdrawCommand(...)) │
156
- └────────┬─────────────────────────────────────────────────────────────────────┘
157
-
158
- │ ① Reads @WorkerProcessor metadata from AccountProcessor
159
- │ ② Extracts accountId from command.accountId property
160
- │ ③ Generates queue name: "account-ACC-123-queue"
161
-
162
-
163
- ┌──────────────────────────────────────────────────────────────────────────────┐
164
- │ REDIS │
165
- │ ┌────────────────────────────────────────────────────────────────────────┐ │
166
- │ │ Queue: account-ACC-123-queue │ │
167
- │ │ ┌─────────────────┬─────────────────┬─────────────────┐ │ │
168
- │ │ │ Job 1 │ Job 2 │ Job 3 │ ... │ │
169
- │ │ │ WithdrawCommand │ DepositCommand │ TransferCommand │ │ │
170
- │ │ │ amount: 80 │ amount: 50 │ amount: 25 │ │ │
171
- │ │ └─────────────────┴─────────────────┴─────────────────┘ │ │
172
- │ └────────────────────────────────────────────────────────────────────────┘ │
173
- │ │
174
- │ ┌────────────────────────────────────────────────────────────────────────┐ │
175
- │ │ Queue: account-ACC-456-queue (different account = different queue) │ │
176
- │ │ ┌─────────────────┐ │ │
177
- │ │ │ Job 1 │ ← Processes in parallel with ACC-123 │ │
178
- │ │ │ WithdrawCommand │ │ │
179
- │ │ └─────────────────┘ │ │
180
- │ └────────────────────────────────────────────────────────────────────────┘ │
181
- └──────────────────────────────────────────────────────────────────────────────┘
182
-
183
- │ ④ BullMQ Worker pulls Job 1 (only one job at a time per queue)
184
-
185
-
186
- ┌──────────────────────────────────────────────────────────────────────────────┐
187
- │ Worker: account-ACC-123-worker │
188
- │ │
189
- │ ┌─────────────────────────────────────────────────────────────────────────┐ │
190
- │ │ ⑤ Lookup "WithdrawCommand" in QueueBus.globalRegistry │ │
191
- │ │ ⑥ Instantiate: Object.assign(new WithdrawCommand(), job.data) │ │
192
- │ │ ⑦ Execute: CommandBus.execute(withdrawCommand) │ │
193
- │ └─────────────────────────────────────────────────────────────────────────┘ │
194
- └────────┬─────────────────────────────────────────────────────────────────────┘
195
-
196
-
197
- ┌──────────────────────────────────────────────────────────────────────────────┐
198
- │ @CommandHandler(WithdrawCommand) │
199
- │ class WithdrawHandler { │
200
- │ async execute(cmd: WithdrawCommand) { │
201
- │ // Safe! No race conditions - guaranteed sequential execution │
202
- │ const balance = await this.repo.getBalance(cmd.accountId); │
203
- │ if (balance < cmd.amount) throw new InsufficientFundsError(); │
204
- │ await this.repo.debit(cmd.accountId, cmd.amount); │
205
- │ } │
206
- │ } │
207
- └──────────────────────────────────────────────────────────────────────────────┘
208
- ```
238
+ - Routes jobs to the correct `@CommandHandler`
239
+ - Terminates idle workers after the configured timeout
209
240
 
210
241
  ---
211
242
 
212
- ## API Reference
243
+ ## Commands and Decorators
213
244
 
214
- ### QueueBus
245
+ ### @QueueEntity(entityType)
215
246
 
216
- The main way to add jobs to queues:
247
+ Marks a command class for queue routing. The `entityType` must match a key in your `entities` config.
217
248
 
218
249
  ```typescript
219
- // Enqueue a single command
220
- await queueBus
221
- .forProcessor(MyProcessor)
222
- .enqueue(new MyCommand(entityId, data));
223
-
224
- // Enqueue and wait for result
225
- const result = await queueBus
226
- .forProcessor(MyProcessor)
227
- .enqueueAndWait(new MyQuery(entityId));
228
-
229
- // Enqueue multiple commands
230
- await queueBus
231
- .forProcessor(MyProcessor)
232
- .enqueueBulk([
233
- new CommandA(entityId),
234
- new CommandB(entityId),
235
- ]);
236
-
237
- // With job options (delay, priority, etc.)
238
- await queueBus
239
- .forProcessor(MyProcessor)
240
- .enqueue(new MyCommand(entityId), {
241
- jobOptions: { delay: 5000, priority: 1 }
242
- });
250
+ @QueueEntity('account')
251
+ export class TransferCommand { ... }
243
252
  ```
244
253
 
245
- ### @WorkerProcessor
254
+ ### @QueueEntityId()
246
255
 
247
- Defines how workers are created for an entity type:
256
+ Marks which property contains the entity ID for queue routing. Only one per class.
248
257
 
249
258
  ```typescript
250
- @WorkerProcessor({
251
- entityType: 'order', // Required
252
- queueName: (id) => `order-${id}-queue`, // Optional
253
- workerName: (id) => `order-${id}-worker`, // Optional
254
- workerConfig: {
255
- concurrency: 1, // Jobs per worker (default: 1)
256
- stalledInterval: 1000, // Check stalled jobs (ms)
257
- lockDuration: 30000, // Job lock duration (ms)
258
- },
259
- })
259
+ @QueueEntity('account')
260
+ export class TransferCommand {
261
+ constructor(
262
+ @QueueEntityId() public readonly sourceAccountId: string, // Routes to source account's queue
263
+ public readonly targetAccountId: string,
264
+ public readonly amount: number,
265
+ ) {}
266
+ }
260
267
  ```
261
268
 
262
- ---
263
-
264
- ## Entity ID Extraction
269
+ ### Alternative: Use defaultEntityId
265
270
 
266
- The `entityId` is automatically extracted from your command's properties:
271
+ If all commands for an entity use the same property name, configure it once:
267
272
 
268
273
  ```typescript
269
- // These property names are checked in order:
270
- // entityId, tableId, userId, id, gameId, playerId
271
-
272
- export class ProcessOrderCommand {
273
- constructor(
274
- public readonly orderId: string, // ✓ 'orderId' contains 'Id' → entityId
275
- public readonly items: string[],
276
- ) {}
274
+ // In module config
275
+ entities: {
276
+ account: {
277
+ defaultEntityId: 'accountId', // Commands without @QueueEntityId use this
278
+ // ...
279
+ },
277
280
  }
278
281
 
279
- // Or use standard names
280
- export class UpdateUserCommand {
282
+ // Then commands don't need @QueueEntityId
283
+ @QueueEntity('account')
284
+ export class WithdrawCommand {
281
285
  constructor(
282
- public readonly userId: string, // Matches 'userId' → entityId
283
- public readonly name: string,
286
+ public readonly accountId: string, // Automatically used
287
+ public readonly amount: number,
284
288
  ) {}
285
289
  }
286
290
  ```
287
291
 
288
292
  ---
289
293
 
290
- ## Scaling with Entity Scalers
291
-
292
- For dynamic worker management based on demand:
294
+ ## Configuration
293
295
 
294
296
  ```typescript
295
- import { Injectable } from '@nestjs/common';
296
- import { EntityScaler, GetActiveEntities, GetDesiredWorkerCount } from 'atomic-queues';
297
-
298
- @EntityScaler({
299
- entityType: 'order',
300
- maxWorkersPerEntity: 1,
301
- })
302
- @Injectable()
303
- export class OrderScaler {
304
- constructor(private readonly orderRepo: OrderRepository) {}
305
-
306
- @GetActiveEntities()
307
- async getActiveOrders(): Promise<string[]> {
308
- // Return IDs that need workers
309
- return this.orderRepo.findPendingOrderIds();
310
- }
311
-
312
- @GetDesiredWorkerCount()
313
- async getWorkerCount(orderId: string): Promise<number> {
314
- return 1; // One worker per order
315
- }
316
- }
297
+ AtomicQueuesModule.forRoot({
298
+ redis: {
299
+ host: 'localhost',
300
+ port: 6379,
301
+ password: 'secret',
302
+ },
303
+
304
+ keyPrefix: 'myapp', // Redis key prefix (default: 'aq')
305
+ enableCronManager: true, // Enable worker lifecycle management
306
+ cronInterval: 5000, // Scaling check interval (ms)
307
+
308
+ workerDefaults: {
309
+ concurrency: 1, // Jobs processed simultaneously
310
+ stalledInterval: 1000, // Stalled job check interval (ms)
311
+ lockDuration: 30000, // Job lock duration (ms)
312
+ heartbeatTTL: 3, // Worker heartbeat TTL (seconds)
313
+ },
314
+
315
+ // OPTIONAL: Per-entity configuration
316
+ // If omitted, uses default naming: {keyPrefix}:{entityType}:{entityId}:queue/worker
317
+ entities: {
318
+ account: {
319
+ defaultEntityId: 'accountId',
320
+ queueName: (id) => `${id}-queue`,
321
+ workerName: (id) => `${id}-worker`,
322
+ maxWorkersPerEntity: 1,
323
+ idleTimeoutSeconds: 15,
324
+ autoSpawn: true, // Default: true
325
+ workerConfig: { // Override defaults per entity
326
+ concurrency: 1,
327
+ lockDuration: 60000,
328
+ },
329
+ },
330
+ order: {
331
+ defaultEntityId: 'orderId',
332
+ queueName: (id) => `order-${id}-queue`,
333
+ idleTimeoutSeconds: 30,
334
+ },
335
+ },
336
+ });
317
337
  ```
318
338
 
319
339
  ---
320
340
 
321
341
  ## Complete Example
322
342
 
323
- A banking service handling critical financial transactions where race conditions could cause overdrafts or double-spending:
343
+ A banking service handling financial transactions:
324
344
 
325
345
  ```typescript
346
+ // ─────────────────────────────────────────────────────────────────
347
+ // app.module.ts
348
+ // ─────────────────────────────────────────────────────────────────
349
+ import { Module } from '@nestjs/common';
350
+ import { CqrsModule } from '@nestjs/cqrs';
351
+ import { AtomicQueuesModule } from 'atomic-queues';
352
+
353
+ @Module({
354
+ imports: [
355
+ CqrsModule,
356
+ AtomicQueuesModule.forRoot({
357
+ redis: { host: 'localhost', port: 6379 },
358
+ keyPrefix: 'banking',
359
+ enableCronManager: true,
360
+ entities: {
361
+ account: {
362
+ queueName: (id) => `${id}-queue`,
363
+ workerName: (id) => `${id}-worker`,
364
+ maxWorkersPerEntity: 1,
365
+ idleTimeoutSeconds: 15,
366
+ workerConfig: {
367
+ concurrency: 1,
368
+ lockDuration: 60000,
369
+ },
370
+ },
371
+ },
372
+ }),
373
+ ],
374
+ providers: [
375
+ AccountService,
376
+ WithdrawHandler,
377
+ DepositHandler,
378
+ TransferHandler,
379
+ ],
380
+ controllers: [AccountController],
381
+ })
382
+ export class AppModule {}
383
+
326
384
  // ─────────────────────────────────────────────────────────────────
327
385
  // commands/withdraw.command.ts
328
386
  // ─────────────────────────────────────────────────────────────────
387
+ import { QueueEntity, QueueEntityId } from 'atomic-queues';
388
+
389
+ @QueueEntity('account')
329
390
  export class WithdrawCommand {
330
391
  constructor(
331
- public readonly accountId: string,
392
+ @QueueEntityId() public readonly accountId: string,
332
393
  public readonly amount: number,
333
394
  public readonly transactionId: string,
334
- public readonly requestedBy: string,
335
395
  ) {}
336
396
  }
337
397
 
338
398
  // ─────────────────────────────────────────────────────────────────
339
399
  // commands/deposit.command.ts
340
400
  // ─────────────────────────────────────────────────────────────────
401
+ import { QueueEntity, QueueEntityId } from 'atomic-queues';
402
+
403
+ @QueueEntity('account')
341
404
  export class DepositCommand {
342
405
  constructor(
343
- public readonly accountId: string,
406
+ @QueueEntityId() public readonly accountId: string,
344
407
  public readonly amount: number,
345
- public readonly transactionId: string,
346
408
  public readonly source: string,
347
409
  ) {}
348
410
  }
@@ -350,9 +412,12 @@ export class DepositCommand {
350
412
  // ─────────────────────────────────────────────────────────────────
351
413
  // commands/transfer.command.ts
352
414
  // ─────────────────────────────────────────────────────────────────
415
+ import { QueueEntity, QueueEntityId } from 'atomic-queues';
416
+
417
+ @QueueEntity('account')
353
418
  export class TransferCommand {
354
419
  constructor(
355
- public readonly accountId: string, // Source account (for queue routing)
420
+ @QueueEntityId() public readonly accountId: string, // Source account
356
421
  public readonly toAccountId: string,
357
422
  public readonly amount: number,
358
423
  public readonly transactionId: string,
@@ -367,46 +432,22 @@ import { WithdrawCommand } from '../commands';
367
432
 
368
433
  @CommandHandler(WithdrawCommand)
369
434
  export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
370
- constructor(
371
- private readonly accountRepo: AccountRepository,
372
- private readonly ledger: LedgerService,
373
- ) {}
435
+ constructor(private readonly accountRepo: AccountRepository) {}
374
436
 
375
437
  async execute(command: WithdrawCommand) {
376
- const { accountId, amount, transactionId } = command;
377
-
378
- // SAFE: No race conditions! This handler runs sequentially per account
379
- // Even if 10 withdrawals arrive simultaneously, they execute one-by-one
438
+ const { accountId, amount } = command;
380
439
 
440
+ // SAFE: Sequential execution per account
381
441
  const account = await this.accountRepo.findById(accountId);
382
442
 
383
443
  if (account.balance < amount) {
384
444
  throw new InsufficientFundsError(accountId, account.balance, amount);
385
445
  }
386
446
 
387
- if (account.status !== 'active') {
388
- throw new AccountFrozenError(accountId);
389
- }
390
-
391
- // Debit the account
392
447
  account.balance -= amount;
393
448
  await this.accountRepo.save(account);
394
449
 
395
- // Record in ledger
396
- await this.ledger.record({
397
- transactionId,
398
- accountId,
399
- type: 'DEBIT',
400
- amount,
401
- balanceAfter: account.balance,
402
- timestamp: new Date(),
403
- });
404
-
405
- return {
406
- success: true,
407
- transactionId,
408
- newBalance: account.balance
409
- };
450
+ return { success: true, newBalance: account.balance };
410
451
  }
411
452
  }
412
453
 
@@ -425,105 +466,34 @@ export class TransferHandler implements ICommandHandler<TransferCommand> {
425
466
  ) {}
426
467
 
427
468
  async execute(command: TransferCommand) {
428
- const { accountId, toAccountId, amount, transactionId } = command;
429
-
430
- // Step 1: Debit source account (already in source account's queue)
431
- const sourceAccount = await this.accountRepo.findById(accountId);
469
+ const { accountId, toAccountId, amount } = command;
432
470
 
433
- if (sourceAccount.balance < amount) {
434
- throw new InsufficientFundsError(accountId, sourceAccount.balance, amount);
471
+ // Debit source (we're in source account's queue)
472
+ const source = await this.accountRepo.findById(accountId);
473
+ if (source.balance < amount) {
474
+ throw new InsufficientFundsError(accountId, source.balance, amount);
435
475
  }
436
476
 
437
- sourceAccount.balance -= amount;
438
- await this.accountRepo.save(sourceAccount);
477
+ source.balance -= amount;
478
+ await this.accountRepo.save(source);
439
479
 
440
- // Step 2: Queue credit to destination account (different queue!)
441
- // This ensures the destination account also processes atomically
442
- await this.queueBus
443
- .forProcessor(AccountProcessor)
444
- .enqueue(new DepositCommand(
445
- toAccountId,
446
- amount,
447
- transactionId,
448
- `transfer:${accountId}`,
449
- ));
480
+ // Credit destination (enqueued to destination's queue)
481
+ await this.queueBus.enqueue(new DepositCommand(
482
+ toAccountId,
483
+ amount,
484
+ `transfer:${accountId}`,
485
+ ));
450
486
 
451
- return { success: true, transactionId };
487
+ return { success: true };
452
488
  }
453
489
  }
454
490
 
455
- // ─────────────────────────────────────────────────────────────────
456
- // account.processor.ts
457
- // ─────────────────────────────────────────────────────────────────
458
- import { Injectable } from '@nestjs/common';
459
- import { WorkerProcessor } from 'atomic-queues';
460
-
461
- @WorkerProcessor({
462
- entityType: 'account',
463
- queueName: (accountId) => `bank-account-${accountId}-queue`,
464
- workerName: (accountId) => `bank-account-${accountId}-worker`,
465
- workerConfig: {
466
- concurrency: 1, // CRITICAL: Must be 1 for financial transactions
467
- lockDuration: 60000, // 60s lock for long transactions
468
- stalledInterval: 5000,
469
- },
470
- })
471
- @Injectable()
472
- export class AccountProcessor {}
473
-
474
- // ─────────────────────────────────────────────────────────────────
475
- // account.scaler.ts - Scale workers based on active accounts
476
- // ─────────────────────────────────────────────────────────────────
477
- import { Injectable } from '@nestjs/common';
478
- import { EntityScaler, GetActiveEntities, GetDesiredWorkerCount } from 'atomic-queues';
479
-
480
- @EntityScaler({
481
- entityType: 'account',
482
- maxWorkersPerEntity: 1, // Never more than 1 worker per account
483
- })
484
- @Injectable()
485
- export class AccountScaler {
486
- constructor(private readonly accountRepo: AccountRepository) {}
487
-
488
- @GetActiveEntities()
489
- async getActiveAccounts(): Promise<string[]> {
490
- // Return accounts with pending transactions
491
- return this.accountRepo.findAccountsWithPendingTransactions();
492
- }
493
-
494
- @GetDesiredWorkerCount()
495
- async getWorkerCount(accountId: string): Promise<number> {
496
- // Always 1 worker per account for atomicity
497
- return 1;
498
- }
499
- }
500
-
501
- // ─────────────────────────────────────────────────────────────────
502
- // account.module.ts
503
- // ─────────────────────────────────────────────────────────────────
504
- import { Module } from '@nestjs/common';
505
- import { CqrsModule } from '@nestjs/cqrs';
506
-
507
- @Module({
508
- imports: [CqrsModule],
509
- providers: [
510
- AccountProcessor,
511
- AccountScaler,
512
- WithdrawHandler, // Commands auto-discovered!
513
- DepositHandler,
514
- TransferHandler,
515
- ],
516
- controllers: [AccountController],
517
- })
518
- export class AccountModule {}
519
-
520
491
  // ─────────────────────────────────────────────────────────────────
521
492
  // account.controller.ts
522
493
  // ─────────────────────────────────────────────────────────────────
523
494
  import { Controller, Post, Body, Param } from '@nestjs/common';
524
495
  import { QueueBus } from 'atomic-queues';
525
- import { AccountProcessor } from './account.processor';
526
- import { WithdrawCommand, DepositCommand, TransferCommand } from './commands';
496
+ import { WithdrawCommand, TransferCommand } from './commands';
527
497
  import { v4 as uuid } from 'uuid';
528
498
 
529
499
  @Controller('accounts')
@@ -533,26 +503,15 @@ export class AccountController {
533
503
  @Post(':accountId/withdraw')
534
504
  async withdraw(
535
505
  @Param('accountId') accountId: string,
536
- @Body() body: { amount: number; requestedBy: string },
506
+ @Body() body: { amount: number },
537
507
  ) {
538
508
  const transactionId = uuid();
539
509
 
540
- // Even if user spam-clicks "Withdraw", each request is queued
541
- // and processed sequentially - no double-withdrawals possible
542
- await this.queueBus
543
- .forProcessor(AccountProcessor)
544
- .enqueue(new WithdrawCommand(
545
- accountId,
546
- body.amount,
547
- transactionId,
548
- body.requestedBy,
549
- ));
550
-
551
- return {
552
- queued: true,
553
- transactionId,
554
- message: 'Withdrawal queued for processing',
555
- };
510
+ await this.queueBus.enqueue(
511
+ new WithdrawCommand(accountId, body.amount, transactionId)
512
+ );
513
+
514
+ return { queued: true, transactionId };
556
515
  }
557
516
 
558
517
  @Post(':accountId/transfer')
@@ -562,99 +521,50 @@ export class AccountController {
562
521
  ) {
563
522
  const transactionId = uuid();
564
523
 
565
- await this.queueBus
566
- .forProcessor(AccountProcessor)
567
- .enqueue(new TransferCommand(
568
- accountId,
569
- body.toAccountId,
570
- body.amount,
571
- transactionId,
572
- ));
573
-
574
- return {
575
- queued: true,
576
- transactionId,
577
- message: 'Transfer queued for processing',
578
- };
524
+ await this.queueBus.enqueue(
525
+ new TransferCommand(accountId, body.toAccountId, body.amount, transactionId)
526
+ );
527
+
528
+ return { queued: true, transactionId };
579
529
  }
580
530
  }
581
531
  ```
582
532
 
583
533
  ---
584
534
 
585
- ## Configuration
586
-
587
- ```typescript
588
- AtomicQueuesModule.forRoot({
589
- redis: {
590
- host: 'localhost',
591
- port: 6379,
592
- password: 'secret',
593
- },
594
-
595
- keyPrefix: 'myapp', // Redis key prefix (default: 'aq')
596
-
597
- enableCronManager: true, // Enable auto-scaling (default: false)
598
- cronInterval: 5000, // Scaling check interval (default: 5000ms)
599
-
600
- verbose: false, // Enable verbose logging (default: false)
601
- // When true, logs service job processing details
602
-
603
- workerDefaults: {
604
- concurrency: 1, // Jobs processed simultaneously
605
- stalledInterval: 1000, // Stalled job check interval
606
- lockDuration: 30000, // Job lock duration
607
- heartbeatTTL: 3, // Worker heartbeat TTL (seconds)
608
- },
609
- });
610
- ```
611
-
612
- ---
613
-
614
- ## Command Registration
535
+ ## Advanced: Custom Worker Processors
615
536
 
616
- By default, atomic-queues **auto-discovers** all commands from your `@CommandHandler` and `@QueryHandler` decorators. No manual registration needed!
537
+ For special cases where you need custom job handling logic, you can still define a `@WorkerProcessor`:
617
538
 
618
- ### Auto-Discovery (Default)
539
+ ```typescript
540
+ import { Injectable } from '@nestjs/common';
541
+ import { WorkerProcessor, JobHandler } from 'atomic-queues';
542
+ import { Job } from 'bullmq';
619
543
 
620
- Commands are automatically discovered when you have CQRS handlers:
544
+ @WorkerProcessor({
545
+ entityType: 'account',
546
+ queueName: (id) => `${id}-queue`,
547
+ workerName: (id) => `${id}-worker`,
548
+ maxWorkersPerEntity: 1,
549
+ idleTimeoutSeconds: 15,
550
+ })
551
+ @Injectable()
552
+ export class AccountProcessor {
553
+ // Custom handler for specific job types
554
+ @JobHandler('special-operation')
555
+ async handleSpecialOperation(job: Job, entityId: string) {
556
+ // Custom logic here
557
+ }
621
558
 
622
- ```typescript
623
- // Your handler - that's all you need!
624
- @CommandHandler(ProcessOrderCommand)
625
- export class ProcessOrderHandler implements ICommandHandler<ProcessOrderCommand> {
626
- async execute(command: ProcessOrderCommand) {
627
- // ProcessOrderCommand is auto-registered with QueueBus
559
+ // Wildcard handler for everything else
560
+ @JobHandler('*')
561
+ async handleAll(job: Job, entityId: string) {
562
+ // Falls back to CQRS routing automatically
628
563
  }
629
564
  }
630
565
  ```
631
566
 
632
- ### Manual Registration (Optional)
633
-
634
- If you need to register commands without handlers, or disable auto-discovery:
635
-
636
- ```typescript
637
- // Disable auto-discovery in config
638
- AtomicQueuesModule.forRoot({
639
- redis: { host: 'localhost', port: 6379 },
640
- autoRegisterCommands: false, // Disable auto-discovery
641
- });
642
-
643
- // Then manually register
644
- QueueBus.registerCommands(ProcessOrderCommand, ShipOrderCommand);
645
- ```
646
-
647
- ---
648
-
649
- ## Why Use atomic-queues?
650
-
651
- | Feature | Without | With atomic-queues |
652
- |---------|---------|-------------------|
653
- | Sequential per-entity | Manual locking | Automatic via queues |
654
- | Race conditions | Possible | Prevented |
655
- | Worker management | Manual | Automatic |
656
- | Horizontal scaling | Complex | Built-in |
657
- | Code organization | Scattered | Clean decorators |
567
+ **Note:** When you define a `@WorkerProcessor` for an entity type, it takes precedence over config-based default registration.
658
568
 
659
569
  ---
660
570