atomic-queues 2.0.0 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +227 -138
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -34,15 +34,21 @@
|
|
|
34
34
|
|
|
35
35
|
## What is atomic-queues?
|
|
36
36
|
|
|
37
|
-
A distributed virtual actor runtime for Node.js
|
|
37
|
+
**A distributed virtual actor runtime for Node.js, built entirely on Redis primitives.**
|
|
38
38
|
|
|
39
|
-
|
|
39
|
+
Think [Microsoft Orleans](https://learn.microsoft.com/en-us/dotnet/orleans/) or [Akka](https://akka.io/) — but for the NestJS ecosystem, requiring nothing beyond a Redis instance you probably already have.
|
|
40
|
+
|
|
41
|
+
Messages addressed to the same entity execute sequentially. Messages addressed to different entities execute in parallel. No distributed locks. No worker processes. No message broker. No BullMQ.
|
|
42
|
+
|
|
43
|
+
```
|
|
44
|
+
npm install atomic-queues ioredis
|
|
45
|
+
```
|
|
40
46
|
|
|
41
47
|
---
|
|
42
48
|
|
|
43
49
|
## The Problem
|
|
44
50
|
|
|
45
|
-
Every distributed system eventually
|
|
51
|
+
Every distributed system eventually builds toward one of two failure modes: **state corruption** from concurrent mutations on the same entity, or **throughput collapse** from the locking mechanisms used to prevent it.
|
|
46
52
|
|
|
47
53
|
```
|
|
48
54
|
Time Request A Request B Database
|
|
@@ -55,9 +61,13 @@ T₃ UPDATE: $100 − $80 = $20 −$60
|
|
|
55
61
|
Result: Balance is −$60. Both withdrawals succeed. Integrity violated.
|
|
56
62
|
```
|
|
57
63
|
|
|
58
|
-
|
|
64
|
+
The standard answers — `SELECT ... FOR UPDATE`, optimistic locking with retries, distributed locks via Redlock or ZooKeeper, serializable transactions — all trade throughput for correctness. Under load, they become bottlenecks. Across services, they become nightmares. And every team ends up inventing some ad-hoc combination of them, poorly, under production pressure.
|
|
65
|
+
|
|
66
|
+
## The Insight
|
|
59
67
|
|
|
60
|
-
|
|
68
|
+
The problem disappears if you change *when* serialization happens. Instead of serializing at the database level (row locks, transaction isolation), serialize at the **message level**: route all operations for a given entity through a single ordered log, and process that log sequentially. Different entities maintain independent logs with zero coordination between them.
|
|
69
|
+
|
|
70
|
+
This is the virtual actor model. It's not new — Erlang/OTP has used it since the 1980s, Orleans shipped it in 2014, Akka has been doing it on the JVM for over a decade. What *is* new is implementing it with nothing beyond Redis and making it native to the NestJS ecosystem.
|
|
61
71
|
|
|
62
72
|
```
|
|
63
73
|
┌─────────────────────────────────────────────────┐
|
|
@@ -67,48 +77,22 @@ atomic-queues routes operations through per-entity message logs. Same entity →
|
|
|
67
77
|
│ │ └──────┘ └──────┘ └──────┘ │ │
|
|
68
78
|
Request C ─┘ │ Sequential ◄────────────┘ │
|
|
69
79
|
└─────────────────────────────────────────────────┘
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
---
|
|
73
|
-
|
|
74
|
-
## Installation
|
|
75
80
|
|
|
76
|
-
|
|
77
|
-
|
|
81
|
+
Meanwhile, account-99, order-7, user-abc — all execute
|
|
82
|
+
in parallel on the same cluster, completely independent.
|
|
78
83
|
```
|
|
79
84
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
```bash
|
|
83
|
-
npm install @nestjs/cqrs # for CQRS surface
|
|
84
|
-
npm install zod zod-to-json-schema # for schema validation in the registry
|
|
85
|
-
```
|
|
85
|
+
This eliminates an entire class of bugs — lost updates, dirty reads, write skew, phantom reads on hot entities — without pessimistic locks, without optimistic retries, and without the `SELECT ... FOR UPDATE` that your DBA tells you not to use under load. The entity itself becomes the consistency boundary, and the consistency is structural rather than transactional.
|
|
86
86
|
|
|
87
87
|
---
|
|
88
88
|
|
|
89
|
-
##
|
|
90
|
-
|
|
91
|
-
### Minimal setup
|
|
92
|
-
|
|
93
|
-
```typescript
|
|
94
|
-
import { Module } from '@nestjs/common';
|
|
95
|
-
import { AtomicQueuesModule } from 'atomic-queues';
|
|
89
|
+
## How It Works
|
|
96
90
|
|
|
97
|
-
|
|
98
|
-
imports: [
|
|
99
|
-
AtomicQueuesModule.forRoot({
|
|
100
|
-
redis: { host: 'localhost', port: 6379 },
|
|
101
|
-
}),
|
|
102
|
-
],
|
|
103
|
-
})
|
|
104
|
-
export class AppModule {}
|
|
105
|
-
```
|
|
91
|
+
### Entities and messages
|
|
106
92
|
|
|
107
|
-
|
|
93
|
+
Everything in atomic-queues is an **entity** that receives **messages**. An entity is identified by a type and an ID — `account:a-42`, `order:o-17`, `user:u-abc`. A message is a command or query addressed to a specific entity instance. You define this relationship with two decorators:
|
|
108
94
|
|
|
109
95
|
```typescript
|
|
110
|
-
import { EntityType, QueueEntityId } from 'atomic-queues';
|
|
111
|
-
|
|
112
96
|
@EntityType('account')
|
|
113
97
|
export class WithdrawCommand {
|
|
114
98
|
constructor(
|
|
@@ -118,53 +102,38 @@ export class WithdrawCommand {
|
|
|
118
102
|
}
|
|
119
103
|
```
|
|
120
104
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
```typescript
|
|
124
|
-
import { QueueBus } from 'atomic-queues';
|
|
125
|
-
|
|
126
|
-
@Injectable()
|
|
127
|
-
export class PaymentService {
|
|
128
|
-
constructor(private readonly queueBus: QueueBus) {}
|
|
129
|
-
|
|
130
|
-
async withdraw(accountId: string, amount: number) {
|
|
131
|
-
await this.queueBus.enqueue(new WithdrawCommand(accountId, amount));
|
|
132
|
-
}
|
|
133
|
-
}
|
|
134
|
-
```
|
|
135
|
-
|
|
136
|
-
That's it. The command is appended to `account:{accountId}`'s message log and executed sequentially by the shared executor pool.
|
|
137
|
-
|
|
138
|
-
---
|
|
139
|
-
|
|
140
|
-
## Three Surfaces
|
|
105
|
+
That's the entire contract. `@EntityType` says "this message targets the `account` entity type." `@QueueEntityId()` says "the value of `accountId` is the entity instance ID." When you enqueue this command, the runtime routes it to the log for `account:{accountId}` and guarantees sequential execution against that specific entity instance, cluster-wide.
|
|
141
106
|
|
|
142
|
-
|
|
107
|
+
### Two levels of abstraction
|
|
143
108
|
|
|
144
|
-
|
|
109
|
+
atomic-queues gives you two ways to handle messages, and they're not different systems — they're two levels of abstraction over the same dispatch engine.
|
|
145
110
|
|
|
146
|
-
|
|
111
|
+
**Actors** are the foundational primitive. An actor class *is* an entity — its fields are the state, its methods are message handlers. The runtime manages its lifecycle: activate on first message, evict from memory on idle, persist state to Redis automatically, restore on reactivation.
|
|
147
112
|
|
|
148
113
|
```typescript
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
await queueBus.forEntity('account').enqueue(new WithdrawCommand(accountId, 100));
|
|
114
|
+
@Actor('account')
|
|
115
|
+
@Injectable()
|
|
116
|
+
export class AccountActor {
|
|
117
|
+
private balance = 0;
|
|
154
118
|
|
|
155
|
-
|
|
156
|
-
|
|
119
|
+
@On(DepositCommand)
|
|
120
|
+
async deposit(msg: DepositCommand) {
|
|
121
|
+
this.balance += msg.amount;
|
|
122
|
+
return this.balance;
|
|
123
|
+
}
|
|
157
124
|
|
|
158
|
-
|
|
159
|
-
|
|
125
|
+
@On(WithdrawCommand)
|
|
126
|
+
async withdraw(msg: WithdrawCommand) {
|
|
127
|
+
if (this.balance < msg.amount) throw new InsufficientFunds();
|
|
128
|
+
this.balance -= msg.amount;
|
|
129
|
+
return this.balance;
|
|
130
|
+
}
|
|
131
|
+
}
|
|
160
132
|
```
|
|
161
133
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
For teams using `@nestjs/cqrs`. Commands and queries route through the actor runtime instead of executing inline.
|
|
134
|
+
**CQRS handlers** are the convenience layer for teams using `@nestjs/cqrs`. You don't write actor classes — you write standard `@CommandHandler` and `@QueryHandler` classes exactly as NestJS CQRS prescribes, and atomic-queues intercepts the dispatch to route them through the same per-entity log and gate system. The handler code doesn't change. The guarantee changes — instead of executing inline on whatever request thread happens to call `commandBus.execute()`, your handler now executes sequentially per entity, cluster-wide.
|
|
165
135
|
|
|
166
136
|
```typescript
|
|
167
|
-
@JobCommand()
|
|
168
137
|
@EntityType('account')
|
|
169
138
|
export class WithdrawCommand {
|
|
170
139
|
constructor(
|
|
@@ -176,69 +145,120 @@ export class WithdrawCommand {
|
|
|
176
145
|
@CommandHandler(WithdrawCommand)
|
|
177
146
|
export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
|
|
178
147
|
async execute(cmd: WithdrawCommand) {
|
|
179
|
-
//
|
|
148
|
+
// This runs sequentially per account — cluster-wide.
|
|
149
|
+
// No locks. No transactions. The dispatch engine guarantees it.
|
|
180
150
|
}
|
|
181
151
|
}
|
|
182
152
|
```
|
|
183
153
|
|
|
184
|
-
|
|
154
|
+
The library auto-discovers `@CommandHandler` and `@QueryHandler` classes at boot and wires them into the dispatch pipeline. Your existing CQRS architecture gets per-entity sequential guarantees without changing a single handler. The CQRS surface *calls into the actor runtime* — it's not a separate execution path.
|
|
185
155
|
|
|
186
|
-
|
|
156
|
+
### Enqueuing messages
|
|
187
157
|
|
|
188
158
|
```typescript
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
@Actor('account')
|
|
192
|
-
@Injectable()
|
|
193
|
-
export class AccountActor {
|
|
194
|
-
private balance = 0;
|
|
159
|
+
// Fire-and-forget
|
|
160
|
+
await queueBus.enqueue(new WithdrawCommand(accountId, 100));
|
|
195
161
|
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
this.balance += msg.amount;
|
|
199
|
-
return this.balance;
|
|
200
|
-
}
|
|
162
|
+
// Enqueue and block until result
|
|
163
|
+
const balance = await queueBus.enqueueAndWait(new GetBalanceQuery(accountId));
|
|
201
164
|
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
if (this.balance < msg.amount) throw new InsufficientFunds();
|
|
205
|
-
this.balance -= msg.amount;
|
|
206
|
-
return this.balance;
|
|
207
|
-
}
|
|
208
|
-
}
|
|
165
|
+
// Scoped to an entity type
|
|
166
|
+
await queueBus.forEntity('account').enqueueBulk([charge1, charge2, charge3]);
|
|
209
167
|
|
|
210
|
-
//
|
|
168
|
+
// Actor-style direct send
|
|
211
169
|
await actorSystem.send('account', accountId, new DepositCommand(100));
|
|
212
170
|
const balance = await actorSystem.sendAndWait('account', accountId, new GetBalanceQuery());
|
|
213
171
|
```
|
|
214
172
|
|
|
215
|
-
|
|
173
|
+
---
|
|
174
|
+
|
|
175
|
+
## The Dispatch Engine
|
|
176
|
+
|
|
177
|
+
Under every API call is the same pipeline: **message → Redis log → Lua scheduler → gate → executor → handler**. Understanding this pipeline is key to understanding what atomic-queues actually guarantees and why it can guarantee it without locks.
|
|
178
|
+
|
|
179
|
+
### Per-entity message logs
|
|
180
|
+
|
|
181
|
+
When you call `enqueue()`, the message is serialized to JSON and appended to a Redis list (`LPUSH aq:log:account:a-42`), and the entity key is added to a global ready set (`SADD aq:ready account:a-42`). A pub/sub notification wakes the executor pool. Three Redis commands, pipelined in one round-trip.
|
|
182
|
+
|
|
183
|
+
The log is the source of truth for ordering. Redis lists are FIFO — `LPUSH` appends to the head, `RPOP` consumes from the tail. Messages for the same entity are always processed in enqueue order.
|
|
184
|
+
|
|
185
|
+
### The dispatch gate
|
|
186
|
+
|
|
187
|
+
The core consistency primitive is the **dispatch gate** — a Redis key per entity (`SET aq:gate:account:a-42 <token> EX 30 NX`). The `NX` flag means only one executor can acquire it. The `EX` TTL means a crashed executor releases it automatically. This is not a distributed lock in the Redlock sense — there's no quorum, no retry loop, no backoff. If the gate is held, the scheduler moves on to the next ready entity. Zero contention between entities, zero blocking within the scheduling loop.
|
|
188
|
+
|
|
189
|
+
### Atomic Lua scheduling
|
|
190
|
+
|
|
191
|
+
A single Lua script runs atomically in Redis to perform the entire dispatch cycle:
|
|
192
|
+
|
|
193
|
+
1. Sample entities from the ready set (`SRANDMEMBER` with batch size 32)
|
|
194
|
+
2. Try to acquire the gate for each candidate (`SET NX EX`)
|
|
195
|
+
3. On first successful acquisition, pop the next message from that entity's log (`RPOP`)
|
|
196
|
+
4. Remove the entity from the ready set if its log is now empty
|
|
197
|
+
|
|
198
|
+
Because Lua scripts execute atomically in Redis, the pick → gate acquisition → message pop sequence cannot be interleaved by another executor on another node. This is what eliminates race conditions — not locks, but atomicity at the Redis command level.
|
|
199
|
+
|
|
200
|
+
### Shared executor pool
|
|
201
|
+
|
|
202
|
+
Traditional queue systems spawn a worker per queue or per entity type. With thousands of entities, that means thousands of blocking Redis connections, thousands of event loops, and a scaling problem that grows linearly with your domain model.
|
|
203
|
+
|
|
204
|
+
atomic-queues uses a **shared executor pool** — a configurable number of concurrent executors per node that dispatch messages from *any* ready entity. One pool can service millions of distinct entities. The pool self-regulates: it drains the ready set until empty or until the concurrency limit is hit, then sleeps until the next pub/sub tickle wakes it. There are no workers to spawn, monitor, or auto-scale.
|
|
205
|
+
|
|
206
|
+
### Gate refresh for long-running handlers
|
|
207
|
+
|
|
208
|
+
If a handler runs longer than the gate TTL, the gate doesn't expire — the executor pool runs a background interval that extends the TTL while the handler is still executing. This prevents false recovery (another node re-dispatching the same message) without requiring an unreasonably large TTL as the safety default.
|
|
209
|
+
|
|
210
|
+
### Multiplexed result collection
|
|
211
|
+
|
|
212
|
+
Request-reply (`enqueueAndWait` / `sendAndWait`) uses a single `PSUBSCRIBE` connection per node for all concurrent result waits. Hundreds or thousands of pending results share one TCP connection to Redis, routed to the correct promise via correlation ID. No connection-per-call, no connection pool exhaustion, no subscriber amplification.
|
|
216
213
|
|
|
217
214
|
---
|
|
218
215
|
|
|
219
216
|
## Cross-Service Communication
|
|
220
217
|
|
|
221
|
-
|
|
218
|
+
This is where atomic-queues stops being a "queue library" and becomes a **distributed coordination primitive**.
|
|
219
|
+
|
|
220
|
+
### The problem it solves
|
|
221
|
+
|
|
222
|
+
In a microservices architecture, the standard way for Service A to tell Service B to do something is: define a gRPC/REST contract, deploy an API gateway or service mesh, handle serialization, implement retries, manage circuit breakers, and hope the schema stays in sync across repos. For async communication, add a message broker (RabbitMQ, Kafka, SQS), define topic/queue naming conventions, implement dead-letter handling, and build consumer groups.
|
|
223
|
+
|
|
224
|
+
atomic-queues replaces all of that with Redis.
|
|
225
|
+
|
|
226
|
+
### How it works
|
|
227
|
+
|
|
228
|
+
Enable the distributed registry and any service connected to the same Redis instance can send typed messages to any entity — regardless of which service owns the handler.
|
|
222
229
|
|
|
223
230
|
```typescript
|
|
224
|
-
//
|
|
231
|
+
// billing-service: defines and handles the entity
|
|
225
232
|
AtomicQueuesModule.forRoot({
|
|
226
233
|
redis: { url: process.env.REDIS_URL },
|
|
227
|
-
registry: {
|
|
228
|
-
enabled: true,
|
|
229
|
-
serviceName: 'billing-service',
|
|
230
|
-
},
|
|
234
|
+
registry: { enabled: true, serviceName: 'billing-service' },
|
|
231
235
|
})
|
|
232
236
|
|
|
233
|
-
//
|
|
237
|
+
// payments-service: sends to it (shared Redis, no code dependency on billing)
|
|
234
238
|
await queueBus.enqueue(new WithdrawCommand(accountId, 100));
|
|
235
239
|
```
|
|
236
240
|
|
|
237
|
-
|
|
241
|
+
When `billing-service` starts, it scans its own `@Actor`, `@CommandHandler`, and `@QueryHandler` classes and publishes **entity contracts** to Redis — a JSON document listing the entity type, accepted messages, and optional JSON schemas, refreshed via heartbeat TTL. When `payments-service` enqueues a message, the registry validates it at the call site *before* it enters the log: entity type exists, message name is accepted, payload matches schema. Errors are immediate and descriptive — not silent dead letters discovered hours later in a DLQ dashboard.
|
|
242
|
+
|
|
243
|
+
### What this replaces
|
|
244
|
+
|
|
245
|
+
Think about what you no longer need:
|
|
238
246
|
|
|
239
|
-
|
|
247
|
+
**No API gateway between services.** Messages go directly into the entity's log via Redis. The "endpoint" is the entity type and message name, not a URL.
|
|
240
248
|
|
|
241
|
-
|
|
249
|
+
**No message broker.** Redis is the transport, the ordering guarantee, and the persistence layer. You don't need RabbitMQ, Kafka, or SQS to get async cross-service communication with ordering guarantees.
|
|
250
|
+
|
|
251
|
+
**No schema registry as a separate service.** The entity contracts live in Redis alongside the message logs. Schema validation happens at the call site. Zod schemas on the producer side serialize to JSON Schema in the registry and validate on every enqueue.
|
|
252
|
+
|
|
253
|
+
**No service discovery.** The registry *is* service discovery. When a service starts, it publishes what it handles. When a service stops, its registrations TTL out. Other services discover capabilities by reading the registry.
|
|
254
|
+
|
|
255
|
+
**No serialization framework.** Messages are JSON. The wire protocol is three Redis commands. No Protobuf compilation step, no `.proto` files, no code generation from IDL. (Though atomic-queues does offer codegen from the live registry — it generates TypeScript interfaces so Service A gets compile-time type safety for messages destined to Service B, without importing Service B's code.)
|
|
256
|
+
|
|
257
|
+
**No separate dead-letter infrastructure.** Failed messages are dead-lettered per entity type in Redis, queryable via the same connection.
|
|
258
|
+
|
|
259
|
+
### Schema validation
|
|
260
|
+
|
|
261
|
+
Attach Zod schemas to message classes for runtime safety across service boundaries:
|
|
242
262
|
|
|
243
263
|
```typescript
|
|
244
264
|
import { Schema } from 'atomic-queues';
|
|
@@ -255,9 +275,13 @@ export class WithdrawCommand {
|
|
|
255
275
|
}
|
|
256
276
|
```
|
|
257
277
|
|
|
258
|
-
|
|
278
|
+
The Zod schema serializes to JSON Schema and stores in the registry. Every service validates payloads against it — even services that don't import your code, even services written in a different language that read the registry directly from Redis.
|
|
259
279
|
|
|
260
|
-
###
|
|
280
|
+
### Entity co-ownership
|
|
281
|
+
|
|
282
|
+
Multiple services can handle different message types on the same entity. Service A handles `DepositCommand` and `WithdrawCommand` on the `account` entity type. Service B handles `FreezeAccountCommand` on the same entity type. The registry merges their contracts automatically. The dispatch gate still ensures single-writer semantics per entity instance, regardless of which service's executor picks up the message.
|
|
283
|
+
|
|
284
|
+
### Contract codegen
|
|
261
285
|
|
|
262
286
|
Generate typed interfaces from the live registry:
|
|
263
287
|
|
|
@@ -265,7 +289,75 @@ Generate typed interfaces from the live registry:
|
|
|
265
289
|
REDIS_URL=redis://localhost:6379 npx atomic-queues generate --ts --output ./generated/contracts.ts
|
|
266
290
|
```
|
|
267
291
|
|
|
268
|
-
|
|
292
|
+
Also supports `--json-schema` for language-agnostic schema export and `--snapshot` for full registry dumps.
|
|
293
|
+
|
|
294
|
+
---
|
|
295
|
+
|
|
296
|
+
## Redis *is* the Protocol
|
|
297
|
+
|
|
298
|
+
This is the most important architectural decision in the project, and it has implications that go far beyond NestJS.
|
|
299
|
+
|
|
300
|
+
The wire protocol is [fully documented](./WIRE-PROTOCOL.md), intentionally simple, and versioned with breaking-change semantics. Enqueuing a message is three Redis commands:
|
|
301
|
+
|
|
302
|
+
```
|
|
303
|
+
LPUSH aq:log:account:a-1 '<message JSON>'
|
|
304
|
+
SADD aq:ready account:a-1
|
|
305
|
+
PUBLISH aq:tickle 1
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
**Any language with a Redis client is a first-class citizen.** A Python data pipeline can enqueue commands to a NestJS-hosted actor. A Go microservice can fire events at entities defined in TypeScript. A Rust executor can run the same Lua scheduling script and compete for gates on equal terms with the Node.js executor pool. A Bash script can trigger a workflow.
|
|
309
|
+
|
|
310
|
+
This is not a feature of any existing mainstream actor framework. Orleans requires the Orleans silo. Akka requires the JVM. Temporal requires the Temporal server with its own database. All of them are monoglot execution environments — actors must be written in the framework's language.
|
|
311
|
+
|
|
312
|
+
atomic-queues is **polyglot by construction**. The coordination happens in Redis, not in the runtime. Any process that speaks the wire protocol participates on equal terms, and the [WIRE-PROTOCOL.md](./WIRE-PROTOCOL.md) includes a complete Python reference client to prove it.
|
|
313
|
+
|
|
314
|
+
This opens architectures that are genuinely difficult to build otherwise:
|
|
315
|
+
|
|
316
|
+
- **Ingest in Go, process in Node.js, analyze in Python.** Each layer speaks Redis. The entity logs are the integration boundary.
|
|
317
|
+
- **Rust executors for CPU-hot-path actors.** The same Lua scheduler, the same gates, the same entity logs. The Rust process is just another executor that happens to be faster. The Node.js side doesn't know or care.
|
|
318
|
+
- **Gradual migration.** Move one entity type's handlers to a different service, a different language, or a different infrastructure — without touching any other service's code. The entity contract in the registry is the interface, not the import statement.
|
|
319
|
+
- **Edge coordination.** An IoT device with a Redis client and 3 commands of knowledge can participate in the same entity model as your cloud services.
|
|
320
|
+
|
|
321
|
+
---
|
|
322
|
+
|
|
323
|
+
## Quick Start
|
|
324
|
+
|
|
325
|
+
```typescript
|
|
326
|
+
import { Module } from '@nestjs/common';
|
|
327
|
+
import { AtomicQueuesModule } from 'atomic-queues';
|
|
328
|
+
|
|
329
|
+
@Module({
|
|
330
|
+
imports: [
|
|
331
|
+
AtomicQueuesModule.forRoot({
|
|
332
|
+
redis: { host: 'localhost', port: 6379 },
|
|
333
|
+
}),
|
|
334
|
+
],
|
|
335
|
+
})
|
|
336
|
+
export class AppModule {}
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
Define a command and enqueue it:
|
|
340
|
+
|
|
341
|
+
```typescript
|
|
342
|
+
@EntityType('account')
|
|
343
|
+
export class WithdrawCommand {
|
|
344
|
+
constructor(
|
|
345
|
+
@QueueEntityId() public readonly accountId: string,
|
|
346
|
+
public readonly amount: number,
|
|
347
|
+
) {}
|
|
348
|
+
}
|
|
349
|
+
|
|
350
|
+
@Injectable()
|
|
351
|
+
export class PaymentService {
|
|
352
|
+
constructor(private readonly queueBus: QueueBus) {}
|
|
353
|
+
|
|
354
|
+
async withdraw(accountId: string, amount: number) {
|
|
355
|
+
await this.queueBus.enqueue(new WithdrawCommand(accountId, amount));
|
|
356
|
+
}
|
|
357
|
+
}
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
The command is appended to `account:{accountId}`'s message log and executed sequentially by the shared executor pool. No handler registration, no worker setup, no queue configuration.
|
|
269
361
|
|
|
270
362
|
---
|
|
271
363
|
|
|
@@ -286,12 +378,12 @@ AtomicQueuesModule.forRoot({
|
|
|
286
378
|
gateTTL: 60,
|
|
287
379
|
retry: { maxAttempts: 5, backoff: 'exponential', backoffDelay: 2000 },
|
|
288
380
|
actorIdleTimeout: 120000,
|
|
289
|
-
statePersistence: true,
|
|
381
|
+
statePersistence: true,
|
|
290
382
|
},
|
|
291
383
|
},
|
|
292
384
|
|
|
293
385
|
registry: {
|
|
294
|
-
enabled: false,
|
|
386
|
+
enabled: false,
|
|
295
387
|
serviceName: 'my-service',
|
|
296
388
|
schemaValidation: false,
|
|
297
389
|
heartbeatInterval: 10000,
|
|
@@ -303,45 +395,44 @@ AtomicQueuesModule.forRoot({
|
|
|
303
395
|
})
|
|
304
396
|
```
|
|
305
397
|
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
## Architecture
|
|
398
|
+
Optional peer dependencies:
|
|
309
399
|
|
|
310
|
-
|
|
400
|
+
```bash
|
|
401
|
+
npm install @nestjs/cqrs # for CQRS handler auto-wiring
|
|
402
|
+
npm install zod zod-to-json-schema # for schema validation in the registry
|
|
403
|
+
```
|
|
311
404
|
|
|
312
|
-
|
|
313
|
-
2. **Tickle**: a pub/sub notification wakes the executor pool.
|
|
314
|
-
3. **Schedule**: a Lua script atomically picks an entity from the ready set, acquires its dispatch gate (`SET NX EX`), and pops the next message.
|
|
315
|
-
4. **Execute**: the handler runs (actor method, CQRS handler, or registered processor).
|
|
316
|
-
5. **Complete**: gate is released, entity is re-added to the ready set if more messages remain.
|
|
405
|
+
---
|
|
317
406
|
|
|
318
|
-
|
|
407
|
+
## Guarantees
|
|
319
408
|
|
|
320
409
|
| Guarantee | Scope | Mechanism |
|
|
321
410
|
|---|---|---|
|
|
322
|
-
| FIFO per entity | Cluster-wide | Redis list (LPUSH
|
|
323
|
-
| Single-writer per entity | Cluster-wide | Gate key (SET NX EX) |
|
|
411
|
+
| FIFO per entity | Cluster-wide | Redis list (`LPUSH`/`RPOP`) |
|
|
412
|
+
| Single-writer per entity | Cluster-wide | Gate key (`SET NX EX`) |
|
|
324
413
|
| At-least-once delivery | Per message | Retry on gate TTL expiry |
|
|
325
414
|
| Parallel across entities | Per node | Executor pool concurrency |
|
|
326
415
|
| Durability | Per message | Redis persistence (AOF/RDB) |
|
|
327
416
|
|
|
328
417
|
### What this does NOT guarantee
|
|
329
418
|
|
|
330
|
-
**Exactly-once processing.** Like every distributed message system, handlers must be idempotent. If an executor
|
|
419
|
+
**Exactly-once processing.** Like every distributed message system — Orleans, Akka, Temporal, Kafka — handlers must be idempotent. If an executor crashes mid-processing, the gate TTL expires and the message retries on another node. This is a fundamental constraint of distributed systems, not a limitation of the library.
|
|
331
420
|
|
|
332
421
|
---
|
|
333
422
|
|
|
334
|
-
##
|
|
335
|
-
|
|
336
|
-
Redis is the protocol. Any language with a Redis client can send messages to atomic-queues entities — three Redis commands:
|
|
337
|
-
|
|
338
|
-
```
|
|
339
|
-
LPUSH {prefix}:log:{entityType}:{entityId} '<message JSON>'
|
|
340
|
-
SADD {prefix}:ready {entityType}:{entityId}
|
|
341
|
-
PUBLISH {prefix}:tickle 1
|
|
342
|
-
```
|
|
423
|
+
## How It Compares
|
|
343
424
|
|
|
344
|
-
|
|
425
|
+
| Capability | BullMQ | Temporal | atomic-queues |
|
|
426
|
+
|---|---|---|---|
|
|
427
|
+
| Per-entity ordering | Manual (named queues) | Workflow-scoped | Built-in, zero config |
|
|
428
|
+
| Cross-entity parallelism | Worker pools | Worker pools | Shared executor pool |
|
|
429
|
+
| Stateful entities | No | Workflow state | Virtual actors |
|
|
430
|
+
| Cross-service messaging | Shared queue names | gRPC | Redis registry + codegen |
|
|
431
|
+
| Polyglot clients | JS/TS only | SDK per language | Any Redis client (3 commands) |
|
|
432
|
+
| Infrastructure required | Redis | Temporal server + DB | Redis only |
|
|
433
|
+
| Distributed locks needed | Yes, for ordering | Internal | None — gates are non-contending |
|
|
434
|
+
| Service discovery | External | Built-in | Built-in (registry) |
|
|
435
|
+
| Schema validation | No | Protobuf | Zod → JSON Schema |
|
|
345
436
|
|
|
346
437
|
---
|
|
347
438
|
|
|
@@ -349,11 +440,9 @@ See [WIRE-PROTOCOL.md](./WIRE-PROTOCOL.md) for the complete specification.
|
|
|
349
440
|
|
|
350
441
|
| Decorator | Purpose |
|
|
351
442
|
|---|---|
|
|
352
|
-
| `@EntityType('type')` | Route a
|
|
443
|
+
| `@EntityType('type')` | Route a message to an entity type |
|
|
353
444
|
| `@QueueEntityId()` | Mark the property holding the entity ID |
|
|
354
445
|
| `@QueueEntity('type', 'prop')` | Combined entity type + ID |
|
|
355
|
-
| `@JobCommand()` | Mark a command for CQRS auto-routing |
|
|
356
|
-
| `@JobQuery()` | Mark a query for CQRS auto-routing |
|
|
357
446
|
| `@Actor('type')` | Declare a virtual actor class |
|
|
358
447
|
| `@On(MessageClass)` | Handle a message type on an actor |
|
|
359
448
|
| `@Schema(zodSchema)` | Attach a Zod schema for registry validation |
|
|
@@ -364,7 +453,7 @@ See [WIRE-PROTOCOL.md](./WIRE-PROTOCOL.md) for the complete specification.
|
|
|
364
453
|
|
|
365
454
|
V2 is a full rewrite of the internals. BullMQ is removed. Workers are removed. The public API is largely preserved.
|
|
366
455
|
|
|
367
|
-
**What stays the same**: `@EntityType`, `@QueueEntityId`, `@QueueEntity`,
|
|
456
|
+
**What stays the same**: `@EntityType`, `@QueueEntityId`, `@QueueEntity`, `queueBus.enqueue()`, `queueBus.forEntity()`, `queueBus.enqueueAndWait()`.
|
|
368
457
|
|
|
369
458
|
**What's removed**: `@WorkerProcessor`, `@JobHandler`, `@EntityScaler`, `@OnSpawnWorker`, `@OnTerminateWorker`, `@GetActiveEntities`, `@GetDesiredWorkerCount`, `.forProcessor()`. All worker and scaling concepts are gone.
|
|
370
459
|
|
package/package.json
CHANGED