atomic-queues 1.2.8 → 1.5.2

package/README.md

@@ -1,836 +1,563 @@
-# atomic-queues
+<p align="center">
+  <img src="https://img.shields.io/npm/v/atomic-queues?style=flat-square&color=cb3837" alt="npm version" />
+  <img src="https://img.shields.io/badge/NestJS-11-ea2845?style=flat-square&logo=nestjs" alt="NestJS 11" />
+  <img src="https://img.shields.io/badge/BullMQ-5-3c873a?style=flat-square" alt="BullMQ 5" />
+  <img src="https://img.shields.io/badge/Redis-7-dc382d?style=flat-square&logo=redis&logoColor=white" alt="Redis 7" />
+  <img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="MIT License" />
+</p>
 
-A NestJS library for atomic, sequential job processing per entity using BullMQ and Redis.
+<h1 align="center">atomic-queues</h1>
 
----
-
-## Overview
-
-**atomic-queues** solves the fundamental concurrency problem in distributed systems: ensuring that operations on the same logical entity execute sequentially, even when requests arrive simultaneously across multiple service instances.
-
-Rather than relying on distributed locks—which introduce contention, latency degradation, and complex failure modes—this library implements a **per-entity queue architecture** where each entity (user account, game table, order, document) has its own dedicated processing queue and worker.
+<p align="center">
+  <strong>Zero-contention, per-entity sequential processing for NestJS.</strong><br/>
+  Distributed. Lock-free.
+</p>
 
 ---
 
-## The Concurrency Problem
+## Why atomic-queues?
 
-### Race Condition Scenario
+Distributed locks (Redlock, advisory locks, optimistic locking) all share the same fundamental flaw: **contention collapse**. When multiple pods fight for the same lock simultaneously, they spend more time retrying failed acquisitions than doing actual work. The harder you push, the slower they go.
 
-Consider a financial system where a user with a $100 balance submits two concurrent $80 withdrawal requests:
-
-```
-Time    Request A                    Request B                    Database State
-─────────────────────────────────────────────────────────────────────────────────
-T₀      SELECT balance → $100        SELECT balance → $100        balance = $100
-T₁      CHECK: $100 >= $80 ✓         CHECK: $100 >= $80 ✓              
-T₂      UPDATE: balance = $20        UPDATE: balance = $20        balance = $20
-T₃                                   UPDATE: balance = -$60       balance = -$60
-─────────────────────────────────────────────────────────────────────────────────
-Result: Both withdrawals succeed. Balance becomes -$60. Integrity violated.
-```
+**atomic-queues** eliminates contention entirely. Instead of locking, each entity gets its own dedicated BullMQ queue. Operations execute sequentially — back-to-back with zero wasted cycles. There's nothing to contend over.
 
-This occurs because both transactions read the balance before either writes, a classic **lost update anomaly**.
+### atomic-queues vs Redlock
 
-### Traditional Solutions and Their Limitations
-
-| Approach | Mechanism | Failure Mode |
-|----------|-----------|--------------|
-| **Distributed Locks (Redlock)** | Acquire lock before operation, release after | Lock contention storms under high throughput; exponential latency degradation; lock holder failure requires TTL expiration |
-| **Database Row Locks** | `SELECT ... FOR UPDATE` | Connection pool exhaustion; deadlock risk in multi-entity transactions; database becomes bottleneck |
-| **Optimistic Concurrency Control** | Version numbers with conditional updates | Retry storms under contention; unbounded retries on hot entities; wasted compute cycles |
-| **Application Semaphores** | In-memory mutex/semaphore | Single-process only; ineffective in horizontally scaled deployments |
-
-**Fundamental limitation**: These approaches attempt to serialize access at the *moment of execution*. Under high contention, this creates a thundering herd where N requests compete for the same resource simultaneously.
+| | Redlock | atomic-queues |
+|---|---|---|
+| **Architecture** | Distributed mutex (quorum-based) | Per-entity queue (sequential) |
+| **Under contention** | Degrades — retry storms, backoff delays | **Constant** — jobs queue up, execute instantly |
+| **Per-entity throughput** | ~20-50 ops/s (heavy contention) | **~200-300 ops/s** (queue-bound, no contention) |
+| **Failure mode** | Silent double-execution (clock drift) | Job stuck in queue (visible, retryable) |
+| **Split-brain risk** | Yes (timing assumptions) | **Impossible** (serial queue) |
+| **Warm-path overhead** | 5-7ms per op (acquire + release) | **0ms** (in-memory hot cache) |
+| **Cold-start** | None | ~2-3ms one-time per entity |
+| **Multi-pod scaling** | Contention increases with pods | **Throughput increases with pods** |
 
 ---
 
-## The Per-Entity Queue Architecture
-
-### Design Principle
-
-Instead of serializing at execution time, **serialize at ingestion time**:
-
-```
-                                    ┌─────────────────────────────────────────┐
-   Request A ─┐                     │         Per-Entity Queue                │
-              │                     │  ┌─────┐ ┌─────┐ ┌─────┐               │
-   Request B ─┼──▶ [Entity Router] ─┼─▶│ Op₁ │→│ Op₂ │→│ Op₃ │→ [Worker] ─┐ │
-              │                     │  └─────┘ └─────┘ └─────┘             │ │
-   Request C ─┘                     │                                      │ │
-                                    │      Sequential Processing ◄─────────┘ │
-                                    └─────────────────────────────────────────┘
-```
-
-Operations targeting the same entity are immediately routed to that entity's queue. A dedicated worker processes operations one at a time, guaranteeing:
-
-1. **Serialized Execution**: Operations execute in FIFO order
-2. **Consistent State Visibility**: Each operation sees the result of all prior operations
-3. **Isolation**: No interleaving of concurrent modifications
-
-### Correctness Under Load
-
-```
-Time    Queue State                 Worker Execution              Database State
-───────────────────────────────────────────────────────────────────────────────────
-T₀      [Withdraw $80, Withdraw $80]                              balance = $100
-T₁      [Withdraw $80]              Process Op₁: $100 - $80       balance = $20
-T₂      []                          Process Op₂: $20 < $80 → REJECT   balance = $20
-───────────────────────────────────────────────────────────────────────────────────
-Result: First withdrawal succeeds. Second is rejected. Integrity preserved.
-```
-
----
-
-## Comparative Analysis
-
-### Behavioral Characteristics
-
-| Characteristic            | Distributed Locks                     | Per-Entity Queues                               |
-|-------------------------  |-------------------------------------- |-------------------                              |
-| **Request Handling**      | Block until lock acquired             | Queue immediately, return                       |
-| **Latency Distribution**  | Bimodal (fast if uncontested)         | Predictable (queue depth × avg processing time) |
-| **Throughput Ceiling**    | Limited by lock contention            | Limited only by worker processing rate          |
-| **Failure Recovery**      | Stuck locks until TTL expiration      | Failed jobs retry or move to dead-letter queue  |
-| **Ordering Guarantees**   | Non-deterministic (race to acquire)   | Deterministic FIFO                              |
-| **Observability**         | Lock wait times difficult to measure  | Queue depth, throughput directly observable     |
+## Table of Contents
 
-### Scalability Profile
-
-```
-Throughput
-    ▲
-    │                                    ╭──── Per-Entity Queues
-    │                                ╭───╯     (linear scaling)
-    │                            ╭───╯
-    │                        ╭───╯
-    │                    ╭───╯
-    │                ╭───╯         ╭────── Distributed Locks
-    │            ╭───╯         ╭───╯       (contention ceiling)
-    │        ╭───╯         ╭───╯
-    │    ╭───╯     ╭───────╯
-    │╭───╯ ╭───────╯
-    ├──────╯
-    └──────────────────────────────────────────────▶ Concurrent Requests
-
-    Lock-based systems hit a contention ceiling where adding more 
-    requests increases wait time faster than throughput.
-    
-    Queue-based systems scale linearly: each entity's queue is 
-    independent, so Entity A's load doesn't affect Entity B.
-```
+- [Why atomic-queues?](#why-atomic-queues)
+- [How It Works](#how-it-works)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Commands & Decorators](#commands--decorators)
+- [Configuration](#configuration)
+- [Distributed Worker Lifecycle](#distributed-worker-lifecycle)
+- [Complete Example](#complete-example)
+- [Advanced: Custom Worker Processors](#advanced-custom-worker-processors)
+- [Performance](#performance)
+- [License](#license)
 
 ---
 
-## Architecture
+## How It Works
 
-### Horizontal Scaling Model
+### The Problem
 
-Workers are distributed across service instances via Redis-based coordination:
+Every distributed system eventually hits this:
 
 ```
-┌─────────────────────────────────────────────────────────────────────────────────┐
-│                              REDIS CLUSTER                                       │
-│  ┌────────────────────────────────────────────────────────────────────────────┐ │
-│  │  Entity Queues                                                             │ │
-│  │  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐              │ │
-│  │  │ account:ACC-001 │ │ account:ACC-002 │ │ account:ACC-003 │  ...         │ │
-│  │  │ [op₁][op₂][op₃] │ │ [op₁]           │ │ [op₁][op₂]      │              │ │
-│  │  └────────┬────────┘ └────────┬────────┘ └────────┬────────┘              │ │
-│  │           │                   │                   │                        │ │
-│  │  ┌────────┴───────────────────┴───────────────────┴────────┐              │ │
-│  │  │              Worker Heartbeat Registry                   │              │ │
-│  │  │  ACC-001 → node-1 | ACC-002 → node-2 | ACC-003 → node-1  │              │ │
-│  │  └──────────────────────────────────────────────────────────┘              │ │
-│  └────────────────────────────────────────────────────────────────────────────┘ │
-└─────────────────────────────────────────────────────────────────────────────────┘
-           │                        │                        │
-           ▼                        ▼                        ▼
-┌─────────────────────┐  ┌─────────────────────┐  ┌─────────────────────┐
-│   Service Node 1    │  │   Service Node 2    │  │   Service Node 3    │
-│                     │  │                     │  │                     │
-│  ┌───────────────┐  │  │  ┌───────────────┐  │  │  ┌───────────────┐  │
-│  │ Worker ACC-001│  │  │  │ Worker ACC-002│  │  │  │ Worker ACC-004│  │
-│  │ Worker ACC-003│  │  │  │ Worker ACC-005│  │  │  │ Worker ACC-006│  │
-│  └───────────────┘  │  │  └───────────────┘  │  │  └───────────────┘  │
-└─────────────────────┘  └─────────────────────┘  └─────────────────────┘
+Time    Request A                    Request B                    Database
+──────────────────────────────────────────────────────────────────────────
+T₀      SELECT balance → $100        SELECT balance → $100        $100
+T₁      CHECK: $100 ≥ $80 ✓          CHECK: $100 ≥ $80 ✓
+T₂      UPDATE: $100 − $80 = $20                                  $20
+T₃                                   UPDATE: $100 − $80 = $20     −$60
+──────────────────────────────────────────────────────────────────────────
+Result: Balance is −$60. Both withdrawals succeed. Integrity violated.
 ```
 
-**Properties:**
-- Each entity has exactly one active worker at any time (enforced via heartbeat TTL)
-- Workers spawn on-demand when jobs arrive for an entity
-- Workers terminate after configurable idle period
-- Node failure → heartbeat expires → worker respawns on healthy node
+### The Solution
 
-### Dynamic Worker Lifecycle
+atomic-queues routes operations through per-entity queues. Same entity → same queue → sequential execution. Different entities → parallel queues → full throughput.
 
 ```
-                    Job Arrives for Entity X
-                              │
-                              ▼
-                    ┌─────────────────────┐
-                    │ Worker exists for X? │
-                    └──────────┬──────────┘
-                               │
-              ┌────────────────┴────────────────┐
-              │ NO                              │ YES
-              ▼                                 ▼
-    ┌─────────────────────┐           ┌─────────────────────┐
-    │ Spawn worker for X  │           │ Job added to queue  │
-    │ Register heartbeat  │           │ Worker will process │
-    └─────────────────────┘           └─────────────────────┘
-              │
-              ▼
-    ┌─────────────────────┐
-    │ Process jobs until  │◄─────── Idle Timeout
-    │ queue empty + idle  │         (configurable)
-    └──────────┬──────────┘
-               │
-               ▼
-    ┌─────────────────────┐
-    │ Worker terminates   │
-    │ Heartbeat expires   │
-    └─────────────────────┘
+                        ┌─────────────────────────────────────────────────┐
+  Request A ─┐          │           Entity: account-42                    │
+             │          │  ┌──────┐  ┌──────┐  ┌──────┐                  │
+  Request B ─┼─► Route ─┼─►│ Op 1 │─►│ Op 2 │─►│ Op 3 │─► [Worker] ──┐  │
+             │          │  └──────┘  └──────┘  └──────┘               │  │
+  Request C ─┘          │                    Sequential ◄─────────────┘  │
+                        └─────────────────────────────────────────────────┘
+
+                        ┌─────────────────────────────────────────────────┐
+  Request D ─┐          │           Entity: account-99                    │
+             │          │  ┌──────┐  ┌──────┐                            │
+  Request E ─┼─► Route ─┼─►│ Op 1 │─►│ Op 2 │─────────► [Worker] ──┐    │
+             │          │  └──────┘  └──────┘                       │    │
+  Request F ─┘          │                    Sequential ◄───────────┘    │
+                        └─────────────────────────────────────────────────┘
+
+                        ▲ These two queues run in PARALLEL across pods ▲
 ```
 
-**Resource Efficiency**: A system with 1 million registered accounts but 10,000 active accounts maintains only 10,000 workers.
+**Key properties:**
+- **One worker per entity** — enforced via Redis heartbeat TTL. No duplicates, ever.
+- **Auto-spawn** — workers materialize when jobs arrive, on the pod that sees them first.
+- **Auto-terminate** — idle workers shut down after a configurable timeout.
+- **Self-healing** — node failure → heartbeat expires → worker respawns on a healthy pod.
+- **Distributed** — workers spread across all pods via atomic `SET NX` claim. No leader election, no single point of failure.
 
 ---
 
-## Use Cases
+## Installation
 
-### Recommended Applications
+```bash
+# npm
+npm install atomic-queues bullmq ioredis
 
-| Domain            | Entity Type     | Operations                                          |
-|-------------------|-----------------|-----------------------------------------------------|
-| **Finance**       | Account, Wallet | Deposits, withdrawals, transfers, balance queries   |
-| **Gaming**        | Game, Match     | Player actions, state transitions, bet processing   |
-| **E-Commerce**    | Order, Cart     | Add/remove items, apply discounts, checkout         |
-| **Collaboration** | Document        | Edits, comments, permission changes                 |
-| **IoT**           | Device          | Command dispatch, state synchronization             |
+# pnpm
+pnpm add atomic-queues bullmq ioredis
 
-### When to Use Alternative Approaches
+# yarn
+yarn add atomic-queues bullmq ioredis
+```
 
-- **Read-heavy workloads**: Use caching layers (Redis, Memcached) or read replicas
-- **Parallelizable operations**: Use standard job queues (BullMQ, SQS) without entity affinity
-- **Fire-and-forget notifications**: Use pub/sub (Redis Pub/Sub, Kafka) without ordering guarantees
-- **Short critical sections (<10ms)**: Distributed locks may suffice if contention is low
+**Peer dependencies:** NestJS 10+, `@nestjs/cqrs` (optional — for auto-routing commands/queries)
 
 ---
 
-## Installation
-
-```bash
-npm install atomic-queues bullmq ioredis
-```
-
 ## Quick Start
 
 ### 1. Configure the Module
 
 ```typescript
 import { Module } from '@nestjs/common';
+import { CqrsModule } from '@nestjs/cqrs';
 import { AtomicQueuesModule } from 'atomic-queues';
 
 @Module({
   imports: [
+    CqrsModule,
     AtomicQueuesModule.forRoot({
       redis: { host: 'localhost', port: 6379 },
       keyPrefix: 'myapp',
+      entities: {
+        account: {
+          queueName: (id) => `account-${id}-queue`,
+          workerName: (id) => `account-${id}-worker`,
+          maxWorkersPerEntity: 1,
+          idleTimeoutSeconds: 15,
+        },
+      },
     }),
   ],
 })
 export class AppModule {}
 ```
 
-### 2. Create Your Commands
+> **Tip:** The `entities` config is optional. Without it, default naming applies: `{keyPrefix}:{entityType}:{entityId}:queue`.
 
-Plain classes - no decorators needed:
+### 2. Define Commands
 
 ```typescript
-// commands/process-order.command.ts
-export class ProcessOrderCommand {
+import { QueueEntity, QueueEntityId } from 'atomic-queues';
+
+@QueueEntity('account')
+export class WithdrawCommand {
   constructor(
-    public readonly orderId: string,
-    public readonly items: string[],
+    @QueueEntityId() public readonly accountId: string,
     public readonly amount: number,
   ) {}
 }
 
-// commands/ship-order.command.ts
-export class ShipOrderCommand {
+@QueueEntity('account')
+export class DepositCommand {
   constructor(
-    public readonly orderId: string,
-    public readonly address: string,
+    @QueueEntityId() public readonly accountId: string,
+    public readonly amount: number,
   ) {}
 }
 ```
 
-### 3. Create a Worker Processor
+### 3. Write Handlers (standard @nestjs/cqrs)
 
 ```typescript
-import { Injectable } from '@nestjs/common';
-import { WorkerProcessor } from 'atomic-queues';
+import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
 
-@WorkerProcessor({
-  entityType: 'order',
-  queueName: (orderId) => `order-${orderId}-queue`,
-  workerName: (orderId) => `order-${orderId}-worker`,
-})
-@Injectable()
-export class OrderProcessor {}
-```
+@CommandHandler(WithdrawCommand)
+export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
+  constructor(private readonly repo: AccountRepository) {}
 
-### 4. Queue Jobs with the Fluent API
+  async execute({ accountId, amount }: WithdrawCommand) {
+    // SAFE: No race conditions. Sequential execution per account.
+    const account = await this.repo.findById(accountId);
 
-Commands are **automatically registered** from your `@CommandHandler` classes - no manual registration needed!
+    if (account.balance < amount) {
+      throw new InsufficientFundsError(accountId, account.balance, amount);
+    }
+
+    account.balance -= amount;
+    await this.repo.save(account);
+  }
+}
+```
+
+### 4. Enqueue Jobs
 
 ```typescript
 import { Injectable } from '@nestjs/common';
 import { QueueBus } from 'atomic-queues';
-import { OrderProcessor } from './order.processor';
-import { ProcessOrderCommand, ShipOrderCommand } from './commands';
 
 @Injectable()
-export class OrderService {
+export class AccountService {
   constructor(private readonly queueBus: QueueBus) {}
 
-  async createOrder(orderId: string, items: string[], amount: number) {
-    // Jobs are queued and processed sequentially per orderId
-    await this.queueBus
-      .forProcessor(OrderProcessor)
-      .enqueue(new ProcessOrderCommand(orderId, items, amount));
-
-    await this.queueBus
-      .forProcessor(OrderProcessor)
-      .enqueue(new ShipOrderCommand(orderId, '123 Main St'));
+  async withdraw(accountId: string, amount: number) {
+    await this.queueBus.enqueue(new WithdrawCommand(accountId, amount));
   }
 }
 ```
 
-That's it! The library automatically:
-- Discovers commands from `@CommandHandler` decorators
-- Creates a queue for each `orderId`
-- Spawns a worker to process jobs sequentially
-- Routes jobs to the correct command handlers
+**That's it.** The library automatically:
+1. Creates a queue for each `accountId` when jobs arrive
+2. Spawns a worker (spread across pods) to process jobs sequentially
+3. Routes jobs to the correct `@CommandHandler` via CQRS
+4. Terminates idle workers after the configured timeout
+5. Self-heals if a pod dies (heartbeat expires → respawn elsewhere)
 
 ---
 
-## How It Works
+## Commands & Decorators
 
-```
-╔═══════════════════════════════════════════════════════════════════════════════╗
-║                              ARCHITECTURE                                      ║
-╚═══════════════════════════════════════════════════════════════════════════════╝
-
-┌──────────────────┐
-│   API Request    │   POST /accounts/ACC-123/withdraw { amount: 80 }
-└────────┬─────────┘
-         │
-         ▼
-┌──────────────────────────────────────────────────────────────────────────────┐
-│  QueueBus.forProcessor(AccountProcessor).enqueue(new WithdrawCommand(...))   │
-└────────┬─────────────────────────────────────────────────────────────────────┘
-         │
-         │  ① Reads @WorkerProcessor metadata from AccountProcessor
-         │  ② Extracts accountId from command.accountId property
-         │  ③ Generates queue name: "account-ACC-123-queue"
-         │
-         ▼
-┌──────────────────────────────────────────────────────────────────────────────┐
-│                              REDIS                                            │
-│  ┌────────────────────────────────────────────────────────────────────────┐  │
-│  │  Queue: account-ACC-123-queue                                          │  │
-│  │  ┌─────────────────┬─────────────────┬─────────────────┐               │  │
-│  │  │ Job 1           │ Job 2           │ Job 3           │  ...          │  │
-│  │  │ WithdrawCommand │ DepositCommand  │ TransferCommand │               │  │
-│  │  │ amount: 80      │ amount: 50      │ amount: 25      │               │  │
-│  │  └─────────────────┴─────────────────┴─────────────────┘               │  │
-│  └────────────────────────────────────────────────────────────────────────┘  │
-│                                                                               │
-│  ┌────────────────────────────────────────────────────────────────────────┐  │
-│  │  Queue: account-ACC-456-queue  (different account = different queue)   │  │
-│  │  ┌─────────────────┐                                                   │  │
-│  │  │ Job 1           │  ← Processes in parallel with ACC-123             │  │
-│  │  │ WithdrawCommand │                                                   │  │
-│  │  └─────────────────┘                                                   │  │
-│  └────────────────────────────────────────────────────────────────────────┘  │
-└──────────────────────────────────────────────────────────────────────────────┘
-         │
-         │  ④ BullMQ Worker pulls Job 1 (only one job at a time per queue)
-         │
-         ▼
-┌──────────────────────────────────────────────────────────────────────────────┐
-│  Worker: account-ACC-123-worker                                              │
-│                                                                               │
-│  ┌─────────────────────────────────────────────────────────────────────────┐ │
-│  │  ⑤ Lookup "WithdrawCommand" in QueueBus.globalRegistry                  │ │
-│  │  ⑥ Instantiate: Object.assign(new WithdrawCommand(), job.data)          │ │
-│  │  ⑦ Execute: CommandBus.execute(withdrawCommand)                         │ │
-│  └─────────────────────────────────────────────────────────────────────────┘ │
-└────────┬─────────────────────────────────────────────────────────────────────┘
-         │
-         ▼
-┌──────────────────────────────────────────────────────────────────────────────┐
-│  @CommandHandler(WithdrawCommand)                                            │
-│  class WithdrawHandler {                                                     │
-│    async execute(cmd: WithdrawCommand) {                                     │
-│      // Safe! No race conditions - guaranteed sequential execution           │
-│      const balance = await this.repo.getBalance(cmd.accountId);              │
-│      if (balance < cmd.amount) throw new InsufficientFundsError();           │
-│      await this.repo.debit(cmd.accountId, cmd.amount);                       │
-│    }                                                                         │
-│  }                                                                           │
-└──────────────────────────────────────────────────────────────────────────────┘
-```
+### `@QueueEntity(entityType)`
 
----
+Marks a command/query class for queue routing.
 
-## API Reference
+```typescript
+@QueueEntity('account')
+export class TransferCommand { ... }
+```
 
-### QueueBus
+### `@QueueEntityId()`
 
-The main way to add jobs to queues:
+Marks the property that contains the entity ID. One per class.
 
 ```typescript
-// Enqueue a single command
-await queueBus
-  .forProcessor(MyProcessor)
-  .enqueue(new MyCommand(entityId, data));
-
-// Enqueue and wait for result
-const result = await queueBus
-  .forProcessor(MyProcessor)
-  .enqueueAndWait(new MyQuery(entityId));
-
-// Enqueue multiple commands
-await queueBus
-  .forProcessor(MyProcessor)
-  .enqueueBulk([
-    new CommandA(entityId),
-    new CommandB(entityId),
-  ]);
-
-// With job options (delay, priority, etc.)
-await queueBus
-  .forProcessor(MyProcessor)
-  .enqueue(new MyCommand(entityId), {
-    jobOptions: { delay: 5000, priority: 1 }
-  });
+@QueueEntity('account')
+export class TransferCommand {
+  constructor(
+    @QueueEntityId() public readonly accountId: string,  // Routes to this account's queue
+    public readonly targetAccountId: string,
+    public readonly amount: number,
+  ) {}
+}
 ```
 
-### @WorkerProcessor
+### `@WorkerProcessor(options)`
 
-Defines how workers are created for an entity type:
+Optional. Define a processor class for custom job handling on top of CQRS auto-routing.
 
 ```typescript
 @WorkerProcessor({
-  entityType: 'order',                              // Required
-  queueName: (id) => `order-${id}-queue`,           // Optional
-  workerName: (id) => `order-${id}-worker`,         // Optional
-  workerConfig: {
-    concurrency: 1,                                 // Jobs per worker (default: 1)
-    stalledInterval: 1000,                          // Check stalled jobs (ms)
-    lockDuration: 30000,                            // Job lock duration (ms)
-  },
+  entityType: 'account',
+  queueName: (id) => `account-${id}-queue`,
+  workerName: (id) => `account-${id}-worker`,
+  maxWorkersPerEntity: 1,
+  idleTimeoutSeconds: 15,
 })
+@Injectable()
+export class AccountProcessor {
+  @JobHandler('special-audit')
+  async handleAudit(job: Job, entityId: string) { ... }
+}
 ```
 
----
+### `@JobHandler(jobName)` / `@JobHandler('*')`
 
-## Entity ID Extraction
+Custom job handlers on a `@WorkerProcessor`. The wildcard `'*'` catches anything not matched by a specific handler.
 
-The `entityId` is automatically extracted from your command's properties:
+---
+
+## Configuration
 
 ```typescript
-// These property names are checked in order:
-// entityId, tableId, userId, id, gameId, playerId
+AtomicQueuesModule.forRoot({
+  // ── Redis connection ──────────────────────────────────────
+  redis: {
+    host: 'redis',
+    port: 6379,
+    password: 'secret',       // optional
+  },
 
-export class ProcessOrderCommand {
-  constructor(
-    public readonly orderId: string,  // ✓ 'orderId' contains 'Id' → entityId
-    public readonly items: string[],
-  ) {}
-}
+  // ── Global settings ───────────────────────────────────────
+  keyPrefix: 'myapp',          // Redis key namespace (default: 'aq')
+  enableCronManager: true,     // Legacy cron-based scaling (optional)
+  cronInterval: 5000,          // Cron tick interval in ms
 
-// Or use standard names
-export class UpdateUserCommand {
-  constructor(
-    public readonly userId: string,   // ✓ Matches 'userId' → entityId
-    public readonly name: string,
-  ) {}
-}
+  // ── Worker defaults ───────────────────────────────────────
+  workerDefaults: {
+    concurrency: 1,            // Jobs processed concurrently per worker
+    stalledInterval: 1000,     // ms between stalled-job checks
+    lockDuration: 30000,       // ms a job is locked during processing
+    heartbeatTTL: 3,           // Heartbeat key TTL in seconds
+  },
+
+  // ── Per-entity configuration (optional) ───────────────────
+  entities: {
+    account: {
+      queueName: (id) => `account-${id}-queue`,
+      workerName: (id) => `account-${id}-worker`,
+      maxWorkersPerEntity: 1,
+      idleTimeoutSeconds: 15,
+      defaultEntityId: 'accountId',
+      workerConfig: {          // Override workerDefaults per entity
+        concurrency: 1,
+        lockDuration: 60000,
+      },
+    },
+  },
+});
 ```
 
 ---
 
-## Scaling with Entity Scalers
+## Distributed Worker Lifecycle
 
-For dynamic worker management based on demand:
+Workers in atomic-queues have a fully automated lifecycle, distributed across all pods with no leader election:
 
-```typescript
-import { Injectable } from '@nestjs/common';
-import { EntityScaler, GetActiveEntities, GetDesiredWorkerCount } from 'atomic-queues';
+```
+  Job arrives                   SET NX claim
+  on any pod  ──────►  ┌──────────────────────┐
+                        │  Pod claims worker?   │
+                        └──────┬───────┬───────┘
+                           YES │       │ NO (another pod won)
+                               ▼       ▼
+                        ┌────────┐  ┌──────────────┐
+                        │ Spawn  │  │ Wait — other │
+                        │ worker │  │ pod handles  │
+                        │ locally│  └──────────────┘
+                        └───┬────┘
+                            ▼
+                     ┌──────────────┐
+                     │  Processing  │◄──── Heartbeat refresh (pipeline)
+                     │  jobs back-  │      every 1s (1 Redis round-trip)
+                     │  to-back     │
+                     └──────┬───────┘
+                            │ No jobs for idleTimeoutSeconds
+                            ▼
+                     ┌──────────────┐
+                     │  Idle sweep  │──── Hot cache eviction
+                     │  closes      │     Heartbeat keys cleaned up
+                     │  worker      │
+                     └──────────────┘
+```
 
-@EntityScaler({
-  entityType: 'order',
-  maxWorkersPerEntity: 1,
-})
-@Injectable()
-export class OrderScaler {
-  constructor(private readonly orderRepo: OrderRepository) {}
+### Hot Cache (v1.5.0+)
 
-  @GetActiveEntities()
-  async getActiveOrders(): Promise<string[]> {
-    // Return IDs that need workers
-    return this.orderRepo.findPendingOrderIds();
-  }
+After a worker is confirmed alive, subsequent job arrivals for that entity hit an **in-memory cache** — zero Redis calls on the warm path. This eliminates the per-job Redis overhead that plagues lock-based approaches.
 
-  @GetDesiredWorkerCount()
-  async getWorkerCount(orderId: string): Promise<number> {
-    return 1; // One worker per order
-  }
-}
-```
+| Path | Redis calls | When |
+|---|---|---|
+| **Hot** (cache hit) | 0 | Worker known alive |
+| **Warm** (cache miss) | 1 (`EXISTS`) | First time seeing entity |
+| **Cold** (no worker) | 1 (`SET NX`) | Worker needs creation |
+
+### SpawnQueueService (v1.4.2+)
+
+For multi-pod deployments, the `SpawnQueueService` distributes worker creation across all pods via a shared BullMQ spawn queue. In v1.5.0, the **direct local spawn** path bypasses this queue entirely — the pod that first sees a job for a new entity claims it with an atomic `SET NX` and spawns the worker locally, saving hundreds of milliseconds.
 
 ---
 
 ## Complete Example
 
-A banking service handling critical financial transactions where race conditions could cause overdrafts or double-spending:
+A banking service with withdrawals, deposits, and cross-account transfers:
 
 ```typescript
-// ─────────────────────────────────────────────────────────────────
-// commands/withdraw.command.ts
-// ─────────────────────────────────────────────────────────────────
+// ── Module ──────────────────────────────────────────────
+import { Module } from '@nestjs/common';
+import { CqrsModule } from '@nestjs/cqrs';
+import { AtomicQueuesModule } from 'atomic-queues';
+
+@Module({
+  imports: [
+    CqrsModule,
+    AtomicQueuesModule.forRoot({
+      redis: { host: 'redis', port: 6379 },
+      keyPrefix: 'banking',
+      entities: {
+        account: {
+          queueName: (id) => `account-${id}-queue`,
+          workerName: (id) => `account-${id}-worker`,
+          maxWorkersPerEntity: 1,
+          idleTimeoutSeconds: 15,
+        },
+      },
+    }),
+  ],
+  providers: [
+    AccountService,
+    WithdrawHandler,
+    DepositHandler,
+    TransferHandler,
+  ],
+})
+export class BankingModule {}
+
+// ── Commands ────────────────────────────────────────────
+import { QueueEntity, QueueEntityId } from 'atomic-queues';
+
+@QueueEntity('account')
 export class WithdrawCommand {
   constructor(
-    public readonly accountId: string,
+    @QueueEntityId() public readonly accountId: string,
     public readonly amount: number,
     public readonly transactionId: string,
-    public readonly requestedBy: string,
   ) {}
 }
 
-// ─────────────────────────────────────────────────────────────────
-// commands/deposit.command.ts
-// ─────────────────────────────────────────────────────────────────
+@QueueEntity('account')
 export class DepositCommand {
   constructor(
-    public readonly accountId: string,
+    @QueueEntityId() public readonly accountId: string,
     public readonly amount: number,
-    public readonly transactionId: string,
     public readonly source: string,
   ) {}
 }
 
-// ─────────────────────────────────────────────────────────────────
-// commands/transfer.command.ts
-// ─────────────────────────────────────────────────────────────────
+@QueueEntity('account')
 export class TransferCommand {
   constructor(
-    public readonly accountId: string,  // Source account (for queue routing)
+    @QueueEntityId() public readonly accountId: string,
     public readonly toAccountId: string,
     public readonly amount: number,
-    public readonly transactionId: string,
   ) {}
 }
 
-// ─────────────────────────────────────────────────────────────────
-// handlers/withdraw.handler.ts
-// ─────────────────────────────────────────────────────────────────
+// ── Handlers ────────────────────────────────────────────
 import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
-import { WithdrawCommand } from '../commands';
 
 @CommandHandler(WithdrawCommand)
 export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
-  constructor(
-    private readonly accountRepo: AccountRepository,
-    private readonly ledger: LedgerService,
-  ) {}
+  constructor(private readonly repo: AccountRepository) {}
 
-  async execute(command: WithdrawCommand) {
-    const { accountId, amount, transactionId } = command;
-    
-    // SAFE: No race conditions! This handler runs sequentially per account
-    // Even if 10 withdrawals arrive simultaneously, they execute one-by-one
-    
-    const account = await this.accountRepo.findById(accountId);
-    
-    if (account.balance < amount) {
-      throw new InsufficientFundsError(accountId, account.balance, amount);
-    }
-    
-    if (account.status !== 'active') {
-      throw new AccountFrozenError(accountId);
-    }
-    
-    // Debit the account
+  async execute({ accountId, amount }: WithdrawCommand) {
+    const account = await this.repo.findById(accountId);
+    if (account.balance < amount) throw new InsufficientFundsError();
     account.balance -= amount;
-    await this.accountRepo.save(account);
-    
-    // Record in ledger
-    await this.ledger.record({
-      transactionId,
-      accountId,
-      type: 'DEBIT',
-      amount,
-      balanceAfter: account.balance,
-      timestamp: new Date(),
-    });
-    
-    return { 
-      success: true, 
-      transactionId, 
-      newBalance: account.balance 
-    };
+    await this.repo.save(account);
   }
 }
 
-// ─────────────────────────────────────────────────────────────────
-// handlers/transfer.handler.ts
-// ─────────────────────────────────────────────────────────────────
-import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
-import { TransferCommand, DepositCommand } from '../commands';
-import { QueueBus } from 'atomic-queues';
-
 @CommandHandler(TransferCommand)
 export class TransferHandler implements ICommandHandler<TransferCommand> {
   constructor(
-    private readonly accountRepo: AccountRepository,
+    private readonly repo: AccountRepository,
     private readonly queueBus: QueueBus,
   ) {}
 
-  async execute(command: TransferCommand) {
-    const { accountId, toAccountId, amount, transactionId } = command;
-    
-    // Step 1: Debit source account (already in source account's queue)
-    const sourceAccount = await this.accountRepo.findById(accountId);
-    
-    if (sourceAccount.balance < amount) {
-      throw new InsufficientFundsError(accountId, sourceAccount.balance, amount);
-    }
-    
-    sourceAccount.balance -= amount;
-    await this.accountRepo.save(sourceAccount);
-    
-    // Step 2: Queue credit to destination account (different queue!)
-    // This ensures the destination account also processes atomically
-    await this.queueBus
-      .forProcessor(AccountProcessor)
-      .enqueue(new DepositCommand(
-        toAccountId,
-        amount,
-        transactionId,
-        `transfer:${accountId}`,
-      ));
-    
-    return { success: true, transactionId };
-  }
-}
-
-// ─────────────────────────────────────────────────────────────────
-// account.processor.ts
-// ─────────────────────────────────────────────────────────────────
-import { Injectable } from '@nestjs/common';
-import { WorkerProcessor } from 'atomic-queues';
-
-@WorkerProcessor({
-  entityType: 'account',
-  queueName: (accountId) => `bank-account-${accountId}-queue`,
-  workerName: (accountId) => `bank-account-${accountId}-worker`,
-  workerConfig: {
-    concurrency: 1,        // CRITICAL: Must be 1 for financial transactions
-    lockDuration: 60000,   // 60s lock for long transactions
-    stalledInterval: 5000,
-  },
-})
-@Injectable()
-export class AccountProcessor {}
-
-// ─────────────────────────────────────────────────────────────────
-// account.scaler.ts - Scale workers based on active accounts
-// ─────────────────────────────────────────────────────────────────
-import { Injectable } from '@nestjs/common';
-import { EntityScaler, GetActiveEntities, GetDesiredWorkerCount } from 'atomic-queues';
-
-@EntityScaler({
-  entityType: 'account',
-  maxWorkersPerEntity: 1,  // Never more than 1 worker per account
-})
-@Injectable()
-export class AccountScaler {
-  constructor(private readonly accountRepo: AccountRepository) {}
-
-  @GetActiveEntities()
-  async getActiveAccounts(): Promise<string[]> {
-    // Return accounts with pending transactions
-    return this.accountRepo.findAccountsWithPendingTransactions();
-  }
-
-  @GetDesiredWorkerCount()
-  async getWorkerCount(accountId: string): Promise<number> {
-    // Always 1 worker per account for atomicity
-    return 1;
+  async execute({ accountId, toAccountId, amount }: TransferCommand) {
+    // Debit source (we're in source account's queue — safe)
+    const source = await this.repo.findById(accountId);
+    if (source.balance < amount) throw new InsufficientFundsError();
+    source.balance -= amount;
+    await this.repo.save(source);
+
+    // Credit destination (enqueued to destination's queue — also safe)
+    await this.queueBus.enqueue(
+      new DepositCommand(toAccountId, amount, `transfer:${accountId}`),
+    );
   }
 }
 
-// ─────────────────────────────────────────────────────────────────
-// account.module.ts
-// ─────────────────────────────────────────────────────────────────
-import { Module } from '@nestjs/common';
-import { CqrsModule } from '@nestjs/cqrs';
-
-@Module({
-  imports: [CqrsModule],
-  providers: [
-    AccountProcessor,
-    AccountScaler,
-    WithdrawHandler,    // Commands auto-discovered!
-    DepositHandler,
-    TransferHandler,
-  ],
-  controllers: [AccountController],
-})
-export class AccountModule {}
-
-// ─────────────────────────────────────────────────────────────────
-// account.controller.ts
-// ─────────────────────────────────────────────────────────────────
+// ── Controller ──────────────────────────────────────────
 import { Controller, Post, Body, Param } from '@nestjs/common';
 import { QueueBus } from 'atomic-queues';
-import { AccountProcessor } from './account.processor';
-import { WithdrawCommand, DepositCommand, TransferCommand } from './commands';
-import { v4 as uuid } from 'uuid';
 
 @Controller('accounts')
 export class AccountController {
   constructor(private readonly queueBus: QueueBus) {}
 
-  @Post(':accountId/withdraw')
-  async withdraw(
-    @Param('accountId') accountId: string,
-    @Body() body: { amount: number; requestedBy: string },
-  ) {
-    const transactionId = uuid();
-    
-    // Even if user spam-clicks "Withdraw", each request is queued
-    // and processed sequentially - no double-withdrawals possible
-    await this.queueBus
-      .forProcessor(AccountProcessor)
-      .enqueue(new WithdrawCommand(
-        accountId,
-        body.amount,
-        transactionId,
-        body.requestedBy,
-      ));
-
-    return { 
-      queued: true, 
-      transactionId,
-      message: 'Withdrawal queued for processing',
-    };
+  @Post(':id/withdraw')
+  async withdraw(@Param('id') id: string, @Body() body: { amount: number }) {
+    await this.queueBus.enqueue(new WithdrawCommand(id, body.amount, uuid()));
+    return { queued: true };
   }
 
-  @Post(':accountId/transfer')
+  @Post(':id/transfer')
   async transfer(
-    @Param('accountId') accountId: string,
-    @Body() body: { toAccountId: string; amount: number },
+    @Param('id') id: string,
+    @Body() body: { to: string; amount: number },
   ) {
-    const transactionId = uuid();
-    
-    await this.queueBus
-      .forProcessor(AccountProcessor)
-      .enqueue(new TransferCommand(
-        accountId,
-        body.toAccountId,
-        body.amount,
-        transactionId,
-      ));
-
-    return { 
-      queued: true, 
-      transactionId,
-      message: 'Transfer queued for processing',
-    };
+    await this.queueBus.enqueue(new TransferCommand(id, body.to, body.amount));
+    return { queued: true };
   }
 }
 ```
 
 ---
 
-## Configuration
-
-```typescript
-AtomicQueuesModule.forRoot({
-  redis: {
-    host: 'localhost',
-    port: 6379,
-    password: 'secret',
-  },
-  
-  keyPrefix: 'myapp',           // Redis key prefix (default: 'aq')
-  
-  enableCronManager: true,       // Enable auto-scaling (default: false)
-  cronInterval: 5000,            // Scaling check interval (default: 5000ms)
-  
-  verbose: false,                // Enable verbose logging (default: false)
-                                 // When true, logs service job processing details
-  
-  workerDefaults: {
-    concurrency: 1,              // Jobs processed simultaneously
-    stalledInterval: 1000,       // Stalled job check interval
-    lockDuration: 30000,         // Job lock duration
-    heartbeatTTL: 3,             // Worker heartbeat TTL (seconds)
-  },
-});
-```
-
----
-
-## Command Registration
+## Advanced: Custom Worker Processors
 
-By default, atomic-queues **auto-discovers** all commands from your `@CommandHandler` and `@QueryHandler` decorators. No manual registration needed!
+For cases where CQRS auto-routing isn't enough, define a `@WorkerProcessor` with explicit `@JobHandler` methods:
 
-### Auto-Discovery (Default)
+```typescript
+import { Injectable } from '@nestjs/common';
+import { WorkerProcessor, JobHandler } from 'atomic-queues';
+import { Job } from 'bullmq';
 
-Commands are automatically discovered when you have CQRS handlers:
+@WorkerProcessor({
+  entityType: 'account',
+  queueName: (id) => `account-${id}-queue`,
+  workerName: (id) => `account-${id}-worker`,
+  maxWorkersPerEntity: 1,
+  idleTimeoutSeconds: 15,
+})
+@Injectable()
+export class AccountProcessor {
+  @JobHandler('high-priority-audit')
+  async handleAudit(job: Job, entityId: string) {
+    // Specific handler for this job type
+  }
 
-```typescript
-// Your handler - that's all you need!
-@CommandHandler(ProcessOrderCommand)
-export class ProcessOrderHandler implements ICommandHandler<ProcessOrderCommand> {
-  async execute(command: ProcessOrderCommand) {
-    // ProcessOrderCommand is auto-registered with QueueBus
+  @JobHandler('*')
+  async handleAll(job: Job, entityId: string) {
+    // Wildcard — catches everything not explicitly handled
+    // Falls back to CQRS routing automatically when not defined
   }
 }
 ```
 
-### Manual Registration (Optional)
+> **Priority order:** Explicit `@JobHandler` → CQRS auto-routing (`@JobCommand`/`@JobQuery`) → Wildcard handler
 
-If you need to register commands without handlers, or disable auto-discovery:
+---
 
-```typescript
-// Disable auto-discovery in config
-AtomicQueuesModule.forRoot({
-  redis: { host: 'localhost', port: 6379 },
-  autoRegisterCommands: false, // Disable auto-discovery
-});
+## Performance
 
-// Then manually register
-QueueBus.registerCommands(ProcessOrderCommand, ShipOrderCommand);
-```
+### Throughput (measured — not estimated)
 
----
+Tested on a 5-pod Kubernetes cluster (OrbStack), 20 concurrent entities, 12,300 orders:
+
+| Metric | Result |
+|---|---|
+| **Phase 1** — 10,000 orders (50 waves × 200 concurrent) | 167 orders/sec |
+| **Phase 2** — 1,000 orders (workers still draining) | 140 orders/sec |
+| **Phase 4** — 1,000 orders (cold start from zero workers) | 176 orders/sec |
+| **Total deductions processed** | 104,004 |
+| **Stock drift** | **0** (all 20 entities) |
+| **Pod distribution** | 5/5 pods actively creating workers |
+| **Worker creates** | 120 |
+| **Idle closures** | 180 |
+
+### Why it's fast
+
+1. **Zero contention** — no locks, no retries, no backoff. Jobs queue and execute.
+2. **Hot cache** — after first check, subsequent job arrivals for an entity incur 0 Redis calls.
+3. **Direct local spawn** — atomic `SET NX` claim, local worker creation. No queue round-trip.
+4. **Pipelined heartbeats** — heartbeat refresh uses a single Redis pipeline (1 round-trip for 2 keys).
+5. **O(1) worker existence check** — global alive key replaces `KEYS` pattern scan.
 
-## Why Use atomic-queues?
+### When to use what
 
-| Feature | Without | With atomic-queues |
-|---------|---------|-------------------|
-| Sequential per-entity | Manual locking | Automatic via queues |
-| Race conditions | Possible | Prevented |
-| Worker management | Manual | Automatic |
-| Horizontal scaling | Complex | Built-in |
-| Code organization | Scattered | Clean decorators |
+| Use case | Recommendation |
+|---|---|
+| High-throughput entity operations (payments, inventory, game state) | **atomic-queues** |
+| Rare, low-frequency mutual exclusion (config updates, migrations) | Redlock / advisory locks |
+| Exactly-once semantics with audit trail | **atomic-queues** (BullMQ job IDs) |
+| Sub-millisecond synchronous response required | Redlock (synchronous acquire) |
+| Multi-pod, many entities, sustained load | **atomic-queues** (contention-free scaling) |
 
 ---