atomic-queues 1.6.1 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +182 -411
- package/dist/cli/generators/json-schema.d.ts +3 -0
- package/dist/cli/generators/json-schema.d.ts.map +1 -0
- package/dist/cli/generators/json-schema.js +31 -0
- package/dist/cli/generators/json-schema.js.map +1 -0
- package/dist/cli/generators/typescript.d.ts +3 -0
- package/dist/cli/generators/typescript.d.ts.map +1 -0
- package/dist/cli/generators/typescript.js +62 -0
- package/dist/cli/generators/typescript.js.map +1 -0
- package/dist/cli/index.d.ts +3 -0
- package/dist/cli/index.d.ts.map +1 -0
- package/dist/cli/index.js +156 -0
- package/dist/cli/index.js.map +1 -0
- package/dist/decorators/actor.decorators.d.ts +4 -0
- package/dist/decorators/actor.decorators.d.ts.map +1 -0
- package/dist/decorators/actor.decorators.js +32 -0
- package/dist/decorators/actor.decorators.js.map +1 -0
- package/dist/decorators/constants.d.ts +5 -12
- package/dist/decorators/constants.d.ts.map +1 -1
- package/dist/decorators/constants.js +10 -16
- package/dist/decorators/constants.js.map +1 -1
- package/dist/decorators/index.d.ts +4 -4
- package/dist/decorators/index.d.ts.map +1 -1
- package/dist/decorators/index.js +4 -4
- package/dist/decorators/index.js.map +1 -1
- package/dist/decorators/interfaces.d.ts +18 -78
- package/dist/decorators/interfaces.d.ts.map +1 -1
- package/dist/decorators/metadata-readers.d.ts +5 -26
- package/dist/decorators/metadata-readers.d.ts.map +1 -1
- package/dist/decorators/metadata-readers.js +16 -33
- package/dist/decorators/metadata-readers.js.map +1 -1
- package/dist/decorators/schema.decorators.d.ts +2 -0
- package/dist/decorators/schema.decorators.d.ts.map +1 -0
- package/dist/decorators/schema.decorators.js +13 -0
- package/dist/decorators/schema.decorators.js.map +1 -0
- package/dist/domain/interfaces/config.interfaces.d.ts +52 -153
- package/dist/domain/interfaces/config.interfaces.d.ts.map +1 -1
- package/dist/domain/interfaces/index.d.ts +0 -7
- package/dist/domain/interfaces/index.d.ts.map +1 -1
- package/dist/domain/interfaces/index.js +0 -7
- package/dist/domain/interfaces/index.js.map +1 -1
- package/dist/domain/interfaces/job.interfaces.d.ts +32 -65
- package/dist/domain/interfaces/job.interfaces.d.ts.map +1 -1
- package/dist/index.d.ts +0 -34
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +0 -39
- package/dist/index.js.map +1 -1
- package/dist/module/atomic-queues.module.d.ts +0 -83
- package/dist/module/atomic-queues.module.d.ts.map +1 -1
- package/dist/module/atomic-queues.module.js +35 -134
- package/dist/module/atomic-queues.module.js.map +1 -1
- package/dist/services/actor-registry/actor-registry.service.d.ts +30 -0
- package/dist/services/actor-registry/actor-registry.service.d.ts.map +1 -0
- package/dist/services/actor-registry/actor-registry.service.js +186 -0
- package/dist/services/actor-registry/actor-registry.service.js.map +1 -0
- package/dist/services/actor-registry/index.d.ts +2 -0
- package/dist/services/actor-registry/index.d.ts.map +1 -0
- package/dist/services/actor-registry/index.js +18 -0
- package/dist/services/actor-registry/index.js.map +1 -0
- package/dist/services/actor-system/actor-system.service.d.ts +19 -0
- package/dist/services/actor-system/actor-system.service.d.ts.map +1 -0
- package/dist/services/actor-system/actor-system.service.js +86 -0
- package/dist/services/actor-system/actor-system.service.js.map +1 -0
- package/dist/services/actor-system/index.d.ts +2 -0
- package/dist/services/actor-system/index.d.ts.map +1 -0
- package/dist/services/{cron-manager → actor-system}/index.js +1 -1
- package/dist/services/actor-system/index.js.map +1 -0
- package/dist/services/command-discovery/command-discovery.service.d.ts +6 -53
- package/dist/services/command-discovery/command-discovery.service.d.ts.map +1 -1
- package/dist/services/command-discovery/command-discovery.service.js +0 -59
- package/dist/services/command-discovery/command-discovery.service.js.map +1 -1
- package/dist/services/constants.d.ts +2 -9
- package/dist/services/constants.d.ts.map +1 -1
- package/dist/services/constants.js +3 -10
- package/dist/services/constants.js.map +1 -1
- package/dist/services/executor-pool/executor-pool.service.d.ts +31 -0
- package/dist/services/executor-pool/executor-pool.service.d.ts.map +1 -0
- package/dist/services/executor-pool/executor-pool.service.js +147 -0
- package/dist/services/executor-pool/executor-pool.service.js.map +1 -0
- package/dist/services/executor-pool/index.d.ts +2 -0
- package/dist/services/executor-pool/index.d.ts.map +1 -0
- package/dist/services/{queue-manager → executor-pool}/index.js +1 -1
- package/dist/services/executor-pool/index.js.map +1 -0
- package/dist/services/gate/gate.service.d.ts +17 -0
- package/dist/services/gate/gate.service.d.ts.map +1 -0
- package/dist/services/gate/gate.service.js +66 -0
- package/dist/services/gate/gate.service.js.map +1 -0
- package/dist/services/gate/index.d.ts +2 -0
- package/dist/services/gate/index.d.ts.map +1 -0
- package/dist/services/{spawn-queue → gate}/index.js +1 -1
- package/dist/services/gate/index.js.map +1 -0
- package/dist/services/handler-executor/handler-executor.service.d.ts +32 -0
- package/dist/services/handler-executor/handler-executor.service.d.ts.map +1 -0
- package/dist/services/handler-executor/handler-executor.service.js +186 -0
- package/dist/services/handler-executor/handler-executor.service.js.map +1 -0
- package/dist/services/handler-executor/index.d.ts +2 -0
- package/dist/services/handler-executor/index.d.ts.map +1 -0
- package/dist/services/handler-executor/index.js +18 -0
- package/dist/services/handler-executor/index.js.map +1 -0
- package/dist/services/index.d.ts +11 -12
- package/dist/services/index.d.ts.map +1 -1
- package/dist/services/index.js +11 -12
- package/dist/services/index.js.map +1 -1
- package/dist/services/log/index.d.ts +2 -0
- package/dist/services/log/index.d.ts.map +1 -0
- package/dist/services/{index-manager → log}/index.js +1 -1
- package/dist/services/log/index.js.map +1 -0
- package/dist/services/log/log.service.d.ts +21 -0
- package/dist/services/log/log.service.d.ts.map +1 -0
- package/dist/services/log/log.service.js +92 -0
- package/dist/services/log/log.service.js.map +1 -0
- package/dist/services/queue-bus/index.d.ts +0 -4
- package/dist/services/queue-bus/index.d.ts.map +1 -1
- package/dist/services/queue-bus/index.js +0 -4
- package/dist/services/queue-bus/index.js.map +1 -1
- package/dist/services/queue-bus/queue-bus.service.d.ts +44 -198
- package/dist/services/queue-bus/queue-bus.service.d.ts.map +1 -1
- package/dist/services/queue-bus/queue-bus.service.js +103 -259
- package/dist/services/queue-bus/queue-bus.service.js.map +1 -1
- package/dist/services/queue-bus/queue-bus.utils.d.ts +0 -28
- package/dist/services/queue-bus/queue-bus.utils.d.ts.map +1 -1
- package/dist/services/queue-bus/queue-bus.utils.js +1 -41
- package/dist/services/queue-bus/queue-bus.utils.js.map +1 -1
- package/dist/services/registry/index.d.ts +4 -0
- package/dist/services/registry/index.d.ts.map +1 -0
- package/dist/services/{queue-events-manager → registry}/index.js +3 -1
- package/dist/services/registry/index.js.map +1 -0
- package/dist/services/registry/registry.service.d.ts +43 -0
- package/dist/services/registry/registry.service.d.ts.map +1 -0
- package/dist/services/registry/registry.service.js +379 -0
- package/dist/services/registry/registry.service.js.map +1 -0
- package/dist/services/registry/registry.types.d.ts +24 -0
- package/dist/services/registry/registry.types.d.ts.map +1 -0
- package/dist/{domain/interfaces/lock.interfaces.js → services/registry/registry.types.js} +1 -1
- package/dist/services/registry/registry.types.js.map +1 -0
- package/dist/services/registry/schema-converter.d.ts +2 -0
- package/dist/services/registry/schema-converter.d.ts.map +1 -0
- package/dist/services/registry/schema-converter.js +27 -0
- package/dist/services/registry/schema-converter.js.map +1 -0
- package/dist/services/result-collector/index.d.ts +2 -0
- package/dist/services/result-collector/index.d.ts.map +1 -0
- package/dist/services/result-collector/index.js +18 -0
- package/dist/services/result-collector/index.js.map +1 -0
- package/dist/services/result-collector/result-collector.service.d.ts +17 -0
- package/dist/services/result-collector/result-collector.service.d.ts.map +1 -0
- package/dist/services/result-collector/result-collector.service.js +92 -0
- package/dist/services/result-collector/result-collector.service.js.map +1 -0
- package/dist/services/scheduler/index.d.ts +2 -0
- package/dist/services/scheduler/index.d.ts.map +1 -0
- package/dist/services/{job-processor → scheduler}/index.js +1 -1
- package/dist/services/scheduler/index.js.map +1 -0
- package/dist/services/scheduler/scheduler.service.d.ts +17 -0
- package/dist/services/scheduler/scheduler.service.d.ts.map +1 -0
- package/dist/services/scheduler/scheduler.service.js +116 -0
- package/dist/services/scheduler/scheduler.service.js.map +1 -0
- package/dist/services/shutdown/index.d.ts +2 -0
- package/dist/services/shutdown/index.d.ts.map +1 -0
- package/dist/services/shutdown/index.js +18 -0
- package/dist/services/shutdown/index.js.map +1 -0
- package/dist/services/shutdown/shutdown.service.d.ts +8 -0
- package/dist/services/shutdown/shutdown.service.d.ts.map +1 -0
- package/dist/services/shutdown/shutdown.service.js +29 -0
- package/dist/services/shutdown/shutdown.service.js.map +1 -0
- package/dist/utils/index.d.ts +3 -1
- package/dist/utils/index.d.ts.map +1 -1
- package/dist/utils/index.js +3 -1
- package/dist/utils/index.js.map +1 -1
- package/dist/utils/naming.utils.d.ts +0 -16
- package/dist/utils/naming.utils.d.ts.map +1 -1
- package/dist/utils/naming.utils.js +0 -29
- package/dist/utils/naming.utils.js.map +1 -1
- package/package.json +19 -11
- package/dist/decorators/legacy.decorators.d.ts +0 -36
- package/dist/decorators/legacy.decorators.d.ts.map +0 -1
- package/dist/decorators/legacy.decorators.js +0 -61
- package/dist/decorators/legacy.decorators.js.map +0 -1
- package/dist/decorators/scaler.decorators.d.ts +0 -65
- package/dist/decorators/scaler.decorators.d.ts.map +0 -1
- package/dist/decorators/scaler.decorators.js +0 -103
- package/dist/decorators/scaler.decorators.js.map +0 -1
- package/dist/decorators/type-guards.d.ts +0 -18
- package/dist/decorators/type-guards.d.ts.map +0 -1
- package/dist/decorators/type-guards.js +0 -32
- package/dist/decorators/type-guards.js.map +0 -1
- package/dist/decorators/worker.decorators.d.ts +0 -58
- package/dist/decorators/worker.decorators.d.ts.map +0 -1
- package/dist/decorators/worker.decorators.js +0 -92
- package/dist/decorators/worker.decorators.js.map +0 -1
- package/dist/domain/interfaces/event.interfaces.d.ts +0 -71
- package/dist/domain/interfaces/event.interfaces.d.ts.map +0 -1
- package/dist/domain/interfaces/event.interfaces.js +0 -3
- package/dist/domain/interfaces/event.interfaces.js.map +0 -1
- package/dist/domain/interfaces/index-tracking.interfaces.d.ts +0 -69
- package/dist/domain/interfaces/index-tracking.interfaces.d.ts.map +0 -1
- package/dist/domain/interfaces/index-tracking.interfaces.js +0 -3
- package/dist/domain/interfaces/index-tracking.interfaces.js.map +0 -1
- package/dist/domain/interfaces/lock.interfaces.d.ts +0 -54
- package/dist/domain/interfaces/lock.interfaces.d.ts.map +0 -1
- package/dist/domain/interfaces/lock.interfaces.js.map +0 -1
- package/dist/domain/interfaces/process.interfaces.d.ts +0 -44
- package/dist/domain/interfaces/process.interfaces.d.ts.map +0 -1
- package/dist/domain/interfaces/process.interfaces.js +0 -3
- package/dist/domain/interfaces/process.interfaces.js.map +0 -1
- package/dist/domain/interfaces/queue.interfaces.d.ts +0 -46
- package/dist/domain/interfaces/queue.interfaces.d.ts.map +0 -1
- package/dist/domain/interfaces/queue.interfaces.js +0 -3
- package/dist/domain/interfaces/queue.interfaces.js.map +0 -1
- package/dist/domain/interfaces/scaling.interfaces.d.ts +0 -62
- package/dist/domain/interfaces/scaling.interfaces.d.ts.map +0 -1
- package/dist/domain/interfaces/scaling.interfaces.js +0 -3
- package/dist/domain/interfaces/scaling.interfaces.js.map +0 -1
- package/dist/domain/interfaces/worker.interfaces.d.ts +0 -120
- package/dist/domain/interfaces/worker.interfaces.d.ts.map +0 -1
- package/dist/domain/interfaces/worker.interfaces.js +0 -3
- package/dist/domain/interfaces/worker.interfaces.js.map +0 -1
- package/dist/services/cron-manager/cron-manager.service.d.ts +0 -199
- package/dist/services/cron-manager/cron-manager.service.d.ts.map +0 -1
- package/dist/services/cron-manager/cron-manager.service.js +0 -583
- package/dist/services/cron-manager/cron-manager.service.js.map +0 -1
- package/dist/services/cron-manager/index.d.ts +0 -2
- package/dist/services/cron-manager/index.d.ts.map +0 -1
- package/dist/services/cron-manager/index.js.map +0 -1
- package/dist/services/index-manager/index-manager.service.d.ts +0 -142
- package/dist/services/index-manager/index-manager.service.d.ts.map +0 -1
- package/dist/services/index-manager/index-manager.service.js +0 -325
- package/dist/services/index-manager/index-manager.service.js.map +0 -1
- package/dist/services/index-manager/index.d.ts +0 -2
- package/dist/services/index-manager/index.d.ts.map +0 -1
- package/dist/services/index-manager/index.js.map +0 -1
- package/dist/services/job-processor/index.d.ts +0 -2
- package/dist/services/job-processor/index.d.ts.map +0 -1
- package/dist/services/job-processor/index.js.map +0 -1
- package/dist/services/job-processor/job-processor.service.d.ts +0 -156
- package/dist/services/job-processor/job-processor.service.d.ts.map +0 -1
- package/dist/services/job-processor/job-processor.service.js +0 -331
- package/dist/services/job-processor/job-processor.service.js.map +0 -1
- package/dist/services/processor-discovery/decorator-discovery.service.d.ts +0 -40
- package/dist/services/processor-discovery/decorator-discovery.service.d.ts.map +0 -1
- package/dist/services/processor-discovery/decorator-discovery.service.js +0 -191
- package/dist/services/processor-discovery/decorator-discovery.service.js.map +0 -1
- package/dist/services/processor-discovery/index.d.ts +0 -6
- package/dist/services/processor-discovery/index.d.ts.map +0 -1
- package/dist/services/processor-discovery/index.js +0 -22
- package/dist/services/processor-discovery/index.js.map +0 -1
- package/dist/services/processor-discovery/processor-discovery.service.d.ts +0 -98
- package/dist/services/processor-discovery/processor-discovery.service.d.ts.map +0 -1
- package/dist/services/processor-discovery/processor-discovery.service.js +0 -258
- package/dist/services/processor-discovery/processor-discovery.service.js.map +0 -1
- package/dist/services/processor-discovery/processor-registry.d.ts +0 -58
- package/dist/services/processor-discovery/processor-registry.d.ts.map +0 -1
- package/dist/services/processor-discovery/processor-registry.js +0 -74
- package/dist/services/processor-discovery/processor-registry.js.map +0 -1
- package/dist/services/processor-discovery/scaling-registration.service.d.ts +0 -60
- package/dist/services/processor-discovery/scaling-registration.service.d.ts.map +0 -1
- package/dist/services/processor-discovery/scaling-registration.service.js +0 -261
- package/dist/services/processor-discovery/scaling-registration.service.js.map +0 -1
- package/dist/services/processor-discovery/worker-factory.service.d.ts +0 -54
- package/dist/services/processor-discovery/worker-factory.service.d.ts.map +0 -1
- package/dist/services/processor-discovery/worker-factory.service.js +0 -185
- package/dist/services/processor-discovery/worker-factory.service.js.map +0 -1
- package/dist/services/queue-bus/entity-target.d.ts +0 -58
- package/dist/services/queue-bus/entity-target.d.ts.map +0 -1
- package/dist/services/queue-bus/entity-target.js +0 -109
- package/dist/services/queue-bus/entity-target.js.map +0 -1
- package/dist/services/queue-bus/queue-bus.types.d.ts +0 -40
- package/dist/services/queue-bus/queue-bus.types.d.ts.map +0 -1
- package/dist/services/queue-bus/queue-bus.types.js +0 -3
- package/dist/services/queue-bus/queue-bus.types.js.map +0 -1
- package/dist/services/queue-bus/queue-target.d.ts +0 -61
- package/dist/services/queue-bus/queue-target.d.ts.map +0 -1
- package/dist/services/queue-bus/queue-target.js +0 -123
- package/dist/services/queue-bus/queue-target.js.map +0 -1
- package/dist/services/queue-events-manager/index.d.ts +0 -2
- package/dist/services/queue-events-manager/index.d.ts.map +0 -1
- package/dist/services/queue-events-manager/index.js.map +0 -1
- package/dist/services/queue-events-manager/queue-events-manager.service.d.ts +0 -120
- package/dist/services/queue-events-manager/queue-events-manager.service.d.ts.map +0 -1
- package/dist/services/queue-events-manager/queue-events-manager.service.js +0 -343
- package/dist/services/queue-events-manager/queue-events-manager.service.js.map +0 -1
- package/dist/services/queue-manager/index.d.ts +0 -2
- package/dist/services/queue-manager/index.d.ts.map +0 -1
- package/dist/services/queue-manager/index.js.map +0 -1
- package/dist/services/queue-manager/queue-manager.service.d.ts +0 -148
- package/dist/services/queue-manager/queue-manager.service.d.ts.map +0 -1
- package/dist/services/queue-manager/queue-manager.service.js +0 -348
- package/dist/services/queue-manager/queue-manager.service.js.map +0 -1
- package/dist/services/resource-lock/index.d.ts +0 -2
- package/dist/services/resource-lock/index.d.ts.map +0 -1
- package/dist/services/resource-lock/index.js +0 -18
- package/dist/services/resource-lock/index.js.map +0 -1
- package/dist/services/resource-lock/resource-lock.service.d.ts +0 -120
- package/dist/services/resource-lock/resource-lock.service.d.ts.map +0 -1
- package/dist/services/resource-lock/resource-lock.service.js +0 -367
- package/dist/services/resource-lock/resource-lock.service.js.map +0 -1
- package/dist/services/service-queue/index.d.ts +0 -3
- package/dist/services/service-queue/index.d.ts.map +0 -1
- package/dist/services/service-queue/index.js +0 -19
- package/dist/services/service-queue/index.js.map +0 -1
- package/dist/services/service-queue/service-queue.service.d.ts +0 -199
- package/dist/services/service-queue/service-queue.service.d.ts.map +0 -1
- package/dist/services/service-queue/service-queue.service.js +0 -617
- package/dist/services/service-queue/service-queue.service.js.map +0 -1
- package/dist/services/service-queue/service-queue.types.d.ts +0 -32
- package/dist/services/service-queue/service-queue.types.d.ts.map +0 -1
- package/dist/services/service-queue/service-queue.types.js +0 -27
- package/dist/services/service-queue/service-queue.types.js.map +0 -1
- package/dist/services/shutdown-state/index.d.ts +0 -2
- package/dist/services/shutdown-state/index.d.ts.map +0 -1
- package/dist/services/shutdown-state/index.js +0 -18
- package/dist/services/shutdown-state/index.js.map +0 -1
- package/dist/services/shutdown-state/shutdown-state.service.d.ts +0 -69
- package/dist/services/shutdown-state/shutdown-state.service.d.ts.map +0 -1
- package/dist/services/shutdown-state/shutdown-state.service.js +0 -127
- package/dist/services/shutdown-state/shutdown-state.service.js.map +0 -1
- package/dist/services/spawn-queue/index.d.ts +0 -2
- package/dist/services/spawn-queue/index.d.ts.map +0 -1
- package/dist/services/spawn-queue/index.js.map +0 -1
- package/dist/services/spawn-queue/spawn-queue.service.d.ts +0 -119
- package/dist/services/spawn-queue/spawn-queue.service.d.ts.map +0 -1
- package/dist/services/spawn-queue/spawn-queue.service.js +0 -273
- package/dist/services/spawn-queue/spawn-queue.service.js.map +0 -1
- package/dist/services/worker-manager/index.d.ts +0 -2
- package/dist/services/worker-manager/index.d.ts.map +0 -1
- package/dist/services/worker-manager/index.js +0 -18
- package/dist/services/worker-manager/index.js.map +0 -1
- package/dist/services/worker-manager/worker-manager.service.d.ts +0 -221
- package/dist/services/worker-manager/worker-manager.service.d.ts.map +0 -1
- package/dist/services/worker-manager/worker-manager.service.js +0 -591
- package/dist/services/worker-manager/worker-manager.service.js.map +0 -1
- package/dist/utils/helpers.d.ts +0 -5
- package/dist/utils/helpers.d.ts.map +0 -1
- package/dist/utils/helpers.js +0 -21
- package/dist/utils/helpers.js.map +0 -1
- package/dist/utils/job.utils.d.ts +0 -50
- package/dist/utils/job.utils.d.ts.map +0 -1
- package/dist/utils/job.utils.js +0 -89
- package/dist/utils/job.utils.js.map +0 -1
package/README.md
CHANGED
|
@@ -26,52 +26,21 @@
|
|
|
26
26
|
<p align="center">
|
|
27
27
|
<img src="https://img.shields.io/npm/v/atomic-queues?style=flat-square&color=cb3837" alt="npm version" />
|
|
28
28
|
<img src="https://img.shields.io/badge/NestJS-11-ea2845?style=flat-square&logo=nestjs" alt="NestJS 11" />
|
|
29
|
-
<img src="https://img.shields.io/badge/BullMQ-5-3c873a?style=flat-square" alt="BullMQ 5" />
|
|
30
29
|
<img src="https://img.shields.io/badge/Redis-7-dc382d?style=flat-square&logo=redis&logoColor=white" alt="Redis 7" />
|
|
31
30
|
<img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="MIT License" />
|
|
32
31
|
</p>
|
|
33
32
|
|
|
34
33
|
---
|
|
35
34
|
|
|
36
|
-
##
|
|
35
|
+
## What is atomic-queues?
|
|
37
36
|
|
|
38
|
-
|
|
37
|
+
A distributed virtual actor runtime for Node.js. Sequential per entity, parallel across entities, zero contention. Three interchangeable API surfaces — queues, CQRS, and actors — backed by one Redis-based execution engine.
|
|
39
38
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
### atomic-queues vs Redlock
|
|
43
|
-
|
|
44
|
-
| | Redlock | atomic-queues |
|
|
45
|
-
|---|---|---|
|
|
46
|
-
| **Architecture** | Distributed mutex (quorum-based) | Per-entity queue (sequential) |
|
|
47
|
-
| **Under contention** | Degrades — retry storms, backoff delays | **Constant** — jobs queue up, execute instantly |
|
|
48
|
-
| **Failure mode** | Silent double-execution (clock drift) | Job stuck in queue (visible, retryable) |
|
|
49
|
-
| **Split-brain risk** | Yes (timing assumptions) | **Impossible** (serial queue) |
|
|
50
|
-
| **Warm-path overhead** | Acquire + release per op | **0 Redis calls** (in-memory hot cache) |
|
|
51
|
-
| **Cold-start** | None | One-time per entity |
|
|
52
|
-
| **Multi-pod scaling** | Contention increases with pods | **Throughput increases with pods** |
|
|
53
|
-
|
|
54
|
-
---
|
|
55
|
-
|
|
56
|
-
## Table of Contents
|
|
57
|
-
|
|
58
|
-
- [Why atomic-queues?](#why-atomic-queues)
|
|
59
|
-
- [How It Works](#how-it-works)
|
|
60
|
-
- [Installation](#installation)
|
|
61
|
-
- [Quick Start](#quick-start)
|
|
62
|
-
- [Commands & Decorators](#commands--decorators)
|
|
63
|
-
- [Configuration](#configuration)
|
|
64
|
-
- [Distributed Worker Lifecycle](#distributed-worker-lifecycle)
|
|
65
|
-
- [Complete Example](#complete-example)
|
|
66
|
-
- [Advanced: Custom Worker Processors](#advanced-custom-worker-processors)
|
|
67
|
-
- [Performance](#performance)
|
|
68
|
-
- [License](#license)
|
|
39
|
+
No BullMQ. No workers. No distributed locks. Just entities, messages, and guarantees.
|
|
69
40
|
|
|
70
41
|
---
|
|
71
42
|
|
|
72
|
-
##
|
|
73
|
-
|
|
74
|
-
### The Problem
|
|
43
|
+
## The Problem
|
|
75
44
|
|
|
76
45
|
Every distributed system eventually hits this:
|
|
77
46
|
|
|
@@ -86,157 +55,76 @@ T₃ UPDATE: $100 − $80 = $20 −$60
|
|
|
86
55
|
Result: Balance is −$60. Both withdrawals succeed. Integrity violated.
|
|
87
56
|
```
|
|
88
57
|
|
|
89
|
-
|
|
58
|
+
## The Solution
|
|
90
59
|
|
|
91
|
-
atomic-queues routes operations through per-entity
|
|
60
|
+
atomic-queues routes operations through per-entity message logs. Same entity → same log → sequential execution. Different entities → parallel logs → full throughput. A shared executor pool dispatches messages via atomic Redis gates — no workers to spawn, no locks to contend.
|
|
92
61
|
|
|
93
62
|
```
|
|
94
63
|
┌─────────────────────────────────────────────────┐
|
|
95
64
|
Request A ─┐ │ Entity: account-42 │
|
|
96
65
|
│ │ ┌──────┐ ┌──────┐ ┌──────┐ │
|
|
97
|
-
Request B ─┼─► Route ─┼─►│
|
|
98
|
-
│ │ └──────┘ └──────┘ └──────┘ │
|
|
99
|
-
Request C ─┘ │
|
|
100
|
-
└─────────────────────────────────────────────────┘
|
|
101
|
-
|
|
102
|
-
┌─────────────────────────────────────────────────┐
|
|
103
|
-
Request D ─┐ │ Entity: account-99 │
|
|
104
|
-
│ │ ┌──────┐ ┌──────┐ │
|
|
105
|
-
Request E ─┼─► Route ─┼─►│ Op 1 │─►│ Op 2 │─────────► [Worker] ──┐ │
|
|
106
|
-
│ │ └──────┘ └──────┘ │ │
|
|
107
|
-
Request F ─┘ │ Sequential ◄───────────┘ │
|
|
66
|
+
Request B ─┼─► Route ─┼─►│ Msg1 │─►│ Msg2 │─►│ Msg3 │─► [Executor] ─┐ │
|
|
67
|
+
│ │ └──────┘ └──────┘ └──────┘ │ │
|
|
68
|
+
Request C ─┘ │ Sequential ◄────────────┘ │
|
|
108
69
|
└─────────────────────────────────────────────────┘
|
|
109
|
-
|
|
110
|
-
▲ These two queues run in PARALLEL across pods ▲
|
|
111
70
|
```
|
|
112
71
|
|
|
113
|
-
**Key properties:**
|
|
114
|
-
- **One worker per entity** — enforced via Redis heartbeat TTL. No duplicates, ever.
|
|
115
|
-
- **Auto-spawn** — workers materialize when jobs arrive, on the pod that sees them first.
|
|
116
|
-
- **Auto-terminate** — idle workers shut down after a configurable timeout.
|
|
117
|
-
- **Self-healing** — node failure → heartbeat expires → worker respawns on a healthy pod.
|
|
118
|
-
- **Distributed** — workers spread across all pods via atomic `SET NX` claim. No leader election, no single point of failure.
|
|
119
|
-
|
|
120
72
|
---
|
|
121
73
|
|
|
122
74
|
## Installation
|
|
123
75
|
|
|
124
76
|
```bash
|
|
125
|
-
npm install atomic-queues
|
|
77
|
+
npm install atomic-queues ioredis
|
|
126
78
|
```
|
|
127
79
|
|
|
128
|
-
|
|
80
|
+
Optional peer dependencies:
|
|
129
81
|
|
|
130
|
-
|
|
82
|
+
```bash
|
|
83
|
+
npm install @nestjs/cqrs # for CQRS surface
|
|
84
|
+
npm install zod zod-to-json-schema # for schema validation in the registry
|
|
85
|
+
```
|
|
131
86
|
|
|
132
87
|
---
|
|
133
88
|
|
|
134
89
|
## Quick Start
|
|
135
90
|
|
|
136
|
-
###
|
|
91
|
+
### Minimal setup
|
|
137
92
|
|
|
138
93
|
```typescript
|
|
139
94
|
import { Module } from '@nestjs/common';
|
|
140
|
-
import { CqrsModule } from '@nestjs/cqrs';
|
|
141
95
|
import { AtomicQueuesModule } from 'atomic-queues';
|
|
142
96
|
|
|
143
97
|
@Module({
|
|
144
98
|
imports: [
|
|
145
|
-
CqrsModule,
|
|
146
99
|
AtomicQueuesModule.forRoot({
|
|
147
100
|
redis: { host: 'localhost', port: 6379 },
|
|
148
|
-
keyPrefix: 'myapp',
|
|
149
|
-
entities: {
|
|
150
|
-
account: {
|
|
151
|
-
queueName: (id) => `account-${id}-queue`,
|
|
152
|
-
workerName: (id) => `account-${id}-worker`,
|
|
153
|
-
maxWorkersPerEntity: 1,
|
|
154
|
-
idleTimeoutSeconds: 15,
|
|
155
|
-
},
|
|
156
|
-
},
|
|
157
101
|
}),
|
|
158
102
|
],
|
|
159
103
|
})
|
|
160
104
|
export class AppModule {}
|
|
161
105
|
```
|
|
162
106
|
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
<details>
|
|
166
|
-
<summary><strong>Async configuration (ConfigService)</strong></summary>
|
|
107
|
+
### Define a command
|
|
167
108
|
|
|
168
109
|
```typescript
|
|
169
|
-
|
|
170
|
-
imports: [ConfigModule],
|
|
171
|
-
useFactory: (config: ConfigService) => ({
|
|
172
|
-
redis: { url: config.get('REDIS_URL') },
|
|
173
|
-
keyPrefix: 'myapp',
|
|
174
|
-
entities: {
|
|
175
|
-
account: {
|
|
176
|
-
queueName: (id) => `account-${id}-queue`,
|
|
177
|
-
workerName: (id) => `account-${id}-worker`,
|
|
178
|
-
},
|
|
179
|
-
},
|
|
180
|
-
}),
|
|
181
|
-
inject: [ConfigService],
|
|
182
|
-
}),
|
|
183
|
-
```
|
|
184
|
-
|
|
185
|
-
</details>
|
|
186
|
-
|
|
187
|
-
### 2. Define Commands
|
|
110
|
+
import { EntityType, QueueEntityId } from 'atomic-queues';
|
|
188
111
|
|
|
189
|
-
|
|
190
|
-
import { QueueEntity, QueueEntityId } from 'atomic-queues';
|
|
191
|
-
|
|
192
|
-
@QueueEntity('account')
|
|
112
|
+
@EntityType('account')
|
|
193
113
|
export class WithdrawCommand {
|
|
194
114
|
constructor(
|
|
195
115
|
@QueueEntityId() public readonly accountId: string,
|
|
196
116
|
public readonly amount: number,
|
|
197
117
|
) {}
|
|
198
118
|
}
|
|
199
|
-
|
|
200
|
-
@QueueEntity('account')
|
|
201
|
-
export class DepositCommand {
|
|
202
|
-
constructor(
|
|
203
|
-
@QueueEntityId() public readonly accountId: string,
|
|
204
|
-
public readonly amount: number,
|
|
205
|
-
) {}
|
|
206
|
-
}
|
|
207
119
|
```
|
|
208
120
|
|
|
209
|
-
###
|
|
121
|
+
### Enqueue it
|
|
210
122
|
|
|
211
123
|
```typescript
|
|
212
|
-
import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
|
|
213
|
-
|
|
214
|
-
@CommandHandler(WithdrawCommand)
|
|
215
|
-
export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
|
|
216
|
-
constructor(private readonly repo: AccountRepository) {}
|
|
217
|
-
|
|
218
|
-
async execute({ accountId, amount }: WithdrawCommand) {
|
|
219
|
-
// SAFE: No race conditions. Sequential execution per account.
|
|
220
|
-
const account = await this.repo.findById(accountId);
|
|
221
|
-
|
|
222
|
-
if (account.balance < amount) {
|
|
223
|
-
throw new InsufficientFundsError(accountId, account.balance, amount);
|
|
224
|
-
}
|
|
225
|
-
|
|
226
|
-
account.balance -= amount;
|
|
227
|
-
await this.repo.save(account);
|
|
228
|
-
}
|
|
229
|
-
}
|
|
230
|
-
```
|
|
231
|
-
|
|
232
|
-
### 4. Enqueue Jobs
|
|
233
|
-
|
|
234
|
-
```typescript
|
|
235
|
-
import { Injectable } from '@nestjs/common';
|
|
236
124
|
import { QueueBus } from 'atomic-queues';
|
|
237
125
|
|
|
238
126
|
@Injectable()
|
|
239
|
-
export class
|
|
127
|
+
export class PaymentService {
|
|
240
128
|
constructor(private readonly queueBus: QueueBus) {}
|
|
241
129
|
|
|
242
130
|
async withdraw(accountId: string, amount: number) {
|
|
@@ -245,361 +133,244 @@ export class AccountService {
|
|
|
245
133
|
}
|
|
246
134
|
```
|
|
247
135
|
|
|
248
|
-
|
|
249
|
-
1. Creates a queue for each `accountId` when jobs arrive
|
|
250
|
-
2. Spawns a worker (spread across pods) to process jobs sequentially
|
|
251
|
-
3. Routes jobs to the correct `@CommandHandler` via CQRS
|
|
252
|
-
4. Terminates idle workers after the configured timeout
|
|
253
|
-
5. Self-heals if a pod dies (heartbeat expires → respawn elsewhere)
|
|
136
|
+
That's it. The command is appended to `account:{accountId}`'s message log and executed sequentially by the shared executor pool.
|
|
254
137
|
|
|
255
138
|
---
|
|
256
139
|
|
|
257
|
-
##
|
|
140
|
+
## Three Surfaces
|
|
141
|
+
|
|
142
|
+
atomic-queues exposes three interchangeable APIs. All three route to the same runtime. Pick whichever fits your mental model.
|
|
258
143
|
|
|
259
|
-
###
|
|
144
|
+
### 1. Queue Surface
|
|
260
145
|
|
|
261
|
-
|
|
146
|
+
The simplest path. Decorate commands, enqueue them.
|
|
262
147
|
|
|
263
148
|
```typescript
|
|
264
|
-
//
|
|
265
|
-
|
|
266
|
-
export class TransferCommand {
|
|
267
|
-
constructor(
|
|
268
|
-
public readonly accountId: string,
|
|
269
|
-
public readonly toAccountId: string,
|
|
270
|
-
public readonly amount: number,
|
|
271
|
-
) {}
|
|
272
|
-
}
|
|
149
|
+
// Enqueue
|
|
150
|
+
await queueBus.enqueue(new WithdrawCommand(accountId, 100));
|
|
273
151
|
|
|
274
|
-
//
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
152
|
+
// Target a specific entity type
|
|
153
|
+
await queueBus.forEntity('account').enqueue(new WithdrawCommand(accountId, 100));
|
|
154
|
+
|
|
155
|
+
// Enqueue and wait for result
|
|
156
|
+
const balance = await queueBus.enqueueAndWait(new GetBalanceQuery(accountId));
|
|
157
|
+
|
|
158
|
+
// Bulk enqueue
|
|
159
|
+
await queueBus.forEntity('account').enqueueBulk([charge1, charge2, charge3]);
|
|
282
160
|
```
|
|
283
161
|
|
|
284
|
-
###
|
|
162
|
+
### 2. CQRS Surface
|
|
285
163
|
|
|
286
|
-
|
|
164
|
+
For teams using `@nestjs/cqrs`. Commands and queries route through the actor runtime instead of executing inline.
|
|
287
165
|
|
|
288
166
|
```typescript
|
|
289
|
-
@
|
|
290
|
-
|
|
167
|
+
@JobCommand()
|
|
168
|
+
@EntityType('account')
|
|
169
|
+
export class WithdrawCommand {
|
|
291
170
|
constructor(
|
|
292
|
-
@QueueEntityId() public readonly accountId: string,
|
|
293
|
-
public readonly targetAccountId: string,
|
|
171
|
+
@QueueEntityId() public readonly accountId: string,
|
|
294
172
|
public readonly amount: number,
|
|
295
173
|
) {}
|
|
296
174
|
}
|
|
297
|
-
```
|
|
298
175
|
|
|
299
|
-
|
|
176
|
+
@CommandHandler(WithdrawCommand)
|
|
177
|
+
export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
|
|
178
|
+
async execute(cmd: WithdrawCommand) {
|
|
179
|
+
// business logic — guaranteed sequential per account
|
|
180
|
+
}
|
|
181
|
+
}
|
|
182
|
+
```
|
|
300
183
|
|
|
301
|
-
###
|
|
184
|
+
### 3. Actor Surface
|
|
302
185
|
|
|
303
|
-
|
|
186
|
+
For stateful entities. The class is the entity. Its methods are message handlers. Its fields are the state.
|
|
304
187
|
|
|
305
188
|
```typescript
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
workerName: (id) => `account-${id}-worker`,
|
|
310
|
-
maxWorkersPerEntity: 1,
|
|
311
|
-
idleTimeoutSeconds: 15,
|
|
312
|
-
})
|
|
189
|
+
import { Actor, On } from 'atomic-queues';
|
|
190
|
+
|
|
191
|
+
@Actor('account')
|
|
313
192
|
@Injectable()
|
|
314
|
-
export class
|
|
315
|
-
|
|
316
|
-
|
|
193
|
+
export class AccountActor {
|
|
194
|
+
private balance = 0;
|
|
195
|
+
|
|
196
|
+
@On(DepositCommand)
|
|
197
|
+
async deposit(msg: DepositCommand) {
|
|
198
|
+
this.balance += msg.amount;
|
|
199
|
+
return this.balance;
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
@On(WithdrawCommand)
|
|
203
|
+
async withdraw(msg: WithdrawCommand) {
|
|
204
|
+
if (this.balance < msg.amount) throw new InsufficientFunds();
|
|
205
|
+
this.balance -= msg.amount;
|
|
206
|
+
return this.balance;
|
|
207
|
+
}
|
|
317
208
|
}
|
|
318
|
-
```
|
|
319
209
|
|
|
320
|
-
|
|
210
|
+
// Usage
|
|
211
|
+
await actorSystem.send('account', accountId, new DepositCommand(100));
|
|
212
|
+
const balance = await actorSystem.sendAndWait('account', accountId, new GetBalanceQuery());
|
|
213
|
+
```
|
|
321
214
|
|
|
322
|
-
|
|
215
|
+
Actor state persists in memory between messages and is automatically saved to Redis on idle eviction.
|
|
323
216
|
|
|
324
217
|
---
|
|
325
218
|
|
|
326
|
-
##
|
|
219
|
+
## Cross-Service Communication
|
|
220
|
+
|
|
221
|
+
Enable the distributed registry and any service connected to the same Redis can send typed messages to any entity — no gRPC, no service mesh, no HTTP endpoints.
|
|
327
222
|
|
|
328
223
|
```typescript
|
|
224
|
+
// Service B: defines and handles the entity
|
|
329
225
|
AtomicQueuesModule.forRoot({
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
password: 'secret', // optional
|
|
335
|
-
},
|
|
336
|
-
|
|
337
|
-
// ── Global settings ───────────────────────────────────────
|
|
338
|
-
keyPrefix: 'myapp', // Redis key namespace (default: 'aq')
|
|
339
|
-
enableCronManager: true, // Legacy cron-based scaling (optional)
|
|
340
|
-
cronInterval: 5000, // Cron tick interval in ms
|
|
341
|
-
|
|
342
|
-
// ── Worker defaults ───────────────────────────────────────
|
|
343
|
-
workerDefaults: {
|
|
344
|
-
concurrency: 1, // Jobs processed concurrently per worker
|
|
345
|
-
stalledInterval: 1000, // ms between stalled-job checks
|
|
346
|
-
lockDuration: 30000, // ms a job is locked during processing
|
|
347
|
-
heartbeatTTL: 3, // Heartbeat key TTL in seconds
|
|
226
|
+
redis: { url: process.env.REDIS_URL },
|
|
227
|
+
registry: {
|
|
228
|
+
enabled: true,
|
|
229
|
+
serviceName: 'billing-service',
|
|
348
230
|
},
|
|
231
|
+
})
|
|
349
232
|
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
account: {
|
|
353
|
-
queueName: (id) => `account-${id}-queue`,
|
|
354
|
-
workerName: (id) => `account-${id}-worker`,
|
|
355
|
-
maxWorkersPerEntity: 1,
|
|
356
|
-
idleTimeoutSeconds: 15,
|
|
357
|
-
|
|
358
|
-
// Fallback property name for entity ID extraction.
|
|
359
|
-
// Used when a command has no @QueueEntityId() decorator
|
|
360
|
-
// and no second argument to @QueueEntity().
|
|
361
|
-
defaultEntityId: 'accountId',
|
|
362
|
-
|
|
363
|
-
workerConfig: { // Override workerDefaults per entity
|
|
364
|
-
concurrency: 1,
|
|
365
|
-
lockDuration: 60000,
|
|
366
|
-
},
|
|
367
|
-
},
|
|
368
|
-
},
|
|
369
|
-
});
|
|
233
|
+
// Service A: sends to it (shared Redis, no code dependency on B)
|
|
234
|
+
await queueBus.enqueue(new WithdrawCommand(accountId, 100));
|
|
370
235
|
```
|
|
371
236
|
|
|
372
|
-
|
|
237
|
+
The registry validates at the call site: entity type exists, message is accepted, payload matches schema (optional). Errors are immediate and clear — not silent dead letters.
|
|
373
238
|
|
|
374
|
-
|
|
239
|
+
### Schema Validation
|
|
375
240
|
|
|
376
|
-
|
|
241
|
+
Attach Zod schemas to message classes for compile-time and runtime safety:
|
|
377
242
|
|
|
378
|
-
```
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
┌──────────────┐
|
|
392
|
-
│ Processing │◄──── Heartbeat refresh (pipeline)
|
|
393
|
-
│ jobs back- │ every 1s (1 Redis round-trip)
|
|
394
|
-
│ to-back │
|
|
395
|
-
└──────┬───────┘
|
|
396
|
-
│ No jobs for idleTimeoutSeconds
|
|
397
|
-
▼
|
|
398
|
-
┌──────────────┐
|
|
399
|
-
│ Idle sweep │──── Hot cache eviction
|
|
400
|
-
│ closes │ Heartbeat keys cleaned up
|
|
401
|
-
│ worker │
|
|
402
|
-
└──────────────┘
|
|
243
|
+
```typescript
|
|
244
|
+
import { Schema } from 'atomic-queues';
|
|
245
|
+
import { z } from 'zod';
|
|
246
|
+
|
|
247
|
+
@Schema(z.object({
|
|
248
|
+
accountId: z.string().uuid(),
|
|
249
|
+
amount: z.number().positive(),
|
|
250
|
+
}))
|
|
251
|
+
@EntityType('account')
|
|
252
|
+
export class WithdrawCommand {
|
|
253
|
+
@QueueEntityId() public readonly accountId: string;
|
|
254
|
+
public readonly amount: number;
|
|
255
|
+
}
|
|
403
256
|
```
|
|
404
257
|
|
|
405
|
-
|
|
258
|
+
Enable `schemaValidation: true` in the registry config. Payload shape is validated against the JSON Schema representation before the message enters the log.
|
|
406
259
|
|
|
407
|
-
|
|
260
|
+
### Codegen
|
|
408
261
|
|
|
409
|
-
|
|
410
|
-
|---|---|---|
|
|
411
|
-
| **Hot** (cache hit) | 0 | Worker known alive |
|
|
412
|
-
| **Warm** (cache miss) | 1 (`EXISTS`) | First time seeing entity |
|
|
413
|
-
| **Cold** (no worker) | 1 (`SET NX`) | Worker needs creation |
|
|
262
|
+
Generate typed interfaces from the live registry:
|
|
414
263
|
|
|
415
|
-
|
|
264
|
+
```bash
|
|
265
|
+
REDIS_URL=redis://localhost:6379 npx atomic-queues generate --ts --output ./generated/contracts.ts
|
|
266
|
+
```
|
|
416
267
|
|
|
417
|
-
|
|
268
|
+
Service A gets fully typed message interfaces without importing Service B's code. Also supports `--json-schema` and `--snapshot`.
|
|
418
269
|
|
|
419
270
|
---
|
|
420
271
|
|
|
421
|
-
##
|
|
422
|
-
|
|
423
|
-
A banking service with withdrawals, deposits, and cross-account transfers:
|
|
272
|
+
## Configuration
|
|
424
273
|
|
|
425
274
|
```typescript
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
import { CqrsModule } from '@nestjs/cqrs';
|
|
429
|
-
import { AtomicQueuesModule } from 'atomic-queues';
|
|
275
|
+
AtomicQueuesModule.forRoot({
|
|
276
|
+
redis: { host: 'localhost', port: 6379 },
|
|
430
277
|
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
redis: { host: 'redis', port: 6379 },
|
|
436
|
-
keyPrefix: 'banking',
|
|
437
|
-
entities: {
|
|
438
|
-
account: {
|
|
439
|
-
queueName: (id) => `account-${id}-queue`,
|
|
440
|
-
workerName: (id) => `account-${id}-worker`,
|
|
441
|
-
maxWorkersPerEntity: 1,
|
|
442
|
-
idleTimeoutSeconds: 15,
|
|
443
|
-
},
|
|
444
|
-
},
|
|
445
|
-
}),
|
|
446
|
-
],
|
|
447
|
-
providers: [
|
|
448
|
-
AccountService,
|
|
449
|
-
WithdrawHandler,
|
|
450
|
-
DepositHandler,
|
|
451
|
-
TransferHandler,
|
|
452
|
-
],
|
|
453
|
-
})
|
|
454
|
-
export class BankingModule {}
|
|
278
|
+
executor: {
|
|
279
|
+
poolSize: 1, // concurrent executors per node
|
|
280
|
+
gateTTL: 30, // seconds before gate expires (safety net)
|
|
281
|
+
},
|
|
455
282
|
|
|
456
|
-
|
|
457
|
-
|
|
283
|
+
entities: {
|
|
284
|
+
account: {
|
|
285
|
+
defaultEntityId: 'accountId',
|
|
286
|
+
gateTTL: 60,
|
|
287
|
+
retry: { maxAttempts: 5, backoff: 'exponential', backoffDelay: 2000 },
|
|
288
|
+
actorIdleTimeout: 120000,
|
|
289
|
+
statePersistence: true, // default: true
|
|
290
|
+
},
|
|
291
|
+
},
|
|
458
292
|
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
}
|
|
293
|
+
registry: {
|
|
294
|
+
enabled: false, // enable for cross-service
|
|
295
|
+
serviceName: 'my-service',
|
|
296
|
+
schemaValidation: false,
|
|
297
|
+
heartbeatInterval: 10000,
|
|
298
|
+
registrationTTL: 30,
|
|
299
|
+
},
|
|
467
300
|
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
public readonly amount: number,
|
|
473
|
-
public readonly source: string,
|
|
474
|
-
) {}
|
|
475
|
-
}
|
|
301
|
+
keyPrefix: 'aq',
|
|
302
|
+
verbose: false,
|
|
303
|
+
})
|
|
304
|
+
```
|
|
476
305
|
|
|
477
|
-
|
|
478
|
-
export class TransferCommand {
|
|
479
|
-
constructor(
|
|
480
|
-
@QueueEntityId() public readonly accountId: string,
|
|
481
|
-
public readonly toAccountId: string,
|
|
482
|
-
public readonly amount: number,
|
|
483
|
-
) {}
|
|
484
|
-
}
|
|
306
|
+
---
|
|
485
307
|
|
|
486
|
-
|
|
487
|
-
import { CommandHandler, ICommandHandler } from '@nestjs/cqrs';
|
|
308
|
+
## Architecture
|
|
488
309
|
|
|
489
|
-
|
|
490
|
-
export class WithdrawHandler implements ICommandHandler<WithdrawCommand> {
|
|
491
|
-
constructor(private readonly repo: AccountRepository) {}
|
|
310
|
+
### How it works
|
|
492
311
|
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
}
|
|
499
|
-
}
|
|
312
|
+
1. **Enqueue**: message is appended to a Redis list (`{prefix}:log:{entityType}:{entityId}`) and the entity is added to the ready set.
|
|
313
|
+
2. **Tickle**: a pub/sub notification wakes the executor pool.
|
|
314
|
+
3. **Schedule**: a Lua script atomically picks an entity from the ready set, acquires its dispatch gate (`SET NX EX`), and pops the next message.
|
|
315
|
+
4. **Execute**: the handler runs (actor method, CQRS handler, or registered processor).
|
|
316
|
+
5. **Complete**: gate is released, entity is re-added to the ready set if more messages remain.
|
|
500
317
|
|
|
501
|
-
|
|
502
|
-
export class TransferHandler implements ICommandHandler<TransferCommand> {
|
|
503
|
-
constructor(
|
|
504
|
-
private readonly repo: AccountRepository,
|
|
505
|
-
private readonly queueBus: QueueBus,
|
|
506
|
-
) {}
|
|
318
|
+
### Guarantees
|
|
507
319
|
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
// Credit destination (enqueued to destination's queue — also safe)
|
|
516
|
-
await this.queueBus.enqueue(
|
|
517
|
-
new DepositCommand(toAccountId, amount, `transfer:${accountId}`),
|
|
518
|
-
);
|
|
519
|
-
}
|
|
520
|
-
}
|
|
320
|
+
| Guarantee | Scope | Mechanism |
|
|
321
|
+
|---|---|---|
|
|
322
|
+
| FIFO per entity | Cluster-wide | Redis list (LPUSH/RPOP) |
|
|
323
|
+
| Single-writer per entity | Cluster-wide | Gate key (SET NX EX) |
|
|
324
|
+
| At-least-once delivery | Per message | Retry on gate TTL expiry |
|
|
325
|
+
| Parallel across entities | Per node | Executor pool concurrency |
|
|
326
|
+
| Durability | Per message | Redis persistence (AOF/RDB) |
|
|
521
327
|
|
|
522
|
-
|
|
523
|
-
import { Controller, Post, Body, Param } from '@nestjs/common';
|
|
524
|
-
import { QueueBus } from 'atomic-queues';
|
|
328
|
+
### What this does NOT guarantee
|
|
525
329
|
|
|
526
|
-
|
|
527
|
-
export class AccountController {
|
|
528
|
-
constructor(private readonly queueBus: QueueBus) {}
|
|
330
|
+
**Exactly-once processing.** Like every distributed message system, handlers must be idempotent. If an executor dies mid-processing, the message retries on another node.
|
|
529
331
|
|
|
530
|
-
|
|
531
|
-
async withdraw(@Param('id') id: string, @Body() body: { amount: number }) {
|
|
532
|
-
await this.queueBus.enqueue(new WithdrawCommand(id, body.amount, uuid()));
|
|
533
|
-
return { queued: true };
|
|
534
|
-
}
|
|
332
|
+
---
|
|
535
333
|
|
|
536
|
-
|
|
537
|
-
async transfer(
|
|
538
|
-
@Param('id') id: string,
|
|
539
|
-
@Body() body: { to: string; amount: number },
|
|
540
|
-
) {
|
|
541
|
-
await this.queueBus.enqueue(new TransferCommand(id, body.to, body.amount));
|
|
542
|
-
return { queued: true };
|
|
543
|
-
}
|
|
544
|
-
}
|
|
545
|
-
```
|
|
334
|
+
## Polyglot Clients
|
|
546
335
|
|
|
547
|
-
|
|
336
|
+
Redis is the protocol. Any language with a Redis client can send messages to atomic-queues entities — three Redis commands:
|
|
548
337
|
|
|
549
|
-
|
|
338
|
+
```
|
|
339
|
+
LPUSH {prefix}:log:{entityType}:{entityId} '<message JSON>'
|
|
340
|
+
SADD {prefix}:ready {entityType}:{entityId}
|
|
341
|
+
PUBLISH {prefix}:tickle 1
|
|
342
|
+
```
|
|
550
343
|
|
|
551
|
-
|
|
344
|
+
See [WIRE-PROTOCOL.md](./WIRE-PROTOCOL.md) for the complete specification.
|
|
552
345
|
|
|
553
|
-
|
|
554
|
-
import { Injectable } from '@nestjs/common';
|
|
555
|
-
import { WorkerProcessor, JobHandler } from 'atomic-queues';
|
|
556
|
-
import { Job } from 'bullmq';
|
|
557
|
-
|
|
558
|
-
@WorkerProcessor({
|
|
559
|
-
entityType: 'account',
|
|
560
|
-
queueName: (id) => `account-${id}-queue`,
|
|
561
|
-
workerName: (id) => `account-${id}-worker`,
|
|
562
|
-
maxWorkersPerEntity: 1,
|
|
563
|
-
idleTimeoutSeconds: 15,
|
|
564
|
-
})
|
|
565
|
-
@Injectable()
|
|
566
|
-
export class AccountProcessor {
|
|
567
|
-
@JobHandler('high-priority-audit')
|
|
568
|
-
async handleAudit(job: Job, entityId: string) {
|
|
569
|
-
// Specific handler for this job type
|
|
570
|
-
}
|
|
346
|
+
---
|
|
571
347
|
|
|
572
|
-
|
|
573
|
-
async handleAll(job: Job, entityId: string) {
|
|
574
|
-
// Wildcard — catches everything not explicitly handled
|
|
575
|
-
// Falls back to CQRS routing automatically when not defined
|
|
576
|
-
}
|
|
577
|
-
}
|
|
578
|
-
```
|
|
348
|
+
## Decorator Reference
|
|
579
349
|
|
|
580
|
-
|
|
350
|
+
| Decorator | Purpose |
|
|
351
|
+
|---|---|
|
|
352
|
+
| `@EntityType('type')` | Route a command/query to an entity type |
|
|
353
|
+
| `@QueueEntityId()` | Mark the property holding the entity ID |
|
|
354
|
+
| `@QueueEntity('type', 'prop')` | Combined entity type + ID |
|
|
355
|
+
| `@JobCommand()` | Mark a command for CQRS auto-routing |
|
|
356
|
+
| `@JobQuery()` | Mark a query for CQRS auto-routing |
|
|
357
|
+
| `@Actor('type')` | Declare a virtual actor class |
|
|
358
|
+
| `@On(MessageClass)` | Handle a message type on an actor |
|
|
359
|
+
| `@Schema(zodSchema)` | Attach a Zod schema for registry validation |
|
|
581
360
|
|
|
582
361
|
---
|
|
583
362
|
|
|
584
|
-
##
|
|
363
|
+
## Migrating from V1
|
|
585
364
|
|
|
586
|
-
|
|
365
|
+
V2 is a full rewrite of the internals. BullMQ is removed. Workers are removed. The public API is largely preserved.
|
|
587
366
|
|
|
588
|
-
|
|
589
|
-
2. **Hot cache** — after first check, subsequent job arrivals for an entity incur 0 Redis calls.
|
|
590
|
-
3. **Direct local spawn** — atomic `SET NX` claim, local worker creation. No queue round-trip.
|
|
591
|
-
4. **Pipelined heartbeats** — heartbeat refresh uses a single Redis pipeline (1 round-trip for 2 keys).
|
|
592
|
-
5. **O(1) worker existence check** — global alive key replaces `KEYS` pattern scan.
|
|
367
|
+
**What stays the same**: `@EntityType`, `@QueueEntityId`, `@QueueEntity`, `@JobCommand`, `@JobQuery`, `queueBus.enqueue()`, `queueBus.forEntity()`, `queueBus.enqueueAndWait()`. These work identically.
|
|
593
368
|
|
|
594
|
-
|
|
369
|
+
**What's removed**: `@WorkerProcessor`, `@JobHandler`, `@EntityScaler`, `@OnSpawnWorker`, `@OnTerminateWorker`, `@GetActiveEntities`, `@GetDesiredWorkerCount`, `.forProcessor()`. All worker and scaling concepts are gone.
|
|
595
370
|
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
| Rare, low-frequency mutual exclusion (config updates, migrations) | Redlock / advisory locks |
|
|
600
|
-
| Exactly-once semantics with audit trail | **atomic-queues** (BullMQ job IDs) |
|
|
601
|
-
| Sub-millisecond synchronous response required | Redlock (synchronous acquire) |
|
|
602
|
-
| Multi-pod, many entities, sustained load | **atomic-queues** (contention-free scaling) |
|
|
371
|
+
**What's new**: `@Actor`, `@On`, `@Schema`, `ActorSystem`, `RegistryService`, distributed registry, codegen CLI.
|
|
372
|
+
|
|
373
|
+
**Migration steps**: (1) remove all `@WorkerProcessor` classes — replace with `@Actor` or configure entity defaults in module config; (2) remove all scaling decorators; (3) run the data migration script to drain in-flight BullMQ jobs to the new log format; (4) remove `bullmq` and `@nestjs/bullmq` from your dependencies.
|
|
603
374
|
|
|
604
375
|
---
|
|
605
376
|
|