@venizia/ignis-docs 0.2.0 → 0.2.1-0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -14
- package/content/best-practices/api-usage-examples.md +39 -19
- package/content/best-practices/architectural-patterns.md +22 -11
- package/content/best-practices/architecture-decisions.md +29 -16
- package/content/best-practices/code-style-standards/advanced-patterns.md +43 -31
- package/content/best-practices/code-style-standards/constants-configuration.md +19 -4
- package/content/best-practices/code-style-standards/control-flow.md +33 -1
- package/content/best-practices/code-style-standards/documentation.md +28 -8
- package/content/best-practices/code-style-standards/function-patterns.md +15 -4
- package/content/best-practices/code-style-standards/index.md +7 -2
- package/content/best-practices/code-style-standards/naming-conventions.md +26 -2
- package/content/best-practices/code-style-standards/route-definitions.md +9 -5
- package/content/best-practices/code-style-standards/tooling.md +14 -1
- package/content/best-practices/code-style-standards/type-safety.md +39 -6
- package/content/best-practices/common-pitfalls.md +59 -35
- package/content/best-practices/contribution-workflow.md +6 -2
- package/content/best-practices/data-modeling.md +78 -62
- package/content/best-practices/deployment-strategies.md +56 -19
- package/content/best-practices/error-handling.md +182 -93
- package/content/best-practices/index.md +6 -0
- package/content/best-practices/performance-optimization.md +26 -13
- package/content/best-practices/security-guidelines.md +35 -9
- package/content/best-practices/testing-strategies.md +7 -2
- package/content/best-practices/troubleshooting-tips.md +27 -17
- package/content/extensions/components/api-reference.md +107 -322
- package/content/extensions/components/authentication/api.md +454 -603
- package/content/extensions/components/authentication/errors.md +121 -498
- package/content/extensions/components/authentication/index.md +88 -801
- package/content/extensions/components/authentication/usage.md +207 -956
- package/content/extensions/components/authorization/api.md +736 -656
- package/content/extensions/components/authorization/errors.md +168 -206
- package/content/extensions/components/authorization/index.md +82 -797
- package/content/extensions/components/authorization/usage.md +194 -527
- package/content/extensions/components/health-check.md +71 -243
- package/content/extensions/components/mail/api.md +504 -287
- package/content/extensions/components/mail/errors.md +73 -61
- package/content/extensions/components/mail/index.md +96 -467
- package/content/extensions/components/mail/usage.md +130 -172
- package/content/extensions/components/request-tracker.md +66 -173
- package/content/extensions/components/socket-io/api.md +195 -13
- package/content/extensions/components/socket-io/errors.md +3 -3
- package/content/extensions/components/socket-io/index.md +50 -337
- package/content/extensions/components/socket-io/usage.md +143 -26
- package/content/extensions/components/static-asset/api.md +410 -142
- package/content/extensions/components/static-asset/errors.md +110 -53
- package/content/extensions/components/static-asset/index.md +79 -608
- package/content/extensions/components/static-asset/usage.md +180 -300
- package/content/extensions/components/websocket/api.md +275 -399
- package/content/extensions/components/websocket/errors.md +47 -56
- package/content/extensions/components/websocket/index.md +74 -407
- package/content/extensions/components/websocket/usage.md +110 -341
- package/content/extensions/helpers/cron/index.md +51 -160
- package/content/extensions/helpers/crypto/index.md +62 -483
- package/content/extensions/helpers/crypto/reference.md +456 -0
- package/content/extensions/helpers/env/index.md +60 -178
- package/content/extensions/helpers/error/index.md +221 -207
- package/content/extensions/helpers/inversion/index.md +65 -556
- package/content/extensions/helpers/inversion/reference.md +522 -0
- package/content/extensions/helpers/kafka/admin.md +20 -1
- package/content/extensions/helpers/kafka/compile-binary.md +41 -35
- package/content/extensions/helpers/kafka/consumer.md +54 -20
- package/content/extensions/helpers/kafka/examples.md +21 -16
- package/content/extensions/helpers/kafka/index.md +80 -610
- package/content/extensions/helpers/kafka/producer.md +134 -5
- package/content/extensions/helpers/kafka/schema-registry.md +45 -70
- package/content/extensions/helpers/logger/hf-logger.md +193 -0
- package/content/extensions/helpers/logger/index.md +64 -563
- package/content/extensions/helpers/logger/pino.md +85 -0
- package/content/extensions/helpers/logger/reference.md +746 -0
- package/content/extensions/helpers/network/api.md +241 -195
- package/content/extensions/helpers/network/index.md +72 -530
- package/content/extensions/helpers/queue/index.md +72 -900
- package/content/extensions/helpers/queue/reference.md +467 -0
- package/content/extensions/helpers/redis/index.md +73 -645
- package/content/extensions/helpers/redis/reference.md +727 -0
- package/content/extensions/helpers/secrets/index.md +66 -0
- package/content/extensions/helpers/socket-io/api.md +305 -203
- package/content/extensions/helpers/socket-io/index.md +66 -432
- package/content/extensions/helpers/storage/api.md +564 -462
- package/content/extensions/helpers/storage/index.md +77 -573
- package/content/extensions/helpers/types/index.md +66 -499
- package/content/extensions/helpers/types/reference.md +650 -0
- package/content/extensions/helpers/uid/index.md +58 -227
- package/content/extensions/helpers/websocket/api.md +329 -216
- package/content/extensions/helpers/websocket/index.md +65 -503
- package/content/extensions/helpers/worker-thread/index.md +58 -396
- package/content/extensions/helpers/worker-thread/reference.md +428 -0
- package/content/guides/core-concepts/persistent/models.md +1 -1
- package/content/guides/core-concepts/persistent/search-typesense.md +3 -3
- package/content/guides/core-concepts/persistent/transactions.md +1 -1
- package/content/guides/core-concepts/secrets-vault.md +177 -0
- package/content/guides/core-concepts/services.md +1 -1
- package/content/guides/migrations/redis-helpers-migration.md +1 -1
- package/content/guides/migrations/unified-connectors-migration.md +2 -2
- package/content/guides/tutorials/ecommerce-api.md +3 -8
- package/content/references/base/application.md +1 -1
- package/content/references/base/connectors.md +79 -136
- package/content/references/base/datasources-reference.md +599 -0
- package/content/references/base/datasources.md +84 -444
- package/content/references/base/dependency-injection.md +17 -39
- package/content/references/base/filter-system/application-usage.md +69 -121
- package/content/references/base/filter-system/array-operators.md +12 -0
- package/content/references/base/filter-system/comparison-operators.md +12 -0
- package/content/references/base/filter-system/default-filter.md +136 -348
- package/content/references/base/filter-system/fields-order-pagination.md +38 -16
- package/content/references/base/filter-system/index.md +106 -257
- package/content/references/base/filter-system/json-filtering.md +12 -2
- package/content/references/base/filter-system/list-operators.md +16 -2
- package/content/references/base/filter-system/logical-operators.md +13 -0
- package/content/references/base/filter-system/null-operators.md +13 -0
- package/content/references/base/filter-system/pattern-matching.md +12 -0
- package/content/references/base/filter-system/quick-reference.md +11 -2
- package/content/references/base/filter-system/range-operators.md +12 -0
- package/content/references/base/filter-system/tips.md +70 -133
- package/content/references/base/filter-system/use-cases.md +156 -233
- package/content/references/base/middlewares.md +35 -21
- package/content/references/base/models-reference.md +886 -0
- package/content/references/base/models.md +80 -1452
- package/content/references/base/repositories/advanced.md +156 -192
- package/content/references/base/repositories/index.md +77 -650
- package/content/references/base/repositories/mixins.md +22 -18
- package/content/references/base/repositories/relations.md +123 -171
- package/content/references/base/repositories/soft-deletable.md +58 -56
- package/content/references/base/secrets.md +263 -0
- package/content/references/base/services.md +2 -2
- package/content/references/configuration/environment-variables.md +48 -4
- package/content/references/configuration/index.md +49 -31
- package/content/references/quick-reference.md +3 -16
- package/content/references/utilities/crypto.md +35 -76
- package/content/references/utilities/date.md +33 -73
- package/content/references/utilities/index.md +1 -1
- package/content/references/utilities/jsx-reference.md +298 -0
- package/content/references/utilities/jsx.md +82 -525
- package/content/references/utilities/module.md +29 -62
- package/content/references/utilities/parse.md +34 -64
- package/content/references/utilities/performance.md +33 -58
- package/content/references/utilities/promise.md +28 -62
- package/content/references/utilities/request.md +57 -218
- package/content/references/utilities/schema.md +43 -137
- package/content/references/utilities/statuses-reference.md +361 -0
- package/content/references/utilities/statuses.md +63 -667
- package/package.json +8 -8
|
@@ -0,0 +1,467 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Queue - Full Reference
|
|
3
|
+
description: Complete reference for BullMQHelper, SequentialQueueHelper, HfQueueHelper, and MQTTClientHelper - every option, method, and state transition
|
|
4
|
+
difficulty: intermediate
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Queue - Full Reference
|
|
8
|
+
|
|
9
|
+
Exhaustive reference for the Queue helper family. For a readable introduction and the most common tasks, start with the [Queue overview](/extensions/helpers/queue/).
|
|
10
|
+
|
|
11
|
+
**Files:**
|
|
12
|
+
|
|
13
|
+
- [`packages/helpers/src/modules/queue/bullmq/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/bullmq/helper.ts) - `BullMQHelper`
|
|
14
|
+
- [`packages/helpers/src/modules/queue/internal/sequential/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/helper.ts) - `SequentialQueueHelper`
|
|
15
|
+
- [`packages/helpers/src/modules/queue/internal/sequential/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/types.ts) - `QueueStatuses`, `TQueueStatus`, `TQueueElement`, `IQueueCallback`
|
|
16
|
+
- [`packages/helpers/src/modules/queue/internal/hf/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/hf/helper.ts) - `HfQueueHelper`
|
|
17
|
+
- [`packages/helpers/src/modules/queue/mqtt/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/mqtt/helper.ts) - `MQTTClientHelper`
|
|
18
|
+
- [`packages/helpers/src/modules/queue/common/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/common/types.ts) - `TBullQueueRole`
|
|
19
|
+
|
|
20
|
+
## Class Overview
|
|
21
|
+
|
|
22
|
+
| Class | Extends | Peer dependency | Use case |
|
|
23
|
+
|-------|---------|------------------|----------|
|
|
24
|
+
| `BullMQHelper` | `BaseHelper` | `bullmq` | Redis-backed job queue - durable, multi-worker background processing |
|
|
25
|
+
| `SequentialQueueHelper` (alias `QueueHelper`) | `BaseHelper` | none | Single-process, in-memory, one-at-a-time sequencing |
|
|
26
|
+
| `HfQueueHelper` | `BaseHelper` | none | Generic O(1) FIFO primitive - single-threaded, no callbacks, no persistence |
|
|
27
|
+
| `MQTTClientHelper` | `BaseHelper` | `mqtt` | MQTT broker pub/sub - IoT and lightweight real-time events |
|
|
28
|
+
|
|
29
|
+
Kafka (`KafkaProducerHelper`, `KafkaConsumerHelper`, `KafkaAdminHelper`, `KafkaSchemaRegistryHelper`) is a fourth backend under the same `queue/` module tree but is documented on its own page - see [Kafka Helpers](/extensions/helpers/kafka/).
|
|
30
|
+
|
|
31
|
+
## Import Paths
|
|
32
|
+
|
|
33
|
+
`BullMQHelper` and `MQTTClientHelper` live behind sub-path exports so their peer dependencies (`bullmq`, `mqtt`) never become hard dependencies of the base package. `SequentialQueueHelper`, `QueueHelper`, `QueueStatuses`, `HfQueueHelper`, and `TBullQueueRole` ship from the root package.
|
|
34
|
+
|
|
35
|
+
```typescript
|
|
36
|
+
// Root package - no peer dependency
|
|
37
|
+
import { SequentialQueueHelper, QueueHelper, QueueStatuses, HfQueueHelper } from '@venizia/ignis-helpers';
|
|
38
|
+
import type { TQueueStatus, TQueueElement, IQueueCallback, IHfQueueNode, TBullQueueRole } from '@venizia/ignis-helpers';
|
|
39
|
+
|
|
40
|
+
// BullMQ - sub-path export
|
|
41
|
+
import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
|
|
42
|
+
|
|
43
|
+
// MQTT - sub-path export
|
|
44
|
+
import { MQTTClientHelper, type IMQTTClientOptions } from '@venizia/ignis-helpers/mqtt';
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## BullMQHelper
|
|
48
|
+
|
|
49
|
+
`Source ->` [`packages/helpers/src/modules/queue/bullmq/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/bullmq/helper.ts)
|
|
50
|
+
|
|
51
|
+
### Constructor
|
|
52
|
+
|
|
53
|
+
```typescript
|
|
54
|
+
import { RedisSingleHelper } from '@venizia/ignis-helpers';
|
|
55
|
+
import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
|
|
56
|
+
|
|
57
|
+
const worker = new BullMQHelper({
|
|
58
|
+
queueName: 'email-queue',
|
|
59
|
+
identifier: 'email-worker',
|
|
60
|
+
role: 'worker',
|
|
61
|
+
redisConnection: redisHelper,
|
|
62
|
+
numberOfWorker: 3,
|
|
63
|
+
lockDuration: 10 * 60 * 1000,
|
|
64
|
+
onWorkerData: async job => {
|
|
65
|
+
console.log(`Processing job ${job.id}:`, job.data);
|
|
66
|
+
return { status: 'sent' };
|
|
67
|
+
},
|
|
68
|
+
onWorkerDataCompleted: async (job, result) => {
|
|
69
|
+
console.log(`Job ${job.id} completed:`, result);
|
|
70
|
+
},
|
|
71
|
+
onWorkerDataFail: async (job, error) => {
|
|
72
|
+
console.error(`Job ${job?.id} failed:`, error.message);
|
|
73
|
+
},
|
|
74
|
+
});
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
`BullMQHelper.newInstance(opts)` is a static factory equivalent to `new BullMQHelper(opts)`.
|
|
78
|
+
|
|
79
|
+
### IBullMQOptions
|
|
80
|
+
|
|
81
|
+
`IBullMQOptions<TQueueElement = any, TQueueResult = any>`
|
|
82
|
+
|
|
83
|
+
| Option | Type | Default | Description |
|
|
84
|
+
|--------|------|---------|-------------|
|
|
85
|
+
| `queueName` | `string` | - | BullMQ queue name. Must be non-empty. |
|
|
86
|
+
| `identifier` | `string` | - | Scoped-logging identifier. |
|
|
87
|
+
| `role` | `TBullQueueRole` (`'queue' \| 'worker'`) | - | `'queue'` initializes a producer; `'worker'` initializes a consumer. |
|
|
88
|
+
| `redisConnection` | `IRedisHelper` | - | Connection backend. The helper calls `duplicateClient()` on it - never a raw ioredis client. |
|
|
89
|
+
| `numberOfWorker` | `number` | `1` | Worker concurrency (BullMQ `Worker` `concurrency` option). Ignored for `role: 'queue'`. |
|
|
90
|
+
| `lockDuration` | `number` | `5400000` (90 minutes) | Job lock duration in milliseconds. Ignored for `role: 'queue'`. |
|
|
91
|
+
| `onWorkerData` | `(job: Job<TQueueElement, TQueueResult>) => Promise<any>` | - | Job processor. If omitted, the worker logs `id`, `name`, `data` at info level and resolves `undefined`. |
|
|
92
|
+
| `onWorkerDataCompleted` | `(job, result) => Promise<void>` | - | Fired on the BullMQ `Worker` `'completed'` event. |
|
|
93
|
+
| `onWorkerDataFail` | `(job \| undefined, error: Error) => Promise<void>` | - | Fired on the BullMQ `Worker` `'failed'` event. `job` may be `undefined`. |
|
|
94
|
+
|
|
95
|
+
> [!IMPORTANT]
|
|
96
|
+
> Pass an `IRedisHelper` instance to `redisConnection`, **not** the raw ioredis client. `BullMQHelper` calls `redisConnection.duplicateClient()` internally to get a dedicated connection per role - one shared Redis helper can back any number of queues and workers.
|
|
97
|
+
|
|
98
|
+
### Configuration lifecycle
|
|
99
|
+
|
|
100
|
+
The constructor calls `configure()`, which switches on `role`:
|
|
101
|
+
|
|
102
|
+
| `role` | Method called | Result |
|
|
103
|
+
|--------|---------------|--------|
|
|
104
|
+
| `'queue'` | `configureQueue()` | Builds `this.queue` (BullMQ `Queue`); attaches an `'error'` listener |
|
|
105
|
+
| `'worker'` | `configureWorker()` | Builds `this.worker` (BullMQ `Worker`); attaches `'completed'`, `'failed'`, `'error'` listeners |
|
|
106
|
+
| missing / other | neither | Logs `'Invalid client role to configure'` and returns - **does not throw** |
|
|
107
|
+
|
|
108
|
+
`configureQueue()` and `configureWorker()` each guard on `queueName`: if it is falsy, the method logs `'Invalid queue name'` / `'Invalid worker name'` and returns without constructing anything - `this.queue` / `this.worker` stay `undefined`. Neither path throws; a misconfigured helper fails later, when you call `.queue.add(...)` or `.worker` on the `undefined` property.
|
|
109
|
+
|
|
110
|
+
> [!NOTE]
|
|
111
|
+
> An `'error'` event with no listener is re-thrown by Node's `EventEmitter` and crashes the process. `BullMQHelper` always attaches an `'error'` listener to both `queue` and `worker` so a transient Redis error is logged instead of taking the process down.
|
|
112
|
+
|
|
113
|
+
`onWorkerDataCompleted` and `onWorkerDataFail` run through an internal hook wrapper that absorbs both synchronous throws and rejected promises, logging them instead of propagating - a broken callback cannot crash the worker or block the next job.
|
|
114
|
+
|
|
115
|
+
### Cluster queue name wrapping
|
|
116
|
+
|
|
117
|
+
- **Detects cluster mode.** `resolveQueueName()` checks whether `redisConnection.getClient()` is an ioredis `Cluster` instance.
|
|
118
|
+
- **Wraps the name in a hash-tag.** When the connection is a cluster and `queueName` does not already start with `{`, the name is wrapped: `email-queue` becomes `{email-queue}`.
|
|
119
|
+
- **Why.** This forces BullMQ's queue keys onto a single cluster slot, which BullMQ requires for cluster mode. You do not need to wrap the name yourself.
|
|
120
|
+
|
|
121
|
+
### Default job options
|
|
122
|
+
|
|
123
|
+
Every job added via `queue.add()` is created with:
|
|
124
|
+
|
|
125
|
+
```typescript
|
|
126
|
+
defaultJobOptions: {
|
|
127
|
+
removeOnComplete: true,
|
|
128
|
+
removeOnFail: true,
|
|
129
|
+
}
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
BullMQ does not retain job records after they finish (success or failure).
|
|
133
|
+
|
|
134
|
+
### Redis Cluster setup
|
|
135
|
+
|
|
136
|
+
`RedisClusterHelper` does not set `maxRetriesPerRequest: null` automatically - set it inside `clusterOptions.redisOptions` yourself, since BullMQ requires it:
|
|
137
|
+
|
|
138
|
+
```typescript
|
|
139
|
+
import { RedisClusterHelper } from '@venizia/ignis-helpers';
|
|
140
|
+
import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
|
|
141
|
+
|
|
142
|
+
const redisHelper = new RedisClusterHelper({
|
|
143
|
+
name: 'cluster-redis',
|
|
144
|
+
nodes: [
|
|
145
|
+
{ host: 'node1.redis.example.com', port: 6379 },
|
|
146
|
+
{ host: 'node2.redis.example.com', port: 6379 },
|
|
147
|
+
{ host: 'node3.redis.example.com', port: 6379 },
|
|
148
|
+
],
|
|
149
|
+
clusterOptions: {
|
|
150
|
+
enableReadyCheck: true,
|
|
151
|
+
scaleReads: 'slave',
|
|
152
|
+
redisOptions: {
|
|
153
|
+
password: 'your-password',
|
|
154
|
+
tls: {},
|
|
155
|
+
maxRetriesPerRequest: null, // required by BullMQ
|
|
156
|
+
},
|
|
157
|
+
},
|
|
158
|
+
});
|
|
159
|
+
|
|
160
|
+
const worker = BullMQHelper.newInstance({
|
|
161
|
+
queueName: 'my-queue',
|
|
162
|
+
identifier: 'cluster-worker',
|
|
163
|
+
role: 'worker',
|
|
164
|
+
redisConnection: redisHelper,
|
|
165
|
+
onWorkerData: async job => {
|
|
166
|
+
// process job
|
|
167
|
+
},
|
|
168
|
+
});
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
### close()
|
|
172
|
+
|
|
173
|
+
```typescript
|
|
174
|
+
await producer.close();
|
|
175
|
+
await consumer.close();
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Calls `worker?.close()` then `queue?.close()` in sequence - both run even if the first fails, so a failing worker close never leaks the queue's Redis connection. If either fails, `close()` throws a single `ApplicationError` aggregating both failure messages after both close attempts have run.
|
|
179
|
+
|
|
180
|
+
### API summary
|
|
181
|
+
|
|
182
|
+
| Member | Returns | Description |
|
|
183
|
+
|--------|---------|--------------|
|
|
184
|
+
| `static newInstance(opts)` | `BullMQHelper` | Factory, equivalent to `new BullMQHelper(opts)`. |
|
|
185
|
+
| `configure()` | `void` | Dispatches to `configureQueue()` or `configureWorker()` based on `role`. Called automatically by the constructor. |
|
|
186
|
+
| `configureQueue()` | `void` | Builds the BullMQ `Queue`. |
|
|
187
|
+
| `configureWorker()` | `void` | Builds the BullMQ `Worker`. |
|
|
188
|
+
| `close()` | `Promise<void>` | Gracefully closes worker and queue connections. |
|
|
189
|
+
| `queue` | `Queue<TQueueElement, TQueueResult>` | Present when `role: 'queue'`. |
|
|
190
|
+
| `worker` | `Worker<TQueueElement, TQueueResult>` | Present when `role: 'worker'`. |
|
|
191
|
+
|
|
192
|
+
## SequentialQueueHelper
|
|
193
|
+
|
|
194
|
+
`Source ->` [`packages/helpers/src/modules/queue/internal/sequential/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/helper.ts)
|
|
195
|
+
|
|
196
|
+
> [!NOTE]
|
|
197
|
+
> `QueueHelper` is a deprecated alias for `SequentialQueueHelper`, kept for backward compatibility. Prefer `SequentialQueueHelper` in new code.
|
|
198
|
+
|
|
199
|
+
A generator-driven, in-memory queue that processes one element at a time within a single process - no Redis, no persistence.
|
|
200
|
+
|
|
201
|
+
### Constructor
|
|
202
|
+
|
|
203
|
+
```typescript
|
|
204
|
+
import { SequentialQueueHelper } from '@venizia/ignis-helpers';
|
|
205
|
+
|
|
206
|
+
const queue = new SequentialQueueHelper<string>({
|
|
207
|
+
identifier: 'task-queue',
|
|
208
|
+
autoDispatch: true,
|
|
209
|
+
onMessage: async ({ identifier, queueElement }) => {
|
|
210
|
+
console.log(`[${identifier}] Processing:`, queueElement.payload);
|
|
211
|
+
},
|
|
212
|
+
onDataEnqueue: async ({ identifier, queueElement }) => {
|
|
213
|
+
console.log(`[${identifier}] Enqueued:`, queueElement.payload);
|
|
214
|
+
},
|
|
215
|
+
onDataDequeue: async ({ identifier, queueElement }) => {
|
|
216
|
+
console.log(`[${identifier}] Dequeued:`, queueElement.payload);
|
|
217
|
+
},
|
|
218
|
+
onStateChange: async ({ identifier, from, to }) => {
|
|
219
|
+
console.log(`[${identifier}] State: ${from} -> ${to}`);
|
|
220
|
+
},
|
|
221
|
+
});
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
### IQueueCallback
|
|
225
|
+
|
|
226
|
+
`IQueueCallback<TElementPayload>`, constructed together with `identifier: string`.
|
|
227
|
+
|
|
228
|
+
| Option | Type | Default | Description |
|
|
229
|
+
|--------|------|---------|-------------|
|
|
230
|
+
| `autoDispatch` | `boolean` | `true` | When `true`, `enqueue()` automatically calls `nextMessage()`. |
|
|
231
|
+
| `onMessage` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Message processor. If omitted, the internal generator logs a warning and never initializes - nothing is ever processed. |
|
|
232
|
+
| `onDataEnqueue` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Fired after an element is pushed onto `storage`. |
|
|
233
|
+
| `onDataDequeue` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Fired after an element is shifted off `storage`. |
|
|
234
|
+
| `onStateChange` | `(opts: { identifier: string; from: TQueueStatus; to: TQueueStatus }) => ValueOrPromise<void>` | - | Fired on every state transition. |
|
|
235
|
+
|
|
236
|
+
All four hooks run through an internal wrapper that absorbs synchronous throws and promise rejections, logging them instead - a broken callback cannot break the state machine.
|
|
237
|
+
|
|
238
|
+
### TQueueElement
|
|
239
|
+
|
|
240
|
+
```typescript
|
|
241
|
+
type TQueueElement<T> = { isLocked: boolean; payload: T };
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
`isLocked` marks whether the head element is currently being handed to `onMessage`; it is distinct from the queue-level `lock()`/`unlock()` state below.
|
|
245
|
+
|
|
246
|
+
### QueueStatuses and the state machine
|
|
247
|
+
|
|
248
|
+
```
|
|
249
|
+
WAITING ──enqueue──> PROCESSING ──done──> WAITING
|
|
250
|
+
│ │
|
|
251
|
+
└──lock()──> LOCKED <─┘
|
|
252
|
+
│
|
|
253
|
+
unlock()──> WAITING
|
|
254
|
+
│
|
|
255
|
+
settle()──> SETTLED (terminal)
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
| State | Value | Description |
|
|
259
|
+
|-------|-------|-------------|
|
|
260
|
+
| `QueueStatuses.WAITING` | `'000_WAITING'` | Idle, ready to process the next element. |
|
|
261
|
+
| `QueueStatuses.PROCESSING` | `'100_PROCESSING'` | Currently running `onMessage` for the head element. |
|
|
262
|
+
| `QueueStatuses.LOCKED` | `'200_LOCKED'` | Paused. Elements can still be enqueued; nothing processes until `unlock()`. |
|
|
263
|
+
| `QueueStatuses.SETTLED` | `'300_SETTLED'` | Terminal. No further elements accepted. |
|
|
264
|
+
|
|
265
|
+
- **Why numeric prefixes.** `'000_...'` .. `'300_...'` make `state >= QueueStatuses.LOCKED` and `state > QueueStatuses.LOCKED` valid string comparisons - `lock()` and `unlock()` use exactly that to guard against re-entering from an invalid state.
|
|
266
|
+
- **Validate an arbitrary string** with `QueueStatuses.isValid(value)`.
|
|
267
|
+
|
|
268
|
+
### Processing loop
|
|
269
|
+
|
|
270
|
+
Internally, `_messageListener()` is a generator that loops `while (true) { yield this.handleMessage(); }`. `nextMessage()` calls `generator.next()` only when `state === WAITING`; any other state logs a warning and returns without advancing. `handleMessage()`:
|
|
271
|
+
|
|
272
|
+
1. Reads the head element (`getElementAt(0)`); returns early if empty or already `isLocked`.
|
|
273
|
+
2. Transitions `WAITING -> PROCESSING` (skipped if already `LOCKED`/`SETTLED`).
|
|
274
|
+
3. Marks the head `isLocked = true`, adds it to `processingEvents`, and awaits `onMessage`.
|
|
275
|
+
4. Dequeues the completed element and removes it from `processingEvents`.
|
|
276
|
+
5. Transitions back to `WAITING` (skipped if `LOCKED`/`SETTLED`).
|
|
277
|
+
6. If `storage` is now empty and `settle()` was requested, transitions to `SETTLED`; otherwise calls `nextMessage()` again to continue the loop.
|
|
278
|
+
|
|
279
|
+
A failing `onMessage` handler is logged and swallowed by the same hook wrapper described above - the element is still dequeued and the loop continues. `onMessage` owns its own retry policy; the queue itself never retries.
|
|
280
|
+
|
|
281
|
+
### Methods
|
|
282
|
+
|
|
283
|
+
| Method | Returns | Description |
|
|
284
|
+
|--------|---------|--------------|
|
|
285
|
+
| `enqueue(payload)` | `Promise<void>` | Pushes an element onto `storage`. No-ops (logs an error) if `state === SETTLED` or settle was requested. No-ops silently if `payload` is falsy. Calls `nextMessage()` when `autoDispatch` is `true`. |
|
|
286
|
+
| `dequeue()` | `TQueueElement<T> \| undefined` | Shifts the head element off `storage` and fires `onDataDequeue`. |
|
|
287
|
+
| `nextMessage()` | `void` | Advances the generator. Only effective when `state === WAITING`; otherwise logs a warning and returns. |
|
|
288
|
+
| `lock()` | `void` | Transitions to `LOCKED`. No-ops (logs an error) if already `LOCKED` or `SETTLED`. |
|
|
289
|
+
| `unlock(opts)` | `void` | `opts: { shouldProcessNextElement?: boolean }` (default `true`). Transitions to `WAITING`; no-ops (logs an error) if `state > LOCKED` (i.e. already `SETTLED`). Calls `nextMessage()` unless `shouldProcessNextElement: false`. |
|
|
290
|
+
| `settle()` | `void` | Sets the settle flag. Transitions immediately to `SETTLED` unless currently `PROCESSING`, in which case `handleMessage()` finishes the transition once `storage` empties. |
|
|
291
|
+
| `isSettled()` | `boolean` | `true` when `state === SETTLED` and `storage` is empty. |
|
|
292
|
+
| `close()` | `void` | Calls `settle()`, then terminates the generator via `generator.return(...)`. |
|
|
293
|
+
| `getElementAt(position)` | `TQueueElement<T>` | Peek at an element by index (`storage[position]`). |
|
|
294
|
+
| `getState()` | `TQueueStatus` | Current state. |
|
|
295
|
+
| `getTotalEvent()` | `number` | Total elements ever enqueued (monotonic counter, not decremented on dequeue). |
|
|
296
|
+
| `getProcessingEvents()` | `Set<TQueueElement<T>>` | The element(s) currently mid-`onMessage`. |
|
|
297
|
+
|
|
298
|
+
> [!NOTE]
|
|
299
|
+
> Once `SETTLED`, `enqueue()` permanently rejects new elements - there is no way to reopen a settled queue. Construct a new `SequentialQueueHelper` instance instead.
|
|
300
|
+
|
|
301
|
+
## HfQueueHelper
|
|
302
|
+
|
|
303
|
+
`Source ->` [`packages/helpers/src/modules/queue/internal/hf/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/hf/helper.ts)
|
|
304
|
+
|
|
305
|
+
- **O(1) FIFO, not a job queue.** A generic, high-frequency, single-consumer primitive for enqueue, dequeue, and cancel - backed by an array plus a moving head index (no `Array.shift()`, which is O(n)).
|
|
306
|
+
- **No callbacks, no state machine, not thread-safe.** It is the low-level queue the pool helper's waiter list is built on, not a job-processing API.
|
|
307
|
+
- **Reach for `SequentialQueueHelper` instead** unless you specifically need a bare FIFO with cancellable entries.
|
|
308
|
+
|
|
309
|
+
```typescript
|
|
310
|
+
import { HfQueueHelper } from '@venizia/ignis-helpers';
|
|
311
|
+
|
|
312
|
+
const queue = new HfQueueHelper<{ task: string }>({ scope: 'my-waiter-queue' });
|
|
313
|
+
|
|
314
|
+
const node = queue.enqueue({ value: { task: 'resize-image' } });
|
|
315
|
+
queue.cancel({ node }); // O(1) removal without scanning
|
|
316
|
+
|
|
317
|
+
const next = queue.dequeue(); // null when empty
|
|
318
|
+
const remaining = queue.drain(); // remove + return every live value, emptying the queue
|
|
319
|
+
```
|
|
320
|
+
|
|
321
|
+
| Member | Returns | Description |
|
|
322
|
+
|--------|---------|--------------|
|
|
323
|
+
| constructor | - | `opts?: { scope?: string }` - unlike other queue helpers, this takes `scope`, not `identifier`. |
|
|
324
|
+
| `size` | `number` (getter) | Count of live (enqueued, not yet dequeued or cancelled) entries. |
|
|
325
|
+
| `enqueue(opts)` | `IHfQueueNode<T>` | `opts: { value: T }`. Appends and returns a node handle. |
|
|
326
|
+
| `dequeue()` | `T \| null` | Removes and returns the next live value in FIFO order, skipping cancelled nodes. `null` when empty. |
|
|
327
|
+
| `cancel(opts)` | `void` | `opts: { node: IHfQueueNode<T> }`. Marks a queued node cancelled; idempotent. |
|
|
328
|
+
| `drain()` | `T[]` | Removes and returns every remaining live value in FIFO order, emptying the queue. |
|
|
329
|
+
|
|
330
|
+
Consumed entries are compacted out of the backing array once the consumed prefix exceeds 256 entries and covers at least half the array, keeping every operation amortized O(1).
|
|
331
|
+
|
|
332
|
+
## MQTTClientHelper
|
|
333
|
+
|
|
334
|
+
`Source ->` [`packages/helpers/src/modules/queue/mqtt/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/mqtt/helper.ts)
|
|
335
|
+
|
|
336
|
+
### Constructor
|
|
337
|
+
|
|
338
|
+
```typescript
|
|
339
|
+
import { MQTTClientHelper } from '@venizia/ignis-helpers/mqtt';
|
|
340
|
+
|
|
341
|
+
const client = new MQTTClientHelper({
|
|
342
|
+
identifier: 'iot-gateway',
|
|
343
|
+
url: 'mqtt://broker.example.com:1883',
|
|
344
|
+
options: { keepalive: 60 },
|
|
345
|
+
onConnect: () => {
|
|
346
|
+
client.subscribe({ topics: ['devices/+/status'] });
|
|
347
|
+
},
|
|
348
|
+
onMessage: ({ topic, message }) => {
|
|
349
|
+
console.log(topic, message.toString());
|
|
350
|
+
},
|
|
351
|
+
onDisconnect: () => console.log('Disconnected'),
|
|
352
|
+
onError: error => console.error('Connection error:', error.message),
|
|
353
|
+
onClose: error => {
|
|
354
|
+
if (error) console.error('Connection closed with error:', error);
|
|
355
|
+
},
|
|
356
|
+
});
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
The constructor calls `configure()` automatically, which connects via `mqtt.connect(url, options)`. Calling `configure()` again on an already-connected client is a no-op (logs and returns).
|
|
360
|
+
|
|
361
|
+
### IMQTTClientOptions
|
|
362
|
+
|
|
363
|
+
| Option | Type | Default | Description |
|
|
364
|
+
|--------|------|---------|-------------|
|
|
365
|
+
| `identifier` | `string` | - | Scoped-logging identifier. |
|
|
366
|
+
| `url` | `string` | - | MQTT broker URL. Must be non-empty - throws an `ApplicationError` (status 500) if empty. |
|
|
367
|
+
| `options` | `mqtt.IClientOptions` | - | MQTT.js client options (username, password, keepalive, ...). |
|
|
368
|
+
| `onMessage` | `(opts: { topic: string; message: Buffer }) => void` | - | Required. Message handler for all subscribed topics. |
|
|
369
|
+
| `onConnect` | `() => void` | - | Fired on the `'connect'` event. |
|
|
370
|
+
| `onDisconnect` | `() => void` | - | Fired on the `'disconnect'` event. |
|
|
371
|
+
| `onError` | `(error: Error) => void` | - | Fired on the `'error'` event. |
|
|
372
|
+
| `onClose` | `(error?: Error) => void` | - | Fired on the `'close'` event. |
|
|
373
|
+
|
|
374
|
+
> [!NOTE]
|
|
375
|
+
> At connect time, `MQTTClientHelper` logs the broker `url` through `redactUrlCredentials()` and `options` through `redactSecrets()`. A password embedded in the URL (`mqtts://user:hunter2@broker:8883`) never reaches the log - only `mqtts://user:[REDACTED]@broker:8883` does.
|
|
376
|
+
|
|
377
|
+
### Methods
|
|
378
|
+
|
|
379
|
+
| Method | Returns | Description |
|
|
380
|
+
|--------|---------|--------------|
|
|
381
|
+
| `configure()` | `void` | Connects to the broker and wires `connect`/`disconnect`/`message`/`error`/`close` listeners. Called automatically by the constructor; a no-op if already connected. |
|
|
382
|
+
| `getClient()` | `mqtt.MqttClient \| undefined` | The underlying MQTT.js client. |
|
|
383
|
+
| `subscribe(opts)` | `Promise<string[]>` | `opts: { topics: string[] }`. Rejects with an `ApplicationError` (status 400) if the client is not connected. |
|
|
384
|
+
| `publish(opts)` | `Promise<{ topic, message }>` | `opts: { topic: string; message: string \| Buffer }`. Rejects with an `ApplicationError` (status 400) if the client is not connected. |
|
|
385
|
+
| `close(opts?)` | `Promise<void>` | `opts?: { isForce?: boolean }` (default `false`). Ends the client via `client.end(isForce, undefined, callback)`, resolving once the driver reports the connection ended. Safe to call more than once. |
|
|
386
|
+
|
|
387
|
+
## Troubleshooting
|
|
388
|
+
|
|
389
|
+
### "Invalid queue name" / "Invalid worker name"
|
|
390
|
+
|
|
391
|
+
**Cause:** `queueName` is empty when `BullMQHelper` configures a queue or worker. This is a **logged error, not a thrown exception** - `this.queue` / `this.worker` are simply never assigned.
|
|
392
|
+
|
|
393
|
+
**Fix:** Provide a non-empty `queueName`:
|
|
394
|
+
|
|
395
|
+
```typescript
|
|
396
|
+
// Wrong - this.queue stays undefined, calling .queue.add(...) throws later
|
|
397
|
+
new BullMQHelper({ queueName: '', role: 'queue', identifier: 'x', redisConnection: redis });
|
|
398
|
+
|
|
399
|
+
// Correct
|
|
400
|
+
new BullMQHelper({ queueName: 'my-email-queue', role: 'queue', identifier: 'x', redisConnection: redis });
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
### "Invalid client role to configure"
|
|
404
|
+
|
|
405
|
+
**Cause:** `role` is missing or not one of `'queue'` / `'worker'`. Logged, not thrown.
|
|
406
|
+
|
|
407
|
+
**Fix:**
|
|
408
|
+
|
|
409
|
+
```typescript
|
|
410
|
+
new BullMQHelper({ role: 'worker', queueName: 'q', identifier: 'x', redisConnection: redis, /* ... */ });
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
### "Invalid url to configure mqtt client!"
|
|
414
|
+
|
|
415
|
+
**Cause:** `url` is empty when constructing `MQTTClientHelper`. **This one throws** an `ApplicationError` (status 500).
|
|
416
|
+
|
|
417
|
+
**Fix:**
|
|
418
|
+
|
|
419
|
+
```typescript
|
|
420
|
+
new MQTTClientHelper({ url: 'mqtt://localhost:1883', identifier: 'x', options: {}, onMessage: () => {} });
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
### "MQTT Client is not available to subscribe topic!" / "... to publish message!"
|
|
424
|
+
|
|
425
|
+
**Cause:** `subscribe()` or `publish()` called before the client finished connecting, or after it disconnected. Rejects with an `ApplicationError` (status 400).
|
|
426
|
+
|
|
427
|
+
**Fix:** Call `subscribe()`/`publish()` from inside `onConnect`, or check `client.getClient()?.connected` first.
|
|
428
|
+
|
|
429
|
+
### Elements not processing in `SequentialQueueHelper`
|
|
430
|
+
|
|
431
|
+
**Checklist:**
|
|
432
|
+
- `onMessage` must be provided - without it, the generator never initializes and nothing is ever processed.
|
|
433
|
+
- Check `getState()` - a `LOCKED` queue accepts `enqueue()` but never processes; call `unlock({ shouldProcessNextElement: true })`.
|
|
434
|
+
- Check `autoDispatch` - if `false`, call `nextMessage()` manually after each `enqueue()`.
|
|
435
|
+
- Check `isSettled()` - a settled queue rejects new elements; construct a new instance.
|
|
436
|
+
|
|
437
|
+
## Import Reference
|
|
438
|
+
|
|
439
|
+
```typescript
|
|
440
|
+
import {
|
|
441
|
+
SequentialQueueHelper,
|
|
442
|
+
QueueHelper, // deprecated alias for SequentialQueueHelper
|
|
443
|
+
QueueStatuses,
|
|
444
|
+
HfQueueHelper,
|
|
445
|
+
} from '@venizia/ignis-helpers';
|
|
446
|
+
|
|
447
|
+
import type {
|
|
448
|
+
TQueueStatus,
|
|
449
|
+
TQueueElement,
|
|
450
|
+
IQueueCallback,
|
|
451
|
+
IHfQueueNode,
|
|
452
|
+
TBullQueueRole,
|
|
453
|
+
} from '@venizia/ignis-helpers';
|
|
454
|
+
|
|
455
|
+
import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
|
|
456
|
+
|
|
457
|
+
import { MQTTClientHelper } from '@venizia/ignis-helpers/mqtt';
|
|
458
|
+
import type { IMQTTClientOptions } from '@venizia/ignis-helpers/mqtt';
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
## See also
|
|
462
|
+
|
|
463
|
+
- [Queue overview](/extensions/helpers/queue/) - introduction and the most common tasks
|
|
464
|
+
- [Redis Helper - Full Reference](/extensions/helpers/redis/reference) - `IRedisHelper`, `duplicateClient()`, and the `maxRetriesPerRequest` defaults per topology
|
|
465
|
+
- [Kafka Helpers](/extensions/helpers/kafka/) - the fourth queueing backend
|
|
466
|
+
- [BullMQ documentation](https://docs.bullmq.io/) - underlying job queue library
|
|
467
|
+
- [MQTT.js](https://github.com/mqttjs/MQTT.js) - underlying MQTT client library
|