@venizia/ignis-docs 0.2.0 → 0.2.1-0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (142) hide show
  1. package/README.md +14 -14
  2. package/content/best-practices/api-usage-examples.md +39 -19
  3. package/content/best-practices/architectural-patterns.md +22 -11
  4. package/content/best-practices/architecture-decisions.md +29 -16
  5. package/content/best-practices/code-style-standards/advanced-patterns.md +43 -31
  6. package/content/best-practices/code-style-standards/constants-configuration.md +19 -4
  7. package/content/best-practices/code-style-standards/control-flow.md +33 -1
  8. package/content/best-practices/code-style-standards/documentation.md +28 -8
  9. package/content/best-practices/code-style-standards/function-patterns.md +15 -4
  10. package/content/best-practices/code-style-standards/index.md +7 -2
  11. package/content/best-practices/code-style-standards/naming-conventions.md +26 -2
  12. package/content/best-practices/code-style-standards/route-definitions.md +9 -5
  13. package/content/best-practices/code-style-standards/tooling.md +14 -1
  14. package/content/best-practices/code-style-standards/type-safety.md +39 -6
  15. package/content/best-practices/common-pitfalls.md +59 -35
  16. package/content/best-practices/contribution-workflow.md +6 -2
  17. package/content/best-practices/data-modeling.md +78 -62
  18. package/content/best-practices/deployment-strategies.md +56 -19
  19. package/content/best-practices/error-handling.md +182 -93
  20. package/content/best-practices/index.md +6 -0
  21. package/content/best-practices/performance-optimization.md +26 -13
  22. package/content/best-practices/security-guidelines.md +35 -9
  23. package/content/best-practices/testing-strategies.md +7 -2
  24. package/content/best-practices/troubleshooting-tips.md +27 -17
  25. package/content/extensions/components/api-reference.md +107 -322
  26. package/content/extensions/components/authentication/api.md +454 -603
  27. package/content/extensions/components/authentication/errors.md +121 -498
  28. package/content/extensions/components/authentication/index.md +88 -801
  29. package/content/extensions/components/authentication/usage.md +207 -956
  30. package/content/extensions/components/authorization/api.md +736 -656
  31. package/content/extensions/components/authorization/errors.md +168 -206
  32. package/content/extensions/components/authorization/index.md +82 -797
  33. package/content/extensions/components/authorization/usage.md +194 -527
  34. package/content/extensions/components/health-check.md +71 -243
  35. package/content/extensions/components/mail/api.md +504 -287
  36. package/content/extensions/components/mail/errors.md +73 -61
  37. package/content/extensions/components/mail/index.md +96 -467
  38. package/content/extensions/components/mail/usage.md +130 -172
  39. package/content/extensions/components/request-tracker.md +66 -173
  40. package/content/extensions/components/socket-io/api.md +195 -13
  41. package/content/extensions/components/socket-io/errors.md +3 -3
  42. package/content/extensions/components/socket-io/index.md +50 -337
  43. package/content/extensions/components/socket-io/usage.md +143 -26
  44. package/content/extensions/components/static-asset/api.md +410 -142
  45. package/content/extensions/components/static-asset/errors.md +110 -53
  46. package/content/extensions/components/static-asset/index.md +79 -608
  47. package/content/extensions/components/static-asset/usage.md +180 -300
  48. package/content/extensions/components/websocket/api.md +275 -399
  49. package/content/extensions/components/websocket/errors.md +47 -56
  50. package/content/extensions/components/websocket/index.md +74 -407
  51. package/content/extensions/components/websocket/usage.md +110 -341
  52. package/content/extensions/helpers/cron/index.md +51 -160
  53. package/content/extensions/helpers/crypto/index.md +62 -483
  54. package/content/extensions/helpers/crypto/reference.md +456 -0
  55. package/content/extensions/helpers/env/index.md +60 -178
  56. package/content/extensions/helpers/error/index.md +221 -207
  57. package/content/extensions/helpers/inversion/index.md +65 -556
  58. package/content/extensions/helpers/inversion/reference.md +522 -0
  59. package/content/extensions/helpers/kafka/admin.md +20 -1
  60. package/content/extensions/helpers/kafka/compile-binary.md +41 -35
  61. package/content/extensions/helpers/kafka/consumer.md +54 -20
  62. package/content/extensions/helpers/kafka/examples.md +21 -16
  63. package/content/extensions/helpers/kafka/index.md +80 -610
  64. package/content/extensions/helpers/kafka/producer.md +134 -5
  65. package/content/extensions/helpers/kafka/schema-registry.md +45 -70
  66. package/content/extensions/helpers/logger/hf-logger.md +193 -0
  67. package/content/extensions/helpers/logger/index.md +64 -563
  68. package/content/extensions/helpers/logger/pino.md +85 -0
  69. package/content/extensions/helpers/logger/reference.md +746 -0
  70. package/content/extensions/helpers/network/api.md +241 -195
  71. package/content/extensions/helpers/network/index.md +72 -530
  72. package/content/extensions/helpers/queue/index.md +72 -900
  73. package/content/extensions/helpers/queue/reference.md +467 -0
  74. package/content/extensions/helpers/redis/index.md +73 -645
  75. package/content/extensions/helpers/redis/reference.md +727 -0
  76. package/content/extensions/helpers/secrets/index.md +66 -0
  77. package/content/extensions/helpers/socket-io/api.md +305 -203
  78. package/content/extensions/helpers/socket-io/index.md +66 -432
  79. package/content/extensions/helpers/storage/api.md +564 -462
  80. package/content/extensions/helpers/storage/index.md +77 -573
  81. package/content/extensions/helpers/types/index.md +66 -499
  82. package/content/extensions/helpers/types/reference.md +650 -0
  83. package/content/extensions/helpers/uid/index.md +58 -227
  84. package/content/extensions/helpers/websocket/api.md +329 -216
  85. package/content/extensions/helpers/websocket/index.md +65 -503
  86. package/content/extensions/helpers/worker-thread/index.md +58 -396
  87. package/content/extensions/helpers/worker-thread/reference.md +428 -0
  88. package/content/guides/core-concepts/persistent/models.md +1 -1
  89. package/content/guides/core-concepts/persistent/search-typesense.md +3 -3
  90. package/content/guides/core-concepts/persistent/transactions.md +1 -1
  91. package/content/guides/core-concepts/secrets-vault.md +177 -0
  92. package/content/guides/core-concepts/services.md +1 -1
  93. package/content/guides/migrations/redis-helpers-migration.md +1 -1
  94. package/content/guides/migrations/unified-connectors-migration.md +2 -2
  95. package/content/guides/tutorials/ecommerce-api.md +3 -8
  96. package/content/references/base/application.md +1 -1
  97. package/content/references/base/connectors.md +79 -136
  98. package/content/references/base/datasources-reference.md +599 -0
  99. package/content/references/base/datasources.md +84 -444
  100. package/content/references/base/dependency-injection.md +17 -39
  101. package/content/references/base/filter-system/application-usage.md +69 -121
  102. package/content/references/base/filter-system/array-operators.md +12 -0
  103. package/content/references/base/filter-system/comparison-operators.md +12 -0
  104. package/content/references/base/filter-system/default-filter.md +136 -348
  105. package/content/references/base/filter-system/fields-order-pagination.md +38 -16
  106. package/content/references/base/filter-system/index.md +106 -257
  107. package/content/references/base/filter-system/json-filtering.md +12 -2
  108. package/content/references/base/filter-system/list-operators.md +16 -2
  109. package/content/references/base/filter-system/logical-operators.md +13 -0
  110. package/content/references/base/filter-system/null-operators.md +13 -0
  111. package/content/references/base/filter-system/pattern-matching.md +12 -0
  112. package/content/references/base/filter-system/quick-reference.md +11 -2
  113. package/content/references/base/filter-system/range-operators.md +12 -0
  114. package/content/references/base/filter-system/tips.md +70 -133
  115. package/content/references/base/filter-system/use-cases.md +156 -233
  116. package/content/references/base/middlewares.md +35 -21
  117. package/content/references/base/models-reference.md +886 -0
  118. package/content/references/base/models.md +80 -1452
  119. package/content/references/base/repositories/advanced.md +156 -192
  120. package/content/references/base/repositories/index.md +77 -650
  121. package/content/references/base/repositories/mixins.md +22 -18
  122. package/content/references/base/repositories/relations.md +123 -171
  123. package/content/references/base/repositories/soft-deletable.md +58 -56
  124. package/content/references/base/secrets.md +263 -0
  125. package/content/references/base/services.md +2 -2
  126. package/content/references/configuration/environment-variables.md +48 -4
  127. package/content/references/configuration/index.md +49 -31
  128. package/content/references/quick-reference.md +3 -16
  129. package/content/references/utilities/crypto.md +35 -76
  130. package/content/references/utilities/date.md +33 -73
  131. package/content/references/utilities/index.md +1 -1
  132. package/content/references/utilities/jsx-reference.md +298 -0
  133. package/content/references/utilities/jsx.md +82 -525
  134. package/content/references/utilities/module.md +29 -62
  135. package/content/references/utilities/parse.md +34 -64
  136. package/content/references/utilities/performance.md +33 -58
  137. package/content/references/utilities/promise.md +28 -62
  138. package/content/references/utilities/request.md +57 -218
  139. package/content/references/utilities/schema.md +43 -137
  140. package/content/references/utilities/statuses-reference.md +361 -0
  141. package/content/references/utilities/statuses.md +63 -667
  142. package/package.json +8 -8
@@ -0,0 +1,467 @@
1
+ ---
2
+ title: Queue - Full Reference
3
+ description: Complete reference for BullMQHelper, SequentialQueueHelper, HfQueueHelper, and MQTTClientHelper - every option, method, and state transition
4
+ difficulty: intermediate
5
+ ---
6
+
7
+ # Queue - Full Reference
8
+
9
+ Exhaustive reference for the Queue helper family. For a readable introduction and the most common tasks, start with the [Queue overview](/extensions/helpers/queue/).
10
+
11
+ **Files:**
12
+
13
+ - [`packages/helpers/src/modules/queue/bullmq/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/bullmq/helper.ts) - `BullMQHelper`
14
+ - [`packages/helpers/src/modules/queue/internal/sequential/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/helper.ts) - `SequentialQueueHelper`
15
+ - [`packages/helpers/src/modules/queue/internal/sequential/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/types.ts) - `QueueStatuses`, `TQueueStatus`, `TQueueElement`, `IQueueCallback`
16
+ - [`packages/helpers/src/modules/queue/internal/hf/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/hf/helper.ts) - `HfQueueHelper`
17
+ - [`packages/helpers/src/modules/queue/mqtt/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/mqtt/helper.ts) - `MQTTClientHelper`
18
+ - [`packages/helpers/src/modules/queue/common/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/common/types.ts) - `TBullQueueRole`
19
+
20
+ ## Class Overview
21
+
22
+ | Class | Extends | Peer dependency | Use case |
23
+ |-------|---------|------------------|----------|
24
+ | `BullMQHelper` | `BaseHelper` | `bullmq` | Redis-backed job queue - durable, multi-worker background processing |
25
+ | `SequentialQueueHelper` (alias `QueueHelper`) | `BaseHelper` | none | Single-process, in-memory, one-at-a-time sequencing |
26
+ | `HfQueueHelper` | `BaseHelper` | none | Generic O(1) FIFO primitive - single-threaded, no callbacks, no persistence |
27
+ | `MQTTClientHelper` | `BaseHelper` | `mqtt` | MQTT broker pub/sub - IoT and lightweight real-time events |
28
+
29
+ Kafka (`KafkaProducerHelper`, `KafkaConsumerHelper`, `KafkaAdminHelper`, `KafkaSchemaRegistryHelper`) is a fourth backend under the same `queue/` module tree but is documented on its own page - see [Kafka Helpers](/extensions/helpers/kafka/).
30
+
31
+ ## Import Paths
32
+
33
+ `BullMQHelper` and `MQTTClientHelper` live behind sub-path exports so their peer dependencies (`bullmq`, `mqtt`) never become hard dependencies of the base package. `SequentialQueueHelper`, `QueueHelper`, `QueueStatuses`, `HfQueueHelper`, and `TBullQueueRole` ship from the root package.
34
+
35
+ ```typescript
36
+ // Root package - no peer dependency
37
+ import { SequentialQueueHelper, QueueHelper, QueueStatuses, HfQueueHelper } from '@venizia/ignis-helpers';
38
+ import type { TQueueStatus, TQueueElement, IQueueCallback, IHfQueueNode, TBullQueueRole } from '@venizia/ignis-helpers';
39
+
40
+ // BullMQ - sub-path export
41
+ import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
42
+
43
+ // MQTT - sub-path export
44
+ import { MQTTClientHelper, type IMQTTClientOptions } from '@venizia/ignis-helpers/mqtt';
45
+ ```
46
+
47
+ ## BullMQHelper
48
+
49
+ `Source ->` [`packages/helpers/src/modules/queue/bullmq/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/bullmq/helper.ts)
50
+
51
+ ### Constructor
52
+
53
+ ```typescript
54
+ import { RedisSingleHelper } from '@venizia/ignis-helpers';
55
+ import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
56
+
57
+ const worker = new BullMQHelper({
58
+ queueName: 'email-queue',
59
+ identifier: 'email-worker',
60
+ role: 'worker',
61
+ redisConnection: redisHelper,
62
+ numberOfWorker: 3,
63
+ lockDuration: 10 * 60 * 1000,
64
+ onWorkerData: async job => {
65
+ console.log(`Processing job ${job.id}:`, job.data);
66
+ return { status: 'sent' };
67
+ },
68
+ onWorkerDataCompleted: async (job, result) => {
69
+ console.log(`Job ${job.id} completed:`, result);
70
+ },
71
+ onWorkerDataFail: async (job, error) => {
72
+ console.error(`Job ${job?.id} failed:`, error.message);
73
+ },
74
+ });
75
+ ```
76
+
77
+ `BullMQHelper.newInstance(opts)` is a static factory equivalent to `new BullMQHelper(opts)`.
78
+
79
+ ### IBullMQOptions
80
+
81
+ `IBullMQOptions<TQueueElement = any, TQueueResult = any>`
82
+
83
+ | Option | Type | Default | Description |
84
+ |--------|------|---------|-------------|
85
+ | `queueName` | `string` | - | BullMQ queue name. Must be non-empty. |
86
+ | `identifier` | `string` | - | Scoped-logging identifier. |
87
+ | `role` | `TBullQueueRole` (`'queue' \| 'worker'`) | - | `'queue'` initializes a producer; `'worker'` initializes a consumer. |
88
+ | `redisConnection` | `IRedisHelper` | - | Connection backend. The helper calls `duplicateClient()` on it - never a raw ioredis client. |
89
+ | `numberOfWorker` | `number` | `1` | Worker concurrency (BullMQ `Worker` `concurrency` option). Ignored for `role: 'queue'`. |
90
+ | `lockDuration` | `number` | `5400000` (90 minutes) | Job lock duration in milliseconds. Ignored for `role: 'queue'`. |
91
+ | `onWorkerData` | `(job: Job<TQueueElement, TQueueResult>) => Promise<any>` | - | Job processor. If omitted, the worker logs `id`, `name`, `data` at info level and resolves `undefined`. |
92
+ | `onWorkerDataCompleted` | `(job, result) => Promise<void>` | - | Fired on the BullMQ `Worker` `'completed'` event. |
93
+ | `onWorkerDataFail` | `(job \| undefined, error: Error) => Promise<void>` | - | Fired on the BullMQ `Worker` `'failed'` event. `job` may be `undefined`. |
94
+
95
+ > [!IMPORTANT]
96
+ > Pass an `IRedisHelper` instance to `redisConnection`, **not** the raw ioredis client. `BullMQHelper` calls `redisConnection.duplicateClient()` internally to get a dedicated connection per role - one shared Redis helper can back any number of queues and workers.
97
+
98
+ ### Configuration lifecycle
99
+
100
+ The constructor calls `configure()`, which switches on `role`:
101
+
102
+ | `role` | Method called | Result |
103
+ |--------|---------------|--------|
104
+ | `'queue'` | `configureQueue()` | Builds `this.queue` (BullMQ `Queue`); attaches an `'error'` listener |
105
+ | `'worker'` | `configureWorker()` | Builds `this.worker` (BullMQ `Worker`); attaches `'completed'`, `'failed'`, `'error'` listeners |
106
+ | missing / other | neither | Logs `'Invalid client role to configure'` and returns - **does not throw** |
107
+
108
+ `configureQueue()` and `configureWorker()` each guard on `queueName`: if it is falsy, the method logs `'Invalid queue name'` / `'Invalid worker name'` and returns without constructing anything - `this.queue` / `this.worker` stay `undefined`. Neither path throws; a misconfigured helper fails later, when you call `.queue.add(...)` or `.worker` on the `undefined` property.
109
+
110
+ > [!NOTE]
111
+ > An `'error'` event with no listener is re-thrown by Node's `EventEmitter` and crashes the process. `BullMQHelper` always attaches an `'error'` listener to both `queue` and `worker` so a transient Redis error is logged instead of taking the process down.
112
+
113
+ `onWorkerDataCompleted` and `onWorkerDataFail` run through an internal hook wrapper that absorbs both synchronous throws and rejected promises, logging them instead of propagating - a broken callback cannot crash the worker or block the next job.
114
+
115
+ ### Cluster queue name wrapping
116
+
117
+ - **Detects cluster mode.** `resolveQueueName()` checks whether `redisConnection.getClient()` is an ioredis `Cluster` instance.
118
+ - **Wraps the name in a hash-tag.** When the connection is a cluster and `queueName` does not already start with `{`, the name is wrapped: `email-queue` becomes `{email-queue}`.
119
+ - **Why.** This forces BullMQ's queue keys onto a single cluster slot, which BullMQ requires for cluster mode. You do not need to wrap the name yourself.
120
+
121
+ ### Default job options
122
+
123
+ Every job added via `queue.add()` is created with:
124
+
125
+ ```typescript
126
+ defaultJobOptions: {
127
+ removeOnComplete: true,
128
+ removeOnFail: true,
129
+ }
130
+ ```
131
+
132
+ BullMQ does not retain job records after they finish (success or failure).
133
+
134
+ ### Redis Cluster setup
135
+
136
+ `RedisClusterHelper` does not set `maxRetriesPerRequest: null` automatically - set it inside `clusterOptions.redisOptions` yourself, since BullMQ requires it:
137
+
138
+ ```typescript
139
+ import { RedisClusterHelper } from '@venizia/ignis-helpers';
140
+ import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
141
+
142
+ const redisHelper = new RedisClusterHelper({
143
+ name: 'cluster-redis',
144
+ nodes: [
145
+ { host: 'node1.redis.example.com', port: 6379 },
146
+ { host: 'node2.redis.example.com', port: 6379 },
147
+ { host: 'node3.redis.example.com', port: 6379 },
148
+ ],
149
+ clusterOptions: {
150
+ enableReadyCheck: true,
151
+ scaleReads: 'slave',
152
+ redisOptions: {
153
+ password: 'your-password',
154
+ tls: {},
155
+ maxRetriesPerRequest: null, // required by BullMQ
156
+ },
157
+ },
158
+ });
159
+
160
+ const worker = BullMQHelper.newInstance({
161
+ queueName: 'my-queue',
162
+ identifier: 'cluster-worker',
163
+ role: 'worker',
164
+ redisConnection: redisHelper,
165
+ onWorkerData: async job => {
166
+ // process job
167
+ },
168
+ });
169
+ ```
170
+
171
+ ### close()
172
+
173
+ ```typescript
174
+ await producer.close();
175
+ await consumer.close();
176
+ ```
177
+
178
+ Calls `worker?.close()` then `queue?.close()` in sequence - both run even if the first fails, so a failing worker close never leaks the queue's Redis connection. If either fails, `close()` throws a single `ApplicationError` aggregating both failure messages after both close attempts have run.
179
+
180
+ ### API summary
181
+
182
+ | Member | Returns | Description |
183
+ |--------|---------|--------------|
184
+ | `static newInstance(opts)` | `BullMQHelper` | Factory, equivalent to `new BullMQHelper(opts)`. |
185
+ | `configure()` | `void` | Dispatches to `configureQueue()` or `configureWorker()` based on `role`. Called automatically by the constructor. |
186
+ | `configureQueue()` | `void` | Builds the BullMQ `Queue`. |
187
+ | `configureWorker()` | `void` | Builds the BullMQ `Worker`. |
188
+ | `close()` | `Promise<void>` | Gracefully closes worker and queue connections. |
189
+ | `queue` | `Queue<TQueueElement, TQueueResult>` | Present when `role: 'queue'`. |
190
+ | `worker` | `Worker<TQueueElement, TQueueResult>` | Present when `role: 'worker'`. |
191
+
192
+ ## SequentialQueueHelper
193
+
194
+ `Source ->` [`packages/helpers/src/modules/queue/internal/sequential/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/helper.ts)
195
+
196
+ > [!NOTE]
197
+ > `QueueHelper` is a deprecated alias for `SequentialQueueHelper`, kept for backward compatibility. Prefer `SequentialQueueHelper` in new code.
198
+
199
+ A generator-driven, in-memory queue that processes one element at a time within a single process - no Redis, no persistence.
200
+
201
+ ### Constructor
202
+
203
+ ```typescript
204
+ import { SequentialQueueHelper } from '@venizia/ignis-helpers';
205
+
206
+ const queue = new SequentialQueueHelper<string>({
207
+ identifier: 'task-queue',
208
+ autoDispatch: true,
209
+ onMessage: async ({ identifier, queueElement }) => {
210
+ console.log(`[${identifier}] Processing:`, queueElement.payload);
211
+ },
212
+ onDataEnqueue: async ({ identifier, queueElement }) => {
213
+ console.log(`[${identifier}] Enqueued:`, queueElement.payload);
214
+ },
215
+ onDataDequeue: async ({ identifier, queueElement }) => {
216
+ console.log(`[${identifier}] Dequeued:`, queueElement.payload);
217
+ },
218
+ onStateChange: async ({ identifier, from, to }) => {
219
+ console.log(`[${identifier}] State: ${from} -> ${to}`);
220
+ },
221
+ });
222
+ ```
223
+
224
+ ### IQueueCallback
225
+
226
+ `IQueueCallback<TElementPayload>`, constructed together with `identifier: string`.
227
+
228
+ | Option | Type | Default | Description |
229
+ |--------|------|---------|-------------|
230
+ | `autoDispatch` | `boolean` | `true` | When `true`, `enqueue()` automatically calls `nextMessage()`. |
231
+ | `onMessage` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Message processor. If omitted, the internal generator logs a warning and never initializes - nothing is ever processed. |
232
+ | `onDataEnqueue` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Fired after an element is pushed onto `storage`. |
233
+ | `onDataDequeue` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Fired after an element is shifted off `storage`. |
234
+ | `onStateChange` | `(opts: { identifier: string; from: TQueueStatus; to: TQueueStatus }) => ValueOrPromise<void>` | - | Fired on every state transition. |
235
+
236
+ All four hooks run through an internal wrapper that absorbs synchronous throws and promise rejections, logging them instead - a broken callback cannot break the state machine.
237
+
238
+ ### TQueueElement
239
+
240
+ ```typescript
241
+ type TQueueElement<T> = { isLocked: boolean; payload: T };
242
+ ```
243
+
244
+ `isLocked` marks whether the head element is currently being handed to `onMessage`; it is distinct from the queue-level `lock()`/`unlock()` state below.
245
+
246
+ ### QueueStatuses and the state machine
247
+
248
+ ```
249
+ WAITING ──enqueue──> PROCESSING ──done──> WAITING
250
+ │ │
251
+ └──lock()──> LOCKED <─┘
252
+
253
+ unlock()──> WAITING
254
+
255
+ settle()──> SETTLED (terminal)
256
+ ```
257
+
258
+ | State | Value | Description |
259
+ |-------|-------|-------------|
260
+ | `QueueStatuses.WAITING` | `'000_WAITING'` | Idle, ready to process the next element. |
261
+ | `QueueStatuses.PROCESSING` | `'100_PROCESSING'` | Currently running `onMessage` for the head element. |
262
+ | `QueueStatuses.LOCKED` | `'200_LOCKED'` | Paused. Elements can still be enqueued; nothing processes until `unlock()`. |
263
+ | `QueueStatuses.SETTLED` | `'300_SETTLED'` | Terminal. No further elements accepted. |
264
+
265
+ - **Why numeric prefixes.** `'000_...'` .. `'300_...'` make `state >= QueueStatuses.LOCKED` and `state > QueueStatuses.LOCKED` valid string comparisons - `lock()` and `unlock()` use exactly that to guard against re-entering from an invalid state.
266
+ - **Validate an arbitrary string** with `QueueStatuses.isValid(value)`.
267
+
268
+ ### Processing loop
269
+
270
+ Internally, `_messageListener()` is a generator that loops `while (true) { yield this.handleMessage(); }`. `nextMessage()` calls `generator.next()` only when `state === WAITING`; any other state logs a warning and returns without advancing. `handleMessage()`:
271
+
272
+ 1. Reads the head element (`getElementAt(0)`); returns early if empty or already `isLocked`.
273
+ 2. Transitions `WAITING -> PROCESSING` (skipped if already `LOCKED`/`SETTLED`).
274
+ 3. Marks the head `isLocked = true`, adds it to `processingEvents`, and awaits `onMessage`.
275
+ 4. Dequeues the completed element and removes it from `processingEvents`.
276
+ 5. Transitions back to `WAITING` (skipped if `LOCKED`/`SETTLED`).
277
+ 6. If `storage` is now empty and `settle()` was requested, transitions to `SETTLED`; otherwise calls `nextMessage()` again to continue the loop.
278
+
279
+ A failing `onMessage` handler is logged and swallowed by the same hook wrapper described above - the element is still dequeued and the loop continues. `onMessage` owns its own retry policy; the queue itself never retries.
280
+
281
+ ### Methods
282
+
283
+ | Method | Returns | Description |
284
+ |--------|---------|--------------|
285
+ | `enqueue(payload)` | `Promise<void>` | Pushes an element onto `storage`. No-ops (logs an error) if `state === SETTLED` or settle was requested. No-ops silently if `payload` is falsy. Calls `nextMessage()` when `autoDispatch` is `true`. |
286
+ | `dequeue()` | `TQueueElement<T> \| undefined` | Shifts the head element off `storage` and fires `onDataDequeue`. |
287
+ | `nextMessage()` | `void` | Advances the generator. Only effective when `state === WAITING`; otherwise logs a warning and returns. |
288
+ | `lock()` | `void` | Transitions to `LOCKED`. No-ops (logs an error) if already `LOCKED` or `SETTLED`. |
289
+ | `unlock(opts)` | `void` | `opts: { shouldProcessNextElement?: boolean }` (default `true`). Transitions to `WAITING`; no-ops (logs an error) if `state > LOCKED` (i.e. already `SETTLED`). Calls `nextMessage()` unless `shouldProcessNextElement: false`. |
290
+ | `settle()` | `void` | Sets the settle flag. Transitions immediately to `SETTLED` unless currently `PROCESSING`, in which case `handleMessage()` finishes the transition once `storage` empties. |
291
+ | `isSettled()` | `boolean` | `true` when `state === SETTLED` and `storage` is empty. |
292
+ | `close()` | `void` | Calls `settle()`, then terminates the generator via `generator.return(...)`. |
293
+ | `getElementAt(position)` | `TQueueElement<T>` | Peek at an element by index (`storage[position]`). |
294
+ | `getState()` | `TQueueStatus` | Current state. |
295
+ | `getTotalEvent()` | `number` | Total elements ever enqueued (monotonic counter, not decremented on dequeue). |
296
+ | `getProcessingEvents()` | `Set<TQueueElement<T>>` | The element(s) currently mid-`onMessage`. |
297
+
298
+ > [!NOTE]
299
+ > Once `SETTLED`, `enqueue()` permanently rejects new elements - there is no way to reopen a settled queue. Construct a new `SequentialQueueHelper` instance instead.
300
+
301
+ ## HfQueueHelper
302
+
303
+ `Source ->` [`packages/helpers/src/modules/queue/internal/hf/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/hf/helper.ts)
304
+
305
+ - **O(1) FIFO, not a job queue.** A generic, high-frequency, single-consumer primitive for enqueue, dequeue, and cancel - backed by an array plus a moving head index (no `Array.shift()`, which is O(n)).
306
+ - **No callbacks, no state machine, not thread-safe.** It is the low-level queue the pool helper's waiter list is built on, not a job-processing API.
307
+ - **Reach for `SequentialQueueHelper` instead** unless you specifically need a bare FIFO with cancellable entries.
308
+
309
+ ```typescript
310
+ import { HfQueueHelper } from '@venizia/ignis-helpers';
311
+
312
+ const queue = new HfQueueHelper<{ task: string }>({ scope: 'my-waiter-queue' });
313
+
314
+ const node = queue.enqueue({ value: { task: 'resize-image' } });
315
+ queue.cancel({ node }); // O(1) removal without scanning
316
+
317
+ const next = queue.dequeue(); // null when empty
318
+ const remaining = queue.drain(); // remove + return every live value, emptying the queue
319
+ ```
320
+
321
+ | Member | Returns | Description |
322
+ |--------|---------|--------------|
323
+ | constructor | - | `opts?: { scope?: string }` - unlike other queue helpers, this takes `scope`, not `identifier`. |
324
+ | `size` | `number` (getter) | Count of live (enqueued, not yet dequeued or cancelled) entries. |
325
+ | `enqueue(opts)` | `IHfQueueNode<T>` | `opts: { value: T }`. Appends and returns a node handle. |
326
+ | `dequeue()` | `T \| null` | Removes and returns the next live value in FIFO order, skipping cancelled nodes. `null` when empty. |
327
+ | `cancel(opts)` | `void` | `opts: { node: IHfQueueNode<T> }`. Marks a queued node cancelled; idempotent. |
328
+ | `drain()` | `T[]` | Removes and returns every remaining live value in FIFO order, emptying the queue. |
329
+
330
+ Consumed entries are compacted out of the backing array once the consumed prefix exceeds 256 entries and covers at least half the array, keeping every operation amortized O(1).
331
+
332
+ ## MQTTClientHelper
333
+
334
+ `Source ->` [`packages/helpers/src/modules/queue/mqtt/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/mqtt/helper.ts)
335
+
336
+ ### Constructor
337
+
338
+ ```typescript
339
+ import { MQTTClientHelper } from '@venizia/ignis-helpers/mqtt';
340
+
341
+ const client = new MQTTClientHelper({
342
+ identifier: 'iot-gateway',
343
+ url: 'mqtt://broker.example.com:1883',
344
+ options: { keepalive: 60 },
345
+ onConnect: () => {
346
+ client.subscribe({ topics: ['devices/+/status'] });
347
+ },
348
+ onMessage: ({ topic, message }) => {
349
+ console.log(topic, message.toString());
350
+ },
351
+ onDisconnect: () => console.log('Disconnected'),
352
+ onError: error => console.error('Connection error:', error.message),
353
+ onClose: error => {
354
+ if (error) console.error('Connection closed with error:', error);
355
+ },
356
+ });
357
+ ```
358
+
359
+ The constructor calls `configure()` automatically, which connects via `mqtt.connect(url, options)`. Calling `configure()` again on an already-connected client is a no-op (logs and returns).
360
+
361
+ ### IMQTTClientOptions
362
+
363
+ | Option | Type | Default | Description |
364
+ |--------|------|---------|-------------|
365
+ | `identifier` | `string` | - | Scoped-logging identifier. |
366
+ | `url` | `string` | - | MQTT broker URL. Must be non-empty - throws an `ApplicationError` (status 500) if empty. |
367
+ | `options` | `mqtt.IClientOptions` | - | MQTT.js client options (username, password, keepalive, ...). |
368
+ | `onMessage` | `(opts: { topic: string; message: Buffer }) => void` | - | Required. Message handler for all subscribed topics. |
369
+ | `onConnect` | `() => void` | - | Fired on the `'connect'` event. |
370
+ | `onDisconnect` | `() => void` | - | Fired on the `'disconnect'` event. |
371
+ | `onError` | `(error: Error) => void` | - | Fired on the `'error'` event. |
372
+ | `onClose` | `(error?: Error) => void` | - | Fired on the `'close'` event. |
373
+
374
+ > [!NOTE]
375
+ > At connect time, `MQTTClientHelper` logs the broker `url` through `redactUrlCredentials()` and `options` through `redactSecrets()`. A password embedded in the URL (`mqtts://user:hunter2@broker:8883`) never reaches the log - only `mqtts://user:[REDACTED]@broker:8883` does.
376
+
377
+ ### Methods
378
+
379
+ | Method | Returns | Description |
380
+ |--------|---------|--------------|
381
+ | `configure()` | `void` | Connects to the broker and wires `connect`/`disconnect`/`message`/`error`/`close` listeners. Called automatically by the constructor; a no-op if already connected. |
382
+ | `getClient()` | `mqtt.MqttClient \| undefined` | The underlying MQTT.js client. |
383
+ | `subscribe(opts)` | `Promise<string[]>` | `opts: { topics: string[] }`. Rejects with an `ApplicationError` (status 400) if the client is not connected. |
384
+ | `publish(opts)` | `Promise<{ topic, message }>` | `opts: { topic: string; message: string \| Buffer }`. Rejects with an `ApplicationError` (status 400) if the client is not connected. |
385
+ | `close(opts?)` | `Promise<void>` | `opts?: { isForce?: boolean }` (default `false`). Ends the client via `client.end(isForce, undefined, callback)`, resolving once the driver reports the connection ended. Safe to call more than once. |
386
+
387
+ ## Troubleshooting
388
+
389
+ ### "Invalid queue name" / "Invalid worker name"
390
+
391
+ **Cause:** `queueName` is empty when `BullMQHelper` configures a queue or worker. This is a **logged error, not a thrown exception** - `this.queue` / `this.worker` are simply never assigned.
392
+
393
+ **Fix:** Provide a non-empty `queueName`:
394
+
395
+ ```typescript
396
+ // Wrong - this.queue stays undefined, calling .queue.add(...) throws later
397
+ new BullMQHelper({ queueName: '', role: 'queue', identifier: 'x', redisConnection: redis });
398
+
399
+ // Correct
400
+ new BullMQHelper({ queueName: 'my-email-queue', role: 'queue', identifier: 'x', redisConnection: redis });
401
+ ```
402
+
403
+ ### "Invalid client role to configure"
404
+
405
+ **Cause:** `role` is missing or not one of `'queue'` / `'worker'`. Logged, not thrown.
406
+
407
+ **Fix:**
408
+
409
+ ```typescript
410
+ new BullMQHelper({ role: 'worker', queueName: 'q', identifier: 'x', redisConnection: redis, /* ... */ });
411
+ ```
412
+
413
+ ### "Invalid url to configure mqtt client!"
414
+
415
+ **Cause:** `url` is empty when constructing `MQTTClientHelper`. **This one throws** an `ApplicationError` (status 500).
416
+
417
+ **Fix:**
418
+
419
+ ```typescript
420
+ new MQTTClientHelper({ url: 'mqtt://localhost:1883', identifier: 'x', options: {}, onMessage: () => {} });
421
+ ```
422
+
423
+ ### "MQTT Client is not available to subscribe topic!" / "... to publish message!"
424
+
425
+ **Cause:** `subscribe()` or `publish()` called before the client finished connecting, or after it disconnected. Rejects with an `ApplicationError` (status 400).
426
+
427
+ **Fix:** Call `subscribe()`/`publish()` from inside `onConnect`, or check `client.getClient()?.connected` first.
428
+
429
+ ### Elements not processing in `SequentialQueueHelper`
430
+
431
+ **Checklist:**
432
+ - `onMessage` must be provided - without it, the generator never initializes and nothing is ever processed.
433
+ - Check `getState()` - a `LOCKED` queue accepts `enqueue()` but never processes; call `unlock({ shouldProcessNextElement: true })`.
434
+ - Check `autoDispatch` - if `false`, call `nextMessage()` manually after each `enqueue()`.
435
+ - Check `isSettled()` - a settled queue rejects new elements; construct a new instance.
436
+
437
+ ## Import Reference
438
+
439
+ ```typescript
440
+ import {
441
+ SequentialQueueHelper,
442
+ QueueHelper, // deprecated alias for SequentialQueueHelper
443
+ QueueStatuses,
444
+ HfQueueHelper,
445
+ } from '@venizia/ignis-helpers';
446
+
447
+ import type {
448
+ TQueueStatus,
449
+ TQueueElement,
450
+ IQueueCallback,
451
+ IHfQueueNode,
452
+ TBullQueueRole,
453
+ } from '@venizia/ignis-helpers';
454
+
455
+ import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
456
+
457
+ import { MQTTClientHelper } from '@venizia/ignis-helpers/mqtt';
458
+ import type { IMQTTClientOptions } from '@venizia/ignis-helpers/mqtt';
459
+ ```
460
+
461
+ ## See also
462
+
463
+ - [Queue overview](/extensions/helpers/queue/) - introduction and the most common tasks
464
+ - [Redis Helper - Full Reference](/extensions/helpers/redis/reference) - `IRedisHelper`, `duplicateClient()`, and the `maxRetriesPerRequest` defaults per topology
465
+ - [Kafka Helpers](/extensions/helpers/kafka/) - the fourth queueing backend
466
+ - [BullMQ documentation](https://docs.bullmq.io/) - underlying job queue library
467
+ - [MQTT.js](https://github.com/mqttjs/MQTT.js) - underlying MQTT client library