@venizia/ignis-docs 0.2.0 → 0.2.1-1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (174) hide show
  1. package/README.md +14 -14
  2. package/content/best-practices/api-usage-examples.md +39 -19
  3. package/content/best-practices/architectural-patterns.md +24 -13
  4. package/content/best-practices/architecture-decisions.md +29 -16
  5. package/content/best-practices/code-style-standards/advanced-patterns.md +43 -31
  6. package/content/best-practices/code-style-standards/constants-configuration.md +19 -4
  7. package/content/best-practices/code-style-standards/control-flow.md +33 -1
  8. package/content/best-practices/code-style-standards/documentation.md +28 -8
  9. package/content/best-practices/code-style-standards/function-patterns.md +15 -4
  10. package/content/best-practices/code-style-standards/index.md +7 -2
  11. package/content/best-practices/code-style-standards/naming-conventions.md +27 -3
  12. package/content/best-practices/code-style-standards/route-definitions.md +9 -5
  13. package/content/best-practices/code-style-standards/tooling.md +14 -1
  14. package/content/best-practices/code-style-standards/type-safety.md +39 -6
  15. package/content/best-practices/common-pitfalls.md +59 -35
  16. package/content/best-practices/contribution-workflow.md +8 -4
  17. package/content/best-practices/data-modeling.md +78 -62
  18. package/content/best-practices/deployment-strategies.md +56 -19
  19. package/content/best-practices/error-handling.md +247 -153
  20. package/content/best-practices/index.md +6 -0
  21. package/content/best-practices/performance-optimization.md +26 -13
  22. package/content/best-practices/security-guidelines.md +35 -9
  23. package/content/best-practices/testing-strategies.md +7 -2
  24. package/content/best-practices/troubleshooting-tips.md +27 -17
  25. package/content/extensions/components/api-reference.md +109 -323
  26. package/content/extensions/components/authentication/api.md +489 -602
  27. package/content/extensions/components/authentication/errors.md +135 -498
  28. package/content/extensions/components/authentication/index.md +89 -801
  29. package/content/extensions/components/authentication/usage.md +231 -955
  30. package/content/extensions/components/authorization/api.md +991 -644
  31. package/content/extensions/components/authorization/errors.md +204 -208
  32. package/content/extensions/components/authorization/getting-started.md +227 -0
  33. package/content/extensions/components/authorization/index.md +88 -795
  34. package/content/extensions/components/authorization/usage.md +219 -528
  35. package/content/extensions/components/health-check.md +77 -243
  36. package/content/extensions/components/index.md +24 -90
  37. package/content/extensions/components/mail/api.md +553 -285
  38. package/content/extensions/components/mail/errors.md +76 -62
  39. package/content/extensions/components/mail/index.md +111 -463
  40. package/content/extensions/components/mail/usage.md +139 -173
  41. package/content/extensions/components/request-tracker.md +70 -173
  42. package/content/extensions/components/socket-io/api.md +462 -785
  43. package/content/extensions/components/socket-io/errors.md +49 -51
  44. package/content/extensions/components/socket-io/index.md +69 -372
  45. package/content/extensions/components/socket-io/usage.md +212 -105
  46. package/content/extensions/components/static-asset/api.md +461 -141
  47. package/content/extensions/components/static-asset/errors.md +121 -53
  48. package/content/extensions/components/static-asset/index.md +84 -606
  49. package/content/extensions/components/static-asset/usage.md +184 -299
  50. package/content/extensions/components/template/index.md +3 -3
  51. package/content/extensions/components/websocket/api.md +311 -404
  52. package/content/extensions/components/websocket/errors.md +47 -56
  53. package/content/extensions/components/websocket/index.md +75 -407
  54. package/content/extensions/components/websocket/usage.md +120 -338
  55. package/content/extensions/helpers/cron/index.md +52 -160
  56. package/content/extensions/helpers/crypto/index.md +67 -480
  57. package/content/extensions/helpers/crypto/reference.md +528 -0
  58. package/content/extensions/helpers/env/index.md +64 -178
  59. package/content/extensions/helpers/error/index.md +286 -196
  60. package/content/extensions/helpers/index.md +61 -47
  61. package/content/extensions/helpers/inversion/index.md +76 -550
  62. package/content/extensions/helpers/inversion/reference.md +530 -0
  63. package/content/extensions/helpers/kafka/admin.md +23 -3
  64. package/content/extensions/helpers/kafka/compile-binary.md +83 -52
  65. package/content/extensions/helpers/kafka/consumer.md +65 -28
  66. package/content/extensions/helpers/kafka/examples.md +46 -253
  67. package/content/extensions/helpers/kafka/index.md +55 -617
  68. package/content/extensions/helpers/kafka/producer.md +156 -22
  69. package/content/extensions/helpers/kafka/schema-registry.md +59 -79
  70. package/content/extensions/helpers/logger/hf-logger.md +220 -0
  71. package/content/extensions/helpers/logger/index.md +78 -552
  72. package/content/extensions/helpers/logger/pino.md +105 -0
  73. package/content/extensions/helpers/logger/reference.md +937 -0
  74. package/content/extensions/helpers/network/api.md +276 -195
  75. package/content/extensions/helpers/network/index.md +87 -524
  76. package/content/extensions/helpers/queue/index.md +80 -897
  77. package/content/extensions/helpers/queue/reference.md +494 -0
  78. package/content/extensions/helpers/redis/index.md +86 -640
  79. package/content/extensions/helpers/redis/reference.md +784 -0
  80. package/content/extensions/helpers/secrets/index.md +136 -0
  81. package/content/extensions/helpers/socket-io/api.md +325 -204
  82. package/content/extensions/helpers/socket-io/index.md +79 -429
  83. package/content/extensions/helpers/storage/api.md +584 -464
  84. package/content/extensions/helpers/storage/index.md +80 -573
  85. package/content/extensions/helpers/types/index.md +75 -495
  86. package/content/extensions/helpers/types/reference.md +689 -0
  87. package/content/extensions/helpers/uid/index.md +176 -189
  88. package/content/extensions/helpers/websocket/api.md +366 -214
  89. package/content/extensions/helpers/websocket/index.md +68 -503
  90. package/content/extensions/helpers/worker-thread/index.md +62 -396
  91. package/content/extensions/helpers/worker-thread/reference.md +428 -0
  92. package/content/extensions/index.md +38 -39
  93. package/content/extensions/src-details/mcp-server.md +96 -548
  94. package/content/guides/core-concepts/persistent/datasources.md +2 -2
  95. package/content/guides/core-concepts/persistent/index.md +5 -1
  96. package/content/guides/core-concepts/persistent/models.md +1 -1
  97. package/content/guides/core-concepts/persistent/pglite.md +218 -0
  98. package/content/guides/core-concepts/persistent/postgres-drivers.md +21 -15
  99. package/content/guides/core-concepts/persistent/search-meilisearch.md +22 -12
  100. package/content/guides/core-concepts/persistent/search-typesense.md +28 -16
  101. package/content/guides/core-concepts/persistent/sqlite.md +296 -0
  102. package/content/guides/core-concepts/persistent/transactions.md +12 -11
  103. package/content/guides/core-concepts/secrets-vault.md +177 -0
  104. package/content/guides/core-concepts/services.md +1 -1
  105. package/content/guides/get-started/5-minute-quickstart.md +59 -206
  106. package/content/guides/get-started/philosophy.md +135 -670
  107. package/content/guides/get-started/setup.md +53 -74
  108. package/content/guides/migrations/redis-helpers-migration.md +5 -4
  109. package/content/guides/migrations/scoped-rbac-migration.md +5 -5
  110. package/content/guides/migrations/unified-connectors-migration.md +8 -7
  111. package/content/guides/tutorials/ecommerce-api.md +3 -8
  112. package/content/guides/tutorials/realtime-chat.md +1 -1
  113. package/content/references/base/application.md +64 -22
  114. package/content/references/base/components.md +3 -3
  115. package/content/references/base/connectors.md +79 -136
  116. package/content/references/base/controllers.md +12 -12
  117. package/content/references/base/datasources-reference.md +600 -0
  118. package/content/references/base/datasources.md +84 -444
  119. package/content/references/base/dependency-injection.md +25 -46
  120. package/content/references/base/filter-system/application-usage.md +95 -123
  121. package/content/references/base/filter-system/array-operators.md +33 -43
  122. package/content/references/base/filter-system/comparison-operators.md +55 -63
  123. package/content/references/base/filter-system/default-filter.md +144 -350
  124. package/content/references/base/filter-system/fields-order-pagination.md +116 -148
  125. package/content/references/base/filter-system/index.md +114 -253
  126. package/content/references/base/filter-system/json-filtering.md +52 -181
  127. package/content/references/base/filter-system/list-operators.md +28 -44
  128. package/content/references/base/filter-system/logical-operators.md +69 -114
  129. package/content/references/base/filter-system/null-operators.md +41 -98
  130. package/content/references/base/filter-system/pattern-matching.md +48 -51
  131. package/content/references/base/filter-system/quick-reference.md +93 -196
  132. package/content/references/base/filter-system/range-operators.md +22 -38
  133. package/content/references/base/filter-system/tips.md +70 -133
  134. package/content/references/base/filter-system/use-cases.md +173 -232
  135. package/content/references/base/grpc-controllers.md +53 -17
  136. package/content/references/base/index.md +5 -3
  137. package/content/references/base/middlewares.md +43 -28
  138. package/content/references/base/models-reference.md +886 -0
  139. package/content/references/base/models.md +81 -1452
  140. package/content/references/base/providers.md +8 -8
  141. package/content/references/base/repositories/advanced.md +281 -419
  142. package/content/references/base/repositories/index.md +84 -644
  143. package/content/references/base/repositories/mixins.md +25 -21
  144. package/content/references/base/repositories/relations.md +194 -375
  145. package/content/references/base/repositories/soft-deletable.md +67 -55
  146. package/content/references/base/secrets.md +267 -0
  147. package/content/references/base/services.md +8 -6
  148. package/content/references/configuration/environment-variables.md +73 -21
  149. package/content/references/configuration/index.md +51 -31
  150. package/content/references/index.md +1 -1
  151. package/content/references/quick-reference.md +3 -16
  152. package/content/references/utilities/crypto.md +35 -76
  153. package/content/references/utilities/date.md +33 -73
  154. package/content/references/utilities/duration.md +85 -0
  155. package/content/references/utilities/index.md +6 -2
  156. package/content/references/utilities/jsx-reference.md +298 -0
  157. package/content/references/utilities/jsx.md +82 -525
  158. package/content/references/utilities/module.md +81 -61
  159. package/content/references/utilities/parse.md +34 -64
  160. package/content/references/utilities/performance.md +33 -58
  161. package/content/references/utilities/promise.md +28 -62
  162. package/content/references/utilities/request.md +58 -218
  163. package/content/references/utilities/retry.md +139 -0
  164. package/content/references/utilities/schema.md +43 -137
  165. package/content/references/utilities/statuses-reference.md +361 -0
  166. package/content/references/utilities/statuses.md +63 -667
  167. package/dist/mcp-server/index.js +0 -0
  168. package/dist/mcp-server/tools/github/list-project-files.tool.d.ts +1 -1
  169. package/dist/mcp-server/tools/github/list-project-files.tool.d.ts.map +1 -1
  170. package/dist/mcp-server/tools/github/list-project-files.tool.js +3 -3
  171. package/dist/mcp-server/tools/github/search-code.tool.js +1 -1
  172. package/dist/mcp-server/tools/github/verify-dependencies.tool.js +1 -1
  173. package/dist/mcp-server/tools/github/view-source-file.tool.js +1 -1
  174. package/package.json +24 -23
@@ -0,0 +1,494 @@
1
+ ---
2
+ title: Queue - Full Reference
3
+ description: Complete reference for BullMQHelper, SequentialQueueHelper, HfQueueHelper, and MQTTClientHelper - every option, method, and state transition
4
+ difficulty: intermediate
5
+ ---
6
+
7
+ # Queue - Full Reference
8
+
9
+ Exhaustive reference for the Queue helper family. For a readable introduction and the most common tasks, start with the [Queue overview](/extensions/helpers/queue/).
10
+
11
+ **Files:**
12
+
13
+ - [`packages/helpers/src/modules/queue/bullmq/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/bullmq/helper.ts) - `BullMQHelper`
14
+ - [`packages/helpers/src/modules/queue/internal/sequential/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/helper.ts) - `SequentialQueueHelper`
15
+ - [`packages/helpers/src/modules/queue/internal/sequential/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/types.ts) - `QueueStatuses`, `TQueueStatus`, `TQueueElement`, `IQueueCallback`
16
+ - [`packages/helpers/src/modules/queue/internal/hf/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/hf/helper.ts) - `HfQueueHelper`
17
+ - [`packages/helpers/src/modules/queue/mqtt/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/mqtt/helper.ts) - `MQTTClientHelper`
18
+ - [`packages/helpers/src/modules/queue/common/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/common/types.ts) - `TBullQueueRole`
19
+
20
+ ## Find what you need
21
+
22
+ | You want to | Go to |
23
+ |---|---|
24
+ | Pick a backend and see import paths | [Class Overview](#class-overview) / [Import Paths](#import-paths) |
25
+ | Construct a producer or worker and see every option | [BullMQHelper](#bullmqhelper) |
26
+ | Understand what a bad `role` or empty `queueName` does | [Configuration lifecycle](#configuration-lifecycle) |
27
+ | Run BullMQ against Redis Cluster | [Redis Cluster setup](#redis-cluster-setup) |
28
+ | Construct an in-memory sequential queue and see every option | [SequentialQueueHelper](#sequentialqueuehelper) |
29
+ | Understand the `WAITING`/`PROCESSING`/`LOCKED`/`SETTLED` state machine | [QueueStatuses and the state machine](#queuestatuses-and-the-state-machine) |
30
+ | Reach for a bare O(1) FIFO instead of a job queue | [HfQueueHelper](#hfqueuehelper) |
31
+ | Construct an MQTT client and see every option | [MQTTClientHelper](#mqttclienthelper) |
32
+ | Fix a thrown or logged error message | [Troubleshooting](#troubleshooting) |
33
+ | Copy the full import list | [Import Reference](#import-reference) |
34
+
35
+ ## Class Overview
36
+
37
+ | Class | Extends | Peer dependency | Use case |
38
+ |-------|---------|------------------|----------|
39
+ | `BullMQHelper` | `BaseHelper` | `bullmq` | Redis-backed job queue - durable, multi-worker background processing |
40
+ | `SequentialQueueHelper` (alias `QueueHelper`) | `BaseHelper` | none | Single-process, in-memory, one-at-a-time sequencing |
41
+ | `HfQueueHelper` | `BaseHelper` | none | Generic O(1) FIFO primitive - single-threaded, no callbacks, no persistence |
42
+ | `MQTTClientHelper` | `BaseHelper` | `mqtt` | MQTT broker pub/sub - IoT and lightweight real-time events |
43
+
44
+ Kafka (`KafkaProducerHelper`, `KafkaConsumerHelper`, `KafkaAdminHelper`, `KafkaSchemaRegistryHelper`) is a fourth backend under the same `queue/` module tree. It's documented on its own page - see [Kafka Helpers](/extensions/helpers/kafka/).
45
+
46
+ ## Import Paths
47
+
48
+ `BullMQHelper` and `MQTTClientHelper` live behind sub-path exports. That keeps their peer dependencies (`bullmq`, `mqtt`) from becoming hard dependencies of the base package. `SequentialQueueHelper`, `QueueHelper`, `QueueStatuses`, `HfQueueHelper`, and `TBullQueueRole` ship from the root package instead.
49
+
50
+ ```typescript
51
+ // Root package - no peer dependency
52
+ import { SequentialQueueHelper, QueueHelper, QueueStatuses, HfQueueHelper } from '@venizia/ignis-helpers';
53
+ import type { TQueueStatus, TQueueElement, IQueueCallback, IHfQueueNode, TBullQueueRole } from '@venizia/ignis-helpers';
54
+
55
+ // BullMQ - sub-path export
56
+ import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
57
+
58
+ // MQTT - sub-path export
59
+ import { MQTTClientHelper, type IMQTTClientOptions } from '@venizia/ignis-helpers/mqtt';
60
+ ```
61
+
62
+ ## BullMQHelper
63
+
64
+ `Source ->` [`packages/helpers/src/modules/queue/bullmq/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/bullmq/helper.ts)
65
+
66
+ ### Constructor
67
+
68
+ ```typescript
69
+ import { RedisSingleHelper } from '@venizia/ignis-helpers';
70
+ import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
71
+
72
+ const worker = new BullMQHelper({
73
+ queueName: 'email-queue',
74
+ identifier: 'email-worker',
75
+ role: 'worker',
76
+ redisConnection: redisHelper,
77
+ numberOfWorker: 3,
78
+ lockDuration: 10 * 60 * 1000,
79
+ onWorkerData: async job => {
80
+ console.log(`Processing job ${job.id}:`, job.data);
81
+ return { status: 'sent' };
82
+ },
83
+ onWorkerDataCompleted: async (job, result) => {
84
+ console.log(`Job ${job.id} completed:`, result);
85
+ },
86
+ onWorkerDataFail: async (job, error) => {
87
+ console.error(`Job ${job?.id} failed:`, error.message);
88
+ },
89
+ });
90
+ ```
91
+
92
+ `BullMQHelper.newInstance(opts)` is a static factory equivalent to `new BullMQHelper(opts)`.
93
+
94
+ ### IBullMQOptions
95
+
96
+ `IBullMQOptions<TQueueElement = any, TQueueResult = any>`
97
+
98
+ | Option | Type | Default | Description |
99
+ |--------|------|---------|-------------|
100
+ | `queueName` | `string` | - | BullMQ queue name. Must be non-empty. |
101
+ | `identifier` | `string` | - | Scoped-logging identifier. |
102
+ | `role` | `TBullQueueRole` (`'queue' \| 'worker'`) | - | `'queue'` initializes a producer; `'worker'` initializes a consumer. |
103
+ | `redisConnection` | `IRedisHelper` | - | Connection backend. The helper calls `duplicateClient()` on it - never a raw ioredis client. |
104
+ | `numberOfWorker` | `number` | `1` | Worker concurrency (BullMQ `Worker` `concurrency` option). Ignored for `role: 'queue'`. |
105
+ | `lockDuration` | `number` | `5400000` (90 minutes) | Job lock duration in milliseconds. Ignored for `role: 'queue'`. |
106
+ | `onWorkerData` | `(job: Job<TQueueElement, TQueueResult>) => Promise<any>` | - | Job processor. If omitted, the worker logs `id`, `name`, `data` at info level and resolves `undefined`. |
107
+ | `onWorkerDataCompleted` | `(job, result) => Promise<void>` | - | Fired on the BullMQ `Worker` `'completed'` event. |
108
+ | `onWorkerDataFail` | `(job \| undefined, error: Error) => Promise<void>` | - | Fired on the BullMQ `Worker` `'failed'` event. `job` may be `undefined`. |
109
+
110
+ > [!IMPORTANT]
111
+ > Pass an `IRedisHelper` instance to `redisConnection`, **not** the raw ioredis client. `BullMQHelper` calls `redisConnection.duplicateClient()` internally to get a dedicated connection per role. One shared Redis helper can back any number of queues and workers.
112
+
113
+ ### Configuration lifecycle
114
+
115
+ The constructor calls `configure()`, which switches on `role`:
116
+
117
+ | `role` | Method called | Result |
118
+ |--------|---------------|--------|
119
+ | `'queue'` | `configureQueue()` | Builds `this.queue` (BullMQ `Queue`); attaches an `'error'` listener |
120
+ | `'worker'` | `configureWorker()` | Builds `this.worker` (BullMQ `Worker`); attaches `'completed'`, `'failed'`, `'error'` listeners |
121
+ | missing (falsy) | neither | Logs `'Invalid client role to configure'` and returns - **does not throw** |
122
+ | any other value | neither | **Silent.** No log, no throw. `this.queue` and `this.worker` both stay `undefined` |
123
+
124
+ > [!WARNING]
125
+ > A typo'd `role` (for example `'Worker'` instead of `'worker'`) hits the "any other value" row above - not the "missing" row. Nothing gets logged. The helper looks constructed, but `.queue` and `.worker` are both `undefined` until you call one and hit a `TypeError`.
126
+
127
+ `configureQueue()` and `configureWorker()` each guard on `queueName`. An empty `queueName` logs `'Invalid queue name'` (or `'Invalid worker name'`) and returns - `this.queue` / `this.worker` stay `undefined`. Neither path throws. A misconfigured helper fails later instead, the first time you call `.queue.add(...)` or use `.worker`.
128
+
129
+ > [!NOTE]
130
+ > An `'error'` event with no listener is re-thrown by Node's `EventEmitter` and crashes the process. `BullMQHelper` always attaches an `'error'` listener to both `queue` and `worker`. A transient Redis error gets logged instead of taking the process down.
131
+
132
+ `onWorkerDataCompleted` and `onWorkerDataFail` run through an internal hook wrapper. It absorbs both synchronous throws and rejected promises, logging them instead of propagating. A broken callback can't crash the worker or block the next job.
133
+
134
+ ### Cluster queue name wrapping
135
+
136
+ - **Detects cluster mode.** `resolveQueueName()` checks whether `redisConnection.getClient()` is an ioredis `Cluster` instance.
137
+ - **Wraps the name in a hash-tag.** When the connection is a cluster and `queueName` does not already start with `{`, the name is wrapped: `email-queue` becomes `{email-queue}`.
138
+ - **Why.** This forces BullMQ's queue keys onto a single cluster slot, which BullMQ requires for cluster mode. You do not need to wrap the name yourself.
139
+
140
+ ### Default job options
141
+
142
+ Every job added via `queue.add()` is created with:
143
+
144
+ ```typescript
145
+ defaultJobOptions: {
146
+ removeOnComplete: true,
147
+ removeOnFail: true,
148
+ }
149
+ ```
150
+
151
+ BullMQ does not retain job records after they finish (success or failure).
152
+
153
+ ### Redis Cluster setup
154
+
155
+ `RedisClusterHelper` does not set `maxRetriesPerRequest: null` automatically - set it inside `clusterOptions.redisOptions` yourself, since BullMQ requires it:
156
+
157
+ ```typescript
158
+ import { RedisClusterHelper } from '@venizia/ignis-helpers';
159
+ import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
160
+
161
+ const redisHelper = new RedisClusterHelper({
162
+ name: 'cluster-redis',
163
+ nodes: [
164
+ { host: 'node1.redis.example.com', port: 6379 },
165
+ { host: 'node2.redis.example.com', port: 6379 },
166
+ { host: 'node3.redis.example.com', port: 6379 },
167
+ ],
168
+ clusterOptions: {
169
+ enableReadyCheck: true,
170
+ scaleReads: 'slave',
171
+ redisOptions: {
172
+ password: 'your-password',
173
+ tls: {},
174
+ maxRetriesPerRequest: null, // required by BullMQ
175
+ },
176
+ },
177
+ });
178
+
179
+ const worker = BullMQHelper.newInstance({
180
+ queueName: 'my-queue',
181
+ identifier: 'cluster-worker',
182
+ role: 'worker',
183
+ redisConnection: redisHelper,
184
+ onWorkerData: async job => {
185
+ // process job
186
+ },
187
+ });
188
+ ```
189
+
190
+ ### close()
191
+
192
+ ```typescript
193
+ await producer.close();
194
+ await consumer.close();
195
+ ```
196
+
197
+ Calls `worker?.close()` then `queue?.close()` in sequence. Both run even if the first fails, so a failing worker close never leaks the queue's Redis connection. If either fails, `close()` throws one `ApplicationError` aggregating both failure messages - only after both close attempts have run.
198
+
199
+ ### API summary
200
+
201
+ | Member | Returns | Description |
202
+ |--------|---------|--------------|
203
+ | `static newInstance(opts)` | `BullMQHelper` | Factory, equivalent to `new BullMQHelper(opts)`. |
204
+ | `configure()` | `void` | Dispatches to `configureQueue()` or `configureWorker()` based on `role`. Called automatically by the constructor. |
205
+ | `configureQueue()` | `void` | Builds the BullMQ `Queue`. |
206
+ | `configureWorker()` | `void` | Builds the BullMQ `Worker`. |
207
+ | `close()` | `Promise<void>` | Gracefully closes worker and queue connections. |
208
+ | `queue` | `Queue<TQueueElement, TQueueResult>` | Present when `role: 'queue'`. |
209
+ | `worker` | `Worker<TQueueElement, TQueueResult>` | Present when `role: 'worker'`. |
210
+
211
+ ## SequentialQueueHelper
212
+
213
+ `Source ->` [`packages/helpers/src/modules/queue/internal/sequential/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/sequential/helper.ts)
214
+
215
+ > [!NOTE]
216
+ > `QueueHelper` is a deprecated alias for `SequentialQueueHelper`, kept for backward compatibility. Prefer `SequentialQueueHelper` in new code.
217
+
218
+ A generator-driven, in-memory queue that processes one element at a time within a single process - no Redis, no persistence.
219
+
220
+ ### Constructor
221
+
222
+ ```typescript
223
+ import { SequentialQueueHelper } from '@venizia/ignis-helpers';
224
+
225
+ const queue = new SequentialQueueHelper<string>({
226
+ identifier: 'task-queue',
227
+ autoDispatch: true,
228
+ onMessage: async ({ identifier, queueElement }) => {
229
+ console.log(`[${identifier}] Processing:`, queueElement.payload);
230
+ },
231
+ onDataEnqueue: async ({ identifier, queueElement }) => {
232
+ console.log(`[${identifier}] Enqueued:`, queueElement.payload);
233
+ },
234
+ onDataDequeue: async ({ identifier, queueElement }) => {
235
+ console.log(`[${identifier}] Dequeued:`, queueElement.payload);
236
+ },
237
+ onStateChange: async ({ identifier, from, to }) => {
238
+ console.log(`[${identifier}] State: ${from} -> ${to}`);
239
+ },
240
+ });
241
+ ```
242
+
243
+ ### IQueueCallback
244
+
245
+ `IQueueCallback<TElementPayload>`, constructed together with `identifier: string`.
246
+
247
+ | Option | Type | Default | Description |
248
+ |--------|------|---------|-------------|
249
+ | `autoDispatch` | `boolean` | `true` | When `true`, `enqueue()` automatically calls `nextMessage()`. |
250
+ | `onMessage` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Message processor. If omitted, the internal generator logs a warning and never initializes - nothing is ever processed. |
251
+ | `onDataEnqueue` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Fired after an element is pushed onto `storage`. |
252
+ | `onDataDequeue` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Fired after an element is shifted off `storage`. |
253
+ | `onStateChange` | `(opts: { identifier: string; from: TQueueStatus; to: TQueueStatus }) => ValueOrPromise<void>` | - | Fired on every state transition. |
254
+
255
+ All four hooks run through an internal wrapper that absorbs synchronous throws and promise rejections, logging them instead. A broken callback can't break the state machine.
256
+
257
+ ### TQueueElement
258
+
259
+ ```typescript
260
+ type TQueueElement<T> = { isLocked: boolean; payload: T };
261
+ ```
262
+
263
+ `isLocked` marks whether the head element is currently being handed to `onMessage`. It is distinct from the queue-level `lock()`/`unlock()` state below.
264
+
265
+ ### QueueStatuses and the state machine
266
+
267
+ ```
268
+ WAITING ──enqueue──> PROCESSING ──done──> WAITING
269
+ │ │
270
+ └──lock()──> LOCKED <─┘
271
+
272
+ unlock()──> WAITING
273
+
274
+ settle()──> SETTLED (terminal)
275
+ ```
276
+
277
+ | State | Value | Description |
278
+ |-------|-------|-------------|
279
+ | `QueueStatuses.WAITING` | `'000_WAITING'` | Idle, ready to process the next element. |
280
+ | `QueueStatuses.PROCESSING` | `'100_PROCESSING'` | Currently running `onMessage` for the head element. |
281
+ | `QueueStatuses.LOCKED` | `'200_LOCKED'` | Paused. Elements can still be enqueued; nothing processes until `unlock()`. |
282
+ | `QueueStatuses.SETTLED` | `'300_SETTLED'` | Terminal. No further elements accepted. |
283
+
284
+ - **Why numeric prefixes.** `'000_...'` through `'300_...'` make `state >= QueueStatuses.LOCKED` and `state > QueueStatuses.LOCKED` valid string comparisons. `lock()` and `unlock()` use exactly that to guard against re-entering from an invalid state.
285
+ - **Validate an arbitrary string** with `QueueStatuses.isValid(value)`.
286
+
287
+ ### Processing loop
288
+
289
+ Internally, `_messageListener()` is a generator that loops `while (true) { yield this.handleMessage(); }`. `nextMessage()` calls `generator.next()` only when `state === WAITING`. Any other state logs a warning and returns without advancing. `handleMessage()`:
290
+
291
+ 1. Reads the head element (`getElementAt(0)`). Returns early if it's empty or already `isLocked`.
292
+ 2. Transitions `WAITING -> PROCESSING` (skipped if already `LOCKED`/`SETTLED`).
293
+ 3. Marks the head `isLocked = true`, adds it to `processingEvents`, and awaits `onMessage`.
294
+ 4. Dequeues the completed element and removes it from `processingEvents`.
295
+ 5. Transitions back to `WAITING` (skipped if `LOCKED`/`SETTLED`).
296
+ 6. Checks `storage`:
297
+
298
+ | `storage` after step 5 | What happens |
299
+ |---|---|
300
+ | Empty, and `settle()` was requested | Transitions to `SETTLED` and stops |
301
+ | Empty, and no `settle()` requested | Stops here. The queue idles until the next `enqueue()` |
302
+ | Not empty | Calls `nextMessage()` to continue the loop immediately |
303
+
304
+ A failing `onMessage` handler is logged and swallowed by the same hook wrapper described above. The element is still dequeued, and the loop continues. `onMessage` owns its own retry policy - the queue itself never retries.
305
+
306
+ ### Methods
307
+
308
+ | Method | Returns | Description |
309
+ |--------|---------|--------------|
310
+ | `enqueue(payload)` | `Promise<void>` | Pushes an element onto `storage`. No-ops (logs an error) if `state === SETTLED` or settle was requested. No-ops silently if `payload` is falsy. Calls `nextMessage()` when `autoDispatch` is `true`. |
311
+ | `dequeue()` | `TQueueElement<T> \| undefined` | Shifts the head element off `storage` and fires `onDataDequeue`. |
312
+ | `nextMessage()` | `void` | Advances the generator. Only effective when `state === WAITING`; otherwise logs a warning and returns. |
313
+ | `lock()` | `void` | Transitions to `LOCKED`. No-ops (logs an error) if already `LOCKED` or `SETTLED`. |
314
+ | `unlock(opts)` | `void` | `opts: { shouldProcessNextElement?: boolean }` (default `true`). Transitions to `WAITING`; no-ops (logs an error) if `state > LOCKED` (i.e. already `SETTLED`). Calls `nextMessage()` unless `shouldProcessNextElement: false`. |
315
+ | `settle()` | `void` | Sets the settle flag. Transitions immediately to `SETTLED` unless currently `PROCESSING`, in which case `handleMessage()` finishes the transition once `storage` empties. |
316
+ | `isSettled()` | `boolean` | `true` when `state === SETTLED` and `storage` is empty. |
317
+ | `close()` | `void` | Calls `settle()`, then terminates the generator via `generator.return(...)`. |
318
+ | `getElementAt(position)` | `TQueueElement<T>` | Peek at an element by index (`storage[position]`). |
319
+ | `getState()` | `TQueueStatus` | Current state. |
320
+ | `getTotalEvent()` | `number` | Total elements ever enqueued (monotonic counter, not decremented on dequeue). |
321
+ | `getProcessingEvents()` | `Set<TQueueElement<T>>` | The element(s) currently mid-`onMessage`. |
322
+
323
+ > [!NOTE]
324
+ > Once `SETTLED`, `enqueue()` permanently rejects new elements - there is no way to reopen a settled queue. Construct a new `SequentialQueueHelper` instance instead.
325
+
326
+ ## HfQueueHelper
327
+
328
+ `Source ->` [`packages/helpers/src/modules/queue/internal/hf/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/hf/helper.ts)
329
+
330
+ - **O(1) FIFO, not a job queue.** A generic, high-frequency, single-consumer primitive for enqueue, dequeue, and cancel. It's backed by an array plus a moving head index - never `Array.shift()`, which is O(n).
331
+ - **No callbacks, no state machine, not thread-safe.** It is the low-level queue the pool helper's waiter list is built on, not a job-processing API.
332
+ - **Reach for `SequentialQueueHelper` instead** unless you specifically need a bare FIFO with cancellable entries.
333
+
334
+ ```typescript
335
+ import { HfQueueHelper } from '@venizia/ignis-helpers';
336
+
337
+ const queue = new HfQueueHelper<{ task: string }>({ scope: 'my-waiter-queue' });
338
+
339
+ const node = queue.enqueue({ value: { task: 'resize-image' } });
340
+ queue.cancel({ node }); // O(1) removal without scanning
341
+
342
+ const next = queue.dequeue(); // null when empty
343
+ const remaining = queue.drain(); // remove + return every live value, emptying the queue
344
+ ```
345
+
346
+ | Member | Returns | Description |
347
+ |--------|---------|--------------|
348
+ | constructor | - | `opts?: { scope?: string }` - unlike other queue helpers, this takes `scope`, not `identifier`. |
349
+ | `size` | `number` (getter) | Count of live (enqueued, not yet dequeued or cancelled) entries. |
350
+ | `enqueue(opts)` | `IHfQueueNode<T>` | `opts: { value: T }`. Appends and returns a node handle. |
351
+ | `dequeue()` | `T \| null` | Removes and returns the next live value in FIFO order, skipping cancelled nodes. `null` when empty. |
352
+ | `cancel(opts)` | `void` | `opts: { node: IHfQueueNode<T> }`. Marks a queued node cancelled; idempotent. |
353
+ | `drain()` | `T[]` | Removes and returns every remaining live value in FIFO order, emptying the queue. |
354
+
355
+ Consumed entries are compacted out of the backing array once the consumed prefix exceeds 256 entries and covers at least half the array. This keeps every operation amortized O(1).
356
+
357
+ ## MQTTClientHelper
358
+
359
+ `Source ->` [`packages/helpers/src/modules/queue/mqtt/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/mqtt/helper.ts)
360
+
361
+ ### Constructor
362
+
363
+ ```typescript
364
+ import { MQTTClientHelper } from '@venizia/ignis-helpers/mqtt';
365
+
366
+ const client = new MQTTClientHelper({
367
+ identifier: 'iot-gateway',
368
+ url: 'mqtt://broker.example.com:1883',
369
+ options: { keepalive: 60 },
370
+ onConnect: () => {
371
+ client.subscribe({ topics: ['devices/+/status'] });
372
+ },
373
+ onMessage: ({ topic, message }) => {
374
+ console.log(topic, message.toString());
375
+ },
376
+ onDisconnect: () => console.log('Disconnected'),
377
+ onError: error => console.error('Connection error:', error.message),
378
+ onClose: error => {
379
+ if (error) console.error('Connection closed with error:', error);
380
+ },
381
+ });
382
+ ```
383
+
384
+ The constructor calls `configure()` automatically, which connects via `mqtt.connect(url, options)`. Calling `configure()` again is a no-op (logs and returns) whenever `this.client` already exists - even if that client later disconnected without you calling `close()`.
385
+
386
+ ### IMQTTClientOptions
387
+
388
+ | Option | Type | Default | Description |
389
+ |--------|------|---------|-------------|
390
+ | `identifier` | `string` | - | Scoped-logging identifier. |
391
+ | `url` | `string` | - | MQTT broker URL. Must be non-empty - throws an `ApplicationError` (status 500) if empty. |
392
+ | `options` | `mqtt.IClientOptions` | - | MQTT.js client options (username, password, keepalive, ...). |
393
+ | `onMessage` | `(opts: { topic: string; message: Buffer }) => void` | - | Required. Message handler for all subscribed topics. |
394
+ | `onConnect` | `() => void` | - | Fired on the `'connect'` event. |
395
+ | `onDisconnect` | `() => void` | - | Fired on the `'disconnect'` event. |
396
+ | `onError` | `(error: Error) => void` | - | Fired on the `'error'` event. |
397
+ | `onClose` | `(error?: Error) => void` | - | Fired on the `'close'` event. |
398
+
399
+ > [!NOTE]
400
+ > At connect time, `MQTTClientHelper` logs the broker `url` through `redactUrlCredentials()` and `options` through `redactSecrets()`. A password embedded in the URL (`mqtts://user:hunter2@broker:8883`) never reaches the log. Only `mqtts://user:[REDACTED]@broker:8883` does.
401
+
402
+ ### Methods
403
+
404
+ | Method | Returns | Description |
405
+ |--------|---------|--------------|
406
+ | `configure()` | `void` | Connects to the broker and wires `connect`/`disconnect`/`message`/`error`/`close` listeners. Called automatically by the constructor. A no-op if `this.client` already exists. |
407
+ | `getClient()` | `mqtt.MqttClient \| undefined` | The underlying MQTT.js client. |
408
+ | `subscribe(opts)` | `Promise<string[]>` | `opts: { topics: string[] }`. Rejects with an `ApplicationError` (status 400) if the client is not connected. |
409
+ | `publish(opts)` | `Promise<{ topic, message }>` | `opts: { topic: string; message: string \| Buffer }`. Rejects with an `ApplicationError` (status 400) if the client is not connected. |
410
+ | `close(opts?)` | `Promise<void>` | `opts?: { isForce?: boolean }` (default `false`). Ends the client via `client.end(isForce, undefined, callback)`, resolving once the driver reports the connection ended. Safe to call more than once. |
411
+
412
+ ## Troubleshooting
413
+
414
+ ### "Invalid queue name" / "Invalid worker name"
415
+
416
+ **Cause:** `queueName` is empty when `BullMQHelper` configures a queue or worker. This is a **logged error, not a thrown exception** - `this.queue` / `this.worker` are never assigned.
417
+
418
+ **Fix:** Provide a non-empty `queueName`:
419
+
420
+ ```typescript
421
+ // Wrong - this.queue stays undefined, calling .queue.add(...) throws later
422
+ new BullMQHelper({ queueName: '', role: 'queue', identifier: 'x', redisConnection: redis });
423
+
424
+ // Correct
425
+ new BullMQHelper({ queueName: 'my-email-queue', role: 'queue', identifier: 'x', redisConnection: redis });
426
+ ```
427
+
428
+ ### "Invalid client role to configure"
429
+
430
+ **Cause:** `role` is missing or empty (falsy). Logged, not thrown.
431
+
432
+ **Also watch for:** a `role` that IS a non-empty string but not `'queue'`/`'worker'` (a typo like `'Worker'`) triggers **no log at all**. `configure()` silently does nothing, and `this.queue`/`this.worker` both stay `undefined` - see [Configuration lifecycle](#configuration-lifecycle).
433
+
434
+ **Fix:**
435
+
436
+ ```typescript
437
+ new BullMQHelper({ role: 'worker', queueName: 'q', identifier: 'x', redisConnection: redis, /* ... */ });
438
+ ```
439
+
440
+ ### "Invalid url to configure mqtt client!"
441
+
442
+ **Cause:** `url` is empty when constructing `MQTTClientHelper`. **This one throws** an `ApplicationError` (status 500).
443
+
444
+ **Fix:**
445
+
446
+ ```typescript
447
+ new MQTTClientHelper({ url: 'mqtt://localhost:1883', identifier: 'x', options: {}, onMessage: () => {} });
448
+ ```
449
+
450
+ ### "MQTT Client is not available to subscribe topic!" / "... to publish message!"
451
+
452
+ **Cause:** `subscribe()` or `publish()` called before the client finished connecting, or after it disconnected. Rejects with an `ApplicationError` (status 400).
453
+
454
+ **Fix:** Call `subscribe()`/`publish()` from inside `onConnect`, or check `client.getClient()?.connected` first.
455
+
456
+ ### Elements not processing in `SequentialQueueHelper`
457
+
458
+ **Checklist:**
459
+ - `onMessage` must be provided - without it, the generator never initializes and nothing is ever processed.
460
+ - Check `getState()`. A `LOCKED` queue accepts `enqueue()` but never processes. Call `unlock({ shouldProcessNextElement: true })` to resume.
461
+ - Check `autoDispatch`. If it's `false`, call `nextMessage()` manually after each `enqueue()`.
462
+ - Check `isSettled()`. A settled queue rejects new elements - construct a new instance instead.
463
+
464
+ ## Import Reference
465
+
466
+ ```typescript
467
+ import {
468
+ SequentialQueueHelper,
469
+ QueueHelper, // deprecated alias for SequentialQueueHelper
470
+ QueueStatuses,
471
+ HfQueueHelper,
472
+ } from '@venizia/ignis-helpers';
473
+
474
+ import type {
475
+ TQueueStatus,
476
+ TQueueElement,
477
+ IQueueCallback,
478
+ IHfQueueNode,
479
+ TBullQueueRole,
480
+ } from '@venizia/ignis-helpers';
481
+
482
+ import { BullMQHelper } from '@venizia/ignis-helpers/bullmq';
483
+
484
+ import { MQTTClientHelper } from '@venizia/ignis-helpers/mqtt';
485
+ import type { IMQTTClientOptions } from '@venizia/ignis-helpers/mqtt';
486
+ ```
487
+
488
+ ## See also
489
+
490
+ - [Queue overview](/extensions/helpers/queue/) - introduction and the most common tasks
491
+ - [Redis Helper - Full Reference](/extensions/helpers/redis/reference) - `IRedisHelper`, `duplicateClient()`, and the `maxRetriesPerRequest` defaults per topology
492
+ - [Kafka Helpers](/extensions/helpers/kafka/) - the fourth queueing backend
493
+ - [BullMQ documentation](https://docs.bullmq.io/) - underlying job queue library
494
+ - [MQTT.js](https://github.com/mqttjs/MQTT.js) - underlying MQTT client library