@venizia/ignis-docs 0.2.1-0 → 0.2.1-1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/content/best-practices/architectural-patterns.md +3 -3
- package/content/best-practices/code-style-standards/naming-conventions.md +1 -1
- package/content/best-practices/contribution-workflow.md +2 -2
- package/content/best-practices/error-handling.md +94 -89
- package/content/best-practices/security-guidelines.md +5 -5
- package/content/extensions/components/api-reference.md +22 -21
- package/content/extensions/components/authentication/api.md +64 -28
- package/content/extensions/components/authentication/errors.md +19 -5
- package/content/extensions/components/authentication/index.md +19 -18
- package/content/extensions/components/authentication/usage.md +55 -30
- package/content/extensions/components/authorization/api.md +351 -84
- package/content/extensions/components/authorization/errors.md +51 -17
- package/content/extensions/components/authorization/getting-started.md +227 -0
- package/content/extensions/components/authorization/index.md +24 -16
- package/content/extensions/components/authorization/usage.md +45 -21
- package/content/extensions/components/health-check.md +15 -9
- package/content/extensions/components/index.md +24 -90
- package/content/extensions/components/mail/api.md +105 -54
- package/content/extensions/components/mail/errors.md +12 -10
- package/content/extensions/components/mail/index.md +32 -13
- package/content/extensions/components/mail/usage.md +26 -18
- package/content/extensions/components/request-tracker.md +18 -14
- package/content/extensions/components/socket-io/api.md +377 -882
- package/content/extensions/components/socket-io/errors.md +49 -51
- package/content/extensions/components/socket-io/index.md +72 -88
- package/content/extensions/components/socket-io/usage.md +107 -117
- package/content/extensions/components/static-asset/api.md +83 -31
- package/content/extensions/components/static-asset/errors.md +18 -7
- package/content/extensions/components/static-asset/index.md +18 -11
- package/content/extensions/components/static-asset/usage.md +11 -6
- package/content/extensions/components/template/index.md +3 -3
- package/content/extensions/components/websocket/api.md +58 -27
- package/content/extensions/components/websocket/errors.md +3 -3
- package/content/extensions/components/websocket/index.md +8 -7
- package/content/extensions/components/websocket/usage.md +21 -8
- package/content/extensions/helpers/cron/index.md +8 -7
- package/content/extensions/helpers/crypto/index.md +16 -8
- package/content/extensions/helpers/crypto/reference.md +96 -24
- package/content/extensions/helpers/env/index.md +14 -10
- package/content/extensions/helpers/error/index.md +99 -23
- package/content/extensions/helpers/index.md +61 -47
- package/content/extensions/helpers/inversion/index.md +23 -6
- package/content/extensions/helpers/inversion/reference.md +30 -22
- package/content/extensions/helpers/kafka/admin.md +3 -2
- package/content/extensions/helpers/kafka/compile-binary.md +69 -44
- package/content/extensions/helpers/kafka/consumer.md +24 -21
- package/content/extensions/helpers/kafka/examples.md +22 -234
- package/content/extensions/helpers/kafka/index.md +30 -62
- package/content/extensions/helpers/kafka/producer.md +31 -26
- package/content/extensions/helpers/kafka/schema-registry.md +19 -14
- package/content/extensions/helpers/logger/hf-logger.md +49 -22
- package/content/extensions/helpers/logger/index.md +37 -12
- package/content/extensions/helpers/logger/pino.md +31 -11
- package/content/extensions/helpers/logger/reference.md +243 -52
- package/content/extensions/helpers/network/api.md +65 -30
- package/content/extensions/helpers/network/index.md +32 -11
- package/content/extensions/helpers/queue/index.md +17 -6
- package/content/extensions/helpers/queue/reference.md +52 -25
- package/content/extensions/helpers/redis/index.md +29 -11
- package/content/extensions/helpers/redis/reference.md +85 -28
- package/content/extensions/helpers/secrets/index.md +82 -12
- package/content/extensions/helpers/socket-io/api.md +40 -21
- package/content/extensions/helpers/socket-io/index.md +26 -10
- package/content/extensions/helpers/storage/api.md +42 -24
- package/content/extensions/helpers/storage/index.md +10 -7
- package/content/extensions/helpers/types/index.md +20 -7
- package/content/extensions/helpers/types/reference.md +53 -14
- package/content/extensions/helpers/uid/index.md +166 -10
- package/content/extensions/helpers/websocket/api.md +52 -13
- package/content/extensions/helpers/websocket/index.md +7 -4
- package/content/extensions/helpers/worker-thread/index.md +9 -5
- package/content/extensions/helpers/worker-thread/reference.md +20 -20
- package/content/extensions/index.md +38 -39
- package/content/extensions/src-details/mcp-server.md +96 -548
- package/content/guides/core-concepts/persistent/datasources.md +2 -2
- package/content/guides/core-concepts/persistent/index.md +5 -1
- package/content/guides/core-concepts/persistent/pglite.md +218 -0
- package/content/guides/core-concepts/persistent/postgres-drivers.md +21 -15
- package/content/guides/core-concepts/persistent/search-meilisearch.md +22 -12
- package/content/guides/core-concepts/persistent/search-typesense.md +26 -14
- package/content/guides/core-concepts/persistent/sqlite.md +296 -0
- package/content/guides/core-concepts/persistent/transactions.md +12 -11
- package/content/guides/get-started/5-minute-quickstart.md +59 -206
- package/content/guides/get-started/philosophy.md +135 -670
- package/content/guides/get-started/setup.md +53 -74
- package/content/guides/migrations/redis-helpers-migration.md +4 -3
- package/content/guides/migrations/scoped-rbac-migration.md +5 -5
- package/content/guides/migrations/unified-connectors-migration.md +7 -6
- package/content/guides/tutorials/realtime-chat.md +1 -1
- package/content/references/base/application.md +63 -21
- package/content/references/base/components.md +3 -3
- package/content/references/base/connectors.md +14 -14
- package/content/references/base/controllers.md +12 -12
- package/content/references/base/datasources-reference.md +32 -31
- package/content/references/base/datasources.md +6 -6
- package/content/references/base/dependency-injection.md +12 -11
- package/content/references/base/filter-system/application-usage.md +54 -30
- package/content/references/base/filter-system/array-operators.md +24 -46
- package/content/references/base/filter-system/comparison-operators.md +47 -67
- package/content/references/base/filter-system/default-filter.md +59 -53
- package/content/references/base/filter-system/fields-order-pagination.md +92 -146
- package/content/references/base/filter-system/index.md +25 -13
- package/content/references/base/filter-system/json-filtering.md +45 -184
- package/content/references/base/filter-system/list-operators.md +23 -53
- package/content/references/base/filter-system/logical-operators.md +63 -121
- package/content/references/base/filter-system/null-operators.md +34 -104
- package/content/references/base/filter-system/pattern-matching.md +40 -55
- package/content/references/base/filter-system/quick-reference.md +86 -198
- package/content/references/base/filter-system/range-operators.md +18 -46
- package/content/references/base/filter-system/tips.md +6 -6
- package/content/references/base/filter-system/use-cases.md +33 -15
- package/content/references/base/grpc-controllers.md +53 -17
- package/content/references/base/index.md +5 -3
- package/content/references/base/middlewares.md +11 -10
- package/content/references/base/models-reference.md +17 -17
- package/content/references/base/models.md +4 -3
- package/content/references/base/providers.md +8 -8
- package/content/references/base/repositories/advanced.md +224 -326
- package/content/references/base/repositories/index.md +19 -6
- package/content/references/base/repositories/mixins.md +5 -5
- package/content/references/base/repositories/relations.md +160 -293
- package/content/references/base/repositories/soft-deletable.md +16 -6
- package/content/references/base/secrets.md +17 -13
- package/content/references/base/services.md +6 -4
- package/content/references/configuration/environment-variables.md +31 -23
- package/content/references/configuration/index.md +6 -4
- package/content/references/index.md +1 -1
- package/content/references/utilities/duration.md +85 -0
- package/content/references/utilities/index.md +5 -1
- package/content/references/utilities/jsx-reference.md +11 -11
- package/content/references/utilities/jsx.md +2 -2
- package/content/references/utilities/module.md +78 -25
- package/content/references/utilities/request.md +2 -1
- package/content/references/utilities/retry.md +139 -0
- package/content/references/utilities/schema.md +2 -2
- package/content/references/utilities/statuses-reference.md +4 -4
- package/content/references/utilities/statuses.md +5 -5
- package/dist/mcp-server/index.js +0 -0
- package/dist/mcp-server/tools/github/list-project-files.tool.d.ts +1 -1
- package/dist/mcp-server/tools/github/list-project-files.tool.d.ts.map +1 -1
- package/dist/mcp-server/tools/github/list-project-files.tool.js +3 -3
- package/dist/mcp-server/tools/github/search-code.tool.js +1 -1
- package/dist/mcp-server/tools/github/verify-dependencies.tool.js +1 -1
- package/dist/mcp-server/tools/github/view-source-file.tool.js +1 -1
- package/package.json +17 -16
|
@@ -8,6 +8,20 @@ difficulty: intermediate
|
|
|
8
8
|
|
|
9
9
|
Exhaustive reference for `BaseNetworkRequest` and its fetchers, the TCP/TLS client-server hierarchy, `NetworkUdpClient`, and every option type. For a readable introduction and the most common tasks, start with the [Network overview](/extensions/helpers/network/).
|
|
10
10
|
|
|
11
|
+
## Find what you need
|
|
12
|
+
|
|
13
|
+
| You're looking for | Go to |
|
|
14
|
+
|---|---|
|
|
15
|
+
| The HTTP client base class and URL helpers | [BaseNetworkRequest](#basenetworkrequest) |
|
|
16
|
+
| Axios-backed HTTP client | [AxiosFetcher](#axiosfetcher) / [AxiosNetworkRequest](#axiosnetworkrequest) |
|
|
17
|
+
| Native-`fetch`-backed HTTP client | [NodeFetcher](#nodefetcher) / [NodeFetchNetworkRequest](#nodefetchnetworkrequest) |
|
|
18
|
+
| Why the `QUERY` method must reach the wire uppercase | [HTTP.Methods](#http-methods) |
|
|
19
|
+
| Secret redaction in request logs | [Request Logging and Redaction](#request-logging-and-redaction) |
|
|
20
|
+
| Plain TCP or TLS server, client tracking, authentication | [BaseNetworkTcpServer](#basenetworktcpserver) |
|
|
21
|
+
| Plain TCP or TLS client, auto-reconnect behavior | [BaseNetworkTcpClient](#basenetworktcpclient) |
|
|
22
|
+
| UDP client and multicast | [NetworkUdpClient](#networkudpclient) |
|
|
23
|
+
| Every option and type in one place | [Types Reference](#types-reference) |
|
|
24
|
+
|
|
11
25
|
**Files:**
|
|
12
26
|
|
|
13
27
|
- [`packages/helpers/src/modules/network/http-request/base-network-request.helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/network/http-request/base-network-request.helper.ts) - `BaseNetworkRequest`
|
|
@@ -48,7 +62,7 @@ AbstractNetworkFetchableHelper<V, RQ, RS> (implements IFetchable, NOT a BaseHel
|
|
|
48
62
|
└── NodeFetcher (V = 'node-fetch')
|
|
49
63
|
```
|
|
50
64
|
|
|
51
|
-
|
|
65
|
+
Every class that extends `BaseHelper` inherits scoped logging via `this.logger`. `AbstractNetworkFetchableHelper` and its two fetchers do **not** extend `BaseHelper`. They accept an optional `logger` parameter per call instead - see [Request Logging and Redaction](#request-logging-and-redaction).
|
|
52
66
|
|
|
53
67
|
---
|
|
54
68
|
|
|
@@ -161,7 +175,7 @@ interface IFetchable<
|
|
|
161
175
|
}
|
|
162
176
|
```
|
|
163
177
|
|
|
164
|
-
All HTTP method shortcuts (`get`, `post`, `put`, `patch`, `delete`, `query`) delegate to `send()` with the `method` field set accordingly. `query` sends the HTTP `QUERY` method ([RFC 9110/10008](https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-method-w-body-10.html))
|
|
178
|
+
All HTTP method shortcuts (`get`, `post`, `put`, `patch`, `delete`, `query`) delegate to `send()` with the `method` field set accordingly. `query` sends the HTTP `QUERY` method ([RFC 9110/10008](https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-method-w-body-10.html)). It behaves like `GET` but carries a body - useful for search payloads too large to fit a query string.
|
|
165
179
|
|
|
166
180
|
### IRequestOptions
|
|
167
181
|
|
|
@@ -175,7 +189,9 @@ interface IRequestOptions {
|
|
|
175
189
|
}
|
|
176
190
|
```
|
|
177
191
|
|
|
178
|
-
### HTTP.Methods
|
|
192
|
+
### HTTP.Methods
|
|
193
|
+
|
|
194
|
+
The const-class every fetcher dispatches on:
|
|
179
195
|
|
|
180
196
|
```typescript
|
|
181
197
|
HTTP.Methods = {
|
|
@@ -188,9 +204,15 @@ type THttpMethod = ValueOf<typeof HTTP.Methods> | Uppercase<ValueOf<typeof HTTP.
|
|
|
188
204
|
|
|
189
205
|
> [!IMPORTANT]
|
|
190
206
|
> - **Every `HTTP.Methods` token is lowercase.** `@hono/zod-openapi` route definitions accept no other case.
|
|
191
|
-
> - **Both fetchers accept either case on input
|
|
192
|
-
> - **This is not cosmetic.** Node's undici
|
|
193
|
-
>
|
|
207
|
+
> - **Both fetchers accept either case on input**, `'post'` or `'POST'`. Each calls `method.toUpperCase()` immediately before dispatching to its transport.
|
|
208
|
+
> - **This is not cosmetic.** Node's undici is the `fetch` implementation on Node - not Bun - and it auto-normalizes only some methods:
|
|
209
|
+
>
|
|
210
|
+
> | Auto-normalized by undici | Sent through verbatim |
|
|
211
|
+
> |---|---|
|
|
212
|
+
> | `DELETE`, `GET`, `HEAD`, `OPTIONS`, `POST`, `PUT` | `PATCH`, `QUERY` |
|
|
213
|
+
>
|
|
214
|
+
> A lowercase `patch` or `query` reaches the server unchanged, and most servers reject it.
|
|
215
|
+
> - **Bun and Axios hide the bug.** Bun's `fetch` and Axios both uppercase every method themselves - Axios via `node:http`. The bug surfaces only once the app runs on Node with undici.
|
|
194
216
|
|
|
195
217
|
---
|
|
196
218
|
|
|
@@ -275,7 +297,7 @@ constructor(opts: {
|
|
|
275
297
|
})
|
|
276
298
|
```
|
|
277
299
|
|
|
278
|
-
`opts.logger`, if provided, logs `'Creating new network request worker instance! Name: %s'` once at construction time
|
|
300
|
+
`opts.logger`, if provided, logs `'Creating new network request worker instance! Name: %s'` once at construction time. This is unrelated to the per-call `logger` argument on `send()`.
|
|
279
301
|
|
|
280
302
|
#### IAxiosRequestOptions
|
|
281
303
|
|
|
@@ -292,7 +314,7 @@ interface IAxiosRequestOptions extends AxiosRequestConfig, IRequestOptions {
|
|
|
292
314
|
> [!NOTE]
|
|
293
315
|
> - **`body` maps to Axios's `data`** field internally.
|
|
294
316
|
> - **Query `params` are serialized** using `node:querystring` via Axios's `paramsSerializer`.
|
|
295
|
-
> - **HTTPS gets an `https.Agent` automatically
|
|
317
|
+
> - **HTTPS gets an `https.Agent` automatically.** Certificate verification is ON, and connections are kept alive. Pass `rejectUnauthorized: false` - on the fetcher, for the whole instance, or on a single request - only when you genuinely mean to accept an unverified certificate. A caller-supplied `httpsAgent` always wins, so a custom CA, pinning or mTLS agent is honoured as given.
|
|
296
318
|
|
|
297
319
|
#### Methods
|
|
298
320
|
|
|
@@ -389,9 +411,9 @@ override async send(opts: INodeFetchRequestOptions, logger?: any): Promise<Respo
|
|
|
389
411
|
Dispatches the request using the native `fetch` API. Behavior:
|
|
390
412
|
|
|
391
413
|
- `method` defaults to `HTTP.Methods.GET` and is uppercased before dispatch.
|
|
392
|
-
- Query `params` are serialized with `node:querystring` and appended to `url
|
|
414
|
+
- Query `params` are serialized with `node:querystring` and appended to `url`. The separator adapts: `?` when the URL carries no query string yet, `&` when it already does - never a double `?`.
|
|
393
415
|
- If `timeout` is provided, an internal `AbortController` aborts the request after that many milliseconds.
|
|
394
|
-
- The internal timeout signal is **composed**, never substituted, with a caller-supplied `signal
|
|
416
|
+
- The internal timeout signal is **composed**, never substituted, with a caller-supplied `signal`. When both are present, IGNIS builds `AbortSignal.any([signal, timeoutController.signal])`. That way a caller aborting its own signal still cancels the request, even while a timeout is also armed.
|
|
395
417
|
- The timer is cleared as soon as the request settles.
|
|
396
418
|
- If `logger` is passed, logs `'URL: %s | Props: %s | Timeout: %s'` at `info` level with the request config run through `redactSecrets()`.
|
|
397
419
|
|
|
@@ -429,7 +451,7 @@ interface INodeFetchNetworkRequestOptions {
|
|
|
429
451
|
If `headers` is a `Headers` instance, it is converted to a plain object via `Object.fromEntries(headers.entries())` before merging with the default.
|
|
430
452
|
|
|
431
453
|
> [!WARNING] `timeout` is per-call, not per-instance
|
|
432
|
-
> `networkOptions` is `RequestInit`, which has no `timeout` field
|
|
454
|
+
> `networkOptions` is `RequestInit`, which has no `timeout` field. Passing one there has no effect on request cancellation. `NodeFetcher.send()` only reads `timeout` from the arguments of each individual `send()`/`get()`/... call. Pass it every time you need an abort:
|
|
433
455
|
>
|
|
434
456
|
> ```typescript
|
|
435
457
|
> await this.getNetworkService().send({ url: '/slow-endpoint', method: 'get', timeout: 5000 });
|
|
@@ -437,15 +459,15 @@ If `headers` is a `Headers` instance, it is converted to a plain object via `Obj
|
|
|
437
459
|
|
|
438
460
|
---
|
|
439
461
|
|
|
440
|
-
### Request Logging
|
|
462
|
+
### Request Logging and Redaction
|
|
441
463
|
|
|
442
|
-
Neither fetcher logs anything by default
|
|
464
|
+
Neither fetcher logs anything by default. `send()` and every shortcut accept an **optional** `logger` as the second argument, guarded internally with `logger?.for(...)`. Pass a logger to get an `info`-level line per request - typically `this.logger`, whether the caller is a `BaseHelper` subclass or extends `BaseNetworkRequest` directly:
|
|
443
465
|
|
|
444
466
|
```typescript
|
|
445
467
|
await this.getNetworkService().post({ url, body }, this.logger);
|
|
446
468
|
```
|
|
447
469
|
|
|
448
|
-
Whenever a logger is passed, the assembled request config
|
|
470
|
+
Whenever a logger is passed, IGNIS runs the assembled request config through `redactSecrets()` before the log line is written. That covers `url`, `method`, `headers`, `body`/`data`, and any other option. Redaction matches by key name, case-insensitively, at any depth, against `SECRET_KEY_PATTERN`:
|
|
449
471
|
|
|
450
472
|
| Key group | Matched spellings |
|
|
451
473
|
|-----------|--------------------|
|
|
@@ -455,7 +477,7 @@ Whenever a logger is passed, the assembled request config - `url`, `method`, `he
|
|
|
455
477
|
|
|
456
478
|
Source: [`redact.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/common/redact.ts) `SECRET_KEY_PATTERN`.
|
|
457
479
|
|
|
458
|
-
A matched value is replaced with `'[REDACTED]'
|
|
480
|
+
A matched value is replaced with `'[REDACTED]'`. `Buffer`/typed-array values under a non-matching key are summarized as `'[Binary N bytes]'` rather than serialized. There is nothing to configure beyond passing the logger - you never call `redactSecrets()` yourself at the call site.
|
|
459
481
|
|
|
460
482
|
---
|
|
461
483
|
|
|
@@ -479,7 +501,7 @@ Abstract TCP server with client tracking, authentication flow, and event delegat
|
|
|
479
501
|
constructor(opts: ITcpSocketServerOptions<SocketServerOptions, SocketServerType, SocketClientType>)
|
|
480
502
|
```
|
|
481
503
|
|
|
482
|
-
**Throws:** `ApplicationError`
|
|
504
|
+
**Throws:** `ApplicationError` when `authenticateOptions.required` is `true` and `duration` is missing, `0`, or negative. Message: `'TCP Server | Invalid authenticate duration | Required duration for authenticateOptions'`.
|
|
483
505
|
|
|
484
506
|
The constructor calls `configure()`, which creates the server via `createServerFn` and starts listening.
|
|
485
507
|
|
|
@@ -495,7 +517,7 @@ The constructor calls `configure()`, which creates the server via `createServerF
|
|
|
495
517
|
| `extraEvents` | `Record<string, (opts) => ValueOrPromise<void>>` | Additional per-client socket events to register |
|
|
496
518
|
|
|
497
519
|
- **Hooks never crash the process.** `onClientData`, `onClientConnected`, `onClientClose`, `onClientError`, `onServerReady`, `onServerError`, and each `extraEvents` entry all run through an internal `invokeHook()` wrapper.
|
|
498
|
-
- **Why it exists.** A hook throwing synchronously inside a raw `net`/`tls` event listener would otherwise be an uncaught exception
|
|
520
|
+
- **Why it exists.** A hook throwing synchronously inside a raw `net`/`tls` event listener would otherwise be an uncaught exception. That crashes the process. `invokeHook()` catches the throw and logs it instead.
|
|
499
521
|
|
|
500
522
|
#### Methods
|
|
501
523
|
|
|
@@ -531,7 +553,7 @@ getServer(): SocketServerType
|
|
|
531
553
|
|
|
532
554
|
##### `doAuthenticate(opts)`
|
|
533
555
|
|
|
534
|
-
Transitions a client's authentication state.
|
|
556
|
+
Transitions a client's authentication state.
|
|
535
557
|
|
|
536
558
|
```typescript
|
|
537
559
|
doAuthenticate(opts: {
|
|
@@ -540,6 +562,11 @@ doAuthenticate(opts: {
|
|
|
540
562
|
}): void
|
|
541
563
|
```
|
|
542
564
|
|
|
565
|
+
| New state | Effect |
|
|
566
|
+
|---|---|
|
|
567
|
+
| `'authenticated'` | Sets `storage.authenticatedAt`; clears the pending kick-timer |
|
|
568
|
+
| `'unauthorized'` / `'authenticating'` | Clears `storage.authenticatedAt` |
|
|
569
|
+
|
|
543
570
|
##### `emit(opts)`
|
|
544
571
|
|
|
545
572
|
Writes data to a specific client's socket. Never throws - each failure case logs and returns instead:
|
|
@@ -569,7 +596,7 @@ async shutdown(): Promise<void>
|
|
|
569
596
|
3. Empties the `clients` registry.
|
|
570
597
|
4. Calls `server.close()` and awaits its callback.
|
|
571
598
|
|
|
572
|
-
- **Why this order.** `server.close()` alone never resolves while a socket is still attached
|
|
599
|
+
- **Why this order.** `server.close()` alone never resolves while a socket is still attached. A caller reaching through `getServer().close()` on a busy server would hang forever.
|
|
573
600
|
- **Idempotent.** A second call is a no-op that resolves cleanly.
|
|
574
601
|
- **Safe on a server that never finished `listen()`.** Logs the resulting `ERR_SERVER_NOT_RUNNING` rather than throwing.
|
|
575
602
|
- **After `shutdown()`**, new connection attempts are refused.
|
|
@@ -668,7 +695,7 @@ Abstract TCP client with auto-reconnect, encoding support, and lifecycle hooks.
|
|
|
668
695
|
constructor(opts: INetworkTcpClientProps<SocketClientOptions, SocketClientType>)
|
|
669
696
|
```
|
|
670
697
|
|
|
671
|
-
Does **not** call `connect()` automatically
|
|
698
|
+
Does **not** call `connect()` automatically. Construction only stores options - call `connect({ resetReconnectCounter })` explicitly.
|
|
672
699
|
|
|
673
700
|
#### Protected Properties
|
|
674
701
|
|
|
@@ -686,7 +713,7 @@ Does **not** call `connect()` automatically - construction only stores options;
|
|
|
686
713
|
protected getLoggableOptions(): unknown
|
|
687
714
|
```
|
|
688
715
|
|
|
689
|
-
Returns `redactSecrets(this.options)`. A TLS client's `options` **is** its private key material
|
|
716
|
+
Returns `redactSecrets(this.options)`. A TLS client's `options` **is** its private key material - `key`/`cert`/`passphrase`. Every internal log call uses this method instead of logging `this.options` directly, or the key would be written to every log file and aggregator downstream.
|
|
690
717
|
|
|
691
718
|
#### Methods
|
|
692
719
|
|
|
@@ -695,7 +722,11 @@ Returns `redactSecrets(this.options)`. A TLS client's `options` **is** its priva
|
|
|
695
722
|
Establishes the connection:
|
|
696
723
|
|
|
697
724
|
- No-op with a log line if already connected (`isConnected()`) or if `options` is empty.
|
|
698
|
-
- Otherwise,
|
|
725
|
+
- Otherwise, in order:
|
|
726
|
+
1. Destroys any stale `client` first.
|
|
727
|
+
2. Creates the socket via `createClientFn`.
|
|
728
|
+
3. Registers `data`/`close`/`error` listeners.
|
|
729
|
+
4. Applies `encoding`, if set.
|
|
699
730
|
|
|
700
731
|
```typescript
|
|
701
732
|
connect(opts: { resetReconnectCounter: boolean }): void
|
|
@@ -764,11 +795,11 @@ handleError(error: any): void // Log
|
|
|
764
795
|
| `reconnect` is `true` and `currentReconnect < maxReconnect` (with `maxReconnect >= 0`) | Reconnect scheduled |
|
|
765
796
|
|
|
766
797
|
> [!IMPORTANT]
|
|
767
|
-
> - **`maxReconnect: -1` does not mean unlimited reconnects - it disables reconnection entirely.**
|
|
768
|
-
> - **A second guard further down the method is dead code.** `if (maxReconnect > -1 && currentReconnect >= maxReconnect)` can never be true
|
|
798
|
+
> - **`maxReconnect: -1` does not mean unlimited reconnects - it disables reconnection entirely.** That is the opposite of the "`-1` = unlimited" convention used elsewhere in the framework - the Redis retry helper, for example.
|
|
799
|
+
> - **A second guard further down the method is dead code.** `if (maxReconnect > -1 && currentReconnect >= maxReconnect)` can never be true. By the time control reaches it, the first guard has already ruled out `currentReconnect >= maxReconnect`.
|
|
769
800
|
> - Source: [`base-tcp-client.helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/network/tcp-socket/base-tcp-client.helper.ts).
|
|
770
801
|
|
|
771
|
-
The reconnect delay is a **fixed 5000 ms
|
|
802
|
+
The reconnect delay is a **fixed 5000 ms**. There is no backoff growth on the TCP/TLS client, unlike the Redis helper's exponential strategy.
|
|
772
803
|
|
|
773
804
|
---
|
|
774
805
|
|
|
@@ -862,15 +893,19 @@ static newInstance(opts: INetworkUdpClientProps): NetworkUdpClient
|
|
|
862
893
|
|
|
863
894
|
##### `connect()`
|
|
864
895
|
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
896
|
+
Creates a `dgram.Socket` and binds it, in order:
|
|
897
|
+
|
|
898
|
+
1. Creates the socket (`type: 'udp4'`, `reuseAddr` from options).
|
|
899
|
+
2. Registers `close`/`error`/`listening`/`message` listeners.
|
|
900
|
+
3. Binds to `port`/`host`.
|
|
901
|
+
|
|
902
|
+
Each listener routes through an internal `invokeHook()` wrapper - the same synchronous-throw guard as the TCP server. `onBind` fires after binding completes; it's the place to join multicast groups via `socket.addMembership(group, iface)`.
|
|
868
903
|
|
|
869
904
|
```typescript
|
|
870
905
|
connect(): void
|
|
871
906
|
```
|
|
872
907
|
|
|
873
|
-
No-op with a log line
|
|
908
|
+
No-op with a log line in two cases: `client` is already set, or `port` fails `Number.isInteger(port) && port >= 0`. Port `0` itself is valid - it means "let the OS assign a free port" - and is accepted.
|
|
874
909
|
|
|
875
910
|
##### `disconnect()`
|
|
876
911
|
|
|
@@ -970,7 +1005,7 @@ interface ITcpSocketServerOptions<
|
|
|
970
1005
|
}
|
|
971
1006
|
```
|
|
972
1007
|
|
|
973
|
-
`onServerError` fires when the underlying `net`/`tls` server emits `'error'`
|
|
1008
|
+
`onServerError` fires when the underlying `net`/`tls` server emits `'error'` - for example `EADDRINUSE` from a port already in use. Without this listener, the event is unhandled and takes the whole process down with it.
|
|
974
1009
|
|
|
975
1010
|
### INetworkTcpClientProps
|
|
976
1011
|
|
|
@@ -6,7 +6,7 @@ difficulty: intermediate
|
|
|
6
6
|
|
|
7
7
|
# Network
|
|
8
8
|
|
|
9
|
-
Network helpers give you HTTP request clients (fetch or Axios), symmetric TCP/TLS socket client-server pairs, and a UDP client
|
|
9
|
+
Network helpers give you HTTP request clients (fetch or Axios), symmetric TCP/TLS socket client-server pairs, and a UDP client. All three share the same options-object construction and scoped logging as the rest of the framework.
|
|
10
10
|
|
|
11
11
|
## In one example
|
|
12
12
|
|
|
@@ -31,16 +31,30 @@ class GitHubApi extends NodeFetchNetworkRequest {
|
|
|
31
31
|
}
|
|
32
32
|
```
|
|
33
33
|
|
|
34
|
-
`getRequestUrl()` joins the constructor's `baseUrl` with the path segments
|
|
34
|
+
`getRequestUrl()` joins the constructor's `baseUrl` with the path segments. `getNetworkService()` returns the underlying fetcher, which exposes `send()` plus `get`/`post`/`put`/`patch`/`delete`/`query` shortcuts.
|
|
35
35
|
|
|
36
36
|
## How it works
|
|
37
37
|
|
|
38
|
-
- **HTTP is two layers.** `BaseNetworkRequest` holds
|
|
39
|
-
|
|
40
|
-
|
|
38
|
+
- **HTTP is two layers.** `BaseNetworkRequest` holds the base URL and delegates every call to an `IFetchable` fetcher. You extend one of two concrete clients:
|
|
39
|
+
|
|
40
|
+
| Client | Wraps | Import from |
|
|
41
|
+
|---|---|---|
|
|
42
|
+
| `NodeFetchNetworkRequest` | native `fetch` | root barrel |
|
|
43
|
+
| `AxiosNetworkRequest` | Axios | `@venizia/ignis-helpers/axios` sub-path |
|
|
44
|
+
|
|
45
|
+
- **`axios` is optional.** It is an optional peer dependency. `AxiosNetworkRequest` is exported only from the `/axios` sub-path, never from the root barrel.
|
|
46
|
+
- **TCP and TLS share one hierarchy.** `BaseNetworkTcpServer`/`BaseNetworkTcpClient` are abstract classes. Each takes a create-function matching its transport, and each concrete pair only pre-wires that function:
|
|
47
|
+
|
|
48
|
+
| Transport | Server class | Client class | Node module |
|
|
49
|
+
|---|---|---|---|
|
|
50
|
+
| Plain TCP | `NetworkTcpServer` | `NetworkTcpClient` | `net` |
|
|
51
|
+
| TLS | `NetworkTlsTcpServer` | `NetworkTlsTcpClient` | `tls` |
|
|
52
|
+
|
|
53
|
+
Constructor options and every method are otherwise identical between the plain and encrypted variants.
|
|
54
|
+
|
|
41
55
|
- **Servers track per-client authentication state.** `unauthorized` → `authenticating` → `authenticated`.
|
|
42
56
|
- **Clients auto-reconnect on a fixed delay.** 5 seconds between attempts, up to `maxRetry` attempts, when `reconnect: true`. See the [Full reference](/extensions/helpers/network/api) for the `maxRetry: -1` edge case.
|
|
43
|
-
- **UDP has no client/server split.** `NetworkUdpClient` is a single class wrapping `node:dgram`
|
|
57
|
+
- **UDP has no client/server split.** `NetworkUdpClient` is a single class wrapping `node:dgram` in UDP4 mode. It supports optional multicast group joining via `onBind`.
|
|
44
58
|
- **Every class extends `BaseHelper`.** `this.logger.for('methodName')` scoped logging is available throughout.
|
|
45
59
|
|
|
46
60
|
**HTTP method case**
|
|
@@ -48,7 +62,7 @@ class GitHubApi extends NodeFetchNetworkRequest {
|
|
|
48
62
|
| Layer | Case | Why |
|
|
49
63
|
|-------|------|-----|
|
|
50
64
|
| `HTTP.Methods.GET`/`.POST`/`.QUERY`/... tokens | always lowercase | Required by `@hono/zod-openapi` route definitions |
|
|
51
|
-
| Wire dispatch | uppercased right before send | Node's undici only auto-normalizes `DELETE`/`GET`/`HEAD`/`OPTIONS`/`POST`/`PUT
|
|
65
|
+
| Wire dispatch | uppercased right before send | Node's undici only auto-normalizes `DELETE`/`GET`/`HEAD`/`OPTIONS`/`POST`/`PUT`. A lowercase `patch` or `query` would go out unchanged |
|
|
52
66
|
|
|
53
67
|
## Common tasks
|
|
54
68
|
|
|
@@ -73,11 +87,18 @@ class PaymentGateway extends AxiosNetworkRequest {
|
|
|
73
87
|
}
|
|
74
88
|
```
|
|
75
89
|
|
|
76
|
-
`AxiosNetworkRequest` applies defaults
|
|
90
|
+
`AxiosNetworkRequest` applies these defaults - override any of them in `networkOptions`:
|
|
91
|
+
|
|
92
|
+
| Setting | Default |
|
|
93
|
+
|---|---|
|
|
94
|
+
| `content-type` header | `application/json; charset=utf-8` |
|
|
95
|
+
| `withCredentials` | `true` |
|
|
96
|
+
| `validateStatus` | `status < 500` |
|
|
97
|
+
| `timeout` | `60000` ms |
|
|
77
98
|
|
|
78
99
|
### Log a request with secrets redacted
|
|
79
100
|
|
|
80
|
-
Pass a logger
|
|
101
|
+
Pass a logger as the second argument to `send()` or any shortcut method - typically `this.logger` from a `BaseHelper` subclass. The fetcher then logs the URL and config at `info` level. It runs the config through `redactSecrets()` first, so values like `Authorization` or `X-API-Key` reach the log as `'[REDACTED]'`. Without a logger argument, nothing is logged.
|
|
81
102
|
|
|
82
103
|
```typescript
|
|
83
104
|
async charge(amount: number, currency: string) {
|
|
@@ -115,11 +136,11 @@ client.connect({ resetReconnectCounter: true });
|
|
|
115
136
|
client.emit({ payload: 'Hello, Server!' });
|
|
116
137
|
```
|
|
117
138
|
|
|
118
|
-
`NetworkTlsTcpServer`/`NetworkTlsTcpClient` use the identical API
|
|
139
|
+
`NetworkTlsTcpServer`/`NetworkTlsTcpClient` use the identical API. Pass certificates in `serverOptions`/`options`, typed `TlsOptions`/`ConnectionOptions` from `node:tls`.
|
|
119
140
|
|
|
120
141
|
### Require TCP client authentication
|
|
121
142
|
|
|
122
|
-
Set `authenticateOptions.required: true` with a positive `duration`
|
|
143
|
+
Set `authenticateOptions.required: true` with a positive `duration` in milliseconds. The constructor throws if `duration` is missing or negative. Clients that never call `doAuthenticate()` within that window are disconnected automatically.
|
|
123
144
|
|
|
124
145
|
```typescript
|
|
125
146
|
const server = new NetworkTcpServer({
|
|
@@ -6,7 +6,7 @@ difficulty: intermediate
|
|
|
6
6
|
|
|
7
7
|
# Queue
|
|
8
8
|
|
|
9
|
-
The Queue helpers
|
|
9
|
+
The Queue helpers move work between processes. Pick a backend: BullMQ for durable jobs, an in-memory queue for single-process sequencing, or MQTT for lightweight pub/sub.
|
|
10
10
|
|
|
11
11
|
## In one example
|
|
12
12
|
|
|
@@ -51,9 +51,20 @@ const worker = BullMQHelper.newInstance({
|
|
|
51
51
|
| In-memory | `SequentialQueueHelper` (alias `QueueHelper`) | none | Sequential, single-process work that does not need persistence |
|
|
52
52
|
| MQTT | `MQTTClientHelper` | `mqtt` | Pub/sub for IoT and lightweight real-time events, not job processing |
|
|
53
53
|
|
|
54
|
-
- **`BullMQHelper` takes one `role` per instance, fixed at construction.**
|
|
55
|
-
|
|
56
|
-
|
|
54
|
+
- **`BullMQHelper` takes one `role` per instance, fixed at construction.**
|
|
55
|
+
|
|
56
|
+
| `role` | Property | Use |
|
|
57
|
+
|---|---|---|
|
|
58
|
+
| `'queue'` | `.queue` (BullMQ `Queue`) | Producers call `.add()` |
|
|
59
|
+
| `'worker'` | `.worker` (BullMQ `Worker`) | Driven by your `onWorkerData` callback |
|
|
60
|
+
|
|
61
|
+
- **One Redis helper backs any number of queues and workers.** The helper always calls `redisConnection.duplicateClient()` to open a dedicated connection for whichever role it owns. It never reuses your client directly.
|
|
62
|
+
- **`SequentialQueueHelper` runs one element at a time**, in a `WAITING -> PROCESSING -> WAITING` loop driven by `onMessage`. Two calls change that loop:
|
|
63
|
+
|
|
64
|
+
| Call | Effect |
|
|
65
|
+
|---|---|
|
|
66
|
+
| `lock()` | Pauses processing at `LOCKED`. Elements still enqueue; nothing processes until `unlock()`. |
|
|
67
|
+
| `settle()` / `close()` | Drains the queue, then moves to the terminal `SETTLED` state. |
|
|
57
68
|
|
|
58
69
|
Full option tables, the complete state machine, and `HfQueueHelper` (the low-level FIFO primitive underneath) are in the [Full reference](/extensions/helpers/queue/reference).
|
|
59
70
|
|
|
@@ -69,7 +80,7 @@ Jobs default to `removeOnComplete: true, removeOnFail: true` - BullMQ does not r
|
|
|
69
80
|
|
|
70
81
|
### Process jobs with a worker
|
|
71
82
|
|
|
72
|
-
`numberOfWorker` sets concurrency
|
|
83
|
+
`numberOfWorker` sets concurrency. `onWorkerDataFail` receives the job (possibly `undefined`) and the error.
|
|
73
84
|
|
|
74
85
|
```typescript
|
|
75
86
|
const worker = BullMQHelper.newInstance({
|
|
@@ -90,7 +101,7 @@ const worker = BullMQHelper.newInstance({
|
|
|
90
101
|
|
|
91
102
|
### The Redis connection requirement
|
|
92
103
|
|
|
93
|
-
Pass an `IRedisHelper` instance, not a raw ioredis client
|
|
104
|
+
Pass an `IRedisHelper` instance, not a raw ioredis client. `BullMQHelper` calls `redisConnection.duplicateClient()` internally.
|
|
94
105
|
|
|
95
106
|
| Redis helper | Sets `maxRetriesPerRequest: null`? |
|
|
96
107
|
|--------------|-------------------------------------|
|
|
@@ -17,6 +17,21 @@ Exhaustive reference for the Queue helper family. For a readable introduction an
|
|
|
17
17
|
- [`packages/helpers/src/modules/queue/mqtt/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/mqtt/helper.ts) - `MQTTClientHelper`
|
|
18
18
|
- [`packages/helpers/src/modules/queue/common/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/common/types.ts) - `TBullQueueRole`
|
|
19
19
|
|
|
20
|
+
## Find what you need
|
|
21
|
+
|
|
22
|
+
| You want to | Go to |
|
|
23
|
+
|---|---|
|
|
24
|
+
| Pick a backend and see import paths | [Class Overview](#class-overview) / [Import Paths](#import-paths) |
|
|
25
|
+
| Construct a producer or worker and see every option | [BullMQHelper](#bullmqhelper) |
|
|
26
|
+
| Understand what a bad `role` or empty `queueName` does | [Configuration lifecycle](#configuration-lifecycle) |
|
|
27
|
+
| Run BullMQ against Redis Cluster | [Redis Cluster setup](#redis-cluster-setup) |
|
|
28
|
+
| Construct an in-memory sequential queue and see every option | [SequentialQueueHelper](#sequentialqueuehelper) |
|
|
29
|
+
| Understand the `WAITING`/`PROCESSING`/`LOCKED`/`SETTLED` state machine | [QueueStatuses and the state machine](#queuestatuses-and-the-state-machine) |
|
|
30
|
+
| Reach for a bare O(1) FIFO instead of a job queue | [HfQueueHelper](#hfqueuehelper) |
|
|
31
|
+
| Construct an MQTT client and see every option | [MQTTClientHelper](#mqttclienthelper) |
|
|
32
|
+
| Fix a thrown or logged error message | [Troubleshooting](#troubleshooting) |
|
|
33
|
+
| Copy the full import list | [Import Reference](#import-reference) |
|
|
34
|
+
|
|
20
35
|
## Class Overview
|
|
21
36
|
|
|
22
37
|
| Class | Extends | Peer dependency | Use case |
|
|
@@ -26,11 +41,11 @@ Exhaustive reference for the Queue helper family. For a readable introduction an
|
|
|
26
41
|
| `HfQueueHelper` | `BaseHelper` | none | Generic O(1) FIFO primitive - single-threaded, no callbacks, no persistence |
|
|
27
42
|
| `MQTTClientHelper` | `BaseHelper` | `mqtt` | MQTT broker pub/sub - IoT and lightweight real-time events |
|
|
28
43
|
|
|
29
|
-
Kafka (`KafkaProducerHelper`, `KafkaConsumerHelper`, `KafkaAdminHelper`, `KafkaSchemaRegistryHelper`) is a fourth backend under the same `queue/` module tree
|
|
44
|
+
Kafka (`KafkaProducerHelper`, `KafkaConsumerHelper`, `KafkaAdminHelper`, `KafkaSchemaRegistryHelper`) is a fourth backend under the same `queue/` module tree. It's documented on its own page - see [Kafka Helpers](/extensions/helpers/kafka/).
|
|
30
45
|
|
|
31
46
|
## Import Paths
|
|
32
47
|
|
|
33
|
-
`BullMQHelper` and `MQTTClientHelper` live behind sub-path exports
|
|
48
|
+
`BullMQHelper` and `MQTTClientHelper` live behind sub-path exports. That keeps their peer dependencies (`bullmq`, `mqtt`) from becoming hard dependencies of the base package. `SequentialQueueHelper`, `QueueHelper`, `QueueStatuses`, `HfQueueHelper`, and `TBullQueueRole` ship from the root package instead.
|
|
34
49
|
|
|
35
50
|
```typescript
|
|
36
51
|
// Root package - no peer dependency
|
|
@@ -93,7 +108,7 @@ const worker = new BullMQHelper({
|
|
|
93
108
|
| `onWorkerDataFail` | `(job \| undefined, error: Error) => Promise<void>` | - | Fired on the BullMQ `Worker` `'failed'` event. `job` may be `undefined`. |
|
|
94
109
|
|
|
95
110
|
> [!IMPORTANT]
|
|
96
|
-
> Pass an `IRedisHelper` instance to `redisConnection`, **not** the raw ioredis client. `BullMQHelper` calls `redisConnection.duplicateClient()` internally to get a dedicated connection per role
|
|
111
|
+
> Pass an `IRedisHelper` instance to `redisConnection`, **not** the raw ioredis client. `BullMQHelper` calls `redisConnection.duplicateClient()` internally to get a dedicated connection per role. One shared Redis helper can back any number of queues and workers.
|
|
97
112
|
|
|
98
113
|
### Configuration lifecycle
|
|
99
114
|
|
|
@@ -103,14 +118,18 @@ The constructor calls `configure()`, which switches on `role`:
|
|
|
103
118
|
|--------|---------------|--------|
|
|
104
119
|
| `'queue'` | `configureQueue()` | Builds `this.queue` (BullMQ `Queue`); attaches an `'error'` listener |
|
|
105
120
|
| `'worker'` | `configureWorker()` | Builds `this.worker` (BullMQ `Worker`); attaches `'completed'`, `'failed'`, `'error'` listeners |
|
|
106
|
-
| missing
|
|
121
|
+
| missing (falsy) | neither | Logs `'Invalid client role to configure'` and returns - **does not throw** |
|
|
122
|
+
| any other value | neither | **Silent.** No log, no throw. `this.queue` and `this.worker` both stay `undefined` |
|
|
123
|
+
|
|
124
|
+
> [!WARNING]
|
|
125
|
+
> A typo'd `role` (for example `'Worker'` instead of `'worker'`) hits the "any other value" row above - not the "missing" row. Nothing gets logged. The helper looks constructed, but `.queue` and `.worker` are both `undefined` until you call one and hit a `TypeError`.
|
|
107
126
|
|
|
108
|
-
`configureQueue()` and `configureWorker()` each guard on `queueName
|
|
127
|
+
`configureQueue()` and `configureWorker()` each guard on `queueName`. An empty `queueName` logs `'Invalid queue name'` (or `'Invalid worker name'`) and returns - `this.queue` / `this.worker` stay `undefined`. Neither path throws. A misconfigured helper fails later instead, the first time you call `.queue.add(...)` or use `.worker`.
|
|
109
128
|
|
|
110
129
|
> [!NOTE]
|
|
111
|
-
> An `'error'` event with no listener is re-thrown by Node's `EventEmitter` and crashes the process. `BullMQHelper` always attaches an `'error'` listener to both `queue` and `worker
|
|
130
|
+
> An `'error'` event with no listener is re-thrown by Node's `EventEmitter` and crashes the process. `BullMQHelper` always attaches an `'error'` listener to both `queue` and `worker`. A transient Redis error gets logged instead of taking the process down.
|
|
112
131
|
|
|
113
|
-
`onWorkerDataCompleted` and `onWorkerDataFail` run through an internal hook wrapper
|
|
132
|
+
`onWorkerDataCompleted` and `onWorkerDataFail` run through an internal hook wrapper. It absorbs both synchronous throws and rejected promises, logging them instead of propagating. A broken callback can't crash the worker or block the next job.
|
|
114
133
|
|
|
115
134
|
### Cluster queue name wrapping
|
|
116
135
|
|
|
@@ -175,7 +194,7 @@ await producer.close();
|
|
|
175
194
|
await consumer.close();
|
|
176
195
|
```
|
|
177
196
|
|
|
178
|
-
Calls `worker?.close()` then `queue?.close()` in sequence
|
|
197
|
+
Calls `worker?.close()` then `queue?.close()` in sequence. Both run even if the first fails, so a failing worker close never leaks the queue's Redis connection. If either fails, `close()` throws one `ApplicationError` aggregating both failure messages - only after both close attempts have run.
|
|
179
198
|
|
|
180
199
|
### API summary
|
|
181
200
|
|
|
@@ -233,7 +252,7 @@ const queue = new SequentialQueueHelper<string>({
|
|
|
233
252
|
| `onDataDequeue` | `(opts: { identifier: string; queueElement: TQueueElement<T> }) => ValueOrPromise<void>` | - | Fired after an element is shifted off `storage`. |
|
|
234
253
|
| `onStateChange` | `(opts: { identifier: string; from: TQueueStatus; to: TQueueStatus }) => ValueOrPromise<void>` | - | Fired on every state transition. |
|
|
235
254
|
|
|
236
|
-
All four hooks run through an internal wrapper that absorbs synchronous throws and promise rejections, logging them instead
|
|
255
|
+
All four hooks run through an internal wrapper that absorbs synchronous throws and promise rejections, logging them instead. A broken callback can't break the state machine.
|
|
237
256
|
|
|
238
257
|
### TQueueElement
|
|
239
258
|
|
|
@@ -241,7 +260,7 @@ All four hooks run through an internal wrapper that absorbs synchronous throws a
|
|
|
241
260
|
type TQueueElement<T> = { isLocked: boolean; payload: T };
|
|
242
261
|
```
|
|
243
262
|
|
|
244
|
-
`isLocked` marks whether the head element is currently being handed to `onMessage
|
|
263
|
+
`isLocked` marks whether the head element is currently being handed to `onMessage`. It is distinct from the queue-level `lock()`/`unlock()` state below.
|
|
245
264
|
|
|
246
265
|
### QueueStatuses and the state machine
|
|
247
266
|
|
|
@@ -262,21 +281,27 @@ WAITING ──enqueue──> PROCESSING ──done──> WAITING
|
|
|
262
281
|
| `QueueStatuses.LOCKED` | `'200_LOCKED'` | Paused. Elements can still be enqueued; nothing processes until `unlock()`. |
|
|
263
282
|
| `QueueStatuses.SETTLED` | `'300_SETTLED'` | Terminal. No further elements accepted. |
|
|
264
283
|
|
|
265
|
-
- **Why numeric prefixes.** `'000_...'`
|
|
284
|
+
- **Why numeric prefixes.** `'000_...'` through `'300_...'` make `state >= QueueStatuses.LOCKED` and `state > QueueStatuses.LOCKED` valid string comparisons. `lock()` and `unlock()` use exactly that to guard against re-entering from an invalid state.
|
|
266
285
|
- **Validate an arbitrary string** with `QueueStatuses.isValid(value)`.
|
|
267
286
|
|
|
268
287
|
### Processing loop
|
|
269
288
|
|
|
270
|
-
Internally, `_messageListener()` is a generator that loops `while (true) { yield this.handleMessage(); }`. `nextMessage()` calls `generator.next()` only when `state === WAITING
|
|
289
|
+
Internally, `_messageListener()` is a generator that loops `while (true) { yield this.handleMessage(); }`. `nextMessage()` calls `generator.next()` only when `state === WAITING`. Any other state logs a warning and returns without advancing. `handleMessage()`:
|
|
271
290
|
|
|
272
|
-
1. Reads the head element (`getElementAt(0)`)
|
|
291
|
+
1. Reads the head element (`getElementAt(0)`). Returns early if it's empty or already `isLocked`.
|
|
273
292
|
2. Transitions `WAITING -> PROCESSING` (skipped if already `LOCKED`/`SETTLED`).
|
|
274
293
|
3. Marks the head `isLocked = true`, adds it to `processingEvents`, and awaits `onMessage`.
|
|
275
294
|
4. Dequeues the completed element and removes it from `processingEvents`.
|
|
276
295
|
5. Transitions back to `WAITING` (skipped if `LOCKED`/`SETTLED`).
|
|
277
|
-
6.
|
|
296
|
+
6. Checks `storage`:
|
|
278
297
|
|
|
279
|
-
|
|
298
|
+
| `storage` after step 5 | What happens |
|
|
299
|
+
|---|---|
|
|
300
|
+
| Empty, and `settle()` was requested | Transitions to `SETTLED` and stops |
|
|
301
|
+
| Empty, and no `settle()` requested | Stops here. The queue idles until the next `enqueue()` |
|
|
302
|
+
| Not empty | Calls `nextMessage()` to continue the loop immediately |
|
|
303
|
+
|
|
304
|
+
A failing `onMessage` handler is logged and swallowed by the same hook wrapper described above. The element is still dequeued, and the loop continues. `onMessage` owns its own retry policy - the queue itself never retries.
|
|
280
305
|
|
|
281
306
|
### Methods
|
|
282
307
|
|
|
@@ -302,7 +327,7 @@ A failing `onMessage` handler is logged and swallowed by the same hook wrapper d
|
|
|
302
327
|
|
|
303
328
|
`Source ->` [`packages/helpers/src/modules/queue/internal/hf/helper.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/internal/hf/helper.ts)
|
|
304
329
|
|
|
305
|
-
- **O(1) FIFO, not a job queue.** A generic, high-frequency, single-consumer primitive for enqueue, dequeue, and cancel
|
|
330
|
+
- **O(1) FIFO, not a job queue.** A generic, high-frequency, single-consumer primitive for enqueue, dequeue, and cancel. It's backed by an array plus a moving head index - never `Array.shift()`, which is O(n).
|
|
306
331
|
- **No callbacks, no state machine, not thread-safe.** It is the low-level queue the pool helper's waiter list is built on, not a job-processing API.
|
|
307
332
|
- **Reach for `SequentialQueueHelper` instead** unless you specifically need a bare FIFO with cancellable entries.
|
|
308
333
|
|
|
@@ -327,7 +352,7 @@ const remaining = queue.drain(); // remove + return every live value, emptying t
|
|
|
327
352
|
| `cancel(opts)` | `void` | `opts: { node: IHfQueueNode<T> }`. Marks a queued node cancelled; idempotent. |
|
|
328
353
|
| `drain()` | `T[]` | Removes and returns every remaining live value in FIFO order, emptying the queue. |
|
|
329
354
|
|
|
330
|
-
Consumed entries are compacted out of the backing array once the consumed prefix exceeds 256 entries and covers at least half the array
|
|
355
|
+
Consumed entries are compacted out of the backing array once the consumed prefix exceeds 256 entries and covers at least half the array. This keeps every operation amortized O(1).
|
|
331
356
|
|
|
332
357
|
## MQTTClientHelper
|
|
333
358
|
|
|
@@ -356,7 +381,7 @@ const client = new MQTTClientHelper({
|
|
|
356
381
|
});
|
|
357
382
|
```
|
|
358
383
|
|
|
359
|
-
The constructor calls `configure()` automatically, which connects via `mqtt.connect(url, options)`. Calling `configure()` again
|
|
384
|
+
The constructor calls `configure()` automatically, which connects via `mqtt.connect(url, options)`. Calling `configure()` again is a no-op (logs and returns) whenever `this.client` already exists - even if that client later disconnected without you calling `close()`.
|
|
360
385
|
|
|
361
386
|
### IMQTTClientOptions
|
|
362
387
|
|
|
@@ -372,13 +397,13 @@ The constructor calls `configure()` automatically, which connects via `mqtt.conn
|
|
|
372
397
|
| `onClose` | `(error?: Error) => void` | - | Fired on the `'close'` event. |
|
|
373
398
|
|
|
374
399
|
> [!NOTE]
|
|
375
|
-
> At connect time, `MQTTClientHelper` logs the broker `url` through `redactUrlCredentials()` and `options` through `redactSecrets()`. A password embedded in the URL (`mqtts://user:hunter2@broker:8883`) never reaches the log
|
|
400
|
+
> At connect time, `MQTTClientHelper` logs the broker `url` through `redactUrlCredentials()` and `options` through `redactSecrets()`. A password embedded in the URL (`mqtts://user:hunter2@broker:8883`) never reaches the log. Only `mqtts://user:[REDACTED]@broker:8883` does.
|
|
376
401
|
|
|
377
402
|
### Methods
|
|
378
403
|
|
|
379
404
|
| Method | Returns | Description |
|
|
380
405
|
|--------|---------|--------------|
|
|
381
|
-
| `configure()` | `void` | Connects to the broker and wires `connect`/`disconnect`/`message`/`error`/`close` listeners. Called automatically by the constructor
|
|
406
|
+
| `configure()` | `void` | Connects to the broker and wires `connect`/`disconnect`/`message`/`error`/`close` listeners. Called automatically by the constructor. A no-op if `this.client` already exists. |
|
|
382
407
|
| `getClient()` | `mqtt.MqttClient \| undefined` | The underlying MQTT.js client. |
|
|
383
408
|
| `subscribe(opts)` | `Promise<string[]>` | `opts: { topics: string[] }`. Rejects with an `ApplicationError` (status 400) if the client is not connected. |
|
|
384
409
|
| `publish(opts)` | `Promise<{ topic, message }>` | `opts: { topic: string; message: string \| Buffer }`. Rejects with an `ApplicationError` (status 400) if the client is not connected. |
|
|
@@ -388,7 +413,7 @@ The constructor calls `configure()` automatically, which connects via `mqtt.conn
|
|
|
388
413
|
|
|
389
414
|
### "Invalid queue name" / "Invalid worker name"
|
|
390
415
|
|
|
391
|
-
**Cause:** `queueName` is empty when `BullMQHelper` configures a queue or worker. This is a **logged error, not a thrown exception** - `this.queue` / `this.worker` are
|
|
416
|
+
**Cause:** `queueName` is empty when `BullMQHelper` configures a queue or worker. This is a **logged error, not a thrown exception** - `this.queue` / `this.worker` are never assigned.
|
|
392
417
|
|
|
393
418
|
**Fix:** Provide a non-empty `queueName`:
|
|
394
419
|
|
|
@@ -402,7 +427,9 @@ new BullMQHelper({ queueName: 'my-email-queue', role: 'queue', identifier: 'x',
|
|
|
402
427
|
|
|
403
428
|
### "Invalid client role to configure"
|
|
404
429
|
|
|
405
|
-
**Cause:** `role` is missing or
|
|
430
|
+
**Cause:** `role` is missing or empty (falsy). Logged, not thrown.
|
|
431
|
+
|
|
432
|
+
**Also watch for:** a `role` that IS a non-empty string but not `'queue'`/`'worker'` (a typo like `'Worker'`) triggers **no log at all**. `configure()` silently does nothing, and `this.queue`/`this.worker` both stay `undefined` - see [Configuration lifecycle](#configuration-lifecycle).
|
|
406
433
|
|
|
407
434
|
**Fix:**
|
|
408
435
|
|
|
@@ -430,9 +457,9 @@ new MQTTClientHelper({ url: 'mqtt://localhost:1883', identifier: 'x', options: {
|
|
|
430
457
|
|
|
431
458
|
**Checklist:**
|
|
432
459
|
- `onMessage` must be provided - without it, the generator never initializes and nothing is ever processed.
|
|
433
|
-
- Check `getState()
|
|
434
|
-
- Check `autoDispatch
|
|
435
|
-
- Check `isSettled()
|
|
460
|
+
- Check `getState()`. A `LOCKED` queue accepts `enqueue()` but never processes. Call `unlock({ shouldProcessNextElement: true })` to resume.
|
|
461
|
+
- Check `autoDispatch`. If it's `false`, call `nextMessage()` manually after each `enqueue()`.
|
|
462
|
+
- Check `isSettled()`. A settled queue rejects new elements - construct a new instance instead.
|
|
436
463
|
|
|
437
464
|
## Import Reference
|
|
438
465
|
|