@venizia/ignis-docs 0.2.0 → 0.2.1-0

package/content/extensions/helpers/kafka/compile-binary.md

@@ -1,41 +1,41 @@
+---
+title: Kafka - Compiling to a Single Binary
+description: The platformaticWasmPlugin() Bun bundler plugin required when compiling an app that imports a Kafka helper
+difficulty: advanced
+---
+
 # Compiling to a Single Binary
 
-`bun build --compile` produces a standalone executable that crashes on startup when the application
-imports any Kafka helper, unless the build registers `platformaticWasmPlugin`.
+This page covers `platformaticWasmPlugin()`, the Bun bundler plugin required when compiling an application that imports any Kafka helper into a standalone `bun build --compile` executable.
 
-```
-ENOENT: no such file or directory, open '/$bunfs/dist/native.wasm'
-```
+## The problem
 
-The failure happens while the module graph is still loading -- before the IGNIS application boots, so
-no log line, no lifecycle hook, and no error handler of yours ever runs.
+- **A compiled binary crashes on startup** if the application imports any Kafka helper and the build did not register the plugin:
 
-## Why it happens
+  ```
+  ENOENT: no such file or directory, open '/$bunfs/dist/native.wasm'
+  ```
 
-`@platformatic/kafka` computes Kafka's CRC32C checksums and lz4/snappy compression in WebAssembly,
-through `@platformatic/wasm-utils`. The default entrypoint of that package reads the wasm payload
-from disk at module load time:
+- **The crash happens before your code runs.** It fires while the module graph is still loading - before the IGNIS application boots, so no log line, no lifecycle hook, and no error handler of yours ever executes.
 
-```javascript
-// @platformatic/wasm-utils/dist/index.js
-const wasm = readFileSync(new URL('../dist/native.wasm', import.meta.url));
-```
+## Why it happens
 
-`bun build --compile` embeds JavaScript modules only -- assets such as `native.wasm` are not carried
-into the executable. Inside the binary, `import.meta.url` resolves against the virtual `/$bunfs`
-filesystem, the file is not there, and the read throws.
+- **The codecs are WebAssembly, not native bindings.** `@platformatic/kafka` computes Kafka's CRC32C checksums and lz4/snappy compression in WebAssembly, through `@platformatic/wasm-utils`.
+- **The default entrypoint reads the wasm payload from disk** at module load time:
 
-Running from source (`bun run`, `bun .`) is unaffected: `node_modules` is on disk, so the read
-succeeds. The bug only exists in compiled binaries.
+  ```javascript
+  // @platformatic/wasm-utils/dist/index.js
+  const wasm = readFileSync(new URL('../dist/native.wasm', import.meta.url));
+  ```
 
-## The fix
+- **`bun build --compile` embeds JavaScript modules only.** Assets such as `native.wasm` are never carried into the executable. Inside the binary, `import.meta.url` resolves against the virtual `/$bunfs` filesystem, the file isn't there, and the read throws.
+- **Running from source is unaffected.** `bun run` / `bun .` keep `node_modules` on disk, so the read succeeds - the bug only exists in compiled binaries.
 
-`@platformatic/wasm-utils` ships a second entrypoint, `@platformatic/wasm-utils/bundled`, exposing
-the same API with the wasm payload inlined as base64 -- no filesystem read. `platformaticWasmPlugin`
-swaps one entrypoint for the other at bundle time.
+## The fix
 
-Compile through a Bun build script instead of the `bun build --compile` CLI, which cannot register
-plugins:
+- **A second entrypoint ships the payload inline.** `@platformatic/wasm-utils/bundled` exposes the same API with the wasm payload inlined as base64 - no filesystem read.
+- **`platformaticWasmPlugin()` swaps one entrypoint for the other** at bundle time, so the rest of your code and `@platformatic/kafka` itself need no changes.
+- **Compile through a Bun build script, not the CLI.** `bun build --compile` on the command line cannot register plugins:
 
 ```typescript
 // scripts/compile.ts
@@ -67,8 +67,7 @@ if (!built.success) {
 }
 ```
 
-The plugin resolves `@platformatic/wasm-utils/bundled` from the importing module's own directory, so
-it works with hoisted and isolated `node_modules` layouts alike, and pins no package version.
+- **The plugin resolves relative to the importing module's own directory** (`Bun.resolveSync` from `dirname(args.importer)`), so it works with hoisted and isolated `node_modules` layouts alike, and pins no package version.
 
 ## Verifying
 
@@ -83,10 +82,17 @@ The binary grows by roughly 76 KB, the base64 form of the 57 KB wasm module.
 
 ## Notes
 
-- Upgrading `@platformatic/kafka` does not remove the need for the plugin: every release to date,
-  including 2.6.1, imports the default `@platformatic/wasm-utils` entrypoint.
-- Applications that never import a Kafka helper need no plugin -- nothing pulls in
-  `@platformatic/wasm-utils`, and the plugin's resolver never fires.
-- Patching `node_modules` during the build achieves the same result, but mutates a dependency in
-  place, pins the store path to one version, and leaves the tree dirty when a build fails. The
-  plugin needs neither.
+- **Upgrading `@platformatic/kafka` does not remove the need for the plugin.** Every release to date imports the default `@platformatic/wasm-utils` entrypoint.
+- **Apps that never import a Kafka helper need no plugin.** Nothing pulls in `@platformatic/wasm-utils`, and the plugin's resolver never fires.
+- **Patching `node_modules` during the build achieves the same result**, but mutates a dependency in place, pins the store path to one version, and leaves the tree dirty when a build fails. The plugin needs neither.
+
+## See also
+
+- [Kafka Overview](./) - the four helpers this plugin protects
+- [Producer](./producer) / [Consumer](./consumer) / [Admin](./admin) - any of these importing into a compiled binary triggers this caveat
+- [Examples & Troubleshooting](./examples) - other Kafka connection and configuration errors
+
+**Files:**
+
+- [`packages/helpers/src/modules/queue/kafka/bundler/platformatic-wasm.plugin.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/bundler/platformatic-wasm.plugin.ts) - `platformaticWasmPlugin`
+- [`packages/helpers/src/modules/queue/kafka/bundler/common/constants.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/bundler/common/constants.ts) - `KafkaBundlerPluginNames`, `PlatformaticWasmSpecifiers`

package/content/extensions/helpers/kafka/consumer.md

@@ -1,6 +1,12 @@
+---
+title: Kafka Consumer
+description: KafkaConsumerHelper - message callbacks, automatic reconnect, and lag monitoring
+difficulty: intermediate
+---
+
 # Consumer
 
-The `KafkaConsumerHelper` wraps `@platformatic/kafka`'s `Consumer` with health tracking, graceful shutdown, message callbacks, consumer group event callbacks, and lag monitoring.
+The `KafkaConsumerHelper` wraps `@platformatic/kafka`'s `Consumer` with health tracking, graceful shutdown, message callbacks, consumer group event callbacks, automatic reconnect, and lag monitoring.
 
 ```typescript
 class KafkaConsumerHelper<
@@ -18,7 +24,7 @@ class KafkaConsumerHelper<
 | `newInstance(opts)` | `static newInstance<K,V,HK,HV>(opts): KafkaConsumerHelper<K,V,HK,HV>` | Factory method |
 | `getConsumer()` | `(): Consumer<K,V,HK,HV>` | Access the underlying `Consumer` |
 | `getStream()` | `(): MessagesStream \| null` | Get the active stream (after `start()`) |
-| `start(opts)` | `(opts: IKafkaConsumeStartOptions): Promise<void>` | Start consuming (creates stream, wires callbacks) |
+| `start(opts)` | `(opts: IKafkaConsumeStartOptions): Promise<void>` | Start consuming (creates stream, wires callbacks, drives automatic reconnect) |
 | `startLagMonitoring(opts)` | `(opts: { topics: string[]; interval?: number }): void` | Start periodic lag monitoring |
 | `stopLagMonitoring()` | `(): void` | Stop lag monitoring |
 | `isHealthy()` | `(): boolean` | `true` when at least one broker connected |
@@ -34,6 +40,8 @@ interface IKafkaConsumerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueTy
   extends IKafkaConnectionOptions
 ```
 
+Plus the shared [Connection & Authentication](./producer#connection--authentication) options (`bootstrapBrokers`, `clientId`, `retries`, `sasl`, `tls`, ...), documented once on the Producer page.
+
 ### Consumer Configuration
 
 | Option | Type | Default | Description |
@@ -47,8 +55,8 @@ interface IKafkaConsumerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueTy
 | `rebalanceTimeout` | `number` | `sessionTimeout` | Max time for rebalance. Defaults to the value of `sessionTimeout` |
 | `highWaterMark` | `number` | `1024` | Stream buffer size (messages) |
 | `minBytes` | `number` | `1` | Min bytes per fetch response |
-| `maxBytes` | `number` | -- | Max bytes per fetch response per partition |
-| `maxWaitTime` | `number` | -- | Max time (ms) broker waits for `minBytes` |
+| `maxBytes` | `number` | `10485760` (10 MB) | Max bytes per fetch response per partition |
+| `maxWaitTime` | `number` | `5000` | Max time (ms) broker waits for `minBytes` |
 | `metadataMaxAge` | `number` | `300000` | Metadata cache TTL (ms) |
 | `groupProtocol` | `'classic' \| 'consumer'` | `'classic'` | Consumer group protocol. `'consumer'` = KIP-848 (Kafka 3.7+) |
 | `groupInstanceId` | `string` | -- | Static group membership ID -- prevents rebalance on restart |
@@ -86,8 +94,6 @@ interface IKafkaConsumerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueTy
 | `onLag` | `TKafkaLagCallback` | Receives `{ lag }` (Offsets map) |
 | `onLagError` | `TKafkaLagErrorCallback` | Receives `{ error }` |
 
-Plus all [Connection Options](./#connection-options).
-
 ## Basic Example
 
 ```typescript
@@ -163,36 +169,43 @@ Stream 'error' event
   -> onMessageError({ error })  (no message available)
 ```
 
-- `onMessage` is the main processing callback -- do your business logic here
-- `onMessageDone` fires only after `onMessage` resolves successfully -- use for logging, metrics, etc.
-- `onMessageError` fires if `onMessage` throws -- use for error tracking. Note that errors from `onMessageDone` also trigger `onMessageError`
-- The stream `'error'` event also calls `onMessageError` (without `message` since it's a stream-level error), but only if `onMessageError` was provided
+- **`onMessage` is the main processing callback** -- do your business logic here.
+- **`onMessageDone` fires only after `onMessage` resolves successfully** -- use it for logging, metrics, and similar side effects. Note that errors thrown from `onMessageDone` also trigger `onMessageError`.
+- **`onMessageError` fires if `onMessage` throws**, and also for the stream's own `'error'` event (without a `message`, since it's a stream-level error).
+- **The stream `'error'` listener is always attached**, whether or not you pass `onMessageError`. An `EventEmitter` `'error'` event with zero listeners is rethrown as an uncaught exception -- without this listener, a pull-style consumer (`start()` + `getStream()`, no `onMessage`) would take the whole process down on the first broker drop.
 
 ## start()
 
-`start()` creates the consume stream and wires all message callbacks. It must be called explicitly after construction.
+`start()` creates the consume stream and wires all message callbacks. It must be called explicitly after construction, and it guards against duplicate starts -- calling it twice logs a warning and returns immediately.
 
 ```typescript
 interface IKafkaConsumeStartOptions {
   topics: string[];
-  mode?: MessagesStreamModeValue;         // Default: 'committed'
-  fallbackMode?: MessagesStreamFallbackModeValue; // Default: 'latest'
+  mode?: MessagesStreamModeValue;
+  fallbackMode?: MessagesStreamFallbackModeValue;
+  reconnectDelayMs?: number;
+  maxReconnectAttempts?: number;
 }
 ```
 
 | Mode | Description |
 |------|-------------|
-| `'committed'` | Resume from last committed offset. **Recommended for production** |
+| `'committed'` (default) | Resume from last committed offset. **Recommended for production** |
 | `'latest'` | Start from the latest offset (skip existing messages) |
 | `'earliest'` | Start from the beginning of the topic |
 | `'manual'` | Start from explicitly provided offsets |
 
 | Fallback | Description |
 |----------|-------------|
-| `'latest'` | Start from latest (default) -- ignore historical messages |
+| `'latest'` (default) | Start from latest -- ignore historical messages |
 | `'earliest'` | Start from beginning -- process all historical messages |
 | `'fail'` | Throw an error |
 
+| Reconnect option | Default | Description |
+|-------------------|---------|-------------|
+| `reconnectDelayMs` | `2000` | Delay before each automatic reconnect attempt |
+| `maxReconnectAttempts` | `5` | Consecutive reconnect attempts before the consume loop gives up |
+
 ```typescript
 // Production pattern
 await helper.start({ topics: ['orders'] });
@@ -200,15 +213,25 @@ await helper.start({ topics: ['orders'] });
 // Replay all historical messages
 await helper.start({ topics: ['orders'], mode: 'earliest' });
 
-// Custom mode
+// Custom mode and reconnect budget
 await helper.start({
   topics: ['orders'],
   mode: 'committed',
   fallbackMode: 'earliest',
+  reconnectDelayMs: 5_000,
+  maxReconnectAttempts: 10,
 });
 ```
 
-Guards against duplicate starts -- calling `start()` twice logs a warning and returns immediately.
+## Automatic reconnect
+
+When `onMessage` is provided, `start()` drives a background consume loop on top of the stream. That loop reconnects on its own:
+
+- **Only the callback-driven loop reconnects.** A pull-style consumer using `getStream()` or `consumer.consume()` directly owns its own retry logic -- the helper's automatic reconnect only runs inside `startConsumeLoop`, which is wired exclusively when `onMessage` is set.
+- **A full broker outage marks the session stale.** When every broker disconnects (`getConnectedBrokerCount()` drops to `0`, via `client:broker:disconnect` or `client:broker:failed`), the helper destroys the current stream immediately instead of waiting for it to error out on its own.
+- **Reconnect rebuilds the client after a stale session.** The next attempt, after `reconnectDelayMs`, constructs a brand-new `@platformatic/kafka` `Consumer` with the original constructor options and swaps it in -- a fresh client forces a clean group rejoin rather than reusing session state Kafka has likely already expired. Active lag monitoring is re-armed on the new client automatically.
+- **Attempts are capped.** After `maxReconnectAttempts` consecutive failures, the consume loop exits and logs an error. Message processing stops until you call `start()` again with a new set of options.
+- **Every retry is logged**, including the attempt number, delay, and current connected-broker count, so a stuck reconnect loop is visible in application logs without extra instrumentation.
 
 ## Lag Monitoring
 
@@ -220,9 +243,7 @@ helper.startLagMonitoring({ topics: ['orders'], interval: 10_000 });
 helper.stopLagMonitoring();
 ```
 
-Lag data is delivered via the `onLag` callback. Errors via `onLagError`.
-
-Guards against duplicate starts -- calling `startLagMonitoring()` twice logs a warning.
+Lag data is delivered via the `onLag` callback. Errors via `onLagError`. `interval` defaults to `30000` ms. Guards against duplicate starts -- calling `startLagMonitoring()` twice logs a warning.
 
 For one-time lag checks, use the underlying consumer directly:
 
@@ -383,3 +404,16 @@ Topic "orders" (3 partitions)
 
 > [!TIP]
 > Create topics with enough partitions for your expected parallelism. You can increase partitions later with `admin.createPartitions()`, but you cannot decrease them.
+
+## See also
+
+- [Kafka Overview](./) - the four helpers, shared health/close API, and the compile-binary caveat
+- [Producer](./producer) - the sending side, plus the shared Connection & Authentication options
+- [Admin](./admin) - create topics and inspect consumer groups from outside the running consumer
+- [Examples & Troubleshooting](./examples) - lag monitoring, exactly-once, and common connection errors
+
+**Files:**
+
+- [`packages/helpers/src/modules/queue/kafka/consumer.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/consumer.ts) - `KafkaConsumerHelper`, consume loop, automatic reconnect
+- [`packages/helpers/src/modules/queue/kafka/base.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/base.ts) - `BaseKafkaHelper`, shared health tracking and shutdown
+- [`packages/helpers/src/modules/queue/kafka/common/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/common/types.ts) - `IKafkaConsumerOptions`, `IKafkaConsumeStartOptions`

package/content/extensions/helpers/kafka/examples.md

@@ -1,3 +1,9 @@
+---
+title: Kafka Examples & Troubleshooting
+description: End-to-end producer, consumer, admin, and IoC examples, plus a common-error lookup table
+difficulty: intermediate
+---
+
 # Examples & Troubleshooting
 
 Complete examples and common issue resolution for the Kafka helpers.
@@ -260,7 +266,7 @@ import {
   KafkaAdminHelper,
 } from '@venizia/ignis-helpers/kafka';
 import { stringSerializers, stringDeserializers } from '@platformatic/kafka';
-import { inject, injectable } from '@venizia/ignis-inversion';
+import { inject } from '@venizia/ignis-inversion';
 
 // Register helpers in the IoC container
 app.bind('kafka.producer').to(
@@ -286,7 +292,6 @@ app.bind('kafka.consumer').to(
 );
 
 // Inject into services
-@injectable()
 export class OrderEventService {
   constructor(
     @inject({ key: 'kafka.producer' }) private producer: KafkaProducerHelper,
@@ -342,20 +347,20 @@ environment:
 - `EXTERNAL` -- used for client connections from outside Docker
 - `CONTROLLER` -- used for KRaft controller communication
 
-## See Also
+## See also
 
-- **Kafka Pages:**
-  - [Overview & Fundamentals](./) -- Connection, serialization, constants, compression
-  - [Producer](./producer) -- Producer helper, transactions, API reference
-  - [Consumer](./consumer) -- Consumer helper, callbacks, lag monitoring, API reference
-  - [Admin](./admin) -- Admin helper & API reference
-  - [Schema Registry](./schema-registry) -- Schema registry helper
+- [Kafka Overview](./) - the four helpers, shared health/close API, and the compile-binary caveat
+- [Producer](./producer) - connection & SASL setup, serialization, compression, transactions
+- [Consumer](./consumer) - message callbacks, automatic reconnect, lag monitoring
+- [Admin](./admin) - topic, group, offset, ACL, and quota management
+- [Schema Registry](./schema-registry) - schema-validated serialization
+- [Compiling to a Single Binary](./compile-binary) - required if any example on this page ships inside a `bun build --compile` binary
+- [Queue Helpers](../queue/) - BullMQ, MQTT, and the in-memory queue
+- [Redis Helper](../redis/) - Redis connection management
 
-- **Other Helpers:**
-  - [Queue Helper](../queue/) -- BullMQ, MQTT, and in-memory queues
-  - [Redis Helper](../redis/) -- Redis connection management
+**Files:**
 
-- **External Resources:**
-  - [@platformatic/kafka](https://github.com/platformatic/kafka) -- Underlying Kafka client library
-  - [Apache Kafka Documentation](https://kafka.apache.org/documentation/) -- Official Kafka docs
-  - [KIP-848](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848) -- New consumer group protocol
+- [`packages/helpers/src/modules/queue/kafka/producer.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/producer.ts) - `KafkaProducerHelper`
+- [`packages/helpers/src/modules/queue/kafka/consumer.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/consumer.ts) - `KafkaConsumerHelper`
+- [`packages/helpers/src/modules/queue/kafka/admin.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/admin.ts) - `KafkaAdminHelper`
+- [`packages/helpers/src/modules/queue/kafka/index.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/index.ts) - the `/kafka` sub-path barrel