@venizia/ignis-docs 0.2.0 → 0.2.1-1

package/content/extensions/helpers/kafka/compile-binary.md

@@ -1,45 +1,16 @@
-# Compiling to a Single Binary
-
-`bun build --compile` produces a standalone executable that crashes on startup when the application
-imports any Kafka helper, unless the build registers `platformaticWasmPlugin`.
-
-```
-ENOENT: no such file or directory, open '/$bunfs/dist/native.wasm'
-```
-
-The failure happens while the module graph is still loading -- before the IGNIS application boots, so
-no log line, no lifecycle hook, and no error handler of yours ever runs.
+---
+title: Kafka - Compiling to a Single Binary
+description: The platformaticKafkaPlugins() Bun bundler plugins required when compiling an app that imports a Kafka helper
+difficulty: advanced
+---
 
-## Why it happens
-
-`@platformatic/kafka` computes Kafka's CRC32C checksums and lz4/snappy compression in WebAssembly,
-through `@platformatic/wasm-utils`. The default entrypoint of that package reads the wasm payload
-from disk at module load time:
-
-```javascript
-// @platformatic/wasm-utils/dist/index.js
-const wasm = readFileSync(new URL('../dist/native.wasm', import.meta.url));
-```
-
-`bun build --compile` embeds JavaScript modules only -- assets such as `native.wasm` are not carried
-into the executable. Inside the binary, `import.meta.url` resolves against the virtual `/$bunfs`
-filesystem, the file is not there, and the read throws.
-
-Running from source (`bun run`, `bun .`) is unaffected: `node_modules` is on disk, so the read
-succeeds. The bug only exists in compiled binaries.
-
-## The fix
-
-`@platformatic/wasm-utils` ships a second entrypoint, `@platformatic/wasm-utils/bundled`, exposing
-the same API with the wasm payload inlined as base64 -- no filesystem read. `platformaticWasmPlugin`
-swaps one entrypoint for the other at bundle time.
+# Compiling to a Single Binary
 
-Compile through a Bun build script instead of the `bun build --compile` CLI, which cannot register
-plugins:
+`platformaticKafkaPlugins()` returns every Bun bundler plugin `@platformatic/kafka` needs to survive `bun build --compile`. Register them whenever you compile an application that imports a Kafka helper. Skip them, and the binary crashes on its first run.
 
 ```typescript
 // scripts/compile.ts
-import { platformaticWasmPlugin } from '@venizia/ignis-helpers/kafka';
+import { platformaticKafkaPlugins } from '@venizia/ignis-helpers/kafka';
 
 const built = await Bun.build({
   entrypoints: ['./dist/index.js'],
@@ -50,7 +21,7 @@ const built = await Bun.build({
     target: process.env.BUN_TARGET ?? 'bun-linux-x64',
     outfile: './dist/bin',
   },
-  plugins: [platformaticWasmPlugin()],
+  plugins: platformaticKafkaPlugins(),
 });
 
 if (!built.success) {
@@ -59,6 +30,8 @@ if (!built.success) {
 }
 ```
 
+`bun build --compile` on the command line cannot register plugins. Compile through a build script instead:
+
 ```json
 {
   "scripts": {
@@ -67,26 +40,84 @@ if (!built.success) {
 }
 ```
 
-The plugin resolves `@platformatic/wasm-utils/bundled` from the importing module's own directory, so
-it works with hoisted and isolated `node_modules` layouts alike, and pins no package version.
+## The two crashes
 
-## Verifying
+| Crash | Plugin that prevents it |
+|---|---|
+| `ENOENT: no such file or directory, open '/$bunfs/dist/native.wasm'` | `platformaticWasmPlugin()` |
+| `Cannot find package 'ajv-draft-04' from '/$bunfs/root/index.js'` | `platformaticRequirePlugin()` |
+
+Both fire while the module graph is still loading, before the IGNIS application boots. No log line, no lifecycle hook, and no error handler of yours ever runs.
+
+Running from source is unaffected. `bun run` and `bun .` keep `node_modules` on disk, so both reads succeed. The crashes exist only in compiled binaries.
 
-A correctly built binary contains no reference to the wasm file on disk:
+## Why the wasm read fails
+
+`@platformatic/kafka` computes Kafka's CRC32C checksums and lz4/snappy compression in WebAssembly, through `@platformatic/wasm-utils`. That package reads its payload from disk at module load:
+
+```javascript
+// @platformatic/wasm-utils/dist/index.js
+const wasm = readFileSync(new URL('../dist/native.wasm', import.meta.url));
+```
+
+`bun build --compile` embeds JavaScript modules only. Assets such as `native.wasm` never reach the executable, so `import.meta.url` resolves against the virtual `/$bunfs` filesystem and the read throws.
+
+`platformaticWasmPlugin()` redirects the import to `@platformatic/wasm-utils/bundled`, the same API with the payload inlined as base64.
+
+## Why the package lookup fails
+
+Since 2.8.0, `@platformatic/kafka` resolves two dependencies at module scope through `createRequire`:
+
+```javascript
+// @platformatic/kafka/dist/registries/confluent-schema-registry.js
+const require = createRequire(import.meta.url);
+const AjvDraft04 = require('ajv-draft-04');
+const draft06MetaSchema = require('ajv/dist/refs/json-schema-draft-06.json');
+```
+
+Three facts make that fatal in a binary:
+
+- **The bundler cannot see through `createRequire`.** Neither module is embedded, and the bare specifier survives verbatim into the binary.
+- **The module always loads.** `dist/index.js` re-exports the schema registry, so importing anything from the package runs those two lines. A schema registry you never use still crashes the binary.
+- **A compiled binary has no `node_modules`.** `import.meta.url` resolves against `/$bunfs` and the lookup fails.
+
+`platformaticRequirePlugin()` rewrites each module-scope `require()` into a static import at bundle time, so the bundler pulls the module into the binary.
+
+> [!WARNING]
+> Installing `ajv-draft-04` as a direct dependency does not fix this. The defect is the resolution mechanism, not a missing package, so the second line fails identically.
+
+## Verifying
 
 ```bash
-grep -c 'native.wasm' ./dist/bin   # 0 -- the payload is inlined
-./dist/bin                         # boots instead of throwing ENOENT
+./dist/bin                           # boots instead of crashing
+grep -ac 'native.wasm'  ./dist/bin   # 0 - the wasm payload is inlined
+grep -ac 'ajv-draft-04' ./dist/bin   # 0 - the package is bundled in
 ```
 
-The binary grows by roughly 76 KB, the base64 form of the 57 KB wasm module.
+The last count is meaningful only for a minified build. An unminified bundle keeps package paths in module comments.
+
+The binary grows by roughly 76 KB for the base64 wasm module, plus the size of the two hoisted packages.
 
 ## Notes
 
-- Upgrading `@platformatic/kafka` does not remove the need for the plugin: every release to date,
-  including 2.6.1, imports the default `@platformatic/wasm-utils` entrypoint.
-- Applications that never import a Kafka helper need no plugin -- nothing pulls in
-  `@platformatic/wasm-utils`, and the plugin's resolver never fires.
-- Patching `node_modules` during the build achieves the same result, but mutates a dependency in
-  place, pins the store path to one version, and leaves the tree dirty when a build fails. The
-  plugin needs neither.
+- **Register the pair, not the individual plugins.** `platformaticKafkaPlugins()` picks up any future addition without another edit to your compile script. `platformaticWasmPlugin()` and `platformaticRequirePlugin()` stay exported for a build that needs one alone.
+- **Upgrading `@platformatic/kafka` does not remove the need for the plugins.** Every release to date imports the default `@platformatic/wasm-utils` entrypoint, and 2.8.0 introduced the `createRequire` calls in a minor release.
+- **An unresolvable specifier now fails the build**, with Bun's own resolution error and `built.success === false`. That is the point of hoisting to a static import: you learn at build time instead of on the first run in production.
+- **Both plugins resolve relative to the importing module's own directory.** That works with hoisted and isolated `node_modules` layouts alike, and pins no package version.
+- **Apps that never import a Kafka helper need no plugins.** Nothing pulls in `@platformatic/wasm-utils` or the schema registry, and neither plugin ever fires.
+- **`platformaticRequirePlugin()` skips `protobufjs` and `@node-rs/crc32`.** Both are optional peers of `@platformatic/kafka`, and a static import would break builds that do not install them.
+- **Patching `node_modules` during the build achieves the same result, but at a cost.** It mutates a dependency in place, pins the store path to one version, and leaves the tree dirty when a build fails. The plugins need none of that.
+
+## See also
+
+- [Kafka Overview](./) - the four helpers these plugins protect
+- [Producer](./producer) / [Consumer](./consumer) / [Admin](./admin) - any of these importing into a compiled binary triggers this caveat
+- [Schema Registry](./schema-registry) - the helper that wraps the module the second crash comes from
+- [Examples & Troubleshooting](./examples) - other Kafka connection and configuration errors
+
+**Files:**
+
+- [`packages/helpers/src/modules/queue/kafka/bundler/plugins.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/bundler/plugins.ts) - `platformaticKafkaPlugins`
+- [`packages/helpers/src/modules/queue/kafka/bundler/platformatic-wasm.plugin.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/bundler/platformatic-wasm.plugin.ts) - `platformaticWasmPlugin`
+- [`packages/helpers/src/modules/queue/kafka/bundler/platformatic-require.plugin.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/bundler/platformatic-require.plugin.ts) - `platformaticRequirePlugin`
+- [`packages/helpers/src/modules/queue/kafka/bundler/common/constants.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/bundler/common/constants.ts) - `KafkaBundlerPluginNames`, `PlatformaticWasmSpecifiers`, `PlatformaticRequireSpecifiers`

package/content/extensions/helpers/kafka/consumer.md

@@ -1,6 +1,12 @@
+---
+title: Kafka Consumer
+description: KafkaConsumerHelper - message callbacks, automatic reconnect, and lag monitoring
+difficulty: intermediate
+---
+
 # Consumer
 
-The `KafkaConsumerHelper` wraps `@platformatic/kafka`'s `Consumer` with health tracking, graceful shutdown, message callbacks, consumer group event callbacks, and lag monitoring.
+The `KafkaConsumerHelper` wraps `@platformatic/kafka`'s `Consumer` with health tracking, graceful shutdown, message callbacks, group callbacks, automatic reconnect, and lag monitoring.
 
 ```typescript
 class KafkaConsumerHelper<
@@ -18,7 +24,7 @@ class KafkaConsumerHelper<
 | `newInstance(opts)` | `static newInstance<K,V,HK,HV>(opts): KafkaConsumerHelper<K,V,HK,HV>` | Factory method |
 | `getConsumer()` | `(): Consumer<K,V,HK,HV>` | Access the underlying `Consumer` |
 | `getStream()` | `(): MessagesStream \| null` | Get the active stream (after `start()`) |
-| `start(opts)` | `(opts: IKafkaConsumeStartOptions): Promise<void>` | Start consuming (creates stream, wires callbacks) |
+| `start(opts)` | `(opts: IKafkaConsumeStartOptions): Promise<void>` | Start consuming (creates stream, wires callbacks, drives automatic reconnect) |
 | `startLagMonitoring(opts)` | `(opts: { topics: string[]; interval?: number }): void` | Start periodic lag monitoring |
 | `stopLagMonitoring()` | `(): void` | Stop lag monitoring |
 | `isHealthy()` | `(): boolean` | `true` when at least one broker connected |
@@ -34,26 +40,28 @@ interface IKafkaConsumerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueTy
   extends IKafkaConnectionOptions
 ```
 
+Plus the shared [Connection & Authentication](./producer#connection--authentication) options (`bootstrapBrokers`, `clientId`, `retries`, `sasl`, `tls`, ...), documented once on the Producer page.
+
 ### Consumer Configuration
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `groupId` | `string` | -- | Consumer group ID. **Required** |
+| `groupId` | `string` | - | Consumer group ID. **Required** |
 | `identifier` | `string` | `'kafka-consumer'` | Scoped logging identifier |
-| `deserializers` | `Partial<Deserializers<K,V,HK,HV>>` | -- | Key/value/header deserializers |
+| `deserializers` | `Partial<Deserializers<K,V,HK,HV>>` | - | Key/value/header deserializers |
 | `autocommit` | `boolean \| number` | `false` | Auto-commit offsets. `true` = default interval, `number` = custom ms |
-| `sessionTimeout` | `number` | `60000` | Session timeout -- consumer removed from group if no heartbeat |
-| `heartbeatInterval` | `number` | `10000` | Heartbeat interval -- must be less than `sessionTimeout` |
+| `sessionTimeout` | `number` | `60000` | Session timeout - consumer removed from group if no heartbeat |
+| `heartbeatInterval` | `number` | `10000` | Heartbeat interval - must be less than `sessionTimeout` |
 | `rebalanceTimeout` | `number` | `sessionTimeout` | Max time for rebalance. Defaults to the value of `sessionTimeout` |
 | `highWaterMark` | `number` | `1024` | Stream buffer size (messages) |
 | `minBytes` | `number` | `1` | Min bytes per fetch response |
-| `maxBytes` | `number` | -- | Max bytes per fetch response per partition |
-| `maxWaitTime` | `number` | -- | Max time (ms) broker waits for `minBytes` |
+| `maxBytes` | `number` | `10485760` (10 MB) | Max bytes per fetch response per partition |
+| `maxWaitTime` | `number` | `5000` | Max time (ms) broker waits for `minBytes` |
 | `metadataMaxAge` | `number` | `300000` | Metadata cache TTL (ms) |
 | `groupProtocol` | `'classic' \| 'consumer'` | `'classic'` | Consumer group protocol. `'consumer'` = KIP-848 (Kafka 3.7+) |
-| `groupInstanceId` | `string` | -- | Static group membership ID -- prevents rebalance on restart |
+| `groupInstanceId` | `string` | - | Static group membership ID - prevents rebalance on restart |
 | `shutdownTimeout` | `number` | `30000` | Graceful shutdown timeout in ms |
-| `registry` | `SchemaRegistry` | -- | Schema registry for auto deser |
+| `registry` | `SchemaRegistry` | - | Schema registry for auto deser |
 
 ### Lifecycle Callbacks
 
@@ -86,8 +94,6 @@ interface IKafkaConsumerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueTy
 | `onLag` | `TKafkaLagCallback` | Receives `{ lag }` (Offsets map) |
 | `onLagError` | `TKafkaLagErrorCallback` | Receives `{ error }` |
 
-Plus all [Connection Options](./#connection-options).
-
 ## Basic Example
 
 ```typescript
@@ -163,36 +169,44 @@ Stream 'error' event
   -> onMessageError({ error })  (no message available)
 ```
 
-- `onMessage` is the main processing callback -- do your business logic here
-- `onMessageDone` fires only after `onMessage` resolves successfully -- use for logging, metrics, etc.
-- `onMessageError` fires if `onMessage` throws -- use for error tracking. Note that errors from `onMessageDone` also trigger `onMessageError`
-- The stream `'error'` event also calls `onMessageError` (without `message` since it's a stream-level error), but only if `onMessageError` was provided
+- **`onMessage` is the main processing callback.** Put your business logic here.
+- **`onMessageDone` fires only after `onMessage` resolves successfully.** Use it for logging, metrics, and similar side effects. An error thrown from `onMessageDone` also triggers `onMessageError`.
+- **`onMessageError` fires if `onMessage` throws.** It also fires for the stream's own `'error'` event. That case carries no `message` - it's a stream-level error, not a per-message one.
+- **The stream `'error'` listener is always attached**, whether or not you pass `onMessageError`. An `EventEmitter` `'error'` event with zero listeners becomes an uncaught exception.
+- **Pull-style consumers benefit too.** Without this listener, a consumer using `start()` + `getStream()` with no `onMessage` would take down the whole process on the first broker drop.
 
 ## start()
 
-`start()` creates the consume stream and wires all message callbacks. It must be called explicitly after construction.
+`start()` creates the consume stream and wires all message callbacks. Call it explicitly after construction - the constructor never calls it for you. It also guards against duplicate starts: calling it twice logs a warning and returns immediately.
 
 ```typescript
 interface IKafkaConsumeStartOptions {
   topics: string[];
-  mode?: MessagesStreamModeValue;         // Default: 'committed'
-  fallbackMode?: MessagesStreamFallbackModeValue; // Default: 'latest'
+  mode?: MessagesStreamModeValue;
+  fallbackMode?: MessagesStreamFallbackModeValue;
+  reconnectDelayMs?: number;
+  maxReconnectAttempts?: number;
 }
 ```
 
 | Mode | Description |
 |------|-------------|
-| `'committed'` | Resume from last committed offset. **Recommended for production** |
+| `'committed'` (default) | Resume from last committed offset. **Recommended for production** |
 | `'latest'` | Start from the latest offset (skip existing messages) |
 | `'earliest'` | Start from the beginning of the topic |
 | `'manual'` | Start from explicitly provided offsets |
 
 | Fallback | Description |
 |----------|-------------|
-| `'latest'` | Start from latest (default) -- ignore historical messages |
-| `'earliest'` | Start from beginning -- process all historical messages |
+| `'latest'` (default) | Start from latest - ignore historical messages |
+| `'earliest'` | Start from beginning - process all historical messages |
 | `'fail'` | Throw an error |
 
+| Reconnect option | Default | Description |
+|-------------------|---------|-------------|
+| `reconnectDelayMs` | `2000` | Delay before each automatic reconnect attempt |
+| `maxReconnectAttempts` | `5` | Consecutive reconnect attempts before the consume loop gives up |
+
 ```typescript
 // Production pattern
 await helper.start({ topics: ['orders'] });
@@ -200,15 +214,27 @@ await helper.start({ topics: ['orders'] });
 // Replay all historical messages
 await helper.start({ topics: ['orders'], mode: 'earliest' });
 
-// Custom mode
+// Custom mode and reconnect budget
 await helper.start({
   topics: ['orders'],
   mode: 'committed',
   fallbackMode: 'earliest',
+  reconnectDelayMs: 5_000,
+  maxReconnectAttempts: 10,
 });
 ```
 
-Guards against duplicate starts -- calling `start()` twice logs a warning and returns immediately.
+## Automatic reconnect
+
+When `onMessage` is provided, `start()` drives a background consume loop on top of the stream. That loop reconnects on its own:
+
+- **Only the callback-driven loop reconnects.** A pull-style consumer using `getStream()` or `consumer.consume()` directly owns its own retry logic. The helper's automatic reconnect runs only inside `startConsumeLoop`, which is wired exclusively when `onMessage` is set.
+- **A full broker outage marks the session stale.** This happens when every broker disconnects - `getConnectedBrokerCount()` drops to `0` via `client:broker:disconnect` or `client:broker:failed` events.
+- **The helper reacts immediately.** It destroys the current stream right away, instead of waiting for the stream to error out on its own.
+- **Reconnect rebuilds the client after a stale session.** The next attempt, after `reconnectDelayMs`, constructs a brand-new `@platformatic/kafka` `Consumer` with the original options and swaps it in.
+- **A fresh client forces a clean group rejoin**, instead of reusing session state Kafka has likely already expired. Lag monitoring re-arms automatically on the new client.
+- **Attempts are capped.** After `maxReconnectAttempts` consecutive failures, the consume loop exits and logs an error. Message processing stops until you call `start()` again with a new set of options.
+- **Every retry is logged**: the attempt number, the delay, and the current connected-broker count. A stuck reconnect loop stays visible in application logs without extra instrumentation.
 
 ## Lag Monitoring
 
@@ -220,9 +246,7 @@ helper.startLagMonitoring({ topics: ['orders'], interval: 10_000 });
 helper.stopLagMonitoring();
 ```
 
-Lag data is delivered via the `onLag` callback. Errors via `onLagError`.
-
-Guards against duplicate starts -- calling `startLagMonitoring()` twice logs a warning.
+Lag data arrives through the `onLag` callback; errors through `onLagError`. `interval` defaults to `30000` ms. Calling `startLagMonitoring()` twice logs a warning instead of starting a second poll loop.
 
 For one-time lag checks, use the underlying consumer directly:
 
@@ -357,7 +381,7 @@ consumer.generationId;   // number
 consumer.assignments;    // GroupAssignment[] | null
 consumer.isActive();     // boolean
 
-// Static membership -- prevents rebalance on restart
+// Static membership - prevents rebalance on restart
 const helper = KafkaConsumerHelper.newInstance({
   ...
   groupInstanceId: 'worker-1',
@@ -383,3 +407,16 @@ Topic "orders" (3 partitions)
 
 > [!TIP]
 > Create topics with enough partitions for your expected parallelism. You can increase partitions later with `admin.createPartitions()`, but you cannot decrease them.
+
+## See also
+
+- [Kafka Overview](./) - the four helpers, shared health/close API, and the compile-binary caveat
+- [Producer](./producer) - the sending side, plus the shared Connection & Authentication options
+- [Admin](./admin) - create topics and inspect consumer groups from outside the running consumer
+- [Examples & Troubleshooting](./examples) - IoC wiring and the common connection-error lookup table
+
+**Files:**
+
+- [`packages/helpers/src/modules/queue/kafka/consumer.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/consumer.ts) - `KafkaConsumerHelper`, consume loop, automatic reconnect
+- [`packages/helpers/src/modules/queue/kafka/base.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/base.ts) - `BaseKafkaHelper`, shared health tracking and shutdown
+- [`packages/helpers/src/modules/queue/kafka/common/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/common/types.ts) - `IKafkaConsumerOptions`, `IKafkaConsumeStartOptions`