@venizia/ignis-docs 0.2.0 → 0.2.1-1

package/content/extensions/helpers/kafka/producer.md

@@ -1,3 +1,9 @@
+---
+title: Kafka Producer
+description: KafkaProducerHelper - connection, SASL, serialization, compression, and transactions
+difficulty: intermediate
+---
+
 # Producer
 
 The `KafkaProducerHelper` wraps `@platformatic/kafka`'s `Producer` with health tracking, graceful shutdown, broker event callbacks, and a transaction helper.
@@ -34,19 +40,61 @@ interface IKafkaProducerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueTy
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
 | `identifier` | `string` | `'kafka-producer'` | Scoped logging identifier |
-| `serializers` | `Partial<Serializers<K,V,HK,HV>>` | -- | Key/value/header serializers |
-| `compression` | `CompressionAlgorithmValue` | -- | `'none'`, `'gzip'`, `'snappy'`, `'lz4'`, `'zstd'` |
-| `acks` | `TKafkaAcks` | -- | Acknowledgment level: `0`, `1`, or `-1` |
-| `idempotent` | `boolean` | -- | Enable idempotent producer (exactly-once within partition) |
-| `transactionalId` | `string` | -- | Transactional ID for exactly-once across partitions |
-| `strict` | `boolean` | `true` | Strict mode -- fail on unknown topics |
+| `serializers` | `Partial<Serializers<K,V,HK,HV>>` | - | Key/value/header serializers |
+| `compression` | `CompressionAlgorithmValue` | - | `'none'`, `'gzip'`, `'snappy'`, `'lz4'`, `'zstd'` |
+| `acks` | `TKafkaAcks` | - | Acknowledgment level: `0` (none), `1` (leader), `-1` (all) |
+| `idempotent` | `boolean` | - | Enable idempotent producer (exactly-once within partition) |
+| `transactionalId` | `string` | - | Transactional ID for exactly-once across partitions |
+| `strict` | `boolean` | `true` | Strict mode - fail on unknown topics |
 | `autocreateTopics` | `boolean` | `false` | Auto-create topics on first produce |
 | `shutdownTimeout` | `number` | `30000` | Graceful shutdown timeout in ms |
-| `registry` | `SchemaRegistry` | -- | Schema registry for auto ser/deser |
-| `onBrokerConnect` | `TKafkaBrokerEventCallback` | -- | Called when broker connects |
-| `onBrokerDisconnect` | `TKafkaBrokerEventCallback` | -- | Called when broker disconnects |
+| `registry` | `SchemaRegistry` | - | Schema registry for auto ser/deser |
+| `onBrokerConnect` | `TKafkaBrokerEventCallback` | - | Called when broker connects |
+| `onBrokerDisconnect` | `TKafkaBrokerEventCallback` | - | Called when broker disconnects |
+
+Plus the shared [Connection & Authentication](#connection--authentication) options below, all inherited from `IKafkaConnectionOptions`.
+
+## Connection & Authentication
+
+`IKafkaProducerOptions`, `IKafkaConsumerOptions`, and `IKafkaAdminOptions` all extend `IKafkaConnectionOptions` - this section applies to all three.
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `bootstrapBrokers` | `string[]` | - | Broker addresses (`host:port`). **Required** |
+| `clientId` | `string` | - | Unique client identifier. **Required** |
+| `retries` | `number` | `3` | Connection retries before failing |
+| `retryDelay` | `number` | `1000` | Delay between retries (ms) |
+| `sasl` | `SASLOptions` | - | SASL authentication |
+| `tls` / `ssl` | `TLSConnectionOptions` | - | TLS options (`ssl` is an alias for `tls`) |
+| `connectTimeout` | `number` | - | TCP connection timeout (ms) |
+| `requestTimeout` | `number` | - | Kafka request timeout (ms) |
+
+**SASL mechanisms** (`@platformatic/kafka` supports five):
+
+| Mechanism | Use case |
+|-----------|----------|
+| `PLAIN` | Username/password (pair with `tls` in production) |
+| `SCRAM-SHA-256` / `SCRAM-SHA-512` | Challenge-response - password never sent in plaintext |
+| `OAUTHBEARER` | Token-based - Azure Event Hubs, Confluent Cloud. `token` accepts a string or an async function that returns one |
+| `GSSAPI` | Kerberos authentication |
 
-Plus all [Connection Options](./#connection-options).
+```typescript
+import fs from 'node:fs';
+import { KafkaProducerHelper } from '@venizia/ignis-helpers/kafka';
+
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['broker1:9093', 'broker2:9093'],
+  clientId: 'order-producer',
+  sasl: { mechanism: 'SCRAM-SHA-512', username: 'kafka-user', password: 'kafka-password' },
+  tls: {
+    ca: fs.readFileSync('/path/to/ca.pem'),
+    cert: fs.readFileSync('/path/to/client-cert.pem'),
+    key: fs.readFileSync('/path/to/client-key.pem'),
+  },
+  connectTimeout: 30_000,
+  requestTimeout: 30_000,
+});
+```
 
 ## Basic Example
 
@@ -92,6 +140,74 @@ await helper.close();
 await helper.close({ isForce: true });
 ```
 
+## Serialization
+
+`@platformatic/kafka`'s default wire format is `Buffer`. The helpers default the generic types to `string`, but that's a type default only. Pass serializers explicitly, or messages travel as raw `Buffer`.
+
+| Export | Type | Description |
+|--------|------|-------------|
+| `stringSerializer` / `stringDeserializer` | `Serializer<string>` / `Deserializer<string>` | UTF-8 string <-> `Buffer` |
+| `jsonSerializer` / `jsonDeserializer` | `Serializer<T>` / `Deserializer<T>` | `JSON.stringify` / `JSON.parse` + UTF-8 |
+| `stringSerializers` / `stringDeserializers` | `Serializers<string,string,string,string>` / `Deserializers<...>` | All four positions (key, value, header key, header value) as string |
+| `serializersFrom(s)` / `deserializersFrom(d)` | `<T>(fn) => Serializers<T,T,T,T>` / `<T>(fn) => Deserializers<T,T,T,T>` | Build all four positions from a single serializer/deserializer |
+
+```typescript
+import { stringSerializers } from '@platformatic/kafka';
+
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: stringSerializers,
+});
+```
+
+```typescript
+import { jsonSerializer, stringSerializer, serializersFrom } from '@platformatic/kafka';
+
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: { ...serializersFrom(jsonSerializer), key: stringSerializer },
+});
+
+await producer.getProducer().send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-123',
+    value: { id: '123', status: 'created', amount: 99 },
+  }],
+});
+```
+
+For schema-validated serialization (Avro, Protobuf, JSON Schema), pass a `registry` instead of `serializers` - see [Schema Registry](./schema-registry).
+
+## Compression
+
+`@platformatic/kafka` supports five compression algorithms:
+
+| Algorithm | Value | Description |
+|-----------|-------|-------------|
+| None | `'none'` | No compression (default) |
+| GZIP | `'gzip'` | Good compression ratio, moderate CPU |
+| Snappy | `'snappy'` | Fast compression, moderate ratio |
+| LZ4 | `'lz4'` | Very fast, good for high-throughput |
+| Zstandard | `'zstd'` | Best ratio, moderate CPU |
+
+```typescript
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: stringSerializers,
+  compression: 'zstd',
+});
+
+// Override per-send
+await helper.getProducer().send({
+  messages: [{ topic: 'logs', key: 'l1', value: largePayload }],
+  compression: 'lz4',
+});
+```
+
 ## Transactions
 
 `runInTransaction()` wraps `beginTransaction()` -> callback -> `commit()` / `abort()` with automatic logging.
@@ -133,6 +249,8 @@ const result = await helper.runInTransaction(async ({ send, addConsumer, addOffs
 });
 ```
 
+If the callback throws, the transaction is automatically aborted and the error re-thrown.
+
 ### Transaction Context
 
 The callback receives an `IKafkaTransactionContext`:
@@ -144,17 +262,17 @@ The callback receives an `IKafkaTransactionContext`:
 | `addConsumer(consumer)` | `(consumer: Consumer) => Promise<void>` | Add a consumer for exactly-once |
 | `addOffset(message)` | `(message: Message) => Promise<void>` | Add consumed message offset to transaction |
 
-If the callback throws, the transaction is automatically aborted and the error re-thrown.
-
 ## Graceful Shutdown
 
-`close()` implements a two-phase shutdown:
+`close()` runs a two-phase shutdown:
 
-1. **Graceful** (default): Calls `close(true)` on the underlying producer (force-flush), with a timeout (`shutdownTimeout`, default 30s)
-2. **Force fallback**: If graceful times out, automatically force-closes
-3. **Force** (`{ isForce: true }`): Immediately calls `close(true)` without timeout protection
+| Phase | When it runs | What happens |
+|---|---|---|
+| Graceful | Default | Calls `close(true)` on the underlying producer, capped at `shutdownTimeout` (default 30s) |
+| Force fallback | Graceful times out | Force-closes automatically, no further waiting |
+| Force | `close({ isForce: true })` | Calls `close(true)` immediately, no timeout |
 
-After `close()`, all broker tracking is cleared and `healthStatus` is set to `'disconnected'`.
+After `close()`, all broker tracking is cleared and `healthStatus` becomes `'disconnected'`.
 
 ```typescript
 // Graceful (recommended)
@@ -166,7 +284,7 @@ await helper.close({ isForce: true });
 
 ## API Reference (`@platformatic/kafka`)
 
-After calling `helper.getProducer()`, you have full access to the `Producer` class:
+After calling `helper.getProducer()`, you have full access to the `Producer` class.
 
 ### `producer.send(options)`
 
@@ -257,11 +375,13 @@ Close the producer connection.
 
 ## Key Partitioning
 
-By default, `@platformatic/kafka` uses **murmur2 hashing** on the message key to determine the target partition:
+By default, `@platformatic/kafka` uses **murmur2 hashing** on the message key to pick the target partition:
 
-- Same key -> always same partition -> guaranteed ordering per key
-- `undefined` key -> round-robin across partitions
-- Explicit `partition` field -> overrides the partitioner
+| Key | Target partition |
+|---|---|
+| Same key, every time | Same partition - ordering is guaranteed per key |
+| `undefined` | Round-robin across partitions |
+| Explicit `partition` field | That partition, overriding the partitioner |
 
 ```typescript
 // Custom partitioner
@@ -272,3 +392,17 @@ await producer.send({
   },
 });
 ```
+
+## See also
+
+- [Kafka Overview](./) - the four helpers, shared health/close API, and the compile-binary caveat
+- [Consumer](./consumer) - the receiving side, including message callbacks and lag monitoring
+- [Schema Registry](./schema-registry) - schema-validated serialization instead of manual `serializers`
+- [Compiling to a Single Binary](./compile-binary) - required when this helper ships inside a `bun build --compile` binary
+- [Examples & Troubleshooting](./examples) - IoC wiring and the common connection-error lookup table
+
+**Files:**
+
+- [`packages/helpers/src/modules/queue/kafka/producer.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/producer.ts) - `KafkaProducerHelper`
+- [`packages/helpers/src/modules/queue/kafka/base.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/base.ts) - `BaseKafkaHelper`, shared health tracking and shutdown
+- [`packages/helpers/src/modules/queue/kafka/common/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/common/types.ts) - `IKafkaProducerOptions`, `IKafkaConnectionOptions`, `IKafkaTransactionContext`

package/content/extensions/helpers/kafka/schema-registry.md

@@ -1,3 +1,9 @@
+---
+title: Kafka Schema Registry
+description: KafkaSchemaRegistryHelper - schema-validated serialization for Avro, Protobuf, and JSON Schema
+difficulty: intermediate
+---
+
 # Schema Registry
 
 The `KafkaSchemaRegistryHelper` wraps `@platformatic/kafka`'s `ConfluentSchemaRegistry`. It provides a centralized schema registry that auto-serializes/deserializes messages using registered schemas (Avro, Protobuf, JSON Schema).
@@ -12,7 +18,10 @@ class KafkaSchemaRegistryHelper<
 ```
 
 > [!NOTE]
-> `KafkaSchemaRegistryHelper` extends `BaseHelper` directly (not `BaseKafkaHelper`) -- it has no broker connection or health tracking. It's a configuration wrapper, not a client.
+> `KafkaSchemaRegistryHelper` extends `BaseHelper` directly (not `BaseKafkaHelper`) - it has no broker connection or health tracking. It's a configuration wrapper, not a client.
+
+> [!WARNING]
+> Compiling to a standalone binary needs `platformaticRequirePlugin()`. `ConfluentSchemaRegistry` resolves `ajv-draft-04` through `createRequire` at module load, which `bun build --compile` cannot see through. See [Compiling to a Single Binary](./compile-binary).
 
 ## Helper API
 
@@ -33,27 +42,35 @@ interface IKafkaSchemaRegistryOptions extends ConfluentSchemaRegistryOptions {
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `url` | `string` | -- | Schema registry URL. **Required** |
-| `auth` | `{ username: string; password: string }` | -- | Basic auth credentials |
-| `protobufTypeMapper` | `ProtobufTypeMapper` | -- | Custom Protobuf type mapper |
-| `jsonValidateSend` | `boolean` | -- | Validate JSON schema on produce |
+| `url` | `string` | - | Schema registry URL. **Required** |
+| `auth` | `{ username: string; password: string }` | - | Basic auth credentials |
+| `protobufTypeMapper` | `ProtobufTypeMapper` | - | Custom Protobuf type mapper |
+| `jsonValidateSend` | `boolean` | - | Validate JSON schema on produce |
 | `identifier` | `string` | `'kafka-schema-registry'` | Scoped logging identifier |
 
-## What Schema Registry Solves
+## What it solves
 
-Without a schema registry, producers and consumers must agree on message format out-of-band. If the producer changes the shape of `value` (adds/removes fields), consumers break silently at runtime.
+Without a schema registry, producers and consumers must agree on message shape out-of-band. If the producer changes the shape of `value` (adds/removes fields), consumers break silently at runtime.
 
-**Schema Registry** is a centralized server (Confluent Schema Registry) that stores and validates schemas (Avro, Protobuf, JSON Schema). It enforces a contract:
+- **Schema Registry is a centralized server** (Confluent Schema Registry) that stores and validates schemas.
+- **It enforces a contract on the way out.** The producer says "I want to send this shape." The registry validates that shape against the registered schema before the message reaches Kafka.
+- **It enforces a contract on the way in.** The consumer asks "what shape is this?" The registry answers with how to deserialize it.
 
-```
-Producer -> "I want to send this shape" -> Schema Registry validates -> Kafka
-Kafka -> Consumer -> "What shape is this?" -> Schema Registry tells -> Deserialize
-```
+| | Without registry | With registry |
+|---|---|---|
+| **Message format** | Raw string, manual `JSON.stringify`/`JSON.parse` | Typed object, auto ser/deser |
+| **Validation** | None - runtime crashes on shape drift | Schema validated before send |
+| **Schema evolution** | Breaks consumers silently | Backward/forward compatibility enforced |
+| **Where schemas live** | Nowhere (tribal knowledge) | Centralized server, e.g. `http://registry:8081` |
+
+Use it when you need schema enforcement and compatibility checks across producers and consumers, especially in multi-team environments. Skip it for simple string or JSON messages where one team controls both sides - coordinate format changes by hand instead.
 
-### Without Schema Registry (raw strings)
+## Without vs. with the registry
+
+Without a registry, serialization is entirely manual and unchecked:
 
 ```typescript
-// Producer -- manually serialize
+// Producer - manually serialize
 const producer = KafkaProducerHelper.newInstance({
   bootstrapBrokers: ['127.0.0.1:29092'],
   clientId: 'order-producer',
@@ -63,11 +80,11 @@ await producer.getProducer().send({
   messages: [{
     topic: 'orders',
     key: 'order-1',
-    value: JSON.stringify({ id: 1, total: 99.99 }),  // <- just a string, no validation
+    value: JSON.stringify({ id: 1, total: 99.99 }),  // <- plain string, no validation
   }],
 });
 
-// Consumer -- manually deserialize, hope the shape is correct
+// Consumer - manually deserialize, hope the shape is correct
 const consumer = KafkaConsumerHelper.newInstance({
   bootstrapBrokers: ['127.0.0.1:29092'],
   clientId: 'order-consumer',
@@ -79,22 +96,28 @@ const consumer = KafkaConsumerHelper.newInstance({
 });
 ```
 
-Problem: if producer adds `{ id: 1, total: 99.99, currency: 'USD' }` or removes `total`, consumer has no way to know until it crashes.
+If the producer adds `{ id: 1, total: 99.99, currency: 'USD' }` or removes `total`, the consumer has no way to know until it crashes.
 
-### With Schema Registry (auto serialize/deserialize)
+With a registry, both sides pass `registry` instead of `serializers`/`deserializers`:
 
 ```typescript
-// 1. Create registry -- points to Confluent Schema Registry server
+import {
+  KafkaSchemaRegistryHelper,
+  KafkaProducerHelper,
+  KafkaConsumerHelper,
+} from '@venizia/ignis-helpers/kafka';
+
+// 1. Create registry - points to Confluent Schema Registry server
 const registry = KafkaSchemaRegistryHelper.newInstance({
   url: 'http://localhost:8081',
   // auth: { username: 'user', password: 'pass' },  // optional
 });
 
-// 2. Producer -- pass registry, it auto-serializes values using registered schema
+// 2. Producer - pass registry, it auto-serializes values using the registered schema
 const producer = KafkaProducerHelper.newInstance({
   bootstrapBrokers: ['127.0.0.1:29092'],
   clientId: 'order-producer',
-  registry: registry.getRegistry(),  // <- registry handles serialization
+  registry: registry.getRegistry(),
 });
 
 await producer.getProducer().send({
@@ -106,61 +129,11 @@ await producer.getProducer().send({
 });
 // If the value doesn't match the registered schema -> error BEFORE sending to Kafka
 
-// 3. Consumer -- pass same registry, it auto-deserializes
+// 3. Consumer - pass the same registry, it auto-deserializes
 const consumer = KafkaConsumerHelper.newInstance({
   bootstrapBrokers: ['127.0.0.1:29092'],
   clientId: 'order-consumer',
   groupId: 'order-group',
-  registry: registry.getRegistry(),  // <- registry handles deserialization
-  onMessage: async ({ message }) => {
-    // message.value is already a typed object, not a raw string
-    console.log(message.value.id, message.value.total);
-  },
-});
-```
-
-### Comparison
-
-| | Without Registry | With Registry |
-|---|---|---|
-| **Message format** | Raw string, manual `JSON.stringify/parse` | Typed object, auto ser/deser |
-| **Validation** | None -- runtime crashes | Schema validated before send |
-| **Schema evolution** | Break consumers silently | Backward/forward compatibility enforced |
-| **Where schemas live** | Nowhere (tribal knowledge) | Centralized server `http://registry:8081` |
-
-You only need it when you want **schema enforcement** across producers/consumers. For simple string messages, skip it entirely.
-
-## Basic Usage
-
-```typescript
-import { KafkaSchemaRegistryHelper, KafkaProducerHelper, KafkaConsumerHelper } from '@venizia/ignis-helpers/kafka';
-
-// 1. Create registry -- points to Confluent Schema Registry server
-const registry = KafkaSchemaRegistryHelper.newInstance({
-  url: 'http://localhost:8081',
-});
-
-// 2. Producer -- registry auto-serializes values using registered schema
-const producer = KafkaProducerHelper.newInstance({
-  bootstrapBrokers: ['localhost:9092'],
-  clientId: 'order-producer',
-  registry: registry.getRegistry(),
-});
-
-await producer.getProducer().send({
-  messages: [{
-    topic: 'orders',
-    key: 'order-1',
-    value: { id: 1, total: 99.99 },  // object, not string -- registry serializes
-  }],
-});
-// If value doesn't match the registered schema -> error BEFORE sending to Kafka
-
-// 3. Consumer -- registry auto-deserializes
-const consumer = KafkaConsumerHelper.newInstance({
-  bootstrapBrokers: ['localhost:9092'],
-  clientId: 'order-consumer',
-  groupId: 'order-group',
   registry: registry.getRegistry(),
   onMessage: async ({ message }) => {
     // message.value is already a typed object, not a raw string
@@ -171,7 +144,7 @@ const consumer = KafkaConsumerHelper.newInstance({
 await consumer.start({ topics: ['orders'] });
 ```
 
-## With Authentication
+## With authentication
 
 ```typescript
 const registry = KafkaSchemaRegistryHelper.newInstance({
@@ -183,9 +156,9 @@ const registry = KafkaSchemaRegistryHelper.newInstance({
 });
 ```
 
-## Alternative: Manual Serializers
+## Alternative: manual serializers
 
-Instead of passing the full registry, you can extract serializers/deserializers for manual use:
+Instead of passing the full registry to `registry`, extract serializers/deserializers for manual use alongside `serializers`/`deserializers`:
 
 ```typescript
 const registry = KafkaSchemaRegistryHelper.newInstance({
@@ -204,11 +177,18 @@ const consumer = KafkaConsumerHelper.newInstance({
   clientId: 'my-consumer',
   groupId: 'my-group',
   deserializers: registry.getDeserializers(),
-  onMessage: async ({ message }) => { ... },
+  onMessage: async ({ message }) => { /* ... */ },
 });
 ```
 
-## When to Use
+## See also
+
+- [Kafka Overview](./) - the four helpers, shared health/close API, and the compile-binary caveat
+- [Producer](./producer) - `registry` as a producer option, plus the manual `serializers` alternative
+- [Consumer](./consumer) - `registry` as a consumer option, plus the manual `deserializers` alternative
+- [Examples & Troubleshooting](./examples) - IoC wiring and the common connection-error lookup table
+
+**Files:**
 
-- **Use schema registry** when you need schema enforcement, validation, and compatibility checks across producers/consumers -- especially in multi-team environments
-- **Skip schema registry** for simple string/JSON messages where both sides are controlled by the same team and format changes are coordinated
+- [`packages/helpers/src/modules/queue/kafka/schema/registry.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/schema/registry.ts) - `KafkaSchemaRegistryHelper`
+- [`packages/helpers/src/modules/queue/kafka/common/types.ts`](https://github.com/VENIZIA-AI/ignis/blob/main/packages/helpers/src/modules/queue/kafka/common/types.ts) - `IKafkaSchemaRegistryOptions`