@venizia/ignis-docs 0.0.7-1 → 0.0.7

package/wiki/references/helpers/kafka/index.md

@@ -1,305 +1,636 @@
-# Kafka <Badge type="warning" text="Experimental" />
+# Kafka
 
-Apache Kafka event streaming with producer, consumer, and admin helpers. Built on [`@platformatic/kafka`](https://github.com/platformatic/kafka).
+Apache Kafka event streaming with producer, consumer, admin, and schema registry helpers. Built on [`@platformatic/kafka`](https://github.com/platformatic/kafka) v1.30.0 — a pure TypeScript Kafka client with zero native dependencies.
 
-> [!WARNING]
-> This helper is **experimental**. The API may change in future releases.
+## Overview
 
-## Quick Reference
+The Kafka module provides four helper classes built on a shared `BaseKafkaHelper` base:
 
-| Class | Extends | Peer Dependency | Use Case |
-|-------|---------|-----------------|----------|
-| **KafkaProducerHelper** | `BaseHelper` | `@platformatic/kafka` | Publish messages to Kafka topics |
-| **KafkaConsumerHelper** | `BaseHelper` | `@platformatic/kafka` | Consume messages from Kafka topics with consumer groups |
-| **KafkaAdminHelper** | `BaseHelper` | `@platformatic/kafka` | Manage topics, partitions, consumer groups, and configs |
+| Class | Wraps | Use Case |
+|-------|-------|----------|
+| `KafkaProducerHelper` | `Producer` | Publish messages, transactions |
+| `KafkaConsumerHelper` | `Consumer` | Consume messages with consumer groups, lag monitoring |
+| `KafkaAdminHelper` | `Admin` | Manage topics, partitions, groups, ACLs, configs |
+| `KafkaSchemaRegistryHelper` | `ConfluentSchemaRegistry` | Schema validation and auto ser/deser |
+
+All helpers (except schema registry) extend `BaseKafkaHelper` which provides:
+
+- **Scoped logging** via `BaseHelper` (Winston with daily rotation)
+- **Health tracking** — `isHealthy()`, `isReady()`, `getHealthStatus()`
+- **Broker event callbacks** — `onBrokerConnect`, `onBrokerDisconnect`
+- **Graceful shutdown** — timeout-based with force fallback
+- **Sensible defaults** via `KafkaDefaults` constants
+- **Factory pattern** via `newInstance()` static method
+
+Use `getProducer()`, `getConsumer()`, or `getAdmin()` to access the full underlying `@platformatic/kafka` API directly.
 
 ### Import Path
 
 ```typescript
+// Helpers & constants
 import {
   KafkaProducerHelper,
   KafkaConsumerHelper,
   KafkaAdminHelper,
+  KafkaSchemaRegistryHelper,
+  BaseKafkaHelper,
   KafkaDefaults,
   KafkaAcks,
-  KafkaConfigResourceTypes,
+  KafkaGroupProtocol,
+  KafkaHealthStatuses,
+  KafkaClientEvents,
 } from '@venizia/ignis-helpers/kafka';
 
+// Types
 import type {
   IKafkaConnectionOptions,
   IKafkaProducerOptions,
   IKafkaConsumerOptions,
   IKafkaAdminOptions,
-  IKafkaProduceMessage,
-  IKafkaSendOptions,
-  IKafkaConsumedMessage,
-  IKafkaCommitOptions,
+  IKafkaConsumeStartOptions,
+  IKafkaSchemaRegistryOptions,
+  IKafkaTransactionContext,
+  IKafkaBaseOptions,
+  TKafkaAcks,
+  TKafkaGroupProtocol,
+  TKafkaHealthStatus,
+  TKafkaBrokerEventCallback,
+  TKafkaMessageCallback,
+  TKafkaMessageDoneCallback,
+  TKafkaMessageErrorCallback,
+  TKafkaGroupJoinCallback,
+  TKafkaGroupLeaveCallback,
+  TKafkaGroupRebalanceCallback,
+  TKafkaHeartbeatErrorCallback,
+  TKafkaLagCallback,
+  TKafkaLagErrorCallback,
+  TKafkaTransactionCallback,
 } from '@venizia/ignis-helpers/kafka';
+
+// @platformatic/kafka (direct usage)
+import {
+  Producer, Consumer, Admin, MessagesStream,
+  stringSerializers, stringDeserializers,
+  stringSerializer, stringDeserializer,
+  jsonSerializer, jsonDeserializer,
+  serializersFrom, deserializersFrom,
+} from '@platformatic/kafka';
+
+import type {
+  Message, MessageToProduce,
+  SendOptions, ConsumeOptions,
+  Serializers, Deserializers,
+  SASLOptions, ConnectionOptions,
+} from '@platformatic/kafka';
 ```
 
-## Installation
+### Installation
 
 ```bash
 bun add @platformatic/kafka
 ```
 
-## Producer
+## Architecture
+
+### Class Hierarchy
+
+```
+BaseHelper (scoped logging, identifier)
+  └── BaseKafkaHelper<TClient> (health tracking, broker events, graceful shutdown)
+        ├── KafkaProducerHelper<K,V,HK,HV>
+        ├── KafkaConsumerHelper<K,V,HK,HV>
+        └── KafkaAdminHelper
+
+BaseHelper
+  └── KafkaSchemaRegistryHelper<K,V,HK,HV>  (no broker connection)
+```
 
-The `KafkaProducerHelper` wraps `@platformatic/kafka`'s `Producer` for publishing messages to Kafka topics.
+### BaseKafkaHelper
+
+All Kafka helpers (except schema registry) extend `BaseKafkaHelper<TClient>`, which provides:
 
 ```typescript
-import { KafkaProducerHelper, KafkaAcks } from '@venizia/ignis-helpers/kafka';
+abstract class BaseKafkaHelper<TClient extends Base<BaseOptions>> extends BaseHelper {
+  // Health
+  isHealthy(): boolean;          // healthStatus === 'connected'
+  isReady(): boolean;            // healthStatus === 'connected' (consumer overrides: + isActive())
+  getHealthStatus(): TKafkaHealthStatus; // 'connected' | 'disconnected' | 'unknown'
+
+  // Shutdown (used by subclasses)
+  protected closeClient(): Promise<void>;
+  protected gracefulCloseClient(): Promise<void>; // races closeClient vs shutdownTimeout
+}
+```
 
-const producer = new KafkaProducerHelper({
-  identifier: 'order-producer',
-  bootstrapBrokers: ['localhost:9092'],
-  acks: KafkaAcks.ALL,
-  autocreateTopics: true,
-  onConnected: () => console.log('Producer connected'),
-  onError: ({ error }) => console.error('Producer error:', error),
-});
+Health status transitions automatically via broker events:
+- `client:broker:connect` → `'connected'`
+- `client:broker:disconnect` → `'disconnected'`
+- `client:broker:failed` → `'disconnected'`
+- `close()` → `'disconnected'`
 
-// Send messages
-await producer.send({
-  messages: [
-    { topic: 'orders', key: 'order-123', value: JSON.stringify({ status: 'created' }) },
-  ],
-});
+## Connection Options
 
-// Send batch across multiple topics
-await producer.sendBatch({
-  topicMessages: [
-    { topic: 'orders', messages: [{ key: 'o1', value: '...' }] },
-    { topic: 'notifications', messages: [{ key: 'n1', value: '...' }] },
-  ],
-});
+All three helpers share a common base interface `IKafkaConnectionOptions` which extends `@platformatic/kafka`'s `ConnectionOptions`.
 
-// Graceful shutdown
-await producer.close();
+```typescript
+interface IKafkaConnectionOptions extends ConnectionOptions {
+  bootstrapBrokers: string[];
+  clientId: string;
+  retries?: number;    // Default: 3
+  retryDelay?: number; // Default: 1000ms
+}
 ```
 
-### IKafkaProducerOptions
+### Full Options Table
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `bootstrapBrokers` | `string[]` | -- | Kafka broker addresses (required) |
-| `identifier` | `string` | -- | Scoped logging identifier |
-| `clientId` | `string` | `'ignis-kafka'` | Kafka client ID |
-| `acks` | `number` | -- | Acknowledgment level (`KafkaAcks.NONE`, `LEADER`, `ALL`) |
-| `autocreateTopics` | `boolean` | -- | Auto-create topics on first produce |
-| `timeout` | `number` | -- | Connection timeout in ms |
-| `retries` | `number \| boolean` | -- | Retry configuration |
-| `retryDelay` | `number` | -- | Delay between retries in ms |
-| `serializers` | `Partial<Serializers>` | -- | Custom key/value/header serializers |
-| `onConnected` | `() => void` | -- | Broker connect callback |
-| `onDisconnected` | `() => void` | -- | Broker disconnect callback |
-| `onError` | `(opts: { error: Error }) => void` | -- | Error callback |
-
-### Producer API
-
-| Method | Returns | Description |
-|--------|---------|-------------|
-| `send(opts)` | `Promise<void>` | Send messages. `opts: { messages: IKafkaProduceMessage[]; acks? }` |
-| `sendBatch(opts)` | `Promise<void>` | Send to multiple topics. `opts: { topicMessages: Array<{ topic; messages }> }` |
-| `getProducer()` | `Producer` | Access the underlying `@platformatic/kafka` Producer |
-| `close()` | `Promise<void>` | Gracefully close the producer connection |
-| `static newInstance(opts)` | `KafkaProducerHelper` | Factory method |
-
-## Consumer
-
-The `KafkaConsumerHelper` provides a stream-based consumer with consumer group support, pause/resume, manual commit, and lag monitoring.
+| `bootstrapBrokers` | `string[]` | — | Kafka broker addresses (`host:port`). **Required** |
+| `clientId` | `string` | — | Unique client identifier. **Required** |
+| `retries` | `number` | `3` | Number of connection retries before failing |
+| `retryDelay` | `number` | `1000` | Delay between retries in milliseconds |
+| `sasl` | `SASLOptions` | — | SASL authentication configuration |
+| `tls` | `TLSConnectionOptions` | — | TLS/SSL connection options |
+| `ssl` | `TLSConnectionOptions` | — | Alias for `tls` |
+| `connectTimeout` | `number` | — | TCP connection timeout in milliseconds |
+| `requestTimeout` | `number` | — | Kafka request timeout in milliseconds |
+
+### Shared Helper Options
+
+These options are available on all three helpers (`IKafkaProducerOptions`, `IKafkaConsumerOptions`, `IKafkaAdminOptions`):
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `identifier` | `string` | `'kafka-{type}'` | Scoped logging identifier |
+| `shutdownTimeout` | `number` | `30000` | Graceful shutdown timeout in ms |
+| `onBrokerConnect` | `TKafkaBrokerEventCallback` | — | Called when broker connects |
+| `onBrokerDisconnect` | `TKafkaBrokerEventCallback` | — | Called when broker disconnects |
+
+### SASL Authentication
+
+`@platformatic/kafka` supports five SASL mechanisms:
+
+| Mechanism | Use Case |
+|-----------|----------|
+| `PLAIN` | Simple username/password (use with TLS in production) |
+| `SCRAM-SHA-256` | Challenge-response, password never sent in plaintext |
+| `SCRAM-SHA-512` | Same as SHA-256 with stronger hash |
+| `OAUTHBEARER` | Token-based (Azure Event Hubs, Confluent Cloud) |
+| `GSSAPI` | Kerberos authentication |
 
 ```typescript
-import { KafkaConsumerHelper } from '@venizia/ignis-helpers/kafka';
+interface SASLOptions {
+  mechanism: 'PLAIN' | 'SCRAM-SHA-256' | 'SCRAM-SHA-512' | 'OAUTHBEARER' | 'GSSAPI';
+  username?: string | CredentialProvider;
+  password?: string | CredentialProvider;
+  token?: string | CredentialProvider;
+  oauthBearerExtensions?: Record<string, string> | CredentialProvider<Record<string, string>>;
+  authenticate?: SASLCustomAuthenticator;
+}
+```
 
-const consumer = new KafkaConsumerHelper({
-  identifier: 'order-consumer',
-  bootstrapBrokers: ['localhost:9092'],
-  groupId: 'order-processing-group',
-  topics: ['orders'],
-  mode: 'latest',
-  autocommit: true,
-  onMessage: async ({ message }) => {
-    console.log(`Topic: ${message.topic}, Partition: ${message.partition}`);
-    console.log(`Key: ${message.key}, Value: ${message.value}`);
-  },
-  onConnected: () => console.log('Consumer connected'),
-  onGroupJoin: ({ groupId, memberId }) => {
-    console.log(`Joined group ${groupId} as ${memberId}`);
+#### SCRAM-SHA-512 Example
+
+```typescript
+const helper = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['broker1:9092', 'broker2:9092', 'broker3:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  sasl: {
+    mechanism: 'SCRAM-SHA-512',
+    username: 'kafka-user',
+    password: 'kafka-password',
   },
-  onError: ({ error }) => console.error('Consumer error:', error),
+  connectTimeout: 30_000,
+  requestTimeout: 30_000,
+  onBrokerConnect: ({ broker }) => console.log(`Connected to ${broker.host}:${broker.port}`),
 });
+```
+
+#### OAUTHBEARER Example
 
-// Start consuming
-await consumer.start();
+```typescript
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['pkc-xxxxx.us-west-2.aws.confluent.cloud:9092'],
+  clientId: 'my-producer',
+  sasl: {
+    mechanism: 'OAUTHBEARER',
+    token: async () => {
+      const response = await fetch('https://auth.example.com/token', { method: 'POST' });
+      const { access_token } = await response.json();
+      return access_token;
+    },
+  },
+  tls: true,
+});
+```
 
-// Pause/resume
-consumer.pause();
-consumer.resume();
+#### TLS Without SASL
 
-// Graceful shutdown
-await consumer.close();
+```typescript
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['broker:9093'],
+  clientId: 'my-producer',
+  tls: {
+    ca: fs.readFileSync('/path/to/ca.pem'),
+    cert: fs.readFileSync('/path/to/client-cert.pem'),
+    key: fs.readFileSync('/path/to/client-key.pem'),
+  },
+});
 ```
 
-### IKafkaConsumerOptions
+## Serialization & Deserialization
 
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `bootstrapBrokers` | `string[]` | -- | Kafka broker addresses (required) |
-| `groupId` | `string` | -- | Consumer group ID (required) |
-| `topics` | `string[]` | -- | Topics to consume (required) |
-| `identifier` | `string` | -- | Scoped logging identifier |
-| `clientId` | `string` | `'ignis-kafka'` | Kafka client ID |
-| `mode` | `'latest' \| 'earliest' \| 'committed'` | `'latest'` | Offset reset strategy |
-| `autocommit` | `boolean \| number` | `true` | Auto-commit offsets (or interval in ms) |
-| `sessionTimeout` | `number` | `30000` | Session timeout in ms |
-| `heartbeatInterval` | `number` | `3000` | Heartbeat interval in ms |
-| `highWaterMark` | `number` | `1024` | Stream high water mark |
-| `maxWaitTime` | `number` | `5000` | Max wait time for fetch in ms |
-| `deserializers` | `Partial<Deserializers>` | -- | Custom key/value/header deserializers |
-| `onMessage` | `(opts: { message }) => ValueOrPromise<void>` | -- | Message handler |
-| `onConnected` | `() => void` | -- | Broker connect callback |
-| `onDisconnected` | `() => void` | -- | Broker disconnect callback |
-| `onGroupJoin` | `(opts: { groupId; memberId }) => void` | -- | Consumer group join callback |
-| `onGroupLeave` | `() => void` | -- | Consumer group leave callback |
-| `onRebalance` | `() => void` | -- | Group rebalance callback |
-| `onLag` | `(opts: { offsets }) => void` | -- | Consumer lag callback |
-| `onError` | `(opts: { error: Error }) => void` | -- | Error callback |
-
-### Consumer API
-
-| Method | Returns | Description |
-|--------|---------|-------------|
-| `start()` | `Promise<void>` | Start consuming messages (fires async consume loop) |
-| `pause()` | `void` | Pause the message stream |
-| `resume()` | `void` | Resume the message stream |
-| `isPaused()` | `boolean` | Check if the stream is paused |
-| `isConsuming()` | `boolean` | Check if the consumer is running |
-| `commit(opts)` | `Promise<void>` | Manually commit offsets |
-| `startLagMonitoring(opts)` | `void` | Start lag monitoring. `opts: { interval: number }` |
-| `stopLagMonitoring()` | `void` | Stop lag monitoring |
-| `getConsumer()` | `Consumer` | Access the underlying `@platformatic/kafka` Consumer |
-| `close()` | `Promise<void>` | Abort consume loop, close stream and consumer |
-| `static newInstance(opts)` | `KafkaConsumerHelper` | Factory method |
-
-### Manual Commit
-
-When `autocommit` is `false`, commit offsets explicitly:
+`@platformatic/kafka`'s default wire format is `Buffer`. The helpers default generic types to `string` (matching common usage), but you must provide serializers/deserializers explicitly.
+
+### Built-in Serializers
+
+| Export | Type | Description |
+|--------|------|-------------|
+| `stringSerializer` | `Serializer<string>` | `string → Buffer` (UTF-8) |
+| `stringDeserializer` | `Deserializer<string>` | `Buffer → string` (UTF-8) |
+| `jsonSerializer` | `Serializer<T>` | `object → Buffer` (JSON.stringify + UTF-8) |
+| `jsonDeserializer` | `Deserializer<T>` | `Buffer → object` (UTF-8 + JSON.parse) |
+| `stringSerializers` | `Serializers<string, string, string, string>` | All four positions as string |
+| `stringDeserializers` | `Deserializers<string, string, string, string>` | All four positions as string |
+
+### Helper Functions
+
+| Export | Signature | Description |
+|--------|-----------|-------------|
+| `serializersFrom(s)` | `<T>(s: Serializer<T>) => Serializers<T, T, T, T>` | Create full serializers from a single serializer |
+| `deserializersFrom(d)` | `<T>(d: Deserializer<T>) => Deserializers<T, T, T, T>` | Create full deserializers from a single deserializer |
+
+### String Serialization
 
 ```typescript
-const consumer = new KafkaConsumerHelper({
-  // ...
-  autocommit: false,
+import { stringSerializers, stringDeserializers } from '@platformatic/kafka';
+
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: stringSerializers,
+});
+
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  deserializers: stringDeserializers,
   onMessage: async ({ message }) => {
-    await processMessage(message);
-    // Commit after successful processing
-    await consumer.commit({
-      offsets: [{
-        topic: message.topic,
-        partition: message.partition,
-        offset: message.offset,
-        leaderEpoch: 0,
-      }],
-    });
+    console.log(message.key, message.value); // both strings
   },
 });
 ```
 
-## Admin
-
-The `KafkaAdminHelper` provides topic, partition, consumer group, and config management.
+### JSON Serialization
 
 ```typescript
-import { KafkaAdminHelper, KafkaConfigResourceTypes } from '@venizia/ignis-helpers/kafka';
+import {
+  jsonSerializer, jsonDeserializer,
+  stringSerializer, stringDeserializer,
+  serializersFrom, deserializersFrom,
+} from '@platformatic/kafka';
+
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: { ...serializersFrom(jsonSerializer), key: stringSerializer },
+});
+
+await producer.getProducer().send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-123',
+    value: { id: '123', status: 'created', amount: 99 },
+  }],
+});
 
-const admin = new KafkaAdminHelper({
-  identifier: 'kafka-admin',
+const consumer = KafkaConsumerHelper.newInstance({
   bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  deserializers: { ...deserializersFrom(jsonDeserializer), key: stringDeserializer },
+  onMessage: async ({ message }) => {
+    console.log(message.value.id, message.value.status); // typed object
+  },
 });
+```
 
-// Topic management
-await admin.createTopics({ topics: ['orders', 'notifications'], partitions: 3, replicas: 1 });
-const topics = await admin.listTopics();
-await admin.deleteTopics({ topics: ['old-topic'] });
+### Schema Registry Serialization
 
-// Partition management
-await admin.createPartitions({ topics: [{ name: 'orders', count: 6 }] });
+For schema-validated serialization (Avro, Protobuf, JSON Schema), use the schema registry helper:
+
+```typescript
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'http://localhost:8081',
+});
 
-// Consumer group management
-const groups = await admin.listGroups();
-const groupDetails = await admin.describeGroups({ groups: ['order-processing-group'] });
-await admin.deleteGroups({ groups: ['stale-group'] });
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  registry: registry.getRegistry(),
+});
 
-// Config management
-const configs = await admin.describeConfigs({
-  resources: [{
-    resourceType: KafkaConfigResourceTypes.TOPIC,
-    resourceName: 'orders',
-  }],
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  registry: registry.getRegistry(),
+  onMessage: async ({ message }) => {
+    // message.value is auto-deserialized using registered schema
+  },
 });
+```
+
+See **[Schema Registry](./schema-registry)** for full documentation.
+
+## Generic Type Parameters
 
-// Metadata
-const meta = await admin.metadata({ topics: ['orders'] });
+All helpers (and their option interfaces) support generic type parameters controlling the serialization types:
 
-// Cleanup
-await admin.close();
+```typescript
+class KafkaProducerHelper<
+  KeyType = string,
+  ValueType = string,
+  HeaderKeyType = string,
+  HeaderValueType = string,
+>
 ```
 
-### Admin API
-
-| Method | Returns | Description |
-|--------|---------|-------------|
-| `createTopics(opts)` | `Promise<any>` | Create topics with partitions and replicas |
-| `deleteTopics(opts)` | `Promise<void>` | Delete topics |
-| `listTopics(opts?)` | `Promise<string[]>` | List topics (optionally include internals) |
-| `metadata(opts?)` | `Promise<any>` | Fetch cluster/topic metadata |
-| `listGroups(opts?)` | `Promise<any>` | List consumer groups (filter by state) |
-| `describeGroups(opts)` | `Promise<any>` | Describe consumer groups |
-| `deleteGroups(opts)` | `Promise<void>` | Delete consumer groups |
-| `listConsumerGroupOffsets(opts)` | `Promise<any>` | List offsets for consumer groups |
-| `alterConsumerGroupOffsets(opts)` | `Promise<void>` | Alter consumer group offsets |
-| `createPartitions(opts)` | `Promise<void>` | Create partitions for topics |
-| `describeConfigs(opts)` | `Promise<any>` | Describe resource configs |
-| `alterConfigs(opts)` | `Promise<void>` | Alter resource configs |
-| `getAdmin()` | `Admin` | Access the underlying `@platformatic/kafka` Admin |
-| `close()` | `Promise<void>` | Close the admin connection |
-| `static newInstance(opts)` | `KafkaAdminHelper` | Factory method |
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `KeyType` | `string` | Message key type after serialization/deserialization |
+| `ValueType` | `string` | Message value type after serialization/deserialization |
+| `HeaderKeyType` | `string` | Header key type |
+| `HeaderValueType` | `string` | Header value type |
+
+> [!NOTE]
+> `@platformatic/kafka` defaults to `Buffer` for all four positions. The helpers default to `string` which is more common for application code. If you don't pass serializers, your messages will be sent/received as `Buffer`.
+
+```typescript
+// Default: string types (most common)
+const helper = KafkaProducerHelper.newInstance({ ... });
+
+// Custom: string keys, JSON object values
+const helper = KafkaProducerHelper.newInstance<string, MyEvent, string, string>({
+  serializers: { ...serializersFrom(jsonSerializer), key: stringSerializer },
+  ...
+});
+```
 
 ## Constants
 
 ### KafkaDefaults
 
+Centralized default values used by all helpers.
+
+```typescript
+import { KafkaDefaults } from '@venizia/ignis-helpers/kafka';
+```
+
+| Constant | Value | Scope | Description |
+|----------|-------|-------|-------------|
+| `RETRIES` | `3` | Shared | Connection retry count |
+| `RETRY_DELAY` | `1000` | Shared | Retry delay in ms |
+| `SHUTDOWN_TIMEOUT` | `30000` | Shared | Graceful shutdown timeout in ms |
+| `STRICT` | `true` | Producer | Fail on unknown topics |
+| `AUTOCREATE_TOPICS` | `false` | Producer | Auto-create topics on produce |
+| `AUTOCOMMIT` | `false` | Consumer | Auto-commit offsets |
+| `SESSION_TIMEOUT` | `30000` | Consumer | Session timeout in ms |
+| `HEARTBEAT_INTERVAL` | `3000` | Consumer | Heartbeat interval in ms |
+| `HIGH_WATER_MARK` | `1024` | Consumer | Stream buffer size (messages) |
+| `MIN_BYTES` | `1` | Consumer | Min bytes per fetch |
+| `METADATA_MAX_AGE` | `300000` | Consumer | Metadata cache TTL in ms |
+| `GROUP_PROTOCOL` | `'classic'` | Consumer | Default group protocol |
+| `CONSUME_MODE` | `'committed'` | Consumer | Default consume mode |
+| `CONSUME_FALLBACK_MODE` | `'latest'` | Consumer | Default consume fallback mode |
+| `LAG_MONITOR_INTERVAL` | `30000` | Consumer | Lag monitoring poll interval in ms |
+
+### KafkaHealthStatuses
+
+Health status values used by all Kafka helpers.
+
+```typescript
+import { KafkaHealthStatuses } from '@venizia/ignis-helpers/kafka';
+```
+
 | Constant | Value | Description |
 |----------|-------|-------------|
-| `CLIENT_ID` | `'ignis-kafka'` | Default Kafka client ID |
-| `SESSION_TIMEOUT` | `30000` | Session timeout (ms) |
-| `HEARTBEAT_INTERVAL` | `3000` | Heartbeat interval (ms) |
-| `MAX_WAIT_TIME` | `5000` | Max fetch wait time (ms) |
-| `HIGH_WATER_MARK` | `1024` | Stream high water mark |
-| `AUTOCOMMIT_INTERVAL` | `100` | Auto-commit interval (ms) |
+| `CONNECTED` | `'connected'` | Broker connection established |
+| `DISCONNECTED` | `'disconnected'` | Broker connection lost or closed |
+| `UNKNOWN` | `'unknown'` | Initial state before first broker event |
+
+### KafkaClientEvents
+
+Event name constants for `@platformatic/kafka` event emitters.
+
+```typescript
+import { KafkaClientEvents } from '@venizia/ignis-helpers/kafka';
+```
+
+| Constant | Value | Scope |
+|----------|-------|-------|
+| `BROKER_CONNECT` | `'client:broker:connect'` | All clients |
+| `BROKER_DISCONNECT` | `'client:broker:disconnect'` | All clients |
+| `BROKER_FAILED` | `'client:broker:failed'` | All clients |
+| `CONSUMER_GROUP_JOIN` | `'consumer:group:join'` | Consumer |
+| `CONSUMER_GROUP_LEAVE` | `'consumer:group:leave'` | Consumer |
+| `CONSUMER_GROUP_REBALANCE` | `'consumer:group:rebalance'` | Consumer |
+| `CONSUMER_HEARTBEAT_ERROR` | `'consumer:heartbeat:error'` | Consumer |
+| `CONSUMER_LAG` | `'consumer:lag'` | Consumer |
+| `CONSUMER_LAG_ERROR` | `'consumer:lag:error'` | Consumer |
+| `STREAM_DATA` | `'data'` | Stream |
+| `STREAM_ERROR` | `'error'` | Stream |
 
 ### KafkaAcks
 
-| Constant | Value | Description |
-|----------|-------|-------------|
-| `NONE` | `0` | No acknowledgment |
-| `LEADER` | `1` | Leader acknowledgment only |
-| `ALL` | `-1` | All replicas must acknowledge |
+Producer acknowledgment levels.
+
+```typescript
+import { KafkaAcks } from '@venizia/ignis-helpers/kafka';
+```
+
+| Constant | Value | Description | Trade-off |
+|----------|-------|-------------|-----------|
+| `NONE` | `0` | No acknowledgment — fire-and-forget | Fastest, no durability guarantee |
+| `LEADER` | `1` | Leader broker acknowledges | Fast, leader-durable |
+| `ALL` | `-1` | All in-sync replicas acknowledge | Slowest, fully durable |
+
+### KafkaGroupProtocol
 
-### KafkaConfigResourceTypes
+Consumer group protocol versions.
+
+```typescript
+import { KafkaGroupProtocol } from '@venizia/ignis-helpers/kafka';
+```
 
 | Constant | Value | Description |
 |----------|-------|-------------|
-| `UNKNOWN` | `0` | Unknown resource type |
-| `TOPIC` | `2` | Topic resource |
-| `BROKER` | `4` | Broker resource |
-| `BROKER_LOGGER` | `8` | Broker logger resource |
+| `CLASSIC` | `'classic'` | Classic consumer group protocol (default, all Kafka versions) |
+| `CONSUMER` | `'consumer'` | New consumer group protocol — KIP-848 (Kafka 3.7+) |
+
+### Derived Types
+
+```typescript
+import type { TKafkaAcks, TKafkaGroupProtocol, TKafkaHealthStatus } from '@venizia/ignis-helpers/kafka';
+
+// TKafkaAcks = 0 | 1 | -1
+// TKafkaGroupProtocol = 'classic' | 'consumer'
+// TKafkaHealthStatus = 'connected' | 'disconnected' | 'unknown'
+```
+
+## Compression
+
+`@platformatic/kafka` supports five compression algorithms:
+
+| Algorithm | Value | Description |
+|-----------|-------|-------------|
+| None | `'none'` | No compression (default) |
+| GZIP | `'gzip'` | Good compression ratio, moderate CPU |
+| Snappy | `'snappy'` | Fast compression, moderate ratio |
+| LZ4 | `'lz4'` | Very fast, good for high-throughput |
+| Zstandard | `'zstd'` | Best ratio, moderate CPU |
+
+```typescript
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: stringSerializers,
+  compression: 'zstd',
+});
+
+// Override per-send
+await helper.getProducer().send({
+  messages: [{ topic: 'logs', key: 'l1', value: largePayload }],
+  compression: 'lz4',
+});
+```
+
+## Quick Usage Comparison
+
+### Construction
+
+```typescript
+// Admin
+const admin = KafkaAdminHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'my-admin',
+  onBrokerConnect: ({ broker }) => console.log(`Connected to ${broker.host}`),
+  onBrokerDisconnect: ({ broker }) => console.log(`Disconnected from ${broker.host}`),
+});
+
+// Producer
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'my-producer',
+  acks: -1,
+  idempotent: true,
+  transactionalId: 'my-tx',
+  onBrokerConnect: ({ broker }) => console.log(`Connected to ${broker.host}`),
+  onBrokerDisconnect: ({ broker }) => console.log(`Disconnected from ${broker.host}`),
+});
+
+// Consumer
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  onBrokerConnect: ({ broker }) => console.log(`Connected to ${broker.host}`),
+  onBrokerDisconnect: ({ broker }) => console.log(`Disconnected from ${broker.host}`),
+  onMessage: async ({ message }) => {
+    console.log('Received:', message.value);
+    await message.commit();
+  },
+  onMessageDone: ({ message }) => console.log('Done:', message.key),
+  onMessageError: ({ error, message }) => console.error('Error:', error),
+  onGroupJoin: ({ groupId, memberId }) => console.log(`Joined ${groupId}`),
+  onGroupLeave: ({ groupId }) => console.log(`Left ${groupId}`),
+  onGroupRebalance: ({ groupId }) => console.log(`Rebalance ${groupId}`),
+  onHeartbeatError: ({ error }) => console.error('Heartbeat:', error),
+  onLag: ({ lag }) => console.log('Lag:', lag),
+  onLagError: ({ error }) => console.error('Lag error:', error),
+});
+```
+
+### Core Operations
+
+| Admin | Producer | Consumer |
+|-------|----------|----------|
+| `admin.getAdmin()` | `producer.getProducer()` | `consumer.getConsumer()` |
+| — | `producer.getProducer().send(...)` | `await consumer.start({ topics: ['t1'] })` |
+| — | `await producer.runInTransaction(async ({ send, addConsumer, addOffset }) => { ... })` | `consumer.startLagMonitoring({ topics: ['t1'], interval: 10_000 })` |
+| — | — | `consumer.stopLagMonitoring()` |
+| — | — | `consumer.getStream()` |
+
+### Health Checks
+
+```typescript
+// All three — identical API
+helper.isHealthy();      // true when broker connected
+helper.isReady();        // Admin/Producer: same as isHealthy()
+                         // Consumer: isHealthy() + consumer.isActive()
+helper.getHealthStatus(); // 'connected' | 'disconnected' | 'unknown'
+```
+
+### Shutdown
+
+```typescript
+// All three — identical API
+await helper.close();                    // graceful (timeout → force fallback)
+await helper.close({ isForce: true });   // immediate force close
+```
+
+### With Schema Registry
+
+```typescript
+const registry = KafkaSchemaRegistryHelper.newInstance({ url: 'http://localhost:8081' });
+
+const producer = KafkaProducerHelper.newInstance({
+  ...,
+  registry: registry.getRegistry(),
+  // or use registry.getSerializers() for manual serializer config
+});
+
+const consumer = KafkaConsumerHelper.newInstance({
+  ...,
+  registry: registry.getRegistry(),
+  // or use registry.getDeserializers() for manual deserializer config
+});
+```
+
+### Transaction (Producer Only)
+
+```typescript
+const result = await producer.runInTransaction(async ({ send, addConsumer, addOffset }) => {
+  // Send messages within transaction
+  const result = await send({
+    messages: [{ topic: 'orders', key: 'o1', value: '{"status":"created"}' }],
+  });
+
+  // Optionally add consumer for exactly-once semantics
+  await addConsumer(consumer.getConsumer());
+  await addOffset(message);
+
+  return result;
+});
+```
+
+## Pages
+
+- **[Producer](./producer)** — Producer helper, transactions, and full `@platformatic/kafka` Producer API reference
+- **[Consumer](./consumer)** — Consumer helper, message callbacks, lag monitoring, and full Consumer API reference
+- **[Admin](./admin)** — Admin helper and full Admin API reference
+- **[Schema Registry](./schema-registry)** — Schema registry helper for Avro/Protobuf/JSON Schema validation
+- **[Examples & Troubleshooting](./examples)** — Complete examples, IoC integration, and troubleshooting guide
 
 ## See Also
 
 - **Other Helpers:**
-  - [Queue Helper](../queue/) -- BullMQ, MQTT, and in-memory queues
-  - [Redis Helper](../redis/) -- Redis connection management
+  - [Queue Helper](../queue/) — BullMQ, MQTT, and in-memory queues
+  - [Redis Helper](../redis/) — Redis connection management
 
 - **External Resources:**
-  - [@platformatic/kafka](https://github.com/platformatic/kafka) -- Underlying Kafka client library
+  - [@platformatic/kafka](https://github.com/platformatic/kafka) — Underlying Kafka client library
+  - [Apache Kafka Documentation](https://kafka.apache.org/documentation/) — Official Kafka docs
+  - [KIP-848](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848) — New consumer group protocol