@venizia/ignis-docs 0.0.7-0 → 0.0.7-2

package/wiki/references/helpers/kafka/index.md

@@ -1,305 +1,482 @@
-# Kafka <Badge type="warning" text="Experimental" />
+# Kafka
 
-Apache Kafka event streaming with producer, consumer, and admin helpers. Built on [`@platformatic/kafka`](https://github.com/platformatic/kafka).
+Apache Kafka event streaming with producer, consumer, and admin helpers. Built on [`@platformatic/kafka`](https://github.com/platformatic/kafka) v1.30.0 — a pure TypeScript Kafka client with zero native dependencies.
 
-> [!WARNING]
-> This helper is **experimental**. The API may change in future releases.
+## Overview
 
-## Quick Reference
+The Kafka module provides three **thin wrapper** classes around `@platformatic/kafka`:
 
-| Class | Extends | Peer Dependency | Use Case |
-|-------|---------|-----------------|----------|
-| **KafkaProducerHelper** | `BaseHelper` | `@platformatic/kafka` | Publish messages to Kafka topics |
-| **KafkaConsumerHelper** | `BaseHelper` | `@platformatic/kafka` | Consume messages from Kafka topics with consumer groups |
-| **KafkaAdminHelper** | `BaseHelper` | `@platformatic/kafka` | Manage topics, partitions, consumer groups, and configs |
+| Class | Wraps | Use Case |
+|-------|-------|----------|
+| `KafkaProducerHelper` | `Producer` | Publish messages to Kafka topics |
+| `KafkaConsumerHelper` | `Consumer` | Consume messages with consumer groups |
+| `KafkaAdminHelper` | `Admin` | Manage topics, partitions, groups, ACLs, configs |
+
+Each helper provides:
+- **Scoped logging** via `BaseHelper` (Winston with daily rotation)
+- **Sensible defaults** via `KafkaDefaults` constants
+- **Factory pattern** via `newInstance()` static method
+- **Lifecycle management** via `close()` method
+
+They do **not** re-implement or passthrough `@platformatic/kafka` methods. Use `getProducer()`, `getConsumer()`, or `getAdmin()` to access the full underlying API directly.
 
 ### Import Path
 
 ```typescript
+// Helpers & constants
 import {
   KafkaProducerHelper,
   KafkaConsumerHelper,
   KafkaAdminHelper,
   KafkaDefaults,
   KafkaAcks,
-  KafkaConfigResourceTypes,
+  KafkaGroupProtocol,
 } from '@venizia/ignis-helpers/kafka';
 
+// Types
 import type {
   IKafkaConnectionOptions,
-  IKafkaProducerOptions,
-  IKafkaConsumerOptions,
-  IKafkaAdminOptions,
-  IKafkaProduceMessage,
-  IKafkaSendOptions,
-  IKafkaConsumedMessage,
-  IKafkaCommitOptions,
+  IKafkaProducerOpts,
+  IKafkaConsumerOpts,
+  IKafkaAdminOpts,
+  TKafkaAcks,
+  TKafkaGroupProtocol,
 } from '@venizia/ignis-helpers/kafka';
+
+// @platformatic/kafka (direct usage)
+import {
+  Producer, Consumer, Admin, MessagesStream,
+  stringSerializers, stringDeserializers,
+  stringSerializer, stringDeserializer,
+  jsonSerializer, jsonDeserializer,
+  serializersFrom, deserializersFrom,
+} from '@platformatic/kafka';
+
+import type {
+  Message, MessageToProduce,
+  SendOptions, ConsumeOptions,
+  Serializers, Deserializers,
+  SASLOptions, ConnectionOptions,
+} from '@platformatic/kafka';
 ```
 
-## Installation
+### Installation
 
 ```bash
 bun add @platformatic/kafka
 ```
 
-## Producer
+---
 
-The `KafkaProducerHelper` wraps `@platformatic/kafka`'s `Producer` for publishing messages to Kafka topics.
+## Connection Options
+
+All three helpers share a common base interface `IKafkaConnectionOptions` which extends `@platformatic/kafka`'s `ConnectionOptions`.
 
 ```typescript
-import { KafkaProducerHelper, KafkaAcks } from '@venizia/ignis-helpers/kafka';
+interface IKafkaConnectionOptions extends ConnectionOptions {
+  bootstrapBrokers: string[];
+  clientId: string;
+  retries?: number;    // Default: 3
+  retryDelay?: number; // Default: 1000ms
+}
+```
 
-const producer = new KafkaProducerHelper({
-  identifier: 'order-producer',
-  bootstrapBrokers: ['localhost:9092'],
-  acks: KafkaAcks.ALL,
-  autocreateTopics: true,
-  onConnected: () => console.log('Producer connected'),
-  onError: ({ error }) => console.error('Producer error:', error),
+### Full Options Table
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `bootstrapBrokers` | `string[]` | — | Kafka broker addresses (`host:port`). **Required** |
+| `clientId` | `string` | — | Unique client identifier. **Required** |
+| `retries` | `number` | `3` | Number of connection retries before failing |
+| `retryDelay` | `number` | `1000` | Delay between retries in milliseconds |
+| `sasl` | `SASLOptions` | — | SASL authentication configuration |
+| `tls` | `TLSConnectionOptions` | — | TLS/SSL connection options |
+| `ssl` | `TLSConnectionOptions` | — | Alias for `tls` |
+| `connectTimeout` | `number` | — | TCP connection timeout in milliseconds |
+| `requestTimeout` | `number` | — | Kafka request timeout in milliseconds |
+
+### SASL Authentication
+
+`@platformatic/kafka` supports five SASL mechanisms:
+
+| Mechanism | Use Case |
+|-----------|----------|
+| `PLAIN` | Simple username/password (use with TLS in production) |
+| `SCRAM-SHA-256` | Challenge-response, password never sent in plaintext |
+| `SCRAM-SHA-512` | Same as SHA-256 with stronger hash |
+| `OAUTHBEARER` | Token-based (Azure Event Hubs, Confluent Cloud) |
+| `GSSAPI` | Kerberos authentication |
+
+```typescript
+interface SASLOptions {
+  mechanism: 'PLAIN' | 'SCRAM-SHA-256' | 'SCRAM-SHA-512' | 'OAUTHBEARER' | 'GSSAPI';
+  username?: string | CredentialProvider;
+  password?: string | CredentialProvider;
+  token?: string | CredentialProvider;
+  oauthBearerExtensions?: Record<string, string> | CredentialProvider<Record<string, string>>;
+  authenticate?: SASLCustomAuthenticator;
+}
+```
+
+#### SCRAM-SHA-512 Example
+
+```typescript
+const helper = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['broker1:9092', 'broker2:9092', 'broker3:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  sasl: {
+    mechanism: 'SCRAM-SHA-512',
+    username: 'kafka-user',
+    password: 'kafka-password',
+  },
+  connectTimeout: 30_000,
+  requestTimeout: 30_000,
 });
+```
 
-// Send messages
-await producer.send({
-  messages: [
-    { topic: 'orders', key: 'order-123', value: JSON.stringify({ status: 'created' }) },
-  ],
+#### OAUTHBEARER Example
+
+```typescript
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['pkc-xxxxx.us-west-2.aws.confluent.cloud:9092'],
+  clientId: 'my-producer',
+  sasl: {
+    mechanism: 'OAUTHBEARER',
+    token: async () => {
+      const response = await fetch('https://auth.example.com/token', { method: 'POST' });
+      const { access_token } = await response.json();
+      return access_token;
+    },
+  },
+  tls: true,
 });
+```
 
-// Send batch across multiple topics
-await producer.sendBatch({
-  topicMessages: [
-    { topic: 'orders', messages: [{ key: 'o1', value: '...' }] },
-    { topic: 'notifications', messages: [{ key: 'n1', value: '...' }] },
-  ],
+#### TLS Without SASL
+
+```typescript
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['broker:9093'],
+  clientId: 'my-producer',
+  tls: {
+    ca: fs.readFileSync('/path/to/ca.pem'),
+    cert: fs.readFileSync('/path/to/client-cert.pem'),
+    key: fs.readFileSync('/path/to/client-key.pem'),
+  },
 });
+```
+
+---
 
-// Graceful shutdown
-await producer.close();
+## Serialization & Deserialization
+
+`@platformatic/kafka`'s default wire format is `Buffer`. The helpers default generic types to `string` (matching common usage), but you must provide serializers/deserializers explicitly.
+
+### Built-in Serializers
+
+| Export | Type | Description |
+|--------|------|-------------|
+| `stringSerializer` | `Serializer<string>` | `string → Buffer` (UTF-8) |
+| `stringDeserializer` | `Deserializer<string>` | `Buffer → string` (UTF-8) |
+| `jsonSerializer` | `Serializer<T>` | `object → Buffer` (JSON.stringify + UTF-8) |
+| `jsonDeserializer` | `Deserializer<T>` | `Buffer → object` (UTF-8 + JSON.parse) |
+| `stringSerializers` | `Serializers<string, string, string, string>` | All four positions as string |
+| `stringDeserializers` | `Deserializers<string, string, string, string>` | All four positions as string |
+
+### Helper Functions
+
+| Export | Signature | Description |
+|--------|-----------|-------------|
+| `serializersFrom(s)` | `<T>(s: Serializer<T>) => Serializers<T, T, T, T>` | Create full serializers from a single serializer |
+| `deserializersFrom(d)` | `<T>(d: Deserializer<T>) => Deserializers<T, T, T, T>` | Create full deserializers from a single deserializer |
+
+### Serializers/Deserializers Interface
+
+```typescript
+interface Serializers<Key, Value, HeaderKey, HeaderValue> {
+  key: SerializerWithHeaders<Key, HeaderKey, HeaderValue>;
+  value: SerializerWithHeaders<Value, HeaderKey, HeaderValue>;
+  headerKey: Serializer<HeaderKey>;
+  headerValue: Serializer<HeaderValue>;
+}
+
+interface Deserializers<Key, Value, HeaderKey, HeaderValue> {
+  key: DeserializerWithHeaders<Key, HeaderKey, HeaderValue>;
+  value: DeserializerWithHeaders<Value, HeaderKey, HeaderValue>;
+  headerKey: Deserializer<HeaderKey>;
+  headerValue: Deserializer<HeaderValue>;
+}
 ```
 
-### IKafkaProducerOptions
+### String Serialization
 
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `bootstrapBrokers` | `string[]` | -- | Kafka broker addresses (required) |
-| `identifier` | `string` | -- | Scoped logging identifier |
-| `clientId` | `string` | `'ignis-kafka'` | Kafka client ID |
-| `acks` | `number` | -- | Acknowledgment level (`KafkaAcks.NONE`, `LEADER`, `ALL`) |
-| `autocreateTopics` | `boolean` | -- | Auto-create topics on first produce |
-| `timeout` | `number` | -- | Connection timeout in ms |
-| `retries` | `number \| boolean` | -- | Retry configuration |
-| `retryDelay` | `number` | -- | Delay between retries in ms |
-| `serializers` | `Partial<Serializers>` | -- | Custom key/value/header serializers |
-| `onConnected` | `() => void` | -- | Broker connect callback |
-| `onDisconnected` | `() => void` | -- | Broker disconnect callback |
-| `onError` | `(opts: { error: Error }) => void` | -- | Error callback |
-
-### Producer API
-
-| Method | Returns | Description |
-|--------|---------|-------------|
-| `send(opts)` | `Promise<void>` | Send messages. `opts: { messages: IKafkaProduceMessage[]; acks? }` |
-| `sendBatch(opts)` | `Promise<void>` | Send to multiple topics. `opts: { topicMessages: Array<{ topic; messages }> }` |
-| `getProducer()` | `Producer` | Access the underlying `@platformatic/kafka` Producer |
-| `close()` | `Promise<void>` | Gracefully close the producer connection |
-| `static newInstance(opts)` | `KafkaProducerHelper` | Factory method |
-
-## Consumer
-
-The `KafkaConsumerHelper` provides a stream-based consumer with consumer group support, pause/resume, manual commit, and lag monitoring.
+The simplest approach — all keys, values, and headers are strings:
 
 ```typescript
-import { KafkaConsumerHelper } from '@venizia/ignis-helpers/kafka';
+import { stringSerializers, stringDeserializers } from '@platformatic/kafka';
 
-const consumer = new KafkaConsumerHelper({
-  identifier: 'order-consumer',
+// Producer
+const producer = KafkaProducerHelper.newInstance({
   bootstrapBrokers: ['localhost:9092'],
-  groupId: 'order-processing-group',
-  topics: ['orders'],
-  mode: 'latest',
-  autocommit: true,
-  onMessage: async ({ message }) => {
-    console.log(`Topic: ${message.topic}, Partition: ${message.partition}`);
-    console.log(`Key: ${message.key}, Value: ${message.value}`);
-  },
-  onConnected: () => console.log('Consumer connected'),
-  onGroupJoin: ({ groupId, memberId }) => {
-    console.log(`Joined group ${groupId} as ${memberId}`);
-  },
-  onError: ({ error }) => console.error('Consumer error:', error),
+  clientId: 'my-producer',
+  serializers: stringSerializers,
 });
 
-// Start consuming
-await consumer.start();
-
-// Pause/resume
-consumer.pause();
-consumer.resume();
-
-// Graceful shutdown
-await consumer.close();
+// Consumer
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  deserializers: stringDeserializers,
+});
 ```
 
-### IKafkaConsumerOptions
+### JSON Serialization
 
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `bootstrapBrokers` | `string[]` | -- | Kafka broker addresses (required) |
-| `groupId` | `string` | -- | Consumer group ID (required) |
-| `topics` | `string[]` | -- | Topics to consume (required) |
-| `identifier` | `string` | -- | Scoped logging identifier |
-| `clientId` | `string` | `'ignis-kafka'` | Kafka client ID |
-| `mode` | `'latest' \| 'earliest' \| 'committed'` | `'latest'` | Offset reset strategy |
-| `autocommit` | `boolean \| number` | `true` | Auto-commit offsets (or interval in ms) |
-| `sessionTimeout` | `number` | `30000` | Session timeout in ms |
-| `heartbeatInterval` | `number` | `3000` | Heartbeat interval in ms |
-| `highWaterMark` | `number` | `1024` | Stream high water mark |
-| `maxWaitTime` | `number` | `5000` | Max wait time for fetch in ms |
-| `deserializers` | `Partial<Deserializers>` | -- | Custom key/value/header deserializers |
-| `onMessage` | `(opts: { message }) => ValueOrPromise<void>` | -- | Message handler |
-| `onConnected` | `() => void` | -- | Broker connect callback |
-| `onDisconnected` | `() => void` | -- | Broker disconnect callback |
-| `onGroupJoin` | `(opts: { groupId; memberId }) => void` | -- | Consumer group join callback |
-| `onGroupLeave` | `() => void` | -- | Consumer group leave callback |
-| `onRebalance` | `() => void` | -- | Group rebalance callback |
-| `onLag` | `(opts: { offsets }) => void` | -- | Consumer lag callback |
-| `onError` | `(opts: { error: Error }) => void` | -- | Error callback |
-
-### Consumer API
-
-| Method | Returns | Description |
-|--------|---------|-------------|
-| `start()` | `Promise<void>` | Start consuming messages (fires async consume loop) |
-| `pause()` | `void` | Pause the message stream |
-| `resume()` | `void` | Resume the message stream |
-| `isPaused()` | `boolean` | Check if the stream is paused |
-| `isConsuming()` | `boolean` | Check if the consumer is running |
-| `commit(opts)` | `Promise<void>` | Manually commit offsets |
-| `startLagMonitoring(opts)` | `void` | Start lag monitoring. `opts: { interval: number }` |
-| `stopLagMonitoring()` | `void` | Stop lag monitoring |
-| `getConsumer()` | `Consumer` | Access the underlying `@platformatic/kafka` Consumer |
-| `close()` | `Promise<void>` | Abort consume loop, close stream and consumer |
-| `static newInstance(opts)` | `KafkaConsumerHelper` | Factory method |
-
-### Manual Commit
-
-When `autocommit` is `false`, commit offsets explicitly:
+For structured data — serialize objects as JSON:
 
 ```typescript
-const consumer = new KafkaConsumerHelper({
-  // ...
-  autocommit: false,
-  onMessage: async ({ message }) => {
-    await processMessage(message);
-    // Commit after successful processing
-    await consumer.commit({
-      offsets: [{
-        topic: message.topic,
-        partition: message.partition,
-        offset: message.offset,
-        leaderEpoch: 0,
-      }],
-    });
-  },
+import {
+  jsonSerializer, jsonDeserializer,
+  stringSerializer, stringDeserializer,
+  serializersFrom, deserializersFrom,
+} from '@platformatic/kafka';
+
+// JSON values with string keys
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: { ...serializersFrom(jsonSerializer), key: stringSerializer },
+});
+
+const p = producer.getProducer();
+await p.send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-123',                                    // string key
+    value: { id: '123', status: 'created', amount: 99 }, // object value → auto-serialized to JSON
+  }],
+});
+
+// Consumer with matching deserializers
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  deserializers: { ...deserializersFrom(jsonDeserializer), key: stringDeserializer },
 });
 ```
 
-## Admin
+### Custom Serialization
 
-The `KafkaAdminHelper` provides topic, partition, consumer group, and config management.
+For advanced use cases (Avro, Protobuf, MessagePack):
 
 ```typescript
-import { KafkaAdminHelper, KafkaConfigResourceTypes } from '@venizia/ignis-helpers/kafka';
+import type { Serializer, Deserializer } from '@platformatic/kafka';
+import * as msgpack from '@msgpack/msgpack';
+
+const msgpackSerializer: Serializer<unknown> = (data) => {
+  if (data === undefined) return undefined;
+  return Buffer.from(msgpack.encode(data));
+};
+
+const msgpackDeserializer: Deserializer<unknown> = (data) => {
+  if (!data) return undefined;
+  return msgpack.decode(data);
+};
 
-const admin = new KafkaAdminHelper({
-  identifier: 'kafka-admin',
+const producer = KafkaProducerHelper.newInstance<string, unknown, string, string>({
   bootstrapBrokers: ['localhost:9092'],
+  clientId: 'msgpack-producer',
+  serializers: {
+    key: stringSerializer,
+    value: msgpackSerializer,
+    headerKey: stringSerializer,
+    headerValue: stringSerializer,
+  },
 });
+```
 
-// Topic management
-await admin.createTopics({ topics: ['orders', 'notifications'], partitions: 3, replicas: 1 });
-const topics = await admin.listTopics();
-await admin.deleteTopics({ topics: ['old-topic'] });
+---
 
-// Partition management
-await admin.createPartitions({ topics: [{ name: 'orders', count: 6 }] });
+## Generic Type Parameters
 
-// Consumer group management
-const groups = await admin.listGroups();
-const groupDetails = await admin.describeGroups({ groups: ['order-processing-group'] });
-await admin.deleteGroups({ groups: ['stale-group'] });
+All three helpers (and their option interfaces) support generic type parameters controlling the serialization types:
 
-// Config management
-const configs = await admin.describeConfigs({
-  resources: [{
-    resourceType: KafkaConfigResourceTypes.TOPIC,
-    resourceName: 'orders',
-  }],
-});
+```typescript
+class KafkaProducerHelper<
+  KeyType = string,
+  ValueType = string,
+  HeaderKeyType = string,
+  HeaderValueType = string,
+>
+```
 
-// Metadata
-const meta = await admin.metadata({ topics: ['orders'] });
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `KeyType` | `string` | Message key type after serialization/deserialization |
+| `ValueType` | `string` | Message value type after serialization/deserialization |
+| `HeaderKeyType` | `string` | Header key type |
+| `HeaderValueType` | `string` | Header value type |
 
-// Cleanup
-await admin.close();
+> [!NOTE]
+> `@platformatic/kafka` defaults to `Buffer` for all four positions. The helpers default to `string` which is more common for application code. If you don't pass serializers, your messages will be sent/received as `Buffer`.
+
+```typescript
+// Default: string types (most common)
+const helper = KafkaProducerHelper.newInstance({ ... });
+
+// Custom: string keys, JSON object values
+const helper = KafkaProducerHelper.newInstance<string, MyEvent, string, string>({
+  serializers: { ...serializersFrom(jsonSerializer), key: stringSerializer },
+  ...
+});
+
+// Custom: Buffer keys, Buffer values (raw wire format)
+const helper = KafkaProducerHelper.newInstance<Buffer, Buffer, Buffer, Buffer>({
+  // No serializers needed — @platformatic/kafka defaults to Buffer
+  ...
+});
 ```
 
-### Admin API
-
-| Method | Returns | Description |
-|--------|---------|-------------|
-| `createTopics(opts)` | `Promise<any>` | Create topics with partitions and replicas |
-| `deleteTopics(opts)` | `Promise<void>` | Delete topics |
-| `listTopics(opts?)` | `Promise<string[]>` | List topics (optionally include internals) |
-| `metadata(opts?)` | `Promise<any>` | Fetch cluster/topic metadata |
-| `listGroups(opts?)` | `Promise<any>` | List consumer groups (filter by state) |
-| `describeGroups(opts)` | `Promise<any>` | Describe consumer groups |
-| `deleteGroups(opts)` | `Promise<void>` | Delete consumer groups |
-| `listConsumerGroupOffsets(opts)` | `Promise<any>` | List offsets for consumer groups |
-| `alterConsumerGroupOffsets(opts)` | `Promise<void>` | Alter consumer group offsets |
-| `createPartitions(opts)` | `Promise<void>` | Create partitions for topics |
-| `describeConfigs(opts)` | `Promise<any>` | Describe resource configs |
-| `alterConfigs(opts)` | `Promise<void>` | Alter resource configs |
-| `getAdmin()` | `Admin` | Access the underlying `@platformatic/kafka` Admin |
-| `close()` | `Promise<void>` | Close the admin connection |
-| `static newInstance(opts)` | `KafkaAdminHelper` | Factory method |
+---
 
 ## Constants
 
 ### KafkaDefaults
 
-| Constant | Value | Description |
-|----------|-------|-------------|
-| `CLIENT_ID` | `'ignis-kafka'` | Default Kafka client ID |
-| `SESSION_TIMEOUT` | `30000` | Session timeout (ms) |
-| `HEARTBEAT_INTERVAL` | `3000` | Heartbeat interval (ms) |
-| `MAX_WAIT_TIME` | `5000` | Max fetch wait time (ms) |
-| `HIGH_WATER_MARK` | `1024` | Stream high water mark |
-| `AUTOCOMMIT_INTERVAL` | `100` | Auto-commit interval (ms) |
+Centralized default values used by all three helpers.
+
+```typescript
+import { KafkaDefaults } from '@venizia/ignis-helpers/kafka';
+```
+
+| Constant | Value | Scope | Used By | Description |
+|----------|-------|-------|---------|-------------|
+| `RETRIES` | `3` | Shared | All helpers | Connection retry count |
+| `RETRY_DELAY` | `1000` | Shared | All helpers | Retry delay in ms |
+| `STRICT` | `true` | Producer | `KafkaProducerHelper` | Fail on unknown topics |
+| `AUTOCREATE_TOPICS` | `false` | Producer | `KafkaProducerHelper` | Auto-create topics on produce |
+| `AUTOCOMMIT` | `false` | Consumer | `KafkaConsumerHelper` | Auto-commit offsets |
+| `SESSION_TIMEOUT` | `30000` | Consumer | `KafkaConsumerHelper` | Session timeout in ms |
+| `HEARTBEAT_INTERVAL` | `3000` | Consumer | `KafkaConsumerHelper` | Heartbeat interval in ms |
+| `HIGH_WATER_MARK` | `1024` | Consumer | `KafkaConsumerHelper` | Stream buffer size (messages) |
+| `MIN_BYTES` | `1` | Consumer | `KafkaConsumerHelper` | Min bytes per fetch |
+| `METADATA_MAX_AGE` | `300000` | Consumer | `KafkaConsumerHelper` | Metadata cache TTL in ms |
+| `GROUP_PROTOCOL` | `'classic'` | Consumer | `KafkaConsumerHelper` | Default group protocol |
 
 ### KafkaAcks
 
-| Constant | Value | Description |
-|----------|-------|-------------|
-| `NONE` | `0` | No acknowledgment |
-| `LEADER` | `1` | Leader acknowledgment only |
-| `ALL` | `-1` | All replicas must acknowledge |
+Producer acknowledgment levels.
+
+```typescript
+import { KafkaAcks } from '@venizia/ignis-helpers/kafka';
+```
+
+| Constant | Value | Description | Trade-off |
+|----------|-------|-------------|-----------|
+| `NONE` | `0` | No acknowledgment — fire-and-forget | Fastest, no durability guarantee |
+| `LEADER` | `1` | Leader broker acknowledges | Fast, leader-durable |
+| `ALL` | `-1` | All in-sync replicas acknowledge | Slowest, fully durable |
+
+**Static methods:**
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `isValid(ack)` | `(ack: number): boolean` | Check if value is a valid ack level |
+| `SCHEME_SET` | `Set<number>` | Set of valid values: `{0, 1, -1}` |
+
+```typescript
+KafkaAcks.ALL;           // -1
+KafkaAcks.isValid(-1);   // true
+KafkaAcks.isValid(2);    // false
+KafkaAcks.SCHEME_SET;    // Set { 0, 1, -1 }
+```
+
+### KafkaGroupProtocol
 
-### KafkaConfigResourceTypes
+Consumer group protocol versions.
+
+```typescript
+import { KafkaGroupProtocol } from '@venizia/ignis-helpers/kafka';
+```
 
 | Constant | Value | Description |
 |----------|-------|-------------|
-| `UNKNOWN` | `0` | Unknown resource type |
-| `TOPIC` | `2` | Topic resource |
-| `BROKER` | `4` | Broker resource |
-| `BROKER_LOGGER` | `8` | Broker logger resource |
+| `CLASSIC` | `'classic'` | Classic consumer group protocol (default, all Kafka versions) |
+| `CONSUMER` | `'consumer'` | New consumer group protocol — KIP-848 (Kafka 3.7+) |
+
+**Static methods:**
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `isValid(mode)` | `(mode: string): boolean` | Check if value is a valid protocol |
+| `SCHEME_SET` | `Set<string>` | Set of valid values: `{'classic', 'consumer'}` |
+
+```typescript
+KafkaGroupProtocol.CLASSIC;            // 'classic'
+KafkaGroupProtocol.isValid('classic');  // true
+KafkaGroupProtocol.isValid('foo');      // false
+```
+
+### Derived Types
+
+```typescript
+import type { TKafkaAcks, TKafkaGroupProtocol } from '@venizia/ignis-helpers/kafka';
+
+// TKafkaAcks = 0 | 1 | -1
+// TKafkaGroupProtocol = 'classic' | 'consumer'
+```
+
+These union types are derived using `TConstValue<T>` from the constant classes.
+
+---
+
+## Compression
+
+`@platformatic/kafka` supports five compression algorithms:
+
+| Algorithm | Value | Description |
+|-----------|-------|-------------|
+| None | `'none'` | No compression (default) |
+| GZIP | `'gzip'` | Good compression ratio, moderate CPU |
+| Snappy | `'snappy'` | Fast compression, moderate ratio |
+| LZ4 | `'lz4'` | Very fast, good for high-throughput |
+| Zstandard | `'zstd'` | Best ratio, moderate CPU |
+
+```typescript
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: stringSerializers,
+  compression: 'zstd', // Applied to all messages by default
+});
+
+// Override per-send
+const producer = helper.getProducer();
+await producer.send({
+  messages: [{ topic: 'logs', key: 'l1', value: largePayload }],
+  compression: 'lz4',
+});
+```
+
+---
+
+## Pages
+
+- **[Producer](./producer)** — Producer helper setup, usage, and full `@platformatic/kafka` Producer API reference
+- **[Consumer](./consumer)** — Consumer helper setup, usage, and full `@platformatic/kafka` Consumer API reference
+- **[Admin](./admin)** — Admin helper setup, usage, and full `@platformatic/kafka` Admin API reference
+- **[Examples & Troubleshooting](./examples)** — Complete examples, IoC integration, and troubleshooting guide
 
 ## See Also
 
 - **Other Helpers:**
-  - [Queue Helper](../queue/) -- BullMQ, MQTT, and in-memory queues
-  - [Redis Helper](../redis/) -- Redis connection management
+  - [Queue Helper](../queue/) — BullMQ, MQTT, and in-memory queues
+  - [Redis Helper](../redis/) — Redis connection management
 
 - **External Resources:**
-  - [@platformatic/kafka](https://github.com/platformatic/kafka) -- Underlying Kafka client library
+  - [@platformatic/kafka](https://github.com/platformatic/kafka) — Underlying Kafka client library
+  - [Apache Kafka Documentation](https://kafka.apache.org/documentation/) — Official Kafka docs
+  - [KIP-848](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848) — New consumer group protocol