@venizia/ignis-docs 0.0.7-2 → 0.0.7

package/wiki/references/helpers/kafka/producer.md

@@ -1,6 +1,6 @@
 # Producer
 
-The `KafkaProducerHelper` is a thin wrapper around `@platformatic/kafka`'s `Producer`. It manages creation, logging, and lifecycle.
+The `KafkaProducerHelper` wraps `@platformatic/kafka`'s `Producer` with health tracking, graceful shutdown, broker event callbacks, and a transaction helper.
 
 ```typescript
 class KafkaProducerHelper<
@@ -8,7 +8,7 @@ class KafkaProducerHelper<
   ValueType = string,
   HeaderKeyType = string,
   HeaderValueType = string,
-> extends BaseHelper
+> extends BaseKafkaHelper<Producer<KeyType, ValueType, HeaderKeyType, HeaderValueType>>
 ```
 
 ## Helper API
@@ -16,26 +16,34 @@ class KafkaProducerHelper<
 | Method | Signature | Description |
 |--------|-----------|-------------|
 | `newInstance(opts)` | `static newInstance<K,V,HK,HV>(opts): KafkaProducerHelper<K,V,HK,HV>` | Factory method |
-| `getProducer()` | `(): Producer<KeyType, ValueType, HeaderKeyType, HeaderValueType>` | Access the underlying `Producer` |
-| `close(isForce?)` | `(isForce?: boolean): Promise<void>` | Close the producer. Default: `force=false` |
+| `getProducer()` | `(): Producer<K,V,HK,HV>` | Access the underlying `Producer` |
+| `runInTransaction(cb)` | `<R>(cb: TKafkaTransactionCallback<R,K,V,HK,HV>): Promise<R>` | Execute callback within a Kafka transaction |
+| `isHealthy()` | `(): boolean` | `true` when broker connected |
+| `isReady()` | `(): boolean` | Same as `isHealthy()` |
+| `getHealthStatus()` | `(): TKafkaHealthStatus` | `'connected'` \| `'disconnected'` \| `'unknown'` |
+| `close(opts?)` | `(opts?: { isForce?: boolean }): Promise<void>` | Close the producer (default: graceful) |
 
-## IKafkaProducerOpts
+## IKafkaProducerOptions
 
 ```typescript
-interface IKafkaProducerOpts<KeyType, ValueType, HeaderKeyType, HeaderValueType>
+interface IKafkaProducerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueType>
   extends IKafkaConnectionOptions
 ```
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
 | `identifier` | `string` | `'kafka-producer'` | Scoped logging identifier |
-| `serializers` | `Partial<Serializers<K,V,HK,HV>>` | — | Key/value/header serializers. **Pass explicitly** |
+| `serializers` | `Partial<Serializers<K,V,HK,HV>>` | — | Key/value/header serializers |
 | `compression` | `CompressionAlgorithmValue` | — | `'none'`, `'gzip'`, `'snappy'`, `'lz4'`, `'zstd'` |
 | `acks` | `TKafkaAcks` | — | Acknowledgment level: `0`, `1`, or `-1` |
 | `idempotent` | `boolean` | — | Enable idempotent producer (exactly-once within partition) |
 | `transactionalId` | `string` | — | Transactional ID for exactly-once across partitions |
 | `strict` | `boolean` | `true` | Strict mode — fail on unknown topics |
 | `autocreateTopics` | `boolean` | `false` | Auto-create topics on first produce |
+| `shutdownTimeout` | `number` | `30000` | Graceful shutdown timeout in ms |
+| `registry` | `SchemaRegistry` | — | Schema registry for auto ser/deser |
+| `onBrokerConnect` | `TKafkaBrokerEventCallback` | — | Called when broker connects |
+| `onBrokerDisconnect` | `TKafkaBrokerEventCallback` | — | Called when broker disconnects |
 
 Plus all [Connection Options](./#connection-options).
 
@@ -51,41 +59,109 @@ const helper = KafkaProducerHelper.newInstance({
   serializers: stringSerializers,
   acks: KafkaAcks.ALL,
   compression: 'gzip',
+  onBrokerConnect: ({ broker }) => console.log(`Connected to ${broker.host}:${broker.port}`),
+  onBrokerDisconnect: ({ broker }) => console.log(`Disconnected from ${broker.host}`),
 });
 
+// Health check
+helper.isHealthy();      // true when connected
+helper.getHealthStatus(); // 'connected' | 'disconnected' | 'unknown'
+
+// Send messages via the underlying producer
 const producer = helper.getProducer();
 
-// Send a single message
 await producer.send({
   messages: [
     { topic: 'orders', key: 'order-123', value: JSON.stringify({ status: 'created' }) },
   ],
 });
 
-// Send multiple messages (batched in a single request)
+// Batch send (single request, multiple messages)
 await producer.send({
   messages: [
     { topic: 'orders', key: 'order-124', value: JSON.stringify({ status: 'created' }) },
-    { topic: 'orders', key: 'order-125', value: JSON.stringify({ status: 'created' }) },
     { topic: 'inventory', key: 'sku-001', value: JSON.stringify({ delta: -1 }) },
   ],
 });
 
+// Graceful close (waits for in-flight, times out after shutdownTimeout → force)
 await helper.close();
+
+// Or force close immediately
+await helper.close({ isForce: true });
 ```
 
-### Generic Types Example
+## Transactions
+
+`runInTransaction()` wraps `beginTransaction()` → callback → `commit()` / `abort()` with automatic logging.
+
+> [!NOTE]
+> Requires `transactionalId` and `idempotent: true` in producer options.
 
 ```typescript
-// Custom: string keys, JSON object values
-const helper = KafkaProducerHelper.newInstance<string, MyEvent, string, string>({
+const helper = KafkaProducerHelper.newInstance({
   bootstrapBrokers: ['localhost:9092'],
-  clientId: 'typed-producer',
-  serializers: { ...serializersFrom(jsonSerializer), key: stringSerializer },
+  clientId: 'tx-producer',
+  serializers: stringSerializers,
+  transactionalId: 'my-tx-id',
+  idempotent: true,
+});
+
+// Simple transaction
+const result = await helper.runInTransaction(async ({ send }) => {
+  return send({
+    messages: [
+      { topic: 'orders', key: 'o1', value: '{"status":"paid"}' },
+      { topic: 'inventory', key: 'sku-1', value: '{"delta":-1}' },
+    ],
+  });
+});
+
+// Exactly-once consume-transform-produce (with consumer offset commit)
+const result = await helper.runInTransaction(async ({ send, addConsumer, addOffset }) => {
+  // Add consumer to transaction (for exactly-once semantics)
+  await addConsumer(consumer.getConsumer());
+
+  // Add the consumed message offset (will be committed on tx.commit())
+  await addOffset(incomingMessage);
+
+  // Produce transformed result
+  return send({
+    messages: [{ topic: 'output', key: incomingMessage.key, value: transformed }],
+  });
 });
 ```
 
----
+### Transaction Context
+
+The callback receives an `IKafkaTransactionContext`:
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `transaction` | `Transaction` | The underlying platformatic transaction |
+| `send(opts)` | `(opts: SendOptions) => Promise<ProduceResult>` | Send messages within the transaction |
+| `addConsumer(consumer)` | `(consumer: Consumer) => Promise<void>` | Add a consumer for exactly-once |
+| `addOffset(message)` | `(message: Message) => Promise<void>` | Add consumed message offset to transaction |
+
+If the callback throws, the transaction is automatically aborted and the error re-thrown.
+
+## Graceful Shutdown
+
+`close()` implements a two-phase shutdown:
+
+1. **Graceful** (default): Waits for in-flight requests to complete, with a timeout (`shutdownTimeout`, default 30s)
+2. **Force fallback**: If graceful times out, automatically force-closes
+3. **Force** (`{ isForce: true }`): Immediately aborts all in-flight requests
+
+```typescript
+// Graceful (recommended)
+await helper.close();
+
+// Force
+await helper.close({ isForce: true });
+```
+
+After `close()`, `healthStatus` is set to `'disconnected'`.
 
 ## API Reference (`@platformatic/kafka`)
 
@@ -114,7 +190,6 @@ interface MessageToProduce<Key, Value, HeaderKey, HeaderValue> {
   headers?: Map<HeaderKey, HeaderValue> | Record<string, HeaderValue>;
 }
 
-// Returns
 interface ProduceResult {
   offsets?: { topic: string; partition: number; offset: bigint }[];
   unwritableNodes?: number[];
@@ -124,11 +199,6 @@ interface ProduceResult {
 **Examples:**
 
 ```typescript
-// Basic send
-await producer.send({
-  messages: [{ topic: 'events', key: 'user-1', value: '{"action":"login"}' }],
-});
-
 // With headers
 await producer.send({
   messages: [{
@@ -148,75 +218,25 @@ await producer.send({
 await producer.send({
   messages: [{ topic: 'events', key: 'e1', value: 'data', partition: 2 }],
 });
-
-// Override compression per-send
-await producer.send({
-  messages: [{ topic: 'logs', key: 'l1', value: largePayload }],
-  compression: 'zstd',
-});
 ```
 
 ### `producer.asStream(options)`
 
 Create a `Writable` stream for high-throughput producing with automatic batching.
 
-```typescript
-interface ProducerStreamOptions<Key, Value, HeaderKey, HeaderValue> {
-  highWaterMark?: number;   // Stream buffer size
-  batchSize?: number;       // Messages per batch
-  batchTime?: number;       // Max ms before flushing batch
-  reportMode?: 'none' | 'batch' | 'message';
-  // ... plus all SendOptions except messages
-}
-```
-
 ```typescript
 const stream = producer.asStream({ batchSize: 100, batchTime: 1000 });
 
-// Write messages — automatically batched
 stream.write({ topic: 'events', key: 'e1', value: '{"type":"click"}' });
 stream.write({ topic: 'events', key: 'e2', value: '{"type":"scroll"}' });
 
-// Listen for batch completion
 stream.on('data', (report) => {
   console.log(`Batch ${report.batchId}: ${report.count} messages sent`);
 });
 
-// Close when done
 await stream.close();
 ```
 
-### `producer.beginTransaction(options?)`
-
-Start a Kafka transaction for exactly-once semantics across multiple topics/partitions.
-
-> [!NOTE]
-> Requires `transactionalId` in producer options and `idempotent: true`.
-
-```typescript
-const producer = new Producer({
-  clientId: 'tx-producer',
-  bootstrapBrokers: ['localhost:9092'],
-  transactionalId: 'my-tx-id',
-  idempotent: true,
-  serializers: stringSerializers,
-});
-
-const tx = await producer.beginTransaction();
-try {
-  await tx.send({
-    messages: [
-      { topic: 'orders', key: 'o1', value: '{"status":"paid"}' },
-      { topic: 'inventory', key: 'sku-1', value: '{"delta":-1}' },
-    ],
-  });
-  await tx.commit();
-} catch (err) {
-  await tx.abort();
-  throw err;
-}
-```
-
 ### `producer.close(force?)`
 
 Close the producer connection.
@@ -234,35 +254,19 @@ Close the producer connection.
 | `coordinatorId` | `number` | Transaction coordinator broker ID |
 | `streamsCount` | `number` | Number of active producer streams |
 
----
-
 ## Key Partitioning
 
 By default, `@platformatic/kafka` uses **murmur2 hashing** on the message key to determine the target partition:
 
-```
-partition = murmur2(key) % numPartitions
-```
-
 - Same key → always same partition → guaranteed ordering per key
 - `undefined` key → round-robin across partitions
 - Explicit `partition` field → overrides the partitioner
 
 ```typescript
-// Key-based routing: all "user-123" messages go to the same partition
-await producer.send({
-  messages: [
-    { topic: 'events', key: 'user-123', value: '{"action":"login"}' },
-    { topic: 'events', key: 'user-123', value: '{"action":"click"}' },  // Same partition
-    { topic: 'events', key: 'user-456', value: '{"action":"login"}' },  // Different partition
-  ],
-});
-
 // Custom partitioner
 await producer.send({
   messages: [{ topic: 'events', key: 'e1', value: 'data' }],
   partitioner: (message) => {
-    // Route by first character of key
     return message.key!.charCodeAt(0) % 3;
   },
 });

package/wiki/references/helpers/kafka/schema-registry.md

@@ -0,0 +1,214 @@
+# Schema Registry
+
+The `KafkaSchemaRegistryHelper` wraps `@platformatic/kafka`'s `ConfluentSchemaRegistry`. It provides a centralized schema registry that auto-serializes/deserializes messages using registered schemas (Avro, Protobuf, JSON Schema).
+
+```typescript
+class KafkaSchemaRegistryHelper<
+  KeyType = string,
+  ValueType = string,
+  HeaderKeyType = string,
+  HeaderValueType = string,
+> extends BaseHelper
+```
+
+> [!NOTE]
+> `KafkaSchemaRegistryHelper` extends `BaseHelper` directly (not `BaseKafkaHelper`) — it has no broker connection or health tracking. It's a configuration wrapper, not a client.
+
+## Helper API
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `newInstance(opts)` | `static newInstance<K,V,HK,HV>(opts): KafkaSchemaRegistryHelper<K,V,HK,HV>` | Factory method |
+| `getRegistry()` | `(): ConfluentSchemaRegistry<K,V,HK,HV>` | Get the registry instance (pass to producer/consumer) |
+| `getSerializers()` | `(): Serializers<K,V,HK,HV>` | Get schema-based serializers |
+| `getDeserializers()` | `(): Deserializers<K,V,HK,HV>` | Get schema-based deserializers |
+
+## IKafkaSchemaRegistryOptions
+
+```typescript
+interface IKafkaSchemaRegistryOptions extends ConfluentSchemaRegistryOptions {
+  identifier?: string; // Default: 'kafka-schema-registry'
+}
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `url` | `string` | — | Schema registry URL. **Required** |
+| `auth` | `{ username: string; password: string }` | — | Basic auth credentials |
+| `protobufTypeMapper` | `ProtobufTypeMapper` | — | Custom Protobuf type mapper |
+| `jsonValidateSend` | `boolean` | — | Validate JSON schema on produce |
+| `identifier` | `string` | `'kafka-schema-registry'` | Scoped logging identifier |
+
+## What Schema Registry Solves
+
+Without a schema registry, producers and consumers must agree on message format out-of-band. If the producer changes the shape of `value` (adds/removes fields), consumers break silently at runtime.
+
+**Schema Registry** is a centralized server (Confluent Schema Registry) that stores and validates schemas (Avro, Protobuf, JSON Schema). It enforces a contract:
+
+```
+Producer → "I want to send this shape" → Schema Registry validates → Kafka
+Kafka → Consumer → "What shape is this?" → Schema Registry tells → Deserialize
+```
+
+### Without Schema Registry (raw strings)
+
+```typescript
+// Producer — manually serialize
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'order-producer',
+});
+
+await producer.getProducer().send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-1',
+    value: JSON.stringify({ id: 1, total: 99.99 }),  // ← just a string, no validation
+  }],
+});
+
+// Consumer — manually deserialize, hope the shape is correct
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'order-consumer',
+  groupId: 'order-group',
+  onMessage: async ({ message }) => {
+    const order = JSON.parse(message.value as string);  // ← pray it matches
+    console.log(order.id, order.total);
+  },
+});
+```
+
+Problem: if producer adds `{ id: 1, total: 99.99, currency: 'USD' }` or removes `total`, consumer has no way to know until it crashes.
+
+### With Schema Registry (auto serialize/deserialize)
+
+```typescript
+// 1. Create registry — points to Confluent Schema Registry server
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'http://localhost:8081',
+  // auth: { username: 'user', password: 'pass' },  // optional
+});
+
+// 2. Producer — pass registry, it auto-serializes values using registered schema
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'order-producer',
+  registry: registry.getRegistry(),  // ← registry handles serialization
+});
+
+await producer.getProducer().send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-1',
+    value: { id: 1, total: 99.99 },  // ← object, not string! Registry serializes it
+  }],
+});
+// If the value doesn't match the registered schema → error BEFORE sending to Kafka
+
+// 3. Consumer — pass same registry, it auto-deserializes
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'order-consumer',
+  groupId: 'order-group',
+  registry: registry.getRegistry(),  // ← registry handles deserialization
+  onMessage: async ({ message }) => {
+    // message.value is already a typed object, not a raw string
+    console.log(message.value.id, message.value.total);
+  },
+});
+```
+
+### Comparison
+
+| | Without Registry | With Registry |
+|---|---|---|
+| **Message format** | Raw string, manual `JSON.stringify/parse` | Typed object, auto ser/deser |
+| **Validation** | None — runtime crashes | Schema validated before send |
+| **Schema evolution** | Break consumers silently | Backward/forward compatibility enforced |
+| **Where schemas live** | Nowhere (tribal knowledge) | Centralized server `http://registry:8081` |
+
+You only need it when you want **schema enforcement** across producers/consumers. For simple string messages, skip it entirely.
+
+## Basic Usage
+
+```typescript
+import { KafkaSchemaRegistryHelper, KafkaProducerHelper, KafkaConsumerHelper } from '@venizia/ignis-helpers/kafka';
+
+// 1. Create registry — points to Confluent Schema Registry server
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'http://localhost:8081',
+});
+
+// 2. Producer — registry auto-serializes values using registered schema
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'order-producer',
+  registry: registry.getRegistry(),
+});
+
+await producer.getProducer().send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-1',
+    value: { id: 1, total: 99.99 },  // object, not string — registry serializes
+  }],
+});
+// If value doesn't match the registered schema → error BEFORE sending to Kafka
+
+// 3. Consumer — registry auto-deserializes
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'order-consumer',
+  groupId: 'order-group',
+  registry: registry.getRegistry(),
+  onMessage: async ({ message }) => {
+    // message.value is already a typed object, not a raw string
+    console.log(message.value.id, message.value.total);
+  },
+});
+
+await consumer.start({ topics: ['orders'] });
+```
+
+## With Authentication
+
+```typescript
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'https://schema-registry.example.com',
+  auth: {
+    username: 'registry-user',
+    password: 'registry-password',
+  },
+});
+```
+
+## Alternative: Manual Serializers
+
+Instead of passing the full registry, you can extract serializers/deserializers for manual use:
+
+```typescript
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'http://localhost:8081',
+});
+
+// Use serializers directly (instead of registry)
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: registry.getSerializers(),
+});
+
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  deserializers: registry.getDeserializers(),
+  onMessage: async ({ message }) => { ... },
+});
+```
+
+## When to Use
+
+- **Use schema registry** when you need schema enforcement, validation, and compatibility checks across producers/consumers — especially in multi-team environments
+- **Skip schema registry** for simple string/JSON messages where both sides are controlled by the same team and format changes are coordinated