@venizia/ignis-docs 0.0.7-1 → 0.0.7

package/wiki/references/helpers/kafka/producer.md

@@ -0,0 +1,273 @@
+# Producer
+
+The `KafkaProducerHelper` wraps `@platformatic/kafka`'s `Producer` with health tracking, graceful shutdown, broker event callbacks, and a transaction helper.
+
+```typescript
+class KafkaProducerHelper<
+  KeyType = string,
+  ValueType = string,
+  HeaderKeyType = string,
+  HeaderValueType = string,
+> extends BaseKafkaHelper<Producer<KeyType, ValueType, HeaderKeyType, HeaderValueType>>
+```
+
+## Helper API
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `newInstance(opts)` | `static newInstance<K,V,HK,HV>(opts): KafkaProducerHelper<K,V,HK,HV>` | Factory method |
+| `getProducer()` | `(): Producer<K,V,HK,HV>` | Access the underlying `Producer` |
+| `runInTransaction(cb)` | `<R>(cb: TKafkaTransactionCallback<R,K,V,HK,HV>): Promise<R>` | Execute callback within a Kafka transaction |
+| `isHealthy()` | `(): boolean` | `true` when broker connected |
+| `isReady()` | `(): boolean` | Same as `isHealthy()` |
+| `getHealthStatus()` | `(): TKafkaHealthStatus` | `'connected'` \| `'disconnected'` \| `'unknown'` |
+| `close(opts?)` | `(opts?: { isForce?: boolean }): Promise<void>` | Close the producer (default: graceful) |
+
+## IKafkaProducerOptions
+
+```typescript
+interface IKafkaProducerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueType>
+  extends IKafkaConnectionOptions
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `identifier` | `string` | `'kafka-producer'` | Scoped logging identifier |
+| `serializers` | `Partial<Serializers<K,V,HK,HV>>` | — | Key/value/header serializers |
+| `compression` | `CompressionAlgorithmValue` | — | `'none'`, `'gzip'`, `'snappy'`, `'lz4'`, `'zstd'` |
+| `acks` | `TKafkaAcks` | — | Acknowledgment level: `0`, `1`, or `-1` |
+| `idempotent` | `boolean` | — | Enable idempotent producer (exactly-once within partition) |
+| `transactionalId` | `string` | — | Transactional ID for exactly-once across partitions |
+| `strict` | `boolean` | `true` | Strict mode — fail on unknown topics |
+| `autocreateTopics` | `boolean` | `false` | Auto-create topics on first produce |
+| `shutdownTimeout` | `number` | `30000` | Graceful shutdown timeout in ms |
+| `registry` | `SchemaRegistry` | — | Schema registry for auto ser/deser |
+| `onBrokerConnect` | `TKafkaBrokerEventCallback` | — | Called when broker connects |
+| `onBrokerDisconnect` | `TKafkaBrokerEventCallback` | — | Called when broker disconnects |
+
+Plus all [Connection Options](./#connection-options).
+
+## Basic Example
+
+```typescript
+import { KafkaProducerHelper, KafkaAcks } from '@venizia/ignis-helpers/kafka';
+import { stringSerializers } from '@platformatic/kafka';
+
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'order-producer',
+  serializers: stringSerializers,
+  acks: KafkaAcks.ALL,
+  compression: 'gzip',
+  onBrokerConnect: ({ broker }) => console.log(`Connected to ${broker.host}:${broker.port}`),
+  onBrokerDisconnect: ({ broker }) => console.log(`Disconnected from ${broker.host}`),
+});
+
+// Health check
+helper.isHealthy();      // true when connected
+helper.getHealthStatus(); // 'connected' | 'disconnected' | 'unknown'
+
+// Send messages via the underlying producer
+const producer = helper.getProducer();
+
+await producer.send({
+  messages: [
+    { topic: 'orders', key: 'order-123', value: JSON.stringify({ status: 'created' }) },
+  ],
+});
+
+// Batch send (single request, multiple messages)
+await producer.send({
+  messages: [
+    { topic: 'orders', key: 'order-124', value: JSON.stringify({ status: 'created' }) },
+    { topic: 'inventory', key: 'sku-001', value: JSON.stringify({ delta: -1 }) },
+  ],
+});
+
+// Graceful close (waits for in-flight, times out after shutdownTimeout → force)
+await helper.close();
+
+// Or force close immediately
+await helper.close({ isForce: true });
+```
+
+## Transactions
+
+`runInTransaction()` wraps `beginTransaction()` → callback → `commit()` / `abort()` with automatic logging.
+
+> [!NOTE]
+> Requires `transactionalId` and `idempotent: true` in producer options.
+
+```typescript
+const helper = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'tx-producer',
+  serializers: stringSerializers,
+  transactionalId: 'my-tx-id',
+  idempotent: true,
+});
+
+// Simple transaction
+const result = await helper.runInTransaction(async ({ send }) => {
+  return send({
+    messages: [
+      { topic: 'orders', key: 'o1', value: '{"status":"paid"}' },
+      { topic: 'inventory', key: 'sku-1', value: '{"delta":-1}' },
+    ],
+  });
+});
+
+// Exactly-once consume-transform-produce (with consumer offset commit)
+const result = await helper.runInTransaction(async ({ send, addConsumer, addOffset }) => {
+  // Add consumer to transaction (for exactly-once semantics)
+  await addConsumer(consumer.getConsumer());
+
+  // Add the consumed message offset (will be committed on tx.commit())
+  await addOffset(incomingMessage);
+
+  // Produce transformed result
+  return send({
+    messages: [{ topic: 'output', key: incomingMessage.key, value: transformed }],
+  });
+});
+```
+
+### Transaction Context
+
+The callback receives an `IKafkaTransactionContext`:
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `transaction` | `Transaction` | The underlying platformatic transaction |
+| `send(opts)` | `(opts: SendOptions) => Promise<ProduceResult>` | Send messages within the transaction |
+| `addConsumer(consumer)` | `(consumer: Consumer) => Promise<void>` | Add a consumer for exactly-once |
+| `addOffset(message)` | `(message: Message) => Promise<void>` | Add consumed message offset to transaction |
+
+If the callback throws, the transaction is automatically aborted and the error re-thrown.
+
+## Graceful Shutdown
+
+`close()` implements a two-phase shutdown:
+
+1. **Graceful** (default): Waits for in-flight requests to complete, with a timeout (`shutdownTimeout`, default 30s)
+2. **Force fallback**: If graceful times out, automatically force-closes
+3. **Force** (`{ isForce: true }`): Immediately aborts all in-flight requests
+
+```typescript
+// Graceful (recommended)
+await helper.close();
+
+// Force
+await helper.close({ isForce: true });
+```
+
+After `close()`, `healthStatus` is set to `'disconnected'`.
+
+## API Reference (`@platformatic/kafka`)
+
+After calling `helper.getProducer()`, you have full access to the `Producer` class:
+
+### `producer.send(options)`
+
+Send messages to one or more topics.
+
+```typescript
+interface SendOptions<Key, Value, HeaderKey, HeaderValue> {
+  messages: MessageToProduce<Key, Value, HeaderKey, HeaderValue>[];
+  acks?: number;
+  compression?: CompressionAlgorithmValue;
+  partitioner?: Partitioner<Key, Value, HeaderKey, HeaderValue>;
+  idempotent?: boolean;
+  autocreateTopics?: boolean;
+}
+
+interface MessageToProduce<Key, Value, HeaderKey, HeaderValue> {
+  topic: string;
+  key?: Key;
+  value?: Value;
+  partition?: number;     // Explicit partition (overrides partitioner)
+  timestamp?: bigint;     // Message timestamp
+  headers?: Map<HeaderKey, HeaderValue> | Record<string, HeaderValue>;
+}
+
+interface ProduceResult {
+  offsets?: { topic: string; partition: number; offset: bigint }[];
+  unwritableNodes?: number[];
+}
+```
+
+**Examples:**
+
+```typescript
+// With headers
+await producer.send({
+  messages: [{
+    topic: 'events',
+    key: 'user-1',
+    value: '{"action":"login"}',
+    headers: { 'x-trace-id': 'abc123', 'x-source': 'auth-service' },
+  }],
+});
+
+// Tombstone (delete compacted key)
+await producer.send({
+  messages: [{ topic: 'users', key: 'user-deleted-123', value: undefined }],
+});
+
+// Explicit partition
+await producer.send({
+  messages: [{ topic: 'events', key: 'e1', value: 'data', partition: 2 }],
+});
+```
+
+### `producer.asStream(options)`
+
+Create a `Writable` stream for high-throughput producing with automatic batching.
+
+```typescript
+const stream = producer.asStream({ batchSize: 100, batchTime: 1000 });
+
+stream.write({ topic: 'events', key: 'e1', value: '{"type":"click"}' });
+stream.write({ topic: 'events', key: 'e2', value: '{"type":"scroll"}' });
+
+stream.on('data', (report) => {
+  console.log(`Batch ${report.batchId}: ${report.count} messages sent`);
+});
+
+await stream.close();
+```
+
+### `producer.close(force?)`
+
+Close the producer connection.
+
+- `force=false` (default): Wait for in-flight requests to complete
+- `force=true`: Abort immediately
+
+### Producer Properties
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `producerId` | `bigint \| undefined` | Assigned producer ID (after idempotent init) |
+| `producerEpoch` | `number \| undefined` | Producer epoch (fencing) |
+| `transaction` | `Transaction \| undefined` | Active transaction (if any) |
+| `coordinatorId` | `number` | Transaction coordinator broker ID |
+| `streamsCount` | `number` | Number of active producer streams |
+
+## Key Partitioning
+
+By default, `@platformatic/kafka` uses **murmur2 hashing** on the message key to determine the target partition:
+
+- Same key → always same partition → guaranteed ordering per key
+- `undefined` key → round-robin across partitions
+- Explicit `partition` field → overrides the partitioner
+
+```typescript
+// Custom partitioner
+await producer.send({
+  messages: [{ topic: 'events', key: 'e1', value: 'data' }],
+  partitioner: (message) => {
+    return message.key!.charCodeAt(0) % 3;
+  },
+});
+```

package/wiki/references/helpers/kafka/schema-registry.md

@@ -0,0 +1,214 @@
+# Schema Registry
+
+The `KafkaSchemaRegistryHelper` wraps `@platformatic/kafka`'s `ConfluentSchemaRegistry`. It provides a centralized schema registry that auto-serializes/deserializes messages using registered schemas (Avro, Protobuf, JSON Schema).
+
+```typescript
+class KafkaSchemaRegistryHelper<
+  KeyType = string,
+  ValueType = string,
+  HeaderKeyType = string,
+  HeaderValueType = string,
+> extends BaseHelper
+```
+
+> [!NOTE]
+> `KafkaSchemaRegistryHelper` extends `BaseHelper` directly (not `BaseKafkaHelper`) — it has no broker connection or health tracking. It's a configuration wrapper, not a client.
+
+## Helper API
+
+| Method | Signature | Description |
+|--------|-----------|-------------|
+| `newInstance(opts)` | `static newInstance<K,V,HK,HV>(opts): KafkaSchemaRegistryHelper<K,V,HK,HV>` | Factory method |
+| `getRegistry()` | `(): ConfluentSchemaRegistry<K,V,HK,HV>` | Get the registry instance (pass to producer/consumer) |
+| `getSerializers()` | `(): Serializers<K,V,HK,HV>` | Get schema-based serializers |
+| `getDeserializers()` | `(): Deserializers<K,V,HK,HV>` | Get schema-based deserializers |
+
+## IKafkaSchemaRegistryOptions
+
+```typescript
+interface IKafkaSchemaRegistryOptions extends ConfluentSchemaRegistryOptions {
+  identifier?: string; // Default: 'kafka-schema-registry'
+}
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `url` | `string` | — | Schema registry URL. **Required** |
+| `auth` | `{ username: string; password: string }` | — | Basic auth credentials |
+| `protobufTypeMapper` | `ProtobufTypeMapper` | — | Custom Protobuf type mapper |
+| `jsonValidateSend` | `boolean` | — | Validate JSON schema on produce |
+| `identifier` | `string` | `'kafka-schema-registry'` | Scoped logging identifier |
+
+## What Schema Registry Solves
+
+Without a schema registry, producers and consumers must agree on message format out-of-band. If the producer changes the shape of `value` (adds/removes fields), consumers break silently at runtime.
+
+**Schema Registry** is a centralized server (Confluent Schema Registry) that stores and validates schemas (Avro, Protobuf, JSON Schema). It enforces a contract:
+
+```
+Producer → "I want to send this shape" → Schema Registry validates → Kafka
+Kafka → Consumer → "What shape is this?" → Schema Registry tells → Deserialize
+```
+
+### Without Schema Registry (raw strings)
+
+```typescript
+// Producer — manually serialize
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'order-producer',
+});
+
+await producer.getProducer().send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-1',
+    value: JSON.stringify({ id: 1, total: 99.99 }),  // ← just a string, no validation
+  }],
+});
+
+// Consumer — manually deserialize, hope the shape is correct
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'order-consumer',
+  groupId: 'order-group',
+  onMessage: async ({ message }) => {
+    const order = JSON.parse(message.value as string);  // ← pray it matches
+    console.log(order.id, order.total);
+  },
+});
+```
+
+Problem: if producer adds `{ id: 1, total: 99.99, currency: 'USD' }` or removes `total`, consumer has no way to know until it crashes.
+
+### With Schema Registry (auto serialize/deserialize)
+
+```typescript
+// 1. Create registry — points to Confluent Schema Registry server
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'http://localhost:8081',
+  // auth: { username: 'user', password: 'pass' },  // optional
+});
+
+// 2. Producer — pass registry, it auto-serializes values using registered schema
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'order-producer',
+  registry: registry.getRegistry(),  // ← registry handles serialization
+});
+
+await producer.getProducer().send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-1',
+    value: { id: 1, total: 99.99 },  // ← object, not string! Registry serializes it
+  }],
+});
+// If the value doesn't match the registered schema → error BEFORE sending to Kafka
+
+// 3. Consumer — pass same registry, it auto-deserializes
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['127.0.0.1:29092'],
+  clientId: 'order-consumer',
+  groupId: 'order-group',
+  registry: registry.getRegistry(),  // ← registry handles deserialization
+  onMessage: async ({ message }) => {
+    // message.value is already a typed object, not a raw string
+    console.log(message.value.id, message.value.total);
+  },
+});
+```
+
+### Comparison
+
+| | Without Registry | With Registry |
+|---|---|---|
+| **Message format** | Raw string, manual `JSON.stringify/parse` | Typed object, auto ser/deser |
+| **Validation** | None — runtime crashes | Schema validated before send |
+| **Schema evolution** | Break consumers silently | Backward/forward compatibility enforced |
+| **Where schemas live** | Nowhere (tribal knowledge) | Centralized server `http://registry:8081` |
+
+You only need it when you want **schema enforcement** across producers/consumers. For simple string messages, skip it entirely.
+
+## Basic Usage
+
+```typescript
+import { KafkaSchemaRegistryHelper, KafkaProducerHelper, KafkaConsumerHelper } from '@venizia/ignis-helpers/kafka';
+
+// 1. Create registry — points to Confluent Schema Registry server
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'http://localhost:8081',
+});
+
+// 2. Producer — registry auto-serializes values using registered schema
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'order-producer',
+  registry: registry.getRegistry(),
+});
+
+await producer.getProducer().send({
+  messages: [{
+    topic: 'orders',
+    key: 'order-1',
+    value: { id: 1, total: 99.99 },  // object, not string — registry serializes
+  }],
+});
+// If value doesn't match the registered schema → error BEFORE sending to Kafka
+
+// 3. Consumer — registry auto-deserializes
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'order-consumer',
+  groupId: 'order-group',
+  registry: registry.getRegistry(),
+  onMessage: async ({ message }) => {
+    // message.value is already a typed object, not a raw string
+    console.log(message.value.id, message.value.total);
+  },
+});
+
+await consumer.start({ topics: ['orders'] });
+```
+
+## With Authentication
+
+```typescript
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'https://schema-registry.example.com',
+  auth: {
+    username: 'registry-user',
+    password: 'registry-password',
+  },
+});
+```
+
+## Alternative: Manual Serializers
+
+Instead of passing the full registry, you can extract serializers/deserializers for manual use:
+
+```typescript
+const registry = KafkaSchemaRegistryHelper.newInstance({
+  url: 'http://localhost:8081',
+});
+
+// Use serializers directly (instead of registry)
+const producer = KafkaProducerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-producer',
+  serializers: registry.getSerializers(),
+});
+
+const consumer = KafkaConsumerHelper.newInstance({
+  bootstrapBrokers: ['localhost:9092'],
+  clientId: 'my-consumer',
+  groupId: 'my-group',
+  deserializers: registry.getDeserializers(),
+  onMessage: async ({ message }) => { ... },
+});
+```
+
+## When to Use
+
+- **Use schema registry** when you need schema enforcement, validation, and compatibility checks across producers/consumers — especially in multi-team environments
+- **Skip schema registry** for simple string/JSON messages where both sides are controlled by the same team and format changes are coordinated