@drarzter/kafka-client 0.3.1 → 0.5.1

package/README.md

@@ -4,11 +4,11 @@
 [![CI](https://github.com/drarzter/kafka-client/actions/workflows/publish.yml/badge.svg)](https://github.com/drarzter/kafka-client/actions/workflows/publish.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 
-Type-safe Kafka client for Node.js. Framework-agnostic core with a first-class NestJS adapter. Built on top of [kafkajs](https://kafka.js.org/).
+Type-safe Kafka client for Node.js. Framework-agnostic core with a first-class NestJS adapter. Built on top of [`@confluentinc/kafka-javascript`](https://github.com/confluentinc/confluent-kafka-javascript) (librdkafka).
 
 ## What is this?
 
-An opinionated, type-safe abstraction over kafkajs. Works standalone (Express, Fastify, raw Node) or as a NestJS DynamicModule. Not a full-featured framework — just a clean, typed layer for producing and consuming Kafka messages.
+An opinionated, type-safe abstraction over `@confluentinc/kafka-javascript` (librdkafka). Works standalone (Express, Fastify, raw Node) or as a NestJS DynamicModule. Not a full-featured framework — just a clean, typed layer for producing and consuming Kafka messages.
 
 ## Why?
 
@@ -22,7 +22,11 @@ An opinionated, type-safe abstraction over kafkajs. Works standalone (Express, F
 - **Partition key support** — route related messages to the same partition
 - **Custom headers** — attach metadata headers to messages
 - **Transactions** — exactly-once semantics with `producer.transaction()`
-- **Consumer interceptors** — before/after/onError hooks for message processing
+- **EventEnvelope** — every consumed message is wrapped in `EventEnvelope<T>` with `eventId`, `correlationId`, `timestamp`, `schemaVersion`, `traceparent`, and Kafka metadata
+- **Correlation ID propagation** — auto-generated on send, auto-propagated through `AsyncLocalStorage` so nested sends inherit the same correlation ID
+- **OpenTelemetry support** — `@drarzter/kafka-client/otel` entrypoint with `otelInstrumentation()` for W3C Trace Context propagation
+- **Consumer interceptors** — before/after/onError hooks with `EventEnvelope` access
+- **Client-wide instrumentation** — `KafkaInstrumentation` hooks for cross-cutting concerns (tracing, metrics)
 - **Auto-create topics** — `autoCreateTopics: true` for dev mode — no need to pre-create topics
 - **Error classes** — `KafkaProcessingError` and `KafkaRetryExhaustedError` with topic, message, and attempt metadata
 - **Health check** — built-in health indicator for monitoring
@@ -37,6 +41,21 @@ See the [Roadmap](./ROADMAP.md) for upcoming features and version history.
 npm install @drarzter/kafka-client
 ```
 
+`@confluentinc/kafka-javascript` uses a native librdkafka addon. On most systems it builds automatically. For faster installs (skips compilation), install the system library first:
+
+```bash
+# Arch / CachyOS
+sudo pacman -S librdkafka
+
+# Debian / Ubuntu
+sudo apt-get install librdkafka-dev
+
+# macOS
+brew install librdkafka
+```
+
+Then install with `BUILD_LIBRDKAFKA=0 npm install`.
+
 For NestJS projects, install peer dependencies: `@nestjs/common`, `@nestjs/core`, `reflect-metadata`, `rxjs`.
 
 For standalone usage (Express, Fastify, raw Node), no extra dependencies needed — import from `@drarzter/kafka-client/core`.
@@ -54,15 +73,31 @@ await kafka.connectProducer();
 // Send
 await kafka.sendMessage(OrderCreated, { orderId: '123', amount: 100 });
 
-// Consume
-await kafka.startConsumer([OrderCreated], async (message, topic) => {
-  console.log(`${topic}:`, message.orderId);
+// Consume — handler receives an EventEnvelope
+await kafka.startConsumer([OrderCreated], async (envelope) => {
+  console.log(`${envelope.topic}:`, envelope.payload.orderId);
 });
 
 // Custom logger (winston, pino, etc.)
 const kafka2 = new KafkaClient('my-app', 'my-group', ['localhost:9092'], {
   logger: myWinstonLogger,
 });
+
+// All module options work in standalone mode too
+const kafka3 = new KafkaClient('my-app', 'my-group', ['localhost:9092'], {
+  autoCreateTopics: true,   // auto-create topics on first use
+  numPartitions: 3,         // partitions for auto-created topics
+  strictSchemas: false,     // disable schema enforcement for string topic keys
+  instrumentation: [...],   // client-wide tracing/metrics hooks
+});
+
+// Health check — available directly, no NestJS needed
+const status = await kafka.checkStatus();
+// { status: 'up', clientId: 'my-app', topics: ['order.created', ...] }
+
+// Stop all consumers without disconnecting the producer or admin
+// Useful when you want to re-subscribe with different options
+await kafka.stopConsumer();
 ```
 
 ## Quick start (NestJS)
@@ -99,7 +134,7 @@ export class AppModule {}
 ```typescript
 // app.service.ts
 import { Injectable } from '@nestjs/common';
-import { InjectKafkaClient, KafkaClient, SubscribeTo } from '@drarzter/kafka-client';
+import { InjectKafkaClient, KafkaClient, SubscribeTo, EventEnvelope } from '@drarzter/kafka-client';
 import { MyTopics } from './types';
 
 @Injectable()
@@ -113,8 +148,8 @@ export class AppService {
   }
 
   @SubscribeTo('hello')
-  async onHello(message: MyTopics['hello']) {
-    console.log('Received:', message.text);
+  async onHello(envelope: EventEnvelope<MyTopics['hello']>) {
+    console.log('Received:', envelope.payload.text);
   }
 }
 ```
@@ -192,7 +227,7 @@ await kafka.transaction(async (tx) => {
 
 // Consuming (decorator)
 @SubscribeTo(OrderCreated)
-async handleOrder(message: OrdersTopicMap['order.created']) { ... }
+async handleOrder(envelope: EventEnvelope<OrdersTopicMap['order.created']>) { ... }
 
 // Consuming (imperative)
 await kafka.startConsumer([OrderCreated], handler);
@@ -217,7 +252,7 @@ import { OrdersTopicMap } from './orders.types';
 export class OrdersModule {}
 ```
 
-`autoCreateTopics` calls `admin.createTopics()` (idempotent — no-op if topic already exists) before the first send/consume for each topic. Useful in development, not recommended for production.
+`autoCreateTopics` calls `admin.createTopics()` (idempotent — no-op if topic already exists) before the first send **and** before each `startConsumer` / `startBatchConsumer` call. librdkafka errors on unknown topics at subscribe time, so consumer-side creation is required. Useful in development, not recommended for production.
 
 Or with `ConfigService`:
 
@@ -301,13 +336,13 @@ import { SubscribeTo } from '@drarzter/kafka-client';
 @Injectable()
 export class OrdersHandler {
   @SubscribeTo('order.created')
-  async handleOrderCreated(message: OrdersTopicMap['order.created'], topic: string) {
-    console.log('New order:', message.orderId);
+  async handleOrderCreated(envelope: EventEnvelope<OrdersTopicMap['order.created']>) {
+    console.log('New order:', envelope.payload.orderId);
   }
 
   @SubscribeTo('order.completed', { retry: { maxRetries: 3 }, dlq: true })
-  async handleOrderCompleted(message: OrdersTopicMap['order.completed'], topic: string) {
-    console.log('Order completed:', message.orderId);
+  async handleOrderCompleted(envelope: EventEnvelope<OrdersTopicMap['order.completed']>) {
+    console.log('Order completed:', envelope.payload.orderId);
   }
 }
 ```
@@ -327,8 +362,8 @@ export class OrdersService implements OnModuleInit {
   async onModuleInit() {
     await this.kafka.startConsumer(
       ['order.created', 'order.completed'],
-      async (message, topic) => {
-        console.log(`${topic}:`, message);
+      async (envelope) => {
+        console.log(`${envelope.topic}:`, envelope.payload);
       },
       {
         retry: { maxRetries: 3, backoffMs: 1000 },
@@ -343,7 +378,7 @@ export class OrdersService implements OnModuleInit {
 
 ### Per-consumer groupId
 
-Override the default consumer group for specific consumers. Each unique `groupId` creates a separate kafkajs Consumer internally:
+Override the default consumer group for specific consumers. Each unique `groupId` creates a separate librdkafka Consumer internally:
 
 ```typescript
 // Default group from constructor
@@ -354,7 +389,7 @@ await kafka.startConsumer(['orders'], auditHandler, { groupId: 'orders-audit' })
 
 // Works with @SubscribeTo too
 @SubscribeTo('orders', { groupId: 'orders-audit' })
-async auditOrders(message) { ... }
+async auditOrders(envelope) { ... }
 ```
 
 **Important:** You cannot mix `eachMessage` and `eachBatch` consumers on the same `groupId`. The library throws a clear error if you try:
@@ -404,7 +439,7 @@ Same with `@SubscribeTo()` — use `clientName` to target a specific named clien
 
 ```typescript
 @SubscribeTo('payment.received', { clientName: 'payments' })  // ← matches name: 'payments'
-async handlePayment(message: PaymentsTopicMap['payment.received']) {
+async handlePayment(envelope: EventEnvelope<PaymentsTopicMap['payment.received']>) {
   // ...
 }
 ```
@@ -460,16 +495,20 @@ await this.kafka.sendBatch('order.created', [
 
 ## Batch consuming
 
-Process messages in batches for higher throughput. The handler receives an array of parsed messages and a `BatchMeta` object with offset management controls:
+Process messages in batches for higher throughput. The handler receives an array of `EventEnvelope`s and a `BatchMeta` object with offset management controls:
 
 ```typescript
 await this.kafka.startBatchConsumer(
   ['order.created'],
-  async (messages, topic, meta) => {
-    // messages: OrdersTopicMap['order.created'][]
-    for (const msg of messages) {
-      await processOrder(msg);
-      meta.resolveOffset(/* ... */);
+  async (envelopes, meta) => {
+    // envelopes: EventEnvelope<OrdersTopicMap['order.created']>[]
+    for (const env of envelopes) {
+      await processOrder(env.payload);
+      meta.resolveOffset(env.offset);
+
+      // Call heartbeat() during long-running batch processing to prevent
+      // the broker from considering the consumer dead (session.timeout.ms)
+      await meta.heartbeat();
     }
     await meta.commitOffsetsIfNecessary();
   },
@@ -477,18 +516,44 @@ await this.kafka.startBatchConsumer(
 );
 ```
 
+With `autoCommit: false` for full manual offset control:
+
+```typescript
+await this.kafka.startBatchConsumer(
+  ['order.created'],
+  async (envelopes, meta) => {
+    for (const env of envelopes) {
+      await processOrder(env.payload);
+      meta.resolveOffset(env.offset);
+    }
+    // commitOffsetsIfNecessary() commits only when autoCommit is off
+    // or when the commit interval has elapsed
+    await meta.commitOffsetsIfNecessary();
+  },
+  { autoCommit: false },
+);
+```
+
 With `@SubscribeTo()`:
 
 ```typescript
 @SubscribeTo('order.created', { batch: true })
-async handleOrders(messages: OrdersTopicMap['order.created'][], topic: string) {
-  // messages is an array
+async handleOrders(envelopes: EventEnvelope<OrdersTopicMap['order.created']>[], meta: BatchMeta) {
+  for (const env of envelopes) { ... }
 }
 ```
 
 Schema validation runs per-message — invalid messages are skipped (DLQ'd if enabled), valid ones are passed to the handler. Retry applies to the whole batch.
 
-`BatchMeta` exposes: `partition`, `highWatermark`, `heartbeat()`, `resolveOffset(offset)`, `commitOffsetsIfNecessary()`.
+`BatchMeta` exposes:
+
+| Property/Method | Description |
+| --------------- | ----------- |
+| `partition` | Partition number for this batch |
+| `highWatermark` | Latest offset in the partition (lag indicator) |
+| `heartbeat()` | Send a heartbeat to keep the consumer session alive — call during long processing loops |
+| `resolveOffset(offset)` | Mark offset as processed (required before `commitOffsetsIfNecessary`) |
+| `commitOffsetsIfNecessary()` | Commit resolved offsets; respects `autoCommit` setting |
 
 ## Transactions
 
@@ -509,24 +574,34 @@ await this.kafka.transaction(async (tx) => {
 });
 ```
 
-`tx.sendBatch()` is also available inside transactions.
+`tx.sendBatch()` is also available inside transactions:
+
+```typescript
+await this.kafka.transaction(async (tx) => {
+  await tx.sendBatch('order.created', [
+    { value: { orderId: '1', userId: '10', amount: 50 }, key: '1' },
+    { value: { orderId: '2', userId: '20', amount: 75 }, key: '2' },
+  ]);
+  // if anything throws, all messages are rolled back
+});
+```
 
 ## Consumer interceptors
 
-Add before/after/onError hooks to message processing:
+Add before/after/onError hooks to message processing. Interceptors receive the full `EventEnvelope`:
 
 ```typescript
 import { ConsumerInterceptor } from '@drarzter/kafka-client';
 
 const loggingInterceptor: ConsumerInterceptor<OrdersTopicMap> = {
-  before: (message, topic) => {
-    console.log(`Processing ${topic}`, message);
+  before: (envelope) => {
+    console.log(`Processing ${envelope.topic}`, envelope.payload);
   },
-  after: (message, topic) => {
-    console.log(`Done ${topic}`);
+  after: (envelope) => {
+    console.log(`Done ${envelope.topic}`);
   },
-  onError: (message, topic, error) => {
-    console.error(`Failed ${topic}:`, error.message);
+  onError: (envelope, error) => {
+    console.error(`Failed ${envelope.topic}:`, error.message);
   },
 };
 
@@ -537,23 +612,53 @@ await this.kafka.startConsumer(['order.created'], handler, {
 
 Multiple interceptors run in order. All hooks are optional.
 
+## Instrumentation
+
+For client-wide cross-cutting concerns (tracing, metrics), use `KafkaInstrumentation` hooks instead of per-consumer interceptors:
+
+```typescript
+import { otelInstrumentation } from '@drarzter/kafka-client/otel';
+
+const kafka = new KafkaClient('my-app', 'my-group', brokers, {
+  instrumentation: [otelInstrumentation()],
+});
+```
+
+`otelInstrumentation()` injects `traceparent` on send, extracts it on consume, and creates `CONSUMER` spans automatically. Requires `@opentelemetry/api` as a peer dependency.
+
+Custom instrumentation:
+
+```typescript
+import { KafkaInstrumentation } from '@drarzter/kafka-client';
+
+const metrics: KafkaInstrumentation = {
+  beforeSend(topic, headers) { /* inject headers, start timer */ },
+  afterSend(topic) { /* record send latency */ },
+  beforeConsume(envelope) { /* start span */ return () => { /* end span */ }; },
+  onConsumeError(envelope, error) { /* record error metric */ },
+};
+```
+
 ## Options reference
 
 ### Send options
 
 Options for `sendMessage()` — the third argument:
 
-| Option    | Default | Description                                      |
-|-----------|---------|--------------------------------------------------|
-| `key`     | —       | Partition key for message routing                 |
-| `headers` | —       | Custom metadata headers (`Record<string, string>`) |
+| Option | Default | Description |
+| ------ | ------- | ----------- |
+| `key` | — | Partition key for message routing |
+| `headers` | — | Custom metadata headers (merged with auto-generated envelope headers) |
+| `correlationId` | auto | Override the auto-propagated correlation ID (default: inherited from ALS context or new UUID) |
+| `schemaVersion` | `1` | Schema version for the payload |
+| `eventId` | auto | Override the auto-generated event ID (UUID v4) |
 
-`sendBatch()` accepts `key` and `headers` per message inside the array items.
+`sendBatch()` accepts the same options per message inside the array items.
 
 ### Consumer options
 
 | Option | Default | Description |
-|--------|---------|-------------|
+| ------ | ------- | ----------- |
 | `groupId` | constructor value | Override consumer group for this subscription |
 | `fromBeginning` | `false` | Read from the beginning of the topic |
 | `autoCommit` | `true` | Auto-commit offsets |
@@ -570,7 +675,7 @@ Options for `sendMessage()` — the third argument:
 Passed to `KafkaModule.register()` or returned from `registerAsync()` factory:
 
 | Option | Default | Description |
-|--------|---------|-------------|
+| ------ | ------- | ----------- |
 | `clientId` | — | Kafka client identifier (required) |
 | `groupId` | — | Default consumer group ID (required) |
 | `brokers` | — | Array of broker addresses (required) |
@@ -579,6 +684,7 @@ Passed to `KafkaModule.register()` or returned from `registerAsync()` factory:
 | `autoCreateTopics` | `false` | Auto-create topics on first send (dev only) |
 | `numPartitions` | `1` | Number of partitions for auto-created topics |
 | `strictSchemas` | `true` | Validate string topic keys against schemas registered via TopicDescriptor |
+| `instrumentation` | `[]` | Client-wide instrumentation hooks (e.g. OTel). Applied to both send and consume paths |
 
 **Module-scoped** (default) — import `KafkaModule` in each module that needs it:
 
@@ -641,7 +747,7 @@ err.cause;            // the original error
 ```typescript
 // In an onError interceptor:
 const interceptor: ConsumerInterceptor<MyTopics> = {
-  onError: (message, topic, error) => {
+  onError: (envelope, error) => {
     if (error instanceof KafkaRetryExhaustedError) {
       console.log(`Failed after ${error.attempts} attempts on ${error.topic}`);
       console.log('Last error:', error.cause);
@@ -658,7 +764,7 @@ When `retry.maxRetries` is set and all attempts fail, `KafkaRetryExhaustedError`
 import { KafkaValidationError } from '@drarzter/kafka-client';
 
 const interceptor: ConsumerInterceptor<MyTopics> = {
-  onError: (message, topic, error) => {
+  onError: (envelope, error) => {
     if (error instanceof KafkaValidationError) {
       console.log(`Bad message on ${error.topic}:`, error.cause?.message);
     }
@@ -707,9 +813,9 @@ await kafka.sendMessage(OrderCreated, { orderId: '1', userId: '2', amount: -5 })
 
 ```typescript
 @SubscribeTo(OrderCreated, { dlq: true })
-async handleOrder(message) {
-  // `message` is guaranteed to match the schema
-  console.log(message.orderId); // string — validated at runtime
+async handleOrder(envelope) {
+  // `envelope.payload` is guaranteed to match the schema
+  console.log(envelope.payload.orderId); // string — validated at runtime
 }
 ```
 
@@ -777,6 +883,8 @@ export class HealthService {
 
 Import from `@drarzter/kafka-client/testing` — zero runtime deps, only `jest` and `@testcontainers/kafka` as peer dependencies.
 
+> Unit tests mock `@confluentinc/kafka-javascript` — no Kafka broker needed. Integration tests use Testcontainers (requires Docker).
+
 #### `createMockKafkaClient<T>()`
 
 Fully typed mock with `jest.fn()` on every `IKafkaClient` method. All methods resolve to sensible defaults:
@@ -830,14 +938,14 @@ it('sends and receives', async () => {
 Options:
 
 | Option | Default | Description |
-|--------|---------|-------------|
+| ------ | ------- | ----------- |
 | `image` | `"confluentinc/cp-kafka:7.7.0"` | Docker image |
 | `transactionWarmup` | `true` | Warm up transaction coordinator on start |
 | `topics` | `[]` | Topics to pre-create (string or `{ topic, numPartitions }`) |
 
 ### Running tests
 
-Unit tests (mocked kafkajs):
+Unit tests (mocked `@confluentinc/kafka-javascript` — no broker needed):
 
 ```bash
 npm test
@@ -851,16 +959,17 @@ npm run test:integration
 
 The integration suite spins up a single-node KRaft Kafka container and tests sending, consuming, batching, transactions, retry + DLQ, interceptors, health checks, and `fromBeginning` — no mocks.
 
-Both suites run in CI on every push to `main`.
+Both suites run in CI on every push to `main` and on pull requests.
 
 ## Project structure
 
-```
+```text
 src/
-├── client/         # Core — KafkaClient, types, topic(), error classes (0 framework deps)
+├── client/         # Core — KafkaClient, types, envelope, consumer pipeline, topic(), errors (0 framework deps)
 ├── nest/           # NestJS adapter — Module, Explorer, decorators, health
 ├── testing/        # Testing utilities — mock client, testcontainer wrapper
 ├── core.ts         # Standalone entrypoint (@drarzter/kafka-client/core)
+├── otel.ts         # OpenTelemetry entrypoint (@drarzter/kafka-client/otel)
 ├── testing.ts      # Testing entrypoint (@drarzter/kafka-client/testing)
 └── index.ts        # Full entrypoint — core + NestJS adapter
 ```