@drarzter/kafka-client 0.3.0 → 0.5.0

package/README.md

@@ -4,11 +4,11 @@
 [![CI](https://github.com/drarzter/kafka-client/actions/workflows/publish.yml/badge.svg)](https://github.com/drarzter/kafka-client/actions/workflows/publish.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 
-Type-safe Kafka client for Node.js. Framework-agnostic core with a first-class NestJS adapter. Built on top of [kafkajs](https://kafka.js.org/).
+Type-safe Kafka client for Node.js. Framework-agnostic core with a first-class NestJS adapter. Built on top of [`@confluentinc/kafka-javascript`](https://github.com/confluentinc/confluent-kafka-javascript) (librdkafka).
 
 ## What is this?
 
-An opinionated, type-safe abstraction over kafkajs. Works standalone (Express, Fastify, raw Node) or as a NestJS DynamicModule. Not a full-featured framework — just a clean, typed layer for producing and consuming Kafka messages.
+An opinionated, type-safe abstraction over `@confluentinc/kafka-javascript` (librdkafka). Works standalone (Express, Fastify, raw Node) or as a NestJS DynamicModule. Not a full-featured framework — just a clean, typed layer for producing and consuming Kafka messages.
 
 ## Why?
 
@@ -22,7 +22,11 @@ An opinionated, type-safe abstraction over kafkajs. Works standalone (Express, F
 - **Partition key support** — route related messages to the same partition
 - **Custom headers** — attach metadata headers to messages
 - **Transactions** — exactly-once semantics with `producer.transaction()`
-- **Consumer interceptors** — before/after/onError hooks for message processing
+- **EventEnvelope** — every consumed message is wrapped in `EventEnvelope<T>` with `eventId`, `correlationId`, `timestamp`, `schemaVersion`, `traceparent`, and Kafka metadata
+- **Correlation ID propagation** — auto-generated on send, auto-propagated through `AsyncLocalStorage` so nested sends inherit the same correlation ID
+- **OpenTelemetry support** — `@drarzter/kafka-client/otel` entrypoint with `otelInstrumentation()` for W3C Trace Context propagation
+- **Consumer interceptors** — before/after/onError hooks with `EventEnvelope` access
+- **Client-wide instrumentation** — `KafkaInstrumentation` hooks for cross-cutting concerns (tracing, metrics)
 - **Auto-create topics** — `autoCreateTopics: true` for dev mode — no need to pre-create topics
 - **Error classes** — `KafkaProcessingError` and `KafkaRetryExhaustedError` with topic, message, and attempt metadata
 - **Health check** — built-in health indicator for monitoring
@@ -37,6 +41,21 @@ See the [Roadmap](./ROADMAP.md) for upcoming features and version history.
 npm install @drarzter/kafka-client
 ```
 
+`@confluentinc/kafka-javascript` uses a native librdkafka addon. On most systems it builds automatically. For faster installs (skips compilation), install the system library first:
+
+```bash
+# Arch / CachyOS
+sudo pacman -S librdkafka
+
+# Debian / Ubuntu
+sudo apt-get install librdkafka-dev
+
+# macOS
+brew install librdkafka
+```
+
+Then install with `BUILD_LIBRDKAFKA=0 npm install`.
+
 For NestJS projects, install peer dependencies: `@nestjs/common`, `@nestjs/core`, `reflect-metadata`, `rxjs`.
 
 For standalone usage (Express, Fastify, raw Node), no extra dependencies needed — import from `@drarzter/kafka-client/core`.
@@ -54,9 +73,9 @@ await kafka.connectProducer();
 // Send
 await kafka.sendMessage(OrderCreated, { orderId: '123', amount: 100 });
 
-// Consume
-await kafka.startConsumer([OrderCreated], async (message, topic) => {
-  console.log(`${topic}:`, message.orderId);
+// Consume — handler receives an EventEnvelope
+await kafka.startConsumer([OrderCreated], async (envelope) => {
+  console.log(`${envelope.topic}:`, envelope.payload.orderId);
 });
 
 // Custom logger (winston, pino, etc.)
@@ -99,7 +118,7 @@ export class AppModule {}
 ```typescript
 // app.service.ts
 import { Injectable } from '@nestjs/common';
-import { InjectKafkaClient, KafkaClient, SubscribeTo } from '@drarzter/kafka-client';
+import { InjectKafkaClient, KafkaClient, SubscribeTo, EventEnvelope } from '@drarzter/kafka-client';
 import { MyTopics } from './types';
 
 @Injectable()
@@ -113,8 +132,8 @@ export class AppService {
   }
 
   @SubscribeTo('hello')
-  async onHello(message: MyTopics['hello']) {
-    console.log('Received:', message.text);
+  async onHello(envelope: EventEnvelope<MyTopics['hello']>) {
+    console.log('Received:', envelope.payload.text);
   }
 }
 ```
@@ -192,7 +211,7 @@ await kafka.transaction(async (tx) => {
 
 // Consuming (decorator)
 @SubscribeTo(OrderCreated)
-async handleOrder(message: OrdersTopicMap['order.created']) { ... }
+async handleOrder(envelope: EventEnvelope<OrdersTopicMap['order.created']>) { ... }
 
 // Consuming (imperative)
 await kafka.startConsumer([OrderCreated], handler);
@@ -217,7 +236,7 @@ import { OrdersTopicMap } from './orders.types';
 export class OrdersModule {}
 ```
 
-`autoCreateTopics` calls `admin.createTopics()` (idempotent — no-op if topic already exists) before the first send/consume for each topic. Useful in development, not recommended for production.
+`autoCreateTopics` calls `admin.createTopics()` (idempotent — no-op if topic already exists) before the first send **and** before each `startConsumer` / `startBatchConsumer` call. librdkafka errors on unknown topics at subscribe time, so consumer-side creation is required. Useful in development, not recommended for production.
 
 Or with `ConfigService`:
 
@@ -301,13 +320,13 @@ import { SubscribeTo } from '@drarzter/kafka-client';
 @Injectable()
 export class OrdersHandler {
   @SubscribeTo('order.created')
-  async handleOrderCreated(message: OrdersTopicMap['order.created'], topic: string) {
-    console.log('New order:', message.orderId);
+  async handleOrderCreated(envelope: EventEnvelope<OrdersTopicMap['order.created']>) {
+    console.log('New order:', envelope.payload.orderId);
   }
 
   @SubscribeTo('order.completed', { retry: { maxRetries: 3 }, dlq: true })
-  async handleOrderCompleted(message: OrdersTopicMap['order.completed'], topic: string) {
-    console.log('Order completed:', message.orderId);
+  async handleOrderCompleted(envelope: EventEnvelope<OrdersTopicMap['order.completed']>) {
+    console.log('Order completed:', envelope.payload.orderId);
   }
 }
 ```
@@ -327,8 +346,8 @@ export class OrdersService implements OnModuleInit {
   async onModuleInit() {
     await this.kafka.startConsumer(
       ['order.created', 'order.completed'],
-      async (message, topic) => {
-        console.log(`${topic}:`, message);
+      async (envelope) => {
+        console.log(`${envelope.topic}:`, envelope.payload);
       },
       {
         retry: { maxRetries: 3, backoffMs: 1000 },
@@ -343,7 +362,7 @@ export class OrdersService implements OnModuleInit {
 
 ### Per-consumer groupId
 
-Override the default consumer group for specific consumers. Each unique `groupId` creates a separate kafkajs Consumer internally:
+Override the default consumer group for specific consumers. Each unique `groupId` creates a separate librdkafka Consumer internally:
 
 ```typescript
 // Default group from constructor
@@ -354,7 +373,7 @@ await kafka.startConsumer(['orders'], auditHandler, { groupId: 'orders-audit' })
 
 // Works with @SubscribeTo too
 @SubscribeTo('orders', { groupId: 'orders-audit' })
-async auditOrders(message) { ... }
+async auditOrders(envelope) { ... }
 ```
 
 **Important:** You cannot mix `eachMessage` and `eachBatch` consumers on the same `groupId`. The library throws a clear error if you try:
@@ -404,7 +423,7 @@ Same with `@SubscribeTo()` — use `clientName` to target a specific named clien
 
 ```typescript
 @SubscribeTo('payment.received', { clientName: 'payments' })  // ← matches name: 'payments'
-async handlePayment(message: PaymentsTopicMap['payment.received']) {
+async handlePayment(envelope: EventEnvelope<PaymentsTopicMap['payment.received']>) {
   // ...
 }
 ```
@@ -460,16 +479,16 @@ await this.kafka.sendBatch('order.created', [
 
 ## Batch consuming
 
-Process messages in batches for higher throughput. The handler receives an array of parsed messages and a `BatchMeta` object with offset management controls:
+Process messages in batches for higher throughput. The handler receives an array of `EventEnvelope`s and a `BatchMeta` object with offset management controls:
 
 ```typescript
 await this.kafka.startBatchConsumer(
   ['order.created'],
-  async (messages, topic, meta) => {
-    // messages: OrdersTopicMap['order.created'][]
-    for (const msg of messages) {
-      await processOrder(msg);
-      meta.resolveOffset(/* ... */);
+  async (envelopes, meta) => {
+    // envelopes: EventEnvelope<OrdersTopicMap['order.created']>[]
+    for (const env of envelopes) {
+      await processOrder(env.payload);
+      meta.resolveOffset(env.offset);
     }
     await meta.commitOffsetsIfNecessary();
   },
@@ -481,8 +500,8 @@ With `@SubscribeTo()`:
 
 ```typescript
 @SubscribeTo('order.created', { batch: true })
-async handleOrders(messages: OrdersTopicMap['order.created'][], topic: string) {
-  // messages is an array
+async handleOrders(envelopes: EventEnvelope<OrdersTopicMap['order.created']>[], meta: BatchMeta) {
+  for (const env of envelopes) { ... }
 }
 ```
 
@@ -513,20 +532,20 @@ await this.kafka.transaction(async (tx) => {
 
 ## Consumer interceptors
 
-Add before/after/onError hooks to message processing:
+Add before/after/onError hooks to message processing. Interceptors receive the full `EventEnvelope`:
 
 ```typescript
 import { ConsumerInterceptor } from '@drarzter/kafka-client';
 
 const loggingInterceptor: ConsumerInterceptor<OrdersTopicMap> = {
-  before: (message, topic) => {
-    console.log(`Processing ${topic}`, message);
+  before: (envelope) => {
+    console.log(`Processing ${envelope.topic}`, envelope.payload);
   },
-  after: (message, topic) => {
-    console.log(`Done ${topic}`);
+  after: (envelope) => {
+    console.log(`Done ${envelope.topic}`);
   },
-  onError: (message, topic, error) => {
-    console.error(`Failed ${topic}:`, error.message);
+  onError: (envelope, error) => {
+    console.error(`Failed ${envelope.topic}:`, error.message);
   },
 };
 
@@ -537,18 +556,48 @@ await this.kafka.startConsumer(['order.created'], handler, {
 
 Multiple interceptors run in order. All hooks are optional.
 
+## Instrumentation
+
+For client-wide cross-cutting concerns (tracing, metrics), use `KafkaInstrumentation` hooks instead of per-consumer interceptors:
+
+```typescript
+import { otelInstrumentation } from '@drarzter/kafka-client/otel';
+
+const kafka = new KafkaClient('my-app', 'my-group', brokers, {
+  instrumentation: [otelInstrumentation()],
+});
+```
+
+`otelInstrumentation()` injects `traceparent` on send, extracts it on consume, and creates `CONSUMER` spans automatically. Requires `@opentelemetry/api` as a peer dependency.
+
+Custom instrumentation:
+
+```typescript
+import { KafkaInstrumentation } from '@drarzter/kafka-client';
+
+const metrics: KafkaInstrumentation = {
+  beforeSend(topic, headers) { /* inject headers, start timer */ },
+  afterSend(topic) { /* record send latency */ },
+  beforeConsume(envelope) { /* start span */ return () => { /* end span */ }; },
+  onConsumeError(envelope, error) { /* record error metric */ },
+};
+```
+
 ## Options reference
 
 ### Send options
 
 Options for `sendMessage()` — the third argument:
 
-| Option    | Default | Description                                      |
-|-----------|---------|--------------------------------------------------|
-| `key`     | —       | Partition key for message routing                 |
-| `headers` | —       | Custom metadata headers (`Record<string, string>`) |
+| Option | Default | Description |
+|--------|---------|-------------|
+| `key` | — | Partition key for message routing |
+| `headers` | — | Custom metadata headers (merged with auto-generated envelope headers) |
+| `correlationId` | auto | Override the auto-propagated correlation ID (default: inherited from ALS context or new UUID) |
+| `schemaVersion` | `1` | Schema version for the payload |
+| `eventId` | auto | Override the auto-generated event ID (UUID v4) |
 
-`sendBatch()` accepts `key` and `headers` per message inside the array items.
+`sendBatch()` accepts the same options per message inside the array items.
 
 ### Consumer options
 
@@ -579,6 +628,7 @@ Passed to `KafkaModule.register()` or returned from `registerAsync()` factory:
 | `autoCreateTopics` | `false` | Auto-create topics on first send (dev only) |
 | `numPartitions` | `1` | Number of partitions for auto-created topics |
 | `strictSchemas` | `true` | Validate string topic keys against schemas registered via TopicDescriptor |
+| `instrumentation` | `[]` | Client-wide instrumentation hooks (e.g. OTel). Applied to both send and consume paths |
 
 **Module-scoped** (default) — import `KafkaModule` in each module that needs it:
 
@@ -641,7 +691,7 @@ err.cause;            // the original error
 ```typescript
 // In an onError interceptor:
 const interceptor: ConsumerInterceptor<MyTopics> = {
-  onError: (message, topic, error) => {
+  onError: (envelope, error) => {
     if (error instanceof KafkaRetryExhaustedError) {
       console.log(`Failed after ${error.attempts} attempts on ${error.topic}`);
       console.log('Last error:', error.cause);
@@ -658,7 +708,7 @@ When `retry.maxRetries` is set and all attempts fail, `KafkaRetryExhaustedError`
 import { KafkaValidationError } from '@drarzter/kafka-client';
 
 const interceptor: ConsumerInterceptor<MyTopics> = {
-  onError: (message, topic, error) => {
+  onError: (envelope, error) => {
     if (error instanceof KafkaValidationError) {
       console.log(`Bad message on ${error.topic}:`, error.cause?.message);
     }
@@ -707,9 +757,9 @@ await kafka.sendMessage(OrderCreated, { orderId: '1', userId: '2', amount: -5 })
 
 ```typescript
 @SubscribeTo(OrderCreated, { dlq: true })
-async handleOrder(message) {
-  // `message` is guaranteed to match the schema
-  console.log(message.orderId); // string — validated at runtime
+async handleOrder(envelope) {
+  // `envelope.payload` is guaranteed to match the schema
+  console.log(envelope.payload.orderId); // string — validated at runtime
 }
 ```
 
@@ -773,7 +823,73 @@ export class HealthService {
 
 ## Testing
 
-Unit tests (mocked kafkajs):
+### Testing utilities
+
+Import from `@drarzter/kafka-client/testing` — zero runtime deps, only `jest` and `@testcontainers/kafka` as peer dependencies.
+
+> Unit tests mock `@confluentinc/kafka-javascript` — no Kafka broker needed. Integration tests use Testcontainers (requires Docker).
+
+#### `createMockKafkaClient<T>()`
+
+Fully typed mock with `jest.fn()` on every `IKafkaClient` method. All methods resolve to sensible defaults:
+
+```typescript
+import { createMockKafkaClient } from '@drarzter/kafka-client/testing';
+
+const kafka = createMockKafkaClient<MyTopics>();
+
+const service = new OrdersService(kafka);
+await service.createOrder();
+
+expect(kafka.sendMessage).toHaveBeenCalledWith(
+  'order.created',
+  expect.objectContaining({ orderId: '123' }),
+);
+
+// Override return values
+kafka.checkStatus.mockResolvedValueOnce({ topics: ['order.created'] });
+
+// Mock rejections
+kafka.sendMessage.mockRejectedValueOnce(new Error('broker down'));
+```
+
+#### `KafkaTestContainer`
+
+Thin wrapper around `@testcontainers/kafka` that handles common setup pain points — transaction coordinator warmup, topic pre-creation:
+
+```typescript
+import { KafkaTestContainer } from '@drarzter/kafka-client/testing';
+import { KafkaClient } from '@drarzter/kafka-client/core';
+
+let container: KafkaTestContainer;
+let brokers: string[];
+
+beforeAll(async () => {
+  container = new KafkaTestContainer({
+    topics: ['orders', { topic: 'payments', numPartitions: 3 }],
+  });
+  brokers = await container.start();
+}, 120_000);
+
+afterAll(() => container.stop());
+
+it('sends and receives', async () => {
+  const kafka = new KafkaClient('test', 'test-group', brokers);
+  // ...
+});
+```
+
+Options:
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `image` | `"confluentinc/cp-kafka:7.7.0"` | Docker image |
+| `transactionWarmup` | `true` | Warm up transaction coordinator on start |
+| `topics` | `[]` | Topics to pre-create (string or `{ topic, numPartitions }`) |
+
+### Running tests
+
+Unit tests (mocked `@confluentinc/kafka-javascript` — no broker needed):
 
 ```bash
 npm test
@@ -787,15 +903,18 @@ npm run test:integration
 
 The integration suite spins up a single-node KRaft Kafka container and tests sending, consuming, batching, transactions, retry + DLQ, interceptors, health checks, and `fromBeginning` — no mocks.
 
-Both suites run in CI on every push to `main`.
+Both suites run in CI on every push to `main` and on pull requests.
 
 ## Project structure
 
-```
+```text
 src/
-├── client/         # Core — KafkaClient, types, topic(), error classes (0 framework deps)
+├── client/         # Core — KafkaClient, types, envelope, consumer pipeline, topic(), errors (0 framework deps)
 ├── nest/           # NestJS adapter — Module, Explorer, decorators, health
+├── testing/        # Testing utilities — mock client, testcontainer wrapper
 ├── core.ts         # Standalone entrypoint (@drarzter/kafka-client/core)
+├── otel.ts         # OpenTelemetry entrypoint (@drarzter/kafka-client/otel)
+├── testing.ts      # Testing entrypoint (@drarzter/kafka-client/testing)
 └── index.ts        # Full entrypoint — core + NestJS adapter
 ```
 

package/dist/chunk-EQQGB2QZ.mjs

@@ -0,0 +1,17 @@
+var __defProp = Object.defineProperty;
+var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __decorateClass = (decorators, target, key, kind) => {
+  var result = kind > 1 ? void 0 : kind ? __getOwnPropDesc(target, key) : target;
+  for (var i = decorators.length - 1, decorator; i >= 0; i--)
+    if (decorator = decorators[i])
+      result = (kind ? decorator(target, key, result) : decorator(result)) || result;
+  if (kind && result) __defProp(target, key, result);
+  return result;
+};
+var __decorateParam = (index, decorator) => (target, key) => decorator(target, key, index);
+
+export {
+  __decorateClass,
+  __decorateParam
+};
+//# sourceMappingURL=chunk-EQQGB2QZ.mjs.map

package/dist/chunk-EQQGB2QZ.mjs.map

@@ -0,0 +1 @@
+{"version":3,"sources":[],"sourcesContent":[],"mappings":"","names":[]}