kafka-crab-js 2.1.0 → 3.0.0-alpha.2

package/README.md

@@ -10,7 +10,81 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
 
 ---
 
-## What's New in Version 2.1.0
+## What's New in Version 3.0.0
+
+### BREAKING CHANGES ⚠️
+
+**OpenTelemetry instrumentation has been moved to a separate package: `kafka-crab-js-otel`**
+
+This change reduces the core package size and makes OTEL an opt-in dependency.
+
+#### Migration from v2.x
+
+**Before (v2.x):**
+```javascript
+import { KafkaClient } from 'kafka-crab-js'
+
+const client = new KafkaClient({
+  brokers: 'localhost:9092',
+  clientId: 'my-app',
+  otel: {
+    serviceName: 'my-service',
+    metrics: { enabled: true },
+  },
+})
+```
+
+**After (v3.x):**
+```javascript
+import { KafkaClient } from 'kafka-crab-js'
+import { enableOtelInstrumentation, endSpan } from 'kafka-crab-js-otel'
+
+// 1. Enable OTEL instrumentation BEFORE creating client
+enableOtelInstrumentation({
+  serviceName: 'my-service',
+  metrics: { enabled: true },
+})
+
+// 2. Create client with diagnostics enabled
+const client = new KafkaClient({
+  brokers: 'localhost:9092',
+  clientId: 'my-app',
+  diagnostics: true, // Required for OTEL to receive events
+})
+
+// 3. For consumers, call endSpan() when processing is complete
+const message = await consumer.recv()
+// ... process message ...
+endSpan(message)
+```
+
+#### Key Changes
+
+| v2.x | v3.x |
+|------|------|
+| `otel` config in KafkaClient | `enableOtelInstrumentation()` from `kafka-crab-js-otel` |
+| Automatic span ending | Call `endSpan(message)` manually for consumers |
+| N/A | `diagnostics: true` required in KafkaClient config |
+| OTEL bundled | Install `kafka-crab-js-otel` separately |
+
+#### New Package Structure
+
+| Package | Description |
+|---------|-------------|
+| `kafka-crab-js` | Core Kafka client (producer, consumer, streams) |
+| `kafka-crab-js-otel` | OpenTelemetry instrumentation (separate install) |
+
+### Internal Improvements
+- Uses Node.js `diagnostics_channel` for observability (zero overhead when not subscribed)
+- Cleaner separation of concerns between core and observability
+- Smaller bundle size for users who don't need OTEL
+
+---
+
+## Previous Versions
+
+<details>
+<summary>What's New in Version 2.1.0</summary>
 
 ### New Features
 
@@ -40,103 +114,25 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
    - GitHub Actions updated to v6
    - Improved caching with actions/cache v4
 
-### Migration from 2.0.x
+</details>
 
-- **No breaking changes** - all existing code continues to work
-- The new `commitMessage()` method is additive - existing `commit()` calls remain valid
-- OTEL improvements are transparent and require no code changes
-
----
-
-## What's New in Version 2.0.0
+<details>
+<summary>What's New in Version 2.0.0</summary>
 
 ### BREAKING CHANGES ⚠️
 
-This major version includes important breaking changes that improve API consistency and memory management:
-
 1. **Consumer Configuration API Changes**:
    - **REMOVED**: `createTopic` field from `ConsumerConfiguration`
    - **Migration**: Use `createTopic` field in `TopicPartitionConfig` instead when subscribing to topics
-   - **Before**: `new KafkaClient().createConsumer({ groupId: 'test', createTopic: true })`
-   - **After**: `consumer.subscribe([{ topic: 'my-topic', createTopic: true }])`
 
 2. **Stream Lifecycle Management**:
    - Stream consumers now properly implement Node.js stream lifecycle methods (`_destroy()`)
    - **Memory leak prevention**: Streams now automatically disconnect Kafka consumers during destruction
-   - Better resource cleanup ensures no hanging connections or memory leaks
-   - Error handling during stream destruction is improved with proper error propagation
-
-3. **Stream Error Handling**:
-   - Stream errors now use `destroy(error)` instead of `emit('error')` for proper stream termination
-   - This ensures streams are properly closed when errors occur, preventing resource leaks
-   - Error propagation follows Node.js stream standards more closely
-
-4. **Rust Configuration Simplification**:
-   - Removed `MAX_BATCH_TIMEOUT` constant - now uses only `DEFAULT_BATCH_TIMEOUT` (1000ms)
-   - Simplified timeout validation logic in Rust layer
-   - Invalid timeout values (< 1ms) now log warnings and use default timeout
-
-5. **Code Quality Improvements**:
-   - Fixed linting violations: short variable names and `any` type usage
-   - Better TypeScript typing throughout the codebase
-   - Removed redundant validation functions for cleaner code
-
-### Major Updates:
-
-1. **Performance Benchmark Suite**:
-   - Added comprehensive benchmark suite comparing performance against `kafkajs`, `node-rdkafka`, and `@platformatic/rdkafka`
-   - Benchmark scripts with automated setup and data generation
-   - Performance testing for both serial and batch processing modes
-   - Supports configurable iterations and message sizes for realistic testing scenarios
-
-2. **Enhanced Stream Consumer Configuration**:
-   - Stream consumer constructor now accepts optional `ReadableOptions` parameter
-   - Default `objectMode: true` for better stream processing ergonomics
-   - Improved batch processing configuration with proper defaults
-
-3. **Flexible Configuration System**:
-   - **BREAKING**: Configuration properties now accept `Record<string, any>` instead of `Record<string, string>`
-   - Enhanced type flexibility for complex configuration values (numbers, booleans, objects)
-   - Better support for advanced librdkafka configuration options
-
-4. **Enhanced Topic Management**:
-   - Added `createTopic`, `numPartitions`, and `replicas` options to `TopicPartitionConfig`
-   - Improved topic creation control during consumer subscription
-   - Better handling of partition assignment and topic setup
-
-5. **Build System Improvements**:
-   - Updated TypeScript configuration with modular `tsconfig.base.json`
-   - Added `typecheck` script for standalone type checking
-   - Enhanced build pipeline with better declaration file generation
-   - Updated to pnpm@10.15.1 for improved dependency management
-
-6. **Development Tools Enhancement**:
-   - Enhanced oxlint configuration with benchmark-specific rules
-   - Added support for benchmark directory linting and formatting
-   - Improved development workflow with better error handling rules
-
-7. **Consumer Topic Creation Control** (v1.9.0):
-   - Added fine-grained control over consumer topic creation behavior
-   - Enhanced consistency in topic management and cleanup processes
-   - Improved handling of topic creation inconsistencies during consumer operations
-
-8. **Enhanced Manual Commit Testing** (v1.9.0):
-   - Consolidated manual commit tests for better reliability and coverage
-   - Added comprehensive integration tests with PostRebalance event handling
-   - Tests now verify offset persistence across consumer restarts and batch processing
-
-9. **Async Consumer Commit** (v1.8.0 - BREAKING):
-   - **BREAKING**: The `consumer.commit()` method is now async and must be awaited
-   - **Before**: `consumer.commit(topic, partition, offset, 'Sync')`
-   - **After**: `await consumer.commit(topic, partition, offset, 'Sync')`
-   - This change improves performance by using `spawn_blocking` for non-blocking operations
 
-### Migration Guide
+3. **Async Consumer Commit**:
+   - **BREAKING**: The `consumer.commit()` method is now async and must be awaited
 
-- **No API changes required** - all public interfaces remain the same
-- **Stream memory management** - streams now automatically clean up resources
-- **Error handling** - streams will properly terminate on errors (existing error handling code still works)
-- **Performance** - simplified timeout logic may provide marginal performance improvements
+</details>
 
 ---
 
@@ -506,12 +502,12 @@ You can find some examples on the [example](https://github.com/flash-tecnologia/
 | Property | Type | Default | Description |
 | --- | --- | --- | --- |
 | `brokers` | `string` || List of brokers to connect to |
-| `clientId` | `string` || Client id to use for the connection |
+| `clientId` | `string` | `"rdkafka"` | Client id to use for the connection |
 | `securityProtocol` | `SecurityProtocol` || Security protocol to use (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL) |
 | `logLevel` | `string` | `info`  | Log level for the client |
 | `brokerAddressFamily` | `string` | `"v4"` | Address family to use for the connection (v4, v6) |
-| `configuration` | `Record<string, any>` | `{}` | Additional configuration options for the client. **v2.0.0+**: Now supports any value type (string, number, boolean, object) |
-| `otel` | `KafkaOtelInstrumentationConfig \| false` | `undefined` | Enable and configure OpenTelemetry instrumentation (see [OpenTelemetry Instrumentation](#opentelemetry-instrumentation)) |
+| `configuration` | `Record<string, any>` | `{}` | Additional configuration options for the client |
+| `diagnostics` | `boolean` | `false` | **v3.0.0+**: Enable diagnostics channel for OTEL instrumentation |
 
 ### ConsumerConfiguration
 
@@ -565,29 +561,49 @@ Both methods support `'Sync'` and `'Async'` commit modes.
 
 You can see the available options here: [librdkafka](https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html).
 
-### OpenTelemetry Configuration
+### OpenTelemetry Instrumentation
 
-OpenTelemetry is enabled by default when the `otel` field is omitted or set to an object. Set `otel: false` to disable instrumentation entirely.
+> **Note:** Starting with v3.0.0, OpenTelemetry instrumentation has been moved to a separate package: [`kafka-crab-js-otel`](https://www.npmjs.com/package/kafka-crab-js-otel)
 
-```ts
+#### Installation
+
+```bash
+npm install kafka-crab-js-otel @opentelemetry/api
+```
+
+#### Usage
+
+```javascript
+import { KafkaClient } from 'kafka-crab-js'
+import { enableOtelInstrumentation, endSpan } from 'kafka-crab-js-otel'
+
+// Enable instrumentation before creating client
+enableOtelInstrumentation({
+  serviceName: 'my-service',
+  metrics: { enabled: true },
+})
+
+// Create client with diagnostics enabled
 const client = new KafkaClient({
-  brokers: 'localhost:29092',
-  clientId: 'payments-service',
-  otel: {
-    serviceName: 'payments-api',
-    ignoreTopics: topic => topic.startsWith('internal.'),
-    messageHook: (span, message) => {
-      span.setAttribute('app.message.key', message.key?.toString('utf8'))
-    },
-  },
+  brokers: 'localhost:9092',
+  clientId: 'my-app',
+  diagnostics: true,
 })
+
+// For consumers, call endSpan() when done processing
+const message = await consumer.recv()
+// ... process message ...
+endSpan(message)
 ```
 
-When instrumentation is active:
+For complete configuration options and examples, see the [kafka-crab-js-otel README](../kafka-crab-js-otel/README.md).
+
+#### Examples
 
-- Producer `send` calls automatically propagate the active span context via Kafka headers while preserving custom headers and their Buffer/string types.
-- Consumer `recv`, `recvBatch`, and stream consumers generate spans that capture consumer group, topic, partition, offset, and batch size.
-- Hooks registered through `messageHook` and `producerHook` execute within the active span context so that additional attributes/events can be attached safely.
+See comprehensive examples in the `example/` directory:
+- `example/otel-tracing-example.mjs` - Complete tracing setup with Jaeger
+- `example/otel-metrics-example.mjs` - Metrics collection with Prometheus
+- `example/README.md` - Full documentation for all examples
 
 ## Performance Benchmarks
 
@@ -723,10 +739,34 @@ Kafka Crab JS offers turnkey tracing for Kafka workloads:
 - **Hook-friendly spans** – Both `messageHook` and `producerHook` callbacks run inside the active span context, simplifying attribute decoration or error handling.
 - **Header normalization helpers** – The runtime automatically normalizes mixed header carriers to Buffers when talking to the native binding, removing the need for manual conversions.
 
+### Consumer span lifecycle (important)
+
+Kafka Crab JS creates consumer `process <topic>` spans, but only your application knows when processing is complete.
+
+- For single-message consumers, call `message.endSpan()` when you're done processing the message.
+- For batch consumers, call `batch.endSpan()` (the returned array is augmented) when you're done processing the batch.
+- To avoid optional chaining in user code, use the helper `endSpan(message)` (or `EndSpan(message)`).
+- For automatic end + error capture, wrap your handler with `await client.otel.processMessage(message, async (msg) => { ... })` (or `processBatch`).
+
+This closes the span(s) and (when metrics are enabled) records `messaging.process.duration`.
+
+If you prefer not to mutate the native `Message` objects, set `otel.decorateMessages=false`. In that mode,
+`recv()`/`recvBatch()` return shallow clones augmented with the same OTEL helper fields.
+
+### Global instrumentation (singleton)
+
+`kafka-crab-js` uses a process-wide OpenTelemetry instrumentation singleton.
+
+- Creating multiple `KafkaClient` instances shares the same instrumentation and configuration.
+- Passing different `otel` configs to different clients updates the same singleton (the latest config wins).
+- For tests, use `resetKafkaInstrumentation()` to clear the singleton between runs.
+
 ### Minimal Setup Example
 
+When using stream consumers, the easiest pattern is calling `endSpan()` in a `finally` block:
+
 ```ts
-import { KafkaClient } from 'kafka-crab-js'
+import { KafkaClient, endSpan } from 'kafka-crab-js'
 import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'
 import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'
 import { AsyncHooksContextManager } from '@opentelemetry/context-async-hooks'
@@ -762,7 +802,12 @@ const consumer = client.createStreamConsumer({
 })
 
 consumer.on('data', message => {
-  console.log(message.headers?.['custom-header']?.toString())
+  try {
+    console.log(message.headers?.['custom-header']?.toString())
+  } finally {
+    // IMPORTANT: close the "process <topic>" span when your app finishes handling the message
+    endSpan(message)
+  }
 })
 ```
 

package/dist/diagnostics/channels.d.ts

@@ -0,0 +1,138 @@
+/**
+ * Kafka Diagnostic Channels
+ *
+ * This module provides Node.js diagnostic channels for Kafka operations.
+ * External observability tools (like OpenTelemetry adapters) can subscribe
+ * to these channels to receive events without coupling to any specific
+ * tracing/metrics implementation.
+ *
+ * @see https://nodejs.org/api/diagnostics_channel.html
+ */
+import { type Channel } from 'node:diagnostics_channel';
+import type { Message, ProducerRecord, RecordMetadata } from '../../js-binding.js';
+export declare const CHANNEL_NAMES: {
+    readonly PRODUCER_SEND_START: "kafka-crab:producer:send:start";
+    readonly PRODUCER_SEND_END: "kafka-crab:producer:send:end";
+    readonly PRODUCER_SEND_ERROR: "kafka-crab:producer:send:error";
+    readonly CONSUMER_RECEIVE_START: "kafka-crab:consumer:receive:start";
+    readonly CONSUMER_RECEIVE_END: "kafka-crab:consumer:receive:end";
+    readonly CONSUMER_PROCESS_START: "kafka-crab:consumer:process:start";
+    readonly CONSUMER_PROCESS_END: "kafka-crab:consumer:process:end";
+    readonly BATCH_RECEIVE_START: "kafka-crab:batch:receive:start";
+    readonly BATCH_RECEIVE_END: "kafka-crab:batch:receive:end";
+    readonly BATCH_PROCESS_START: "kafka-crab:batch:process:start";
+    readonly BATCH_PROCESS_END: "kafka-crab:batch:process:end";
+};
+export type ChannelName = (typeof CHANNEL_NAMES)[keyof typeof CHANNEL_NAMES];
+/**
+ * Base event interface with common fields
+ */
+export interface BaseEvent {
+    /** Timestamp when the event occurred (ms since epoch) */
+    timestamp: number;
+    /** Client ID if available */
+    clientId?: string;
+    /** Server address if configured */
+    serverAddress?: string;
+    /** Server port if configured */
+    serverPort?: number;
+    /**
+     * Context object for adapters to attach data (e.g., spans).
+     * Subscribers can mutate this to pass data between start/end events.
+     */
+    context: Record<PropertyKey, unknown>;
+}
+export interface ProducerSendStartEvent extends BaseEvent {
+    topic: string;
+    record: ProducerRecord;
+    messageCount: number;
+}
+export interface ProducerSendEndEvent extends BaseEvent {
+    topic: string;
+    record: ProducerRecord;
+    metadata?: RecordMetadata[];
+    durationMs: number;
+    error?: Error;
+}
+export interface ProducerSendErrorEvent extends BaseEvent {
+    topic: string;
+    record: ProducerRecord;
+    durationMs: number;
+    error: Error;
+}
+export interface ConsumerReceiveStartEvent extends BaseEvent {
+    groupId?: string;
+}
+export interface ConsumerReceiveEndEvent extends BaseEvent {
+    message: Message | null;
+    groupId?: string;
+    durationMs: number;
+    error?: Error;
+}
+export interface ConsumerProcessStartEvent extends BaseEvent {
+    message: Message;
+    groupId?: string;
+}
+export interface ConsumerProcessEndEvent extends BaseEvent {
+    message: Message;
+    groupId?: string;
+    durationMs: number;
+    error?: Error;
+}
+export interface BatchReceiveStartEvent extends BaseEvent {
+    groupId?: string;
+    requestedSize: number;
+    timeoutMs: number;
+}
+export interface BatchReceiveEndEvent extends BaseEvent {
+    messages: Message[];
+    groupId?: string;
+    durationMs: number;
+    error?: Error;
+}
+export interface BatchProcessStartEvent extends BaseEvent {
+    messages: Message[];
+    groupId?: string;
+}
+export interface BatchProcessEndEvent extends BaseEvent {
+    messages: Message[];
+    groupId?: string;
+    durationMs: number;
+    error?: Error;
+}
+export type KafkaEvent = ProducerSendStartEvent | ProducerSendEndEvent | ConsumerReceiveStartEvent | ConsumerReceiveEndEvent | ConsumerProcessStartEvent | ConsumerProcessEndEvent | BatchReceiveStartEvent | BatchReceiveEndEvent | BatchProcessStartEvent | BatchProcessEndEvent;
+/**
+ * Typed channel wrapper for type-safe publish/subscribe
+ */
+export interface TypedChannel<TEvent> {
+    readonly name: string;
+    readonly channel: Channel;
+    hasSubscribers: boolean;
+    publish(event: TEvent): void;
+    subscribe(handler: (event: TEvent, name: string | symbol) => void): void;
+    unsubscribe(handler: (event: TEvent, name: string | symbol) => void): void;
+}
+export declare const producerSendStartChannel: TypedChannel<ProducerSendStartEvent>;
+export declare const producerSendEndChannel: TypedChannel<ProducerSendEndEvent>;
+export declare const producerSendErrorChannel: TypedChannel<ProducerSendErrorEvent>;
+export declare const consumerReceiveStartChannel: TypedChannel<ConsumerReceiveStartEvent>;
+export declare const consumerReceiveEndChannel: TypedChannel<ConsumerReceiveEndEvent>;
+export declare const consumerProcessStartChannel: TypedChannel<ConsumerProcessStartEvent>;
+export declare const consumerProcessEndChannel: TypedChannel<ConsumerProcessEndEvent>;
+export declare const batchReceiveStartChannel: TypedChannel<BatchReceiveStartEvent>;
+export declare const batchReceiveEndChannel: TypedChannel<BatchReceiveEndEvent>;
+export declare const batchProcessStartChannel: TypedChannel<BatchProcessStartEvent>;
+export declare const batchProcessEndChannel: TypedChannel<BatchProcessEndEvent>;
+export declare const channels: {
+    readonly producerSendStart: TypedChannel<ProducerSendStartEvent>;
+    readonly producerSendEnd: TypedChannel<ProducerSendEndEvent>;
+    readonly producerSendError: TypedChannel<ProducerSendErrorEvent>;
+    readonly consumerReceiveStart: TypedChannel<ConsumerReceiveStartEvent>;
+    readonly consumerReceiveEnd: TypedChannel<ConsumerReceiveEndEvent>;
+    readonly consumerProcessStart: TypedChannel<ConsumerProcessStartEvent>;
+    readonly consumerProcessEnd: TypedChannel<ConsumerProcessEndEvent>;
+    readonly batchReceiveStart: TypedChannel<BatchReceiveStartEvent>;
+    readonly batchReceiveEnd: TypedChannel<BatchReceiveEndEvent>;
+    readonly batchProcessStart: TypedChannel<BatchProcessStartEvent>;
+    readonly batchProcessEnd: TypedChannel<BatchProcessEndEvent>;
+};

package/dist/diagnostics/index.d.ts

@@ -0,0 +1,7 @@
+/**
+ * Diagnostics module entry point
+ *
+ * Re-exports all channels and types for external consumption.
+ */
+export { type BaseEvent, batchProcessEndChannel, type BatchProcessEndEvent, batchProcessStartChannel, type BatchProcessStartEvent, batchReceiveEndChannel, type BatchReceiveEndEvent, batchReceiveStartChannel, type BatchReceiveStartEvent, CHANNEL_NAMES, type ChannelName, channels, consumerProcessEndChannel, type ConsumerProcessEndEvent, consumerProcessStartChannel, type ConsumerProcessStartEvent, consumerReceiveEndChannel, type ConsumerReceiveEndEvent, consumerReceiveStartChannel, type ConsumerReceiveStartEvent, type KafkaEvent, producerSendEndChannel, type ProducerSendEndEvent, producerSendErrorChannel, type ProducerSendErrorEvent, producerSendStartChannel, type ProducerSendStartEvent, type TypedChannel, } from './channels.js';
+export { type DiagnosticInstrumentationConfig, instrumentBatchReceive, instrumentConsumerReceive, instrumentProducerSend, } from './instrumentation.js';

package/dist/diagnostics/instrumentation.d.ts

@@ -0,0 +1,31 @@
+/**
+ * Diagnostic Channel-based Kafka Instrumentation
+ *
+ * This module provides instrumentation that publishes events to Node.js
+ * diagnostic channels instead of directly calling OpenTelemetry APIs.
+ * Observability tools can subscribe to these channels to receive events.
+ */
+import type { KafkaConsumer, KafkaProducer, Message, ProducerRecord, RecordMetadata } from '../../js-binding.js';
+/**
+ * Configuration for diagnostic channel instrumentation
+ */
+export interface DiagnosticInstrumentationConfig {
+    /** Client ID for attribution */
+    clientId?: string;
+    /** Server address for attribution */
+    serverAddress?: string;
+    /** Server port for attribution */
+    serverPort?: number;
+}
+/**
+ * Instruments a producer send method to publish diagnostic events
+ */
+export declare function instrumentProducerSend(originalSend: (record: ProducerRecord) => Promise<RecordMetadata[]>, config?: DiagnosticInstrumentationConfig): (this: KafkaProducer, record: ProducerRecord) => Promise<RecordMetadata[]>;
+/**
+ * Instruments a consumer receive method to publish diagnostic events
+ */
+export declare function instrumentConsumerReceive(originalReceive: () => Promise<Message | null>, groupId?: string, config?: DiagnosticInstrumentationConfig): (this: KafkaConsumer) => Promise<Message | null>;
+/**
+ * Instruments a consumer batch receive method to publish diagnostic events
+ */
+export declare function instrumentBatchReceive(originalBatchReceive: (size: number, timeoutMs: number) => Promise<Message[]>, groupId?: string, config?: DiagnosticInstrumentationConfig): (this: KafkaConsumer, size: number, timeoutMs: number) => Promise<Message[]>;