kafka-crab-js 4.0.0-beta.2 → 4.0.0

package/README.md

@@ -1,907 +1,1131 @@
-<div align="center">
+# Kafka Crab JS
 
-# 🦀 Kafka Crab JS 🦀
+Kafka Crab JS is a native Kafka client for Node.js and TypeScript. It uses Rust, NAPI-RS, and librdkafka to reduce
+JavaScript heap pressure, expose Kafka's mature native client behavior, and push high-throughput consumer workloads
+through a small TypeScript-friendly API.
 
-A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It is built using Rust and librdkafka, providing a high-performance and feature-rich Kafka client.
-
-[![npm version](https://img.shields.io/npm/v/kafka-crab-js.svg)](https://www.npmjs.com/package/kafka-crab-js)
+[![npm beta version](https://img.shields.io/badge/npm%20beta-v4.0.0--beta.3-blue.svg)](https://www.npmjs.com/package/kafka-crab-js)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-</div>
-
----
+## Version 4
+
+`kafka-crab-js` v4 is the current beta line. It focuses on high-throughput consumers, lower JavaScript heap pressure,
+and a cleaner monorepo release model under [`flash-tecnologia/crab-js`](https://github.com/flash-tecnologia/crab-js).
+
+The main v4 additions are:
+
+- Native Web Stream consumers through `client.createWebStreamConsumer()`, with explicit `serial` and `batch` modes.
+- Direct batch consumption through `consumer.recvBatch(size, timeoutMs)` for workers that can process chunks.
+- Compact native batch streaming internally, reducing native-to-JavaScript marshalling overhead before expanding to the
+  public `Message[]` API.
+- Serial Web Stream prefetching, so message-by-message code still benefits from small native batches.
+- Diagnostics-channel events in the core package, with OpenTelemetry tracing and metrics moved to the optional
+  `kafka-crab-js-otel` package.
+- Updated TypeScript exports where runtime enum-like values are now string literal types, avoiding missing runtime
+  exports in ESM/CommonJS consumers.
+
+For production adoption, treat v4 as a beta release: validate it with your broker settings, payload sizes, partition
+count, commit strategy, and shutdown path before replacing an existing Kafka client.
+
+## Why Kafka Crab JS?
+
+KafkaJS is a strong default for many Node.js services. Kafka Crab JS is for the cases where Kafka throughput, memory
+pressure, native batching, or librdkafka behavior start to matter.
+
+| Need                                | What Kafka Crab JS Provides                                                              |
+| ----------------------------------- | ---------------------------------------------------------------------------------------- |
+| Higher consumer throughput          | Native batch receive paths and Web Stream batch consumers.                               |
+| Lower JavaScript heap pressure      | Message fetch and Kafka protocol work happen in Rust/librdkafka before crossing into JS. |
+| Predictable Kafka offset handling   | `commitMessage()` commits `message.offset + 1`, the value Kafka expects.                 |
+| Production Kafka tuning             | Advanced librdkafka settings are passed through with their original names.               |
+| Node.js ecosystem integration       | Direct APIs, Node.js `Readable` streams, native Web Streams, and CommonJS/ESM exports.   |
+| Observability without core coupling | Diagnostics-channel events power the optional `kafka-crab-js-otel` package.              |
+
+In the local isolated consumer benchmark snapshot, `kafka-crab-js v4 (stream, batch)` reached about
+`1.17M messages/sec`, ahead of the measured `@platformatic/kafka` and KafkaJS batch scenarios, while using lower RSS
+delta and peak heap than both. Treat benchmark numbers as workload-specific; the full methodology and memory/GC
+breakdown are documented in [BENCHMARKS.md](../../BENCHMARKS.md) and [Performance Benchmarks](#performance-benchmarks).
+
+## Highlights
+
+- Native librdkafka client exposed through a TypeScript-friendly API.
+- Producer, direct consumer, Node.js `Readable` stream, and v4 native Web Stream consumer APIs.
+- High-throughput batch receive APIs for workloads that can process more than one message at a time.
+- Compact batch streaming and serial prefetching to reduce native boundary crossings.
+- Manual commit helpers that commit `message.offset + 1` correctly.
+- Optional diagnostics-channel instrumentation for OpenTelemetry through `kafka-crab-js-otel`.
+- Advanced librdkafka settings are forwarded through `configuration` without forcing a custom allowlist.
+
+## Requirements
+
+- Node.js `>= 22`.
+- A Kafka broker reachable from the Node.js process.
+- No separate librdkafka install is required for the published binaries.
+
+## Table Of Contents
+
+1. [Version 4](#version-4)
+2. [Installation](#installation)
+3. [Module Usage](#module-usage)
+4. [Quick Start](#quick-start)
+5. [Message Model](#message-model)
+6. [Choosing A Consumer API](#choosing-a-consumer-api)
+7. [Producer API](#producer-api)
+8. [Consumer API](#consumer-api)
+9. [Stream Consumers](#stream-consumers)
+10. [Batching, Backpressure, And Tuning](#batching-backpressure-and-tuning)
+11. [Configuration](#configuration)
+12. [OpenTelemetry](#opentelemetry)
+13. [Performance Benchmarks](#performance-benchmarks)
+14. [Migration Notes](#migration-notes)
+15. [Troubleshooting](#troubleshooting)
+16. [Development](#development)
+17. [License](#license)
 
-## What's New in Version 4.0.0
+## Installation
 
-### BREAKING CHANGES ⚠️
+```bash
+npm install kafka-crab-js
+```
 
-**`CommitMode`, `KafkaEventName`, `PartitionPosition`, and `SecurityProtocol` are no longer runtime exports from `kafka-crab-js`.**
+```bash
+pnpm add kafka-crab-js
+```
 
-These names are now type-only at the root package API. This change applies to the `kafka-crab-js` root import surface and is intended as a major-version change.
+```bash
+yarn add kafka-crab-js
+```
 
-#### Migration from v3.x
+## Module Usage
 
-**Before (v3.x):**
+The package publishes both ESM and CommonJS entry points:
 
 ```ts
-import { CommitMode, SecurityProtocol } from 'kafka-crab-js'
+import { KafkaClient } from 'kafka-crab-js'
+import type { Message, ProducerRecord } from 'kafka-crab-js'
 ```
 
-**After for type-only usage:**
-
-```ts
-import type { CommitMode, SecurityProtocol } from 'kafka-crab-js'
+```js
+const { KafkaClient } = require('kafka-crab-js')
 ```
 
-**After for runtime usage:**
+The public TypeScript types are generated from the native NAPI contract and the JavaScript wrapper. Runtime enum-like
+values such as commit modes and security protocols are string literals, not runtime objects:
 
 ```ts
-const commitMode: 'Sync' | 'Async' = 'Sync'
-const securityProtocol: 'Plaintext' | 'Ssl' | 'SaslPlaintext' | 'SaslSsl' = 'Plaintext'
-const eventName: 'PreRebalance' | 'PostRebalance' | 'CommitCallback' = 'PreRebalance'
-const partitionPosition: 'Beginning' | 'End' | 'Stored' | 'Invalid' = 'Beginning'
+await consumer.commitMessage(message, 'Sync')
+
+const client = new KafkaClient({
+  brokers: 'localhost:9092',
+  securityProtocol: 'Plaintext',
+})
 ```
 
-#### Replacement Values
+## Quick Start
 
-| Name                | Replacement string literal values                       |
-| ------------------- | ------------------------------------------------------- |
-| `CommitMode`        | `'Sync' \| 'Async'`                                     |
-| `KafkaEventName`    | `'PreRebalance' \| 'PostRebalance' \| 'CommitCallback'` |
-| `PartitionPosition` | `'Beginning' \| 'End' \| 'Stored' \| 'Invalid'`         |
-| `SecurityProtocol`  | `'Plaintext' \| 'Ssl' \| 'SaslPlaintext' \| 'SaslSsl'`  |
+### Produce Messages
 
-## What's New in Version 3.0.0
+```ts
+import { KafkaClient } from 'kafka-crab-js'
 
-### BREAKING CHANGES ⚠️
+const client = new KafkaClient({
+  brokers: 'localhost:9092',
+  clientId: 'orders-api',
+  securityProtocol: 'Plaintext',
+})
 
-**OpenTelemetry instrumentation has been moved to a separate package: `kafka-crab-js-otel`**
+const producer = client.createProducer()
 
-This change reduces the core package size and makes OTEL an opt-in dependency.
+const metadata = await producer.send({
+  topic: 'orders',
+  messages: [
+    {
+      key: Buffer.from('order-123'),
+      payload: Buffer.from(JSON.stringify({ id: 'order-123', status: 'created' })),
+      headers: {
+        'content-type': Buffer.from('application/json'),
+      },
+    },
+  ],
+})
 
-#### Migration from v2.x
+console.log(metadata)
+```
 
-**Before (v2.x):**
+### Consume One Message At A Time
 
-```javascript
+```ts
 import { KafkaClient } from 'kafka-crab-js'
 
 const client = new KafkaClient({
   brokers: 'localhost:9092',
-  clientId: 'my-app',
-  otel: {
-    serviceName: 'my-service',
-    metrics: { enabled: true },
+  clientId: 'orders-worker',
+  securityProtocol: 'Plaintext',
+})
+
+const consumer = client.createConsumer({
+  groupId: 'orders-worker',
+  enableAutoCommit: false,
+  configuration: {
+    'auto.offset.reset': 'earliest',
   },
 })
-```
 
-**After (v3.x):**
+await consumer.subscribe('orders')
 
-```javascript
-import { KafkaClient } from 'kafka-crab-js'
-import { enableOtelInstrumentation, endSpan } from 'kafka-crab-js-otel'
+try {
+  while (true) {
+    const message = await consumer.recv()
+    if (!message) {
+      break
+    }
 
-// 1. Enable OTEL instrumentation BEFORE creating client
-// Note: serviceName is set via OTEL SDK Resource, not here
-enableOtelInstrumentation({
-  metrics: { enabled: true },
-})
+    const value = JSON.parse(message.payload.toString('utf8'))
+    console.log({ value, topic: message.topic, partition: message.partition, offset: message.offset })
 
-// 2. Create client with diagnostics enabled
-const client = new KafkaClient({
-  brokers: 'localhost:9092',
-  clientId: 'my-app',
-  diagnostics: true, // Required for OTEL to receive events
+    await consumer.commitMessage(message, 'Sync')
+  }
+} finally {
+  await consumer.disconnect()
+}
+```
+
+### Consume In Batches
+
+```ts
+const consumer = client.createConsumer({
+  groupId: 'orders-batch-worker',
+  enableAutoCommit: false,
+  configuration: {
+    'auto.offset.reset': 'earliest',
+  },
 })
 
-// 3. For consumers, call endSpan() when processing is complete
-const message = await consumer.recv()
-// ... process message ...
-endSpan(message)
+await consumer.subscribe('orders')
+
+try {
+  while (true) {
+    const batch = await consumer.recvBatch(500, 50)
+    if (batch.length === 0) {
+      continue
+    }
+
+    for (const message of batch) {
+      await processOrder(message.payload)
+      await consumer.commitMessage(message, 'Async')
+    }
+  }
+} finally {
+  await consumer.disconnect()
+}
 ```
 
-#### Key Changes
+## Message Model
 
-| v2.x                         | v3.x                                                    |
-| ---------------------------- | ------------------------------------------------------- |
-| `otel` config in KafkaClient | `enableOtelInstrumentation()` from `kafka-crab-js-otel` |
-| Automatic span ending        | Call `endSpan(message)` manually for consumers          |
-| N/A                          | `diagnostics: true` required in KafkaClient config      |
-| OTEL bundled                 | Install `kafka-crab-js-otel` separately                 |
+Messages use `Buffer` values because the Kafka protocol treats keys, payloads, and headers as bytes. Decode them at the
+edge of your application instead of assuming text:
 
-#### New Package Structure
+```ts
+import type { Message, MessageProducer, RecordMetadata } from 'kafka-crab-js'
+
+const outgoing: MessageProducer = {
+  key: Buffer.from('order-123'),
+  payload: Buffer.from(JSON.stringify({ id: 'order-123' })),
+  headers: {
+    'content-type': Buffer.from('application/json'),
+    source: Buffer.from('orders-api'),
+  },
+}
 
-| Package              | Description                                      |
-| -------------------- | ------------------------------------------------ |
-| `kafka-crab-js`      | Core Kafka client (producer, consumer, streams)  |
-| `kafka-crab-js-otel` | OpenTelemetry instrumentation (separate install) |
+function decodeJsonMessage(message: Message) {
+  return JSON.parse(message.payload.toString('utf8')) as unknown
+}
 
-### Internal Improvements
+function logDelivery(records: RecordMetadata[]) {
+  for (const record of records) {
+    console.log(`${record.topic}[${record.partition}]@${record.offset}`)
+  }
+}
+```
 
-- Uses Node.js `diagnostics_channel` for observability (zero overhead when not subscribed)
-- Cleaner separation of concerns between core and observability
-- Smaller bundle size for users who don't need OTEL
+`Message` values contain the consumed Kafka offset. When committing manually, commit the next offset. Prefer
+`commitMessage()` unless you intentionally need to calculate offsets yourself:
 
----
+```ts
+await consumer.commitMessage(message, 'Sync')
+await consumer.commit(message.topic, message.partition, message.offset + 1, 'Async')
+```
 
-## Previous Versions
+Keys are optional. An omitted key keeps Kafka producer partitioning semantics for keyless records. An empty
+`Buffer.alloc(0)` is still a present key.
 
-<details>
-<summary>What's New in Version 2.1.0</summary>
+## Choosing A Consumer API
 
-### New Features
+| API                                | Emits                                                        | Best For                                               |
+| ---------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------ |
+| `consumer.recv()`                  | `Message \| null`                                            | Simple sequential workers and explicit control.        |
+| `consumer.recvBatch()`             | `Message[]`                                                  | High-throughput workers that can process chunks.       |
+| `consumer.recvStream()`            | Web `ReadableStream<Message>`                                | Pull-based Web Stream workflows.                       |
+| `consumer.recvBatchStream()`       | Web `ReadableStream<Message[]>`                              | Native batch streaming with explicit consumer control. |
+| `client.createStreamConsumer()`    | Node.js `Readable` emitting `Message`                        | Existing Node stream pipelines.                        |
+| `client.createWebStreamConsumer()` | Web `ReadableStream<Message>` or `ReadableStream<Message[]>` | v4 high-throughput stream consumers.                   |
 
-1. **Simplified Message Commits with `commitMessage()`**:
-   - New convenience method that accepts a message and commit mode directly
-   - Automatically handles `offset + 1` increment internally - no more manual offset arithmetic
-   - Available on both `KafkaConsumer` and stream consumers
-   - **Before** (v2.0.0):
-     ```javascript
-     const message = await consumer.recv()
-     await consumer.commit(message.topic, message.partition, message.offset + 1, 'Sync')
-     ```
-   - **After** (v2.1.0):
-     ```javascript
-     const message = await consumer.recv()
-     await consumer.commitMessage(message, 'Sync')
-     ```
+Two details matter for choosing correctly:
 
-2. **Enhanced OpenTelemetry Support**:
-   - Improved OTEL context propagation for better distributed tracing
-   - Safe handling when OTEL SDK is not installed (no-op behavior)
-   - Better span context management across producer and consumer operations
-   - Seamless integration with standard OTEL SDK setup
+- `createStreamConsumer()` is a Node.js `Readable` compatibility API. Even when `batchSize > 1`, it emits individual
+  `Message` objects and uses batching internally to reduce native boundary crossings.
+- `createWebStreamConsumer()` returns a discriminated object. In serial mode it emits `Message`; in batch mode it emits
+  `Message[]`. Use this API when your code can process whole batches directly.
 
-3. **CI/CD Improvements**:
-   - Updated to Node.js 24 support
-   - GitHub Actions updated to v6
-   - Improved caching with actions/cache v4
+Recommended starting points:
 
-</details>
+- Use `consumer.recv()` for straightforward workers where one handler processes one message and then commits.
+- Use `consumer.recvBatch()` when the handler can process chunks, flush work in groups, or commit after a batch.
+- Use `createStreamConsumer()` when existing code already expects a Node.js `Readable`.
+- Use `createWebStreamConsumer()` for new v4 stream code, especially when you can keep batch chunks intact.
 
-<details>
-<summary>What's New in Version 2.0.0</summary>
+## Producer API
 
-### BREAKING CHANGES ⚠️
+Create a producer from a `KafkaClient`:
 
-1. **Consumer Configuration API Changes**:
-   - **REMOVED**: `createTopic` field from `ConsumerConfiguration`
-   - **Migration**: Use `createTopic` field in `TopicPartitionConfig` instead when subscribing to topics
+```ts
+const producer = client.createProducer({
+  autoFlush: true,
+  queueTimeout: 5000,
+  configuration: {
+    'compression.type': 'lz4',
+    acks: 'all',
+  },
+})
+```
 
-2. **Stream Lifecycle Management**:
-   - Stream consumers now properly implement Node.js stream lifecycle methods (`_destroy()`)
-   - **Memory leak prevention**: Streams now automatically disconnect Kafka consumers during destruction
+Send one or more messages:
 
-3. **Async Consumer Commit**:
-   - **BREAKING**: The `consumer.commit()` method is now async and must be awaited
+```ts
+const records = await producer.send({
+  topic: 'orders',
+  messages: [
+    { key: Buffer.from('1'), payload: Buffer.from('first') },
+    { key: Buffer.from('2'), payload: Buffer.from('second') },
+  ],
+})
+```
 
-</details>
+When `autoFlush` is enabled, `send()` waits for delivery confirmation and returns `RecordMetadata[]`. When `autoFlush`
+is disabled, `send()` buffers messages and `flush()` sends pending messages:
 
----
+```ts
+const producer = client.createProducer({ autoFlush: false })
 
-## Features
+await producer.send({
+  topic: 'orders',
+  messages: [{ payload: Buffer.from('buffered') }],
+})
 
-- 🦀 Simple and intuitive API
-- 🚀 High-performance message processing
-- 🔄 Automatic reconnection handling
-- 🎯 Type-safe interfaces (TypeScript support)
-- ⚡ Async/await support
-- 🛠️ Configurable consumer and producer options
-- 📊 Stream processing support with configurable stream options
-- 📦 Message batching capabilities
-- 🔍 Comprehensive error handling
-- 📈 Performance benchmarking suite
-- 🔧 Flexible configuration system supporting complex data types
+const records = await producer.flush()
+console.log(records)
+```
 
-## Table of Contents
+`producer.inFlightCount()` returns the number of messages sent but not yet acknowledged.
 
-1. [Installation](#installation)
-2. [Quick Start](#quick-start)
-3. [Consumer Examples](#basic-consumer-setup)
-4. [Producer Examples](#producer-examples)
-5. [Stream Processing](#stream-processing)
-6. [Configuration](#configuration)
-7. [Performance Benchmarks](#performance-benchmarks)
-8. [Best Practices](#best-practices)
-9. [Contributing](#contributing)
-10. [License](#license)
-11. [OpenTelemetry Instrumentation](#opentelemetry-instrumentation)
+### Delivery Semantics
 
-## Installation
+`send()` writes records to librdkafka and, with the default `autoFlush: true`, waits for delivery results before
+resolving. Delivery metadata can still contain a per-record `error`, so production code should inspect it when failed
+records must be retried or reported:
 
-```bash
-npm install kafka-crab-js
-# or
-yarn add kafka-crab-js
-# or
-pnpm add kafka-crab-js
-```
+```ts
+const records = await producer.send({
+  topic: 'orders',
+  messages: [{ payload: Buffer.from('created') }],
+})
 
-## Quick Start
+for (const record of records) {
+  if (record.error) {
+    throw new Error(`Kafka delivery failed: ${record.error.message}`)
+  }
+}
+```
 
-### Basic Consumer Setup
+With `autoFlush: false`, `send()` returns an empty array and leaves delivery confirmation to the next `flush()` call.
+This can improve throughput for bursty producers, but shutdown code must flush before the process exits:
 
-```javascript
-import { KafkaClient } from 'kafka-crab-js'
-async function run() {
-  const kafkaClient = new KafkaClient({
-    brokers: 'localhost:29092',
-    clientId: 'foo-client',
-    logLevel: 'debug',
-    brokerAddressFamily: 'v4',
-  })
-
-  // Create consumer with topic creation control
-  const consumer = kafkaClient.createConsumer({
-    groupId: 'foo-group',
-    configuration: {
-      'auto.offset.reset': 'earliest',
-      'enable.auto.commit': false, // Use manual commit for better control
-    },
-  })
+```ts
+import type { RecordMetadata } from 'kafka-crab-js'
 
-  // Subscribe with topic creation options
-  await consumer.subscribe([
-    {
-      topic: 'foo',
-      createTopic: true,
-      numPartitions: 3,
-    },
-  ])
-
-  const message = await consumer.recv()
-  const { payload, partition, offset, topic } = message
-  console.log({
-    topic,
-    partition,
-    offset,
-    value: payload.toString(),
-  })
-
-  // Manual commit - two options:
-  // Option 1 (v2.1.0+): Simplified with commitMessage
-  await consumer.commitMessage(message, 'Sync')
+const producer = client.createProducer({ autoFlush: false })
 
-  // Option 2: Traditional commit with manual offset increment
-  // await consumer.commit(topic, partition, offset + 1, 'Sync');
+function assertAllDelivered(records: RecordMetadata[]) {
+  for (const record of records) {
+    if (record.error) {
+      throw new Error(`Kafka delivery failed: ${record.error.message}`)
+    }
+  }
+}
 
-  consumer.unsubscribe()
+try {
+  await producer.send({ topic: 'orders', messages })
+  const records = await producer.flush()
+  assertAllDelivered(records)
+} finally {
+  const remaining = await producer.flush()
+  assertAllDelivered(remaining)
 }
+```
+
+## Consumer API
+
+### Subscribe
+
+The simplest subscription accepts a topic string:
 
-await run()
+```ts
+await consumer.subscribe('orders')
 ```
 
-### Basic Producer Setup
+Use `TopicPartitionConfig[]` when you need offsets, partition-specific offsets, or topic creation:
 
-```javascript
-import { KafkaClient } from 'kafka-crab-js'
+```ts
+await consumer.subscribe([
+  {
+    topic: 'orders',
+    createTopic: true,
+    numPartitions: 3,
+    replicas: 1,
+    allOffsets: { position: 'Beginning' },
+  },
+])
+```
 
-const kafkaClient = new KafkaClient({
-  brokers: 'localhost:29092',
-  clientId: 'my-client-id',
-  logLevel: 'info',
-  brokerAddressFamily: 'v4',
-})
+For partition-specific offsets:
 
-// Producer configuration is now optional with sensible defaults
-const producer = kafkaClient.createProducer({
-  configuration: {
-    'message.timeout.ms': 5000, // Now supports number values
-    'batch.size': 16384,
-    'compression.type': 'snappy',
+```ts
+await consumer.subscribe([
+  {
+    topic: 'orders',
+    partitionOffset: [
+      {
+        partition: 0,
+        offset: { offset: 42 },
+      },
+    ],
   },
-})
+])
+```
+
+### Receive
+
+`recv()` waits for one message and returns `null` when the consumer is disconnected:
 
-const message = {
-  id: 1,
-  name: 'Sample Message',
-  timestamp: new Date().toISOString(),
+```ts
+const message = await consumer.recv()
+if (message) {
+  await handleMessage(message)
 }
+```
 
-const result = await producer.send({
-  topic: 'my-topic',
-  messages: [
-    {
-      payload: Buffer.from(JSON.stringify(message)),
-    },
-  ],
+`recvBatch(size, timeoutMs)` returns up to `size` messages:
+
+```ts
+const messages = await consumer.recvBatch(1000, 100)
+```
+
+An empty array means the timeout elapsed without a full batch. It does not mean the consumer is closed.
+
+### Commit
+
+Use `commitMessage()` for normal message processing. It commits `message.offset + 1`, which is the offset Kafka expects:
+
+```ts
+await consumer.commitMessage(message, 'Sync')
+```
+
+Use `commit()` when you already calculated the offset:
+
+```ts
+await consumer.commit(message.topic, message.partition, message.offset + 1, 'Async')
+```
+
+Commit modes are string literal values:
+
+```ts
+type CommitMode = 'Sync' | 'Async'
+```
+
+For at-least-once processing, disable auto commit, process the message successfully, then commit. If processing fails
+before the commit, Kafka can redeliver that message after restart or rebalance:
+
+```ts
+const consumer = client.createConsumer({
+  groupId: 'orders-worker',
+  enableAutoCommit: false,
 })
 
-const errors = result.map((r) => r.error).filter(Boolean)
-if (errors.length > 0) {
-  console.error('Error sending message:', errors)
-} else {
-  console.log('Message sent. Offset:', result)
+const message = await consumer.recv()
+if (message) {
+  await processOrder(message)
+  await consumer.commitMessage(message, 'Sync')
 }
 ```
 
-## Stream Processing
+`'Sync'` waits for the broker commit response. `'Async'` schedules the commit through librdkafka and returns sooner,
+which can be useful in high-throughput workers that tolerate the usual async commit tradeoff.
 
-### Enhanced Stream Consumer Example
+### Pause, Resume, Seek, And Assignment
 
-```javascript
-import { KafkaClient } from 'kafka-crab-js'
+```ts
+consumer.pause()
+consumer.resume()
 
-const kafkaClient = new KafkaClient({
-  brokers: 'localhost:29092',
-  clientId: 'my-client-id',
-  logLevel: 'info',
-  brokerAddressFamily: 'v4',
+consumer.seek('orders', 0, { position: 'Beginning' })
+consumer.seek('orders', 0, { offset: 42 })
+
+const assignment = consumer.assignment()
+const subscription = consumer.getSubscription()
+```
+
+Consumer events are available through `onEvents()`:
+
+```ts
+consumer.onEvents((error, event) => {
+  if (error) {
+    console.error(error)
+    return
+  }
+
+  console.log(event.name, event.payload)
 })
+```
 
-// Stream consumer with custom ReadableOptions (v2.0.0+)
-const kafkaStream = kafkaClient.createStreamConsumer(
-  {
-    groupId: `my-group-id`,
-    enableAutoCommit: true,
-  },
-  {
-    objectMode: true, // Default in v2.0.0+
-    highWaterMark: 1024,
-    encoding: null,
-  },
-)
+Event names are currently `PreRebalance`, `PostRebalance`, and `CommitCallback`. The payload includes the topic-partition
+list associated with the event and may include an `error` string for commit or rebalance failures.
 
-await kafkaStream.subscribe([
-  { topic: 'foo', createTopic: true },
-  { topic: 'bar', createTopic: true },
-])
+### Cleanup
+
+Always disconnect direct consumers:
+
+```ts
+try {
+  await consumer.subscribe('orders')
+  // consume...
+} finally {
+  await consumer.disconnect()
+}
+```
+
+## Stream Consumers
+
+### Node.js Readable Stream
 
-kafkaStream.on('data', (message) => {
-  console.log('>>> Message received:', {
-    payload: message.payload.toString(),
-    offset: message.offset,
-    partition: message.partition,
-    topic: message.topic,
-  })
+Use `createStreamConsumer()` when integrating with Node.js stream tooling. It emits `Message` objects.
 
-  if (message.offset > 10) {
-    kafkaStream.destroy()
+```ts
+const stream = client.createStreamConsumer({
+  groupId: 'orders-stream',
+  enableAutoCommit: false,
+  batchSize: 256,
+  batchTimeout: 50,
+  configuration: {
+    'auto.offset.reset': 'earliest',
+  },
+})
+
+await stream.subscribe('orders')
+
+stream.on('data', async (message) => {
+  try {
+    await processOrder(message.payload)
+    await stream.commitMessage(message, 'Async')
+  } catch (error) {
+    stream.destroy(error instanceof Error ? error : new Error(String(error)))
   }
 })
 
-kafkaStream.on('close', () => {
-  kafkaStream.unsubscribe()
-  console.log('Stream ended')
+stream.on('error', (error) => {
+  console.error('stream error', error)
+})
+
+stream.on('close', () => {
+  console.log('stream closed')
 })
 ```
 
-## Producer Examples
+Destroy stream consumers instead of calling `disconnect()` directly. The stream cleanup path cancels the source reader,
+unsubscribes, and disconnects the native consumer:
 
-### Batch Message Production
+```ts
+stream.destroy()
+```
 
-```javascript
-const kafkaClient = new KafkaClient({
-  brokers: 'localhost:29092',
-  clientId: 'my-client-id',
-  brokerAddressFamily: 'v4',
-})
+### Native Web Stream
 
-// Enhanced producer with flexible configuration
-const producer = kafkaClient.createProducer({
+Use `createWebStreamConsumer()` when you want a Web Stream and clear serial/batch typing.
+
+Serial mode is selected when `batchSize` is omitted, `0`, or `1`:
+
+```ts
+const webConsumer = client.createWebStreamConsumer({
+  groupId: 'orders-web-serial',
+  serialPrefetchSize: 64,
+  serialPrefetchTimeout: 1,
   configuration: {
-    'batch.size': 50000, // Number value supported
-    'linger.ms': 10, // Number value supported
-    'compression.type': 'lz4',
-    'enable.idempotence': true, // Boolean value supported
+    'auto.offset.reset': 'earliest',
   },
 })
 
-const messages = Array.from({ length: 100 }, (_, i) => ({
-  payload: Buffer.from(
-    JSON.stringify({
-      _id: i,
-      name: `Batch Message ${i}`,
-      timestamp: new Date().toISOString(),
-    }),
-  ),
-}))
+await webConsumer.consumer.subscribe('orders')
+
+const reader = webConsumer.stream.getReader()
 
 try {
-  const result = await producer.send({
-    topic: 'my-topic',
-    messages,
-  })
-  console.log('Batch sent. Offset:', result)
-  console.assert(result.length === 100)
-} catch (error) {
-  console.error('Batch error:', error)
+  const { value: message, done } = await reader.read()
+  if (!done && message) {
+    await processOrder(message.payload)
+    await webConsumer.consumer.commitMessage(message, 'Async')
+  }
+} finally {
+  await reader.cancel()
+  await webConsumer.consumer.disconnect()
 }
 ```
 
-### Producer with Keys and Headers
+Batch mode is selected with `batchSize > 1` and emits `Message[]` chunks:
 
-```javascript
-async function produceWithMetadata() {
-  const producer = kafkaClient.createProducer({
-    configuration: {
-      acks: 'all',
-      retries: 5,
-      'max.in.flight.requests.per.connection': 1,
-    },
-  })
+```ts
+const webConsumer = client.createWebStreamConsumer({
+  groupId: 'orders-web-batch',
+  batchSize: 1024,
+  batchTimeout: 10,
+  configuration: {
+    'auto.offset.reset': 'earliest',
+  },
+})
+
+if (webConsumer.mode === 'batch') {
+  await webConsumer.consumer.subscribe('orders')
+  const reader = webConsumer.stream.getReader()
 
   try {
-    await producer.send({
-      topic: 'user-events',
-      messages: [
-        {
-          key: 'user-123',
-          payload: Buffer.from(
-            JSON.stringify({
-              userId: 123,
-              action: 'update',
-            }),
-          ),
-          headers: {
-            'correlation-id': 'txn-123',
-            source: 'user-service',
-          },
-        },
-      ],
-    })
-  } catch (error) {
-    console.error('Error:', error)
+    const { value: batch, done } = await reader.read()
+    if (!done && batch) {
+      for (const message of batch) {
+        await processOrder(message.payload)
+      }
+    }
+  } finally {
+    await reader.cancel()
+    await webConsumer.consumer.disconnect()
   }
 }
 ```
 
-### Reconnecting Kafka Consumer
+## Batching, Backpressure, And Tuning
 
-```javascript
-import { KafkaClient } from 'kafka-crab-js'
+The main throughput lever is how many messages cross the native-to-JavaScript boundary per call.
 
-const kafkaClient = new KafkaClient({
-  brokers: 'localhost:29092',
-  clientId: 'reconnect-test',
-  logLevel: 'debug',
-  brokerAddressFamily: 'v4',
+| Knob                    | Applies To                                      | Effect                                                                 |
+| ----------------------- | ----------------------------------------------- | ---------------------------------------------------------------------- |
+| `recvBatch(size, ms)`   | Direct consumer                                 | Pulls up to `size` messages, waiting up to `ms` for data.              |
+| `batchSize`             | Node stream and Web Stream batch modes          | Sets the maximum native batch size used by the stream.                 |
+| `batchTimeout`          | Node stream and Web Stream batch modes          | Bounds how long a partially filled batch waits before being emitted.   |
+| `serialPrefetchSize`    | `createWebStreamConsumer()` serial mode         | Pulls small native batches and flattens them into individual messages. |
+| `serialPrefetchTimeout` | `createWebStreamConsumer()` serial mode         | Timeout for the serial-mode prefetch batch.                            |
+| `streamOptions`         | `createStreamConsumer()` Node.js `Readable` API | Lets Node stream `highWaterMark` participate in backpressure.          |
+
+Use larger batches when throughput matters and your handler can process arrays efficiently. Use smaller batches and
+shorter timeouts when tail latency matters more than total throughput. Very large batches can increase RSS because more
+payloads, keys, headers, and metadata must be retained at once.
+
+The v4 Web Stream serial path defaults to `serialPrefetchSize: 64` and `serialPrefetchTimeout: 1`. This keeps the public
+serial API message-by-message while reducing native boundary crossings. Batch mode defaults `batchTimeout` to `1000`
+milliseconds when it is omitted.
+
+For broker fetch tuning, pass librdkafka settings through `configuration`. Keep these values aligned with your expected
+message size:
+
+```ts
+const consumer = client.createConsumer({
+  groupId: 'orders-worker',
   configuration: {
-    'auto.offset.reset': 'earliest',
-    'session.timeout.ms': 30000,
-    'heartbeat.interval.ms': 10000,
+    'fetch.min.bytes': 1,
+    'fetch.wait.max.ms': 10,
+    'fetch.max.bytes': 1_048_576,
+    'max.partition.fetch.bytes': 1_048_576,
+    'message.max.bytes': 1_000_000,
   },
 })
+```
 
-/**
- * Creates and configures a new Kafka stream consumer
- */
-async function createConsumer() {
-  const kafkaStream = kafkaClient.createStreamConsumer(
-    {
-      groupId: 'reconnect-test',
-      enableAutoCommit: true,
-    },
-    {
-      highWaterMark: 100,
-      objectMode: true,
-    },
-  )
-
-  await kafkaStream.subscribe([
-    { topic: 'foo', createTopic: true },
-    { topic: 'bar', createTopic: true },
-  ])
-  return kafkaStream
-}
-
-/**
- * Starts a Kafka consumer with auto-restart capability
- */
-async function startConsumer() {
-  let counter = 0
-  let retryCount = 0
-  const MAX_RETRIES = 5
-  const RETRY_DELAY = 5000 // 5 seconds
-
-  async function handleRetry() {
-    if (retryCount < MAX_RETRIES) {
-      retryCount++
-      console.log(
-        `Attempting to restart consumer (attempt ${retryCount}/${MAX_RETRIES}) in ${RETRY_DELAY / 1000} seconds...`,
-      )
-      setTimeout(setupConsumerWithRetry, RETRY_DELAY)
-    } else {
-      console.error(`Maximum retry attempts (${MAX_RETRIES}) reached. Stopping consumer.`)
-      process.exit(1)
-    }
-  }
+`fetch.max.bytes` must be greater than or equal to `message.max.bytes`. If this invariant is broken, librdkafka rejects
+the consumer configuration before the benchmark or application starts.
 
-  async function setupConsumerWithRetry() {
-    try {
-      const kafkaStream = await createConsumer()
-      retryCount = 0 // Reset retry count on successful connection
-
-      console.log('Starting consumer')
-
-      kafkaStream.on('data', (message) => {
-        counter++
-        console.log('>>> Message received:', {
-          counter,
-          payload: message.payload.toString(),
-          offset: message.offset,
-          partition: message.partition,
-          topic: message.topic,
-        })
-      })
-
-      kafkaStream.on('error', async (error) => {
-        console.error('Stream error:', error)
-        handleRetry()
-      })
-
-      kafkaStream.on('close', () => {
-        console.log('Stream ended')
-        try {
-          kafkaStream.unsubscribe()
-        } catch (unsubError) {
-          console.error('Error unsubscribing:', unsubError)
-        }
-      })
-    } catch (error) {
-      console.error('Error setting up consumer:', error)
-      handleRetry()
-    }
-  }
+## Configuration
 
-  await setupConsumerWithRetry()
-}
+### KafkaClient
 
-await startConsumer()
+```ts
+const client = new KafkaClient({
+  brokers: 'localhost:9092',
+  clientId: 'orders-service',
+  securityProtocol: 'Plaintext',
+  logLevel: 'info',
+  brokerAddressFamily: 'v4',
+  diagnostics: true,
+  configuration: {
+    'socket.keepalive.enable': true,
+  },
+})
 ```
 
-### Examples
+| Option                | Type                                                   | Default   | Description                                                               |
+| --------------------- | ------------------------------------------------------ | --------- | ------------------------------------------------------------------------- |
+| `brokers`             | `string`                                               | required  | Comma-separated broker list, for example `localhost:9092,localhost:9093`. |
+| `clientId`            | `string`                                               | `rdkafka` | Client identifier sent to Kafka.                                          |
+| `securityProtocol`    | `'Plaintext' \| 'Ssl' \| 'SaslPlaintext' \| 'SaslSsl'` |           | Security protocol.                                                        |
+| `logLevel`            | `string`                                               | `error`   | librdkafka log level.                                                     |
+| `brokerAddressFamily` | `string`                                               | `v4`      | Address family hint such as `v4` or `any`.                                |
+| `diagnostics`         | `boolean`                                              | `true`    | Enables diagnostic-channel events used by `kafka-crab-js-otel`.           |
+| `configuration`       | `Record<string, any>`                                  |           | Additional librdkafka client settings.                                    |
 
-You can find some examples on the [example](https://github.com/flash-tecnologia/kafka-crab-js/tree/main/example) folder of this project.
+`configuration` is passed through to librdkafka after values are converted to strings. Use the original librdkafka
+property names, for example `sasl.mechanism`, `queued.min.messages`, or `fetch.wait.max.ms`.
 
-## Configuration
+Common connection examples:
 
-### KafkaConfiguration
+```ts
+const localClient = new KafkaClient({
+  brokers: 'localhost:9092',
+  clientId: 'orders-local',
+  securityProtocol: 'Plaintext',
+  brokerAddressFamily: 'v4',
+})
+```
 
-| Property              | Type                  | Default     | Description                                                         |
-| --------------------- | --------------------- | ----------- | ------------------------------------------------------------------- |
-| `brokers`             | `string`              |             | List of brokers to connect to                                       |
-| `clientId`            | `string`              | `"rdkafka"` | Client id to use for the connection                                 |
-| `securityProtocol`    | `SecurityProtocol`    |             | Security protocol to use (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL) |
-| `logLevel`            | `string`              | `info`      | Log level for the client                                            |
-| `brokerAddressFamily` | `string`              | `"v4"`      | Address family to use for the connection (v4, v6)                   |
-| `configuration`       | `Record<string, any>` | `{}`        | Additional configuration options for the client                     |
-| `diagnostics`         | `boolean`             | `false`     | **v3.0.0+**: Enable diagnostics channel for OTEL instrumentation    |
+```ts
+const saslClient = new KafkaClient({
+  brokers: process.env.KAFKA_BROKERS!,
+  clientId: 'orders-worker',
+  securityProtocol: 'SaslSsl',
+  configuration: {
+    'sasl.mechanism': 'PLAIN',
+    'sasl.username': process.env.KAFKA_USERNAME!,
+    'sasl.password': process.env.KAFKA_PASSWORD!,
+  },
+})
+```
 
 ### ConsumerConfiguration
 
-| Property               | Type                  | Default | Description                               |
-| ---------------------- | --------------------- | ------- | ----------------------------------------- |
-| `groupId`              | `string`              |         | Consumer group ID                         |
-| `enableAutoCommit`     | `boolean`             | `true`  | Enable automatic offset commits           |
-| `configuration`        | `Record<string, any>` | `{}`    | Additional consumer configuration options |
-| `fetchMetadataTimeout` | `number`              | `60000` | Timeout for fetching metadata (ms)        |
-| `maxBatchMessages`     | `number`              | `1000`  | Maximum messages in a batch operation     |
+```ts
+const consumer = client.createConsumer({
+  groupId: 'orders-worker',
+  enableAutoCommit: false,
+  fetchMetadataTimeout: 5000,
+  configuration: {
+    'auto.offset.reset': 'earliest',
+    'enable.auto.commit': false,
+    'fetch.min.bytes': 1,
+  },
+})
+```
 
-### Consumer Commit Methods
+| Option                 | Type                  | Description                                |
+| ---------------------- | --------------------- | ------------------------------------------ |
+| `groupId`              | `string`              | Kafka consumer group id.                   |
+| `enableAutoCommit`     | `boolean`             | Convenience flag for auto commit behavior. |
+| `fetchMetadataTimeout` | `number`              | Metadata fetch timeout in milliseconds.    |
+| `configuration`        | `Record<string, any>` | Additional librdkafka consumer settings.   |
 
-kafka-crab-js provides two methods for committing offsets:
+`enableAutoCommit` is a convenience option that sets librdkafka `enable.auto.commit`. You can also pass
+`'enable.auto.commit'` in `configuration`, but using both with different values makes the intent hard to read.
 
-| Method          | Signature                                | Description                                                             |
-| --------------- | ---------------------------------------- | ----------------------------------------------------------------------- |
-| `commit`        | `commit(topic, partition, offset, mode)` | Traditional commit - you must calculate `offset + 1`                    |
-| `commitMessage` | `commitMessage(message, mode)`           | **v2.1.0+**: Simplified commit - automatically handles offset increment |
+For new services that need explicit processing guarantees, start with manual commits:
 
-```javascript
-// Using commitMessage (recommended for v2.1.0+)
-const message = await consumer.recv()
-await consumer.commitMessage(message, 'Sync')
+```ts
+const consumer = client.createConsumer({
+  groupId: 'orders-worker',
+  enableAutoCommit: false,
+  configuration: {
+    'auto.offset.reset': 'earliest',
+    'enable.partition.eof': false,
+  },
+})
+```
 
-// Using commit (traditional)
-const message = await consumer.recv()
-await consumer.commit(message.topic, message.partition, message.offset + 1, 'Sync')
+### ProducerConfiguration
+
+```ts
+const producer = client.createProducer({
+  autoFlush: true,
+  queueTimeout: 5000,
+  configuration: {
+    'compression.type': 'lz4',
+  },
+})
 ```
 
-Both methods support `'Sync'` and `'Async'` commit modes.
+| Option          | Type                  | Description                                             |
+| --------------- | --------------------- | ------------------------------------------------------- |
+| `autoFlush`     | `boolean`             | When enabled, `send()` waits for delivery confirmation. |
+| `queueTimeout`  | `number`              | Queue timeout in milliseconds.                          |
+| `configuration` | `Record<string, any>` | Additional librdkafka producer settings.                |
 
-### ProducerConfiguration
+Producer settings are workload-specific. A low-latency producer might prefer smaller buffering windows, while a
+throughput-oriented producer can allow librdkafka to coalesce more work:
 
-| Property        | Type                  | Default | Description                               |
-| --------------- | --------------------- | ------- | ----------------------------------------- |
-| `queueTimeout`  | `number`              | `5000`  | Queue timeout in milliseconds             |
-| `autoFlush`     | `boolean`             | `true`  | Enable automatic message flushing         |
-| `configuration` | `Record<string, any>` | `{}`    | Additional producer configuration options |
+```ts
+const producer = client.createProducer({
+  autoFlush: false,
+  queueTimeout: 5000,
+  configuration: {
+    acks: 'all',
+    'compression.type': 'lz4',
+    'queue.buffering.max.ms': 10,
+    'queue.buffering.max.messages': 100000,
+  },
+})
+```
 
 ### TopicPartitionConfig
 
-| Property          | Type                     | Default | Description                                           |
-| ----------------- | ------------------------ | ------- | ----------------------------------------------------- |
-| `topic`           | `string`                 |         | Topic name                                            |
-| `allOffsets`      | `OffsetModel`            |         | Offset configuration for all partitions               |
-| `partitionOffset` | `Array<PartitionOffset>` |         | Per-partition offset configuration                    |
-| `createTopic`     | `boolean`                | `false` | **v2.0.0+**: Create topic if it doesn't exist         |
-| `numPartitions`   | `number`                 | `1`     | **v2.0.0+**: Number of partitions when creating topic |
-| `replicas`        | `number`                 | `1`     | **v2.0.0+**: Number of replicas when creating topic   |
+| Option            | Type                | Description                                      |
+| ----------------- | ------------------- | ------------------------------------------------ |
+| `topic`           | `string`            | Topic name.                                      |
+| `allOffsets`      | `OffsetModel`       | Offset model applied to all assigned partitions. |
+| `partitionOffset` | `PartitionOffset[]` | Partition-specific offset model.                 |
+| `createTopic`     | `boolean`           | Create the topic before subscribing.             |
+| `numPartitions`   | `number`            | Partition count when creating a topic.           |
+| `replicas`        | `number`            | Replica count when creating a topic.             |
 
-You can see the available options here: [librdkafka](https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html).
+Offset models:
 
-### OpenTelemetry Instrumentation
+```ts
+{
+  position: 'Beginning'
+}
+{
+  position: 'End'
+}
+{
+  position: 'Stored'
+}
+{
+  position: 'Invalid'
+}
+{
+  offset: 42
+}
+```
 
-> **Note:** Starting with v3.0.0, OpenTelemetry instrumentation has been moved to a separate package: [`kafka-crab-js-otel`](https://www.npmjs.com/package/kafka-crab-js-otel)
+### Type-Only Literals
 
-#### Installation
+In v4, these names are TypeScript-only exports:
+
+```ts
+import type { CommitMode, KafkaEventName, PartitionPosition, SecurityProtocol } from 'kafka-crab-js'
+```
+
+Use literal values at runtime:
+
+```ts
+const commitMode: CommitMode = 'Sync'
+const securityProtocol: SecurityProtocol = 'Plaintext'
+```
+
+## OpenTelemetry
+
+OpenTelemetry support lives in the separate `kafka-crab-js-otel` package. The core package emits diagnostic-channel
+events when `diagnostics !== false`; the OTEL package subscribes to those events.
+
+Install the instrumentation package and OpenTelemetry dependencies:
 
 ```bash
-npm install kafka-crab-js-otel @opentelemetry/api
+npm install kafka-crab-js-otel @opentelemetry/api @opentelemetry/sdk-node
 ```
 
-#### Usage
+Enable instrumentation before creating `KafkaClient`:
 
-```javascript
+```ts
 import { KafkaClient } from 'kafka-crab-js'
 import { enableOtelInstrumentation, endSpan } from 'kafka-crab-js-otel'
 
-// Enable instrumentation before creating client
-// Note: serviceName is configured via OTEL SDK Resource
 enableOtelInstrumentation({
   metrics: { enabled: true },
 })
 
-// Create client with diagnostics enabled
 const client = new KafkaClient({
   brokers: 'localhost:9092',
-  clientId: 'my-app',
+  clientId: 'orders-worker',
   diagnostics: true,
 })
 
-// For consumers, call endSpan() when done processing
+const consumer = client.createConsumer({ groupId: 'orders-worker' })
+await consumer.subscribe('orders')
+
 const message = await consumer.recv()
-// ... process message ...
-endSpan(message)
+if (message) {
+  try {
+    await processOrder(message.payload)
+  } finally {
+    endSpan(message)
+  }
+}
 ```
 
-For complete configuration options and examples, see the [kafka-crab-js-otel README](../kafka-crab-js-otel/README.md).
-
-#### Examples
-
-See comprehensive examples in the `example/` directory:
-
-- `example/otel-tracing-example.mjs` - Complete tracing setup with Jaeger
-- `example/otel-metrics-example.mjs` - Metrics collection with Prometheus
-- `example/README.md` - Full documentation for all examples
+See the [OpenTelemetry package README](../kafka-crab-js-otel/README.md) for full tracing and metrics setup.
 
 ## Performance Benchmarks
 
-### Running Benchmarks
+The repository includes a benchmark suite that compares `kafka-crab-js` with KafkaJS and `@platformatic/kafka`. The
+latest captured run in [BENCHMARKS.md](../../BENCHMARKS.md) was recorded on May 25, 2026 on a MacBook Pro with Apple M1,
+Node.js `v24.16.0`, and a local three-broker `apache/kafka:4.0.0` cluster.
 
-kafka-crab-js v2.0.0+ includes a comprehensive benchmark suite to compare performance against other popular Kafka clients:
+From the repository root:
 
 ```bash
-cd ../benchmark
+podman compose up -d
 
-# Set up benchmark environment (requires Kafka running locally)
-vp install
-vp run setup:consumer
+cd benchmarks/kafka
+
+# Set up benchmark data. Requires Kafka running locally.
+KAFKA_BROKERS=127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094 vp run setup:consumer
 
-# Run consumer performance benchmarks
-vp run benchmark
+# Run the default isolated-process memory benchmark.
+KAFKA_BROKERS=127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094 vp run benchmark
 ```
 
-### Benchmark Results
+The default benchmark runs one child Node.js process per selected scenario. This avoids carrying V8 heap pages, native
+allocator arenas, sockets, librdkafka state, and Kafka metadata from one client into the next client's measurement.
 
-_Benchmarks run on macOS with Apple M1 chip processing 50,000 messages (December 2024)_
+### Consumer Benchmark Snapshot
 
-```
-╔════════════════════════╤═════════╤══════════════════╤═══════════╤══════════════════════════╗
-║ Slower tests           │ Samples │           Result │ Tolerance │ Difference with previous ║
-╟────────────────────────┼─────────┼──────────────────┼───────────┼──────────────────────────╢
-║ kafkajs                │   50000 │    834.12 op/sec │ ±  0.22 % │                          ║
-║ node-rdkafka (evented) │   84115 │  24922.67 op/sec │ ± 74.82 % │ + 2887.91 %              ║
-║ kafka-crab-js (serial) │   50000 │  43213.86 op/sec │ ±  3.46 % │ + 73.39 %                ║
-║ node-rdkafka (stream)  │   50000 │  49805.32 op/sec │ ± 27.10 % │ + 15.25 %                ║
-╟────────────────────────┼─────────┼──────────────────┼───────────┼──────────────────────────╢
-║ Fastest test           │ Samples │           Result │ Tolerance │ Difference with previous ║
-╟────────────────────────┼─────────┼──────────────────┼───────────┼──────────────────────────╢
-║ kafka-crab-js (batch)  │   50000 │ 205985.31 op/sec │ ± 16.53 % │ + 313.58 %               ║
-╚════════════════════════╧═════════╧══════════════════╧═══════════╧══════════════════════════╝
+The captured run below is equivalent to:
+
+```bash
+cd benchmarks/kafka
+KAFKA_BROKERS=127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094 vp run benchmark
 ```
 
-The benchmark suite compares:
+The run uses the default isolated-process lifecycle-memory benchmark mode.
 
-- **kafka-crab-js (serial)**: Single message processing - **43,214 ops/sec**
-- **kafka-crab-js (batch)**: Batch message processing - **205,985 ops/sec** (fastest)
-- **node-rdkafka (evented)**: Event-based processing - **24,923 ops/sec**
-- **node-rdkafka (stream)**: Stream-based processing - **49,805 ops/sec**
-- **kafkajs**: Official KafkaJS client - **834 ops/sec**
+![Consumer benchmark snapshot](./assets/consumer-benchmark-snapshot.svg)
 
-Performance characteristics:
+#### Throughput
 
-- **52x faster than kafkajs** in serial mode, **247x faster in batch mode**
-- **High throughput**: Batch processing provides 4.8x performance improvement over serial mode
-- **Low latency**: Optimized for both single and batch message processing
-- **Memory efficient**: Lock-free data structures minimize memory overhead
-- **Concurrent processing**: Zero-contention concurrent operations
+| Rank | Scenario                            |                Result | Relative |
+| ---: | ----------------------------------- | --------------------: | -------: |
+|    1 | `kafka-crab-js v4 (stream, batch)`  | `1,167,396.71 op/sec` | `100.0%` |
+|    2 | `@platformatic/kafka`               |   `653,355.35 op/sec` |  `56.0%` |
+|    3 | `kafka-crab-js v4 (stream, serial)` |   `558,968.71 op/sec` |  `47.9%` |
+|    4 | `KafkaJS (eachBatch)`               |   `478,411.67 op/sec` |  `41.0%` |
+|    5 | `KafkaJS (eachMessage)`             |   `431,519.93 op/sec` |  `37.0%` |
+|    6 | `KafkaJS (eachMessage, concurrent)` |   `427,265.29 op/sec` |  `36.6%` |
 
-### Benchmark Configuration
+`kafka-crab-js v4 (stream, batch)` is the fastest scenario in this run. It is about `78.7%` faster than
+`@platformatic/kafka` and about `144.0%` faster than `KafkaJS (eachBatch)`. Its tolerance was `+/- 4.40%`, so the exact
+gap should still be treated as a benchmark snapshot rather than a universal constant.
 
-You can customize benchmark parameters in `benchmark/utils/definitions.ts`:
+For message-by-message consumption, `kafka-crab-js v4 (stream, serial)` is close to KafkaJS throughput while using much
+less lifecycle memory. In this run it is about `29.5%` faster than `KafkaJS (eachMessage)` and about `30.8%` faster
+than the concurrent KafkaJS `eachMessage` scenario.
 
-```typescript
-export const topic = 'benchmarks'
-export const brokers = ['localhost:9092', 'localhost:9093', 'localhost:9094']
+#### Memory And GC
 
-// Benchmark parameters can be adjusted in consumer.ts:
-const iterations = 10_000 // Number of messages to process
-const maxBytes = 200 // Maximum message size
-```
+| Scenario                            |   RSS delta |  Peak heap |    GC time | GC share | Notes                                      |
+| ----------------------------------- | ----------: | ---------: | ---------: | -------: | ------------------------------------------ |
+| `kafka-crab-js v4 (stream, serial)` |  `64.7 MiB` | `14.6 MiB` | `30.46 ms` |  `3.41%` | Lowest lifecycle RSS and heap.             |
+| `kafka-crab-js v4 (stream, batch)`  | `106.0 MiB` | `29.5 MiB` | `21.69 ms` |  `5.06%` | Best throughput, efficiency, and GC time.  |
+| `KafkaJS (eachMessage, concurrent)` | `178.3 MiB` | `72.2 MiB` | `83.77 ms` |  `7.16%` | Concurrency did not improve this workload. |
+| `KafkaJS (eachMessage)`             | `186.3 MiB` | `91.6 MiB` | `90.15 ms` |  `7.78%` | Highest peak heap and max GC pause.        |
+| `KafkaJS (eachBatch)`               | `193.4 MiB` | `83.8 MiB` | `63.05 ms` |  `6.03%` | KafkaJS batch baseline.                    |
+| `@platformatic/kafka`               | `216.0 MiB` | `87.4 MiB` | `74.27 ms` |  `9.71%` | Fastest non-crab competitor.               |
 
-## Best Practices
+Lifecycle memory includes module loading, client creation, subscription, consumption, cleanup, and retained RSS after
+cleanup. GC metrics use the narrower first-to-last-message window used for throughput.
 
-### Error Handling
+The most important memory result is efficiency, not just raw RSS. `kafka-crab-js v4 (stream, batch)` delivered
+`11016 op/sec/MiB`, the best score in the run. `kafka-crab-js v4 (stream, serial)` delivered `8635 op/sec/MiB`, ahead
+of `@platformatic/kafka` at `3025 op/sec/MiB` and `KafkaJS (eachBatch)` at `2474 op/sec/MiB`.
 
-- Always wrap async operations in try-catch blocks
-- Implement proper error logging and monitoring
-- Handle both operational and programming errors separately
+### Benchmark Interpretation
 
-### Performance
+- Use `kafka-crab-js v4 (stream, batch)` when raw throughput or memory efficiency is the priority.
+- Use `kafka-crab-js v4 (stream, serial)` when message-by-message processing needs the lowest lifecycle RSS and heap
+  pressure.
+- Compare batch scenarios with batch scenarios and message scenarios with message scenarios.
+- `@platformatic/kafka` is the strongest non-crab throughput competitor in this run, but it used more RSS, heap, and GC
+  time than both v4 scenarios.
+- KafkaJS `eachMessage` concurrency did not help this workload.
 
-- Use batch operations for high-throughput scenarios
-- Configure appropriate batch sizes and compression settings
-- Monitor and tune consumer group performance
-- Leverage the benchmark suite to optimize your specific use case
+### Benchmark Configuration
 
-### Configuration (v2.0.0+)
+Captured benchmark settings:
 
-- Use the flexible configuration system with proper data types:
-  ```javascript
-  const config = {
-    'batch.size': 16384, // number
-    'compression.type': 'snappy', // string
-    'enable.idempotence': true, // boolean
-    retries: 5, // number
-  }
-  ```
+```bash
+BENCHMARK_ITERATIONS=100000
+BENCHMARK_RUNS=5
+BENCHMARK_FETCH_MIN_BYTES=1
+BENCHMARK_FETCH_WAIT_MS=10
+BENCHMARK_MAX_BYTES=2048
+BENCHMARK_BATCH_SIZE=4096
+BENCHMARK_BATCH_TIMEOUT_MS=2
+BENCHMARK_KAFKAJS_EACH_MESSAGE_CONCURRENCY=3
+BENCHMARK_MEMORY=1
+BENCHMARK_MEMORY_SAMPLE_MS=100
+BENCHMARK_MEMORY_SETTLE_MS=100
+BENCHMARK_TOPIC=benchmarks
+BENCHMARK_PARTITIONS=3
+```
 
-### Message Processing
+See [BENCHMARKS.md](../../BENCHMARKS.md) for the latest captured run and the
+[benchmark README](../../benchmarks/kafka/README.md) for the full benchmark methodology and environment variables.
 
-- Validate message formats before processing
-- Implement proper serialization/deserialization
-- Handle message ordering when required
-- Use topic creation options for better topic management
+## Migration Notes
 
-### Stream Processing (v2.0.0+)
+### v4 Type-Only Runtime Exports
 
-- Configure appropriate `ReadableOptions` for your use case
-- Use `objectMode: true` for structured message processing
-- Set appropriate `highWaterMark` based on memory constraints
+`CommitMode`, `KafkaEventName`, `PartitionPosition`, and `SecurityProtocol` are no longer runtime exports from
+`kafka-crab-js`.
 
-## Contributing
+Before:
 
-1. Fork the repository
-2. Create your feature branch (`git checkout -b feature/amazing-feature`)
-3. Run the benchmark suite to ensure performance isn't degraded
-4. Commit your changes (`git commit -m 'Add some amazing feature'`)
-5. Push to the branch (`git push origin feature/amazing-feature`)
-6. Open a Pull Request
+```ts
+import { CommitMode, SecurityProtocol } from 'kafka-crab-js'
+```
 
-### Development Commands
+After:
 
-```bash
-# Build the project
-vp run build
+```ts
+import type { CommitMode, SecurityProtocol } from 'kafka-crab-js'
 
-# Run type checking
-vp run typecheck
+const commitMode: CommitMode = 'Sync'
+const securityProtocol: SecurityProtocol = 'Plaintext'
+```
 
-# Run linting and formatting
-vp check
+### v3 OpenTelemetry Split
 
-# Run benchmarks
-cd ../benchmark
-vp run benchmark
-```
+OpenTelemetry instrumentation moved from the core package to `kafka-crab-js-otel` in v3. This keeps the core package
+small and makes tracing/metrics opt-in.
 
-## OpenTelemetry Instrumentation
+## Troubleshooting
 
-> **Note:** Starting with v3.0.0, OpenTelemetry instrumentation has been moved to a separate package: [`kafka-crab-js-otel`](https://www.npmjs.com/package/kafka-crab-js-otel)
+### The Consumer Does Not Receive Existing Messages
 
-Kafka Crab JS offers turnkey tracing for Kafka workloads:
+Use an offset reset policy and a fresh consumer group:
 
-- **Seamless propagation** – Producer instrumentation injects `traceparent`/`tracestate` into Kafka headers while retaining any existing headers (including `Buffer` values) so downstream systems continue to see custom metadata.
-- **Consumer & stream coverage** – Standard consumers, batch consumers, and `createStreamConsumer` streams emit spans that include consumer group, topic, partition, offset, and batch size semantics.
-- **Hook-friendly spans** – Both `messageHook` and `producerHook` callbacks run inside the active span context, simplifying attribute decoration or error handling.
-- **Zero overhead when disabled** – Uses Node.js `diagnostics_channel` for near-zero cost when OTEL is not active.
+```ts
+const consumer = client.createConsumer({
+  groupId: `orders-worker-${Date.now()}`,
+  configuration: {
+    'auto.offset.reset': 'earliest',
+  },
+})
+```
 
-### Consumer span lifecycle (important)
+Or subscribe with an explicit starting position:
 
-Kafka Crab JS creates consumer `process <topic>` spans, but only your application knows when processing is complete.
+```ts
+await consumer.subscribe([{ topic: 'orders', allOffsets: { position: 'Beginning' } }])
+```
 
-- For single-message consumers, call `endSpan(message)` when you're done processing the message.
-- For batch consumers, call `endSpan(batch)` when you're done processing the batch.
-- The `endSpan()` helper is exported from `kafka-crab-js-otel`.
+### The Process Hangs On Shutdown
 
-This closes the span(s) and (when metrics are enabled) records `messaging.process.duration`.
+Direct consumers should call `disconnect()`:
 
-### Global instrumentation (singleton)
+```ts
+await consumer.disconnect()
+```
 
-`kafka-crab-js-otel` uses a process-wide OpenTelemetry instrumentation singleton.
+Node stream consumers should call `destroy()`:
 
-- Call `enableOtelInstrumentation()` **before** creating any `KafkaClient` instances.
-- Creating multiple `KafkaClient` instances shares the same instrumentation.
-- For tests, use `resetKafkaInstrumentation()` to clear the singleton between runs.
+```ts
+stream.destroy()
+```
 
-### Minimal Setup Example
+Web stream consumers should cancel the reader and disconnect the underlying consumer:
 
-When using stream consumers, the easiest pattern is calling `endSpan()` in a `finally` block:
+```ts
+await reader.cancel()
+await webConsumer.consumer.disconnect()
+```
+
+### Localhost Resolves To IPv6 But Kafka Listens On IPv4
+
+Set `brokerAddressFamily: 'v4'`:
 
 ```ts
-import { KafkaClient } from 'kafka-crab-js'
-import { enableOtelInstrumentation, endSpan } from 'kafka-crab-js-otel'
-import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'
-import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base'
-import { AsyncHooksContextManager } from '@opentelemetry/context-async-hooks'
-import { context } from '@opentelemetry/api'
-
-// 1. Set up OpenTelemetry SDK
-const provider = new NodeTracerProvider()
-provider.addSpanProcessor(new SimpleSpanProcessor(exporter))
-provider.register()
-context.setGlobalContextManager(new AsyncHooksContextManager().enable())
-
-// 2. Enable kafka-crab-js instrumentation BEFORE creating client
-enableOtelInstrumentation({
-  captureMessagePayload: true,
-  captureMessageHeaders: true,
-  producerHook: (span) => span.setAttribute('messaging.client.kind', 'producer'),
+const client = new KafkaClient({
+  brokers: 'localhost:9092',
+  brokerAddressFamily: 'v4',
 })
+```
 
-// 3. Create client with diagnostics enabled
+### Need Advanced Kafka Settings
+
+Pass librdkafka settings through `configuration`:
+
+```ts
 const client = new KafkaClient({
-  brokers: 'localhost:29092',
-  clientId: 'orders-api',
-  diagnostics: true, // Required for OTEL to receive events
+  brokers: 'localhost:9092',
+  configuration: {
+    'socket.keepalive.enable': true,
+    'metadata.max.age.ms': 300000,
+  },
 })
+```
 
-// 4. Use producer - spans are created automatically
-const producer = client.createProducer()
-await producer.send({
-  topic: 'orders',
-  messages: [
-    {
-      payload: Buffer.from(JSON.stringify({ orderId: '123' })),
-      headers: { 'custom-header': 'foo' },
-    },
-  ],
-})
+Consumer and producer configuration objects also accept their own `configuration` maps.
 
-// 5. Use stream consumer with proper cleanup
-const consumer = client.createStreamConsumer({
-  groupId: 'orders-consumer',
-  enableAutoCommit: false,
-})
+### `fetch.max.bytes` Must Be At Least `message.max.bytes`
 
-await consumer.subscribe('orders')
+librdkafka validates fetch limits at consumer creation time. If you tune fetch sizes for benchmarks or production and
+set `message.max.bytes` higher than `fetch.max.bytes`, the consumer will fail to start with an error similar to:
 
-consumer.on('data', (message) => {
-  try {
-    console.log(message.headers?.['custom-header']?.toString())
-  } finally {
-    // IMPORTANT: close the "process <topic>" span when your app finishes handling the message
-    endSpan(message)
-  }
-})
+```text
+`fetch.max.bytes` must be >= `message.max.bytes`
+```
+
+Keep the fetch caps aligned:
 
-// Proper cleanup for stream consumers - use destroy() not disconnect()
-consumer.on('close', () => console.log('Consumer closed'))
-// consumer.destroy()
+```ts
+const consumer = client.createConsumer({
+  groupId: 'orders-worker',
+  configuration: {
+    'message.max.bytes': 1_000_000,
+    'fetch.max.bytes': 1_048_576,
+    'max.partition.fetch.bytes': 1_048_576,
+  },
+})
 ```
 
-To disable OTEL, simply don't call `enableOtelInstrumentation()` or set `diagnostics: false` in the client config.
+### Benchmark Numbers Move Between Runs
 
-## License
+Kafka benchmarks are sensitive to CPU frequency, power mode, broker state, topic data already in page cache, and V8 heap
+history. The benchmark package defaults to isolated child processes for memory mode to reduce cross-scenario
+contamination, but you should still compare multiple runs and focus on large, repeatable gaps.
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+## Development
 
----
+From the repository root:
 
-<div align="center">
+```bash
+vp install
+vp run build
+vp run test
+vp run test:integration
+vp check
+```
 
-[![Built with Rust](https://img.shields.io/badge/Built%20with-Rust-orange)](https://www.rust-lang.org/)
+Useful package-local commands:
+
+```bash
+vp run --filter kafka-crab-js build
+vp run --filter kafka-crab-js test
+vp run --filter kafka-crab-js test:integration
+```
+
+Integration tests require Kafka at `localhost:9092` unless `KAFKA_BROKERS` is set.
+
+## License
 
-</div>
+MIT