kafka-crab-js 1.9.1 → 2.0.0

package/README.md

@@ -10,32 +10,97 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
 
 ---
 
-## What's New in Version 1.9.0
+## What's New in Version 2.0.0
+
+### BREAKING CHANGES ⚠️
+
+This major version includes important breaking changes that improve API consistency and memory management:
+
+1. **Consumer Configuration API Changes**:
+   - **REMOVED**: `createTopic` field from `ConsumerConfiguration`
+   - **Migration**: Use `createTopic` field in `TopicPartitionConfig` instead when subscribing to topics
+   - **Before**: `new KafkaClient().createConsumer({ groupId: 'test', createTopic: true })`
+   - **After**: `consumer.subscribe([{ topic: 'my-topic', createTopic: true }])`
+
+2. **Stream Lifecycle Management**:
+   - Stream consumers now properly implement Node.js stream lifecycle methods (`_destroy()`, `_final()`)
+   - **Memory leak prevention**: Streams now automatically disconnect Kafka consumers during destruction
+   - Better resource cleanup ensures no hanging connections or memory leaks
+   - Error handling during stream destruction is improved with proper error propagation
+
+3. **Stream Error Handling**:
+   - Stream errors now use `destroy(error)` instead of `emit('error')` for proper stream termination
+   - This ensures streams are properly closed when errors occur, preventing resource leaks
+   - Error propagation follows Node.js stream standards more closely
+
+4. **Rust Configuration Simplification**:
+   - Removed `MAX_BATCH_TIMEOUT` constant - now uses only `DEFAULT_BATCH_TIMEOUT` (1000ms)
+   - Simplified timeout validation logic in Rust layer
+   - Invalid timeout values (< 1ms) now log warnings and use default timeout
+
+5. **Code Quality Improvements**:
+   - Fixed linting violations: short variable names and `any` type usage
+   - Better TypeScript typing throughout the codebase
+   - Removed redundant validation functions for cleaner code
+
+### Migration Guide
+
+- **No API changes required** - all public interfaces remain the same
+- **Stream memory management** - streams now automatically clean up resources
+- **Error handling** - streams will properly terminate on errors (existing error handling code still works)
+- **Performance** - simplified timeout logic may provide marginal performance improvements
+
+---
+
+## What's New in Version 2.0.0
 
 ### Major Updates:
-1. **Consumer Topic Creation Control**:
+
+1. **Performance Benchmark Suite**:
+   - Added comprehensive benchmark suite comparing performance against `kafkajs`, `node-rdkafka`, and `@platformatic/rdkafka`
+   - Benchmark scripts with automated setup and data generation
+   - Performance testing for both serial and batch processing modes
+   - Supports configurable iterations and message sizes for realistic testing scenarios
+
+2. **Enhanced Stream Consumer Configuration**:
+   - Stream consumer constructor now accepts optional `ReadableOptions` parameter
+   - Default `objectMode: true` for better stream processing ergonomics
+   - Improved batch processing configuration with proper defaults
+
+3. **Flexible Configuration System**:
+   - **BREAKING**: Configuration properties now accept `Record<string, any>` instead of `Record<string, string>`
+   - Enhanced type flexibility for complex configuration values (numbers, booleans, objects)
+   - Better support for advanced librdkafka configuration options
+
+4. **Enhanced Topic Management**:
+   - Added `createTopic`, `numPartitions`, and `replicas` options to `TopicPartitionConfig`
+   - Improved topic creation control during consumer subscription
+   - Better handling of partition assignment and topic setup
+
+5. **Build System Improvements**:
+   - Updated TypeScript configuration with modular `tsconfig.base.json`
+   - Added `typecheck` script for standalone type checking
+   - Enhanced build pipeline with better declaration file generation
+   - Updated to pnpm@10.15.1 for improved dependency management
+
+6. **Development Tools Enhancement**:
+   - Enhanced oxlint configuration with benchmark-specific rules
+   - Added support for benchmark directory linting and formatting
+   - Improved development workflow with better error handling rules
+
+### Previous Version (1.9.0) Updates:
+7. **Consumer Topic Creation Control**:
    - Added fine-grained control over consumer topic creation behavior
    - Enhanced consistency in topic management and cleanup processes
    - Improved handling of topic creation inconsistencies during consumer operations
 
-2. **Enhanced Manual Commit Testing**:
+8. **Enhanced Manual Commit Testing**:
    - Consolidated manual commit tests for better reliability and coverage
    - Added comprehensive integration tests with PostRebalance event handling
    - Tests now verify offset persistence across consumer restarts and batch processing
-   - Improved timing and synchronization in integration test scenarios
-
-3. **Optional Producer Configuration**:
-   - Producer configuration is now optional with sensible defaults
-   - Enhanced commit logging for better debugging and monitoring
-   - Improved flexibility in producer setup and configuration
-
-4. **Build System Improvements**:
-   - Added Cargo.lock tracking for reproducible builds across environments
-   - Enhanced dependency management and version consistency
-   - Updated TypeScript bindings and CommonJS exports
 
 ### Previous Version (1.8.0) Breaking Changes:
-5. **Async Consumer Commit**:
+9. **Async Consumer Commit**:
    - **BREAKING**: The `consumer.commit()` method is now async and must be awaited
    - **Before**: `consumer.commit(topic, partition, offset, 'Sync')`
    - **After**: `await consumer.commit(topic, partition, offset, 'Sync')`
@@ -51,9 +116,11 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
 - 🎯 Type-safe interfaces (TypeScript support)
 - ⚡ Async/await support
 - 🛠️ Configurable consumer and producer options
-- 📊 Stream processing support
+- 📊 Stream processing support with configurable stream options
 - 📦 Message batching capabilities
 - 🔍 Comprehensive error handling
+- 📈 Performance benchmarking suite
+- 🔧 Flexible configuration system supporting complex data types
 
 ## Table of Contents
 
@@ -63,9 +130,10 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
 4. [Producer Examples](#producer-examples)
 5. [Stream Processing](#stream-processing)
 6. [Configuration](#configuration)
-7. [Best Practices](#best-practices)
-8. [Contributing](#contributing)
-9. [License](#license)
+7. [Performance Benchmarks](#performance-benchmarks)
+8. [Best Practices](#best-practices)
+9. [Contributing](#contributing)
+10. [License](#license)
 
 ## Installation
 
@@ -91,12 +159,21 @@ async function run() {
     brokerAddressFamily: 'v4',
   });
 
-  // Create consumer
+  // Create consumer with topic creation control
   const consumer = kafkaClient.createConsumer({
     groupId: 'foo-group',
+    configuration: {
+      'auto.offset.reset': 'earliest',
+      'enable.auto.commit': false, // Use manual commit for better control
+    }
   });
 
-  await consumer.subscribe([{ topic: 'foo' }]);
+  // Subscribe with topic creation options
+  await consumer.subscribe([{
+    topic: 'foo',
+    createTopic: true,
+    numPartitions: 3
+  }]);
 
   const message = await consumer.recv();
   const { payload, partition, offset, topic } = message;
@@ -111,7 +188,6 @@ async function run() {
   await consumer.commit(topic, partition, offset + 1, 'Sync');
 
   consumer.unsubscribe();
-
 }
 
 await run();
@@ -129,7 +205,14 @@ const kafkaClient = new KafkaClient({
   brokerAddressFamily: 'v4',
 });
 
-const producer = kafkaClient.createProducer({ configuration: { 'message.timeout.ms': '5000' } });
+// Producer configuration is now optional with sensible defaults
+const producer = kafkaClient.createProducer({
+  configuration: {
+    'message.timeout.ms': 5000,  // Now supports number values
+    'batch.size': 16384,
+    'compression.type': 'snappy'
+  }
+});
 
 const message = {
   id: 1,
@@ -150,12 +233,11 @@ if (errors.length > 0) {
 } else {
   console.log('Message sent. Offset:', result);
 }
-
 ```
 
 ## Stream Processing
 
-### Stream Consumer Example
+### Enhanced Stream Consumer Example
 
 ```javascript
 import { KafkaClient } from 'kafka-crab-js';
@@ -167,24 +249,38 @@ const kafkaClient = new KafkaClient({
   brokerAddressFamily: 'v4',
 });
 
+// Stream consumer with custom ReadableOptions (v1.10.0+)
 const kafkaStream = kafkaClient.createStreamConsumer({
-  groupId: `my-groud-id`,
+  groupId: `my-group-id`,
   enableAutoCommit: true,
+}, {
+  objectMode: true,  // Default in v1.10.0+
+  highWaterMark: 1024,
+  encoding: null
 });
 
-await kafkaStream.subscribe([{ topic: 'foo' }, { topic: 'bar' }])
+await kafkaStream.subscribe([
+  { topic: 'foo', createTopic: true },
+  { topic: 'bar', createTopic: true }
+]);
 
 kafkaStream.on('data', (message) => {
-  console.log('>>> Message received:', { payload: message.payload.toString(), offset: message.offset, partition: message.partition, topic: message.topic })
+  console.log('>>> Message received:', {
+    payload: message.payload.toString(),
+    offset: message.offset,
+    partition: message.partition,
+    topic: message.topic
+  });
+
   if (message.offset > 10) {
     kafkaStream.destroy();
   }
-})
+});
 
 kafkaStream.on('close', () => {
   kafkaStream.unsubscribe();
-  console.log('Stream ended')
-})
+  console.log('Stream ended');
+});
 ```
 
 ## Producer Examples
@@ -197,7 +293,16 @@ const kafkaClient = new KafkaClient({
   clientId: 'my-client-id',
   brokerAddressFamily: 'v4',
 });
-const producer = kafkaClient.createProducer({});
+
+// Enhanced producer with flexible configuration
+const producer = kafkaClient.createProducer({
+  configuration: {
+    'batch.size': 50000,      // Number value supported
+    'linger.ms': 10,          // Number value supported
+    'compression.type': 'lz4',
+    'enable.idempotence': true  // Boolean value supported
+  }
+});
 
 const messages = Array.from({ length: 100 }, (_, i) => ({
   payload: Buffer.from(JSON.stringify({
@@ -223,11 +328,17 @@ try {
 
 ```javascript
 async function produceWithMetadata() {
-  const producer = await kafkaCrab.createProducer({ config });
+  const producer = kafkaClient.createProducer({
+    configuration: {
+      'acks': 'all',
+      'retries': 5,
+      'max.in.flight.requests.per.connection': 1
+    }
+  });
 
   try {
     await producer.send({
-      topic,
+      topic: 'user-events',
       messages: [{
         key: 'user-123',
         payload: Buffer.from(JSON.stringify({
@@ -258,6 +369,8 @@ const kafkaClient = new KafkaClient({
   brokerAddressFamily: 'v4',
   configuration: {
     'auto.offset.reset': 'earliest',
+    'session.timeout.ms': 30000,
+    'heartbeat.interval.ms': 10000
   },
 })
 
@@ -268,10 +381,14 @@ async function createConsumer() {
   const kafkaStream = kafkaClient.createStreamConsumer({
     groupId: 'reconnect-test',
     enableAutoCommit: true,
+  }, {
+    highWaterMark: 100,
+    objectMode: true
   })
+
   await kafkaStream.subscribe([
-    { topic: 'foo' },
-    { topic: 'bar' },
+    { topic: 'foo', createTopic: true },
+    { topic: 'bar', createTopic: true },
   ])
   return kafkaStream
 }
@@ -343,7 +460,7 @@ await startConsumer()
 
 ### Examples
 
-You can find some examples on the [example](https://github.com/mthssdrbrg/kafka-crab-js/tree/main/example) folder of this project.
+You can find some examples on the [example](https://github.com/flash-tecnologia/kafka-crab-js/tree/main/example) folder of this project.
 
 ## Configuration
 
@@ -356,13 +473,101 @@ You can find some examples on the [example](https://github.com/mthssdrbrg/kafka-
 | `brokers` | `string` || List of brokers to connect to |
 | `clientId` | `string` || Client id to use for the connection |
 | `securityProtocol` | `SecurityProtocol` || Security protocol to use (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL) |
-| `logLevel` | `string` | `info`  | Client id to use for the connection |
+| `logLevel` | `string` | `info`  | Log level for the client |
 | `brokerAddressFamily` | `string` | `"v4"` | Address family to use for the connection (v4, v6) |
-| `configuration` | `Map<string, string>` | `{}` | Additional configuration options for the client. See [librdkafka](https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html) |
+| `configuration` | `Record<string, any>` | `{}` | Additional configuration options for the client. **v1.10.0+**: Now supports any value type (string, number, boolean, object) |
+
+### ConsumerConfiguration
+
+| Property | Type | Default | Description |
+| --- | --- | --- | --- |
+| `groupId` | `string` || Consumer group ID |
+| `enableAutoCommit` | `boolean` | `true` | Enable automatic offset commits |
+| `configuration` | `Record<string, any>` | `{}` | Additional consumer configuration options |
+| `fetchMetadataTimeout` | `number` | `60000` | Timeout for fetching metadata (ms) |
+| `maxBatchMessages` | `number` | `1000` | Maximum messages in a batch operation |
+
+### ProducerConfiguration
+
+| Property | Type | Default | Description |
+| --- | --- | --- | --- |
+| `queueTimeout` | `number` | `5000` | Queue timeout in milliseconds |
+| `autoFlush` | `boolean` | `true` | Enable automatic message flushing |
+| `configuration` | `Record<string, any>` | `{}` | Additional producer configuration options |
+
+### TopicPartitionConfig
+
+| Property | Type | Default | Description |
+| --- | --- | --- | --- |
+| `topic` | `string` || Topic name |
+| `allOffsets` | `OffsetModel` || Offset configuration for all partitions |
+| `partitionOffset` | `Array<PartitionOffset>` || Per-partition offset configuration |
+| `createTopic` | `boolean` | `false` | **v2.0.x+**: Create topic if it doesn't exist |
+| `numPartitions` | `number` | `1` | **v2.0.x+**: Number of partitions when creating topic |
+| `replicas` | `number` | `1` | **v2.0.x+**: Number of replicas when creating topic |
 
 You can see the available options here: [librdkafka](https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html).
 
+## Performance Benchmarks
+
+### Running Benchmarks
+
+kafka-crab-js v1.10.0+ includes a comprehensive benchmark suite to compare performance against other popular Kafka clients:
+
+```bash
+# Set up benchmark environment (requires Kafka running locally)
+pnpm add -D tsx  # For running TypeScript files directly
+npx tsx benchmark/utils/setup.ts
+
+# Run consumer performance benchmarks
+node benchmark/consumer.ts
+```
+
+### Benchmark Results
+
+*Benchmarks run on macOS with Apple M1 chip processing 50,000 messages*
+
+```
+╔════════════════════════╤═════════╤══════════════════╤═══════════╤══════════════════════════╗
+║ Slower tests           │ Samples │           Result │ Tolerance │ Difference with previous ║
+╟────────────────────────┼─────────┼──────────────────┼───────────┼──────────────────────────╢
+║ kafkajs                │   50000 │   2316.67 op/sec │ ±  1.12 % │                          ║
+║ kafka-crab-js (serial) │   50000 │  38414.57 op/sec │ ±  2.97 % │ + 1558.18 %              ║
+║ node-rdkafka (stream)  │   50000 │  43681.44 op/sec │ ± 13.98 % │ + 13.71 %                ║
+║ node-rdkafka (evented) │   53841 │  74137.97 op/sec │ ± 69.30 % │ + 69.72 %                ║
+╟────────────────────────┼─────────┼──────────────────┼───────────┼──────────────────────────╢
+║ Fastest test           │ Samples │           Result │ Tolerance │ Difference with previous ║
+╟────────────────────────┼─────────┼──────────────────┼───────────┼──────────────────────────╢
+║ kafka-crab-js (batch)  │   50000 │ 153533.53 op/sec │ ± 15.98 % │ + 107.09 %               ║
+╚════════════════════════╧═════════╧══════════════════╧═══════════╧══════════════════════════╝
+```
+
+The benchmark suite compares:
+- **kafka-crab-js (serial)**: Single message processing - **38,415 ops/sec**
+- **kafka-crab-js (batch)**: Batch message processing - **153,534 ops/sec** (fastest)
+- **node-rdkafka (evented)**: Event-based processing - **74,138 ops/sec**
+- **node-rdkafka (stream)**: Stream-based processing - **43,681 ops/sec**
+- **kafkajs**: Official KafkaJS client - **2,317 ops/sec**
+
+Performance characteristics:
+- **17x faster than kafkajs** in serial mode, **66x faster in batch mode**
+- **High throughput**: Batch processing provides 4x performance improvement over serial mode
+- **Low latency**: Optimized for both single and batch message processing
+- **Memory efficient**: Lock-free data structures minimize memory overhead
+- **Concurrent processing**: Zero-contention concurrent operations
+
+### Benchmark Configuration
 
+You can customize benchmark parameters in `benchmark/utils/definitions.ts`:
+
+```typescript
+export const topic = 'benchmarks'
+export const brokers = ['localhost:9092', 'localhost:9093', 'localhost:9094']
+
+// Benchmark parameters can be adjusted in consumer.ts:
+const iterations = 10_000  // Number of messages to process
+const maxBytes = 200       // Maximum message size
+```
 
 ## Best Practices
 
@@ -373,21 +578,60 @@ You can see the available options here: [librdkafka](https://docs.confluent.io/p
 
 ### Performance
 - Use batch operations for high-throughput scenarios
-- Configure appropriate batch sizes and compression
+- Configure appropriate batch sizes and compression settings
 - Monitor and tune consumer group performance
+- Leverage the benchmark suite to optimize your specific use case
+
+### Configuration (v1.10.0+)
+- Use the flexible configuration system with proper data types:
+  ```javascript
+  const config = {
+    'batch.size': 16384,           // number
+    'compression.type': 'snappy',  // string
+    'enable.idempotence': true,    // boolean
+    'retries': 5                   // number
+  }
+  ```
 
 ### Message Processing
 - Validate message formats before processing
 - Implement proper serialization/deserialization
 - Handle message ordering when required
+- Use topic creation options for better topic management
+
+### Stream Processing (v1.10.0+)
+- Configure appropriate `ReadableOptions` for your use case
+- Use `objectMode: true` for structured message processing
+- Set appropriate `highWaterMark` based on memory constraints
 
 ## Contributing
 
 1. Fork the repository
 2. Create your feature branch (`git checkout -b feature/amazing-feature`)
-3. Commit your changes (`git commit -m 'Add some amazing feature'`)
-4. Push to the branch (`git push origin feature/amazing-feature`)
-5. Open a Pull Request
+3. Run the benchmark suite to ensure performance isn't degraded
+4. Commit your changes (`git commit -m 'Add some amazing feature'`)
+5. Push to the branch (`git push origin feature/amazing-feature`)
+6. Open a Pull Request
+
+### Development Commands
+
+```bash
+# Build the project
+pnpm build
+
+# Run type checking
+pnpm typecheck
+
+# Run linting
+pnpm lint
+
+# Format code
+pnpm fmt
+
+# Run benchmarks
+npx tsx benchmark/utils/setup.ts
+npx tsx benchmark/consumer.ts
+```
 
 ## License