kafka-crab-js 1.9.0 โ†’ 2.0.0-beta.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -10,36 +10,102 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
10
10
 
11
11
  ---
12
12
 
13
- ## What's New in Version 1.8.0
13
+ ## What's New in Version 2.0.0
14
14
 
15
- ### ๐Ÿšจ Breaking Changes:
16
- 1. **Async Consumer Commit**:
15
+ ### BREAKING CHANGES โš ๏ธ
16
+
17
+ This major version includes important breaking changes that improve API consistency and memory management:
18
+
19
+ 1. **Consumer Configuration API Changes**:
20
+ - **REMOVED**: `createTopic` field from `ConsumerConfiguration`
21
+ - **Migration**: Use `createTopic` field in `TopicPartitionConfig` instead when subscribing to topics
22
+ - **Before**: `new KafkaClient().createConsumer({ groupId: 'test', createTopic: true })`
23
+ - **After**: `consumer.subscribe([{ topic: 'my-topic', createTopic: true }])`
24
+
25
+ 2. **Stream Lifecycle Management**:
26
+ - Stream consumers now properly implement Node.js stream lifecycle methods (`_destroy()`, `_final()`)
27
+ - **Memory leak prevention**: Streams now automatically disconnect Kafka consumers during destruction
28
+ - Better resource cleanup ensures no hanging connections or memory leaks
29
+ - Error handling during stream destruction is improved with proper error propagation
30
+
31
+ 3. **Stream Error Handling**:
32
+ - Stream errors now use `destroy(error)` instead of `emit('error')` for proper stream termination
33
+ - This ensures streams are properly closed when errors occur, preventing resource leaks
34
+ - Error propagation follows Node.js stream standards more closely
35
+
36
+ 4. **Rust Configuration Simplification**:
37
+ - Removed `MAX_BATCH_TIMEOUT` constant - now uses only `DEFAULT_BATCH_TIMEOUT` (1000ms)
38
+ - Simplified timeout validation logic in Rust layer
39
+ - Invalid timeout values (< 1ms) now log warnings and use default timeout
40
+
41
+ 5. **Code Quality Improvements**:
42
+ - Fixed linting violations: short variable names and `any` type usage
43
+ - Better TypeScript typing throughout the codebase
44
+ - Removed redundant validation functions for cleaner code
45
+
46
+ ### Migration Guide
47
+
48
+ - **No API changes required** - all public interfaces remain the same
49
+ - **Stream memory management** - streams now automatically clean up resources
50
+ - **Error handling** - streams will properly terminate on errors (existing error handling code still works)
51
+ - **Performance** - simplified timeout logic may provide marginal performance improvements
52
+
53
+ ---
54
+
55
+ ## What's New in Version 1.10.0
56
+
57
+ ### Major Updates:
58
+
59
+ 1. **Performance Benchmark Suite**:
60
+ - Added comprehensive benchmark suite comparing performance against `kafkajs`, `node-rdkafka`, and `@platformatic/rdkafka`
61
+ - Benchmark scripts with automated setup and data generation
62
+ - Performance testing for both serial and batch processing modes
63
+ - Supports configurable iterations and message sizes for realistic testing scenarios
64
+
65
+ 2. **Enhanced Stream Consumer Configuration**:
66
+ - Stream consumer constructor now accepts optional `ReadableOptions` parameter
67
+ - Default `objectMode: true` for better stream processing ergonomics
68
+ - Improved batch processing configuration with proper defaults
69
+
70
+ 3. **Flexible Configuration System**:
71
+ - **BREAKING**: Configuration properties now accept `Record<string, any>` instead of `Record<string, string>`
72
+ - Enhanced type flexibility for complex configuration values (numbers, booleans, objects)
73
+ - Better support for advanced librdkafka configuration options
74
+
75
+ 4. **Enhanced Topic Management**:
76
+ - Added `createTopic` and `numPartitions` options to `TopicPartitionConfig`
77
+ - Improved topic creation control during consumer subscription
78
+ - Better handling of partition assignment and topic setup
79
+
80
+ 5. **Build System Improvements**:
81
+ - Updated TypeScript configuration with modular `tsconfig.base.json`
82
+ - Added `typecheck` script for standalone type checking
83
+ - Enhanced build pipeline with better declaration file generation
84
+ - Updated to pnpm@10.15.1 for improved dependency management
85
+
86
+ 6. **Development Tools Enhancement**:
87
+ - Enhanced oxlint configuration with benchmark-specific rules
88
+ - Added support for benchmark directory linting and formatting
89
+ - Improved development workflow with better error handling rules
90
+
91
+ ### Previous Version (1.9.0) Updates:
92
+ 7. **Consumer Topic Creation Control**:
93
+ - Added fine-grained control over consumer topic creation behavior
94
+ - Enhanced consistency in topic management and cleanup processes
95
+ - Improved handling of topic creation inconsistencies during consumer operations
96
+
97
+ 8. **Enhanced Manual Commit Testing**:
98
+ - Consolidated manual commit tests for better reliability and coverage
99
+ - Added comprehensive integration tests with PostRebalance event handling
100
+ - Tests now verify offset persistence across consumer restarts and batch processing
101
+
102
+ ### Previous Version (1.8.0) Breaking Changes:
103
+ 9. **Async Consumer Commit**:
17
104
  - **BREAKING**: The `consumer.commit()` method is now async and must be awaited
18
105
  - **Before**: `consumer.commit(topic, partition, offset, 'Sync')`
19
106
  - **After**: `await consumer.commit(topic, partition, offset, 'Sync')`
20
107
  - This change improves performance by using `spawn_blocking` for non-blocking operations
21
108
 
22
- ### Major Updates:
23
- 2. **Enhanced Manual Commit Testing**:
24
- - Added comprehensive integration tests for manual commit functionality
25
- - Tests now use PostRebalance events for proper timing instead of arbitrary timeouts
26
- - Covers both sync and async commit modes with offset verification
27
- - Tests offset persistence across consumer restarts and batch processing scenarios
28
-
29
- 3. **Improved Error Messages**:
30
- - Standardized all error messages to use consistent "Failed to [action]" format
31
- - Enhanced error clarity and specificity throughout the codebase
32
- - Fixed grammar and spelling issues in error messages
33
-
34
- 4. **Updated Dependencies**:
35
- - Upgraded all dependencies to their latest versions for improved performance and security
36
- - Updated TypeScript bindings and CommonJS exports
37
- - Synchronized pnpm-lock.yaml with new dependency versions
38
-
39
- 5. **Arc-based Consumer Architecture**:
40
- - Improved internal architecture using `Arc<StreamConsumer>` for better memory management
41
- - Enhanced thread safety for commit operations
42
-
43
109
  ---
44
110
 
45
111
  ## Features
@@ -50,9 +116,11 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
50
116
  - ๐ŸŽฏ Type-safe interfaces (TypeScript support)
51
117
  - โšก Async/await support
52
118
  - ๐Ÿ› ๏ธ Configurable consumer and producer options
53
- - ๐Ÿ“Š Stream processing support
119
+ - ๐Ÿ“Š Stream processing support with configurable stream options
54
120
  - ๐Ÿ“ฆ Message batching capabilities
55
121
  - ๐Ÿ” Comprehensive error handling
122
+ - ๐Ÿ“ˆ Performance benchmarking suite
123
+ - ๐Ÿ”ง Flexible configuration system supporting complex data types
56
124
 
57
125
  ## Table of Contents
58
126
 
@@ -62,9 +130,10 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
62
130
  4. [Producer Examples](#producer-examples)
63
131
  5. [Stream Processing](#stream-processing)
64
132
  6. [Configuration](#configuration)
65
- 7. [Best Practices](#best-practices)
66
- 8. [Contributing](#contributing)
67
- 9. [License](#license)
133
+ 7. [Performance Benchmarks](#performance-benchmarks)
134
+ 8. [Best Practices](#best-practices)
135
+ 9. [Contributing](#contributing)
136
+ 10. [License](#license)
68
137
 
69
138
  ## Installation
70
139
 
@@ -90,12 +159,21 @@ async function run() {
90
159
  brokerAddressFamily: 'v4',
91
160
  });
92
161
 
93
- // Create consumer
162
+ // Create consumer with topic creation control
94
163
  const consumer = kafkaClient.createConsumer({
95
164
  groupId: 'foo-group',
165
+ configuration: {
166
+ 'auto.offset.reset': 'earliest',
167
+ 'enable.auto.commit': false, // Use manual commit for better control
168
+ }
96
169
  });
97
170
 
98
- await consumer.subscribe([{ topic: 'foo' }]);
171
+ // Subscribe with topic creation options
172
+ await consumer.subscribe([{
173
+ topic: 'foo',
174
+ createTopic: true,
175
+ numPartitions: 3
176
+ }]);
99
177
 
100
178
  const message = await consumer.recv();
101
179
  const { payload, partition, offset, topic } = message;
@@ -110,7 +188,6 @@ async function run() {
110
188
  await consumer.commit(topic, partition, offset + 1, 'Sync');
111
189
 
112
190
  consumer.unsubscribe();
113
-
114
191
  }
115
192
 
116
193
  await run();
@@ -128,7 +205,14 @@ const kafkaClient = new KafkaClient({
128
205
  brokerAddressFamily: 'v4',
129
206
  });
130
207
 
131
- const producer = kafkaClient.createProducer({ configuration: { 'message.timeout.ms': '5000' } });
208
+ // Producer configuration is now optional with sensible defaults
209
+ const producer = kafkaClient.createProducer({
210
+ configuration: {
211
+ 'message.timeout.ms': 5000, // Now supports number values
212
+ 'batch.size': 16384,
213
+ 'compression.type': 'snappy'
214
+ }
215
+ });
132
216
 
133
217
  const message = {
134
218
  id: 1,
@@ -149,12 +233,11 @@ if (errors.length > 0) {
149
233
  } else {
150
234
  console.log('Message sent. Offset:', result);
151
235
  }
152
-
153
236
  ```
154
237
 
155
238
  ## Stream Processing
156
239
 
157
- ### Stream Consumer Example
240
+ ### Enhanced Stream Consumer Example
158
241
 
159
242
  ```javascript
160
243
  import { KafkaClient } from 'kafka-crab-js';
@@ -166,24 +249,38 @@ const kafkaClient = new KafkaClient({
166
249
  brokerAddressFamily: 'v4',
167
250
  });
168
251
 
252
+ // Stream consumer with custom ReadableOptions (v1.10.0+)
169
253
  const kafkaStream = kafkaClient.createStreamConsumer({
170
- groupId: `my-groud-id`,
254
+ groupId: `my-group-id`,
171
255
  enableAutoCommit: true,
256
+ }, {
257
+ objectMode: true, // Default in v1.10.0+
258
+ highWaterMark: 1024,
259
+ encoding: null
172
260
  });
173
261
 
174
- await kafkaStream.subscribe([{ topic: 'foo' }, { topic: 'bar' }])
262
+ await kafkaStream.subscribe([
263
+ { topic: 'foo', createTopic: true },
264
+ { topic: 'bar', createTopic: true }
265
+ ]);
175
266
 
176
267
  kafkaStream.on('data', (message) => {
177
- console.log('>>> Message received:', { payload: message.payload.toString(), offset: message.offset, partition: message.partition, topic: message.topic })
268
+ console.log('>>> Message received:', {
269
+ payload: message.payload.toString(),
270
+ offset: message.offset,
271
+ partition: message.partition,
272
+ topic: message.topic
273
+ });
274
+
178
275
  if (message.offset > 10) {
179
276
  kafkaStream.destroy();
180
277
  }
181
- })
278
+ });
182
279
 
183
280
  kafkaStream.on('close', () => {
184
281
  kafkaStream.unsubscribe();
185
- console.log('Stream ended')
186
- })
282
+ console.log('Stream ended');
283
+ });
187
284
  ```
188
285
 
189
286
  ## Producer Examples
@@ -196,7 +293,16 @@ const kafkaClient = new KafkaClient({
196
293
  clientId: 'my-client-id',
197
294
  brokerAddressFamily: 'v4',
198
295
  });
199
- const producer = kafkaClient.createProducer({});
296
+
297
+ // Enhanced producer with flexible configuration
298
+ const producer = kafkaClient.createProducer({
299
+ configuration: {
300
+ 'batch.size': 50000, // Number value supported
301
+ 'linger.ms': 10, // Number value supported
302
+ 'compression.type': 'lz4',
303
+ 'enable.idempotence': true // Boolean value supported
304
+ }
305
+ });
200
306
 
201
307
  const messages = Array.from({ length: 100 }, (_, i) => ({
202
308
  payload: Buffer.from(JSON.stringify({
@@ -222,11 +328,17 @@ try {
222
328
 
223
329
  ```javascript
224
330
  async function produceWithMetadata() {
225
- const producer = await kafkaCrab.createProducer({ config });
331
+ const producer = kafkaClient.createProducer({
332
+ configuration: {
333
+ 'acks': 'all',
334
+ 'retries': 5,
335
+ 'max.in.flight.requests.per.connection': 1
336
+ }
337
+ });
226
338
 
227
339
  try {
228
340
  await producer.send({
229
- topic,
341
+ topic: 'user-events',
230
342
  messages: [{
231
343
  key: 'user-123',
232
344
  payload: Buffer.from(JSON.stringify({
@@ -257,6 +369,8 @@ const kafkaClient = new KafkaClient({
257
369
  brokerAddressFamily: 'v4',
258
370
  configuration: {
259
371
  'auto.offset.reset': 'earliest',
372
+ 'session.timeout.ms': 30000,
373
+ 'heartbeat.interval.ms': 10000
260
374
  },
261
375
  })
262
376
 
@@ -267,10 +381,14 @@ async function createConsumer() {
267
381
  const kafkaStream = kafkaClient.createStreamConsumer({
268
382
  groupId: 'reconnect-test',
269
383
  enableAutoCommit: true,
384
+ }, {
385
+ highWaterMark: 100,
386
+ objectMode: true
270
387
  })
388
+
271
389
  await kafkaStream.subscribe([
272
- { topic: 'foo' },
273
- { topic: 'bar' },
390
+ { topic: 'foo', createTopic: true },
391
+ { topic: 'bar', createTopic: true },
274
392
  ])
275
393
  return kafkaStream
276
394
  }
@@ -342,7 +460,7 @@ await startConsumer()
342
460
 
343
461
  ### Examples
344
462
 
345
- You can find some examples on the [example](https://github.com/mthssdrbrg/kafka-crab-js/tree/main/example) folder of this project.
463
+ You can find some examples on the [example](https://github.com/flash-tecnologia/kafka-crab-js/tree/main/example) folder of this project.
346
464
 
347
465
  ## Configuration
348
466
 
@@ -355,13 +473,100 @@ You can find some examples on the [example](https://github.com/mthssdrbrg/kafka-
355
473
  | `brokers` | `string` || List of brokers to connect to |
356
474
  | `clientId` | `string` || Client id to use for the connection |
357
475
  | `securityProtocol` | `SecurityProtocol` || Security protocol to use (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL) |
358
- | `logLevel` | `string` | `info` | Client id to use for the connection |
476
+ | `logLevel` | `string` | `info` | Log level for the client |
359
477
  | `brokerAddressFamily` | `string` | `"v4"` | Address family to use for the connection (v4, v6) |
360
- | `configuration` | `Map<string, string>` | `{}` | Additional configuration options for the client. See [librdkafka](https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html) |
478
+ | `configuration` | `Record<string, any>` | `{}` | Additional configuration options for the client. **v1.10.0+**: Now supports any value type (string, number, boolean, object) |
479
+
480
+ ### ConsumerConfiguration
481
+
482
+ | Property | Type | Default | Description |
483
+ | --- | --- | --- | --- |
484
+ | `groupId` | `string` || Consumer group ID |
485
+ | `enableAutoCommit` | `boolean` | `true` | Enable automatic offset commits |
486
+ | `configuration` | `Record<string, any>` | `{}` | Additional consumer configuration options |
487
+ | `fetchMetadataTimeout` | `number` | `60000` | Timeout for fetching metadata (ms) |
488
+ | `maxBatchMessages` | `number` | `1000` | Maximum messages in a batch operation |
489
+
490
+ ### ProducerConfiguration
491
+
492
+ | Property | Type | Default | Description |
493
+ | --- | --- | --- | --- |
494
+ | `queueTimeout` | `number` | `5000` | Queue timeout in milliseconds |
495
+ | `autoFlush` | `boolean` | `true` | Enable automatic message flushing |
496
+ | `configuration` | `Record<string, any>` | `{}` | Additional producer configuration options |
497
+
498
+ ### TopicPartitionConfig
499
+
500
+ | Property | Type | Default | Description |
501
+ | --- | --- | --- | --- |
502
+ | `topic` | `string` || Topic name |
503
+ | `allOffsets` | `OffsetModel` || Offset configuration for all partitions |
504
+ | `partitionOffset` | `Array<PartitionOffset>` || Per-partition offset configuration |
505
+ | `createTopic` | `boolean` | `false` | **v1.10.0+**: Create topic if it doesn't exist |
506
+ | `numPartitions` | `number` | `1` | **v1.10.0+**: Number of partitions when creating topic |
361
507
 
362
508
  You can see the available options here: [librdkafka](https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html).
363
509
 
510
+ ## Performance Benchmarks
511
+
512
+ ### Running Benchmarks
513
+
514
+ kafka-crab-js v1.10.0+ includes a comprehensive benchmark suite to compare performance against other popular Kafka clients:
515
+
516
+ ```bash
517
+ # Set up benchmark environment (requires Kafka running locally)
518
+ pnpm add -D tsx # For running TypeScript files directly
519
+ npx tsx benchmark/utils/setup.ts
520
+
521
+ # Run consumer performance benchmarks
522
+ node benchmark/consumer.ts
523
+ ```
524
+
525
+ ### Benchmark Results
526
+
527
+ *Benchmarks run on macOS with Apple M1 chip processing 50,000 messages*
528
+
529
+ ```
530
+ โ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•คโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•คโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•คโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•คโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—
531
+ โ•‘ Slower tests โ”‚ Samples โ”‚ Result โ”‚ Tolerance โ”‚ Difference with previous โ•‘
532
+ โ•Ÿโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ข
533
+ โ•‘ kafkajs โ”‚ 50000 โ”‚ 2316.67 op/sec โ”‚ ยฑ 1.12 % โ”‚ โ•‘
534
+ โ•‘ kafka-crab-js (serial) โ”‚ 50000 โ”‚ 38414.57 op/sec โ”‚ ยฑ 2.97 % โ”‚ + 1558.18 % โ•‘
535
+ โ•‘ node-rdkafka (stream) โ”‚ 50000 โ”‚ 43681.44 op/sec โ”‚ ยฑ 13.98 % โ”‚ + 13.71 % โ•‘
536
+ โ•‘ node-rdkafka (evented) โ”‚ 53841 โ”‚ 74137.97 op/sec โ”‚ ยฑ 69.30 % โ”‚ + 69.72 % โ•‘
537
+ โ•Ÿโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ข
538
+ โ•‘ Fastest test โ”‚ Samples โ”‚ Result โ”‚ Tolerance โ”‚ Difference with previous โ•‘
539
+ โ•Ÿโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ข
540
+ โ•‘ kafka-crab-js (batch) โ”‚ 50000 โ”‚ 153533.53 op/sec โ”‚ ยฑ 15.98 % โ”‚ + 107.09 % โ•‘
541
+ โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•งโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•งโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•งโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•งโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
542
+ ```
364
543
 
544
+ The benchmark suite compares:
545
+ - **kafka-crab-js (serial)**: Single message processing - **38,415 ops/sec**
546
+ - **kafka-crab-js (batch)**: Batch message processing - **153,534 ops/sec** (fastest)
547
+ - **node-rdkafka (evented)**: Event-based processing - **74,138 ops/sec**
548
+ - **node-rdkafka (stream)**: Stream-based processing - **43,681 ops/sec**
549
+ - **kafkajs**: Official KafkaJS client - **2,317 ops/sec**
550
+
551
+ Performance characteristics:
552
+ - **17x faster than kafkajs** in serial mode, **66x faster in batch mode**
553
+ - **High throughput**: Batch processing provides 4x performance improvement over serial mode
554
+ - **Low latency**: Optimized for both single and batch message processing
555
+ - **Memory efficient**: Lock-free data structures minimize memory overhead
556
+ - **Concurrent processing**: Zero-contention concurrent operations
557
+
558
+ ### Benchmark Configuration
559
+
560
+ You can customize benchmark parameters in `benchmark/utils/definitions.ts`:
561
+
562
+ ```typescript
563
+ export const topic = 'benchmarks'
564
+ export const brokers = ['localhost:9092', 'localhost:9093', 'localhost:9094']
565
+
566
+ // Benchmark parameters can be adjusted in consumer.ts:
567
+ const iterations = 10_000 // Number of messages to process
568
+ const maxBytes = 200 // Maximum message size
569
+ ```
365
570
 
366
571
  ## Best Practices
367
572
 
@@ -372,21 +577,60 @@ You can see the available options here: [librdkafka](https://docs.confluent.io/p
372
577
 
373
578
  ### Performance
374
579
  - Use batch operations for high-throughput scenarios
375
- - Configure appropriate batch sizes and compression
580
+ - Configure appropriate batch sizes and compression settings
376
581
  - Monitor and tune consumer group performance
582
+ - Leverage the benchmark suite to optimize your specific use case
583
+
584
+ ### Configuration (v1.10.0+)
585
+ - Use the flexible configuration system with proper data types:
586
+ ```javascript
587
+ const config = {
588
+ 'batch.size': 16384, // number
589
+ 'compression.type': 'snappy', // string
590
+ 'enable.idempotence': true, // boolean
591
+ 'retries': 5 // number
592
+ }
593
+ ```
377
594
 
378
595
  ### Message Processing
379
596
  - Validate message formats before processing
380
597
  - Implement proper serialization/deserialization
381
598
  - Handle message ordering when required
599
+ - Use topic creation options for better topic management
600
+
601
+ ### Stream Processing (v1.10.0+)
602
+ - Configure appropriate `ReadableOptions` for your use case
603
+ - Use `objectMode: true` for structured message processing
604
+ - Set appropriate `highWaterMark` based on memory constraints
382
605
 
383
606
  ## Contributing
384
607
 
385
608
  1. Fork the repository
386
609
  2. Create your feature branch (`git checkout -b feature/amazing-feature`)
387
- 3. Commit your changes (`git commit -m 'Add some amazing feature'`)
388
- 4. Push to the branch (`git push origin feature/amazing-feature`)
389
- 5. Open a Pull Request
610
+ 3. Run the benchmark suite to ensure performance isn't degraded
611
+ 4. Commit your changes (`git commit -m 'Add some amazing feature'`)
612
+ 5. Push to the branch (`git push origin feature/amazing-feature`)
613
+ 6. Open a Pull Request
614
+
615
+ ### Development Commands
616
+
617
+ ```bash
618
+ # Build the project
619
+ pnpm build
620
+
621
+ # Run type checking
622
+ pnpm typecheck
623
+
624
+ # Run linting
625
+ pnpm lint
626
+
627
+ # Format code
628
+ pnpm fmt
629
+
630
+ # Run benchmarks
631
+ npx tsx benchmark/utils/setup.ts
632
+ npx tsx benchmark/consumer.ts
633
+ ```
390
634
 
391
635
  ## License
392
636
 
@@ -398,4 +642,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
398
642
 
399
643
  [![Built with Rust](https://img.shields.io/badge/Built%20with-Rust-orange)](https://www.rust-lang.org/)
400
644
 
401
- </div>
645
+ </div>