kafka-crab-js 1.9.0 โ 2.0.0-beta.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +297 -53
- package/dist/index.cjs +109 -83
- package/dist/index.cjs.map +3 -3
- package/dist/index.d.ts +4 -1
- package/dist/index.js +107 -83
- package/dist/index.js.map +3 -3
- package/dist/kafka-client.d.ts +12 -5
- package/dist/streams/base-kafka-stream-readable.d.ts +58 -0
- package/dist/streams/kafka-batch-stream-readable.d.ts +31 -0
- package/dist/streams/kafka-stream-readable.d.ts +12 -0
- package/js-binding.cjs +46 -46
- package/js-binding.d.ts +10 -9
- package/js-binding.js +46 -46
- package/package.json +11 -10
- package/dist/kafka-stream-readable.d.ts +0 -76
package/README.md
CHANGED
|
@@ -10,36 +10,102 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
|
|
|
10
10
|
|
|
11
11
|
---
|
|
12
12
|
|
|
13
|
-
## What's New in Version
|
|
13
|
+
## What's New in Version 2.0.0
|
|
14
14
|
|
|
15
|
-
###
|
|
16
|
-
|
|
15
|
+
### BREAKING CHANGES โ ๏ธ
|
|
16
|
+
|
|
17
|
+
This major version includes important breaking changes that improve API consistency and memory management:
|
|
18
|
+
|
|
19
|
+
1. **Consumer Configuration API Changes**:
|
|
20
|
+
- **REMOVED**: `createTopic` field from `ConsumerConfiguration`
|
|
21
|
+
- **Migration**: Use `createTopic` field in `TopicPartitionConfig` instead when subscribing to topics
|
|
22
|
+
- **Before**: `new KafkaClient().createConsumer({ groupId: 'test', createTopic: true })`
|
|
23
|
+
- **After**: `consumer.subscribe([{ topic: 'my-topic', createTopic: true }])`
|
|
24
|
+
|
|
25
|
+
2. **Stream Lifecycle Management**:
|
|
26
|
+
- Stream consumers now properly implement Node.js stream lifecycle methods (`_destroy()`, `_final()`)
|
|
27
|
+
- **Memory leak prevention**: Streams now automatically disconnect Kafka consumers during destruction
|
|
28
|
+
- Better resource cleanup ensures no hanging connections or memory leaks
|
|
29
|
+
- Error handling during stream destruction is improved with proper error propagation
|
|
30
|
+
|
|
31
|
+
3. **Stream Error Handling**:
|
|
32
|
+
- Stream errors now use `destroy(error)` instead of `emit('error')` for proper stream termination
|
|
33
|
+
- This ensures streams are properly closed when errors occur, preventing resource leaks
|
|
34
|
+
- Error propagation follows Node.js stream standards more closely
|
|
35
|
+
|
|
36
|
+
4. **Rust Configuration Simplification**:
|
|
37
|
+
- Removed `MAX_BATCH_TIMEOUT` constant - now uses only `DEFAULT_BATCH_TIMEOUT` (1000ms)
|
|
38
|
+
- Simplified timeout validation logic in Rust layer
|
|
39
|
+
- Invalid timeout values (< 1ms) now log warnings and use default timeout
|
|
40
|
+
|
|
41
|
+
5. **Code Quality Improvements**:
|
|
42
|
+
- Fixed linting violations: short variable names and `any` type usage
|
|
43
|
+
- Better TypeScript typing throughout the codebase
|
|
44
|
+
- Removed redundant validation functions for cleaner code
|
|
45
|
+
|
|
46
|
+
### Migration Guide
|
|
47
|
+
|
|
48
|
+
- **No API changes required** - all public interfaces remain the same
|
|
49
|
+
- **Stream memory management** - streams now automatically clean up resources
|
|
50
|
+
- **Error handling** - streams will properly terminate on errors (existing error handling code still works)
|
|
51
|
+
- **Performance** - simplified timeout logic may provide marginal performance improvements
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## What's New in Version 1.10.0
|
|
56
|
+
|
|
57
|
+
### Major Updates:
|
|
58
|
+
|
|
59
|
+
1. **Performance Benchmark Suite**:
|
|
60
|
+
- Added comprehensive benchmark suite comparing performance against `kafkajs`, `node-rdkafka`, and `@platformatic/rdkafka`
|
|
61
|
+
- Benchmark scripts with automated setup and data generation
|
|
62
|
+
- Performance testing for both serial and batch processing modes
|
|
63
|
+
- Supports configurable iterations and message sizes for realistic testing scenarios
|
|
64
|
+
|
|
65
|
+
2. **Enhanced Stream Consumer Configuration**:
|
|
66
|
+
- Stream consumer constructor now accepts optional `ReadableOptions` parameter
|
|
67
|
+
- Default `objectMode: true` for better stream processing ergonomics
|
|
68
|
+
- Improved batch processing configuration with proper defaults
|
|
69
|
+
|
|
70
|
+
3. **Flexible Configuration System**:
|
|
71
|
+
- **BREAKING**: Configuration properties now accept `Record<string, any>` instead of `Record<string, string>`
|
|
72
|
+
- Enhanced type flexibility for complex configuration values (numbers, booleans, objects)
|
|
73
|
+
- Better support for advanced librdkafka configuration options
|
|
74
|
+
|
|
75
|
+
4. **Enhanced Topic Management**:
|
|
76
|
+
- Added `createTopic` and `numPartitions` options to `TopicPartitionConfig`
|
|
77
|
+
- Improved topic creation control during consumer subscription
|
|
78
|
+
- Better handling of partition assignment and topic setup
|
|
79
|
+
|
|
80
|
+
5. **Build System Improvements**:
|
|
81
|
+
- Updated TypeScript configuration with modular `tsconfig.base.json`
|
|
82
|
+
- Added `typecheck` script for standalone type checking
|
|
83
|
+
- Enhanced build pipeline with better declaration file generation
|
|
84
|
+
- Updated to pnpm@10.15.1 for improved dependency management
|
|
85
|
+
|
|
86
|
+
6. **Development Tools Enhancement**:
|
|
87
|
+
- Enhanced oxlint configuration with benchmark-specific rules
|
|
88
|
+
- Added support for benchmark directory linting and formatting
|
|
89
|
+
- Improved development workflow with better error handling rules
|
|
90
|
+
|
|
91
|
+
### Previous Version (1.9.0) Updates:
|
|
92
|
+
7. **Consumer Topic Creation Control**:
|
|
93
|
+
- Added fine-grained control over consumer topic creation behavior
|
|
94
|
+
- Enhanced consistency in topic management and cleanup processes
|
|
95
|
+
- Improved handling of topic creation inconsistencies during consumer operations
|
|
96
|
+
|
|
97
|
+
8. **Enhanced Manual Commit Testing**:
|
|
98
|
+
- Consolidated manual commit tests for better reliability and coverage
|
|
99
|
+
- Added comprehensive integration tests with PostRebalance event handling
|
|
100
|
+
- Tests now verify offset persistence across consumer restarts and batch processing
|
|
101
|
+
|
|
102
|
+
### Previous Version (1.8.0) Breaking Changes:
|
|
103
|
+
9. **Async Consumer Commit**:
|
|
17
104
|
- **BREAKING**: The `consumer.commit()` method is now async and must be awaited
|
|
18
105
|
- **Before**: `consumer.commit(topic, partition, offset, 'Sync')`
|
|
19
106
|
- **After**: `await consumer.commit(topic, partition, offset, 'Sync')`
|
|
20
107
|
- This change improves performance by using `spawn_blocking` for non-blocking operations
|
|
21
108
|
|
|
22
|
-
### Major Updates:
|
|
23
|
-
2. **Enhanced Manual Commit Testing**:
|
|
24
|
-
- Added comprehensive integration tests for manual commit functionality
|
|
25
|
-
- Tests now use PostRebalance events for proper timing instead of arbitrary timeouts
|
|
26
|
-
- Covers both sync and async commit modes with offset verification
|
|
27
|
-
- Tests offset persistence across consumer restarts and batch processing scenarios
|
|
28
|
-
|
|
29
|
-
3. **Improved Error Messages**:
|
|
30
|
-
- Standardized all error messages to use consistent "Failed to [action]" format
|
|
31
|
-
- Enhanced error clarity and specificity throughout the codebase
|
|
32
|
-
- Fixed grammar and spelling issues in error messages
|
|
33
|
-
|
|
34
|
-
4. **Updated Dependencies**:
|
|
35
|
-
- Upgraded all dependencies to their latest versions for improved performance and security
|
|
36
|
-
- Updated TypeScript bindings and CommonJS exports
|
|
37
|
-
- Synchronized pnpm-lock.yaml with new dependency versions
|
|
38
|
-
|
|
39
|
-
5. **Arc-based Consumer Architecture**:
|
|
40
|
-
- Improved internal architecture using `Arc<StreamConsumer>` for better memory management
|
|
41
|
-
- Enhanced thread safety for commit operations
|
|
42
|
-
|
|
43
109
|
---
|
|
44
110
|
|
|
45
111
|
## Features
|
|
@@ -50,9 +116,11 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
|
|
|
50
116
|
- ๐ฏ Type-safe interfaces (TypeScript support)
|
|
51
117
|
- โก Async/await support
|
|
52
118
|
- ๐ ๏ธ Configurable consumer and producer options
|
|
53
|
-
- ๐ Stream processing support
|
|
119
|
+
- ๐ Stream processing support with configurable stream options
|
|
54
120
|
- ๐ฆ Message batching capabilities
|
|
55
121
|
- ๐ Comprehensive error handling
|
|
122
|
+
- ๐ Performance benchmarking suite
|
|
123
|
+
- ๐ง Flexible configuration system supporting complex data types
|
|
56
124
|
|
|
57
125
|
## Table of Contents
|
|
58
126
|
|
|
@@ -62,9 +130,10 @@ A lightweight, flexible, and reliable Kafka client for JavaScript/TypeScript. It
|
|
|
62
130
|
4. [Producer Examples](#producer-examples)
|
|
63
131
|
5. [Stream Processing](#stream-processing)
|
|
64
132
|
6. [Configuration](#configuration)
|
|
65
|
-
7. [
|
|
66
|
-
8. [
|
|
67
|
-
9. [
|
|
133
|
+
7. [Performance Benchmarks](#performance-benchmarks)
|
|
134
|
+
8. [Best Practices](#best-practices)
|
|
135
|
+
9. [Contributing](#contributing)
|
|
136
|
+
10. [License](#license)
|
|
68
137
|
|
|
69
138
|
## Installation
|
|
70
139
|
|
|
@@ -90,12 +159,21 @@ async function run() {
|
|
|
90
159
|
brokerAddressFamily: 'v4',
|
|
91
160
|
});
|
|
92
161
|
|
|
93
|
-
// Create consumer
|
|
162
|
+
// Create consumer with topic creation control
|
|
94
163
|
const consumer = kafkaClient.createConsumer({
|
|
95
164
|
groupId: 'foo-group',
|
|
165
|
+
configuration: {
|
|
166
|
+
'auto.offset.reset': 'earliest',
|
|
167
|
+
'enable.auto.commit': false, // Use manual commit for better control
|
|
168
|
+
}
|
|
96
169
|
});
|
|
97
170
|
|
|
98
|
-
|
|
171
|
+
// Subscribe with topic creation options
|
|
172
|
+
await consumer.subscribe([{
|
|
173
|
+
topic: 'foo',
|
|
174
|
+
createTopic: true,
|
|
175
|
+
numPartitions: 3
|
|
176
|
+
}]);
|
|
99
177
|
|
|
100
178
|
const message = await consumer.recv();
|
|
101
179
|
const { payload, partition, offset, topic } = message;
|
|
@@ -110,7 +188,6 @@ async function run() {
|
|
|
110
188
|
await consumer.commit(topic, partition, offset + 1, 'Sync');
|
|
111
189
|
|
|
112
190
|
consumer.unsubscribe();
|
|
113
|
-
|
|
114
191
|
}
|
|
115
192
|
|
|
116
193
|
await run();
|
|
@@ -128,7 +205,14 @@ const kafkaClient = new KafkaClient({
|
|
|
128
205
|
brokerAddressFamily: 'v4',
|
|
129
206
|
});
|
|
130
207
|
|
|
131
|
-
|
|
208
|
+
// Producer configuration is now optional with sensible defaults
|
|
209
|
+
const producer = kafkaClient.createProducer({
|
|
210
|
+
configuration: {
|
|
211
|
+
'message.timeout.ms': 5000, // Now supports number values
|
|
212
|
+
'batch.size': 16384,
|
|
213
|
+
'compression.type': 'snappy'
|
|
214
|
+
}
|
|
215
|
+
});
|
|
132
216
|
|
|
133
217
|
const message = {
|
|
134
218
|
id: 1,
|
|
@@ -149,12 +233,11 @@ if (errors.length > 0) {
|
|
|
149
233
|
} else {
|
|
150
234
|
console.log('Message sent. Offset:', result);
|
|
151
235
|
}
|
|
152
|
-
|
|
153
236
|
```
|
|
154
237
|
|
|
155
238
|
## Stream Processing
|
|
156
239
|
|
|
157
|
-
### Stream Consumer Example
|
|
240
|
+
### Enhanced Stream Consumer Example
|
|
158
241
|
|
|
159
242
|
```javascript
|
|
160
243
|
import { KafkaClient } from 'kafka-crab-js';
|
|
@@ -166,24 +249,38 @@ const kafkaClient = new KafkaClient({
|
|
|
166
249
|
brokerAddressFamily: 'v4',
|
|
167
250
|
});
|
|
168
251
|
|
|
252
|
+
// Stream consumer with custom ReadableOptions (v1.10.0+)
|
|
169
253
|
const kafkaStream = kafkaClient.createStreamConsumer({
|
|
170
|
-
groupId: `my-
|
|
254
|
+
groupId: `my-group-id`,
|
|
171
255
|
enableAutoCommit: true,
|
|
256
|
+
}, {
|
|
257
|
+
objectMode: true, // Default in v1.10.0+
|
|
258
|
+
highWaterMark: 1024,
|
|
259
|
+
encoding: null
|
|
172
260
|
});
|
|
173
261
|
|
|
174
|
-
await kafkaStream.subscribe([
|
|
262
|
+
await kafkaStream.subscribe([
|
|
263
|
+
{ topic: 'foo', createTopic: true },
|
|
264
|
+
{ topic: 'bar', createTopic: true }
|
|
265
|
+
]);
|
|
175
266
|
|
|
176
267
|
kafkaStream.on('data', (message) => {
|
|
177
|
-
console.log('>>> Message received:', {
|
|
268
|
+
console.log('>>> Message received:', {
|
|
269
|
+
payload: message.payload.toString(),
|
|
270
|
+
offset: message.offset,
|
|
271
|
+
partition: message.partition,
|
|
272
|
+
topic: message.topic
|
|
273
|
+
});
|
|
274
|
+
|
|
178
275
|
if (message.offset > 10) {
|
|
179
276
|
kafkaStream.destroy();
|
|
180
277
|
}
|
|
181
|
-
})
|
|
278
|
+
});
|
|
182
279
|
|
|
183
280
|
kafkaStream.on('close', () => {
|
|
184
281
|
kafkaStream.unsubscribe();
|
|
185
|
-
console.log('Stream ended')
|
|
186
|
-
})
|
|
282
|
+
console.log('Stream ended');
|
|
283
|
+
});
|
|
187
284
|
```
|
|
188
285
|
|
|
189
286
|
## Producer Examples
|
|
@@ -196,7 +293,16 @@ const kafkaClient = new KafkaClient({
|
|
|
196
293
|
clientId: 'my-client-id',
|
|
197
294
|
brokerAddressFamily: 'v4',
|
|
198
295
|
});
|
|
199
|
-
|
|
296
|
+
|
|
297
|
+
// Enhanced producer with flexible configuration
|
|
298
|
+
const producer = kafkaClient.createProducer({
|
|
299
|
+
configuration: {
|
|
300
|
+
'batch.size': 50000, // Number value supported
|
|
301
|
+
'linger.ms': 10, // Number value supported
|
|
302
|
+
'compression.type': 'lz4',
|
|
303
|
+
'enable.idempotence': true // Boolean value supported
|
|
304
|
+
}
|
|
305
|
+
});
|
|
200
306
|
|
|
201
307
|
const messages = Array.from({ length: 100 }, (_, i) => ({
|
|
202
308
|
payload: Buffer.from(JSON.stringify({
|
|
@@ -222,11 +328,17 @@ try {
|
|
|
222
328
|
|
|
223
329
|
```javascript
|
|
224
330
|
async function produceWithMetadata() {
|
|
225
|
-
const producer =
|
|
331
|
+
const producer = kafkaClient.createProducer({
|
|
332
|
+
configuration: {
|
|
333
|
+
'acks': 'all',
|
|
334
|
+
'retries': 5,
|
|
335
|
+
'max.in.flight.requests.per.connection': 1
|
|
336
|
+
}
|
|
337
|
+
});
|
|
226
338
|
|
|
227
339
|
try {
|
|
228
340
|
await producer.send({
|
|
229
|
-
topic,
|
|
341
|
+
topic: 'user-events',
|
|
230
342
|
messages: [{
|
|
231
343
|
key: 'user-123',
|
|
232
344
|
payload: Buffer.from(JSON.stringify({
|
|
@@ -257,6 +369,8 @@ const kafkaClient = new KafkaClient({
|
|
|
257
369
|
brokerAddressFamily: 'v4',
|
|
258
370
|
configuration: {
|
|
259
371
|
'auto.offset.reset': 'earliest',
|
|
372
|
+
'session.timeout.ms': 30000,
|
|
373
|
+
'heartbeat.interval.ms': 10000
|
|
260
374
|
},
|
|
261
375
|
})
|
|
262
376
|
|
|
@@ -267,10 +381,14 @@ async function createConsumer() {
|
|
|
267
381
|
const kafkaStream = kafkaClient.createStreamConsumer({
|
|
268
382
|
groupId: 'reconnect-test',
|
|
269
383
|
enableAutoCommit: true,
|
|
384
|
+
}, {
|
|
385
|
+
highWaterMark: 100,
|
|
386
|
+
objectMode: true
|
|
270
387
|
})
|
|
388
|
+
|
|
271
389
|
await kafkaStream.subscribe([
|
|
272
|
-
{ topic: 'foo' },
|
|
273
|
-
{ topic: 'bar' },
|
|
390
|
+
{ topic: 'foo', createTopic: true },
|
|
391
|
+
{ topic: 'bar', createTopic: true },
|
|
274
392
|
])
|
|
275
393
|
return kafkaStream
|
|
276
394
|
}
|
|
@@ -342,7 +460,7 @@ await startConsumer()
|
|
|
342
460
|
|
|
343
461
|
### Examples
|
|
344
462
|
|
|
345
|
-
You can find some examples on the [example](https://github.com/
|
|
463
|
+
You can find some examples on the [example](https://github.com/flash-tecnologia/kafka-crab-js/tree/main/example) folder of this project.
|
|
346
464
|
|
|
347
465
|
## Configuration
|
|
348
466
|
|
|
@@ -355,13 +473,100 @@ You can find some examples on the [example](https://github.com/mthssdrbrg/kafka-
|
|
|
355
473
|
| `brokers` | `string` || List of brokers to connect to |
|
|
356
474
|
| `clientId` | `string` || Client id to use for the connection |
|
|
357
475
|
| `securityProtocol` | `SecurityProtocol` || Security protocol to use (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL) |
|
|
358
|
-
| `logLevel` | `string` | `info` |
|
|
476
|
+
| `logLevel` | `string` | `info` | Log level for the client |
|
|
359
477
|
| `brokerAddressFamily` | `string` | `"v4"` | Address family to use for the connection (v4, v6) |
|
|
360
|
-
| `configuration` | `
|
|
478
|
+
| `configuration` | `Record<string, any>` | `{}` | Additional configuration options for the client. **v1.10.0+**: Now supports any value type (string, number, boolean, object) |
|
|
479
|
+
|
|
480
|
+
### ConsumerConfiguration
|
|
481
|
+
|
|
482
|
+
| Property | Type | Default | Description |
|
|
483
|
+
| --- | --- | --- | --- |
|
|
484
|
+
| `groupId` | `string` || Consumer group ID |
|
|
485
|
+
| `enableAutoCommit` | `boolean` | `true` | Enable automatic offset commits |
|
|
486
|
+
| `configuration` | `Record<string, any>` | `{}` | Additional consumer configuration options |
|
|
487
|
+
| `fetchMetadataTimeout` | `number` | `60000` | Timeout for fetching metadata (ms) |
|
|
488
|
+
| `maxBatchMessages` | `number` | `1000` | Maximum messages in a batch operation |
|
|
489
|
+
|
|
490
|
+
### ProducerConfiguration
|
|
491
|
+
|
|
492
|
+
| Property | Type | Default | Description |
|
|
493
|
+
| --- | --- | --- | --- |
|
|
494
|
+
| `queueTimeout` | `number` | `5000` | Queue timeout in milliseconds |
|
|
495
|
+
| `autoFlush` | `boolean` | `true` | Enable automatic message flushing |
|
|
496
|
+
| `configuration` | `Record<string, any>` | `{}` | Additional producer configuration options |
|
|
497
|
+
|
|
498
|
+
### TopicPartitionConfig
|
|
499
|
+
|
|
500
|
+
| Property | Type | Default | Description |
|
|
501
|
+
| --- | --- | --- | --- |
|
|
502
|
+
| `topic` | `string` || Topic name |
|
|
503
|
+
| `allOffsets` | `OffsetModel` || Offset configuration for all partitions |
|
|
504
|
+
| `partitionOffset` | `Array<PartitionOffset>` || Per-partition offset configuration |
|
|
505
|
+
| `createTopic` | `boolean` | `false` | **v1.10.0+**: Create topic if it doesn't exist |
|
|
506
|
+
| `numPartitions` | `number` | `1` | **v1.10.0+**: Number of partitions when creating topic |
|
|
361
507
|
|
|
362
508
|
You can see the available options here: [librdkafka](https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html).
|
|
363
509
|
|
|
510
|
+
## Performance Benchmarks
|
|
511
|
+
|
|
512
|
+
### Running Benchmarks
|
|
513
|
+
|
|
514
|
+
kafka-crab-js v1.10.0+ includes a comprehensive benchmark suite to compare performance against other popular Kafka clients:
|
|
515
|
+
|
|
516
|
+
```bash
|
|
517
|
+
# Set up benchmark environment (requires Kafka running locally)
|
|
518
|
+
pnpm add -D tsx # For running TypeScript files directly
|
|
519
|
+
npx tsx benchmark/utils/setup.ts
|
|
520
|
+
|
|
521
|
+
# Run consumer performance benchmarks
|
|
522
|
+
node benchmark/consumer.ts
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
### Benchmark Results
|
|
526
|
+
|
|
527
|
+
*Benchmarks run on macOS with Apple M1 chip processing 50,000 messages*
|
|
528
|
+
|
|
529
|
+
```
|
|
530
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
531
|
+
โ Slower tests โ Samples โ Result โ Tolerance โ Difference with previous โ
|
|
532
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโข
|
|
533
|
+
โ kafkajs โ 50000 โ 2316.67 op/sec โ ยฑ 1.12 % โ โ
|
|
534
|
+
โ kafka-crab-js (serial) โ 50000 โ 38414.57 op/sec โ ยฑ 2.97 % โ + 1558.18 % โ
|
|
535
|
+
โ node-rdkafka (stream) โ 50000 โ 43681.44 op/sec โ ยฑ 13.98 % โ + 13.71 % โ
|
|
536
|
+
โ node-rdkafka (evented) โ 53841 โ 74137.97 op/sec โ ยฑ 69.30 % โ + 69.72 % โ
|
|
537
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโข
|
|
538
|
+
โ Fastest test โ Samples โ Result โ Tolerance โ Difference with previous โ
|
|
539
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโข
|
|
540
|
+
โ kafka-crab-js (batch) โ 50000 โ 153533.53 op/sec โ ยฑ 15.98 % โ + 107.09 % โ
|
|
541
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
542
|
+
```
|
|
364
543
|
|
|
544
|
+
The benchmark suite compares:
|
|
545
|
+
- **kafka-crab-js (serial)**: Single message processing - **38,415 ops/sec**
|
|
546
|
+
- **kafka-crab-js (batch)**: Batch message processing - **153,534 ops/sec** (fastest)
|
|
547
|
+
- **node-rdkafka (evented)**: Event-based processing - **74,138 ops/sec**
|
|
548
|
+
- **node-rdkafka (stream)**: Stream-based processing - **43,681 ops/sec**
|
|
549
|
+
- **kafkajs**: Official KafkaJS client - **2,317 ops/sec**
|
|
550
|
+
|
|
551
|
+
Performance characteristics:
|
|
552
|
+
- **17x faster than kafkajs** in serial mode, **66x faster in batch mode**
|
|
553
|
+
- **High throughput**: Batch processing provides 4x performance improvement over serial mode
|
|
554
|
+
- **Low latency**: Optimized for both single and batch message processing
|
|
555
|
+
- **Memory efficient**: Lock-free data structures minimize memory overhead
|
|
556
|
+
- **Concurrent processing**: Zero-contention concurrent operations
|
|
557
|
+
|
|
558
|
+
### Benchmark Configuration
|
|
559
|
+
|
|
560
|
+
You can customize benchmark parameters in `benchmark/utils/definitions.ts`:
|
|
561
|
+
|
|
562
|
+
```typescript
|
|
563
|
+
export const topic = 'benchmarks'
|
|
564
|
+
export const brokers = ['localhost:9092', 'localhost:9093', 'localhost:9094']
|
|
565
|
+
|
|
566
|
+
// Benchmark parameters can be adjusted in consumer.ts:
|
|
567
|
+
const iterations = 10_000 // Number of messages to process
|
|
568
|
+
const maxBytes = 200 // Maximum message size
|
|
569
|
+
```
|
|
365
570
|
|
|
366
571
|
## Best Practices
|
|
367
572
|
|
|
@@ -372,21 +577,60 @@ You can see the available options here: [librdkafka](https://docs.confluent.io/p
|
|
|
372
577
|
|
|
373
578
|
### Performance
|
|
374
579
|
- Use batch operations for high-throughput scenarios
|
|
375
|
-
- Configure appropriate batch sizes and compression
|
|
580
|
+
- Configure appropriate batch sizes and compression settings
|
|
376
581
|
- Monitor and tune consumer group performance
|
|
582
|
+
- Leverage the benchmark suite to optimize your specific use case
|
|
583
|
+
|
|
584
|
+
### Configuration (v1.10.0+)
|
|
585
|
+
- Use the flexible configuration system with proper data types:
|
|
586
|
+
```javascript
|
|
587
|
+
const config = {
|
|
588
|
+
'batch.size': 16384, // number
|
|
589
|
+
'compression.type': 'snappy', // string
|
|
590
|
+
'enable.idempotence': true, // boolean
|
|
591
|
+
'retries': 5 // number
|
|
592
|
+
}
|
|
593
|
+
```
|
|
377
594
|
|
|
378
595
|
### Message Processing
|
|
379
596
|
- Validate message formats before processing
|
|
380
597
|
- Implement proper serialization/deserialization
|
|
381
598
|
- Handle message ordering when required
|
|
599
|
+
- Use topic creation options for better topic management
|
|
600
|
+
|
|
601
|
+
### Stream Processing (v1.10.0+)
|
|
602
|
+
- Configure appropriate `ReadableOptions` for your use case
|
|
603
|
+
- Use `objectMode: true` for structured message processing
|
|
604
|
+
- Set appropriate `highWaterMark` based on memory constraints
|
|
382
605
|
|
|
383
606
|
## Contributing
|
|
384
607
|
|
|
385
608
|
1. Fork the repository
|
|
386
609
|
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
|
|
387
|
-
3.
|
|
388
|
-
4.
|
|
389
|
-
5.
|
|
610
|
+
3. Run the benchmark suite to ensure performance isn't degraded
|
|
611
|
+
4. Commit your changes (`git commit -m 'Add some amazing feature'`)
|
|
612
|
+
5. Push to the branch (`git push origin feature/amazing-feature`)
|
|
613
|
+
6. Open a Pull Request
|
|
614
|
+
|
|
615
|
+
### Development Commands
|
|
616
|
+
|
|
617
|
+
```bash
|
|
618
|
+
# Build the project
|
|
619
|
+
pnpm build
|
|
620
|
+
|
|
621
|
+
# Run type checking
|
|
622
|
+
pnpm typecheck
|
|
623
|
+
|
|
624
|
+
# Run linting
|
|
625
|
+
pnpm lint
|
|
626
|
+
|
|
627
|
+
# Format code
|
|
628
|
+
pnpm fmt
|
|
629
|
+
|
|
630
|
+
# Run benchmarks
|
|
631
|
+
npx tsx benchmark/utils/setup.ts
|
|
632
|
+
npx tsx benchmark/consumer.ts
|
|
633
|
+
```
|
|
390
634
|
|
|
391
635
|
## License
|
|
392
636
|
|
|
@@ -398,4 +642,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
|
|
|
398
642
|
|
|
399
643
|
[](https://www.rust-lang.org/)
|
|
400
644
|
|
|
401
|
-
</div>
|
|
645
|
+
</div>
|