@drarzter/kafka-client 0.5.2 → 0.5.5

package/README.md

@@ -6,10 +6,57 @@
 
 Type-safe Kafka client for Node.js. Framework-agnostic core with a first-class NestJS adapter. Built on top of [`@confluentinc/kafka-javascript`](https://github.com/confluentinc/confluent-kafka-javascript) (librdkafka).
 
+## Table of contents
+
+- [What is this?](#what-is-this)
+- [Why?](#why)
+- [Installation](#installation)
+- [Standalone usage](#standalone-usage-no-nestjs)
+- [Quick start (NestJS)](#quick-start-nestjs)
+- [Usage](#usage)
+  - [1. Define your topic map](#1-define-your-topic-map)
+  - [2. Register the module](#2-register-the-module)
+  - [3. Inject and use](#3-inject-and-use)
+- [Consuming messages](#consuming-messages)
+  - [Declarative: @SubscribeTo()](#declarative-subscribeto)
+  - [Imperative: startConsumer()](#imperative-startconsumer)
+- [Multiple consumer groups](#multiple-consumer-groups)
+- [Partition key](#partition-key)
+- [Message headers](#message-headers)
+- [Batch sending](#batch-sending)
+- [Batch consuming](#batch-consuming)
+- [Transactions](#transactions)
+- [Consumer interceptors](#consumer-interceptors)
+- [Instrumentation](#instrumentation)
+- [Options reference](#options-reference)
+- [Error classes](#error-classes)
+- [Retry topic chain](#retry-topic-chain)
+- [stopConsumer](#stopconsumer)
+- [Consumer handles](#consumer-handles)
+- [onMessageLost](#onmessagelost)
+- [onRebalance](#onrebalance)
+- [Consumer lag](#consumer-lag)
+- [Handler timeout warning](#handler-timeout-warning)
+- [Schema validation](#schema-validation)
+- [Health check](#health-check)
+- [Testing](#testing)
+- [Project structure](#project-structure)
+
 ## What is this?
 
 An opinionated, type-safe abstraction over `@confluentinc/kafka-javascript` (librdkafka). Works standalone (Express, Fastify, raw Node) or as a NestJS DynamicModule. Not a full-featured framework — just a clean, typed layer for producing and consuming Kafka messages.
 
+**This library exists so you don't have to think about:**
+
+- rebalance edge cases
+- retry loops and backoff scheduling
+- dead letter queue wiring
+- transaction coordinator warmup
+- graceful shutdown and offset commit pitfalls
+- silent message loss
+
+Safe by default. Configurable when you need it. Escape hatches for when you know what you're doing.
+
 ## Why?
 
 - **Typed topics** — you define a map of topic -> message shape, and the compiler won't let you send wrong data to wrong topic
@@ -666,7 +713,9 @@ Options for `sendMessage()` — the third argument:
 | `retry.backoffMs` | `1000` | Base delay for exponential backoff in ms |
 | `retry.maxBackoffMs` | `30000` | Maximum delay cap for exponential backoff in ms |
 | `dlq` | `false` | Send to `{topic}.dlq` after all retries exhausted — message carries `x-dlq-*` metadata headers |
+| `retryTopics` | `false` | Route failed messages through `{topic}.retry` instead of sleeping in-process (see [Retry topic chain](#retry-topic-chain)) |
 | `interceptors` | `[]` | Array of before/after/onError hooks |
+| `handlerTimeoutMs` | — | Log a warning if the handler hasn't resolved within this window (ms) — does not cancel the handler |
 | `batch` | `false` | (decorator only) Use `startBatchConsumer` instead of `startConsumer` |
 | `subscribeRetry.retries` | `5` | Max attempts for `consumer.subscribe()` when topic doesn't exist yet |
 | `subscribeRetry.backoffMs` | `5000` | Delay between subscribe retry attempts (ms) |
@@ -687,6 +736,7 @@ Passed to `KafkaModule.register()` or returned from `registerAsync()` factory:
 | `strictSchemas` | `true` | Validate string topic keys against schemas registered via TopicDescriptor |
 | `instrumentation` | `[]` | Client-wide instrumentation hooks (e.g. OTel). Applied to both send and consume paths |
 | `onMessageLost` | — | Called when a message is silently dropped without DLQ — use to alert, log to external systems, or trigger fallback logic |
+| `onRebalance` | — | Called on every partition assign/revoke event across all consumers created by this client |
 
 **Module-scoped** (default) — import `KafkaModule` in each module that needs it:
 
@@ -774,6 +824,64 @@ const interceptor: ConsumerInterceptor<MyTopics> = {
 };
 ```
 
+## Retry topic chain
+
+By default, retry is handled in-process: the consumer sleeps between attempts while holding the partition. With `retryTopics: true`, failed messages are routed to a `<topic>.retry` Kafka topic instead. A companion consumer auto-starts on `<topic>.retry` (group `<groupId>-retry`), waits until the scheduled retry time, then calls the same handler.
+
+Benefits over in-process retry:
+
+- **Durable** — retry messages survive a consumer restart
+- **Non-blocking** — the original consumer is free immediately; the retry consumer pauses only the specific partition being delayed, so other partitions continue processing
+
+```typescript
+await kafka.startConsumer(['orders.created'], handler, {
+  retry: { maxRetries: 3, backoffMs: 1000, maxBackoffMs: 30_000 },
+  dlq: true,
+  retryTopics: true,   // ← opt in
+});
+```
+
+Message flow with `maxRetries: 2`:
+
+```text
+orders.created  →  handler fails  →  orders.created.retry  (attempt 1, delay ~1 s)
+                                   →  handler fails  →  orders.created.retry  (attempt 2, delay ~2 s)
+                                                      →  handler fails  →  orders.created.dlq
+```
+
+The retry topic messages carry scheduling headers (`x-retry-attempt`, `x-retry-after`, `x-retry-original-topic`, `x-retry-max-retries`) that the companion consumer reads automatically — no manual configuration needed.
+
+> **Note:** `retryTopics` requires `retry` to be set — an error is thrown at startup if `retry` is missing. Currently only applies to `startConsumer`; batch consumers (`startBatchConsumer`) use in-process retry regardless.
+
+## stopConsumer
+
+Stop all consumers or a specific group:
+
+```typescript
+// Stop a specific consumer group
+await kafka.stopConsumer('my-group');
+
+// Stop all consumers
+await kafka.stopConsumer();
+```
+
+`stopConsumer(groupId)` disconnects and removes only that group's consumer, leaving other groups running. Useful when you want to pause processing for a specific topic without restarting the whole client.
+
+## Consumer handles
+
+`startConsumer()` and `startBatchConsumer()` return a `ConsumerHandle` instead of `void`. Use it to stop a specific consumer without needing to remember the group ID:
+
+```typescript
+const handle = await kafka.startConsumer(['orders'], handler);
+
+console.log(handle.groupId); // e.g. "my-group"
+
+// Later — stop only this consumer, producer stays connected
+await handle.stop();
+```
+
+`handle.stop()` is equivalent to `kafka.stopConsumer(handle.groupId)`. Useful in lifecycle methods or when you need to conditionally stop one consumer while others keep running.
+
 ## onMessageLost
 
 By default, if a consumer handler throws and `dlq` is not enabled, the message is logged and dropped. Use `onMessageLost` to catch these silent losses:
@@ -799,6 +907,59 @@ const kafka = new KafkaClient('my-app', 'my-group', ['localhost:9092'], {
 
 It does NOT fire when `dlq: true` — in that case the message is preserved in `{topic}.dlq`.
 
+## onRebalance
+
+React to partition rebalance events without patching the consumer. Useful for flushing in-flight state before partitions are revoked, or for logging/metrics:
+
+```typescript
+const kafka = new KafkaClient('my-app', 'my-group', ['localhost:9092'], {
+  onRebalance: (type, partitions) => {
+    // type       — 'assign' | 'revoke'
+    // partitions — Array<{ topic: string; partition: number }>
+    console.log(`Rebalance ${type}:`, partitions);
+  },
+});
+```
+
+- `'assign'` fires when this instance receives new partitions (e.g. on startup or when another consumer leaves the group).
+- `'revoke'` fires when partitions are taken away (e.g. another consumer joins).
+
+The callback is applied to **every** consumer created by this client. If you need per-consumer rebalance handling, use separate `KafkaClient` instances.
+
+## Consumer lag
+
+Query consumer group lag per partition via the admin API — no external tooling needed:
+
+```typescript
+const lag = await kafka.getConsumerLag();
+// [{ topic: 'orders', partition: 0, lag: 12 }, ...]
+
+// Or for a different group:
+const lag2 = await kafka.getConsumerLag('another-group');
+```
+
+Lag is computed as `brokerHighWatermark − lastCommittedOffset`. A partition with a committed offset of `-1` (nothing ever committed) reports full lag equal to the high watermark.
+
+Returns an empty array when the group has no committed offsets at all.
+
+## Handler timeout warning
+
+Catch stuck handlers before they silently starve a partition. Set `handlerTimeoutMs` on `startConsumer` or `startBatchConsumer`:
+
+```typescript
+await kafka.startConsumer(['orders'], handler, {
+  handlerTimeoutMs: 5_000, // warn if handler hasn't resolved after 5 s
+});
+```
+
+If the handler hasn't resolved within the window, a `warn` is logged:
+
+```text
+[KafkaClient:my-app] Handler for topic "orders" has not resolved after 5000ms — possible stuck handler
+```
+
+The handler is **not** cancelled — the warning is diagnostic only. Combine with `retry` to automatically give up after a fixed number of slow attempts.
+
 ## Schema validation
 
 Add runtime message validation using any library with a `.parse()` method — Zod, Valibot, ArkType, or a custom validator. No extra dependency required.