@drarzter/kafka-client 0.6.6 → 0.6.9

package/README.md

@@ -33,6 +33,9 @@ Type-safe Kafka client for Node.js. Framework-agnostic core with a first-class N
 - [Deduplication (Lamport Clock)](#deduplication-lamport-clock)
 - [Retry topic chain](#retry-topic-chain)
 - [stopConsumer](#stopconsumer)
+- [Pause and resume](#pause-and-resume)
+- [Reset consumer offsets](#reset-consumer-offsets)
+- [DLQ replay](#dlq-replay)
 - [Graceful shutdown](#graceful-shutdown)
 - [Consumer handles](#consumer-handles)
 - [onMessageLost](#onmessagelost)
@@ -620,7 +623,7 @@ await kafka.startBatchConsumer(
 | Property/Method | Description |
 | --------------- | ----------- |
 | `partition` | Partition number for this batch |
-| `highWatermark` | Latest offset in the partition (lag indicator) |
+| `highWatermark` | Latest offset in the partition (`string`). `null` when the message is replayed via a retry topic consumer — in that path the broker high-watermark is not available. Guard against `null` before computing lag |
 | `heartbeat()` | Send a heartbeat to keep the consumer session alive — call during long processing loops |
 | `resolveOffset(offset)` | Mark offset as processed (required before `commitOffsetsIfNecessary`) |
 | `commitOffsetsIfNecessary()` | Commit resolved offsets; respects `autoCommit` setting |
@@ -743,6 +746,55 @@ type BeforeConsumeResult =
 
 When multiple instrumentations each provide a `wrap`, they compose in declaration order — the first instrumentation's `wrap` is the outermost.
 
+### Lifecycle event hooks
+
+Three additional hooks fire for specific events in the consume pipeline:
+
+| Hook | When called | Arguments |
+| ---- | ----------- | --------- |
+| `onMessage` | Handler successfully processed a message | `(envelope)` — use as a success counter for error-rate calculations |
+| `onRetry` | A message is queued for another attempt (in-process backoff or routed to a retry topic) | `(envelope, attempt, maxRetries)` |
+| `onDlq` | A message is routed to the dead letter queue | `(envelope, reason)` — reason is `'handler-error'`, `'validation-error'`, or `'lamport-clock-duplicate'` |
+| `onDuplicate` | A duplicate is detected via Lamport Clock | `(envelope, strategy)` — strategy is `'drop'`, `'dlq'`, or `'topic'` |
+
+```typescript
+const myInstrumentation: KafkaInstrumentation = {
+  onMessage(envelope) {
+    metrics.increment('kafka.processed', { topic: envelope.topic });
+  },
+  onRetry(envelope, attempt, maxRetries) {
+    console.warn(`Retrying ${envelope.topic} — attempt ${attempt}/${maxRetries}`);
+  },
+  onDlq(envelope, reason) {
+    alertingSystem.send({ topic: envelope.topic, reason });
+  },
+  onDuplicate(envelope, strategy) {
+    metrics.increment('kafka.duplicate', { topic: envelope.topic, strategy });
+  },
+};
+```
+
+### Built-in metrics
+
+`KafkaClient` maintains lightweight in-process event counters independently of any instrumentation:
+
+```typescript
+// Global snapshot — aggregate across all topics
+const snapshot = kafka.getMetrics();
+// { processedCount: number; retryCount: number; dlqCount: number; dedupCount: number }
+
+// Per-topic snapshot
+const orderMetrics = kafka.getMetrics('order.created');
+// { processedCount: 5, retryCount: 1, dlqCount: 0, dedupCount: 0 }
+
+kafka.resetMetrics();                // reset all counters
+kafka.resetMetrics('order.created'); // reset only one topic's counters
+```
+
+Passing a topic name that has not seen any events returns a zero-valued snapshot — it never throws.
+
+Counters are incremented in the same code paths that fire the corresponding hooks — they are always active regardless of whether any instrumentation is configured.
+
 ## Options reference
 
 ### Send options
@@ -795,6 +847,7 @@ Passed to `KafkaModule.register()` or returned from `registerAsync()` factory:
 | `numPartitions` | `1` | Number of partitions for auto-created topics |
 | `strictSchemas` | `true` | Validate string topic keys against schemas registered via TopicDescriptor |
 | `instrumentation` | `[]` | Client-wide instrumentation hooks (e.g. OTel). Applied to both send and consume paths |
+| `transactionalId` | `${clientId}-tx` | Transactional producer ID for `transaction()` calls. Must be unique per producer instance across the cluster — two instances sharing the same ID will be fenced by Kafka. The client logs a warning when the same ID is registered twice within one process |
 | `onMessageLost` | — | Called when a message is silently dropped without DLQ — use to alert, log to external systems, or trigger fallback logic |
 | `onRebalance` | — | Called on every partition assign/revoke event across all consumers created by this client |
 
@@ -983,7 +1036,7 @@ By default, retry is handled in-process: the consumer sleeps between attempts wh
 
 Benefits over in-process retry:
 
-- **Durable** — retry messages survive a consumer restart; routing between levels and to DLQ is exactly-once via Kafka transactions
+- **Durable** — retry messages survive a consumer restart; all routing (main → retry.1, level N → N+1, retry → DLQ) is exactly-once via Kafka transactions
 - **Non-blocking** — the original consumer is free immediately; each level consumer only pauses its specific partition during the delay window, so other partitions continue processing
 - **Isolated** — each retry level has its own consumer group, so a slow level 3 consumer never blocks a level 1 consumer
 
@@ -1015,9 +1068,9 @@ Each level consumer uses `consumer.pause → sleep(remaining) → consumer.resum
 
 The retry topic messages carry scheduling headers (`x-retry-attempt`, `x-retry-after`, `x-retry-original-topic`, `x-retry-max-retries`) that each level consumer reads automatically — no manual configuration needed.
 
-> **Delivery guarantee:** routing within the retry chain (retry.N → retry.N+1 and retry.N → DLQ) is **exactly-once** — each routing step is wrapped in a Kafka transaction via `sendOffsetsToTransaction`, so the produce and the consumer offset commit happen atomically. A crash at any point rolls back the transaction: the message is redelivered and the routing is retried, with no duplicate in the next level. If the EOS transaction itself fails (broker unavailable), the offset is not committed and the message stays safely in the retry topic until the broker recovers.
+> **Delivery guarantee:** the entire retry chain — including the **main consumer → retry.1** boundary — is **exactly-once**. Every routing step (main → retry.1, retry.N → retry.N+1, retry.N → DLQ) is wrapped in a Kafka transaction via `sendOffsetsToTransaction`: the produce and the consumer offset commit happen atomically. A crash at any point rolls back the transaction: the message is redelivered and the routing is retried, with no duplicate in the next level. If the EOS transaction fails (broker unavailable), the offset stays uncommitted and the message is safely redelivered — it is never lost.
 >
-> The remaining at-least-once window is at the **main consumer → retry.1** boundary: the main consumer uses `autoCommit: true` by default, so if it crashes after routing to `retry.1` but before autoCommit fires, the message may appear twice in `retry.1`. This is the standard Kafka at-least-once trade-off for any consumer using autoCommit. Design handlers to be idempotent if this edge case is unacceptable.
+> The standard Kafka at-least-once guarantee still applies at the handler level: if your handler succeeds but the process crashes before the manual offset commit completes, the message is redelivered to the handler. Design handlers to be idempotent.
 >
 > **Startup validation:** `retryTopics` requires `retry` to be set — an error is thrown at startup if `retry` is missing. When `autoCreateTopics: false`, all `{topic}.retry.N` topics are validated to exist at startup and a clear error lists any missing ones. With `autoCreateTopics: true` the check is skipped — topics are created automatically by the `ensureTopic` path. Supported by both `startConsumer` and `startBatchConsumer`.
 
@@ -1037,6 +1090,75 @@ await kafka.stopConsumer();
 
 `stopConsumer(groupId)` disconnects and removes only that group's consumer, leaving other groups running. Useful when you want to pause processing for a specific topic without restarting the whole client.
 
+## Pause and resume
+
+Temporarily stop delivering messages from specific partitions without disconnecting the consumer:
+
+```typescript
+// Pause partition 0 of 'orders' (default group)
+kafka.pauseConsumer(undefined, [{ topic: 'orders', partitions: [0] }]);
+
+// Resume it later
+kafka.resumeConsumer(undefined, [{ topic: 'orders', partitions: [0] }]);
+
+// Target a specific consumer group, multiple partitions
+kafka.pauseConsumer('payments-group', [{ topic: 'payments', partitions: [0, 1] }]);
+```
+
+The first argument is the consumer group ID — pass `undefined` to target the default group. A warning is logged if the group is not found.
+
+Pausing is non-destructive: the consumer stays connected and Kafka preserves the partition assignment for as long as the group session is alive. Messages accumulate in the topic and are delivered once the consumer resumes. Typical use: apply backpressure when a downstream dependency (e.g. a database) is temporarily overloaded.
+
+## Reset consumer offsets
+
+Seek a consumer group's committed offsets to the beginning or end of a topic:
+
+```typescript
+// Seek to the beginning — re-process all existing messages
+await kafka.resetOffsets(undefined, 'orders', 'earliest');
+
+// Seek to the end — skip existing messages, process only new ones
+await kafka.resetOffsets(undefined, 'orders', 'latest');
+
+// Target a specific consumer group
+await kafka.resetOffsets('payments-group', 'orders', 'earliest');
+```
+
+**Important:** the consumer for the specified group must be stopped before calling `resetOffsets`. An error is thrown if the group is currently running — this prevents the reset from racing with an active offset commit.
+
+## DLQ replay
+
+Re-publish messages from a dead letter queue back to the original topic:
+
+```typescript
+// Re-publish all messages from 'orders.dlq' → 'orders'
+const result = await kafka.replayDlq('orders');
+// { replayed: 42, skipped: 0 }
+```
+
+Options:
+
+| Option | Default | Description |
+| ------ | ------- | ----------- |
+| `targetTopic` | `x-dlq-original-topic` header | Override the destination topic |
+| `dryRun` | `false` | Count messages without sending |
+| `filter` | — | `(headers) => boolean` — skip messages where the callback returns `false` |
+
+```typescript
+// Dry run — see how many messages would be replayed
+const dry = await kafka.replayDlq('orders', { dryRun: true });
+
+// Route to a different topic
+const result = await kafka.replayDlq('orders', { targetTopic: 'orders.v2' });
+
+// Only replay messages with a specific correlation ID
+const filtered = await kafka.replayDlq('orders', {
+  filter: (headers) => headers['x-correlation-id'] === 'corr-123',
+});
+```
+
+`replayDlq` creates a temporary consumer group that reads the DLQ topic up to the high-watermark at the time of the call — messages published after replay starts are not included. DLQ metadata headers (`x-dlq-original-topic`, `x-dlq-error-message`, `x-dlq-error-stack`, `x-dlq-failed-at`, `x-dlq-attempt-count`) are stripped from the replayed messages; all other headers (e.g. `x-correlation-id`) are preserved.
+
 ## Graceful shutdown
 
 `disconnect()` now drains in-flight handlers before tearing down connections — no messages are silently cut off mid-processing.