@drarzter/kafka-client 0.7.0 → 0.7.2

package/README.md

@@ -20,12 +20,15 @@ Type-safe Kafka client for Node.js. Framework-agnostic core with a first-class N
 - [Consuming messages](#consuming-messages)
   - [Declarative: @SubscribeTo()](#declarative-subscribeto)
   - [Imperative: startConsumer()](#imperative-startconsumer)
+  - [Regex topic subscription](#regex-topic-subscription)
   - [Iterator: consume()](#iterator-consume)
 - [Multiple consumer groups](#multiple-consumer-groups)
 - [Partition key](#partition-key)
 - [Message headers](#message-headers)
 - [Batch sending](#batch-sending)
 - [Batch consuming](#batch-consuming)
+- [Tombstone messages](#tombstone-messages)
+- [Compression](#compression)
 - [Transactions](#transactions)
 - [Consumer interceptors](#consumer-interceptors)
 - [Instrumentation](#instrumentation)
@@ -36,13 +39,17 @@ Type-safe Kafka client for Node.js. Framework-agnostic core with a first-class N
 - [stopConsumer](#stopconsumer)
 - [Pause and resume](#pause-and-resume)
 - [Circuit breaker](#circuit-breaker)
+  - [getCircuitState](#getcircuitstate)
 - [Reset consumer offsets](#reset-consumer-offsets)
 - [Seek to offset](#seek-to-offset)
+- [Seek to timestamp](#seek-to-timestamp)
 - [Message TTL](#message-ttl)
 - [DLQ replay](#dlq-replay)
+- [Admin API](#admin-api)
 - [Graceful shutdown](#graceful-shutdown)
 - [Consumer handles](#consumer-handles)
 - [onMessageLost](#onmessagelost)
+- [onTtlExpired](#onttlexpired)
 - [onRebalance](#onrebalance)
 - [Consumer lag](#consumer-lag)
 - [Handler timeout warning](#handler-timeout-warning)
@@ -94,6 +101,11 @@ Safe by default. Configurable when you need it. Escape hatches for when you know
 - **Message TTL** — `messageTtlMs` drops or DLQs messages older than a configurable threshold, preventing stale events from poisoning downstream systems after a lag spike
 - **Circuit breaker** — `circuitBreaker` option applies a sliding-window breaker per topic-partition; pauses delivery on repeated DLQ failures and resumes after a configurable recovery window
 - **Seek to offset** — `seekToOffset(groupId, assignments)` seeks individual partitions to explicit offsets for fine-grained replay
+- **Tombstone messages** — `sendTombstone(topic, key)` sends a null-value record to compact a key out of a log-compacted topic; all instrumentation hooks still fire
+- **Regex topic subscription** — `startConsumer([/^orders\..+/], handler)` subscribes using a pattern; the broker routes matching topics to the consumer dynamically
+- **Compression** — per-send `compression` option (`gzip`, `snappy`, `lz4`, `zstd`) in `SendOptions` and `BatchSendOptions`
+- **Partition assignment strategy** — `partitionAssigner` in `ConsumerOptions` chooses between `cooperative-sticky` (default), `roundrobin`, and `range`
+- **Admin API** — `listConsumerGroups()`, `describeTopics()`, `deleteRecords()` for group inspection, partition metadata, and message deletion
 
 See the [Roadmap](./ROADMAP.md) for upcoming features and version history.
 
@@ -436,6 +448,27 @@ export class OrdersService implements OnModuleInit {
 }
 ```
 
+### Regex topic subscription
+
+Subscribe to multiple topics matching a pattern — the broker dynamically routes any topic whose name matches the regex to this consumer:
+
+```typescript
+// Subscribe to all topics starting with "orders."
+await kafka.startConsumer([/^orders\..+/], handler);
+
+// Mix regexes and literal strings
+await kafka.startConsumer([/^payments\..+/, 'audit.global'], handler);
+```
+
+Works with `startBatchConsumer` and `@SubscribeTo` too:
+
+```typescript
+@SubscribeTo(/^events\..+/)
+async handleEvent(envelope: EventEnvelope<any>) { ... }
+```
+
+> **Limitation:** `retryTopics: true` is incompatible with regex subscriptions — the library cannot derive static retry topic names from a pattern. An error is thrown at startup if both are combined.
+
 ### Iterator: consume()
 
 Stream messages from a single topic as an `AsyncIterableIterator` — useful for scripts, one-off tasks, or any context where you prefer `for await` over a callback:
@@ -465,6 +498,20 @@ for await (const envelope of kafka.consume('orders', {
 
 `break`, `return`, or any early exit from the loop calls the iterator's `return()` method, which closes the internal queue and calls `handle.stop()` on the background consumer.
 
+**Backpressure** — use `queueHighWaterMark` to prevent unbounded queue growth when processing is slower than the message rate:
+
+```typescript
+for await (const envelope of kafka.consume('orders', {
+  queueHighWaterMark: 100, // pause partition when queue reaches 100 messages
+})) {
+  await slowProcessing(envelope.payload); // resumes when queue drains below 50
+}
+```
+
+The partition is paused when the internal queue reaches `queueHighWaterMark` and automatically resumed when it drains below 50%. Without this option the queue is unbounded.
+
+**Error propagation** — if the consumer fails to start (e.g. broker unreachable), the error surfaces on the next `next()` / `for await` iteration rather than being silently swallowed.
+
 ## Multiple consumer groups
 
 ### Per-consumer groupId
@@ -665,6 +712,36 @@ await kafka.startBatchConsumer(
 | `resolveOffset(offset)` | Mark offset as processed (required before `commitOffsetsIfNecessary`) |
 | `commitOffsetsIfNecessary()` | Commit resolved offsets; respects `autoCommit` setting |
 
+## Tombstone messages
+
+Send a null-value Kafka record to compact a specific key out of a log-compacted topic:
+
+```typescript
+// Delete the record with key "user-123" from the log-compacted "users" topic
+await kafka.sendTombstone('users', 'user-123');
+
+// With custom headers
+await kafka.sendTombstone('users', 'user-123', { 'x-reason': 'gdpr-deletion' });
+```
+
+`sendTombstone` skips envelope headers, schema validation, and the Lamport clock — the record value is literally `null`, as required by Kafka's log compaction protocol. Both `beforeSend` and `afterSend` instrumentation hooks still fire so tracing works correctly.
+
+## Compression
+
+Reduce network bandwidth with per-send compression. Supported codecs: `'gzip'`, `'snappy'`, `'lz4'`, `'zstd'`:
+
+```typescript
+import { CompressionType } from '@drarzter/kafka-client/core';
+
+// Single message
+await kafka.sendMessage('events', payload, { compression: 'gzip' });
+
+// Batch
+await kafka.sendBatch('events', messages, { compression: 'snappy' });
+```
+
+Compression is applied at the Kafka message-set level — the broker decompresses transparently on the consumer side. `'snappy'` and `'lz4'` offer the best throughput/CPU trade-off for most workloads; `'gzip'` gives the highest compression ratio; `'zstd'` balances both.
+
 ## Transactions
 
 Send multiple messages atomically with exactly-once semantics:
@@ -845,8 +922,9 @@ Options for `sendMessage()` — the third argument:
 | `correlationId` | auto | Override the auto-propagated correlation ID (default: inherited from ALS context or new UUID) |
 | `schemaVersion` | `1` | Schema version for the payload |
 | `eventId` | auto | Override the auto-generated event ID (UUID v4) |
+| `compression` | — | Compression codec for the message set: `'gzip'`, `'snappy'`, `'lz4'`, `'zstd'`; omit to send uncompressed |
 
-`sendBatch()` accepts the same options per message inside the array items.
+`sendBatch()` accepts `compression` as a top-level option (not per-message); all other options are per-message inside the array items.
 
 ### Consumer options
 
@@ -871,7 +949,10 @@ Options for `sendMessage()` — the third argument:
 | `circuitBreaker.recoveryMs` | `30_000` | Milliseconds to wait in OPEN state before entering HALF_OPEN |
 | `circuitBreaker.windowSize` | `threshold × 2, min 10` | Sliding window size in messages |
 | `circuitBreaker.halfOpenSuccesses` | `1` | Consecutive successes in HALF_OPEN required to close the circuit |
+| `queueHighWaterMark` | unbounded | Max messages buffered in the `consume()` iterator queue before the partition is paused; resumes at 50% drain. Only applies to `consume()` |
 | `batch` | `false` | (decorator only) Use `startBatchConsumer` instead of `startConsumer` |
+| `partitionAssigner` | `'cooperative-sticky'` | Partition assignment strategy: `'cooperative-sticky'` (minimal movement on rebalance, best for horizontal scaling), `'roundrobin'` (even distribution), `'range'` (contiguous partition ranges) |
+| `onTtlExpired` | — | Per-consumer override of the client-level `onTtlExpired` callback; takes precedence when set. Receives `TtlExpiredContext` — same shape as the client-level hook |
 | `subscribeRetry.retries` | `5` | Max attempts for `consumer.subscribe()` when topic doesn't exist yet |
 | `subscribeRetry.backoffMs` | `5000` | Delay between subscribe retry attempts (ms) |
 
@@ -892,6 +973,7 @@ Passed to `KafkaModule.register()` or returned from `registerAsync()` factory:
 | `instrumentation` | `[]` | Client-wide instrumentation hooks (e.g. OTel). Applied to both send and consume paths |
 | `transactionalId` | `${clientId}-tx` | Transactional producer ID for `transaction()` calls. Must be unique per producer instance across the cluster — two instances sharing the same ID will be fenced by Kafka. The client logs a warning when the same ID is registered twice within one process |
 | `onMessageLost` | — | Called when a message is silently dropped without DLQ — use to alert, log to external systems, or trigger fallback logic |
+| `onTtlExpired` | — | Called when a message is dropped due to TTL expiration (`messageTtlMs`) and `dlq` is not enabled; receives `{ topic, ageMs, messageTtlMs, headers }` |
 | `onRebalance` | — | Called on every partition assign/revoke event across all consumers created by this client |
 
 **Module-scoped** (default) — import `KafkaModule` in each module that needs it:
@@ -1200,6 +1282,41 @@ Options:
 | `windowSize` | `threshold × 2, min 10` | Sliding window size in messages |
 | `halfOpenSuccesses` | `1` | Consecutive successes in HALF_OPEN required to close the circuit |
 
+### getCircuitState
+
+Inspect the current circuit breaker state for a partition — useful for health endpoints and dashboards:
+
+```typescript
+const state = kafka.getCircuitState('orders', 0);
+// undefined — circuit not configured or never tripped
+// { status: 'closed', failures: 2, windowSize: 10 }
+// { status: 'open',   failures: 5, windowSize: 10 }
+// { status: 'half-open', failures: 0, windowSize: 0 }
+
+// With explicit group ID:
+const state = kafka.getCircuitState('orders', 0, 'payments-group');
+```
+
+Returns `undefined` when `circuitBreaker` is not configured for the group or the circuit has never been tripped (state is lazily initialised on the first DLQ event).
+
+**Instrumentation hooks** — react to state transitions via `KafkaInstrumentation`:
+
+```typescript
+const kafka = new KafkaClient('svc', 'group', brokers, {
+  instrumentation: [{
+    onCircuitOpen(topic, partition) {
+      metrics.increment('circuit_open', { topic, partition });
+    },
+    onCircuitHalfOpen(topic, partition) {
+      logger.log(`Circuit probing ${topic}[${partition}]`);
+    },
+    onCircuitClose(topic, partition) {
+      metrics.increment('circuit_close', { topic, partition });
+    },
+  }],
+});
+```
+
 ## Reset consumer offsets
 
 Seek a consumer group's committed offsets to the beginning or end of a topic:
@@ -1239,6 +1356,29 @@ The first argument is the consumer group ID — pass `undefined` to target the d
 
 **Important:** the consumer for the specified group must be stopped before calling `seekToOffset`. An error is thrown if the group is currently running.
 
+## Seek to timestamp
+
+Seek partitions to the offset nearest to a specific point in time — useful for replaying events that occurred after a known incident or deployment:
+
+```typescript
+const ts = new Date('2024-06-01T12:00:00Z').getTime(); // Unix ms
+
+await kafka.seekToTimestamp(undefined, [
+  { topic: 'orders', partition: 0, timestamp: ts },
+  { topic: 'orders', partition: 1, timestamp: ts },
+]);
+
+// Multiple topics in one call
+await kafka.seekToTimestamp('payments-group', [
+  { topic: 'payments', partition: 0, timestamp: ts },
+  { topic: 'refunds',  partition: 0, timestamp: ts },
+]);
+```
+
+Uses `admin.fetchTopicOffsetsByTime` under the hood. If no offset exists at the requested timestamp (e.g. the partition is empty or the timestamp is in the future), the partition falls back to `-1` (end of topic — new messages only).
+
+**Important:** the consumer group must be stopped before seeking. Assignments for the same topic are batched into a single `admin.setOffsets` call.
+
 ## Message TTL
 
 Drop or route expired messages using `messageTtlMs` in `ConsumerOptions`:
@@ -1253,7 +1393,7 @@ await kafka.startConsumer(['orders'], handler, {
 The TTL is evaluated against the `x-timestamp` header stamped on every outgoing message by the producer. Messages whose age at consumption time exceeds `messageTtlMs` are:
 
 - **Routed to DLQ** with `x-dlq-reason: ttl-expired` when `dlq: true`
-- **Dropped** (calling `onMessageLost`) otherwise
+- **Dropped** (calling `onTtlExpired` if configured) otherwise
 
 Typical use: prevent stale events from poisoning downstream systems after a consumer lag spike — e.g. discard order events or push notifications that are no longer actionable.
 
@@ -1290,6 +1430,50 @@ const filtered = await kafka.replayDlq('orders', {
 
 `replayDlq` creates a temporary consumer group that reads the DLQ topic up to the high-watermark at the time of the call — messages published after replay starts are not included. DLQ metadata headers (`x-dlq-original-topic`, `x-dlq-error-message`, `x-dlq-error-stack`, `x-dlq-failed-at`, `x-dlq-attempt-count`) are stripped from the replayed messages; all other headers (e.g. `x-correlation-id`) are preserved.
 
+## Admin API
+
+Inspect consumer groups, topic metadata, and delete records via the built-in admin client — no separate connection needed.
+
+### `listConsumerGroups()`
+
+Returns all consumer groups known to the broker:
+
+```typescript
+const groups = await kafka.listConsumerGroups();
+// [{ groupId: 'orders-group', state: 'Stable' }, ...]
+```
+
+`state` reflects the librdkafka group state: `'Stable'`, `'PreparingRebalance'`, `'CompletingRebalance'`, `'Dead'`, or `'Empty'`.
+
+### `describeTopics(topics?)`
+
+Fetch partition metadata (leader, replicas, ISR) for one or more topics. Omit `topics` to describe all topics the client knows about:
+
+```typescript
+const info = await kafka.describeTopics(['orders.created', 'payments.received']);
+// [
+//   {
+//     name: 'orders.created',
+//     partitions: [{ partition: 0, leader: 1, replicas: [1, 2], isr: [1, 2] }],
+//   },
+//   ...
+// ]
+```
+
+### `deleteRecords(topic, partitions)`
+
+Truncate a topic by deleting all records up to (but not including) a given offset:
+
+```typescript
+// Delete all records in partition 0 up to offset 500
+await kafka.deleteRecords('orders.created', [
+  { partition: 0, offset: '500' },
+  { partition: 1, offset: '200' },
+]);
+```
+
+Pass `offset: '-1'` to delete all records in a partition (truncate completely).
+
 ## Graceful shutdown
 
 `disconnect()` now drains in-flight handlers before tearing down connections — no messages are silently cut off mid-processing.
@@ -1361,6 +1545,40 @@ const kafka = new KafkaClient('my-app', 'my-group', ['localhost:9092'], {
 
 In the normal case (`dlq: true`, DLQ send succeeds), `onMessageLost` does NOT fire — the message is preserved in `{topic}.dlq`.
 
+> **Note:** TTL-expired messages do **not** trigger `onMessageLost`. Use `onTtlExpired` to observe them separately.
+
+## onTtlExpired
+
+Called when a message is dropped because it exceeded `messageTtlMs` and `dlq` is not enabled. Fires instead of `onMessageLost` for expired messages:
+
+```typescript
+import { KafkaClient, TtlExpiredContext } from '@drarzter/kafka-client/core';
+
+const kafka = new KafkaClient('my-app', 'my-group', ['localhost:9092'], {
+  onTtlExpired: (ctx: TtlExpiredContext) => {
+    // ctx.topic         — topic the message came from
+    // ctx.ageMs         — actual age of the message at drop time
+    // ctx.messageTtlMs  — the configured threshold
+    // ctx.headers       — original message headers
+    logger.warn(`Stale message on ${ctx.topic}: ${ctx.ageMs}ms old (limit ${ctx.messageTtlMs}ms)`);
+  },
+});
+```
+
+When `dlq: true`, expired messages are routed to DLQ instead and `onTtlExpired` is **not** called.
+
+`onTtlExpired` can also be set per-consumer via `ConsumerOptions.onTtlExpired`. The consumer-level value takes precedence over the client-level one, so you can use different handlers for different topics:
+
+```typescript
+await kafka.startConsumer(['orders'], handler, {
+  messageTtlMs: 30_000,
+  onTtlExpired: (ctx) => {
+    // overrides the client-level onTtlExpired for this consumer only
+    ordersMetrics.increment('ttl_expired', { topic: ctx.topic });
+  },
+});
+```
+
 ## onRebalance
 
 React to partition rebalance events without patching the consumer. Useful for flushing in-flight state before partitions are revoked, or for logging/metrics: