@basaltkit/queue-kafka 1.0.2 → 1.1.1

package/README.md

@@ -70,7 +70,7 @@ await Job.dispatch(payload, { delay: '5m' }) // → throws UnsupportedJobOptionE
 // with onUnsupported: 'warn' (default) → warns once and runs immediately
 ```
 
-> If what you need is *streaming*/pub-sub (rather than jobs with retry/delay), the natural fit in Basalt is usually `@basaltkit/events`, not `@basaltkit/queue`.
+**If what you need is *streaming*/pub-sub** (rather than jobs with retry/delay), the natural fit in Basalt is usually `@basaltkit/events`, not `@basaltkit/queue`.
 
 ## How it works
 
@@ -94,8 +94,57 @@ The worker's concurrency is passed as `partitionsConsumedConcurrently` — actua
 | `retrySuffix` | `string` | `'.retry'` | Suffix for the retry topic. |
 | `deadSuffix` | `string` | `'.dead'` | Suffix for the dead-letter topic. |
 | `client` | `KafkaClient` | kafkajs | Injectable client — used in tests without a broker. |
+| `onError` | `(error: unknown, info: { source: 'consumer' \| 'producer'; queue?: string }) => void` | contextual `console.error` | The driver's single fault channel — see below. |
 
-Implements the `QueueDriver` contract from `@basaltkit/queue`.
+Implements the `QueueDriver` contract from `@basaltkit/queue`. It does **not** implement the optional `stats` / `retryFailed`, so `basalt queue:stats` and `basalt queue:retry` report the operation as unsupported — use your Kafka tooling for consumer-group lag instead.
+
+### Failure hooks
+
+`onError` is the only callback, and its default (`console.error` with the source and queue) is
+never silent. There is no `onJobFailed`: a job that exhausts `attempts` is produced to
+`<topic>.dead`, which *is* the report — monitor that topic.
+
+| `source` | Raised when | What the driver does next |
+|---|---|---|
+| `'consumer'` | A worker's `connect`/`subscribe`/`run` rejected at boot (broker unreachable, missing topic, bad ACLs). | Reports and stops. Without this the rejection would float and be process-fatal, and the app would report healthy with **zero** workers. |
+| `'producer'` | The retry / dead-letter **re-publish itself failed** while handling a failed job. | Reports, then **rethrows** — see below. |
+
+### Redelivery when the DLQ produce fails
+
+The subtle case. A job's handler throws, so the driver tries to re-route the message — to
+`<topic>.retry` if attempts remain, otherwise to `<topic>.dead`. If *that* produce also fails
+(the producer lost its broker connection, the dead topic doesn't exist, the request timed out),
+the failed job exists nowhere but in the message currently being consumed.
+
+kafkajs auto-commits offsets after `eachMessage` **resolves**. So the driver:
+
+1. reports the publish failure through `onError({ source: 'producer', queue })`, then
+2. **rethrows** it, so `eachMessage` rejects and the offset is **not** committed.
+
+Kafka then redelivers the same message and the driver tries the whole thing again — at-least-once
+rather than a job that quietly evaporated during a producer outage. It is the Kafka equivalent of
+RabbitMQ leaving a message unacked.
+
+Two consequences worth planning for:
+
+- **Handlers must be idempotent.** A redelivered message re-runs the handler that already failed,
+  and a message whose re-publish succeeded is never redelivered — but a partition stalls on the
+  failing message while the producer is down, so ordered downstream work backs up behind it.
+- **A normal failure path does commit.** When the re-publish *succeeds*, the failure is
+  considered handled: the offset commits and the retry copy carries the incremented
+  `x-basalt-attempt` header. Redelivery only happens on the produce failure itself.
+
+### Exported errors
+
+This driver throws no error classes of its own. `UnsupportedJobOptionError`
+(`QUEUE_UNSUPPORTED_OPTION`) comes from `@basaltkit/queue` when a `delay`/`priority` dispatch
+meets `onUnsupported: 'throw'`; everything else surfaces through `onError`.
+
+### Hard limits
+
+The attempt counters travel in message headers, which any producer on the topic could write, so
+the consumer clamps the `x-basalt-attempts` it reads to at most **50**. A crafted message cannot
+drive an unbounded retry loop.
 
 ## How it connects to other modules
 

package/dist/index.d.ts

@@ -47,6 +47,17 @@ export interface KafkaDriverOptions {
     deadSuffix?: string;
     /** Injectable client — defaults to kafkajs. Tests pass a fake. */
     client?: KafkaClient;
+    /**
+     * Called on infrastructure faults the driver cannot recover in line — a
+     * worker's connect/subscribe/run failing at boot (otherwise the app reports
+     * healthy with ZERO workers and the rejection is process-fatal), or a
+     * retry/dead-letter re-publish failing inside the consume callback. Same
+     * pattern as the rabbitmq/sqs drivers. Default: console.error with context.
+     */
+    onError?: (error: unknown, info: {
+        source: 'consumer' | 'producer';
+        queue?: string;
+    }) => void;
 }
 /**
  * Kafka driver for `@basaltkit/queue`. Kafka is a log, not a task queue, so this
@@ -73,6 +84,7 @@ export declare class KafkaQueueDriver implements QueueDriver {
     private readonly groupId;
     private readonly retrySuffix;
     private readonly deadSuffix;
+    private readonly onError;
     constructor(options: KafkaDriverOptions);
     setExecutor(executor: JobExecutor): void;
     add(queue: string, jobName: string, data: unknown, options: AddJobOptions): Promise<void>;

package/dist/index.js

@@ -41,11 +41,15 @@ export class KafkaQueueDriver {
     groupId;
     retrySuffix;
     deadSuffix;
+    onError;
     constructor(options) {
         this.options = options;
         this.groupId = options.groupId ?? 'basalt-queue';
         this.retrySuffix = options.retrySuffix ?? '.retry';
         this.deadSuffix = options.deadSuffix ?? '.dead';
+        this.onError =
+            options.onError ??
+                ((error, info) => console.error(`[basalt:queue] kafka ${info.source} error${info.queue ? ` (queue "${info.queue}")` : ''}:`, error));
     }
     setExecutor(executor) {
         this.executor = executor;
@@ -70,7 +74,8 @@ export class KafkaQueueDriver {
         });
     }
     startWorker(queue, options = {}) {
-        void (async () => {
+        ;
+        (async () => {
             const consumer = (await this.client()).consumer({ groupId: this.groupId });
             this.consumers.push(consumer);
             await consumer.connect();
@@ -80,7 +85,12 @@ export class KafkaQueueDriver {
                 eachMessage: ({ message }) => this.handle(queue, message),
                 ...(options.concurrency !== undefined ? { partitionsConsumedConcurrently: options.concurrency } : {}),
             });
-        })();
+        })().catch((error) => {
+            // A broker-connect/subscribe failure at boot must be VISIBLE — otherwise
+            // the app reports healthy with zero workers, and the floating rejection
+            // would be process-fatal by Node's default.
+            this.onError(error, { source: 'consumer', queue });
+        });
     }
     async close() {
         const producer = await this.producerPromise?.catch(() => undefined);
@@ -102,15 +112,26 @@ export class KafkaQueueDriver {
             // Clamp the max-attempts read from the (untrusted) message to a hard
             // ceiling so a crafted `attempts` can't drive a retry-amplification loop.
             const attempts = Math.min(Number(headers[HEADER.attempts] ?? 1) || 1, MAX_ATTEMPTS);
-            const producer = await this.producer();
-            if (attempt < attempts) {
-                await producer.send({
-                    topic: this.retryTopic(queue),
-                    messages: [{ value, headers: { ...headers, [HEADER.attempt]: String(attempt + 1) } }],
-                });
+            try {
+                const producer = await this.producer();
+                if (attempt < attempts) {
+                    await producer.send({
+                        topic: this.retryTopic(queue),
+                        messages: [{ value, headers: { ...headers, [HEADER.attempt]: String(attempt + 1) } }],
+                    });
+                }
+                else {
+                    await producer.send({ topic: this.deadTopic(queue), messages: [{ value, headers }] });
+                }
             }
-            else {
-                await producer.send({ topic: this.deadTopic(queue), messages: [{ value, headers }] });
+            catch (publishError) {
+                // The failed job could not be re-routed. Surface the fault, then
+                // RETHROW so eachMessage rejects and the offset is NOT committed —
+                // kafkajs redelivers the message (at-least-once) instead of the job
+                // silently vanishing on a producer outage. (Rabbitmq keeps the message
+                // unacked for the same reason; Kafka's equivalent is not committing.)
+                this.onError(publishError, { source: 'producer', queue });
+                throw publishError;
             }
         }
         // Offsets auto-commit after eachMessage resolves — a re-routed failure is

package/package.json

@@ -1,9 +1,13 @@
 {
   "name": "@basaltkit/queue-kafka",
-  "version": "1.0.2",
+  "version": "1.1.1",
+  "engines": {
+    "node": ">=22.5.0"
+  },
   "description": "Kafka driver for @basaltkit/queue: produce/consume jobs with retry and dead-letter topics (no delayed delivery or priority — Kafka has neither).",
   "license": "MIT",
   "type": "module",
+  "sideEffects": false,
   "exports": {
     ".": {
       "types": "./dist/index.d.ts",
@@ -14,7 +18,7 @@
     "dist"
   ],
   "dependencies": {
-    "@basaltkit/queue": "^1.2.1"
+    "@basaltkit/queue": "^1.4.1"
   },
   "peerDependencies": {
     "kafkajs": "^2.0.0"