@basaltkit/queue-kafka 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Machize Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,97 @@
+# @basaltkit/queue-kafka
+
+**Apache Kafka** driver for [`@basaltkit/queue`](https://www.npmjs.com/package/@basaltkit/queue): runs your jobs by producing and consuming messages on Kafka topics, without changing your job code. You need this package when your data platform is already built on Kafka and you want to process background work on the same infrastructure.
+
+## What this module solves
+
+`@basaltkit/queue` defines **jobs** in an abstract way and picks the backend via a *driver*. This package provides a driver that talks to **Kafka**: jobs are produced to a topic and consumed by a *consumer group*.
+
+`defineJob`, `dispatch`, the workers and context propagation all stay the same — you only swap the driver.
+
+## Installation
+
+```bash
+pnpm add @basaltkit/queue-kafka kafkajs
+```
+
+`kafkajs` is a **peer dependency**. You need a reachable Kafka cluster.
+
+## Getting started in 5 minutes
+
+```ts
+import { createApp } from '@basaltkit/core'
+import { queuePlugin, defineJob } from '@basaltkit/queue'
+import { KafkaQueueDriver } from '@basaltkit/queue-kafka'
+
+const IndexDocument = defineJob<{ id: string }>({
+  name: 'index-document',
+  queue: 'indexing',
+  attempts: 3,
+  async handle({ id }) {
+    // ... index it
+  },
+})
+
+const app = await createApp({
+  plugins: [
+    queuePlugin({
+      driver: new KafkaQueueDriver({ brokers: ['localhost:9092'], clientId: 'my-app' }),
+      jobs: [IndexDocument],
+      workers: [{ queue: 'indexing', concurrency: 4 }],
+    }),
+  ],
+}).boot()
+
+await IndexDocument.dispatch({ id: 'doc-1' })
+```
+
+## Being honest about Kafka
+
+Kafka is a **distributed log**, not a *task queue* — and the driver is deliberately honest about that in its `capabilities`:
+
+| Capability | Supported | Why |
+|---|:---:|---|
+| `delayed` (delayed delivery) | ❌ | Kafka doesn't delay messages. |
+| `priority` | ❌ | Kafka has no message priority. |
+| `retries` | ✅ | Via a *retry topic* that the worker also consumes. |
+| `backoff` | ❌ | No delay between attempts (Kafka doesn't defer). |
+
+Since the driver **declares** this, a job that requests `delay` or `priority` is caught by `@basaltkit/queue`'s `onUnsupported` policy:
+
+```ts
+queuePlugin({ driver: new KafkaQueueDriver({ brokers }), onUnsupported: 'throw' })
+await Job.dispatch(payload, { delay: '5m' }) // → throws UnsupportedJobOptionError
+// with onUnsupported: 'warn' (default) → warns once and runs immediately
+```
+
+> If what you need is *streaming*/pub-sub (rather than jobs with retry/delay), the natural fit in Basalt is usually `@basaltkit/events`, not `@basaltkit/queue`.
+
+## How it works
+
+For each queue `t`:
+
+- **`t`** — the main topic where jobs are produced.
+- **`t.retry`** — retry topic that the worker also subscribes to; a failed job is re-produced here with the attempt counter incremented.
+- **`t.dead`** — dead-letter topic for jobs that exhausted their attempts.
+
+The worker's concurrency is passed as `partitionsConsumedConcurrently` — actual parallelism is limited by the topic's **number of partitions**, not an arbitrary number.
+
+## API reference
+
+### `new KafkaQueueDriver(options)`
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `brokers` | `string[]` | — (required) | List of brokers, e.g. `['localhost:9092']`. |
+| `clientId` | `string` | `'basalt'` | Kafka client id. |
+| `groupId` | `string` | `'basalt-queue'` | Workers' consumer group. |
+| `retrySuffix` | `string` | `'.retry'` | Suffix for the retry topic. |
+| `deadSuffix` | `string` | `'.dead'` | Suffix for the dead-letter topic. |
+| `client` | `KafkaClient` | kafkajs | Injectable client — used in tests without a broker. |
+
+Implements the `QueueDriver` contract from `@basaltkit/queue`.
+
+## How it connects to other modules
+
+- **`@basaltkit/queue`** — this is a driver for that package; the jobs API comes from there.
+- Sibling drivers: [`@basaltkit/queue-rabbitmq`](https://www.npmjs.com/package/@basaltkit/queue-rabbitmq) and [`@basaltkit/queue-sqs`](https://www.npmjs.com/package/@basaltkit/queue-sqs).

package/dist/index.d.ts

@@ -0,0 +1,91 @@
+import { QueueDriver, DriverCapabilities, JobExecutor, AddJobOptions } from '@basaltkit/queue';
+
+/** The subset of a kafkajs client this driver uses. */
+interface KafkaClient {
+    producer(): KafkaProducer;
+    consumer(config: {
+        groupId: string;
+    }): KafkaConsumer;
+}
+interface KafkaProducer {
+    connect(): Promise<void>;
+    send(record: {
+        topic: string;
+        messages: {
+            value: string;
+            headers?: Record<string, string>;
+        }[];
+    }): Promise<unknown>;
+    disconnect(): Promise<void>;
+}
+interface KafkaMessage {
+    value: Uint8Array | string | null;
+    headers?: Record<string, Uint8Array | string | undefined>;
+}
+interface KafkaConsumer {
+    connect(): Promise<void>;
+    subscribe(subscription: {
+        topic: string;
+        fromBeginning?: boolean;
+    }): Promise<void>;
+    run(config: {
+        eachMessage: (payload: {
+            topic: string;
+            message: KafkaMessage;
+        }) => Promise<void>;
+        partitionsConsumedConcurrently?: number;
+    }): Promise<void>;
+    disconnect(): Promise<void>;
+}
+interface KafkaDriverOptions {
+    brokers: string[];
+    clientId?: string;
+    /** Consumer group used by workers. Default 'basalt-queue'. */
+    groupId?: string;
+    /** Suffix for the retry topic. Default '.retry'. */
+    retrySuffix?: string;
+    /** Suffix for the dead-letter topic. Default '.dead'. */
+    deadSuffix?: string;
+    /** Injectable client — defaults to kafkajs. Tests pass a fake. */
+    client?: KafkaClient;
+}
+/**
+ * Kafka driver for `@basaltkit/queue`. Kafka is a log, not a task queue, so this
+ * driver is deliberately honest about what it can't do:
+ *
+ * - `delayed` and `priority` are NOT supported (Kafka has neither). With the
+ *   queue's `onUnsupported: 'throw'` policy a delayed/priority dispatch fails
+ *   loudly; with the default 'warn' it logs and proceeds without them.
+ * - `retries` are supported via a retry topic (`<topic>.retry`) the worker also
+ *   consumes; exhausted jobs go to `<topic>.dead`. There is no backoff delay
+ *   (Kafka can't defer a message), so `backoff` is not supported either.
+ *
+ * Worker concurrency is bounded by the topic's partition count, not the
+ * `concurrency` number (passed through as `partitionsConsumedConcurrently`).
+ */
+declare class KafkaQueueDriver implements QueueDriver {
+    private readonly options;
+    readonly name = "kafka";
+    readonly capabilities: DriverCapabilities;
+    private executor;
+    private clientPromise;
+    private producerPromise;
+    private readonly consumers;
+    private readonly groupId;
+    private readonly retrySuffix;
+    private readonly deadSuffix;
+    constructor(options: KafkaDriverOptions);
+    setExecutor(executor: JobExecutor): void;
+    add(queue: string, jobName: string, data: unknown, options: AddJobOptions): Promise<void>;
+    startWorker(queue: string, options?: {
+        concurrency?: number;
+    }): void;
+    close(): Promise<void>;
+    private handle;
+    private client;
+    private producer;
+    private retryTopic;
+    private deadTopic;
+}
+
+export { type KafkaClient, type KafkaConsumer, type KafkaDriverOptions, type KafkaMessage, type KafkaProducer, KafkaQueueDriver };

package/dist/index.js

@@ -0,0 +1,127 @@
+// src/index.ts
+var HEADER = {
+  job: "x-basalt-job",
+  attempt: "x-basalt-attempt",
+  attempts: "x-basalt-attempts"
+};
+var defaultClient = async (options) => {
+  const specifier = "kafkajs";
+  const mod = await import(specifier);
+  return new mod.Kafka({ clientId: options.clientId ?? "basalt", brokers: options.brokers });
+};
+var KafkaQueueDriver = class {
+  constructor(options) {
+    this.options = options;
+    this.groupId = options.groupId ?? "basalt-queue";
+    this.retrySuffix = options.retrySuffix ?? ".retry";
+    this.deadSuffix = options.deadSuffix ?? ".dead";
+  }
+  options;
+  name = "kafka";
+  capabilities = {
+    delayed: false,
+    priority: false,
+    retries: true,
+    backoff: false
+  };
+  executor;
+  clientPromise;
+  producerPromise;
+  consumers = [];
+  groupId;
+  retrySuffix;
+  deadSuffix;
+  setExecutor(executor) {
+    this.executor = executor;
+  }
+  async add(queue, jobName, data, options) {
+    void options;
+    const producer = await this.producer();
+    await producer.send({
+      topic: queue,
+      messages: [
+        {
+          value: JSON.stringify(data),
+          headers: {
+            [HEADER.job]: jobName,
+            [HEADER.attempt]: "1",
+            [HEADER.attempts]: String(options.attempts)
+          }
+        }
+      ]
+    });
+  }
+  startWorker(queue, options = {}) {
+    void (async () => {
+      const consumer = (await this.client()).consumer({ groupId: this.groupId });
+      this.consumers.push(consumer);
+      await consumer.connect();
+      await consumer.subscribe({ topic: queue, fromBeginning: false });
+      await consumer.subscribe({ topic: this.retryTopic(queue), fromBeginning: false });
+      await consumer.run({
+        eachMessage: ({ message }) => this.handle(queue, message),
+        ...options.concurrency !== void 0 ? { partitionsConsumedConcurrently: options.concurrency } : {}
+      });
+    })();
+  }
+  async close() {
+    const producer = await this.producerPromise?.catch(() => void 0);
+    await producer?.disconnect();
+    await Promise.all(this.consumers.map((consumer) => consumer.disconnect()));
+  }
+  // --- internals -----------------------------------------------------------
+  async handle(queue, message) {
+    if (message.value === null) return;
+    const headers = decodeHeaders(message.headers);
+    const jobName = headers[HEADER.job] ?? "";
+    const value = asString(message.value);
+    try {
+      await this.executor?.(jobName, JSON.parse(value));
+    } catch {
+      const attempt = Number(headers[HEADER.attempt] ?? 1);
+      const attempts = Number(headers[HEADER.attempts] ?? 1);
+      const producer = await this.producer();
+      if (attempt < attempts) {
+        await producer.send({
+          topic: this.retryTopic(queue),
+          messages: [{ value, headers: { ...headers, [HEADER.attempt]: String(attempt + 1) } }]
+        });
+      } else {
+        await producer.send({ topic: this.deadTopic(queue), messages: [{ value, headers }] });
+      }
+    }
+  }
+  client() {
+    if (!this.clientPromise) {
+      this.clientPromise = this.options.client ? Promise.resolve(this.options.client) : defaultClient(this.options);
+    }
+    return this.clientPromise;
+  }
+  producer() {
+    if (!this.producerPromise) {
+      this.producerPromise = (async () => {
+        const producer = (await this.client()).producer();
+        await producer.connect();
+        return producer;
+      })();
+    }
+    return this.producerPromise;
+  }
+  retryTopic(queue) {
+    return `${queue}${this.retrySuffix}`;
+  }
+  deadTopic(queue) {
+    return `${queue}${this.deadSuffix}`;
+  }
+};
+var asString = (value) => typeof value === "string" ? value : new TextDecoder().decode(value);
+var decodeHeaders = (headers) => {
+  const out = {};
+  for (const [key, value] of Object.entries(headers ?? {})) {
+    if (value !== void 0) out[key] = asString(value);
+  }
+  return out;
+};
+export {
+  KafkaQueueDriver
+};

package/package.json

@@ -0,0 +1,51 @@
+{
+  "name": "@basaltkit/queue-kafka",
+  "version": "1.0.0",
+  "description": "Kafka driver for @basaltkit/queue: produce/consume jobs with retry and dead-letter topics (no delayed delivery or priority — Kafka has neither).",
+  "license": "MIT",
+  "type": "module",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
+  "files": [
+    "dist"
+  ],
+  "dependencies": {
+    "@basaltkit/queue": "^1.0.0"
+  },
+  "peerDependencies": {
+    "kafkajs": "^2.0.0"
+  },
+  "devDependencies": {
+    "@types/node": "^22.15.0",
+    "tsup": "^8.4.0",
+    "typescript": "^5.8.0",
+    "vitest": "^3.1.0",
+    "@basaltkit/tsconfig": "^0.24.0"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Zebedeu/basalt.git",
+    "directory": "packages/queue-kafka"
+  },
+  "homepage": "https://github.com/Zebedeu/basalt/tree/main/packages/queue-kafka#readme",
+  "bugs": "https://github.com/Zebedeu/basalt/issues",
+  "keywords": [
+    "basalt",
+    "typescript",
+    "queue",
+    "kafka",
+    "kafkajs"
+  ],
+  "scripts": {
+    "build": "tsup src/index.ts --format esm --dts --clean",
+    "test": "vitest run",
+    "typecheck": "tsc --noEmit"
+  }
+}