@basaltkit/queue 1.3.0 → 1.4.0

package/README.md

@@ -119,7 +119,10 @@ await GenerateInvoice.dispatch({ orderId: 'o-1' }, { delay: '10m' })
 await GenerateInvoice.dispatch({ orderId: 'o-2' }, { priority: 1 })
 ```
 
-Note: with the sync driver, `delay` is ignored — the job runs immediately.
+Note: with the sync driver, `delay` and `priority` are ignored — the job runs immediately.
+That is not silent: the sync driver declares `delayed: false` / `priority: false`, so the
+`onUnsupported` policy fires (by default a one-off `console.warn` per job+feature). Set
+`onUnsupported: 'throw'` in production if a delay is load-bearing.
 
 ### Context propagation (tenant, requestId…)
 
@@ -232,8 +235,15 @@ console.log(driver.executed) // [{ queue: 'default', jobName: 'demo', attempts:
 | `queue` | `string` | No | `'default'` | Name of the queue the job goes into. |
 | `attempts` | `number` | No | `1` | Maximum number of attempts on failure. |
 | `backoff` | `JobBackoff` | No | — | Wait strategy between attempts. |
+| `removeOnComplete` | `JobRetention` | No | the `queuePlugin` default | Redis retention for this job once it completes — overrides the plugin-wide default. |
+| `removeOnFail` | `JobRetention` | No | the `queuePlugin` default | Redis retention for this job once it fails permanently — overrides the plugin-wide default. |
 | `handle` | `(payload: T) => void \| Promise<void>` | Yes | — | The function that does the work (runs on the worker). |
 
+`JobRetention` = `boolean | number | { age?: DurationInput; count?: number }`. `true` removes
+the job as soon as it finishes, `false` keeps it forever, a number keeps that many most-recent,
+and `{ age, count }` caps by both. Only the BullMQ driver stores finished jobs, so only it
+honours retention; the sync driver stores nothing and ignores it.
+
 The returned object (`JobDefinition<T>`) exposes:
 
 - `dispatch(payload, options?)` — puts the job on the queue. Throws `JobNotRegisteredError` if the job hasn't yet been registered with a `QueueManager`.
@@ -264,6 +274,16 @@ Basalt plugin that registers a `QueueManager` in the container under the `QUEUE`
 | `connection` | `string \| ConnectionOptions` | No | — | Redis URL (`redis://…` or `rediss://…`) or ioredis options. With a value → BullMQ driver; without one → sync driver. |
 | `driver` | `QueueDriver` | No | — | Custom driver — takes precedence over `connection`. |
 | `workers` | `{ queue: string; concurrency?: number }[]` | No | `[]` | Queues whose workers start in this process on boot. |
+| `onUnsupported` | `'throw' \| 'warn' \| 'ignore'` | No | `'warn'` | What happens when a dispatch uses an option the active driver cannot honour (a delay on Kafka, a priority on SQS). `'warn'` logs once per job+feature and proceeds; `'throw'` raises `UnsupportedJobOptionError` — set it in production when the option is load-bearing; `'ignore'` is silent. |
+| `removeOnComplete` | `JobRetention` | No | driver default (BullMQ keeps the last `1000`) | Default Redis retention for completed jobs. A job's own `removeOnComplete` wins. |
+| `removeOnFail` | `JobRetention` | No | driver default (BullMQ `false` — keep all) | Default retention for failed jobs. Keeping all is deliberate (inspection and `queue:retry`); set e.g. `{ age: '14d' }` so failures don't grow unbounded. |
+| `onError` | `(error, { queue, source }) => void` | No | `console.error` with context | Infra faults from the driver's broker client (Redis down). Forwarded to the driver built from `connection`; ignored when you pass your own `driver`. |
+| `onJobFailed` | `({ queue, job, jobId?, error }) => void` | No | `console.error` with context | A job exhausted its retries. Forwarded to the driver built from `connection`; ignored when you pass your own `driver`. |
+
+**Failure callbacks work on the shorthand path.** `queuePlugin({ connection, onError,
+onJobFailed })` forwards both to the BullMQ driver it builds. They are ignored when you
+supply your own `driver` — that driver owns its own callbacks, so configure them there.
+See [Failure hooks](#failure-hooks) below.
 
 ```ts
 import { QUEUE } from '@basaltkit/queue'
@@ -274,12 +294,23 @@ const manager = app.container.get(QUEUE) // get the QueueManager from the contai
 
 | Method | Signature | Description |
 |---|---|---|
-| `constructor` | `new QueueManager(driver: QueueDriver)` | Creates the manager over a driver. |
+| `constructor` | `new QueueManager(driver: QueueDriver, options?: QueueManagerOptions)` | Creates the manager over a driver. |
 | `register` | `(job) => this` | Registers a job and wires its `dispatch`. |
-| `dispatch` | `<T>(job, payload: T, options?: DispatchOptions) => Promise<void>` | Validates and puts the job on the queue. Auto-registers the job if not registered yet. |
+| `dispatch` | `<T>(job, payload: T, options?: DispatchOptions) => Promise<void>` | Validates, snapshots the request context, checks the driver's capabilities, and enqueues. Auto-registers the job if not registered yet. |
 | `work` | `(queue = 'default', { concurrency? }?) => void` | Starts a worker for the queue (no-op on the sync driver). |
+| `stats` | `(queue = 'default') => Promise<QueueStats \| undefined>` | Job counts per state, or `undefined` when the driver can't introspect (sync). |
+| `retryFailed` | `(queue = 'default', { limit? }?) => Promise<number \| undefined>` | Re-enqueues failed jobs; returns the count, or `undefined` when the driver doesn't support it. |
 | `close` | `() => Promise<void>` | Closes workers and connections. |
 
+`QueueManagerOptions`:
+
+| Option | Type | Default | Purpose |
+|---|---|---|---|
+| `onUnsupported` | `'throw' \| 'warn' \| 'ignore'` | `'warn'` | Same policy as the plugin option. |
+| `warn` | `(message: string) => void` | `console.warn` | Where `'warn'` diagnostics go — point it at your logger. |
+| `removeOnComplete` | `JobRetention` | driver default | Default retention for completed jobs. |
+| `removeOnFail` | `JobRetention` | driver default | Default retention for failed jobs. |
+
 ### `queuedOn<T>(bus, manager, event, handler, options?): () => void`
 
 Creates the event→job bridge. Returns the subscription cancel function.
@@ -294,17 +325,94 @@ Creates the event→job bridge. Returns the subscription cancel function.
 
 ### Drivers
 
-- **`class SyncQueueDriver`** — runs inline on `dispatch`, honors `attempts` (immediate retry). Public property `executed: { queue, jobName, attempts }[]` with the execution history. For testing and dev without Redis.
-- **`class BullmqQueueDriver`** — production over Redis. `new BullmqQueueDriver({ connection })`, where `connection` is a Redis URL or ioredis options (`BullmqDriverOptions`). Completed jobs are cleaned up (keeps 1000); failed jobs are kept.
-- **`interface QueueDriver`** (Advanced) — contract for custom drivers: `setExecutor(executor)`, `add(queue, jobName, data, options: AddJobOptions)`, `startWorker(queue, { concurrency? })`, `close()`. Helper types: `AddJobOptions`, `JobExecutor`.
+- **`class SyncQueueDriver`** — runs inline on `dispatch`, honors `attempts` (immediate retry). Public property `executed: { queue, jobName, attempts }[]` with the execution history (capped at 1000 entries). For testing and dev without Redis.
+- **`class BullmqQueueDriver`** — production over Redis; see the options table below.
+- **`interface QueueDriver`** (Advanced) — contract for custom drivers: `setExecutor(executor)`, `add(queue, jobName, data, options: AddJobOptions)`, `startWorker(queue, { concurrency? })`, optional `stats(queue)` / `retryFailed(queue, { limit? })`, `close()`, plus the optional `name` and `capabilities` fields. Helper types: `AddJobOptions`, `JobExecutor`, `QueueStats`, `DriverCapabilities`.
+
+#### `new BullmqQueueDriver(options: BullmqDriverOptions)`
+
+| Option | Type | Default | Purpose |
+|---|---|---|---|
+| `connection` | `string \| ConnectionOptions` | — (required) | Redis URL (`redis://…`, `rediss://…` → TLS) or ioredis options. |
+| `onError` | `(error: unknown, info: { queue: string; source: 'worker' \| 'queue' }) => void` | `console.error` with the queue name and source | Infra errors from BullMQ's `Worker`/`Queue` EventEmitters (Redis down, connection reset). Node makes an unlistened `'error'` event **fatal**, so the driver always attaches a listener — override this to route the fault into your logger/alerting. |
+| `onJobFailed` | `(info: { queue: string; job: string; jobId?: string; error: unknown }) => void` | `console.error` naming the job, its id and the queue | A job **exhausted its retries** (BullMQ's `'failed'` event). Without it, permanently-failed jobs are only visible by polling `queue:stats`. |
+
+Retention defaults applied by this driver when neither the job nor the plugin sets one:
+completed → keep the last `1000`; failed → keep all.
+
+#### `DriverCapabilities`
+
+Each driver declares what its backend honours — `{ delayed, priority, retries, backoff }`, all
+`boolean`. `QueueManager.dispatch` compares a dispatch's options against them and applies
+`onUnsupported`. A driver that omits `capabilities` is treated as fully capable.
+
+| Driver | `delayed` | `priority` | `retries` | `backoff` |
+|---|:---:|:---:|:---:|:---:|
+| `bullmq` | ✅ | ✅ | ✅ | ✅ |
+| `sync` | ❌ | ❌ | ✅ | ❌ |
+| [`rabbitmq`](https://www.npmjs.com/package/@basaltkit/queue-rabbitmq) | ✅ | ✅ | ✅ | ✅ |
+| [`sqs`](https://www.npmjs.com/package/@basaltkit/queue-sqs) | ✅ (≤ 15 min) | ❌ | ✅ | ✅ |
+| [`kafka`](https://www.npmjs.com/package/@basaltkit/queue-kafka) | ❌ | ❌ | ✅ | ❌ |
+
+### Failure hooks
+
+The queue has no hook bus of its own — failures are reported through driver callbacks, and every
+one of them has a **non-silent default**, so nothing disappears if you configure nothing.
+
+| Hook | Where it lives | Receives | Default when unset |
+|---|---|---|---|
+| `onError` | `QueuePluginOptions` or `BullmqDriverOptions` | `(error, { queue, source: 'worker' \| 'queue' })` | `console.error` with queue + source |
+| `onJobFailed` | `QueuePluginOptions` or `BullmqDriverOptions` | `({ queue, job, jobId?, error })` | `console.error` naming the job |
+| `warn` | `QueueManagerOptions` | `(message: string)` | `console.warn`, once per job+feature |
+| `onError` | [`RabbitmqDriverOptions`](https://www.npmjs.com/package/@basaltkit/queue-rabbitmq) | `(error, { source: 'connection' \| 'channel' })` | `console.error` |
+| `onError` | [`KafkaDriverOptions`](https://www.npmjs.com/package/@basaltkit/queue-kafka) | `(error, { source: 'consumer' \| 'producer'; queue? })` | `console.error` |
+| `onError` | [`SqsDriverOptions`](https://www.npmjs.com/package/@basaltkit/queue-sqs) | `(error, { queue })` | `console.error`, then an `errorPauseMs` pause |
+
+Only the BullMQ driver has `onJobFailed`; the broker drivers route an exhausted job to their own
+dead-letter destination (`q.dead`, `<topic>.dead`, `<queue>-dead`) instead.
+
+Override the BullMQ callbacks right on the plugin — `queuePlugin({ connection })` forwards
+them to the driver it builds:
+
+```ts
+import { queuePlugin } from '@basaltkit/queue'
+
+queuePlugin({
+  connection: process.env.REDIS_URL!,
+  jobs: [SendWelcomeEmail],
+  workers: [{ queue: 'default', concurrency: 5 }],
+  onError: (error, { queue, source }) => logger.error({ err: error, queue, source }, 'queue infra error'),
+  onJobFailed: ({ queue, job, jobId, error }) =>
+    logger.error({ err: error, queue, job, jobId }, 'job failed permanently'),
+})
+```
+
+Passing your own `driver` bypasses that forwarding — a supplied driver owns its callbacks,
+so set them in its constructor:
+
+```ts
+import { queuePlugin, BullmqQueueDriver } from '@basaltkit/queue'
+
+queuePlugin({
+  jobs: [SendWelcomeEmail],
+  driver: new BullmqQueueDriver({
+    connection: process.env.REDIS_URL!,
+    onError: (error, { queue, source }) => logger.error({ err: error, queue, source }, 'queue infra error'),
+  }),
+})
+```
 
 ### Exported errors
 
-| Class | Code | When it occurs |
+| Error | Code | When |
 |---|---|---|
-| `JobValidationError` | `JOB_INVALID` | Payload doesn't pass the `schema` (has `.job` and `.issues`). |
-| `JobNotRegisteredError` | `QUEUE_JOB_NOT_REGISTERED` | `dispatch` before registering the job with a manager. |
-| `UnknownJobError` | `QUEUE_UNKNOWN_JOB` | The job reached the worker but isn't registered in that process. |
+| `JobValidationError` | `JOB_INVALID` | The payload failed the job's `schema` — on `dispatch` and again on the worker. Carries `.job` and `.issues`. |
+| `JobNotRegisteredError` | `QUEUE_JOB_NOT_REGISTERED` | `job.dispatch()` was called before the job was registered in a `QueueManager`. |
+| `UnknownJobError` | `QUEUE_UNKNOWN_JOB` | A job reached the worker but is not registered in that process — producer and worker registered different job lists. |
+| `UnsupportedJobOptionError` | `QUEUE_UNSUPPORTED_OPTION` | With `onUnsupported: 'throw'`, a dispatch used an option the active driver's `capabilities` do not include. `status = 500`. |
+
+Errors thrown outside these classes come from the driver's client (ioredis, amqplib, kafkajs,
+the AWS SDK) and reach you through that driver's `onError`.
 
 ### Token
 
@@ -327,9 +435,23 @@ The data passed to `dispatch` doesn't match the `schema`. The error includes `is
 **In dev, `delay` doesn't work.**
 The sync driver always runs immediately. Delays, timed backoff, and priority only have a real effect with the BullMQ driver (with `connection`).
 
+**`[basalt/queue] The "sync" driver does not support …` in the logs.**
+The `onUnsupported` policy caught a dispatch option the active driver can't honour. It warns once per job+feature. Either switch to a driver that supports it, drop the option, or set `onUnsupported: 'ignore'` if you accept the degradation.
+
+**A job failed for good and nothing was logged.**
+Only the BullMQ driver reports exhausted jobs, via `onJobFailed` (default `console.error`). If you replaced it with a no-op, you removed the only signal. On the broker drivers, inspect the dead-letter destination (`q.dead`, `<topic>.dead`, `<queue>-dead`).
+
 **Do I need Redis to run the tests?**
 No. Without `connection`, the plugin uses `SyncQueueDriver`. You can also instantiate the driver directly and inspect `driver.executed`.
 
+## Sync driver semantics
+
+The inline sync driver (the default without a `connection`) is at-most-once:
+handler errors reject `dispatch()` and an exhausted job is lost. Selecting it
+implicitly in production logs a boot warning — pass `driver: new
+SyncQueueDriver()` to opt in deliberately. Its `executed[]` history is capped
+at 1000 entries.
+
 ## How it connects to other modules
 
 - **`@basaltkit/core`** — provides `createApp`/`definePlugin` (`queuePlugin` is a core plugin), the ALS context (`runWithContext`/`ctx`) propagated to workers, `parseDuration` (formats `'30s'`, `'10m'`), and the base `BasaltError` class.

package/dist/drivers/sync.d.ts

@@ -3,6 +3,15 @@ import type { AddJobOptions, JobExecutor, QueueDriver } from '../driver.js';
  * Synchronous driver: executes the job inline on dispatch, honoring `attempts`
  * (immediate retry). It is the driver for tests and Redis-less dev — the
  * equivalent of Laravel's `sync` queue driver.
+ *
+ * Semantics to be aware of (they differ from a broker-backed driver):
+ * - **At-most-once.** A job that exhausts its inline retries is LOST — there is
+ *   no persistence and no later redelivery.
+ * - **Errors propagate to the dispatcher.** `job.dispatch()` rejects when the
+ *   handler fails (useful in tests/dev; a broker driver would return
+ *   immediately and retry in the background).
+ * Deploying this driver to production is almost always a misconfiguration —
+ * `queuePlugin` warns when it is selected by default there.
  */
 export declare class SyncQueueDriver implements QueueDriver {
     readonly name = "sync";
@@ -22,5 +31,8 @@ export declare class SyncQueueDriver implements QueueDriver {
     setExecutor(executor: JobExecutor): void;
     add(queue: string, jobName: string, data: unknown, options: AddJobOptions): Promise<void>;
     startWorker(): void;
+    /** Appends to `executed[]`, evicting the oldest entries past the cap so a
+     * long-running process on this driver cannot leak memory unboundedly. */
+    private record;
     close(): Promise<void>;
 }

package/dist/drivers/sync.js

@@ -1,7 +1,18 @@
+/** `executed[]` keeps at most this many entries (oldest evicted first). */
+const EXECUTED_HISTORY_LIMIT = 1000;
 /**
  * Synchronous driver: executes the job inline on dispatch, honoring `attempts`
  * (immediate retry). It is the driver for tests and Redis-less dev — the
  * equivalent of Laravel's `sync` queue driver.
+ *
+ * Semantics to be aware of (they differ from a broker-backed driver):
+ * - **At-most-once.** A job that exhausts its inline retries is LOST — there is
+ *   no persistence and no later redelivery.
+ * - **Errors propagate to the dispatcher.** `job.dispatch()` rejects when the
+ *   handler fails (useful in tests/dev; a broker driver would return
+ *   immediately and retry in the background).
+ * Deploying this driver to production is almost always a misconfiguration —
+ * `queuePlugin` warns when it is selected by default there.
  */
 export class SyncQueueDriver {
     name = 'sync';
@@ -19,18 +30,26 @@ export class SyncQueueDriver {
         for (let attempt = 1; attempt <= options.attempts; attempt++) {
             try {
                 await this.executor?.(jobName, data);
-                this.executed.push({ queue, jobName, attempts: attempt });
+                this.record({ queue, jobName, attempts: attempt });
                 return;
             }
             catch (error) {
                 lastError = error;
             }
         }
-        this.executed.push({ queue, jobName, attempts: options.attempts });
+        this.record({ queue, jobName, attempts: options.attempts });
         throw lastError;
     }
     startWorker() {
         // no-op: add() already executes inline
     }
+    /** Appends to `executed[]`, evicting the oldest entries past the cap so a
+     * long-running process on this driver cannot leak memory unboundedly. */
+    record(entry) {
+        this.executed.push(entry);
+        if (this.executed.length > EXECUTED_HISTORY_LIMIT) {
+            this.executed.splice(0, this.executed.length - EXECUTED_HISTORY_LIMIT);
+        }
+    }
     async close() { }
 }

package/dist/index.d.ts

@@ -38,5 +38,17 @@ export interface QueuePluginOptions {
      * and retries) — set e.g. `{ age: '14d' }` so failures don't grow unbounded.
      */
     removeOnFail?: JobRetention;
+    /**
+     * Infra errors from the driver's broker client (e.g. Redis down). Forwarded
+     * to the driver built from `connection`; default `console.error` with context.
+     * Ignored when you pass your own `driver` — configure it on the driver then.
+     */
+    onError?: BullmqDriverOptions['onError'];
+    /**
+     * A job exhausted its retries. Forwarded to the driver built from
+     * `connection`; default `console.error` with context. Ignored when you pass
+     * your own `driver` — configure it on the driver then.
+     */
+    onJobFailed?: BullmqDriverOptions['onJobFailed'];
 }
 export declare function queuePlugin(options?: QueuePluginOptions): import("@basaltkit/core").BasaltPlugin<unknown>;

package/dist/index.js

@@ -14,10 +14,29 @@ export function queuePlugin(options = {}) {
         register({ container }) {
             registerQueueCommands(container);
             container.singleton(QUEUE, () => {
-                const driver = options.driver ??
-                    (options.connection
-                        ? new BullmqQueueDriver({ connection: options.connection })
-                        : new SyncQueueDriver());
+                let driver = options.driver;
+                if (!driver) {
+                    if (options.connection) {
+                        driver = new BullmqQueueDriver({
+                            connection: options.connection,
+                            ...(options.onError !== undefined ? { onError: options.onError } : {}),
+                            ...(options.onJobFailed !== undefined ? { onJobFailed: options.onJobFailed } : {}),
+                        });
+                    }
+                    else {
+                        driver = new SyncQueueDriver();
+                        if (process.env['NODE_ENV'] === 'production') {
+                            // The silent default without a Redis connection is the inline sync
+                            // driver: at-most-once, no background retries, handler errors
+                            // propagate into the dispatching request. Deliberate sync use in
+                            // production stays possible — pass `driver: new SyncQueueDriver()`
+                            // explicitly to silence this.
+                            console.warn('[basalt:queue] No `connection` (Redis) configured — falling back to the inline sync driver. ' +
+                                'Jobs run at-most-once inside the dispatching request and are lost on failure. ' +
+                                'Configure a Redis `connection` for production, or pass `driver: new SyncQueueDriver()` to opt in explicitly.');
+                        }
+                    }
+                }
                 const manager = new QueueManager(driver, {
                     ...(options.onUnsupported !== undefined ? { onUnsupported: options.onUnsupported } : {}),
                     ...(options.removeOnComplete !== undefined ? { removeOnComplete: options.removeOnComplete } : {}),

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@basaltkit/queue",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "description": "Basalt queues on top of BullMQ: declarative jobs with Zod payloads, context propagation (tenant/requestId) to workers and a sync driver for tests.",
   "license": "MIT",
   "type": "module",
@@ -15,8 +15,8 @@
   ],
   "dependencies": {
     "bullmq": "^6.2.1",
-    "@basaltkit/core": "^1.3.0",
-    "@basaltkit/events": "^1.0.1"
+    "@basaltkit/events": "^1.1.0",
+    "@basaltkit/core": "^1.3.0"
   },
   "devDependencies": {
     "@types/node": "^26.3.0",