@warlock.js/queue 5.32.0 → 5.34.0

package/llms-full.txt

@@ -13,7 +13,7 @@ description: 'Configure `@warlock.js/queue`: the declarative `src/config/queue.t
 
 # Configure the queue
 
-Redis is required. The config is declarative; the connector applies it.
+BullMQ is the default and requires Redis. Select `driver: "local"` for durable SQLite without Redis (Node >= 22.15), or `driver: "memory"` for tests; see `local-queues` for working examples, leases, and cancellation. The config is declarative; the connector applies it.
 
 ```ts title="src/config/queue.ts"
 import type { QueueConfig } from "@warlock.js/queue";
@@ -168,6 +168,131 @@ Jobs are not run — call the handler to test it. Opt out with `tests.fakes: { q
 - Defining the same name again replaces the handler (this is what keeps dev reloads working), so keep names unique.
 
 
+## local-queues  `@warlock.js/queue/local-queues/SKILL.md`
+
+---
+name: local-queues
+description: 'Run @warlock.js/queue without Redis: driver: "local" for durable single-host SQLite jobs using node:sqlite on Node >= 22.15, or driver: "memory" for real handler execution in tests. Covers local.path, leases, crash recovery, polling, concurrency, cancellation, graceful shutdown, and at-least-once delivery. Use for local/desktop background jobs, testing workers without Redis, or configuring SQLite queues.'
+---
+
+# Local and memory queues
+
+Select a backend in the usual queue config; register `queueConnector()` as usual.
+No extra dependency is needed. Existing config defaults to BullMQ.
+
+```ts title="src/config/queue.ts"
+import type { QueueConfig } from "@warlock.js/queue";
+
+export default {
+  driver: "local",
+  prefix: "my-app",
+  local: {
+    path: "/var/lib/my-app/jobs.sqlite",
+    leaseDuration: 30_000,
+    pollInterval: 100,
+  },
+  workers: { concurrency: 3, shutdownTimeout: 5000 },
+  defaultJobOptions: { attempts: 3, backoff: { type: "exponential", delay: 1000 } },
+} satisfies QueueConfig;
+```
+
+Node's built-in SQLite is available without flags from Node 22.13, still
+experimental in Node 22.15. A no-flags import/query was verified on Node
+22.15.0. Require Node >= 22.15 with `node:sqlite` available; unavailable
+runtimes throw on selecting local storage. Sources:
+[Node changelog](https://nodejs.org/en/blog/release/v22.13.0),
+[Node 22.15 API](https://nodejs.org/download/release/v22.15.0/docs/api/sqlite.html).
+BullMQ/memory do not load SQLite.
+
+## Define, dispatch, inspect, and retry
+
+```ts
+import { defineJob, failedJobs, retryFailedJob } from "@warlock.js/queue";
+
+const resize = defineJob({
+  name: "images.resize",
+  async handle(payload: { imageId: string }, ctx) {
+    ctx.signal?.throwIfAborted();
+    await ctx.progress(50);
+    await ctx.log(`resizing ${payload.imageId}`);
+    return { ready: true };
+  },
+});
+
+const { id } = await resize.dispatch(
+  { imageId: "42" },
+  {
+    jobId: "resize-42",
+    delay: "2s",
+    priority: 1,
+    attempts: 4,
+    backoff: { type: "fixed", delay: 500 },
+  },
+);
+const snapshot = await resize.find(id);
+console.log(snapshot?.state, snapshot?.result);
+for (const job of await failedJobs()) await job.retry();
+// For an id known to be failed: await retryFailedJob(id);
+```
+
+The same API supports fixed/numeric or exponential backoff, priority/FIFO,
+retention, context propagation, `UnrecoverableJobError`, and failed-job retry.
+Payload/context/result/progress are JSON data. Attempts include the first
+execution; crashed leases re-deliver without consuming a settled attempt.
+
+## Memory tests and scripts
+
+```ts
+import {
+  closeQueue,
+  defineJob,
+  setDispatchInterceptor,
+  setQueueConfig,
+  startWorkers,
+} from "@warlock.js/queue";
+
+setDispatchInterceptor(undefined); // opt out of queueFake for real execution
+setQueueConfig({ driver: "memory", workers: { concurrency: 2 } });
+const double = defineJob({ name: "numbers.double", handle: (value: number) => value * 2 });
+await startWorkers();
+const { id } = await double.dispatch(21);
+try {
+  const deadline = Date.now() + 5000;
+  while ((await double.find(id))?.state !== "completed") {
+    if (Date.now() >= deadline) throw new Error("Job did not complete");
+    await new Promise((resolve) => setTimeout(resolve, 10));
+  }
+  console.log((await double.find(id))?.result); // 42
+} finally {
+  await closeQueue({ timeout: 1000 });
+}
+```
+
+Memory storage belongs to one process and is cleared by `closeQueue()`.
+Use temporary SQLite files for restart/crash tests. `workers.enabled: false`
+and process-role gating work for every backend.
+
+## Durability and shutdown rules
+
+- Use the same absolute file path, namespace (`prefix`), and job definitions
+  in every producer/worker. Default file: `.warlock/queue.sqlite`, relative
+  to cwd. Parents are created; use writable local disk, never a network share.
+- SQLite WAL plus FULL synchronous writes persist jobs. Short synchronous
+  transactions claim jobs atomically. A unique token fences every renewal,
+  progress update, and completion; an expired owner cannot write.
+- The lease defaults to 30 seconds, renews every third of its duration, and
+  must be at least 30 ms. Polling defaults to 100 ms, minimum 1 ms. Concurrency
+  is per queue per process. Avoid blocking the event loop longer than the lease.
+- Shutdown stops claims immediately, finishes handlers within the timeout,
+  then aborts `ctx.signal` and releases remaining leases. Observe cancellation
+  in handlers and operations. Await close before changing config or restarting.
+- A crashed process's jobs re-deliver after lease expiry. Delivery is at least
+  once: make external side effects idempotent. Expired/released handlers that
+  ignore cancellation can keep running even though queue writes are fenced.
+- `getQueue()` and bull-board require BullMQ. Use `find()` / `failedJobs()`
+  with local/memory queues. Prefer BullMQ for multi-host/high-volume workloads.
+
+
 ## manage-failed-jobs  `@warlock.js/queue/manage-failed-jobs/SKILL.md`
 
 ---
@@ -249,12 +374,12 @@ description: "@warlock.js/queue overview for orientation and choosing the focuse
 
 # `@warlock.js/queue` — overview
 
-Durable background jobs. Jobs are stored in **Redis** (required), retried on failure, and processed by workers — inside the app process by default.
+Durable background jobs. Jobs are stored in **Redis** (default), local SQLite without Redis, or memory for tests, retried on failure, and processed by workers — inside the app process by default.
 
 ## Mental model
 
 - **Job definition** — `defineJob({ name, handle })` registers a handler by name and returns a typed job.
-- **Dispatch** — `job.dispatch(payload, options)` stores the job in Redis. Any process with the same definition and a worker can run it.
+- **Dispatch** — `job.dispatch(payload, options)` stores the job in the configured backend. Any process with the same definition and a worker can run it.
 - **Worker** — one per queue name, started by `queueConnector()` (or `startWorkers()`); routes each job to the handler with its name.
 - **Shutdown** — workers stop taking jobs, active jobs finish (up to `workers.shutdownTimeout`), then connections close.
 
@@ -264,6 +389,7 @@ Durable background jobs. Jobs are stored in **Redis** (required), retried on fai
 
 ## Skills index
 
+- `local-queues` — SQLite/memory configuration, leases, crash recovery, cancellation.
 - `configure-queue` — `src/config/queue.ts`, `queueConnector()`, workers on/off, shutdown.
 - `define-jobs` — `defineJob`, `dispatch` options, retries, progress, `find`.
 - `manage-failed-jobs` — `failedJobs`, `retryFailedJob`, the bull-board dashboard.

package/llms.txt

@@ -2,12 +2,13 @@
 
 > Package: `@warlock.js/queue`
 
-> Durable background jobs for Warlock.js on BullMQ + Redis: defineJob, dispatch with delay/priority, retries with backoff, progress, failed-job access, in-process workers with graceful shutdown.
+> Background jobs for Warlock.js on BullMQ + Redis, local SQLite, or memory: defineJob, dispatch with delay/priority, retries with backoff, progress, failed-job access, in-process workers with graceful shutdown.
 
 ## Skills
 
 - [configure-queue](@warlock.js/queue/configure-queue/SKILL.md): Configure `@warlock.js/queue`: the declarative `src/config/queue.ts` (`QueueConfig` — `connection`, `prefix`, `defaultQueue`, `defaultJobOptions`, `workers: { enabled, concurrency, shutdownTimeout }`), registering `queueConnector()` in `warlock.config.ts > connectors`, running a dispatch-only process, and graceful shutdown. Programmatic `setQueueConfig` / `startWorkers` / `closeQueue` for scripts and tests. Triggers: `QueueConfig`, `queueConnector`, `setQueueConfig`, `startWorkers`, `closeQueue`, `workers.enabled`, `shutdownTimeout`; "configure the queue", "connect BullMQ to Redis", "disable workers in the web process", "graceful shutdown of jobs". Skip: writing jobs — the `define-jobs` topic.
 - [define-jobs](@warlock.js/queue/define-jobs/SKILL.md): Define and dispatch background jobs with `@warlock.js/queue`: `defineJob({ name, queue?, attempts?, backoff?, removeOnComplete?, removeOnFail?, handle(payload, ctx) })` returns a typed job; `job.dispatch(payload, { delay, priority, jobId, attempts, backoff })`; the handler context (`id`, `attempt`, `maxAttempts`, `progress()`, `log()`); reading a job with `job.find(id)`. Triggers: `setDispatchInterceptor`, `queueFake`, `defineJob`, `.dispatch(`, `ctx.progress`, `JobContext`, `DispatchOptions`, `backoff`, `attempts`, `priority`, `jobId`; "run this in the background", "retry with backoff", "delay a job", "job progress", "idempotent dispatch". Skip: config and workers — the `configure-queue` topic; failed jobs — the `manage-failed-jobs` topic.
+- [local-queues](@warlock.js/queue/local-queues/SKILL.md): Run @warlock.js/queue without Redis: driver: "local" for durable single-host SQLite jobs using node:sqlite on Node >= 22.15, or driver: "memory" for real handler execution in tests. Covers local.path, leases, crash recovery, polling, concurrency, cancellation, graceful shutdown, and at-least-once delivery. Use for local/desktop background jobs, testing workers without Redis, or configuring SQLite queues.
 - [manage-failed-jobs](@warlock.js/queue/manage-failed-jobs/SKILL.md): Inspect and retry failed `@warlock.js/queue` jobs: `failedJobs({ queue, start, end })` returns `FailedJob[]` (`id`, `name`, `payload`, `attemptsMade`, `failedReason`, `stacktrace`, `failedAt`, `retry()`), `retryFailedJob(id, { queue })` (throws `FailedJobNotFoundError`), and the bull-board UI — `warlock add bull-board` (requires: ["queue"], so it installs queue first when missing) writes a `dashboard: { enabled, path, middleware }` block to `src/config/queue.ts`; a guard middleware is required in production or boot throws `QueueDashboardUnguardedError`. Manual `queueDashboard(server, { basePath, queues, middleware })` also available — needs `@bull-board/api` + `@bull-board/fastify`, loaded only on call, missing ones throw `QueueDashboardDependencyError`. Triggers: `failedJobs`, `retryFailedJob`, `queueDashboard`, `bull-board`, `dashboard.enabled`, `QueueDashboardUnguardedError`; "list failed jobs", "retry a failed job", "queue dashboard", "job admin UI". Skip: defining retries — the `define-jobs` topic.
 - [overview](@warlock.js/queue/overview/SKILL.md): @warlock.js/queue overview for orientation and choosing the focused skill for the task.
 - [queue-notifications](@warlock.js/queue/queue-notifications/SKILL.md): Send `@warlock.js/notifications` `.queue()` deliveries through BullMQ with `bullmqQueue({ queue?, attempts?, backoff? })` from `@warlock.js/notifications` — put it in the `queue` slot of `src/config/notifications.ts`; the queue workers deliver. Honours `SendOptions.delay` (number = seconds, or "10m"), retries a failing `channel.send`, and fails at once for a channel missing from the worker config. Triggers: `bullmqQueue`, `NotificationConfig.queue`, `QueuePackageNotInstalledError`; "queue notifications with BullMQ", "retry notification delivery", "delayed notification". Skip: the herald backend — the `queue-notifications` topic of the `warlock-js-notifications` skill.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@warlock.js/queue",
-  "description": "Durable background jobs for Warlock.js on BullMQ + Redis: defineJob, dispatch with delay/priority, retries with backoff, progress, failed-job access, in-process workers with graceful shutdown.",
+  "description": "Background jobs for Warlock.js on BullMQ + Redis, local SQLite, or memory: defineJob, dispatch with delay/priority, retries with backoff, progress, failed-job access, in-process workers with graceful shutdown.",
   "warlock": {
     "environment": "server"
   },
@@ -11,8 +11,8 @@
   "peerDependencies": {
     "@bull-board/api": "^9.0.0",
     "@bull-board/fastify": "^9.0.0",
-    "@warlock.js/core": "5.32.0",
-    "@warlock.js/logger": "5.32.0"
+    "@warlock.js/core": "5.34.0",
+    "@warlock.js/logger": "5.34.0"
   },
   "peerDependenciesMeta": {
     "@bull-board/api": {
@@ -38,7 +38,7 @@
   ],
   "author": "hassanzohdy",
   "license": "MIT",
-  "version": "5.32.0",
+  "version": "5.34.0",
   "main": "./cjs/index.cjs",
   "module": "./esm/index.mjs",
   "types": "./esm/index.d.mts",

package/skills/configure-queue/SKILL.md

@@ -5,7 +5,7 @@ description: 'Configure `@warlock.js/queue`: the declarative `src/config/queue.t
 
 # Configure the queue
 
-Redis is required. The config is declarative; the connector applies it.
+BullMQ is the default and requires Redis. Select `driver: "local"` for durable SQLite without Redis (Node >= 22.15), or `driver: "memory"` for tests; see `local-queues` for working examples, leases, and cancellation. The config is declarative; the connector applies it.
 
 ```ts title="src/config/queue.ts"
 import type { QueueConfig } from "@warlock.js/queue";

package/skills/index.md

@@ -1,10 +1,10 @@
 ---
-description: 'Durable background jobs on BullMQ + Redis: retries, delays, priorities, workers, failed-job recovery. Exports `defineJob`, `queueConnector`, `startWorkers`, `closeQueue`, `getQueue`, `failedJobs`, `retryFailedJob`, `mountQueueDashboard`, `QueueConfig`, `QueueJob`. Use for: "run this in the background", "retry a failed job", "delay a job", "survive restarts", "job dashboard". Not this package: cron or recurring schedules → @warlock.js/scheduler; broker pub/sub messaging → @warlock.js/herald; core `Queue` is an in-memory batcher.'
+description: 'Background jobs on BullMQ + Redis, local SQLite, or memory: retries, delays, priorities, workers, failed-job recovery. Exports `defineJob`, `queueConnector`, `startWorkers`, `closeQueue`, `getQueue`, `failedJobs`, `retryFailedJob`, `mountQueueDashboard`, `QueueConfig`, `QueueJob`. Use for: "run this in the background", "retry a failed job", "delay a job", "survive restarts", "job dashboard". Not this package: cron or recurring schedules → @warlock.js/scheduler; broker pub/sub messaging → @warlock.js/herald; core `Queue` is an in-memory batcher.'
 ---
 
 # @warlock.js/queue
 
-Jobs are stored in Redis (required; not bundled), retried on failure, and run by workers, in the app process by default. `defineJob({ name, handle })` registers a handler and returns a typed job; `job.dispatch(payload, options)` enqueues it. Any process with the same definition and a worker can run it. Server-only.
+Jobs are stored in Redis (default; not bundled), local SQLite without Redis, or memory for tests, retried on failure, and run by workers, in the app process by default. `defineJob({ name, handle })` registers a handler and returns a typed job; `job.dispatch(payload, options)` enqueues it. Any process with the same definition and a worker can run it. Server-only.
 
 ## The 80% path
 
@@ -22,3 +22,5 @@ Jobs are stored in Redis (required; not bundled), retried on failure, and run by
 - Guard the dashboard with middleware; an unguarded one is refused on purpose.
 - Core `Queue` is an in-memory batcher, lost on exit; use this package when work must survive.
 - Do not value-import from client code.
+
+Use the `local-queues` topic for `driver: "local"` / `"memory"`, Node requirements, leases, crash recovery, and cancellation.

package/skills/local-queues/SKILL.md

@@ -0,0 +1,121 @@
+---
+name: local-queues
+description: 'Run @warlock.js/queue without Redis: driver: "local" for durable single-host SQLite jobs using node:sqlite on Node >= 22.15, or driver: "memory" for real handler execution in tests. Covers local.path, leases, crash recovery, polling, concurrency, cancellation, graceful shutdown, and at-least-once delivery. Use for local/desktop background jobs, testing workers without Redis, or configuring SQLite queues.'
+---
+
+# Local and memory queues
+
+Select a backend in the usual queue config; register `queueConnector()` as usual.
+No extra dependency is needed. Existing config defaults to BullMQ.
+
+```ts title="src/config/queue.ts"
+import type { QueueConfig } from "@warlock.js/queue";
+
+export default {
+  driver: "local",
+  prefix: "my-app",
+  local: {
+    path: "/var/lib/my-app/jobs.sqlite",
+    leaseDuration: 30_000,
+    pollInterval: 100,
+  },
+  workers: { concurrency: 3, shutdownTimeout: 5000 },
+  defaultJobOptions: { attempts: 3, backoff: { type: "exponential", delay: 1000 } },
+} satisfies QueueConfig;
+```
+
+Node's built-in SQLite is available without flags from Node 22.13, still
+experimental in Node 22.15. A no-flags import/query was verified on Node
+22.15.0. Require Node >= 22.15 with `node:sqlite` available; unavailable
+runtimes throw on selecting local storage. Sources:
+[Node changelog](https://nodejs.org/en/blog/release/v22.13.0),
+[Node 22.15 API](https://nodejs.org/download/release/v22.15.0/docs/api/sqlite.html).
+BullMQ/memory do not load SQLite.
+
+## Define, dispatch, inspect, and retry
+
+```ts
+import { defineJob, failedJobs, retryFailedJob } from "@warlock.js/queue";
+
+const resize = defineJob({
+  name: "images.resize",
+  async handle(payload: { imageId: string }, ctx) {
+    ctx.signal?.throwIfAborted();
+    await ctx.progress(50);
+    await ctx.log(`resizing ${payload.imageId}`);
+    return { ready: true };
+  },
+});
+
+const { id } = await resize.dispatch(
+  { imageId: "42" },
+  {
+    jobId: "resize-42",
+    delay: "2s",
+    priority: 1,
+    attempts: 4,
+    backoff: { type: "fixed", delay: 500 },
+  },
+);
+const snapshot = await resize.find(id);
+console.log(snapshot?.state, snapshot?.result);
+for (const job of await failedJobs()) await job.retry();
+// For an id known to be failed: await retryFailedJob(id);
+```
+
+The same API supports fixed/numeric or exponential backoff, priority/FIFO,
+retention, context propagation, `UnrecoverableJobError`, and failed-job retry.
+Payload/context/result/progress are JSON data. Attempts include the first
+execution; crashed leases re-deliver without consuming a settled attempt.
+
+## Memory tests and scripts
+
+```ts
+import {
+  closeQueue,
+  defineJob,
+  setDispatchInterceptor,
+  setQueueConfig,
+  startWorkers,
+} from "@warlock.js/queue";
+
+setDispatchInterceptor(undefined); // opt out of queueFake for real execution
+setQueueConfig({ driver: "memory", workers: { concurrency: 2 } });
+const double = defineJob({ name: "numbers.double", handle: (value: number) => value * 2 });
+await startWorkers();
+const { id } = await double.dispatch(21);
+try {
+  const deadline = Date.now() + 5000;
+  while ((await double.find(id))?.state !== "completed") {
+    if (Date.now() >= deadline) throw new Error("Job did not complete");
+    await new Promise((resolve) => setTimeout(resolve, 10));
+  }
+  console.log((await double.find(id))?.result); // 42
+} finally {
+  await closeQueue({ timeout: 1000 });
+}
+```
+
+Memory storage belongs to one process and is cleared by `closeQueue()`.
+Use temporary SQLite files for restart/crash tests. `workers.enabled: false`
+and process-role gating work for every backend.
+
+## Durability and shutdown rules
+
+- Use the same absolute file path, namespace (`prefix`), and job definitions
+  in every producer/worker. Default file: `.warlock/queue.sqlite`, relative
+  to cwd. Parents are created; use writable local disk, never a network share.
+- SQLite WAL plus FULL synchronous writes persist jobs. Short synchronous
+  transactions claim jobs atomically. A unique token fences every renewal,
+  progress update, and completion; an expired owner cannot write.
+- The lease defaults to 30 seconds, renews every third of its duration, and
+  must be at least 30 ms. Polling defaults to 100 ms, minimum 1 ms. Concurrency
+  is per queue per process. Avoid blocking the event loop longer than the lease.
+- Shutdown stops claims immediately, finishes handlers within the timeout,
+  then aborts `ctx.signal` and releases remaining leases. Observe cancellation
+  in handlers and operations. Await close before changing config or restarting.
+- A crashed process's jobs re-deliver after lease expiry. Delivery is at least
+  once: make external side effects idempotent. Expired/released handlers that
+  ignore cancellation can keep running even though queue writes are fenced.
+- `getQueue()` and bull-board require BullMQ. Use `find()` / `failedJobs()`
+  with local/memory queues. Prefer BullMQ for multi-host/high-volume workloads.

package/skills/overview/SKILL.md

@@ -5,12 +5,12 @@ description: "@warlock.js/queue overview for orientation and choosing the focuse
 
 # `@warlock.js/queue` — overview
 
-Durable background jobs. Jobs are stored in **Redis** (required), retried on failure, and processed by workers — inside the app process by default.
+Durable background jobs. Jobs are stored in **Redis** (default), local SQLite without Redis, or memory for tests, retried on failure, and processed by workers — inside the app process by default.
 
 ## Mental model
 
 - **Job definition** — `defineJob({ name, handle })` registers a handler by name and returns a typed job.
-- **Dispatch** — `job.dispatch(payload, options)` stores the job in Redis. Any process with the same definition and a worker can run it.
+- **Dispatch** — `job.dispatch(payload, options)` stores the job in the configured backend. Any process with the same definition and a worker can run it.
 - **Worker** — one per queue name, started by `queueConnector()` (or `startWorkers()`); routes each job to the handler with its name.
 - **Shutdown** — workers stop taking jobs, active jobs finish (up to `workers.shutdownTimeout`), then connections close.
 
@@ -20,6 +20,7 @@ Durable background jobs. Jobs are stored in **Redis** (required), retried on fai
 
 ## Skills index
 
+- `local-queues` — SQLite/memory configuration, leases, crash recovery, cancellation.
 - `configure-queue` — `src/config/queue.ts`, `queueConnector()`, workers on/off, shutdown.
 - `define-jobs` — `defineJob`, `dispatch` options, retries, progress, `find`.
 - `manage-failed-jobs` — `failedJobs`, `retryFailedJob`, the bull-board dashboard.