bunqueue 2.8.31 → 2.8.33

package/README.md

@@ -14,7 +14,7 @@
 
 <p align="center">
   High-performance job queue for Bun. Built for AI agents and automation.<br/>
-  Zero external dependencies. MCP-native. TypeScript-first.
+  Zero external infrastructure. MCP-native. TypeScript-first.
 </p>
 
 <p align="center">
@@ -63,15 +63,15 @@ That's it. Queue + Worker in one object. No Redis, no config, no setup.
 
 ## 🪶 Featherweight install
 
-`bun add bunqueue` pulls **7 packages and 5.4 MB** — and that includes the binary
+`bun add bunqueue` pulls **7 packages and 5.5 MB** — and that includes the binary
 queue server, the CLI, and the MCP server. Most queue libraries pull a Redis
 client and its dependency tree before you've queued a single job.
 
 | | Before (2.7.x) | Now | |
 | --- | --- | --- | --- |
-| **`node_modules`** | 93 MB | **5.4 MB** | **−94%** |
+| **`node_modules`** | 93 MB | **5.5 MB** | **−94%** |
 | **packages** | 117 | **7** | **−110** |
-| **cold install** | ~6 s | **~1.2 s** | **~5× faster** |
+| **cold install** | ~6 s | **~1.7 s** | **~3.5× faster** |
 
 Just **2 runtime dependencies** (`croner` + `msgpackr`). SQLite, S3, HTTP and
 WebSocket are all Bun built-ins. The MCP SDK is an optional peer dependency —
@@ -336,7 +336,7 @@ const app = new Bunqueue('tasks', {
 });
 ```
 
-A job with priority 1, after 5 min: 3, after 10 min: 5, … capped at 100.
+A job with priority 1 becomes eligible after 5 min, then gains +2 on every 60s tick: 3, 5, 7, … capped at 100.
 
 ### Deduplication
 
@@ -362,7 +362,7 @@ Override per-job: `await app.add('task', data, { deduplication: { id: 'my-id', t
 
 ### Debouncing
 
-Coalesce rapid same-name jobs. Only the last one in the TTL window gets processed:
+Attach a default debounce id to same-name jobs. Each job gets `debounce: { id: jobName, ttl }` unless the per-job options already set one:
 
 ```typescript
 const app = new Bunqueue('search', {
@@ -373,9 +373,11 @@ const app = new Bunqueue('search', {
 
 await app.add('search', { query: 'h' });
 await app.add('search', { query: 'he' });
-await app.add('search', { query: 'hello' });  // only this one processes
+await app.add('search', { query: 'hello' });  // all three share the debounce id 'search'
 ```
 
+The debounce id and TTL are stored on the job (BullMQ v5 compatible metadata, visible via `job.opts.debounce`), but they do not suppress duplicates by themselves in the current engine. To actually coalesce rapid duplicates, use [Deduplication](#deduplication) with `replace: true`: a duplicate arriving within the TTL window replaces the pending job, giving last-write-wins semantics.
+
 ### Rate Limiting
 
 ```typescript
@@ -385,15 +387,13 @@ const app = new Bunqueue('api', {
   rateLimit: { max: 100, duration: 1000 },
 });
 
-// Per-group rate limiting (e.g., per customer)
+// Per-group cap (e.g., per customer): with groupKey set, max limits
+// concurrently active jobs per group (duration is not applied in this mode)
 const app2 = new Bunqueue('api', {
   embedded: true,
   processor: async (job) => callAPI(job.data),
   rateLimit: { max: 10, duration: 1000, groupKey: 'customerId' },
 });
-
-app.setGlobalRateLimit(50, 1000);
-app.removeGlobalRateLimit();
 ```
 
 ### DLQ (Dead Letter Queue)
@@ -418,7 +418,7 @@ app.retryDlq();                      // retry all
 app.purgeDlq();                      // clear all
 ```
 
-Failure reasons: `explicit_fail`, `max_attempts_exceeded`, `timeout`, `stalled`, `ttl_expired`, `worker_lost`.
+Failure reasons: `explicit_fail`, `max_attempts_exceeded`, `timeout`, `stalled`, `ttl_expired`, `worker_lost`, `unknown`.
 
 ### Cron Jobs
 
@@ -471,7 +471,8 @@ await app.close(true); // force shutdown
 
 ```typescript
 const counts = await queue.getJobCountsAsync();
-// { waiting, prioritized, active, completed, failed, delayed, paused }
+// { waiting, prioritized, active, completed, failed, delayed,
+//   'waiting-children', paused }
 
 // Failed jobs (those that exhausted their attempts) are enumerable by state —
 // the failed list reflects the same jobs that `failed` counts.
@@ -661,10 +662,10 @@ bunx bunqueue-dashboard
 
 - **MCP server included** — 73 tools, 5 resources, 3 prompts. AI agents get full control out of the box
 - **BullMQ-compatible API** — Same `Queue`, `Worker`, `QueueEvents`
-- **Zero dependencies** — No Redis, no MongoDB
-- **Tiny footprint** — 5.4 MB installed, 7 packages (just 2 runtime deps: `croner` + `msgpackr`)
+- **Zero external infrastructure** — No Redis, no MongoDB
+- **Tiny footprint** — 5.5 MB installed, 7 packages (just 2 runtime deps: `croner` + `msgpackr`)
 - **SQLite persistence** — Survives restarts, WAL mode for concurrent access
-- **Up to 286K ops/sec** — [Verified benchmarks](https://bunqueue.dev/guide/benchmarks/)
+- **Up to 630K ops/sec** — [Verified benchmarks](https://bunqueue.dev/guide/benchmarks/)
 
 ## Built for AI Agents (MCP Server)
 
@@ -690,8 +691,8 @@ bunqueue is the **first job queue with native MCP support**. AI agents get a ful
 # bunqueue-mcp is a binary bundled inside the bunqueue package (not a standalone npm package),
 # so install bunqueue first, then connect Claude Code:
 bun add bunqueue
-# Since v2.8.0 the MCP SDK is an optional peer dependency. Queue-only users skip it entirely:
-# a clean install drops from 93 MB / 117 packages to 5.4 MB / 7 packages (−94%, up to ~5× faster on a cold install).
+# Since v2.8.1 the MCP SDK is an optional peer dependency. Queue-only users skip it entirely:
+# a clean install drops from 93 MB / 117 packages to 5.5 MB / 7 packages (−94%, ~3.5× faster on a cold install).
 # bunx --package=bunqueue does NOT auto-install optional peers, so install the SDK once to run the MCP server:
 bun add @modelcontextprotocol/sdk
 claude mcp add bunqueue -- bunx bunqueue-mcp
@@ -731,7 +732,7 @@ Supports **embedded** (local SQLite) and **TCP** (remote server) modes. [Full MC
 - **Agentic workflows** — agents push jobs, workers process, agents monitor results
 - Single-server deployments
 - Prototypes and MVPs
-- Moderate to high workloads (up to 286K ops/sec)
+- Moderate to high workloads (up to 630K ops/sec)
 - Teams that want to avoid Redis operational overhead
 - Embedded use cases (CLI tools, edge functions, serverless)
 
@@ -745,7 +746,7 @@ Supports **embedded** (local SQLite) and **TCP** (remote server) modes. [Full MC
 
 If you're already running Redis, BullMQ is great — battle-tested and feature-rich.
 
-bunqueue is for when you **don't want to run Redis**. SQLite with WAL mode handles surprisingly high throughput for single-node deployments (tested up to 286K ops/sec). You get persistence, priorities, delays, retries, cron jobs, and DLQ — without the operational overhead of another service.
+bunqueue is for when you **don't want to run Redis**. SQLite with WAL mode handles surprisingly high throughput for single-node deployments (tested up to 630K ops/sec on bulk push). You get persistence, priorities, delays, retries, cron jobs, and DLQ — without the operational overhead of another service.
 
 ## Install
 
@@ -765,13 +766,15 @@ bunqueue runs in two modes depending on your architecture:
 | ---------------- | ------------------------------------- | -------------------------------------------- |
 | **How it works** | Queue runs inside your process        | Standalone server, clients connect via TCP   |
 | **Setup**        | `bun add bunqueue`                    | `docker run` or `bunqueue start`             |
-| **Performance**  | 286K ops/sec                          | 149K ops/sec                                 |
+| **Performance**  | 630K ops/sec (bulk push)              | 90K ops/sec (push)                           |
 | **Best for**     | Single-process apps, CLIs, serverless | Multiple workers, separate producer/consumer |
 | **Scaling**      | Same process only                     | Multiple clients across machines             |
 
 ### Embedded Mode
 
-Everything runs in your process. No server, no network, no setup.
+Everything runs in your process. No server, no network, no setup. Without a
+data path the queue is in-memory and lost on restart: pass
+`dataPath: './data/app.db'` (or set `BUNQUEUE_DATA_PATH`) to persist jobs.
 
 ```typescript
 import { Queue, Worker } from 'bunqueue/client';
@@ -827,8 +830,8 @@ full feature parity, so producers and workers can live anywhere in your stack:
 
 | Where your code runs | Install | Status |
 | -------------------- | ------- | ------ |
-| **Node.js ≥ 20, Deno ≥ 2, Bun, Cloudflare Workers** | [`npm install bunqueue-client`](https://www.npmjs.com/package/bunqueue-client) | 58/58 e2e per runtime + 11/11 inside workerd |
-| **Python ≥ 3.9** | PyPI coming soon — today: `pip install "git+https://github.com/egeominotti/bunqueue.git#subdirectory=sdk/python"` | 44/44 e2e |
+| **Node.js ≥ 20, Deno ≥ 2, Bun, Cloudflare Workers** | [`npm install bunqueue-client`](https://www.npmjs.com/package/bunqueue-client) | 100/100 e2e per runtime + 16/16 inside workerd |
+| **Python ≥ 3.9** | PyPI coming soon — today: `pip install "git+https://github.com/egeominotti/bunqueue.git#subdirectory=sdk/python"` | 78/78 e2e |
 
 ```typescript
 // Node.js / Deno / Cloudflare Workers
@@ -909,8 +912,8 @@ SQLite handles surprisingly high throughput for single-node deployments:
 
 | Mode     | Peak Throughput | Use Case            |
 | -------- | --------------- | ------------------- |
-| Embedded | 286K ops/sec    | Same process        |
-| TCP      | 149K ops/sec    | Distributed workers |
+| Embedded | 630K ops/sec (bulk push) | Same process        |
+| TCP      | 90K ops/sec (push)       | Distributed workers |
 
 > Run `bun run bench` to verify on your hardware. [Full benchmark methodology →](https://bunqueue.dev/guide/benchmarks/)
 

package/dist/application/backgroundTasks.js

@@ -201,9 +201,11 @@ export function recover(ctx) {
     const now = Date.now();
     // === PHASE 1: Recover active jobs (were processing when server stopped) ===
     // These jobs are considered "stalled" and need to be retried or moved to DLQ
-    let activeOffset = 0;
     while (true) {
-        const activeJobs = ctx.storage.loadActiveJobs(RECOVERY_BATCH_SIZE, activeOffset);
+        // Every successfully handled row leaves state='active' (retry -> waiting, or
+        // delete -> DLQ/discard). Always read the first page: advancing OFFSET over
+        // this shrinking result set would skip up to half of the active rows.
+        const activeJobs = ctx.storage.loadActiveJobs(RECOVERY_BATCH_SIZE, 0);
         if (activeJobs.length === 0)
             break;
         for (const job of activeJobs) {
@@ -248,22 +250,20 @@ export function recover(ctx) {
                 ctx.storage.deleteJob(job.id);
             }
             else {
-                // Retry: put back in queue with backoff (uses backoffConfig if present)
+                // Persist the retry here, but let Phase 2 be the single authoritative
+                // enqueue path. Enqueuing in both phases replaces the priority-queue entry
+                // by ID but increments queued/delayed counters twice.
                 job.runAt = now + calculateBackoff(job);
-                shard.getQueue(job.queue).push(job);
-                const isDelayed = job.runAt > now;
-                shard.incrementQueued(job.id, isDelayed, job.createdAt, job.queue, job.runAt);
-                ctx.jobIndex.set(job.id, { type: 'queue', shardIdx: idx, queueName: job.queue });
                 ctx.storage.updateForRetry(job);
             }
             ctx.registerQueueName(job.queue);
         }
-        activeOffset += activeJobs.length;
         if (activeJobs.length < RECOVERY_BATCH_SIZE)
             break;
     }
     // === PHASE 2: Load pending jobs ===
     let offset = 0;
+    const corruptPendingJobs = [];
     // Load pending jobs in batches to avoid memory spikes
     while (true) {
         const jobs = ctx.storage.loadPendingJobs(RECOVERY_BATCH_SIZE, offset);
@@ -276,7 +276,10 @@ export function recover(ctx) {
             // Route the job to the DLQ rather than enqueuing it as ready (out-of-order
             // execution) or parking it in waitingDeps forever (unbounded leak).
             if (isCorruptDependsOn(job)) {
-                quarantineCorruptDependsOn(ctx, job);
+                // Keep the SQLite pending result set stable until OFFSET pagination has
+                // finished. Deleting this row now shifts a healthy row across the next
+                // page boundary and leaves it absent from the in-memory queue.
+                corruptPendingJobs.push(job);
                 continue;
             }
             // Check if job has unmet dependencies
@@ -313,6 +316,11 @@ export function recover(ctx) {
         if (jobs.length < RECOVERY_BATCH_SIZE)
             break;
     }
+    // Quarantine only after the paginated scan so deletions cannot move page
+    // boundaries. loadDlq() below restores these entries into memory exactly once.
+    for (const job of corruptPendingJobs) {
+        quarantineCorruptDependsOn(ctx, job);
+    }
     // Load DLQ entries
     const dlqEntries = ctx.storage.loadDlq();
     for (const [queue, entries] of dlqEntries) {

package/dist/application/dependencyProcessor.js

@@ -76,7 +76,12 @@ function promoteJobsToQueue(jobsToPromote, shard, ctx, shardIdx) {
         }
     }
     if (jobsToPromote.length > 0) {
-        shard.notify();
+        const promotedByQueue = new Map();
+        for (const job of jobsToPromote) {
+            promotedByQueue.set(job.queue, (promotedByQueue.get(job.queue) ?? 0) + 1);
+        }
+        for (const [queue, count] of promotedByQueue)
+            shard.notifyBatch(queue, count);
         for (const job of jobsToPromote) {
             ctx.dashboardEmit?.('job:dependencies-resolved', { jobId: String(job.id), queue: job.queue });
         }

package/dist/application/dlqManager.js

@@ -274,6 +274,6 @@ function requeueCompletedJob(job, ctx) {
     ctx.completedJobs.delete(job.id);
     ctx.jobResults.delete(job.id);
     ctx.storage?.updateForRetry(job);
-    shard.notify();
+    shard.notify(job.queue);
     return 1;
 }

package/dist/application/lockManager.js

@@ -144,7 +144,7 @@ function requeueExpiredJob(opts) {
     const isDelayed = job.runAt > now;
     shard.incrementQueued(jobId, isDelayed, job.createdAt, job.queue, job.runAt);
     ctx.jobIndex.set(jobId, { type: 'queue', shardIdx: idx, queueName: job.queue });
-    shard.notify();
+    shard.notify(job.queue);
     ctx.eventsManager.broadcast({
         eventType: "stalled" /* EventType.Stalled */,
         jobId,

package/dist/application/operations/ack.js

@@ -26,7 +26,11 @@ export async function ackJob(jobId, result, ctx) {
     }
     const idx = shardIndex(job.queue);
     await withWriteLock(ctx.shardLocks[idx], () => {
-        ctx.shards[idx].releaseJobResources(job.queue, job.uniqueKey, job.groupId);
+        const shard = ctx.shards[idx];
+        shard.releaseJobResources(job.queue, job.uniqueKey, job.groupId);
+        // Releasing either queue concurrency or an active FIFO group can make a
+        // parked job eligible. Wake only waiters for this queue.
+        shard.notify(job.queue);
     });
     // Release customId so it can be reused
     if (job.customId && ctx.customIdMap) {
@@ -179,6 +183,9 @@ export async function failJob(jobId, error, ctx, unrecoverable = false, stack) {
         else {
             moveFailedJobToDlq(job, jobId, error, shard, ctx);
         }
+        // Terminal failure and retry both release queue concurrency (and possibly
+        // an active FIFO group), so another job may now be eligible.
+        shard.notify(job.queue);
     });
     ctx.broadcast({
         eventType: 'failed',
@@ -234,6 +241,9 @@ export async function ackJobBatch(jobIds, ctx) {
     // Step 3-4: Group by queue shard and release
     const byQueueShard = groupByQueueShard(extractedJobs);
     await releaseResources(byQueueShard, batchCtx);
+    for (const [idx, jobs] of byQueueShard) {
+        notifyReleasedQueues(ctx.shards[idx], jobs);
+    }
     // Step 5: Finalize
     finalizeBatchAck(extractedJobs, ctx, false);
 }
@@ -260,6 +270,18 @@ export async function ackJobBatchWithResults(items, ctx) {
     // Step 3-4: Group by queue shard and release
     const byQueueShard = groupByQueueShard(extractedJobs);
     await releaseResources(byQueueShard, batchCtx);
+    for (const [idx, jobs] of byQueueShard) {
+        notifyReleasedQueues(ctx.shards[idx], jobs);
+    }
     // Step 5: Finalize with results
     finalizeBatchAck(extractedJobs, ctx, true);
 }
+/** Wake queue-scoped waiters for concurrency/group slots released by a batch. */
+function notifyReleasedQueues(shard, jobs) {
+    const releasedByQueue = new Map();
+    for (const job of jobs) {
+        releasedByQueue.set(job.queue, (releasedByQueue.get(job.queue) ?? 0) + 1);
+    }
+    for (const [queue, count] of releasedByQueue)
+        shard.notifyBatch(queue, count);
+}

package/dist/application/operations/jobManagement.js

@@ -200,11 +200,19 @@ export async function moveJobToDelayed(jobId, delay, ctx) {
     const queueName = job.queue;
     await withWriteLock(ctx.shardLocks[idx], () => {
         const shard = ctx.shards[idx];
+        // Release the concurrency slot (+group+uniqueKey) acquired at pull before
+        // re-queueing, mirroring moveActiveToWait (jobStateTransitions.ts) --
+        // otherwise the slot leaks and setConcurrency(N) wedges after N moves.
+        // Dropping the uniqueKey reservation on requeue matches the sibling
+        // requeue paths (failJob retry, moveActiveToWait, requeueExpiredJob).
+        shard.releaseJobResources(job.queue, job.uniqueKey, job.groupId);
         shard.getQueue(job.queue).push(job);
         // Update running counters for O(1) stats and temporal index (job is delayed since delay > 0)
         const isDelayed = job.runAt > now;
         shard.incrementQueued(jobId, isDelayed, job.createdAt, job.queue, job.runAt);
         ctx.jobIndex.set(jobId, { type: 'queue', shardIdx: idx, queueName: job.queue });
+        // The freed slot may unblock waiting jobs for long-pollers.
+        shard.notify(job.queue);
     });
     // Persist the move so the delay survives a restart: the on-disk row was
     // `state='active'` with the old run_at, which recovery would otherwise treat as a
@@ -250,9 +258,29 @@ export async function discardJob(jobId, ctx) {
     if (job) {
         const validJob = job; // Local reference for closure
         const idx = shardIndex(validJob.queue);
+        const fromProcessing = location.type === 'processing';
         const entry = await withWriteLock(ctx.shardLocks[idx], () => {
+            const shard = ctx.shards[idx];
+            if (fromProcessing) {
+                // The pull that activated this job acquired a concurrency slot (and
+                // possibly a FIFO group); return them before the DLQ move or the slot
+                // leaks and setConcurrency(N) wedges after N discards. The uniqueKey
+                // is freed too: every sibling terminal DLQ path releases the full
+                // reservation on entry (failJob, ack.ts; handleMaxStallsExceeded,
+                // lockManager.ts), so the DLQ entry never keeps it.
+                shard.releaseJobResources(validJob.queue, validJob.uniqueKey, validJob.groupId);
+                // The freed slot may unblock waiting jobs for long-pollers.
+                shard.notify(validJob.queue);
+            }
+            else if (validJob.uniqueKey) {
+                // Queue branch: the waiting job never acquired a slot or group at
+                // pull, so a full releaseJobResources would free a slot legitimately
+                // held by ANOTHER active job. Release only the uniqueKey reservation,
+                // matching cancelJob and the DLQ-entry semantics above.
+                shard.releaseUniqueKey(validJob.queue, validJob.uniqueKey);
+            }
             // addToDlq already updates dlq counter and returns entry
-            const dlqEntry = ctx.shards[idx].addToDlq(validJob);
+            const dlqEntry = shard.addToDlq(validJob);
             ctx.jobIndex.set(jobId, { type: 'dlq', queueName: validJob.queue });
             return dlqEntry;
         });

package/dist/application/operations/jobStateTransitions.js

@@ -33,7 +33,7 @@ export async function moveActiveToWait(jobId, ctx) {
         shard.getQueue(job.queue).push(job);
         shard.incrementQueued(jobId, false, job.createdAt, job.queue, job.runAt);
         ctx.jobIndex.set(jobId, { type: 'queue', shardIdx: idx, queueName: job.queue });
-        shard.notify();
+        shard.notify(job.queue);
     });
     ctx.eventsManager.broadcast({
         eventType: 'waiting',