bunqueue 2.8.31 → 2.8.32

package/README.md

@@ -63,15 +63,15 @@ That's it. Queue + Worker in one object. No Redis, no config, no setup.
 
 ## 🪶 Featherweight install
 
-`bun add bunqueue` pulls **7 packages and 5.4 MB** — and that includes the binary
+`bun add bunqueue` pulls **7 packages and 5.5 MB** — and that includes the binary
 queue server, the CLI, and the MCP server. Most queue libraries pull a Redis
 client and its dependency tree before you've queued a single job.
 
 | | Before (2.7.x) | Now | |
 | --- | --- | --- | --- |
-| **`node_modules`** | 93 MB | **5.4 MB** | **−94%** |
+| **`node_modules`** | 93 MB | **5.5 MB** | **−94%** |
 | **packages** | 117 | **7** | **−110** |
-| **cold install** | ~6 s | **~1.2 s** | **~5× faster** |
+| **cold install** | ~6 s | **~1.7 s** | **~3.5× faster** |
 
 Just **2 runtime dependencies** (`croner` + `msgpackr`). SQLite, S3, HTTP and
 WebSocket are all Bun built-ins. The MCP SDK is an optional peer dependency —
@@ -336,7 +336,7 @@ const app = new Bunqueue('tasks', {
 });
 ```
 
-A job with priority 1, after 5 min: 3, after 10 min: 5, … capped at 100.
+A job with priority 1 becomes eligible after 5 min, then gains +2 on every 60s tick: 3, 5, 7, … capped at 100.
 
 ### Deduplication
 
@@ -362,7 +362,7 @@ Override per-job: `await app.add('task', data, { deduplication: { id: 'my-id', t
 
 ### Debouncing
 
-Coalesce rapid same-name jobs. Only the last one in the TTL window gets processed:
+Attach a default debounce id to same-name jobs. Each job gets `debounce: { id: jobName, ttl }` unless the per-job options already set one:
 
 ```typescript
 const app = new Bunqueue('search', {
@@ -373,9 +373,11 @@ const app = new Bunqueue('search', {
 
 await app.add('search', { query: 'h' });
 await app.add('search', { query: 'he' });
-await app.add('search', { query: 'hello' });  // only this one processes
+await app.add('search', { query: 'hello' });  // all three share the debounce id 'search'
 ```
 
+The debounce id and TTL are stored on the job (BullMQ v5 compatible metadata, visible via `job.opts.debounce`), but they do not suppress duplicates by themselves in the current engine. To actually coalesce rapid duplicates, use [Deduplication](#deduplication) with `replace: true`: a duplicate arriving within the TTL window replaces the pending job, giving last-write-wins semantics.
+
 ### Rate Limiting
 
 ```typescript
@@ -385,15 +387,13 @@ const app = new Bunqueue('api', {
   rateLimit: { max: 100, duration: 1000 },
 });
 
-// Per-group rate limiting (e.g., per customer)
+// Per-group cap (e.g., per customer): with groupKey set, max limits
+// concurrently active jobs per group (duration is not applied in this mode)
 const app2 = new Bunqueue('api', {
   embedded: true,
   processor: async (job) => callAPI(job.data),
   rateLimit: { max: 10, duration: 1000, groupKey: 'customerId' },
 });
-
-app.setGlobalRateLimit(50, 1000);
-app.removeGlobalRateLimit();
 ```
 
 ### DLQ (Dead Letter Queue)
@@ -418,7 +418,7 @@ app.retryDlq();                      // retry all
 app.purgeDlq();                      // clear all
 ```
 
-Failure reasons: `explicit_fail`, `max_attempts_exceeded`, `timeout`, `stalled`, `ttl_expired`, `worker_lost`.
+Failure reasons: `explicit_fail`, `max_attempts_exceeded`, `timeout`, `stalled`, `ttl_expired`, `worker_lost`, `unknown`.
 
 ### Cron Jobs
 
@@ -662,9 +662,9 @@ bunx bunqueue-dashboard
 - **MCP server included** — 73 tools, 5 resources, 3 prompts. AI agents get full control out of the box
 - **BullMQ-compatible API** — Same `Queue`, `Worker`, `QueueEvents`
 - **Zero dependencies** — No Redis, no MongoDB
-- **Tiny footprint** — 5.4 MB installed, 7 packages (just 2 runtime deps: `croner` + `msgpackr`)
+- **Tiny footprint** — 5.5 MB installed, 7 packages (just 2 runtime deps: `croner` + `msgpackr`)
 - **SQLite persistence** — Survives restarts, WAL mode for concurrent access
-- **Up to 286K ops/sec** — [Verified benchmarks](https://bunqueue.dev/guide/benchmarks/)
+- **Up to 630K ops/sec** — [Verified benchmarks](https://bunqueue.dev/guide/benchmarks/)
 
 ## Built for AI Agents (MCP Server)
 
@@ -690,8 +690,8 @@ bunqueue is the **first job queue with native MCP support**. AI agents get a ful
 # bunqueue-mcp is a binary bundled inside the bunqueue package (not a standalone npm package),
 # so install bunqueue first, then connect Claude Code:
 bun add bunqueue
-# Since v2.8.0 the MCP SDK is an optional peer dependency. Queue-only users skip it entirely:
-# a clean install drops from 93 MB / 117 packages to 5.4 MB / 7 packages (−94%, up to ~5× faster on a cold install).
+# Since v2.8.1 the MCP SDK is an optional peer dependency. Queue-only users skip it entirely:
+# a clean install drops from 93 MB / 117 packages to 5.5 MB / 7 packages (−94%, ~3.5× faster on a cold install).
 # bunx --package=bunqueue does NOT auto-install optional peers, so install the SDK once to run the MCP server:
 bun add @modelcontextprotocol/sdk
 claude mcp add bunqueue -- bunx bunqueue-mcp
@@ -731,7 +731,7 @@ Supports **embedded** (local SQLite) and **TCP** (remote server) modes. [Full MC
 - **Agentic workflows** — agents push jobs, workers process, agents monitor results
 - Single-server deployments
 - Prototypes and MVPs
-- Moderate to high workloads (up to 286K ops/sec)
+- Moderate to high workloads (up to 630K ops/sec)
 - Teams that want to avoid Redis operational overhead
 - Embedded use cases (CLI tools, edge functions, serverless)
 
@@ -745,7 +745,7 @@ Supports **embedded** (local SQLite) and **TCP** (remote server) modes. [Full MC
 
 If you're already running Redis, BullMQ is great — battle-tested and feature-rich.
 
-bunqueue is for when you **don't want to run Redis**. SQLite with WAL mode handles surprisingly high throughput for single-node deployments (tested up to 286K ops/sec). You get persistence, priorities, delays, retries, cron jobs, and DLQ — without the operational overhead of another service.
+bunqueue is for when you **don't want to run Redis**. SQLite with WAL mode handles surprisingly high throughput for single-node deployments (tested up to 630K ops/sec on bulk push). You get persistence, priorities, delays, retries, cron jobs, and DLQ — without the operational overhead of another service.
 
 ## Install
 
@@ -765,13 +765,15 @@ bunqueue runs in two modes depending on your architecture:
 | ---------------- | ------------------------------------- | -------------------------------------------- |
 | **How it works** | Queue runs inside your process        | Standalone server, clients connect via TCP   |
 | **Setup**        | `bun add bunqueue`                    | `docker run` or `bunqueue start`             |
-| **Performance**  | 286K ops/sec                          | 149K ops/sec                                 |
+| **Performance**  | 630K ops/sec (bulk push)              | 90K ops/sec (push)                           |
 | **Best for**     | Single-process apps, CLIs, serverless | Multiple workers, separate producer/consumer |
 | **Scaling**      | Same process only                     | Multiple clients across machines             |
 
 ### Embedded Mode
 
-Everything runs in your process. No server, no network, no setup.
+Everything runs in your process. No server, no network, no setup. Without a
+data path the queue is in-memory and lost on restart: pass
+`dataPath: './data/app.db'` (or set `BUNQUEUE_DATA_PATH`) to persist jobs.
 
 ```typescript
 import { Queue, Worker } from 'bunqueue/client';
@@ -827,8 +829,8 @@ full feature parity, so producers and workers can live anywhere in your stack:
 
 | Where your code runs | Install | Status |
 | -------------------- | ------- | ------ |
-| **Node.js ≥ 20, Deno ≥ 2, Bun, Cloudflare Workers** | [`npm install bunqueue-client`](https://www.npmjs.com/package/bunqueue-client) | 58/58 e2e per runtime + 11/11 inside workerd |
-| **Python ≥ 3.9** | PyPI coming soon — today: `pip install "git+https://github.com/egeominotti/bunqueue.git#subdirectory=sdk/python"` | 44/44 e2e |
+| **Node.js ≥ 20, Deno ≥ 2, Bun, Cloudflare Workers** | [`npm install bunqueue-client`](https://www.npmjs.com/package/bunqueue-client) | 100/100 e2e per runtime + 16/16 inside workerd |
+| **Python ≥ 3.9** | PyPI coming soon — today: `pip install "git+https://github.com/egeominotti/bunqueue.git#subdirectory=sdk/python"` | 78/78 e2e |
 
 ```typescript
 // Node.js / Deno / Cloudflare Workers
@@ -909,8 +911,8 @@ SQLite handles surprisingly high throughput for single-node deployments:
 
 | Mode     | Peak Throughput | Use Case            |
 | -------- | --------------- | ------------------- |
-| Embedded | 286K ops/sec    | Same process        |
-| TCP      | 149K ops/sec    | Distributed workers |
+| Embedded | 630K ops/sec (bulk push) | Same process        |
+| TCP      | 90K ops/sec (push)       | Distributed workers |
 
 > Run `bun run bench` to verify on your hardware. [Full benchmark methodology →](https://bunqueue.dev/guide/benchmarks/)
 

package/dist/application/operations/jobManagement.js

@@ -200,11 +200,19 @@ export async function moveJobToDelayed(jobId, delay, ctx) {
     const queueName = job.queue;
     await withWriteLock(ctx.shardLocks[idx], () => {
         const shard = ctx.shards[idx];
+        // Release the concurrency slot (+group+uniqueKey) acquired at pull before
+        // re-queueing, mirroring moveActiveToWait (jobStateTransitions.ts) --
+        // otherwise the slot leaks and setConcurrency(N) wedges after N moves.
+        // Dropping the uniqueKey reservation on requeue matches the sibling
+        // requeue paths (failJob retry, moveActiveToWait, requeueExpiredJob).
+        shard.releaseJobResources(job.queue, job.uniqueKey, job.groupId);
         shard.getQueue(job.queue).push(job);
         // Update running counters for O(1) stats and temporal index (job is delayed since delay > 0)
         const isDelayed = job.runAt > now;
         shard.incrementQueued(jobId, isDelayed, job.createdAt, job.queue, job.runAt);
         ctx.jobIndex.set(jobId, { type: 'queue', shardIdx: idx, queueName: job.queue });
+        // The freed slot may unblock waiting jobs for long-pollers.
+        shard.notify();
     });
     // Persist the move so the delay survives a restart: the on-disk row was
     // `state='active'` with the old run_at, which recovery would otherwise treat as a
@@ -250,9 +258,29 @@ export async function discardJob(jobId, ctx) {
     if (job) {
         const validJob = job; // Local reference for closure
         const idx = shardIndex(validJob.queue);
+        const fromProcessing = location.type === 'processing';
         const entry = await withWriteLock(ctx.shardLocks[idx], () => {
+            const shard = ctx.shards[idx];
+            if (fromProcessing) {
+                // The pull that activated this job acquired a concurrency slot (and
+                // possibly a FIFO group); return them before the DLQ move or the slot
+                // leaks and setConcurrency(N) wedges after N discards. The uniqueKey
+                // is freed too: every sibling terminal DLQ path releases the full
+                // reservation on entry (failJob, ack.ts; handleMaxStallsExceeded,
+                // lockManager.ts), so the DLQ entry never keeps it.
+                shard.releaseJobResources(validJob.queue, validJob.uniqueKey, validJob.groupId);
+                // The freed slot may unblock waiting jobs for long-pollers.
+                shard.notify();
+            }
+            else if (validJob.uniqueKey) {
+                // Queue branch: the waiting job never acquired a slot or group at
+                // pull, so a full releaseJobResources would free a slot legitimately
+                // held by ANOTHER active job. Release only the uniqueKey reservation,
+                // matching cancelJob and the DLQ-entry semantics above.
+                shard.releaseUniqueKey(validJob.queue, validJob.uniqueKey);
+            }
             // addToDlq already updates dlq counter and returns entry
-            const dlqEntry = ctx.shards[idx].addToDlq(validJob);
+            const dlqEntry = shard.addToDlq(validJob);
             ctx.jobIndex.set(jobId, { type: 'dlq', queueName: validJob.queue });
             return dlqEntry;
         });

package/dist/client/tcpPool.js

@@ -50,6 +50,8 @@ export class TcpConnectionPool {
                 pingInterval: this.options.pingInterval,
                 maxPingFailures: this.options.maxPingFailures,
                 maxCommandTimeouts: this.options.maxCommandTimeouts,
+                pipelining: this.options.pipelining,
+                maxInFlight: this.options.maxInFlight,
             });
             // Socket-level errors (e.g. TLS handshake failures, protocol garbage) are
             // emitted as 'error' events; without a listener EventEmitter would throw
@@ -190,7 +192,12 @@ function getPoolKey(options) {
     // TLS config must differentiate pools too: a TLS pool and a plaintext pool
     // to the same host:port are NOT interchangeable.
     const tlsKey = options?.tls ? JSON.stringify(options.tls) : '0';
-    return `${host}:${port}:${poolSize}:${tokenHash}:${tlsKey}`;
+    // pipelining/maxInFlight shape per-connection behavior: without them in the
+    // key, two Queues with different windows would silently share whichever
+    // pool was created first.
+    const pipelining = (options?.pipelining ?? true) ? '1' : '0';
+    const maxInFlight = options?.maxInFlight ?? 100;
+    return `${host}:${port}:${poolSize}:${tokenHash}:${tlsKey}:${pipelining}:${maxInFlight}`;
 }
 /** Get or create shared connection pool */
 export function getSharedPool(options) {

package/dist/infrastructure/server/handlers/core.js

@@ -5,6 +5,7 @@
 import * as resp from '../../../domain/types/response';
 import { jobId } from '../../../domain/types/job';
 import { validateQueueName, validateJobData, validateJobOptions, validateNumericField, } from '../protocol';
+import { validatePushBatchJobs } from './pushBatchValidation';
 /** Handle PUSH command */
 export async function handlePush(cmd, ctx, reqId) {
     const queueError = validateQueueName(cmd.queue);
@@ -86,11 +87,11 @@ export async function handlePushBatch(cmd, ctx, reqId) {
     const queueError = validateQueueName(cmd.queue);
     if (queueError)
         return resp.error(queueError, reqId);
-    for (const job of cmd.jobs) {
-        const dataError = validateJobData(job.data);
-        if (dataError)
-            return resp.error(dataError, reqId);
-    }
+    // PUSH parity: per-job data/option bounds + dependsOn existence gate
+    // (extended to earlier same-batch custom ids for intra-batch chains).
+    const jobsError = validatePushBatchJobs(cmd.jobs, ctx);
+    if (jobsError)
+        return resp.error(jobsError, reqId);
     const ids = await ctx.queueManager.pushBatch(cmd.queue, cmd.jobs);
     return resp.batch(ids, reqId);
 }
@@ -128,6 +129,10 @@ export async function handlePullBatch(cmd, ctx, reqId) {
     const countError = validateNumericField(cmd.count, 'count', { min: 1, max: 1000 });
     if (countError)
         return resp.error(countError, reqId);
+    // Validate timeout (same bound as PULL)
+    const timeoutError = validateNumericField(cmd.timeout, 'timeout', { min: 0, max: 60000 });
+    if (timeoutError)
+        return resp.error(timeoutError, reqId);
     // If owner is provided, use lock-based pull
     if (cmd.owner) {
         const { jobs, tokens } = await ctx.queueManager.pullBatchWithLock(cmd.queue, cmd.count, cmd.owner, cmd.timeout ?? 0, cmd.lockTtl);
@@ -139,8 +144,9 @@ export async function handlePullBatch(cmd, ctx, reqId) {
         }
         return resp.pulledJobs(jobs, tokens, reqId);
     }
-    // Standard pull (no locks, but still track for client release)
-    const jobs = await ctx.queueManager.pullBatch(cmd.queue, cmd.count, 0);
+    // Standard pull (no locks, but still track for client release) — the
+    // non-owner branch honors cmd.timeout exactly like the owner branch and PULL.
+    const jobs = await ctx.queueManager.pullBatch(cmd.queue, cmd.count, cmd.timeout ?? 0);
     if (ctx.clientId) {
         for (const job of jobs) {
             ctx.queueManager.registerClientJob(ctx.clientId, job.id);

package/dist/infrastructure/server/handlers/pushBatchValidation.d.ts

@@ -0,0 +1,23 @@
+/**
+ * PUSHB Validation
+ * Per-job validation for batch pushes with PUSH parity: the same
+ * validateJobData/validateJobOptions bounds and the same dependsOn
+ * existence gate handlePush enforces (core.ts).
+ */
+import type { JobInput } from '../../../domain/types/job';
+import type { HandlerContext } from '../types';
+/**
+ * Validate every job of a PUSHB batch. Returns an error message naming the
+ * offending index, or null when the whole batch is valid.
+ *
+ * The dependsOn gate is EXTENDED beyond the PUSH one: a dependency may also
+ * reference ANY OTHER job of the same batch via its custom id (the only ids
+ * a client can know before the batch is applied). Order within the batch is
+ * deliberately irrelevant: the default-on auto-batcher groups concurrent
+ * add() calls in arbitrary order, so a child can legitimately precede its
+ * parent in the array, and the readiness layer (waitingDeps) resolves the
+ * chain once the whole batch is applied. Self-references are rejected, they
+ * can never resolve. jobIndex/completedJobs/depCompletions are hoisted out
+ * of the loop because PUSHB is the bulk hot path.
+ */
+export declare function validatePushBatchJobs(jobs: JobInput[], ctx: HandlerContext): string | null;

package/dist/infrastructure/server/handlers/pushBatchValidation.js

@@ -0,0 +1,70 @@
+/**
+ * PUSHB Validation
+ * Per-job validation for batch pushes with PUSH parity: the same
+ * validateJobData/validateJobOptions bounds and the same dependsOn
+ * existence gate handlePush enforces (core.ts).
+ */
+import { validateJobData, validateJobOptions } from '../protocol';
+/**
+ * Validate every job of a PUSHB batch. Returns an error message naming the
+ * offending index, or null when the whole batch is valid.
+ *
+ * The dependsOn gate is EXTENDED beyond the PUSH one: a dependency may also
+ * reference ANY OTHER job of the same batch via its custom id (the only ids
+ * a client can know before the batch is applied). Order within the batch is
+ * deliberately irrelevant: the default-on auto-batcher groups concurrent
+ * add() calls in arbitrary order, so a child can legitimately precede its
+ * parent in the array, and the readiness layer (waitingDeps) resolves the
+ * chain once the whole batch is applied. Self-references are rejected, they
+ * can never resolve. jobIndex/completedJobs/depCompletions are hoisted out
+ * of the loop because PUSHB is the bulk hot path.
+ */
+export function validatePushBatchJobs(jobs, ctx) {
+    const jobIndex = ctx.queueManager.getJobIndex();
+    const completedJobs = ctx.queueManager.getCompletedJobs();
+    const depCompletions = ctx.queueManager.getDepCompletions();
+    // Custom ids of ALL jobs in this batch: they become real job ids the
+    // moment the batch is applied. Allocated lazily (most batches have neither
+    // custom ids nor dependencies).
+    let batchIds = null;
+    for (const job of jobs) {
+        if (job.customId) {
+            batchIds ??= new Set();
+            batchIds.add(job.customId);
+        }
+    }
+    for (let i = 0; i < jobs.length; i++) {
+        const job = jobs[i];
+        const dataError = validateJobData(job.data);
+        if (dataError)
+            return `jobs[${i}]: ${dataError}`;
+        const optionsError = validateJobOptions({
+            priority: job.priority,
+            delay: job.delay,
+            timeout: job.timeout,
+            maxAttempts: job.maxAttempts,
+            backoff: job.backoff,
+            ttl: job.ttl,
+        });
+        if (optionsError)
+            return `jobs[${i}]: ${optionsError}`;
+        if (job.dependsOn && job.dependsOn.length > 0) {
+            for (const depId of job.dependsOn) {
+                const key = String(depId);
+                if (key === job.customId) {
+                    return `jobs[${i}]: Job cannot depend on itself: ${key}`;
+                }
+                const exists = jobIndex.has(depId) ||
+                    completedJobs.has(depId) ||
+                    // removeOnComplete parents leave only a bare completion id behind;
+                    // the gate must honor it exactly like PUSH does.
+                    depCompletions.has(depId) ||
+                    (batchIds?.has(key) ?? false);
+                if (!exists) {
+                    return `jobs[${i}]: Dependency job not found: ${key}`;
+                }
+            }
+        }
+    }
+    return null;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bunqueue",
-  "version": "2.8.31",
+  "version": "2.8.32",
   "description": "High-performance job queue for Bun & AI agents. SQLite persistence, cron scheduling, priorities, retries, DLQ, webhooks, native MCP server. Zero external dependencies.",
   "type": "module",
   "main": "dist/main.js",