bunqueue 2.8.30 → 2.8.32

package/README.md

@@ -63,15 +63,15 @@ That's it. Queue + Worker in one object. No Redis, no config, no setup.
 
 ## 🪶 Featherweight install
 
-`bun add bunqueue` pulls **7 packages and 5.4 MB** — and that includes the binary
+`bun add bunqueue` pulls **7 packages and 5.5 MB** — and that includes the binary
 queue server, the CLI, and the MCP server. Most queue libraries pull a Redis
 client and its dependency tree before you've queued a single job.
 
 | | Before (2.7.x) | Now | |
 | --- | --- | --- | --- |
-| **`node_modules`** | 93 MB | **5.4 MB** | **−94%** |
+| **`node_modules`** | 93 MB | **5.5 MB** | **−94%** |
 | **packages** | 117 | **7** | **−110** |
-| **cold install** | ~6 s | **~1.2 s** | **~5× faster** |
+| **cold install** | ~6 s | **~1.7 s** | **~3.5× faster** |
 
 Just **2 runtime dependencies** (`croner` + `msgpackr`). SQLite, S3, HTTP and
 WebSocket are all Bun built-ins. The MCP SDK is an optional peer dependency —
@@ -336,7 +336,7 @@ const app = new Bunqueue('tasks', {
 });
 ```
 
-A job with priority 1, after 5 min: 3, after 10 min: 5, … capped at 100.
+A job with priority 1 becomes eligible after 5 min, then gains +2 on every 60s tick: 3, 5, 7, … capped at 100.
 
 ### Deduplication
 
@@ -362,7 +362,7 @@ Override per-job: `await app.add('task', data, { deduplication: { id: 'my-id', t
 
 ### Debouncing
 
-Coalesce rapid same-name jobs. Only the last one in the TTL window gets processed:
+Attach a default debounce id to same-name jobs. Each job gets `debounce: { id: jobName, ttl }` unless the per-job options already set one:
 
 ```typescript
 const app = new Bunqueue('search', {
@@ -373,9 +373,11 @@ const app = new Bunqueue('search', {
 
 await app.add('search', { query: 'h' });
 await app.add('search', { query: 'he' });
-await app.add('search', { query: 'hello' });  // only this one processes
+await app.add('search', { query: 'hello' });  // all three share the debounce id 'search'
 ```
 
+The debounce id and TTL are stored on the job (BullMQ v5 compatible metadata, visible via `job.opts.debounce`), but they do not suppress duplicates by themselves in the current engine. To actually coalesce rapid duplicates, use [Deduplication](#deduplication) with `replace: true`: a duplicate arriving within the TTL window replaces the pending job, giving last-write-wins semantics.
+
 ### Rate Limiting
 
 ```typescript
@@ -385,15 +387,13 @@ const app = new Bunqueue('api', {
   rateLimit: { max: 100, duration: 1000 },
 });
 
-// Per-group rate limiting (e.g., per customer)
+// Per-group cap (e.g., per customer): with groupKey set, max limits
+// concurrently active jobs per group (duration is not applied in this mode)
 const app2 = new Bunqueue('api', {
   embedded: true,
   processor: async (job) => callAPI(job.data),
   rateLimit: { max: 10, duration: 1000, groupKey: 'customerId' },
 });
-
-app.setGlobalRateLimit(50, 1000);
-app.removeGlobalRateLimit();
 ```
 
 ### DLQ (Dead Letter Queue)
@@ -418,7 +418,7 @@ app.retryDlq();                      // retry all
 app.purgeDlq();                      // clear all
 ```
 
-Failure reasons: `explicit_fail`, `max_attempts_exceeded`, `timeout`, `stalled`, `ttl_expired`, `worker_lost`.
+Failure reasons: `explicit_fail`, `max_attempts_exceeded`, `timeout`, `stalled`, `ttl_expired`, `worker_lost`, `unknown`.
 
 ### Cron Jobs
 
@@ -662,9 +662,9 @@ bunx bunqueue-dashboard
 - **MCP server included** — 73 tools, 5 resources, 3 prompts. AI agents get full control out of the box
 - **BullMQ-compatible API** — Same `Queue`, `Worker`, `QueueEvents`
 - **Zero dependencies** — No Redis, no MongoDB
-- **Tiny footprint** — 5.4 MB installed, 7 packages (just 2 runtime deps: `croner` + `msgpackr`)
+- **Tiny footprint** — 5.5 MB installed, 7 packages (just 2 runtime deps: `croner` + `msgpackr`)
 - **SQLite persistence** — Survives restarts, WAL mode for concurrent access
-- **Up to 286K ops/sec** — [Verified benchmarks](https://bunqueue.dev/guide/benchmarks/)
+- **Up to 630K ops/sec** — [Verified benchmarks](https://bunqueue.dev/guide/benchmarks/)
 
 ## Built for AI Agents (MCP Server)
 
@@ -690,8 +690,8 @@ bunqueue is the **first job queue with native MCP support**. AI agents get a ful
 # bunqueue-mcp is a binary bundled inside the bunqueue package (not a standalone npm package),
 # so install bunqueue first, then connect Claude Code:
 bun add bunqueue
-# Since v2.8.0 the MCP SDK is an optional peer dependency. Queue-only users skip it entirely:
-# a clean install drops from 93 MB / 117 packages to 5.4 MB / 7 packages (−94%, up to ~5× faster on a cold install).
+# Since v2.8.1 the MCP SDK is an optional peer dependency. Queue-only users skip it entirely:
+# a clean install drops from 93 MB / 117 packages to 5.5 MB / 7 packages (−94%, ~3.5× faster on a cold install).
 # bunx --package=bunqueue does NOT auto-install optional peers, so install the SDK once to run the MCP server:
 bun add @modelcontextprotocol/sdk
 claude mcp add bunqueue -- bunx bunqueue-mcp
@@ -731,7 +731,7 @@ Supports **embedded** (local SQLite) and **TCP** (remote server) modes. [Full MC
 - **Agentic workflows** — agents push jobs, workers process, agents monitor results
 - Single-server deployments
 - Prototypes and MVPs
-- Moderate to high workloads (up to 286K ops/sec)
+- Moderate to high workloads (up to 630K ops/sec)
 - Teams that want to avoid Redis operational overhead
 - Embedded use cases (CLI tools, edge functions, serverless)
 
@@ -745,7 +745,7 @@ Supports **embedded** (local SQLite) and **TCP** (remote server) modes. [Full MC
 
 If you're already running Redis, BullMQ is great — battle-tested and feature-rich.
 
-bunqueue is for when you **don't want to run Redis**. SQLite with WAL mode handles surprisingly high throughput for single-node deployments (tested up to 286K ops/sec). You get persistence, priorities, delays, retries, cron jobs, and DLQ — without the operational overhead of another service.
+bunqueue is for when you **don't want to run Redis**. SQLite with WAL mode handles surprisingly high throughput for single-node deployments (tested up to 630K ops/sec on bulk push). You get persistence, priorities, delays, retries, cron jobs, and DLQ — without the operational overhead of another service.
 
 ## Install
 
@@ -765,13 +765,15 @@ bunqueue runs in two modes depending on your architecture:
 | ---------------- | ------------------------------------- | -------------------------------------------- |
 | **How it works** | Queue runs inside your process        | Standalone server, clients connect via TCP   |
 | **Setup**        | `bun add bunqueue`                    | `docker run` or `bunqueue start`             |
-| **Performance**  | 286K ops/sec                          | 149K ops/sec                                 |
+| **Performance**  | 630K ops/sec (bulk push)              | 90K ops/sec (push)                           |
 | **Best for**     | Single-process apps, CLIs, serverless | Multiple workers, separate producer/consumer |
 | **Scaling**      | Same process only                     | Multiple clients across machines             |
 
 ### Embedded Mode
 
-Everything runs in your process. No server, no network, no setup.
+Everything runs in your process. No server, no network, no setup. Without a
+data path the queue is in-memory and lost on restart: pass
+`dataPath: './data/app.db'` (or set `BUNQUEUE_DATA_PATH`) to persist jobs.
 
 ```typescript
 import { Queue, Worker } from 'bunqueue/client';
@@ -827,8 +829,8 @@ full feature parity, so producers and workers can live anywhere in your stack:
 
 | Where your code runs | Install | Status |
 | -------------------- | ------- | ------ |
-| **Node.js ≥ 20, Deno ≥ 2, Bun, Cloudflare Workers** | [`npm install bunqueue-client`](https://www.npmjs.com/package/bunqueue-client) | 58/58 e2e per runtime + 11/11 inside workerd |
-| **Python ≥ 3.9** | PyPI coming soon — today: `pip install "git+https://github.com/egeominotti/bunqueue.git#subdirectory=sdk/python"` | 44/44 e2e |
+| **Node.js ≥ 20, Deno ≥ 2, Bun, Cloudflare Workers** | [`npm install bunqueue-client`](https://www.npmjs.com/package/bunqueue-client) | 100/100 e2e per runtime + 16/16 inside workerd |
+| **Python ≥ 3.9** | PyPI coming soon — today: `pip install "git+https://github.com/egeominotti/bunqueue.git#subdirectory=sdk/python"` | 78/78 e2e |
 
 ```typescript
 // Node.js / Deno / Cloudflare Workers
@@ -909,8 +911,8 @@ SQLite handles surprisingly high throughput for single-node deployments:
 
 | Mode     | Peak Throughput | Use Case            |
 | -------- | --------------- | ------------------- |
-| Embedded | 286K ops/sec    | Same process        |
-| TCP      | 149K ops/sec    | Distributed workers |
+| Embedded | 630K ops/sec (bulk push) | Same process        |
+| TCP      | 90K ops/sec (push)       | Distributed workers |
 
 > Run `bun run bench` to verify on your hardware. [Full benchmark methodology →](https://bunqueue.dev/guide/benchmarks/)
 

package/dist/application/operations/jobManagement.js

@@ -200,11 +200,19 @@ export async function moveJobToDelayed(jobId, delay, ctx) {
     const queueName = job.queue;
     await withWriteLock(ctx.shardLocks[idx], () => {
         const shard = ctx.shards[idx];
+        // Release the concurrency slot (+group+uniqueKey) acquired at pull before
+        // re-queueing, mirroring moveActiveToWait (jobStateTransitions.ts) --
+        // otherwise the slot leaks and setConcurrency(N) wedges after N moves.
+        // Dropping the uniqueKey reservation on requeue matches the sibling
+        // requeue paths (failJob retry, moveActiveToWait, requeueExpiredJob).
+        shard.releaseJobResources(job.queue, job.uniqueKey, job.groupId);
         shard.getQueue(job.queue).push(job);
         // Update running counters for O(1) stats and temporal index (job is delayed since delay > 0)
         const isDelayed = job.runAt > now;
         shard.incrementQueued(jobId, isDelayed, job.createdAt, job.queue, job.runAt);
         ctx.jobIndex.set(jobId, { type: 'queue', shardIdx: idx, queueName: job.queue });
+        // The freed slot may unblock waiting jobs for long-pollers.
+        shard.notify();
     });
     // Persist the move so the delay survives a restart: the on-disk row was
     // `state='active'` with the old run_at, which recovery would otherwise treat as a
@@ -250,9 +258,29 @@ export async function discardJob(jobId, ctx) {
     if (job) {
         const validJob = job; // Local reference for closure
         const idx = shardIndex(validJob.queue);
+        const fromProcessing = location.type === 'processing';
         const entry = await withWriteLock(ctx.shardLocks[idx], () => {
+            const shard = ctx.shards[idx];
+            if (fromProcessing) {
+                // The pull that activated this job acquired a concurrency slot (and
+                // possibly a FIFO group); return them before the DLQ move or the slot
+                // leaks and setConcurrency(N) wedges after N discards. The uniqueKey
+                // is freed too: every sibling terminal DLQ path releases the full
+                // reservation on entry (failJob, ack.ts; handleMaxStallsExceeded,
+                // lockManager.ts), so the DLQ entry never keeps it.
+                shard.releaseJobResources(validJob.queue, validJob.uniqueKey, validJob.groupId);
+                // The freed slot may unblock waiting jobs for long-pollers.
+                shard.notify();
+            }
+            else if (validJob.uniqueKey) {
+                // Queue branch: the waiting job never acquired a slot or group at
+                // pull, so a full releaseJobResources would free a slot legitimately
+                // held by ANOTHER active job. Release only the uniqueKey reservation,
+                // matching cancelJob and the DLQ-entry semantics above.
+                shard.releaseUniqueKey(validJob.queue, validJob.uniqueKey);
+            }
             // addToDlq already updates dlq counter and returns entry
-            const dlqEntry = ctx.shards[idx].addToDlq(validJob);
+            const dlqEntry = shard.addToDlq(validJob);
             ctx.jobIndex.set(jobId, { type: 'dlq', queueName: validJob.queue });
             return dlqEntry;
         });

package/dist/application/operations/pull.js

@@ -47,18 +47,45 @@ function tryDequeueNextJob(shard, queue, now, ctx) {
     if (job.timeline.length < MAX_TIMELINE_ENTRIES) {
         job.timeline.push({ state: 'active', timestamp: now });
     }
+    // Make the queue -> processing transition atomic for observers. From the
+    // pop above, the job is no longer findable in the shard queue, so a
+    // jobIndex entry still saying 'queue' would make getJob/getJobState return
+    // a false null/'unknown' for an existing job (JOB -> null -> JOB(active)
+    // flicker), and cleanOrphanedJobIndex would even drop its index entry.
+    // Insert into processingShards and flip the index HERE, in the same
+    // synchronous critical section as the pop (the caller holds the shard
+    // write lock; JS is single-threaded, so no other task can interleave).
+    // Writing processingShards without its RWLock is safe: until this
+    // statement the index said 'queue', so no id-targeted critical section
+    // (ack/fail/discard operate only on ids they find as 'processing') can be
+    // mid-operation on this id; a sync Map.set cannot corrupt a paused
+    // critical section; and lock-free sync access to processingShards is
+    // established practice (stallDetection phases 1/2, collectActiveJobs,
+    // getJobProgress). Acquiring the async RWLock here would instead hold the
+    // hot shard write lock across an await.
+    const procIdx = processingShardIndex(job.id);
+    ctx.processingShards[procIdx].set(job.id, job);
+    ctx.jobIndex.set(job.id, { type: 'processing', shardIdx: procIdx });
     return { status: 'job', job };
 }
 /**
- * Move job to processing shard and broadcast
+ * Finish the handoff of a dequeued job: persist active state and broadcast.
+ *
+ * The job was ALREADY inserted into processingShards and jobIndex was flipped
+ * to 'processing' inside the dequeue critical section (tryDequeueNextJob),
+ * atomically with the queue pop, so readers never see a stale 'queue'
+ * location. Only the bookkeeping that may follow an await happens here.
+ *
+ * Returns false when the job is no longer in processingShards: a management
+ * op (discardJob, moveJobToDelayed, obliterate) claimed it between the
+ * dequeue and this call. The claimer owns the job now; the pull must NOT
+ * deliver it to a worker nor mark it active.
  */
-async function moveToProcessing(job, queue, ctx) {
+function finalizeProcessing(job, queue, ctx) {
     const procIdx = processingShardIndex(job.id);
+    if (!ctx.processingShards[procIdx].has(job.id))
+        return false;
     const now = Date.now();
-    await withWriteLock(ctx.processingLocks[procIdx], () => {
-        ctx.processingShards[procIdx].set(job.id, job);
-    });
-    ctx.jobIndex.set(job.id, { type: 'processing', shardIdx: procIdx });
     try {
         ctx.storage?.markActive(job.id, job.startedAt ?? now, job.timeline);
     }
@@ -74,74 +101,59 @@ async function moveToProcessing(job, queue, ctx) {
         jobId: job.id,
         timestamp: now,
     });
+    return true;
 }
 /**
- * Move multiple jobs to processing shards (optimized: groups by shard)
+ * Finish the handoff for a batch of dequeued jobs.
+ * Returns the jobs actually delivered (claimed jobs are skipped, see above).
  */
-async function moveToProcessingBatch(jobs, queue, ctx) {
-    // Group by processing shard for efficiency
-    const byProcShard = new Map();
+function finalizeProcessingBatch(jobs, queue, ctx) {
+    const delivered = [];
     for (const job of jobs) {
-        const procIdx = processingShardIndex(job.id);
-        const shardJobs = byProcShard.get(procIdx) ?? [];
-        if (shardJobs.length === 0)
-            byProcShard.set(procIdx, shardJobs);
-        shardJobs.push(job);
-    }
-    // Add to processing shards in parallel
-    const lockPromises = [];
-    for (const [procIdx, shardJobs] of byProcShard) {
-        lockPromises.push(withWriteLock(ctx.processingLocks[procIdx], () => {
-            for (const job of shardJobs) {
-                ctx.processingShards[procIdx].set(job.id, job);
-            }
-        }));
-    }
-    await Promise.all(lockPromises);
-    // Update indexes and broadcast
-    const now = Date.now();
-    for (const job of jobs) {
-        const procIdx = processingShardIndex(job.id);
-        ctx.jobIndex.set(job.id, { type: 'processing', shardIdx: procIdx });
-        try {
-            ctx.storage?.markActive(job.id, job.startedAt ?? now, job.timeline);
-        }
-        catch {
-            // Non-fatal: job is already in processingShards (in-memory source of truth).
-        }
-        ctx.totalPulled.value++;
-        throughputTracker.pullRate.increment();
-        ctx.broadcast({
-            eventType: 'pulled',
-            queue,
-            jobId: job.id,
-            timestamp: now,
-        });
+        if (finalizeProcessing(job, queue, ctx))
+            delivered.push(job);
     }
+    return delivered;
 }
 /**
- * Requeue a job that was popped from the priority queue but failed to move to processing.
- * Restores the job to the shard queue, removes it from processingShards, and notifies waiters.
+ * Requeue a job whose handoff to the worker failed after dequeue.
+ * The dequeue critical section moved the job to processingShards and flipped
+ * jobIndex to 'processing'; this restores both, keeping the same atomicity
+ * rule: the job becomes findable in the queue (push + index flip in ONE
+ * shard-lock section) BEFORE the processing entry is dropped, so a reader
+ * never sees a location that misses. Lock order shard -> processing matches
+ * the documented hierarchy.
  */
 async function requeueJob(job, queue, idx, ctx) {
-    // Remove from processingShards (may have been added before the failure)
     const procIdx = processingShardIndex(job.id);
-    await withWriteLock(ctx.processingLocks[procIdx], () => {
-        ctx.processingShards[procIdx].delete(job.id);
-    });
-    // Reset job state (was modified in tryDequeueNextJob)
-    job.startedAt = null;
-    // Push back to queue
+    let requeued = false;
     await withWriteLock(ctx.shardLocks[idx], () => {
+        // A management op (discardJob, moveJobToDelayed) may have claimed the job
+        // from processingShards after the failed handoff; it owns the job now.
+        if (!ctx.processingShards[procIdx].has(job.id))
+            return;
         const shard = ctx.shards[idx];
         if (job.groupId)
             shard.releaseGroup(queue, job.groupId);
         shard.releaseConcurrency(queue);
+        // Reset job state (was modified in tryDequeueNextJob)
+        job.startedAt = null;
         shard.getQueue(queue).push(job);
         shard.incrementQueued(job.id, false, job.createdAt, queue, job.runAt);
+        ctx.jobIndex.set(job.id, { type: 'queue', shardIdx: idx, queueName: queue });
         shard.notify();
+        requeued = true;
+    });
+    if (!requeued)
+        return;
+    await withWriteLock(ctx.processingLocks[procIdx], () => {
+        // A concurrent pull may have re-popped the job between the two lock
+        // sections and flipped the index back to 'processing'; do not delete the
+        // (re-inserted, same-object) entry out from under it.
+        if (ctx.jobIndex.get(job.id)?.type !== 'processing') {
+            ctx.processingShards[procIdx].delete(job.id);
+        }
     });
-    ctx.jobIndex.set(job.id, { type: 'queue', shardIdx: idx, queueName: queue });
 }
 /**
  * Pull next job from queue
@@ -153,14 +165,18 @@ export async function pullJob(queue, timeoutMs, ctx) {
     while (true) {
         const job = await tryPullFromShard(queue, idx, ctx);
         if (job) {
+            let delivered = false;
             try {
-                await moveToProcessing(job, queue, ctx);
+                delivered = finalizeProcessing(job, queue, ctx);
             }
             catch {
-                // Safety net: if moveToProcessing fails, push job back to prevent loss
+                // Safety net: if the handoff fails, push job back to prevent loss
                 await requeueJob(job, queue, idx, ctx);
                 return null;
             }
+            // Claimed by a management op during the handoff: try the next job
+            if (!delivered)
+                continue;
             latencyTracker.pull.observe((Bun.nanoseconds() - startNs) / 1e6);
             return job;
         }
@@ -216,21 +232,27 @@ export async function pullJobBatch(queue, count, timeoutMs, ctx) {
     while (true) {
         const jobs = await tryPullBatchFromShard(queue, idx, count, ctx);
         if (jobs.length > 0) {
+            let delivered;
             try {
-                await moveToProcessingBatch(jobs, queue, ctx);
+                delivered = finalizeProcessingBatch(jobs, queue, ctx);
             }
             catch {
-                // Safety net: push all jobs back to prevent loss
+                // Safety net: push all jobs back to prevent loss (requeueJob skips
+                // jobs a management op already claimed)
                 for (const job of jobs) {
                     await requeueJob(job, queue, idx, ctx);
                 }
                 return [];
             }
-            if (jobs.length > 1) {
-                ctx.dashboardEmit?.('batch:pulled', { queue, count: jobs.length });
+            if (delivered.length > 0) {
+                if (delivered.length > 1) {
+                    ctx.dashboardEmit?.('batch:pulled', { queue, count: delivered.length });
+                }
+                latencyTracker.pull.observe((Bun.nanoseconds() - startNs) / 1e6);
+                return delivered;
             }
-            latencyTracker.pull.observe((Bun.nanoseconds() - startNs) / 1e6);
-            return jobs;
+            // Every dequeued job was claimed by a management op: fall through to
+            // the timeout/wait logic below (nothing to deliver this round)
         }
         // No jobs available, check timeout
         // Cache Date.now() to avoid multiple syscalls per iteration

package/dist/application/operations/queryOperations.js

@@ -6,51 +6,75 @@ import { withReadLock } from '../../shared/lock';
 import { shardIndex } from '../../shared/hash';
 /** Get job by ID */
 export async function getJob(jobId, ctx) {
-    const location = ctx.jobIndex.get(jobId);
-    if (!location) {
-        // Fallback: jobIndex may not be populated after restart for completed/DLQ jobs.
-        // Consult SQLite directly so getJob survives recovery.
-        if (ctx.storage) {
-            const job = ctx.storage.getJob(jobId);
-            if (job)
-                return job;
-            const dlqEntry = ctx.storage.getDlqEntry(jobId);
-            if (dlqEntry)
-                return dlqEntry.job;
-        }
-        return ctx.completedJobsData.get(jobId) ?? null;
-    }
-    switch (location.type) {
-        case 'queue': {
-            return await withReadLock(ctx.shardLocks[location.shardIdx], () => {
-                const shard = ctx.shards[location.shardIdx];
-                return (shard.getQueue(location.queueName).find(jobId) ??
-                    shard.waitingDeps.get(jobId) ??
-                    shard.waitingChildren.get(jobId) ??
-                    null);
-            });
-        }
-        case 'processing': {
-            return await withReadLock(ctx.processingLocks[location.shardIdx], () => {
-                return ctx.processingShards[location.shardIdx].get(jobId) ?? null;
-            });
-        }
-        case 'completed':
-            return ctx.storage?.getJob(jobId) ?? ctx.completedJobsData.get(jobId) ?? null;
-        case 'dlq': {
+    // The location snapshot is only valid until the first await: while this
+    // reader waits on the shard read lock (writers have priority), a concurrent
+    // pull can pop the job and move it queue -> processing. The pull updates
+    // jobIndex atomically with the pop (tryDequeueNextJob), so on a miss the
+    // index re-read below is authoritative: an identity change means the job
+    // MOVED, and we chase the new location instead of returning a false null
+    // for a job that still exists (getJob(id) === null must be permanent for
+    // never-reused uuidv7 ids). Bounded: every extra pass requires a further
+    // state transition of this very job (queue -> active -> completed/removed),
+    // so 4 passes cover any realistic chase; index entries are always replaced
+    // (never mutated), which makes the identity comparison exact.
+    for (let pass = 0; pass < 4; pass++) {
+        const location = ctx.jobIndex.get(jobId);
+        if (!location) {
+            // Fallback: jobIndex may not be populated after restart for completed/DLQ jobs.
+            // Consult SQLite directly so getJob survives recovery.
             if (ctx.storage) {
-                const dlqEntry = ctx.storage.getDlqEntry(jobId);
-                if (dlqEntry)
-                    return dlqEntry.job;
                 const job = ctx.storage.getJob(jobId);
                 if (job)
                     return job;
+                const dlqEntry = ctx.storage.getDlqEntry(jobId);
+                if (dlqEntry)
+                    return dlqEntry.job;
             }
-            const dlqShardIdx = shardIndex(location.queueName);
-            const dlqJobs = ctx.shards[dlqShardIdx].getDlq(location.queueName);
-            return dlqJobs.find((j) => j.id === jobId) ?? null;
+            return ctx.completedJobsData.get(jobId) ?? null;
         }
+        let found = null;
+        switch (location.type) {
+            case 'queue': {
+                found = await withReadLock(ctx.shardLocks[location.shardIdx], () => {
+                    const shard = ctx.shards[location.shardIdx];
+                    return (shard.getQueue(location.queueName).find(jobId) ??
+                        shard.waitingDeps.get(jobId) ??
+                        shard.waitingChildren.get(jobId) ??
+                        null);
+                });
+                break;
+            }
+            case 'processing': {
+                found = await withReadLock(ctx.processingLocks[location.shardIdx], () => {
+                    return ctx.processingShards[location.shardIdx].get(jobId) ?? null;
+                });
+                break;
+            }
+            case 'completed':
+                found = ctx.storage?.getJob(jobId) ?? ctx.completedJobsData.get(jobId) ?? null;
+                break;
+            case 'dlq': {
+                if (ctx.storage) {
+                    const dlqEntry = ctx.storage.getDlqEntry(jobId);
+                    if (dlqEntry)
+                        return dlqEntry.job;
+                    const job = ctx.storage.getJob(jobId);
+                    if (job)
+                        return job;
+                }
+                const dlqShardIdx = shardIndex(location.queueName);
+                const dlqJobs = ctx.shards[dlqShardIdx].getDlq(location.queueName);
+                found = dlqJobs.find((j) => j.id === jobId) ?? null;
+                break;
+            }
+        }
+        if (found)
+            return found;
+        if (ctx.jobIndex.get(jobId) === location)
+            return null; // no move: genuine miss
+        // The job moved while we held the stale snapshot: chase it.
     }
+    return null;
 }
 /** Get job result */
 export function getJobResult(jobId, ctx) {
@@ -121,46 +145,56 @@ function resolveStateFromStorage(jobId, storage) {
 }
 /** Get job state by ID */
 export async function getJobState(jobId, ctx) {
-    const location = ctx.jobIndex.get(jobId);
-    // Check completed set first (fast path)
-    if (ctx.completedJobs.has(jobId)) {
-        return "completed" /* JobState.Completed */;
-    }
-    if (!location) {
-        return resolveStateFromStorage(jobId, ctx.storage);
-    }
-    switch (location.type) {
-        case 'queue': {
-            // Check if job is delayed, waiting, or waiting for children/deps
-            const result = await withReadLock(ctx.shardLocks[location.shardIdx], () => {
-                const shard = ctx.shards[location.shardIdx];
-                const queueJob = shard.getQueue(location.queueName).find(jobId);
-                if (queueJob)
-                    return { job: queueJob, waitingDeps: false, waitingChildren: false };
-                const depsJob = shard.waitingDeps.get(jobId);
-                if (depsJob)
-                    return { job: depsJob, waitingDeps: true, waitingChildren: false };
-                const childrenJob = shard.waitingChildren.get(jobId);
-                if (childrenJob)
-                    return { job: childrenJob, waitingDeps: false, waitingChildren: true };
-                return null;
-            });
-            if (!result)
-                return 'unknown';
-            if (result.waitingDeps || result.waitingChildren)
-                return 'waiting-children';
-            const now = Date.now();
-            if (result.job.runAt > now)
-                return "delayed" /* JobState.Delayed */;
-            return result.job.priority > 0 ? "prioritized" /* JobState.Prioritized */ : "waiting" /* JobState.Waiting */;
-        }
-        case 'processing':
-            return "active" /* JobState.Active */;
-        case 'completed':
+    // Same stale-snapshot chase as getJob: a 'queue' location read before the
+    // read-lock await may be outdated by a concurrent pull (queue -> processing)
+    // by the time the lookup runs. On a miss with a changed index entry, retry
+    // with the fresh location instead of reporting a false 'unknown'.
+    for (let pass = 0; pass < 4; pass++) {
+        const location = ctx.jobIndex.get(jobId);
+        // Check completed set first (fast path)
+        if (ctx.completedJobs.has(jobId)) {
             return "completed" /* JobState.Completed */;
-        case 'dlq':
-            return "failed" /* JobState.Failed */;
+        }
+        if (!location) {
+            return resolveStateFromStorage(jobId, ctx.storage);
+        }
+        switch (location.type) {
+            case 'queue': {
+                // Check if job is delayed, waiting, or waiting for children/deps
+                const result = await withReadLock(ctx.shardLocks[location.shardIdx], () => {
+                    const shard = ctx.shards[location.shardIdx];
+                    const queueJob = shard.getQueue(location.queueName).find(jobId);
+                    if (queueJob)
+                        return { job: queueJob, waitingDeps: false, waitingChildren: false };
+                    const depsJob = shard.waitingDeps.get(jobId);
+                    if (depsJob)
+                        return { job: depsJob, waitingDeps: true, waitingChildren: false };
+                    const childrenJob = shard.waitingChildren.get(jobId);
+                    if (childrenJob)
+                        return { job: childrenJob, waitingDeps: false, waitingChildren: true };
+                    return null;
+                });
+                if (!result) {
+                    if (ctx.jobIndex.get(jobId) === location)
+                        return 'unknown'; // no move
+                    break; // moved mid-lookup: chase the new location
+                }
+                if (result.waitingDeps || result.waitingChildren)
+                    return 'waiting-children';
+                const now = Date.now();
+                if (result.job.runAt > now)
+                    return "delayed" /* JobState.Delayed */;
+                return result.job.priority > 0 ? "prioritized" /* JobState.Prioritized */ : "waiting" /* JobState.Waiting */;
+            }
+            case 'processing':
+                return "active" /* JobState.Active */;
+            case 'completed':
+                return "completed" /* JobState.Completed */;
+            case 'dlq':
+                return "failed" /* JobState.Failed */;
+        }
     }
+    return 'unknown';
 }
 /** Collect completed jobs for a queue from index + storage */
 function collectCompletedJobs(queue, ctx, maxCollect) {

package/dist/client/tcpPool.js

@@ -50,6 +50,8 @@ export class TcpConnectionPool {
                 pingInterval: this.options.pingInterval,
                 maxPingFailures: this.options.maxPingFailures,
                 maxCommandTimeouts: this.options.maxCommandTimeouts,
+                pipelining: this.options.pipelining,
+                maxInFlight: this.options.maxInFlight,
             });
             // Socket-level errors (e.g. TLS handshake failures, protocol garbage) are
             // emitted as 'error' events; without a listener EventEmitter would throw
@@ -190,7 +192,12 @@ function getPoolKey(options) {
     // TLS config must differentiate pools too: a TLS pool and a plaintext pool
     // to the same host:port are NOT interchangeable.
     const tlsKey = options?.tls ? JSON.stringify(options.tls) : '0';
-    return `${host}:${port}:${poolSize}:${tokenHash}:${tlsKey}`;
+    // pipelining/maxInFlight shape per-connection behavior: without them in the
+    // key, two Queues with different windows would silently share whichever
+    // pool was created first.
+    const pipelining = (options?.pipelining ?? true) ? '1' : '0';
+    const maxInFlight = options?.maxInFlight ?? 100;
+    return `${host}:${port}:${poolSize}:${tokenHash}:${tlsKey}:${pipelining}:${maxInFlight}`;
 }
 /** Get or create shared connection pool */
 export function getSharedPool(options) {

package/dist/infrastructure/server/handlers/core.js

@@ -5,6 +5,7 @@
 import * as resp from '../../../domain/types/response';
 import { jobId } from '../../../domain/types/job';
 import { validateQueueName, validateJobData, validateJobOptions, validateNumericField, } from '../protocol';
+import { validatePushBatchJobs } from './pushBatchValidation';
 /** Handle PUSH command */
 export async function handlePush(cmd, ctx, reqId) {
     const queueError = validateQueueName(cmd.queue);
@@ -86,11 +87,11 @@ export async function handlePushBatch(cmd, ctx, reqId) {
     const queueError = validateQueueName(cmd.queue);
     if (queueError)
         return resp.error(queueError, reqId);
-    for (const job of cmd.jobs) {
-        const dataError = validateJobData(job.data);
-        if (dataError)
-            return resp.error(dataError, reqId);
-    }
+    // PUSH parity: per-job data/option bounds + dependsOn existence gate
+    // (extended to earlier same-batch custom ids for intra-batch chains).
+    const jobsError = validatePushBatchJobs(cmd.jobs, ctx);
+    if (jobsError)
+        return resp.error(jobsError, reqId);
     const ids = await ctx.queueManager.pushBatch(cmd.queue, cmd.jobs);
     return resp.batch(ids, reqId);
 }
@@ -128,6 +129,10 @@ export async function handlePullBatch(cmd, ctx, reqId) {
     const countError = validateNumericField(cmd.count, 'count', { min: 1, max: 1000 });
     if (countError)
         return resp.error(countError, reqId);
+    // Validate timeout (same bound as PULL)
+    const timeoutError = validateNumericField(cmd.timeout, 'timeout', { min: 0, max: 60000 });
+    if (timeoutError)
+        return resp.error(timeoutError, reqId);
     // If owner is provided, use lock-based pull
     if (cmd.owner) {
         const { jobs, tokens } = await ctx.queueManager.pullBatchWithLock(cmd.queue, cmd.count, cmd.owner, cmd.timeout ?? 0, cmd.lockTtl);
@@ -139,8 +144,9 @@ export async function handlePullBatch(cmd, ctx, reqId) {
         }
         return resp.pulledJobs(jobs, tokens, reqId);
     }
-    // Standard pull (no locks, but still track for client release)
-    const jobs = await ctx.queueManager.pullBatch(cmd.queue, cmd.count, 0);
+    // Standard pull (no locks, but still track for client release) — the
+    // non-owner branch honors cmd.timeout exactly like the owner branch and PULL.
+    const jobs = await ctx.queueManager.pullBatch(cmd.queue, cmd.count, cmd.timeout ?? 0);
     if (ctx.clientId) {
         for (const job of jobs) {
             ctx.queueManager.registerClientJob(ctx.clientId, job.id);

package/dist/infrastructure/server/handlers/pushBatchValidation.d.ts

@@ -0,0 +1,23 @@
+/**
+ * PUSHB Validation
+ * Per-job validation for batch pushes with PUSH parity: the same
+ * validateJobData/validateJobOptions bounds and the same dependsOn
+ * existence gate handlePush enforces (core.ts).
+ */
+import type { JobInput } from '../../../domain/types/job';
+import type { HandlerContext } from '../types';
+/**
+ * Validate every job of a PUSHB batch. Returns an error message naming the
+ * offending index, or null when the whole batch is valid.
+ *
+ * The dependsOn gate is EXTENDED beyond the PUSH one: a dependency may also
+ * reference ANY OTHER job of the same batch via its custom id (the only ids
+ * a client can know before the batch is applied). Order within the batch is
+ * deliberately irrelevant: the default-on auto-batcher groups concurrent
+ * add() calls in arbitrary order, so a child can legitimately precede its
+ * parent in the array, and the readiness layer (waitingDeps) resolves the
+ * chain once the whole batch is applied. Self-references are rejected, they
+ * can never resolve. jobIndex/completedJobs/depCompletions are hoisted out
+ * of the loop because PUSHB is the bulk hot path.
+ */
+export declare function validatePushBatchJobs(jobs: JobInput[], ctx: HandlerContext): string | null;

package/dist/infrastructure/server/handlers/pushBatchValidation.js

@@ -0,0 +1,70 @@
+/**
+ * PUSHB Validation
+ * Per-job validation for batch pushes with PUSH parity: the same
+ * validateJobData/validateJobOptions bounds and the same dependsOn
+ * existence gate handlePush enforces (core.ts).
+ */
+import { validateJobData, validateJobOptions } from '../protocol';
+/**
+ * Validate every job of a PUSHB batch. Returns an error message naming the
+ * offending index, or null when the whole batch is valid.
+ *
+ * The dependsOn gate is EXTENDED beyond the PUSH one: a dependency may also
+ * reference ANY OTHER job of the same batch via its custom id (the only ids
+ * a client can know before the batch is applied). Order within the batch is
+ * deliberately irrelevant: the default-on auto-batcher groups concurrent
+ * add() calls in arbitrary order, so a child can legitimately precede its
+ * parent in the array, and the readiness layer (waitingDeps) resolves the
+ * chain once the whole batch is applied. Self-references are rejected, they
+ * can never resolve. jobIndex/completedJobs/depCompletions are hoisted out
+ * of the loop because PUSHB is the bulk hot path.
+ */
+export function validatePushBatchJobs(jobs, ctx) {
+    const jobIndex = ctx.queueManager.getJobIndex();
+    const completedJobs = ctx.queueManager.getCompletedJobs();
+    const depCompletions = ctx.queueManager.getDepCompletions();
+    // Custom ids of ALL jobs in this batch: they become real job ids the
+    // moment the batch is applied. Allocated lazily (most batches have neither
+    // custom ids nor dependencies).
+    let batchIds = null;
+    for (const job of jobs) {
+        if (job.customId) {
+            batchIds ??= new Set();
+            batchIds.add(job.customId);
+        }
+    }
+    for (let i = 0; i < jobs.length; i++) {
+        const job = jobs[i];
+        const dataError = validateJobData(job.data);
+        if (dataError)
+            return `jobs[${i}]: ${dataError}`;
+        const optionsError = validateJobOptions({
+            priority: job.priority,
+            delay: job.delay,
+            timeout: job.timeout,
+            maxAttempts: job.maxAttempts,
+            backoff: job.backoff,
+            ttl: job.ttl,
+        });
+        if (optionsError)
+            return `jobs[${i}]: ${optionsError}`;
+        if (job.dependsOn && job.dependsOn.length > 0) {
+            for (const depId of job.dependsOn) {
+                const key = String(depId);
+                if (key === job.customId) {
+                    return `jobs[${i}]: Job cannot depend on itself: ${key}`;
+                }
+                const exists = jobIndex.has(depId) ||
+                    completedJobs.has(depId) ||
+                    // removeOnComplete parents leave only a bare completion id behind;
+                    // the gate must honor it exactly like PUSH does.
+                    depCompletions.has(depId) ||
+                    (batchIds?.has(key) ?? false);
+                if (!exists) {
+                    return `jobs[${i}]: Dependency job not found: ${key}`;
+                }
+            }
+        }
+    }
+    return null;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bunqueue",
-  "version": "2.8.30",
+  "version": "2.8.32",
   "description": "High-performance job queue for Bun & AI agents. SQLite persistence, cron scheduling, priorities, retries, DLQ, webhooks, native MCP server. Zero external dependencies.",
   "type": "module",
   "main": "dist/main.js",