bunqueue-client 0.1.3 → 0.1.5

package/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 Egeo Minotti
+Copyright (c) 2026 Egeo Minotti
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/README.md

@@ -8,11 +8,11 @@ The bunqueue server runs on Bun, distributed as a binary or a Docker image. This
 
 | Runtime | Status | Notes |
 |---|---|---|
-| Node.js 20 or later | Supported, 72/72 e2e and 8/8 integration tests | ESM. TypeScript files run directly on Node 22 or later via `--experimental-strip-types` |
-| Bun | Supported, 72/72 e2e and 8/8 integration tests | No additional configuration required |
-| Deno 2 or later | Supported, 72/72 e2e and 8/8 integration tests | Uses `node:` builtins and the npm `msgpackr` package |
+| Node.js 20 or later | Supported, 81/81 e2e and 8/8 integration tests | ESM. TypeScript files run directly on Node 22 or later via `--experimental-strip-types` |
+| Bun | Supported, 81/81 e2e and 8/8 integration tests | No additional configuration required |
+| Deno 2 or later | Supported, 81/81 e2e and 8/8 integration tests | Uses `node:` builtins and the npm `msgpackr` package |
 | tsx, ts-node, vitest, jest | Supported | These environments execute on Node.js |
-| Cloudflare Workers | Supported, 11/11 e2e tests inside workerd | Requires the `nodejs_compat` compatibility flag. The runtime is request scoped, so long lived worker loops are not available: consume in batches from Cron Triggers or Durable Object alarms, a pattern covered by the test suite. TLS connections require a publicly trusted certificate |
+| Cloudflare Workers | Supported, 16/16 e2e tests inside workerd, including Simple Mode and the full API surface | Requires the `nodejs_compat` compatibility flag. The runtime is request scoped, so long lived worker loops are not available: consume in batches from Cron Triggers or Durable Object alarms, a pattern covered by the test suite. TLS connections require a publicly trusted certificate |
 | Browser | Not supported | Raw TCP sockets are unavailable. Use the server HTTP API instead |
 
 Portability is guaranteed by design: the client relies exclusively on `node:*` builtins (`net`, `tls`, `events`, `crypto`, `os`), uses no `Bun.*` globals and no runtime specific imports, and carries a single runtime dependency, `msgpackr`.
@@ -24,15 +24,37 @@ npm install bunqueue-client
 # or: bun add bunqueue-client / pnpm add bunqueue-client / deno add npm:bunqueue-client
 ```
 
-## Quick start
+## Quick start, step by step
 
-Sixty seconds from zero to a working queue. Step 1, start the server (requires [Bun](https://bun.sh), or use the Docker image):
+Every step from zero to a production ready queue.
+
+### Step 1. Run the bunqueue server
+
+The server is the only component that requires [Bun](https://bun.sh). Pick one:
 
 ```bash
+# Option A: one command, no install (requires Bun)
 bunx bunqueue start
+
+# Option B: Docker, with persistent data
+docker run -d --name bunqueue \
+  -p 6789:6789 -p 6790:6790 \
+  -v bunqueue-data:/app/data \
+  ghcr.io/egeominotti/bunqueue:latest
+```
+
+Port 6789 is the TCP protocol (what this client uses), port 6790 is the HTTP API with `/health`, `/metrics`, and dashboard endpoints.
+
+### Step 2. Install the client
+
+```bash
+npm install bunqueue-client
+# or: bun add bunqueue-client / pnpm add bunqueue-client / deno add npm:bunqueue-client
 ```
 
-Step 2, create `app.ts`: add a job and process it, in the same file for the sake of the demo:
+### Step 3. Add your first job and process it
+
+Create `app.ts`, one file for the sake of the demo:
 
 ```typescript
 import { Queue, Worker } from 'bunqueue-client';
@@ -51,7 +73,7 @@ await queue.add('greet', { name: 'world' });
 queue.close();
 ```
 
-Step 3, run it with the runtime you already use:
+Run it with the runtime you already use:
 
 ```bash
 node --experimental-strip-types app.ts    # Node 22 or later
@@ -66,7 +88,53 @@ processing { name: 'world' }
 completed 019f40a5-... { greeted: 'world' }
 ```
 
-That is the whole model: the server owns state, retries, and scheduling, your code only adds and processes. In production the producer and the worker are separate services, often in different languages: the [Python client](https://github.com/egeominotti/bunqueue/tree/main/sdk/python) speaks the same protocol against the same queue. Defaults are `host: 'localhost'`, `port: 6789`, so constructors need no options on a local setup.
+Defaults are `host: 'localhost'` and `port: 6789`, so constructors need no options on a local setup.
+
+### Step 4. Split producer and worker
+
+In production the producer and the worker are separate services, often in different languages. The producer is typically an API endpoint:
+
+```typescript
+// api-service: adds jobs, no processing
+import { Queue } from 'bunqueue-client';
+const queue = new Queue('emails', { host: 'queue.internal', port: 6789 });
+await queue.add('welcome', { to: 'user@example.com' }, { attempts: 3 });
+```
+
+The worker is a long running process:
+
+```typescript
+// worker-service: processes jobs, no HTTP
+import { Worker } from 'bunqueue-client';
+new Worker('emails', sendEmail, { host: 'queue.internal', port: 6789, concurrency: 10 });
+```
+
+The [Python client](https://github.com/egeominotti/bunqueue/tree/main/sdk/python) speaks the same protocol against the same queue, so the worker can be a Python service instead.
+
+### Step 5. Observe and operate
+
+```typescript
+await queue.getJobCounts();      // { waiting, active, completed, failed, delayed, ... }
+await queue.getDlq();            // jobs that exhausted their retries
+await queue.retryDlq();          // send them back to the queue
+await queue.getWorkers();        // connected workers
+await queue.getStats();          // throughput and totals
+```
+
+Or hit the HTTP side: `curl http://localhost:6790/health`.
+
+### Step 6. Go to production
+
+```typescript
+const queue = new Queue('emails', {
+  host: 'queue.example.com',
+  port: 6789,
+  token: process.env.BUNQUEUE_TOKEN,   // server started with AUTH_TOKENS=...
+  tls: true,                            // or { caFile: './ca.pem' }
+});
+```
+
+Checklist: set `AUTH_TOKENS` on the server, enable TLS (`TLS_CERT_FILE`/`TLS_KEY_FILE`), mount a volume for the SQLite data path, monitor `/health` and `/metrics`, and size worker `concurrency` to your workload. Full guide: [bunqueue.dev/guide/deployment](https://bunqueue.dev/guide/deployment/).
 
 ## Producing jobs
 
@@ -209,7 +277,7 @@ Authentication uses server side tokens (`AUTH_TOKENS`). Transport security uses
 | Area | Capabilities |
 |---|---|
 | Queue | `add`, `addBulk`, full `JobOptions`: priority, delay, attempts, backoff, ttl, timeout, jobId, deduplication, dependsOn, tags, groupId, lifo, removeOnComplete, removeOnFail, durable, repeat, debounce |
-| Query | `getJob`, `getJobByCustomId`, `getJobs` with per state helpers, state, result, progress, `waitForJob`, counts, counts per priority, children values, job logs |
+| Query | `getJob`, `getJobByCustomId`, `getJobs` with per state helpers, state, result, progress, `waitForJob` (throws on timeout, BullMQ contract), counts, counts per priority, children values, job logs |
 | Control | pause, resume, drain, obliterate, clean, remove, discard, promote, `retryJob`, `retryJobs`, move to wait or delayed, change priority or delay, update data, extend lock |
 | Dead letter queue | `getDlq`, `retryDlq`, `purgeDlq`, DLQ configuration |
 | Administration | rate limiting, global concurrency, stall configuration, webhooks, stats, metrics, `listQueues`, `getWorkers` |

package/dist/bunqueue/bunqueue.js

@@ -105,8 +105,10 @@ export class Bunqueue {
     }
     // ------------------------------------------------- core processing pipeline
     async processJob(job) {
-        if (this.rateGate)
+        if (this.rateGate) {
+            this.rateGate.prune(); // evict fully-expired groups (high-cardinality groupKey)
             await this.rateGate.acquire(this.rateGate.groupFor(job.data));
+        }
         // Circuit breaker check
         if (this.cb?.isOpen()) {
             throw new Error('Circuit breaker is open');

package/dist/bunqueue/rate-gate.d.ts

@@ -18,4 +18,10 @@ export declare class RateGate {
     groupFor(data: unknown): string;
     /** Wait until the group's window has room, then record the start. */
     acquire(group: string): Promise<void>;
+    /**
+     * Drop groups whose window is fully expired. Called on each acquire cycle
+     * boundary by the owner; without it a high-cardinality groupKey (e.g. one
+     * group per user id) grows the map forever.
+     */
+    prune(): void;
 }

package/dist/bunqueue/rate-gate.js

@@ -45,4 +45,17 @@ export class RateGate {
             await sleep(Math.max(oldest + this.duration - now, 10));
         }
     }
+    /**
+     * Drop groups whose window is fully expired. Called on each acquire cycle
+     * boundary by the owner; without it a high-cardinality groupKey (e.g. one
+     * group per user id) grows the map forever.
+     */
+    prune() {
+        const now = Date.now();
+        for (const [group, window] of this.windows) {
+            if (window.length === 0 || now - window[window.length - 1] >= this.duration) {
+                this.windows.delete(group);
+            }
+        }
+    }
 }

package/dist/connection.d.ts

@@ -25,6 +25,8 @@ export declare class Connection {
     private connectGeneration;
     private failedAttempts;
     private nextAttemptAt;
+    private readonly maxCommandTimeouts;
+    private consecutiveTimeouts;
     constructor(options?: ConnectionOptions);
     get isConnected(): boolean;
     /**
@@ -48,5 +50,11 @@ export declare class Connection {
     /** Close permanently; in-flight commands reject. */
     close(): void;
     private handleData;
+    /**
+     * A dead/half-open link makes every command time out while the socket still
+     * looks connected. After maxCommandTimeouts consecutive timeouts, tear down
+     * so the next call() reconnects instead of wedging (mirrors #94).
+     */
+    private noteTimeout;
     private teardown;
 }

package/dist/connection.js

@@ -27,6 +27,10 @@ export class Connection {
     connectGeneration = 0;
     failedAttempts = 0;
     nextAttemptAt = 0;
+    // Half-open recovery (#94): after this many consecutive command timeouts the
+    // socket is presumed dead and torn down so the next call reconnects.
+    maxCommandTimeouts = 3;
+    consecutiveTimeouts = 0;
     constructor(options = {}) {
         this.host = options.host ?? 'localhost';
         this.port = options.port ?? 6789;
@@ -79,6 +83,9 @@ export class Connection {
     async doConnect() {
         const socket = await openSocket(this.host, this.port, this.tls, this.connectTimeoutMs);
         socket.setNoDelay(true);
+        // TCP keepalive (~15s idle) surfaces a half-open link (cloud LB/NAT idle
+        // drop with no FIN/RST) in seconds instead of ~tcp_retries2 minutes.
+        socket.setKeepAlive(true, 15_000);
         this.parser.clear();
         this.socket = socket;
         socket.on('data', (chunk) => this.handleData(chunk));
@@ -86,6 +93,13 @@ export class Connection {
         socket.on('close', () => this.teardown());
         this.connected = true;
         this.connectGeneration += 1;
+        this.consecutiveTimeouts = 0;
+        // INVARIANT (H3): connected is flipped true before Auth, which is safe
+        // ONLY because call() writes the Auth frame synchronously — there is no
+        // `await` between this line and the Auth socket.write, so no concurrent
+        // call() can interleave a frame ahead of Auth on the wire. Do NOT insert
+        // an await here or before the Auth call, or a command could race ahead of
+        // Auth (the Python SDK guards this with a lock; JS relies on this ordering).
         if (this.token) {
             try {
                 await this.call({ cmd: 'Auth', token: this.token });
@@ -111,14 +125,17 @@ export class Connection {
         this.reqCounter = (this.reqCounter + 1) & 0x7fffffff;
         const reqId = String(this.reqCounter);
         const payload = pack({ ...compact(command), reqId });
+        const gen = this.connectGeneration; // snapshot: a timeout must not tear down a newer conn
         return new Promise((resolve, reject) => {
             const timer = setTimeout(() => {
                 this.pending.delete(reqId);
+                this.noteTimeout(gen);
                 reject(new CommandTimeoutError(`no response for ${command.cmd} within timeout`));
             }, timeoutMs ?? this.commandTimeoutMs);
             this.pending.set(reqId, {
                 resolve: (response) => {
                     clearTimeout(timer);
+                    this.consecutiveTimeouts = 0; // any reply means the link is alive
                     if (!response.ok) {
                         reject(new CommandError(String(response.error ?? 'unknown server error')));
                     }
@@ -196,6 +213,20 @@ export class Connection {
             }
         }
     }
+    /**
+     * A dead/half-open link makes every command time out while the socket still
+     * looks connected. After maxCommandTimeouts consecutive timeouts, tear down
+     * so the next call() reconnects instead of wedging (mirrors #94).
+     */
+    noteTimeout(gen) {
+        // A timeout from an already-replaced connection must not tear down (or
+        // miscount against) the current one.
+        if (gen !== this.connectGeneration)
+            return;
+        this.consecutiveTimeouts += 1;
+        if (this.consecutiveTimeouts >= this.maxCommandTimeouts)
+            this.teardown();
+    }
     teardown() {
         this.connected = false;
         const socket = this.socket;

package/dist/flow.js

@@ -4,6 +4,7 @@
  * UpdateParent fix-up, rollback via Cancel on failure).
  */
 import { Connection } from './connection.js';
+import { CommandError } from './errors.js';
 import { compact } from './frame.js';
 import { Job } from './job.js';
 import { jobPayload, wireJobOptions } from './types.js';
@@ -43,7 +44,7 @@ export class FlowProducer {
     }
     /** Fetch a flow tree starting from a job id (recursive over childrenIds). */
     getFlow(opts) {
-        return this.fetchNode(opts.id, opts.depth ?? Number.POSITIVE_INFINITY, opts.maxChildren);
+        return this.fetchNode(opts.id, opts.depth ?? Number.POSITIVE_INFINITY, opts.maxChildren, new Set());
     }
     /** Add a sequential chain: step[0] → step[1] → ... via dependsOn. */
     async addChain(steps) {
@@ -156,8 +157,24 @@ export class FlowProducer {
         }
         return parentJobId;
     }
-    async fetchNode(id, depth, maxChildren) {
-        const response = await this.connection.call({ cmd: 'GetJob', id });
+    async fetchNode(id, depth, maxChildren, visited) {
+        if (visited.has(id))
+            return null; // cycle guard: id already on the current path
+        visited.add(id);
+        // A missing job — the root, or a child removed via removeOnComplete/cancel
+        // (childrenIds is a static push-time list, never pruned) — yields null and
+        // is skipped, returning the surviving partial tree instead of throwing.
+        let response;
+        try {
+            response = await this.connection.call({ cmd: 'GetJob', id });
+        }
+        catch (err) {
+            // Only 'Job not found' means a removed node; a real server error must not
+            // masquerade as a missing child and yield a misleading partial tree.
+            if (err instanceof CommandError && /not found/i.test(err.message))
+                return null;
+            throw err;
+        }
         const raw = response.job;
         if (!raw)
             return null;
@@ -167,7 +184,7 @@ export class FlowProducer {
         const limit = maxChildren ?? job.childrenIds.length;
         const children = [];
         for (const childId of job.childrenIds.slice(0, limit)) {
-            const child = await this.fetchNode(childId, depth - 1, maxChildren);
+            const child = await this.fetchNode(childId, depth - 1, maxChildren, visited);
             if (child)
                 children.push(child);
         }

package/dist/queue-control.js

@@ -48,7 +48,9 @@ export const controlMethods = {
             await this.call({ cmd: 'RetryCompleted', queue: this.name });
             return;
         }
-        await this.call(compact({ cmd: 'RetryDlq', queue: this.name, count: opts.count }));
+        // `count` is accepted for API parity but not sent: the server has no
+        // partial RetryDlq — it retries the whole DLQ (BullMQ semantics).
+        await this.call({ cmd: 'RetryDlq', queue: this.name });
     },
     async retryCompleted(id) {
         await this.call(compact({ cmd: 'RetryCompleted', queue: this.name, id }));

package/dist/queue-query.d.ts

@@ -32,7 +32,14 @@ export declare const queryMethods: {
     removeChildDependency(this: Ctx, id: string): Promise<void>;
     /** Remove all still-unprocessed children of a parent job. */
     removeUnprocessedChildren(this: Ctx, id: string): Promise<void>;
-    /** Block until the job finishes; returns its result. */
+    /**
+     * Block until the job completes; returns its result.
+     * The server's WaitJob waiter resolves only on completion, replying
+     * `{ok:true, completed:false}` (no result) otherwise — so returning undefined
+     * would be indistinguishable from a genuine undefined result. On
+     * non-completion we probe the state: a `failed` job throws CommandError (it
+     * will not complete), everything else throws CommandTimeoutError.
+     */
     waitForJob<R = unknown>(this: Ctx, id: string, ttlMs?: number): Promise<R>;
     /** BullMQ v5 alias for waitForJob (queueEvents param unused over TCP). */
     waitJobUntilFinished<R = unknown>(this: Ctx, id: string, _queueEvents?: unknown, ttlMs?: number): Promise<R>;
@@ -50,7 +57,7 @@ export declare const queryMethods: {
     getWaitingChildrenCount(this: Ctx): Promise<number>;
     count(this: Ctx): Promise<number>;
     getCountsPerPriority(this: Ctx): Promise<Record<string, number>>;
-    addJobLog(this: Ctx, id: string, message: string): Promise<void>;
+    addJobLog(this: Ctx, id: string, message: string, level?: "info" | "warn" | "error"): Promise<void>;
     getJobLogs(this: Ctx, id: string, start?: number, end?: number): Promise<string[]>;
     clearJobLogs(this: Ctx, id: string, keepLogs?: number): Promise<void>;
 };

package/dist/queue-query.js

@@ -2,7 +2,7 @@
  * Queue query surface: job lookup, state, results, counts, logs, children.
  * Methods are merged onto Queue.prototype by queue.ts.
  */
-import { CommandError } from './errors.js';
+import { CommandError, CommandTimeoutError } from './errors.js';
 import { compact } from './frame.js';
 import { Job } from './job.js';
 function unwrapValues(response) {
@@ -53,7 +53,9 @@ export const queryMethods = {
         return jobs.map((raw) => new Job(raw, this.connection));
     },
     getWaiting(start, end) {
-        return this.getJobs({ state: ['waiting', 'prioritized'], start, end });
+        // Only the 'waiting' bucket — prioritized jobs live in a separate bucket
+        // (getPrioritized), matching BullMQ and the Python SDK / reference client.
+        return this.getJobs({ state: 'waiting', start, end });
     },
     getDelayed(start, end) {
         return this.getJobs({ state: 'delayed', start, end });
@@ -98,9 +100,28 @@ export const queryMethods = {
     async removeUnprocessedChildren(id) {
         await this.call({ cmd: 'RemoveUnprocessedChildren', id });
     },
-    /** Block until the job finishes; returns its result. */
+    /**
+     * Block until the job completes; returns its result.
+     * The server's WaitJob waiter resolves only on completion, replying
+     * `{ok:true, completed:false}` (no result) otherwise — so returning undefined
+     * would be indistinguishable from a genuine undefined result. On
+     * non-completion we probe the state: a `failed` job throws CommandError (it
+     * will not complete), everything else throws CommandTimeoutError.
+     */
     async waitForJob(id, ttlMs = 30_000) {
         const response = await this.call({ cmd: 'WaitJob', id, timeout: ttlMs }, ttlMs + 5000);
+        if (response.completed !== true) {
+            let state;
+            try {
+                state = await this.getJobState(id);
+            }
+            catch {
+                /* ignore probe failure; fall through to timeout */
+            }
+            if (state === 'failed')
+                throw new CommandError(`job ${id} failed before completion`);
+            throw new CommandTimeoutError(`waitUntilFinished timed out after ${ttlMs}ms`);
+        }
         return response.result;
     },
     /** BullMQ v5 alias for waitForJob (queueEvents param unused over TCP). */
@@ -120,8 +141,9 @@ export const queryMethods = {
         return response.counts;
     },
     async getWaitingCount() {
-        const counts = await this.getJobCounts();
-        return counts.waiting + counts.prioritized;
+        // 'waiting' only — prioritized jobs are counted by getPrioritizedCount,
+        // matching BullMQ and the Python SDK / reference client.
+        return (await this.getJobCounts()).waiting;
     },
     async getActiveCount() {
         return (await this.getJobCounts()).active;
@@ -150,14 +172,20 @@ export const queryMethods = {
         return (response.counts ?? response.data ?? {});
     },
     // --------------------------------------------------------------------- logs
-    async addJobLog(id, message) {
-        await this.call({ cmd: 'AddLog', id, message });
+    async addJobLog(id, message, level) {
+        await this.call(compact({ cmd: 'AddLog', id, message, level }));
     },
     async getJobLogs(id, start, end) {
         const response = await this.call(compact({ cmd: 'GetLogs', id, start, end }));
         const data = (response.data ?? {});
         const logs = (data.logs ?? response.logs ?? []);
-        return logs.map((row) => (typeof row === 'string' ? row : String(row.message ?? row)));
+        // Format as `[level] message` (reference client parity); never drop level.
+        return logs.map((row) => {
+            if (typeof row === 'string')
+                return row;
+            const r = row;
+            return r.level ? `[${r.level}] ${r.message}` : String(r.message ?? row);
+        });
     },
     async clearJobLogs(id, keepLogs) {
         await this.call(compact({ cmd: 'ClearLogs', id, keepLogs }));

package/dist/queue.js

@@ -50,10 +50,18 @@ export class Queue {
     }
     /** Add many jobs in one round-trip; returns Job stubs. */
     async addBulk(jobs) {
-        const inputs = jobs.map((entry) => ({
-            data: jobPayload(entry.name, entry.data),
-            ...wireJobOptions(entry.opts),
-        }));
+        const inputs = jobs.map((entry) => {
+            const opts = wireJobOptions(entry.opts);
+            // PUSHB entries are JobInput, whose custom-id field is `customId` —
+            // unlike single PUSH which renames `jobId`->`customId` server-side.
+            // Without this the batch custom id is silently dropped (idempotency /
+            // getJobByCustomId broken).
+            if (opts.jobId !== undefined) {
+                opts.customId = opts.jobId;
+                delete opts.jobId;
+            }
+            return { data: jobPayload(entry.name, entry.data), ...opts };
+        });
         const response = await this.call({ cmd: 'PUSHB', queue: this.name, jobs: inputs });
         const ids = (response.ids ?? []);
         return ids.map((id, i) => new Job({ id, queue: this.name, data: inputs[i].data }, this.connection));

package/dist/worker.js

@@ -107,7 +107,9 @@ export class Worker extends WorkerBase {
         catch (err) {
             this.failedCount += 1;
             const error = err instanceof Error ? err : new Error(String(err));
-            const stack = (error.stack ?? error.message).split('\n').slice(-MAX_STACK_LINES);
+            // Keep the FIRST lines: in a JS stack the message + throw site lead, so
+            // slice(0,N) preserves them (slice(-N) would drop them on long stacks).
+            const stack = (error.stack ?? error.message).split('\n').slice(0, MAX_STACK_LINES);
             await this.safeCall(compact({
                 cmd: 'FAIL',
                 id: job.id,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "bunqueue-client",
-  "version": "0.1.3",
+  "version": "0.1.5",
   "description": "Cross-runtime TypeScript client for the bunqueue job queue server — Node.js, Bun, Deno and Cloudflare Workers",
   "license": "MIT",
   "type": "module",
@@ -29,7 +29,7 @@
     "lint": "biome lint src tests",
     "format": "biome format --write src tests",
     "check": "biome check src tests",
-    "test:workers": "bun pm pack --destination tests/workers && cd tests/workers && mv bunqueue-client-*.tgz bunqueue-client.tgz && bun install && node run.mjs"
+    "test:workers": "bun pm pack --destination tests/workers && cd tests/workers && mv bunqueue-client-*.tgz bunqueue-client.tgz && rm -rf node_modules/bunqueue-client bun.lock && bun install --force && node run.mjs"
   },
   "keywords": [
     "queue",

package/src/bunqueue/bunqueue.ts

@@ -120,7 +120,10 @@ export class Bunqueue<T = unknown, R = unknown> {
   // ------------------------------------------------- core processing pipeline
 
   private async processJob(job: Job<T>): Promise<R> {
-    if (this.rateGate) await this.rateGate.acquire(this.rateGate.groupFor(job.data));
+    if (this.rateGate) {
+      this.rateGate.prune(); // evict fully-expired groups (high-cardinality groupKey)
+      await this.rateGate.acquire(this.rateGate.groupFor(job.data));
+    }
     // Circuit breaker check
     if (this.cb?.isOpen()) {
       throw new Error('Circuit breaker is open');

package/src/bunqueue/rate-gate.ts

@@ -50,4 +50,18 @@ export class RateGate {
       await sleep(Math.max(oldest + this.duration - now, 10));
     }
   }
+
+  /**
+   * Drop groups whose window is fully expired. Called on each acquire cycle
+   * boundary by the owner; without it a high-cardinality groupKey (e.g. one
+   * group per user id) grows the map forever.
+   */
+  prune(): void {
+    const now = Date.now();
+    for (const [group, window] of this.windows) {
+      if (window.length === 0 || now - window[window.length - 1] >= this.duration) {
+        this.windows.delete(group);
+      }
+    }
+  }
 }

package/src/connection.ts

@@ -40,6 +40,10 @@ export class Connection {
   private connectGeneration = 0;
   private failedAttempts = 0;
   private nextAttemptAt = 0;
+  // Half-open recovery (#94): after this many consecutive command timeouts the
+  // socket is presumed dead and torn down so the next call reconnects.
+  private readonly maxCommandTimeouts = 3;
+  private consecutiveTimeouts = 0;
 
   constructor(options: ConnectionOptions = {}) {
     this.host = options.host ?? 'localhost';
@@ -96,6 +100,9 @@ export class Connection {
   private async doConnect(): Promise<void> {
     const socket = await openSocket(this.host, this.port, this.tls, this.connectTimeoutMs);
     socket.setNoDelay(true);
+    // TCP keepalive (~15s idle) surfaces a half-open link (cloud LB/NAT idle
+    // drop with no FIN/RST) in seconds instead of ~tcp_retries2 minutes.
+    socket.setKeepAlive(true, 15_000);
     this.parser.clear();
     this.socket = socket;
 
@@ -105,7 +112,14 @@ export class Connection {
 
     this.connected = true;
     this.connectGeneration += 1;
+    this.consecutiveTimeouts = 0;
 
+    // INVARIANT (H3): connected is flipped true before Auth, which is safe
+    // ONLY because call() writes the Auth frame synchronously — there is no
+    // `await` between this line and the Auth socket.write, so no concurrent
+    // call() can interleave a frame ahead of Auth on the wire. Do NOT insert
+    // an await here or before the Auth call, or a command could race ahead of
+    // Auth (the Python SDK guards this with a lock; JS relies on this ordering).
     if (this.token) {
       try {
         await this.call({ cmd: 'Auth', token: this.token });
@@ -129,16 +143,19 @@ export class Connection {
     this.reqCounter = (this.reqCounter + 1) & 0x7fffffff;
     const reqId = String(this.reqCounter);
     const payload = pack({ ...compact(command), reqId });
+    const gen = this.connectGeneration; // snapshot: a timeout must not tear down a newer conn
 
     return new Promise<Response>((resolve, reject) => {
       const timer = setTimeout(() => {
         this.pending.delete(reqId);
+        this.noteTimeout(gen);
         reject(new CommandTimeoutError(`no response for ${command.cmd} within timeout`));
       }, timeoutMs ?? this.commandTimeoutMs);
 
       this.pending.set(reqId, {
         resolve: (response) => {
           clearTimeout(timer);
+          this.consecutiveTimeouts = 0; // any reply means the link is alive
           if (!response.ok) {
             reject(new CommandError(String(response.error ?? 'unknown server error')));
           } else {
@@ -216,6 +233,19 @@ export class Connection {
     }
   }
 
+  /**
+   * A dead/half-open link makes every command time out while the socket still
+   * looks connected. After maxCommandTimeouts consecutive timeouts, tear down
+   * so the next call() reconnects instead of wedging (mirrors #94).
+   */
+  private noteTimeout(gen: number): void {
+    // A timeout from an already-replaced connection must not tear down (or
+    // miscount against) the current one.
+    if (gen !== this.connectGeneration) return;
+    this.consecutiveTimeouts += 1;
+    if (this.consecutiveTimeouts >= this.maxCommandTimeouts) this.teardown();
+  }
+
   private teardown(): void {
     this.connected = false;
     const socket = this.socket;

package/src/flow.ts

@@ -5,6 +5,7 @@
  */
 
 import { Connection } from './connection.js';
+import { CommandError } from './errors.js';
 import type {
   FlowJob,
   FlowProducerOptions,
@@ -54,7 +55,12 @@ export class FlowProducer {
 
   /** Fetch a flow tree starting from a job id (recursive over childrenIds). */
   getFlow<T = unknown>(opts: GetFlowOptions): Promise<JobNode<T> | null> {
-    return this.fetchNode<T>(opts.id, opts.depth ?? Number.POSITIVE_INFINITY, opts.maxChildren);
+    return this.fetchNode<T>(
+      opts.id,
+      opts.depth ?? Number.POSITIVE_INFINITY,
+      opts.maxChildren,
+      new Set()
+    );
   }
 
   /** Add a sequential chain: step[0] → step[1] → ... via dependsOn. */
@@ -205,9 +211,23 @@ export class FlowProducer {
   private async fetchNode<T>(
     id: string,
     depth: number,
-    maxChildren?: number
+    maxChildren: number | undefined,
+    visited: Set<string>
   ): Promise<JobNode<T> | null> {
-    const response = await this.connection.call({ cmd: 'GetJob', id });
+    if (visited.has(id)) return null; // cycle guard: id already on the current path
+    visited.add(id);
+    // A missing job — the root, or a child removed via removeOnComplete/cancel
+    // (childrenIds is a static push-time list, never pruned) — yields null and
+    // is skipped, returning the surviving partial tree instead of throwing.
+    let response: Awaited<ReturnType<Connection['call']>>;
+    try {
+      response = await this.connection.call({ cmd: 'GetJob', id });
+    } catch (err) {
+      // Only 'Job not found' means a removed node; a real server error must not
+      // masquerade as a missing child and yield a misleading partial tree.
+      if (err instanceof CommandError && /not found/i.test(err.message)) return null;
+      throw err;
+    }
     const raw = response.job as Record<string, unknown> | null;
     if (!raw) return null;
     const job = new Job<T>(raw, this.connection);
@@ -216,7 +236,7 @@ export class FlowProducer {
     const limit = maxChildren ?? job.childrenIds.length;
     const children: JobNode<T>[] = [];
     for (const childId of job.childrenIds.slice(0, limit)) {
-      const child = await this.fetchNode<T>(childId, depth - 1, maxChildren);
+      const child = await this.fetchNode<T>(childId, depth - 1, maxChildren, visited);
       if (child) children.push(child);
     }
     return { job, children: children.length > 0 ? children : undefined };

package/src/queue-control.ts

@@ -72,11 +72,9 @@ export const controlMethods = {
       await this.call({ cmd: 'RetryCompleted', queue: this.name });
       return;
     }
-    await this.call(
-      compact({ cmd: 'RetryDlq', queue: this.name, count: opts.count }) as {
-        cmd: string;
-      }
-    );
+    // `count` is accepted for API parity but not sent: the server has no
+    // partial RetryDlq — it retries the whole DLQ (BullMQ semantics).
+    await this.call({ cmd: 'RetryDlq', queue: this.name });
   },
 
   async retryCompleted(this: Ctx, id?: string): Promise<void> {

package/src/queue-query.ts

@@ -3,7 +3,7 @@
  * Methods are merged onto Queue.prototype by queue.ts.
  */
 
-import { CommandError } from './errors.js';
+import { CommandError, CommandTimeoutError } from './errors.js';
 import { compact } from './frame.js';
 import { Job } from './job.js';
 import type { Queue } from './queue.js';
@@ -66,7 +66,9 @@ export const queryMethods = {
   },
 
   getWaiting<T = unknown>(this: Ctx, start?: number, end?: number): Promise<Job<T>[]> {
-    return this.getJobs({ state: ['waiting', 'prioritized'], start, end });
+    // Only the 'waiting' bucket — prioritized jobs live in a separate bucket
+    // (getPrioritized), matching BullMQ and the Python SDK / reference client.
+    return this.getJobs({ state: 'waiting', start, end });
   },
 
   getDelayed<T = unknown>(this: Ctx, start?: number, end?: number): Promise<Job<T>[]> {
@@ -125,9 +127,26 @@ export const queryMethods = {
     await this.call({ cmd: 'RemoveUnprocessedChildren', id });
   },
 
-  /** Block until the job finishes; returns its result. */
+  /**
+   * Block until the job completes; returns its result.
+   * The server's WaitJob waiter resolves only on completion, replying
+   * `{ok:true, completed:false}` (no result) otherwise — so returning undefined
+   * would be indistinguishable from a genuine undefined result. On
+   * non-completion we probe the state: a `failed` job throws CommandError (it
+   * will not complete), everything else throws CommandTimeoutError.
+   */
   async waitForJob<R = unknown>(this: Ctx, id: string, ttlMs = 30_000): Promise<R> {
     const response = await this.call({ cmd: 'WaitJob', id, timeout: ttlMs }, ttlMs + 5000);
+    if (response.completed !== true) {
+      let state: string | undefined;
+      try {
+        state = await this.getJobState(id);
+      } catch {
+        /* ignore probe failure; fall through to timeout */
+      }
+      if (state === 'failed') throw new CommandError(`job ${id} failed before completion`);
+      throw new CommandTimeoutError(`waitUntilFinished timed out after ${ttlMs}ms`);
+    }
     return response.result as R;
   },
 
@@ -157,8 +176,9 @@ export const queryMethods = {
   },
 
   async getWaitingCount(this: Ctx): Promise<number> {
-    const counts = await this.getJobCounts();
-    return counts.waiting + counts.prioritized;
+    // 'waiting' only — prioritized jobs are counted by getPrioritizedCount,
+    // matching BullMQ and the Python SDK / reference client.
+    return (await this.getJobCounts()).waiting;
   },
 
   async getActiveCount(this: Ctx): Promise<number> {
@@ -197,8 +217,13 @@ export const queryMethods = {
 
   // --------------------------------------------------------------------- logs
 
-  async addJobLog(this: Ctx, id: string, message: string): Promise<void> {
-    await this.call({ cmd: 'AddLog', id, message });
+  async addJobLog(
+    this: Ctx,
+    id: string,
+    message: string,
+    level?: 'info' | 'warn' | 'error'
+  ): Promise<void> {
+    await this.call(compact({ cmd: 'AddLog', id, message, level }) as { cmd: string });
   },
 
   async getJobLogs(this: Ctx, id: string, start?: number, end?: number): Promise<string[]> {
@@ -209,7 +234,12 @@ export const queryMethods = {
     );
     const data = (response.data ?? {}) as Raw;
     const logs = (data.logs ?? response.logs ?? []) as unknown[];
-    return logs.map((row) => (typeof row === 'string' ? row : String((row as Raw).message ?? row)));
+    // Format as `[level] message` (reference client parity); never drop level.
+    return logs.map((row) => {
+      if (typeof row === 'string') return row;
+      const r = row as Raw;
+      return r.level ? `[${r.level}] ${r.message}` : String(r.message ?? row);
+    });
   },
 
   async clearJobLogs(this: Ctx, id: string, keepLogs?: number): Promise<void> {

package/src/queue.ts

@@ -72,10 +72,18 @@ export class Queue<T = unknown> {
 
   /** Add many jobs in one round-trip; returns Job stubs. */
   async addBulk(jobs: BulkJobEntry<T>[]): Promise<Job<T>[]> {
-    const inputs = jobs.map((entry) => ({
-      data: jobPayload(entry.name, entry.data),
-      ...wireJobOptions(entry.opts),
-    }));
+    const inputs = jobs.map((entry) => {
+      const opts = wireJobOptions(entry.opts);
+      // PUSHB entries are JobInput, whose custom-id field is `customId` —
+      // unlike single PUSH which renames `jobId`->`customId` server-side.
+      // Without this the batch custom id is silently dropped (idempotency /
+      // getJobByCustomId broken).
+      if (opts.jobId !== undefined) {
+        opts.customId = opts.jobId;
+        delete opts.jobId;
+      }
+      return { data: jobPayload(entry.name, entry.data), ...opts };
+    });
     const response = await this.call({ cmd: 'PUSHB', queue: this.name, jobs: inputs });
     const ids = (response.ids ?? []) as string[];
     return ids.map(

package/src/worker.ts

@@ -127,7 +127,9 @@ export class Worker<T = unknown, R = unknown> extends WorkerBase {
     } catch (err) {
       this.failedCount += 1;
       const error = err instanceof Error ? err : new Error(String(err));
-      const stack = (error.stack ?? error.message).split('\n').slice(-MAX_STACK_LINES);
+      // Keep the FIRST lines: in a JS stack the message + throw site lead, so
+      // slice(0,N) preserves them (slice(-N) would drop them on long stacks).
+      const stack = (error.stack ?? error.message).split('\n').slice(0, MAX_STACK_LINES);
       await this.safeCall(
         compact({
           cmd: 'FAIL',