power-queues 2.0.13 → 2.0.15

package/README.md

@@ -1,15 +1,21 @@
-# power-queues
-## High-Performance Redis Streams Queue for Node.js
+# power-queues - High‑Performance Redis Streams Queue Engine for Node.js
 
-Ultra-fast, fault-tolerant, Lua-optimized distributed task queue built on Redis Streams.  
-Supports **bulk XADD**, **idempotent jobs**, **retries**, **DLQ**, **stuck-task recovery**, **batching**, and **consumer groups**.  
-Designed for large-scale microservices, telemetry pipelines, and high-load systems.
+Production‑ready, lightweight and highly scalable
+queue engine built directly on **Redis Streams + Lua scripts**.
+It is designed for real‑world distributed systems that require **high
+throughput**, **idempotent task execution**, **automatic recovery**, and
+**predictable performance under heavy load**.
+
+Unlike traditional Redis‑based queues that rely on lists or heavy
+abstractions, power-queues focuses on **low‑level control**, **atomic
+operations**, and **minimal overhead**, making it ideal for high‑load
+backends, microservices, schedulers, telemetry pipelines, and data
+processing clusters.
 
 <p align="center">
   <img src="https://img.shields.io/badge/redis-streams-red?logo=redis" />
   <img src="https://img.shields.io/badge/nodejs-queue-green?logo=node.js" />
   <img src="https://img.shields.io/badge/typescript-ready-blue?logo=typescript" />
-  <img src="https://img.shields.io/badge/nestjs-support-ea2845?logo=nestjs" />
   <img src="https://img.shields.io/badge/license-MIT-lightgrey" />
   <img src="https://img.shields.io/badge/status-production-success" />
 </p>
@@ -23,134 +29,278 @@ Full documentation is available here:
 
 ---
 
-## 🚀 Features
+## 🚀 Key Features
+
+### **1. Ultra‑Fast Bulk XADD (Lua‑Powered)**
+
+-   Adds thousands of messages per second using optimized Lua scripts.
+-   Minimizes round‑trips to Redis.
+-   Supports batching based on:
+    -   number of tasks
+    -   number of Redis arguments (safe upper bound)
+-   Outperforms typical list‑based queues and generic abstractions.
+
+---
+
+### **2. Built‑in Idempotent Workers**
+
+Every task can carry an `idemKey`, guaranteeing **exactly‑once
+execution** even under: - worker crashes
+- network interruptions
+- duplicate task submissions
+- process restarts
+
+Idempotency includes: - Lock key
+- Start key
+- Done key
+- TTL‑managed execution lock
+- Automatic release on failure
+- Heartbeat mechanism
+- Waiting on TTL for contended executions
+
+This makes the engine ideal for: - payment processing
+- external API calls
+- high‑value jobs
+- distributed pipelines
+
+---
+
+### **3. Stuck Task Recovery (Advanced Stream Scanning)**
+
+If a worker crashes mid‑execution, power-queues automatically detects: -
+abandoned tasks
+- stalled locks
+- unfinished start keys
+
+The engine then recovers these tasks back to active processing safely
+and efficiently.
+
+---
+
+### **4. High‑Throughput Workers**
+
+-   Batch execution support
+-   Parallel or sequential processing mode
+-   Configurable worker loop interval
+-   Individual and batch‑level error hooks
+-   Safe retry flow with per‑task attempt counters
+
+---
+
+### **5. Native DLQ (Dead‑Letter Queue)**
+
+When retries reach the configured limit: - the task is moved into
+`${stream}:dlq`
+- includes: payload, attempt count, job, timestamp, error text
+- fully JSON‑safe
+
+Perfect for monitoring or later re‑processing.
+
+---
+
+### **6. Zero‑Overhead Serialization**
+
+power-queues uses: - safe JSON encoding
+- optional "flat" key/value task format
+- predictable and optimized payload transformation
+
+This keeps Redis memory layout clean and eliminates overhead.
+
+---
+
+### **7. Complete Set of Lifecycle Hooks**
+
+You can extend any part of the execution flow:
+
+-   `onSelected`
+-   `onExecute`
+-   `onSuccess`
+-   `onError`
+-   `onRetry`
+-   `onBatchError`
+-   `onReady`
+
+This allows full integration with: - monitoring systems
+- logging pipelines
+- external APM tools
+- domain logic
+
+---
+
+### **8. Atomic Script Loading + NOSCRIPT Recovery**
+
+Scripts are: - loaded once
+- cached
+- auto‑reloaded if Redis restarts
+- executed safely via SHA‑based calls
 
-- ⚡ **Bulk XADD** — send thousands of tasks in a single Redis call  
-- 🔁 **Retries & attempt tracking**  
-- 🧠 **Idempotent job execution** (Lua locks, TTL, start/done keys)  
-- 🧹 **Stuck task recovery** (XAUTOCLAIM + Lua-based recovery)  
-- 🌀 **Consumer groups + batching**  
-- 📥 **Dead Letter Queue (DLQ)**  
-- 🔐 **Stream trimming, approx/exact maxlen, minid window**  
-- 🧱 **Fully async, high-throughput, production-ready**  
+Ensures resilience in failover scenarios.
+
+---
+
+### **9. Job Progress Tracking**
+
+Optional per‑job counters: - `job:ok` - `job:err` - `job:ready`
+
+Useful for UI dashboards and real‑time job progress visualization.
 
 ---
 
 ## 📦 Installation
 
-```bash
+``` bash
 npm install power-queues
 ```
-
+OR
+```bash
+yarn add power-redis
+```
 ---
 
 ## 🧪 Quick Start
 
-```ts
-import { QueueService } from './queue.service';
+``` ts
+const queue = new PowerQueues({
+  stream: "email",
+  group: "workers",
+});
 
-const queue = new QueueService();
+await queue.loadScripts(true);
 
-// Add tasks
-await queue.addTasks('my_queue', [
-  { payload: { foo: 'bar' } },
-  { payload: { a: 1, b: 2 } },
+await queue.addTasks("email", [
+  { payload: { type: "welcome", userId: 42 } },
 ]);
+```
+
+Worker:
 
-// Run worker
-queue.runQueue();
+``` ts
+class EmailWorker extends PowerQueues {
+  async onExecute(id, payload) {
+    await sendEmail(payload);
+  }
+}
 ```
 
 ---
 
-## 🔧 Add Tasks (Bulk)
+## ⚙ power-queues vs Existing Solutions
 
-```ts
-await queue.addTasks('mass_polling', largeArray, {
-  approx: true,
-  minidWindowMs: 30000,
-  maxlen: largeArray.length,
-});
-```
+|Feature               |power-queues    |BullMQ      |Bee-Queue   |Custom Streams|
+|----------------------|----------------|----------- |------------|--------------|
+|Bulk XADD (Lua)       |✅ Yes          |❌ No       |❌ No       |Rare          |
+|Idempotent workers    |✅ Built-in     |Partial     |❌ No       |❌ No         |
+|Stuck-task recovery   |✅ Advanced     |Basic       |❌ No       |Manual        |
+|Heartbeats            |✅ Yes          |Limited     |❌ No       |Manual        |
+|Retry logic           |✅ Flexible     |Good        |Basic       |Manual        |
+|DLQ                   |✅ Native       |Basic       |❌ No       |Manual        |
+|Pure Streams          |✅ Yes          |Partial     |❌ No       |Yes           |
+|Lua optimization      |✅ Strong       |Minimal     |❌ No       |Manual        |
+|Throughput            |🔥 Very high    |High        |Medium      |Depends       |
+|Overhead              |Low             |Medium      |Low         |Very high     |
 
----
+## 🛠 When to Choose power-queues
 
-## 🏗️ Worker Hooks
+Use this engine if you need:
 
-You can override:
+### **✔ High performance under load**
 
-- `onExecute`
-- `onSuccess`
-- `onError`
-- `onRetry`
-- `onBatchError`
-- `onSelected`
-- `onReady`
+Millions of tasks per hour? No problem.
 
-Example:
+### **✔ Strong idempotent guarantees**
 
-```ts
-async onExecute(id, payload) {
-  console.log('executing', id, payload);
-}
-```
+Exactly‑once processing for critical operations.
 
----
+### **✔ Low‑level control without heavy abstractions**
 
-## 🧱 Architecture Overview
+No magic, no hidden states - everything is explicit.
 
-```
-Producer → Redis Stream → Consumer Group → Worker → DLQ (optional)
-```
+### **✔ Predictable behavior in distributed environments**
 
-- Redis Streams store tasks  
-- Lua scripts handle trimming, idempotency, stuck recovery  
-- Workers fetch tasks via XREADGROUP or Lua select  
-- Tasks executed, ACKed, or sent to DLQ  
+Even with frequent worker restarts.
+
+### **✔ Production‑grade reliability**
+
+Backpressure, recovery, retries, dead-lettering - all included.
 
 ---
 
-## 🗄️ Dead Letter Queue (DLQ)
+## 🏗️ Project Structure & Architecture
 
-Failed tasks after `workerMaxRetries` automatically go to:
+-   Redis Streams for messaging
+-   Lua scripts for atomic operations
+-   JavaScript/TypeScript API
+-   Full worker lifecycle management
+-   Configurable backpressure & contention handling
+-   Optional job‑level progress tracking
 
-```
-<stream>:dlq
-```
+---
+
+## 🧩 Extensibility
+
+power-queues is ideal for building:
+
+-   task schedulers
+-   distributed cron engines
+-   ETL pipelines
+-   telemetry processors
+-   notification workers
+-   device monitoring systems
+-   AI job pipelines
+-   high-frequency background jobs
 
 ---
 
-## 🧩 Idempotency
+## 🧱 Reliability First
 
-Guaranteed by 3 keys:
+Every part of the engine is designed to prevent:
 
-- `doneKey`
-- `lockKey`
-- `startKey`
+-   double execution
+-   stuck tasks
+-   orphan locks
+-   lost messages
+-   zombie workers
+-   script desynchronization
 
-This prevents double-execution during retries, crashes, or concurrency.
+The heartbeat + TTL strategy guarantees that no task is "lost" even in
+chaotic cluster environments.
 
 ---
 
-## 🚀 Performance
+## 🏷️ SEO‑Optimized Keywords (Non‑Spam)
+
+power-queues is relevant for:
 
-- 10,000+ XADDs/sec  
-- Bulk mode: 50,000 operations in one request  
-- Extremely low CPU usage due to Lua trimming  
+-   Redis Streams queue engine
+-   Node.js stream-based queue
+-   idempotent task processing
+-   high‑performance job queue for Node.js
+-   Redis Lua queue
+-   distributed worker engine
+-   scalable background jobs
+-   enterprise-grade Redis queue
+-   microservices task runner
+-   fault-tolerant queue for Node.js
 
 ---
 
-## 🏷️ SEO Keywords
+## 📝 License
 
-```
-redis streams, redis queue, task queue, job queue, nodejs queue, nestjs queue,
-bulk xadd, distributed queue system, background jobs, retries, dlq,
-idempotency, redis lua scripts, microservices, high-performance queue,
-high-throughput, batching, concurrency control
-```
+MIT - free for commercial and private use.
 
 ---
 
-## 📜 License
+## ⭐ Why This Project Exists
+
+Most Node.js queue libraries are: - too slow
+- too abstract
+- not idempotent
+- not safe for financial or mission‑critical workloads
+
+power-queues was built to solve real production problems where: -
+*duplicate tasks cost money*,
+- *workers are unstable*,
+- *tasks must survive restarts*,
+- *performance matters at scale*.
 
-MIT
+If these things matter to you - this engine will feel like home.

package/dist/index.cjs

@@ -260,7 +260,6 @@ var PowerQueues = class extends import_power_redis.PowerRedis {
   constructor() {
     super(...arguments);
     this.abort = new AbortController();
-    this.strictCheckingConnection = ["true", "on", "yes", "y", "1"].includes(String(process.env.REDIS_STRICT_CHECK_CONNECTION ?? "").trim().toLowerCase());
     this.scripts = {};
     this.addingBatchTasksCount = 800;
     this.addingBatchKeysLimit = 1e4;
@@ -311,7 +310,7 @@ var PowerQueues = class extends import_power_redis.PowerRedis {
         }
         const tasksP = await this.onSelected(tasks);
         const ids = await this.execute((0, import_full_utils.isArrFilled)(tasksP) ? tasksP : tasks);
-        if ((0, import_full_utils.isArrFilled)(tasks)) {
+        if ((0, import_full_utils.isArrFilled)(ids)) {
           await this.approve(ids);
         }
       } catch (err) {
@@ -454,7 +453,7 @@ var PowerQueues = class extends import_power_redis.PowerRedis {
       }
       const pairs = flat.length / 2;
       if ((0, import_full_utils.isNumNZ)(pairs)) {
-        throw new Error('Task must have "payload" or "flat".');
+        throw new Error('Task "flat" must contain at least one field/value pair.');
       }
       argv.push(String(id));
       argv.push(String(pairs));
@@ -507,7 +506,13 @@ var PowerQueues = class extends import_power_redis.PowerRedis {
     return batches;
   }
   keysLength(task) {
-    return 2 + ("flat" in task && Array.isArray(task.flat) && task.flat.length ? task.flat.length : Object.keys(task).length * 2);
+    if ("flat" in task && Array.isArray(task.flat) && task.flat.length) {
+      return 2 + task.flat.length;
+    }
+    if ("payload" in task && (0, import_full_utils.isObj)(task.payload)) {
+      return 2 + Object.keys(task.payload).length * 2;
+    }
+    return 2 + Object.keys(task).length * 2;
   }
   attemptsKey(id) {
     const safeStream = this.stream.replace(/[^\w:\-]/g, "_");
@@ -674,7 +679,8 @@ var PowerQueues = class extends import_power_redis.PowerRedis {
               error: String(err?.message || err),
               createdAt,
               job,
-              id
+              id,
+              attempt: 0
             }
           }]);
           await this.clearAttempts(id);
@@ -777,42 +783,63 @@ var PowerQueues = class extends import_power_redis.PowerRedis {
       if (signal?.aborted) {
         return resolve();
       }
+      let delay;
+      if (ttl > 0) {
+        const base = Math.max(25, Math.min(ttl, 5e3));
+        const jitter = Math.floor(Math.min(base, 200) * Math.random());
+        delay = base + jitter;
+      } else {
+        delay = 5 + Math.floor(Math.random() * 15);
+      }
       const t = setTimeout(() => {
         if (signal) {
           signal.removeEventListener("abort", onAbort);
         }
         resolve();
-      }, ttl > 0 ? 25 + Math.floor(Math.random() * 50) : 5 + Math.floor(Math.random() * 15));
+      }, delay);
       t.unref?.();
       function onAbort() {
         clearTimeout(t);
         resolve();
       }
-      signal?.addEventListener("abort", onAbort, { once: true });
+      signal?.addEventListener?.("abort", onAbort, { once: true });
     });
   }
+  async sendHeartbeat(keys) {
+    try {
+      const r1 = await this.redis.pexpire(keys.lockKey, this.workerExecuteLockTimeoutMs);
+      const r2 = await this.redis.pexpire(keys.startKey, this.workerExecuteLockTimeoutMs);
+      const ok1 = Number(r1 || 0) === 1;
+      const ok2 = Number(r2 || 0) === 1;
+      return ok1 || ok2;
+    } catch {
+      return false;
+    }
+  }
   heartbeat(keys) {
     if (this.workerExecuteLockTimeoutMs <= 0) {
       return;
     }
-    let timer, alive = true, hbFails = 0;
     const workerHeartbeatTimeoutMs = Math.max(1e3, Math.floor(Math.max(5e3, this.workerExecuteLockTimeoutMs | 0) / 4));
+    let timer;
+    let alive = true;
+    let hbFails = 0;
     const stop = () => {
       alive = false;
       if (timer) {
         clearTimeout(timer);
       }
     };
-    const onAbort = () => stop();
     const signal = this.signal();
+    const onAbort = () => stop();
     signal?.addEventListener?.("abort", onAbort, { once: true });
     const tick = async () => {
       if (!alive) {
         return;
       }
       try {
-        const r = await this.heartbeat(keys);
-        hbFails = r ? 0 : hbFails + 1;
+        const ok = await this.sendHeartbeat(keys);
+        hbFails = ok ? 0 : hbFails + 1;
         if (hbFails >= 3) {
           throw new Error("Heartbeat lost.");
         }
@@ -823,9 +850,11 @@ var PowerQueues = class extends import_power_redis.PowerRedis {
           return;
         }
       }
-      timer = setTimeout(tick, workerHeartbeatTimeoutMs).unref?.();
+      timer = setTimeout(tick, workerHeartbeatTimeoutMs);
+      timer.unref?.();
     };
-    timer = setTimeout(tick, workerHeartbeatTimeoutMs).unref?.();
+    timer = setTimeout(tick, workerHeartbeatTimeoutMs);
+    timer.unref?.();
     return () => {
       signal?.removeEventListener?.("abort", onAbort);
       stop();

package/dist/index.d.cts

@@ -34,7 +34,6 @@ type Task = {
 declare class PowerQueues extends PowerRedis {
     abort: AbortController;
     redis: IORedisLike;
-    readonly strictCheckingConnection: boolean;
     readonly scripts: Record<string, SavedScript>;
     readonly addingBatchTasksCount: number;
     readonly addingBatchKeysLimit: number;
@@ -94,6 +93,7 @@ declare class PowerQueues extends PowerRedis {
     private selectStuck;
     private selectFresh;
     private waitAbortable;
+    private sendHeartbeat;
     private heartbeat;
     private normalizeEntries;
     private values;

package/dist/index.d.ts

@@ -34,7 +34,6 @@ type Task = {
 declare class PowerQueues extends PowerRedis {
     abort: AbortController;
     redis: IORedisLike;
-    readonly strictCheckingConnection: boolean;
     readonly scripts: Record<string, SavedScript>;
     readonly addingBatchTasksCount: number;
     readonly addingBatchKeysLimit: number;
@@ -94,6 +93,7 @@ declare class PowerQueues extends PowerRedis {
     private selectStuck;
     private selectFresh;
     private waitAbortable;
+    private sendHeartbeat;
     private heartbeat;
     private normalizeEntries;
     private values;

package/dist/index.js

@@ -243,7 +243,6 @@ var PowerQueues = class extends PowerRedis {
   constructor() {
     super(...arguments);
     this.abort = new AbortController();
-    this.strictCheckingConnection = ["true", "on", "yes", "y", "1"].includes(String(process.env.REDIS_STRICT_CHECK_CONNECTION ?? "").trim().toLowerCase());
     this.scripts = {};
     this.addingBatchTasksCount = 800;
     this.addingBatchKeysLimit = 1e4;
@@ -294,7 +293,7 @@ var PowerQueues = class extends PowerRedis {
         }
         const tasksP = await this.onSelected(tasks);
         const ids = await this.execute(isArrFilled(tasksP) ? tasksP : tasks);
-        if (isArrFilled(tasks)) {
+        if (isArrFilled(ids)) {
           await this.approve(ids);
         }
       } catch (err) {
@@ -437,7 +436,7 @@ var PowerQueues = class extends PowerRedis {
       }
       const pairs = flat.length / 2;
       if (isNumNZ(pairs)) {
-        throw new Error('Task must have "payload" or "flat".');
+        throw new Error('Task "flat" must contain at least one field/value pair.');
       }
       argv.push(String(id));
       argv.push(String(pairs));
@@ -490,7 +489,13 @@ var PowerQueues = class extends PowerRedis {
     return batches;
   }
   keysLength(task) {
-    return 2 + ("flat" in task && Array.isArray(task.flat) && task.flat.length ? task.flat.length : Object.keys(task).length * 2);
+    if ("flat" in task && Array.isArray(task.flat) && task.flat.length) {
+      return 2 + task.flat.length;
+    }
+    if ("payload" in task && isObj(task.payload)) {
+      return 2 + Object.keys(task.payload).length * 2;
+    }
+    return 2 + Object.keys(task).length * 2;
   }
   attemptsKey(id) {
     const safeStream = this.stream.replace(/[^\w:\-]/g, "_");
@@ -657,7 +662,8 @@ var PowerQueues = class extends PowerRedis {
               error: String(err?.message || err),
               createdAt,
               job,
-              id
+              id,
+              attempt: 0
             }
           }]);
           await this.clearAttempts(id);
@@ -760,42 +766,63 @@ var PowerQueues = class extends PowerRedis {
       if (signal?.aborted) {
         return resolve();
       }
+      let delay;
+      if (ttl > 0) {
+        const base = Math.max(25, Math.min(ttl, 5e3));
+        const jitter = Math.floor(Math.min(base, 200) * Math.random());
+        delay = base + jitter;
+      } else {
+        delay = 5 + Math.floor(Math.random() * 15);
+      }
       const t = setTimeout(() => {
         if (signal) {
           signal.removeEventListener("abort", onAbort);
         }
         resolve();
-      }, ttl > 0 ? 25 + Math.floor(Math.random() * 50) : 5 + Math.floor(Math.random() * 15));
+      }, delay);
       t.unref?.();
       function onAbort() {
         clearTimeout(t);
         resolve();
       }
-      signal?.addEventListener("abort", onAbort, { once: true });
+      signal?.addEventListener?.("abort", onAbort, { once: true });
     });
   }
+  async sendHeartbeat(keys) {
+    try {
+      const r1 = await this.redis.pexpire(keys.lockKey, this.workerExecuteLockTimeoutMs);
+      const r2 = await this.redis.pexpire(keys.startKey, this.workerExecuteLockTimeoutMs);
+      const ok1 = Number(r1 || 0) === 1;
+      const ok2 = Number(r2 || 0) === 1;
+      return ok1 || ok2;
+    } catch {
+      return false;
+    }
+  }
   heartbeat(keys) {
     if (this.workerExecuteLockTimeoutMs <= 0) {
       return;
     }
-    let timer, alive = true, hbFails = 0;
     const workerHeartbeatTimeoutMs = Math.max(1e3, Math.floor(Math.max(5e3, this.workerExecuteLockTimeoutMs | 0) / 4));
+    let timer;
+    let alive = true;
+    let hbFails = 0;
     const stop = () => {
       alive = false;
       if (timer) {
         clearTimeout(timer);
       }
     };
-    const onAbort = () => stop();
     const signal = this.signal();
+    const onAbort = () => stop();
     signal?.addEventListener?.("abort", onAbort, { once: true });
     const tick = async () => {
       if (!alive) {
         return;
       }
       try {
-        const r = await this.heartbeat(keys);
-        hbFails = r ? 0 : hbFails + 1;
+        const ok = await this.sendHeartbeat(keys);
+        hbFails = ok ? 0 : hbFails + 1;
         if (hbFails >= 3) {
           throw new Error("Heartbeat lost.");
         }
@@ -806,9 +833,11 @@ var PowerQueues = class extends PowerRedis {
           return;
         }
       }
-      timer = setTimeout(tick, workerHeartbeatTimeoutMs).unref?.();
+      timer = setTimeout(tick, workerHeartbeatTimeoutMs);
+      timer.unref?.();
     };
-    timer = setTimeout(tick, workerHeartbeatTimeoutMs).unref?.();
+    timer = setTimeout(tick, workerHeartbeatTimeoutMs);
+    timer.unref?.();
     return () => {
       signal?.removeEventListener?.("abort", onAbort);
       stop();

package/package.json

@@ -1,7 +1,7 @@
 {
 	"name": "power-queues",
-	"version": "2.0.13",
-	"description": "Base classes for implementing custom queues in redis under high load conditions based on nestjs.",
+	"version": "2.0.15",
+	"description": "High-performance Redis Streams queue for Node.js with Lua-powered bulk XADD, idempotent workers, heartbeat locks, stuck-task recovery, retries, DLQ, and distributed processing.",
 	"author": "ihor-bielchenko",
 	"license": "MIT",
 	"repository": {
@@ -81,11 +81,8 @@
 		"power-redis"
 	],
 	"dependencies": {
-		"@nestjs-labs/nestjs-ioredis": "^11.0.4",
-		"@nestjs/common": "^11.1.8",
-		"full-utils": "^2.0.3",
-		"ioredis": "^5.8.2",
-		"power-redis": "^2.0.6",
+		"full-utils": "^2.0.5",
+		"power-redis": "^2.0.8",
 		"uuid": "^13.0.0"
 	}
 }