power-queues 2.0.12 → 2.0.13

package/README.md

@@ -1,146 +1,156 @@
 # power-queues
-A production-ready, **Redis-backed queue runner** with visibility timeouts, delayed scheduling, chainable queues, retries with exponential backoff + jitter, and heartbeat renewals — built on top of a thin ```PowerRedis``` abstraction.
+## High-Performance Redis Streams Queue for Node.js
 
-This module exposes a base abstract class PowerQueue which you extend and implement. It supports:
+Ultra-fast, fault-tolerant, Lua-optimized distributed task queue built on Redis Streams.  
+Supports **bulk XADD**, **idempotent jobs**, **retries**, **DLQ**, **stuck-task recovery**, **batching**, and **consumer groups**.  
+Designed for large-scale microservices, telemetry pipelines, and high-load systems.
 
-- FIFO reservation (`LMOVE LEFT->RIGHT` preferred, `RPOPLPUSH` fallback) via Lua scripts or client loop.
-- **Visibility timeout (VT)** with periodic heartbeat (ZSET scores store deadlines).
-- **Delayed tasks** (ZSET with future timestamps) and promotion to ready list.
-- **Retry policy** with exponential backoff + jitter and a **fail** queue after max attempts.
-- **Chain mode**: automatically forward a task through a list of queues and call lifecycle hooks.
-- Iteration loop with promote/requeue passes, concurrency slicing, and status counters with TTL.
+<p align="center">
+  <img src="https://img.shields.io/badge/redis-streams-red?logo=redis" />
+  <img src="https://img.shields.io/badge/nodejs-queue-green?logo=node.js" />
+  <img src="https://img.shields.io/badge/typescript-ready-blue?logo=typescript" />
+  <img src="https://img.shields.io/badge/nestjs-support-ea2845?logo=nestjs" />
+  <img src="https://img.shields.io/badge/license-MIT-lightgrey" />
+  <img src="https://img.shields.io/badge/status-production-success" />
+</p>
 
-It provides the core logic for **visibility timeouts**, **task retries**, **delayed scheduling**, **chain forwarding**, and **concurrent workers** — without any external dependencies.
+---
 
-> Built on top of [**PowerRedis**](https://github.com/ihor-bielchenko/power-redis)  
-> Uses utilities from [**full-utils**](https://github.com/ihor-bielchenko/full-utils)
+## 📚 Documentation
 
-## API (with examples)
-Below are brief excerpts. Full JSDoc: power-queues.docs.ihor.bielchenko.com.
+Full documentation is available here:  
+👉 **https://power-queues.docs.ihor.bielchenko.com**
 
-## Overview
+---
 
-PowerQueue is not a “ready-made” message broker like BullMQ — it’s a **foundation** for building your own  
-custom Redis-based queues with full control over task lifecycle and execution flow.
+## 🚀 Features
 
-It manages:
+- ⚡ **Bulk XADD** — send thousands of tasks in a single Redis call  
+- 🔁 **Retries & attempt tracking**  
+- 🧠 **Idempotent job execution** (Lua locks, TTL, start/done keys)  
+- 🧹 **Stuck task recovery** (XAUTOCLAIM + Lua-based recovery)  
+- 🌀 **Consumer groups + batching**  
+- 📥 **Dead Letter Queue (DLQ)**  
+- 🔐 **Stream trimming, approx/exact maxlen, minid window**  
+- 🧱 **Fully async, high-throughput, production-ready**  
 
-- `LIST` for **ready tasks** (RPUSH producers, LMOVE/RPOPLPUSH consumers)
-- `LIST` for **processing tasks** (temporary visibility list)
-- `ZSET` for **visibility timeouts** (invisible tasks until acknowledged or expired)
-- `ZSET` for **delayed scheduling** (future tasks activation)
+---
 
-Ideal for:
-- high-performance Redis-based backends,
-- telemetry and distributed job systems,
-- retryable or delayed workloads,
-- cron-like task promotion and requeueing.
+## 📦 Installation
 
+```bash
+npm install power-queues
+```
 
 ---
 
-## Key Features
+## 🧪 Quick Start
 
-- **Visibility timeout** — automatic requeue if worker crashes or fails to ack in time  
-- **Delayed tasks** — schedule jobs for future execution (e.g., 5 minutes later)  
-- **Concurrent processing** — process multiple jobs in parallel  
-- **Retry logic** — exponential backoff with configurable limits  
-- **Lifecycle hooks** — extend and customize every phase  
-- **Task chaining** — forward results between queues  
-- **Lua-optimized reservation** — atomic batch pulls via LMOVE or RPOPLPUSH fallback  
-- **Zero dependencies** — built purely on Node.js + PowerRedis + full-utils
+```ts
+import { QueueService } from './queue.service';
 
-## Installation
+const queue = new QueueService();
 
-```bash
-npm install power-queue
-# or
-yarn add power-queue
+// Add tasks
+await queue.addTasks('my_queue', [
+  { payload: { foo: 'bar' } },
+  { payload: { a: 1, b: 2 } },
+]);
+
+// Run worker
+queue.runQueue();
 ```
-PowerQueue works with any Redis client compatible with IORedis interface.
-
-## Example Usage
-### 1. Create your custom queue
-```javascript
-import { PowerQueue } from 'power-queue';
-
-class EmailQueue extends PowerQueue {
-  /**
-   * Process a single task payload.
-   */
-  async execute(task) {
-    console.log('📨 Sending email:', task.payload);
-
-    // Simulate async work
-    await this.wait(200);
-
-    // Optionally return result
-    return { status: 'sent', timestamp: Date.now() };
-  }
-}
 
-export const emailQueue = new EmailQueue({
-  redis: { host: '127.0.0.1', port: 6379 },
+---
+
+## 🔧 Add Tasks (Bulk)
+
+```ts
+await queue.addTasks('mass_polling', largeArray, {
+  approx: true,
+  minidWindowMs: 30000,
+  maxlen: largeArray.length,
 });
+```
 
+---
+
+## 🏗️ Worker Hooks
+
+You can override:
+
+- `onExecute`
+- `onSuccess`
+- `onError`
+- `onRetry`
+- `onBatchError`
+- `onSelected`
+- `onReady`
+
+Example:
+
+```ts
+async onExecute(id, payload) {
+  console.log('executing', id, payload);
+}
 ```
-### 2. Enqueue a new task
-```javascript
-await emailQueue.enqueue('emails', {
-  to: 'user@example.com',
-  subject: 'Welcome!',
-  body: 'Hello and thanks for joining!',
-});
 
+---
+
+## 🧱 Architecture Overview
+
+```
+Producer → Redis Stream → Consumer Group → Worker → DLQ (optional)
+```
+
+- Redis Streams store tasks  
+- Lua scripts handle trimming, idempotency, stuck recovery  
+- Workers fetch tasks via XREADGROUP or Lua select  
+- Tasks executed, ACKed, or sent to DLQ  
+
+---
+
+## 🗄️ Dead Letter Queue (DLQ)
+
+Failed tasks after `workerMaxRetries` automatically go to:
+
+```
+<stream>:dlq
 ```
-### 3. Start processing
-```javascript
-emailQueue.start('emails');
+
+---
+
+## 🧩 Idempotency
+
+Guaranteed by 3 keys:
+
+- `doneKey`
+- `lockKey`
+- `startKey`
+
+This prevents double-execution during retries, crashes, or concurrency.
+
+---
+
+## 🚀 Performance
+
+- 10,000+ XADDs/sec  
+- Bulk mode: 50,000 operations in one request  
+- Extremely low CPU usage due to Lua trimming  
+
+---
+
+## 🏷️ SEO Keywords
 
 ```
-That’s it — PowerQueue will:
-- Pull tasks from Redis LIST atomically
-- Move them into processing + visibility sets
-- Retry or requeue on error or timeout
-- Handle parallel processing with full isolation
-
-## Configuration Options
-Each property is strongly typed and fully documented in TypeDoc.
-| Property               | Type                                | Default | Description                                    |
-| ---------------------- | ----------------------------------- | ------- | ---------------------------------------------- |
-| `iterationTimeout`     | `number`                            | `1000`  | Delay (ms) between polling loops               |
-| `portionLength`        | `number`                            | `1000`  | Max number of tasks per batch pull             |
-| `expireStatusSec`      | `number`                            | `300`   | TTL for progress tracking                      |
-| `maxAttempts`          | `number`                            | `1`     | Max retry attempts before marking as failed    |
-| `concurrency`          | `number`                            | `32`    | Parallel runners per queue                     |
-| `visibilityTimeoutSec` | `number`                            | `60`    | How long a task stays invisible before requeue |
-| `retryBaseSec`         | `number`                            | `1`     | Base delay for exponential backoff             |
-| `retryMaxSec`          | `number`                            | `3600`  | Max retry delay                                |
-| `runners`              | `Map<string, { running: boolean }>` | —       | Active queue execution flags                   |
-| `processingRaw`        | `Map<string, string>`               | —       | Tracks raw Redis entries for acknowledgment    |
-| `heartbeatTimers`      | `Map<string, NodeJS.Timeout>`       | —       | Maintains active visibility extensions         |
-
-
-## Design Principles
-### Minimalism
-Only core Redis primitives are used — no unnecessary abstractions or event emitters.
-
-### Predictability
-All queue state is transparent and Redis-inspectable:
-- ```LIST``` for ready and processing states
-- ```ZSET``` for timeouts and scheduling
-
-### Resilience
-Every step (reserve, ack, retry, promote) can recover after process restart.
-
-### Extensibility
-Implement custom behaviors by overriding:
-```javascript
-async beforeExecute(task) {}
-async afterExecute(task, result) {}
-async onError(task, error) {}
-async onRetry(task, delaySec) {}
+redis streams, redis queue, task queue, job queue, nodejs queue, nestjs queue,
+bulk xadd, distributed queue system, background jobs, retries, dlq,
+idempotency, redis lua scripts, microservices, high-performance queue,
+high-throughput, batching, concurrency control
 ```
 
-## License
-Use freely in your own projects. Add proper notices if you publish a package (MIT/Apache-2.0, etc.).
+---
+
+## 📜 License
+
+MIT

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "power-queues",
-	"version": "2.0.12",
+	"version": "2.0.13",
 	"description": "Base classes for implementing custom queues in redis under high load conditions based on nestjs.",
 	"author": "ihor-bielchenko",
 	"license": "MIT",