@koala42/redis-highway 0.2.4 → 0.2.9

package/README.md

@@ -1,16 +1,14 @@
 # @koala42/redis-highway
 
-High performance Redis stream-based queue for Node.js. Supports Redis single instances and Valkey single instances
+High performance Redis stream-based queue for Node.js. Supports Redis single instances and Valkey single instances.
+Designed for high throughput and massive concurrency with low overhead.
 
-## Missing features ATM
-- Better gracefull shutdown handling
-- In worker process functions expose more than just data: T like job id and current status
-- Option to customize/enable/disable metrics
-- Enable custom logger for workers and producers instead of console.logs
-
-
-## Roadmap
-- Support redis cluster, that is probably possible only for job payloads and DLQ, since the stream is only one
+## Features
+- **Lightweight**: Uses optimized Lua scripts and pipelines for maximum performance and reduced I/O.
+- **Micro-Batching**: Supports batch processing for high-volume message consumption.
+- **Granular Retries**: Consumer group isolation - if one group fails, only that group retries.
+- **Reliability**: Auto-claiming of stuck messages (crashed consumers) and Dead Letter Queue (DLQ) support.
+- **Metrics**: Built-in tracking for throughput, queue depth, DLQ size, and retries. Prometheus export ready.
 
 ## Installation
 
@@ -27,12 +25,13 @@ import { Redis } from 'ioredis';
 import { Producer } from '@koala42/redis-highway';
 
 const redis = new Redis();
-const producer = new Producer(redis, 'my-stream');
+const producer = new Producer<{hello: string}>(redis, 'my-stream');
 
 // Send job
 await producer.push(
-  JSON.stringify({ hello: 'world' }), // Message serialization is not done automatically
-  ['group-A', 'group-B'] // Target specific consumer groups
+  { hello: 'world' }, // Type-safe payload
+  ['group-A', 'group-B'], // Target specific consumer groups
+  { ttl: 3600 } // Optional: expiration time in seconds
 );
 ```
 
@@ -42,17 +41,55 @@ await producer.push(
 import { Redis } from 'ioredis';
 import { Worker } from '@koala42/redis-highway';
 
-class MyWorker extends Worker<T> {
-  async process(data: T) {
-    console.log('Processing:', data);
-    // throw new Error('fail'); // Automatic retry/DLQ logic
+class MyWorker extends Worker<{hello: string}> {
+  async process(data: {hello: string}) {
+    console.log('Processing:', data.hello);
+    // throw new Error('fail'); // Triggers automatic retry logic
   }
 }
 
 const redis = new Redis();
-const worker = new MyWorker(redis, 'group-A', 'my-stream');
+const worker = new MyWorker(
+  redis,
+  {
+    groupName: 'group-A',
+    streamName: 'my-stream',
+    concurrency: 10 // Number of concurrent jobs to process
+  }
+);
 
 await worker.start();
+
+// To stop gracefully
+// await worker.stop();
+```
+
+### Batch Worker
+Process messages in batches for higher throughput.
+
+```typescript
+import { Redis } from 'ioredis';
+import { BatchWorker } from '@koala42/redis-highway';
+
+class MyBatchWorker extends BatchWorker<{hello: string}> {
+  async process(batchedData: {hello: string}[]) {
+    console.log(`Processing batch of ${batchedData.length} items`);
+    // Example: Bulk insert into database
+  }
+}
+
+const batchWorker = new MyBatchWorker(
+  redis,
+  {
+    groupName: 'group-B',
+    streamName: 'my-stream',
+    concurrency: 50, // Total items processing limit
+    batchSize: 10,   // Items per batch
+    maxFetchCount: 50
+  }
+);
+
+await batchWorker.start();
 ```
 
 ### Metrics
@@ -62,63 +99,94 @@ import { Metrics } from '@koala42/redis-highway';
 
 const metrics = new Metrics(redis, 'my-stream');
 
-// Prometheus format
-const payload = await metrics.getPrometheusMetrics(['group-A']);
+// Get raw metrics object
+const stats = await metrics.getMetrics(['group-A', 'group-B']);
+console.log(stats.throughput);
+
+// Get Prometheus formatted string
+const promMetrics = await metrics.getPrometheusMetrics(['group-A'], 'my_app_queue');
 ```
 
-## Usage with NestJS
+## Configuration
+
+### Worker Options
+The second argument to `Worker` and `BatchWorker` constructors is the primary configuration object.
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `groupName` | string | **Required**. The consumer group name (e.g., 'email-service'). |
+| `streamName` | string | **Required**. The Redis stream key. |
+| `concurrency` | number | **Required**. Maximum number of messages processed in parallel by this worker instance. |
+| `batchSize` | number | **Required (BatchWorker only)**. Number of messages to process in a single call. |
+| `maxFetchCount` | number | **Required (BatchWorker only)**. limit for XREADGROUP count. |
+
+### Control Options
+The third argument is for fine-tuning retry and recovery behavior.
 
 ```typescript
+const worker = new MyWorker(redis, { ... }, {
+  maxRetries: 3,         // Default: 3
+  blockTimeMs: 2000,     // Default: 2000. XREADGROUP block time.
+  minIdleTimeMs: 120000, // Default: 2 minutes. Time before a message is considered stuck.
+  claimIntervalMs: 120000,// Default: 2 minutes. How often to check for stuck messages.
+  collectMetrics: true    // Default: true. Enable throughput tracking.
+});
+```
 
-// Producer
+| Option | Default | Description |
+|--------|---------|-------------|
+| `maxRetries` | 3 | Number of times to retry a failed message before moving it to DLQ. |
+| `blockTimeMs` | 2000 | Redis blocking timeout for fetching new messages (in ms). |
+| `minIdleTimeMs` | 120000 | Messages pending longer than this are candidates for auto-claim (recovery). |
+| `claimIntervalMs` | 120000 | Interval for checking and claiming stuck messages. |
+| `collectMetrics` | true | If true, increments throughput counters in Redis. |
+
+## Usage with NestJS
+
+```typescript
+// Producer Service
 @Injectable()
 export class EntryService {
-  privater readonly producer: Producer;
-  
-  constructor(){
-    this.producer = new Producer(
-      new Redis(...), // Or reuse existing ioredis connection
-      'my-stream'
-    )
+  private readonly producer: Producer<MyPayload>;
+
+  constructor(@InjectRedis() private readonly redis: Redis) {
+    this.producer = new Producer(this.redis, 'my-stream');
   }
-  
-  public async sth(): Promise<void>{
-    await producer.push(
-      JSON.stringify({ hello: 'world' }), // Message serialization is not done automatically
-      ['group-A', 'group-B'] // Target specific consumer groups
-    );
+
+  async addToQueue(data: MyPayload) {
+    await this.producer.push(data, ['group-A']);
   }
 }
 
-
-// Processor
+// Worker Service
 @Injectable()
-export class ProcessorService extends Worker<T> implements OnModuleInit, OnModuleDestroy {
-  constructor(){
-    super(
-      new Redis(...), // or reuse existing redis conn
-      'group-A',
-      'my-stream',
-      50 // concurrency
-    )
+export class ProcessorService extends Worker<MyPayload> implements OnModuleInit, OnModuleDestroy {
+  constructor(@InjectRedis() redis: Redis) {
+    super(redis, {
+      groupName: 'group-A',
+      streamName: 'my-stream',
+      concurrency: 50
+    });
   }
-  
-  async onModuleInit(): Promise<void>{
-    await this.start()
+
+  async onModuleInit() {
+    await this.start();
   }
-  
-  onModuleDestroy(){
-    this.stop()
+
+  async onModuleDestroy() {
+    await this.stop();
   }
-  
-  async process(data: T): Promise<void>{
-    console.log("Processing job", JSON.stringify(data))
+
+  async process(data: MyPayload) {
+    // Process your job here
   }
 }
-````
+```
 
-## Features
-- **Lightweight**: Uses light Lua scripts and pipelines wherever possible, making it highly concurrents for inserts and for processing as well, because of the reduced I/O load compared to BullMQ
-- **Granular Retries**: If one group fails, only that group retries.
-- **DLQ**: Dead Letter Queue support after max retries.
-- **Metrics**: Throughput, Waiting, DLQ, Prometheus export.
+## Roadmap & Missing Features
+tracked in [Github Issues](https://github.com/Koala42/redis-highway/issues)
+
+## AI Usage Disclosure
+- AI will not be used for the development, ever
+- AI may be used to do code reviews
+- AI may be used to write unit tests

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@koala42/redis-highway",
-  "version": "0.2.4",
+  "version": "0.2.9",
   "description": "High performance redis queue",
   "license": "MIT",
   "author": {
@@ -8,6 +8,11 @@
     "name": "David Stranava",
     "url": "https://github.com/stranavad"
   },
+  "repository": {
+    "url": "https://github.com/Koala42/redis-highway",
+    "type": "github",
+    "directory": "packages/redis-highway"
+  },
   "type": "commonjs",
   "main": "dist/src/index.js",
   "types": "dist/src/index.d.ts",
@@ -16,7 +21,8 @@
   ],
   "scripts": {
     "clean": "rimraf dist",
-    "test:all": "vitest run test/queue.spec.ts test/batch-worker.spec.ts",
+    "test": "vitest run test",
+    "test:all": "vitest run test",
     "build": "npm run clean && tsc",
     "prepublish": "npm run build"
   },
@@ -36,4 +42,4 @@
     "typescript": "^5.9.3",
     "vitest": "^4.0.16"
   }
-}
+}