glide-mq 0.3.0 → 0.4.1

package/README.md

@@ -4,21 +4,41 @@ High-performance message queue for Node.js, built on Valkey/Redis Streams with [
 
 ## Performance
 
+### Processing throughput
+
 | Concurrency | Throughput |
 |-------------|-----------|
 | c=1 | 4,376 jobs/s |
-| c=10 | 20,979 jobs/s |
-| c=50 | 44,643 jobs/s |
+| c=5 | 14,925 jobs/s |
+| c=10 | 15,504 jobs/s |
+| c=50 | 48,077 jobs/s |
+
+### Bulk add (addBulk with Batch API)
+
+| Jobs | Serial | Batch | Speedup |
+|------|--------|-------|---------|
+| 200 | 76ms | 14ms | 5.4x |
+| 1,000 | 228ms | 18ms | 12.7x |
+
+### Payload compression
+
+| Mode | Stored size (15KB payload) | Savings |
+|------|--------------------------|---------|
+| Plain | 15,327 bytes | - |
+| Gzip | 331 bytes | 98% |
 
 No-op processor, Valkey 8.0, single node.
 
 ## Why
 
-- **Streams-first** - uses Redis Streams + consumer groups + PEL instead of Lists + BRPOPLPUSH. Fewer moving parts, built-in at-least-once delivery.
+- **Streams-first** - Redis Streams + consumer groups + PEL instead of Lists + BRPOPLPUSH. Fewer moving parts, built-in at-least-once delivery.
 - **Server Functions** - single `FUNCTION LOAD` instead of dozens of EVAL scripts. Persistent across restarts, no NOSCRIPT cache-miss errors.
 - **1 RTT per job** - `completeAndFetchNext` combines job completion + next job fetch + activation in a single FCALL round trip.
 - **Cluster-native** - hash-tagged keys from day one. No afterthought `{braces}` requirement.
 - **Native bindings** - built on [@glidemq/speedkey](https://github.com/avifenesh/speedkey) (valkey-glide with Rust core + NAPI).
+- **AZ-Affinity** - route reads to same-AZ replicas, reducing cross-AZ latency and AWS costs by up to 75%.
+- **Batch pipelining** - `addBulk` uses GLIDE's Batch API for single round-trip bulk operations (12.7x faster than serial).
+- **Transparent compression** - gzip payloads with zero-config decompression on workers (98% savings on repetitive data).
 
 ## Install
 
@@ -26,7 +46,7 @@ No-op processor, Valkey 8.0, single node.
 npm install glide-mq
 ```
 
-Requires Node.js 20+ and a running Valkey (7.2+) or Redis (6.2+) instance.
+Requires Node.js 20+ and a running Valkey (7.0+) or Redis (7.0+) instance for FUNCTION support.
 
 ## Quick Start
 
@@ -68,6 +88,7 @@ worker.on('failed', (job, err) => console.log(`Job ${job.id} failed: ${err.messa
 - **Global concurrency** - limit active jobs across all workers
 - **Job retention** - removeOnComplete/removeOnFail (count, age-based)
 - **Priorities** - encoded in sorted set scores, FIFO within same priority
+- **Compression** - transparent gzip for job payloads (Node.js zlib, zero deps)
 
 ### Workflows
 - **FlowProducer** - atomic parent-child job trees with nested flows
@@ -79,7 +100,13 @@ worker.on('failed', (job, err) => console.log(`Job ${job.id} failed: ${err.messa
 - **QueueEvents** - stream-based event subscription (added, completed, failed, stalled, etc.)
 - **Job schedulers** - cron patterns and fixed intervals for repeatable jobs
 - **Metrics** - getJobCounts, getMetrics
-- **OpenTelemetry** - optional spans for Queue.add, Worker.process, FlowProducer.add
+- **OpenTelemetry** - automatic tracing spans for Queue.add and FlowProducer.add operations (optional peer dependency)
+
+### Cloud-Native (GLIDE-exclusive)
+- **AZ-Affinity routing** - route reads to same-AZ replicas for lower latency and reduced cross-AZ costs
+- **IAM authentication** - native AWS ElastiCache/MemoryDB auth with auto-token refresh
+- **Batch API** - single round-trip bulk operations via GLIDE's non-atomic pipeline
+- **Multiplexed connections** - single connection per node instead of connection pools
 
 ### Operations
 - **Graceful shutdown** - SIGTERM/SIGINT handler, waits for active jobs
@@ -93,8 +120,16 @@ worker.on('failed', (job, err) => console.log(`Job ${job.id} failed: ${err.messa
 
 ```typescript
 const queue = new Queue('name', {
-  connection: { addresses: [{ host, port }], clusterMode: false },
-  prefix: 'glide', // key prefix (default: 'glide')
+  connection: {
+    addresses: [{ host, port }],
+    clusterMode: false,
+    readFrom: ReadFrom.AZAffinity,  // route reads to same-AZ replicas
+    clientAz: 'us-east-1a',
+    credentials: { password: 'secret' },
+    // or IAM: { type: 'iam', serviceType: 'elasticache', region: 'us-east-1', userId: 'user', clusterName: 'my-cluster' }
+  },
+  prefix: 'glide',
+  compression: 'gzip',  // transparent payload compression
 });
 
 await queue.add('jobName', data, {
@@ -107,6 +142,12 @@ await queue.add('jobName', data, {
   deduplication: { id: 'unique-key', mode: 'simple' },
 });
 
+// Bulk add - 12.7x faster than serial via Batch API
+await queue.addBulk([
+  { name: 'job1', data: { a: 1 } },
+  { name: 'job2', data: { a: 2 } },
+]);
+
 await queue.pause();
 await queue.resume();
 await queue.getJobCounts();  // { waiting, active, delayed, completed, failed }
@@ -118,6 +159,7 @@ await queue.close();
 
 ```typescript
 const worker = new Worker('name', async (job) => {
+  await job.log('Starting processing');
   await job.updateProgress(50);
   return result;
 }, {
@@ -127,6 +169,10 @@ const worker = new Worker('name', async (job) => {
   stalledInterval: 30000,
   lockDuration: 30000,
   limiter: { max: 100, duration: 60000 },
+  deadLetterQueue: { name: 'failed-jobs' },
+  backoffStrategies: {
+    custom: (attemptsMade, err) => attemptsMade * 1000,
+  },
 });
 
 worker.on('completed', (job, result) => {});
@@ -149,17 +195,23 @@ await flow.add({
   ],
 });
 
-// Chain: A -> B -> C (sequential)
-await chain(connection, 'tasks', [
+// Chain: A -> B -> C (sequential, each step receives previous result)
+await chain('tasks', [
   { name: 'step1', data: {} },
   { name: 'step2', data: {} },
-]);
+], connection);
 
-// Group: A + B + C (parallel)
-await group(connection, 'tasks', [
+// Group: A + B + C (parallel, parent completes when all children done)
+await group('tasks', [
   { name: 'task1', data: {} },
   { name: 'task2', data: {} },
-]);
+], connection);
+
+// Chord: run group in parallel, then callback with all results
+await chord('tasks', [
+  { name: 'task1', data: {} },
+  { name: 'task2', data: {} },
+], { name: 'aggregate', data: {} }, connection);
 ```
 
 ### Events
@@ -173,18 +225,149 @@ events.on('failed', ({ jobId, failedReason }) => {});
 events.on('stalled', ({ jobId }) => {});
 ```
 
+### OpenTelemetry
+
+```typescript
+// Install optional peer dependency
+// npm install @opentelemetry/api
+
+// Automatic tracing for Queue.add and FlowProducer.add
+const queue = new Queue('tasks', { connection });
+await queue.add('job', data);  // Creates span: glide-mq.queue.add
+
+// Custom tracer (optional)
+import { setTracer } from 'glide-mq';
+setTracer(customTracerInstance);
+```
+
+### Graceful Shutdown
+
+```typescript
+import { gracefulShutdown } from 'glide-mq';
+
+const queue = new Queue('tasks', { connection });
+const worker = new Worker('tasks', processor, { connection });
+const events = new QueueEvents('tasks', { connection });
+
+// Registers SIGTERM/SIGINT handlers and resolves when all components are closed
+await gracefulShutdown([queue, worker, events]);
+```
+
 ## Cluster Mode
 
 ```typescript
 const connection = {
   addresses: [{ host: 'cluster-node', port: 7000 }],
   clusterMode: true,
+  readFrom: ReadFrom.AZAffinity,
+  clientAz: 'us-east-1a',
 };
 
 // Everything works the same - keys are hash-tagged automatically
 const queue = new Queue('tasks', { connection });
 ```
 
+## Testing Mode
+
+glide-mq ships a built-in in-memory backend so you can unit-test your job processors **without a running Valkey instance**.
+
+```typescript
+import { TestQueue, TestWorker } from 'glide-mq/testing';
+
+const queue = new TestQueue('tasks');
+
+const worker = new TestWorker(queue, async (job) => {
+  return { processed: job.data };
+});
+
+worker.on('completed', (job, result) => {
+  console.log(`Job ${job.id} done:`, result);
+});
+
+worker.on('failed', (job, err) => {
+  console.error(`Job ${job.id} failed:`, err.message);
+});
+
+await queue.add('send-email', { to: 'user@example.com' });
+
+// Inspect state without touching Valkey
+const counts = await queue.getJobCounts();
+// { waiting: 0, active: 0, delayed: 0, completed: 1, failed: 0 }
+
+// Search jobs by name or data fields
+const jobs = await queue.searchJobs({ name: 'send-email', state: 'completed' });
+
+await worker.close();
+await queue.close();
+```
+
+`TestQueue` and `TestWorker` mirror the real `Queue` / `Worker` public API (add, addBulk, getJob, getJobs, getJobCounts, pause, resume, events, retries, concurrency), making it straightforward to swap implementations between test and production code.
+
+## Dashboard
+
+glide-mq exposes a REST + Server-Sent Events API that can be consumed by the [`@glidemq/dashboard`](https://github.com/avifenesh/glidemq-dashboard) UI package.
+
+### Quick start with the built-in demo server
+
+```bash
+cd demo
+npm install
+npm run dashboard   # starts http://localhost:3000
+```
+
+### REST endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/queues` | List all queues with counts and metrics |
+| `GET` | `/api/queues/:name` | Queue details + recent jobs |
+| `GET` | `/api/queues/:name/jobs/:id` | Single job details, state, logs |
+| `POST` | `/api/queues/:name/jobs` | Add a new job |
+| `POST` | `/api/queues/:name/pause` | Pause a queue |
+| `POST` | `/api/queues/:name/resume` | Resume a queue |
+| `POST` | `/api/queues/:name/jobs/:id/retry` | Retry a failed job |
+| `DELETE` | `/api/queues/:name/jobs/:id` | Remove a job |
+| `POST` | `/api/queues/:name/drain` | Drain all waiting jobs |
+| `POST` | `/api/queues/:name/obliterate` | Obliterate queue and all data |
+| `GET` | `/api/events` | SSE stream for real-time job events |
+
+### Real-time events via SSE
+
+```javascript
+const es = new EventSource('http://localhost:3000/api/events');
+es.onmessage = ({ data }) => {
+  const { queue, event, jobId } = JSON.parse(data);
+  // event: 'added' | 'completed' | 'failed' | 'progress' | 'stalled' | 'heartbeat'
+  console.log(`[${queue}] ${event} – job ${jobId}`);
+};
+```
+
+### Embedding the dashboard server in your own Express app
+
+```typescript
+import express from 'express';
+import { Queue, QueueEvents } from 'glide-mq';
+
+const app = express();
+app.use(express.json());
+
+const queues: Record<string, Queue> = {
+  orders: new Queue('orders', { connection }),
+  payments: new Queue('payments', { connection }),
+};
+
+app.get('/api/queues', async (_req, res) => {
+  const data = await Promise.all(
+    Object.entries(queues).map(async ([name, q]) => ({
+      name,
+      counts: await q.getJobCounts(),
+      isPaused: await q.isPaused(),
+    })),
+  );
+  res.json(data);
+});
+```
+
 ## License
 
 Apache-2.0

package/demo/README.md

@@ -0,0 +1,169 @@
+# glide-mq Demo Application
+
+Comprehensive demonstration of all glide-mq features simulating a complete e-commerce platform.
+
+## Features Demonstrated
+
+### Core Queue Operations
+- Job processing with progress tracking
+- Bulk operations with Batch API
+- Priority queues with FIFO ordering
+- Delayed/scheduled jobs
+- Job retries with exponential backoff
+- Dead letter queue handling
+
+### Advanced Features
+- Deduplication (simple, throttle, debounce)
+- Rate limiting (sliding window)
+- Global concurrency limits
+- Job retention policies
+- Transparent gzip compression
+- Timeout handling
+
+### Workflows
+- FlowProducer for parent-child job trees
+- Chain pattern for sequential pipelines
+- Group pattern for parallel execution
+- Complex nested workflows
+
+### Observability
+- Real-time event streaming
+- Progress tracking
+- Metrics and job counts
+- Job logs and state tracking
+- Dashboard API integration
+
+## Setup
+
+1. Install dependencies:
+```bash
+cd demo
+npm install
+```
+
+2. Ensure Valkey/Redis is running:
+```bash
+# Single node on port 6379
+valkey-server
+
+# Or for cluster mode (optional)
+# Ports 7000-7005
+```
+
+## Running the Demo
+
+### Option 1: Full Demo with All Features
+```bash
+npm start
+```
+
+This launches:
+- 10 different queue types (orders, payments, inventory, etc.)
+- 10 specialized workers with different configurations
+- 12 demo scenarios showcasing various features
+- Real-time metrics display every 5 seconds
+
+### Option 2: Dashboard Server
+```bash
+npm run dashboard
+```
+
+Opens dashboard API server on http://localhost:3000
+
+Features:
+- REST API for queue management
+- Server-Sent Events for real-time updates
+- Queue pause/resume controls
+- Job retry/remove operations
+- Metrics and monitoring
+
+## Demo Scenarios
+
+1. **Simple Order Processing** - Basic job with progress tracking
+2. **Bulk Inventory Update** - 20 jobs added via addBulk
+3. **Priority Tasks** - Jobs with different priority levels
+4. **Scheduled Notifications** - Delayed job execution
+5. **Payment with Retries** - Automatic retry on failure
+6. **Deduplicated Analytics** - Prevents duplicate events
+7. **E-commerce Workflow** - Complex parent-child job tree
+8. **Sequential Pipeline** - Chain pattern for data processing
+9. **Parallel Broadcast** - Group pattern for notifications
+10. **Large Report Generation** - Compression for big payloads
+11. **Rate-Limited Recommendations** - Throttled processing
+12. **Timeout Handling** - Job timeout demonstration
+
+## Queue Types
+
+- **orders** - Order processing with progress tracking
+- **payments** - Payment processing with retry logic
+- **inventory** - Rate-limited inventory updates
+- **shipping** - Shipping label generation
+- **notifications** - Multi-channel notifications (email/SMS/push)
+- **analytics** - Event aggregation and metrics
+- **recommendations** - ML recommendation generation
+- **reports** - Long-running report generation
+- **priority-tasks** - Priority-based task execution
+- **dead-letter** - Failed job investigation
+
+## API Endpoints (Dashboard Server)
+
+- `GET /api/queues` - List all queues with counts
+- `GET /api/queues/:name` - Queue details and jobs
+- `GET /api/queues/:name/jobs/:id` - Specific job details
+- `POST /api/queues/:name/jobs` - Add new job
+- `POST /api/queues/:name/pause` - Pause queue
+- `POST /api/queues/:name/resume` - Resume queue
+- `POST /api/queues/:name/jobs/:id/retry` - Retry failed job
+- `DELETE /api/queues/:name/jobs/:id` - Remove job
+- `POST /api/queues/:name/drain` - Drain queue
+- `POST /api/queues/:name/obliterate` - Obliterate queue
+- `GET /api/events` - SSE stream for real-time updates
+
+## Monitoring
+
+The demo displays real-time metrics including:
+- Queue counts (waiting, active, delayed, completed, failed)
+- Job progress updates
+- Success/failure events
+- Stalled job detection
+- Worker status
+
+## Architecture
+
+```
+┌─────────────┐     ┌─────────────┐     ┌─────────────┐
+│   Producer  │────▶│    Queue    │────▶│   Worker    │
+│  (Demo App) │     │   (Valkey)  │     │ (Processor) │
+└─────────────┘     └─────────────┘     └─────────────┘
+                           │
+                           ▼
+                    ┌─────────────┐
+                    │  Dashboard  │
+                    │   (API/UI)  │
+                    └─────────────┘
+```
+
+## Customization
+
+Edit `index.ts` to:
+- Add new queue types
+- Modify worker configurations
+- Create custom scenarios
+- Adjust timing and concurrency
+- Change retry strategies
+
+## Troubleshooting
+
+- Ensure Valkey/Redis is running on localhost:6379
+- Check Node.js version is 20+
+- Verify all dependencies installed
+- Monitor console for error messages
+- Use dashboard API to inspect job details
+
+## Performance Tips
+
+- Increase worker concurrency for higher throughput
+- Use bulk operations for batch processing
+- Enable compression for large payloads
+- Configure appropriate retention policies
+- Use priority queues for critical tasks