glide-mq 0.4.0 → 0.4.2

package/README.md

@@ -46,7 +46,7 @@ No-op processor, Valkey 8.0, single node.
 npm install glide-mq
 ```
 
-Requires Node.js 20+ and a running Valkey (7.2+) or Redis (6.2+) instance.
+Requires Node.js 20+ and a running Valkey (7.0+) or Redis (7.0+) instance for FUNCTION support.
 
 ## Quick Start
 
@@ -100,7 +100,7 @@ worker.on('failed', (job, err) => console.log(`Job ${job.id} failed: ${err.messa
 - **QueueEvents** - stream-based event subscription (added, completed, failed, stalled, etc.)
 - **Job schedulers** - cron patterns and fixed intervals for repeatable jobs
 - **Metrics** - getJobCounts, getMetrics
-- **OpenTelemetry** - optional spans for Queue.add, Worker.process, FlowProducer.add
+- **OpenTelemetry** - automatic tracing spans for Queue.add and FlowProducer.add operations (optional peer dependency)
 
 ### Cloud-Native (GLIDE-exclusive)
 - **AZ-Affinity routing** - route reads to same-AZ replicas for lower latency and reduced cross-AZ costs
@@ -195,17 +195,23 @@ await flow.add({
   ],
 });
 
-// Chain: A -> B -> C (sequential)
-await chain(connection, 'tasks', [
+// Chain: A -> B -> C (sequential, each step receives previous result)
+await chain('tasks', [
   { name: 'step1', data: {} },
   { name: 'step2', data: {} },
-]);
+], connection);
 
-// Group: A + B + C (parallel)
-await group(connection, 'tasks', [
+// Group: A + B + C (parallel, parent completes when all children done)
+await group('tasks', [
   { name: 'task1', data: {} },
   { name: 'task2', data: {} },
-]);
+], connection);
+
+// Chord: run group in parallel, then callback with all results
+await chord('tasks', [
+  { name: 'task1', data: {} },
+  { name: 'task2', data: {} },
+], { name: 'aggregate', data: {} }, connection);
 ```
 
 ### Events
@@ -219,6 +225,34 @@ events.on('failed', ({ jobId, failedReason }) => {});
 events.on('stalled', ({ jobId }) => {});
 ```
 
+### OpenTelemetry
+
+```typescript
+// Install optional peer dependency
+// npm install @opentelemetry/api
+
+// Automatic tracing for Queue.add and FlowProducer.add
+const queue = new Queue('tasks', { connection });
+await queue.add('job', data);  // Creates span: glide-mq.queue.add
+
+// Custom tracer (optional)
+import { setTracer } from 'glide-mq';
+setTracer(customTracerInstance);
+```
+
+### Graceful Shutdown
+
+```typescript
+import { gracefulShutdown } from 'glide-mq';
+
+const queue = new Queue('tasks', { connection });
+const worker = new Worker('tasks', processor, { connection });
+const events = new QueueEvents('tasks', { connection });
+
+// Registers SIGTERM/SIGINT handlers and resolves when all components are closed
+await gracefulShutdown([queue, worker, events]);
+```
+
 ## Cluster Mode
 
 ```typescript
@@ -233,6 +267,107 @@ const connection = {
 const queue = new Queue('tasks', { connection });
 ```
 
+## Testing Mode
+
+glide-mq ships a built-in in-memory backend so you can unit-test your job processors **without a running Valkey instance**.
+
+```typescript
+import { TestQueue, TestWorker } from 'glide-mq/testing';
+
+const queue = new TestQueue('tasks');
+
+const worker = new TestWorker(queue, async (job) => {
+  return { processed: job.data };
+});
+
+worker.on('completed', (job, result) => {
+  console.log(`Job ${job.id} done:`, result);
+});
+
+worker.on('failed', (job, err) => {
+  console.error(`Job ${job.id} failed:`, err.message);
+});
+
+await queue.add('send-email', { to: 'user@example.com' });
+
+// Inspect state without touching Valkey
+const counts = await queue.getJobCounts();
+// { waiting: 0, active: 0, delayed: 0, completed: 1, failed: 0 }
+
+// Search jobs by name or data fields
+const jobs = await queue.searchJobs({ name: 'send-email', state: 'completed' });
+
+await worker.close();
+await queue.close();
+```
+
+`TestQueue` and `TestWorker` mirror the real `Queue` / `Worker` public API (add, addBulk, getJob, getJobs, getJobCounts, pause, resume, events, retries, concurrency), making it straightforward to swap implementations between test and production code.
+
+## Dashboard
+
+glide-mq exposes a REST + Server-Sent Events API that can be consumed by the [`@glidemq/dashboard`](https://github.com/avifenesh/glidemq-dashboard) UI package.
+
+### Quick start with the built-in demo server
+
+```bash
+cd demo
+npm install
+npm run dashboard   # starts http://localhost:3000
+```
+
+### REST endpoints
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/queues` | List all queues with counts and metrics |
+| `GET` | `/api/queues/:name` | Queue details + recent jobs |
+| `GET` | `/api/queues/:name/jobs/:id` | Single job details, state, logs |
+| `POST` | `/api/queues/:name/jobs` | Add a new job |
+| `POST` | `/api/queues/:name/pause` | Pause a queue |
+| `POST` | `/api/queues/:name/resume` | Resume a queue |
+| `POST` | `/api/queues/:name/jobs/:id/retry` | Retry a failed job |
+| `DELETE` | `/api/queues/:name/jobs/:id` | Remove a job |
+| `POST` | `/api/queues/:name/drain` | Drain all waiting jobs |
+| `POST` | `/api/queues/:name/obliterate` | Obliterate queue and all data |
+| `GET` | `/api/events` | SSE stream for real-time job events |
+
+### Real-time events via SSE
+
+```javascript
+const es = new EventSource('http://localhost:3000/api/events');
+es.onmessage = ({ data }) => {
+  const { queue, event, jobId } = JSON.parse(data);
+  // event: 'added' | 'completed' | 'failed' | 'progress' | 'stalled' | 'heartbeat'
+  console.log(`[${queue}] ${event} – job ${jobId}`);
+};
+```
+
+### Embedding the dashboard server in your own Express app
+
+```typescript
+import express from 'express';
+import { Queue, QueueEvents } from 'glide-mq';
+
+const app = express();
+app.use(express.json());
+
+const queues: Record<string, Queue> = {
+  orders: new Queue('orders', { connection }),
+  payments: new Queue('payments', { connection }),
+};
+
+app.get('/api/queues', async (_req, res) => {
+  const data = await Promise.all(
+    Object.entries(queues).map(async ([name, q]) => ({
+      name,
+      counts: await q.getJobCounts(),
+      isPaused: await q.isPaused(),
+    })),
+  );
+  res.json(data);
+});
+```
+
 ## License
 
 Apache-2.0

package/demo/README.md

@@ -0,0 +1,169 @@
+# glide-mq Demo Application
+
+Comprehensive demonstration of all glide-mq features simulating a complete e-commerce platform.
+
+## Features Demonstrated
+
+### Core Queue Operations
+- Job processing with progress tracking
+- Bulk operations with Batch API
+- Priority queues with FIFO ordering
+- Delayed/scheduled jobs
+- Job retries with exponential backoff
+- Dead letter queue handling
+
+### Advanced Features
+- Deduplication (simple, throttle, debounce)
+- Rate limiting (sliding window)
+- Global concurrency limits
+- Job retention policies
+- Transparent gzip compression
+- Timeout handling
+
+### Workflows
+- FlowProducer for parent-child job trees
+- Chain pattern for sequential pipelines
+- Group pattern for parallel execution
+- Complex nested workflows
+
+### Observability
+- Real-time event streaming
+- Progress tracking
+- Metrics and job counts
+- Job logs and state tracking
+- Dashboard API integration
+
+## Setup
+
+1. Install dependencies:
+```bash
+cd demo
+npm install
+```
+
+2. Ensure Valkey/Redis is running:
+```bash
+# Single node on port 6379
+valkey-server
+
+# Or for cluster mode (optional)
+# Ports 7000-7005
+```
+
+## Running the Demo
+
+### Option 1: Full Demo with All Features
+```bash
+npm start
+```
+
+This launches:
+- 10 different queue types (orders, payments, inventory, etc.)
+- 10 specialized workers with different configurations
+- 12 demo scenarios showcasing various features
+- Real-time metrics display every 5 seconds
+
+### Option 2: Dashboard Server
+```bash
+npm run dashboard
+```
+
+Opens dashboard API server on http://localhost:3000
+
+Features:
+- REST API for queue management
+- Server-Sent Events for real-time updates
+- Queue pause/resume controls
+- Job retry/remove operations
+- Metrics and monitoring
+
+## Demo Scenarios
+
+1. **Simple Order Processing** - Basic job with progress tracking
+2. **Bulk Inventory Update** - 20 jobs added via addBulk
+3. **Priority Tasks** - Jobs with different priority levels
+4. **Scheduled Notifications** - Delayed job execution
+5. **Payment with Retries** - Automatic retry on failure
+6. **Deduplicated Analytics** - Prevents duplicate events
+7. **E-commerce Workflow** - Complex parent-child job tree
+8. **Sequential Pipeline** - Chain pattern for data processing
+9. **Parallel Broadcast** - Group pattern for notifications
+10. **Large Report Generation** - Compression for big payloads
+11. **Rate-Limited Recommendations** - Throttled processing
+12. **Timeout Handling** - Job timeout demonstration
+
+## Queue Types
+
+- **orders** - Order processing with progress tracking
+- **payments** - Payment processing with retry logic
+- **inventory** - Rate-limited inventory updates
+- **shipping** - Shipping label generation
+- **notifications** - Multi-channel notifications (email/SMS/push)
+- **analytics** - Event aggregation and metrics
+- **recommendations** - ML recommendation generation
+- **reports** - Long-running report generation
+- **priority-tasks** - Priority-based task execution
+- **dead-letter** - Failed job investigation
+
+## API Endpoints (Dashboard Server)
+
+- `GET /api/queues` - List all queues with counts
+- `GET /api/queues/:name` - Queue details and jobs
+- `GET /api/queues/:name/jobs/:id` - Specific job details
+- `POST /api/queues/:name/jobs` - Add new job
+- `POST /api/queues/:name/pause` - Pause queue
+- `POST /api/queues/:name/resume` - Resume queue
+- `POST /api/queues/:name/jobs/:id/retry` - Retry failed job
+- `DELETE /api/queues/:name/jobs/:id` - Remove job
+- `POST /api/queues/:name/drain` - Drain queue
+- `POST /api/queues/:name/obliterate` - Obliterate queue
+- `GET /api/events` - SSE stream for real-time updates
+
+## Monitoring
+
+The demo displays real-time metrics including:
+- Queue counts (waiting, active, delayed, completed, failed)
+- Job progress updates
+- Success/failure events
+- Stalled job detection
+- Worker status
+
+## Architecture
+
+```
+┌─────────────┐     ┌─────────────┐     ┌─────────────┐
+│   Producer  │────▶│    Queue    │────▶│   Worker    │
+│  (Demo App) │     │   (Valkey)  │     │ (Processor) │
+└─────────────┘     └─────────────┘     └─────────────┘
+                           │
+                           ▼
+                    ┌─────────────┐
+                    │  Dashboard  │
+                    │   (API/UI)  │
+                    └─────────────┘
+```
+
+## Customization
+
+Edit `index.ts` to:
+- Add new queue types
+- Modify worker configurations
+- Create custom scenarios
+- Adjust timing and concurrency
+- Change retry strategies
+
+## Troubleshooting
+
+- Ensure Valkey/Redis is running on localhost:6379
+- Check Node.js version is 20+
+- Verify all dependencies installed
+- Monitor console for error messages
+- Use dashboard API to inspect job details
+
+## Performance Tips
+
+- Increase worker concurrency for higher throughput
+- Use bulk operations for batch processing
+- Enable compression for large payloads
+- Configure appropriate retention policies
+- Use priority queues for critical tasks