queen-mq 0.1.1 → 0.1.3

package/README.md

@@ -9,9 +9,11 @@
 
 [Quick Start](#-quick-start) • [Client Examples](#-client-examples) • [Server Setup](#-server-setup) • [Core Concepts](#-core-concepts) • [API Reference](#-http-api-reference) • [Dashboard](#-dashboard)
 
-</div>
+<p align="center">
+  <img src="assets/queen-logo.svg" alt="Queen Logo" width="120" />
+</p>
 
-![Queen Dashboard](assets/dashboard.png)
+</div>
 
 ---
 
@@ -22,12 +24,14 @@
 ### Why Queen?
 
 **🚀 Developer-First API**
-- **4 methods, infinite patterns**: `queue()`, `push()`, `take()`, `ack()`
+- **4 core methods, infinite patterns**: `queue()`, `push()`, `take()`, `ack()`
 - **Async iteration**: Process messages with familiar `for await` syntax
+- **Batch processing**: Use `takeBatch()` for 250k+ msg/sec throughput on millions of messages
 - **Smart addressing**: `orders/urgent@workers` - queue, partition, and consumer group in one
 
 **⚡ Production-Ready Performance**
-- **10,000+ msg/sec** throughput with sub-10ms latency
+- **100,000+ msg/sec** throughput with cursor-based consumption
+- **Constant-time batch operations** - O(batch_size) regardless of queue depth
 - **Long polling** for event-driven, real-time message delivery
 - **Partition locking** prevents duplicate processing across consumers
 - **Connection pooling** and optimized batch operations
@@ -49,6 +53,7 @@
 - **Rich Analytics**: Throughput, lag, queue depth metrics
 - **Message Browser**: Search, inspect, and retry messages
 - **System Health**: Database, memory, and performance metrics
+- **Cursor Tracking**: Monitor consumption progress per consumer group
 
 ### Use Cases
 
@@ -67,6 +72,7 @@
 - [Client Examples](#-client-examples)
 - [Server Setup](#-server-setup)
 - [Core Concepts](#-core-concepts)
+- [Cursor-Based Consumption Strategy](#-cursor-based-consumption-strategy)
 - [HTTP API Reference](#-http-api-reference)
 - [Dashboard](#-dashboard)
 - [Configuration](#-configuration)
@@ -128,6 +134,33 @@ npm install
 node init-db.js
 ```
 
+### Running the Server
+
+**Option 1: Standalone Server (Traditional)**
+
+```bash
+npm start
+```
+
+**Option 2: Programmatic Server (New in v0.1.2)**
+
+```javascript
+import { QueenServer } from 'queen-mq';
+
+// Start server with default config
+const server = await QueenServer();
+
+// Or with custom options
+const server = await QueenServer({
+  port: 3000,
+  host: '127.0.0.1'
+});
+
+console.log(`Server running at http://${server.host}:${server.port}`);
+```
+
+This allows you to embed Queen MQ directly in your application, run multiple instances, or easily start/stop servers in tests. See [PROGRAMMATIC_SERVER.md](./PROGRAMMATIC_SERVER.md) for more details.
+
 ### Set Environment (Optional)
 
 ```bash
@@ -252,6 +285,26 @@ for await (const message of client.take('orders/urgent')) {
   await processUrgentOrder(message.data);
   await client.ack(message);
 }
+
+// Use takeBatch to get arrays of messages (higher throughput)
+for await (const messages of client.takeBatch('orders', { 
+  batch: 1000,     // Fetch 1000 at a time
+  wait: true 
+})) {
+  // messages is an array of up to 1000 messages
+  console.log(`Processing batch of ${messages.length} messages`);
+  
+  try {
+    // Process entire batch
+    await processBatch(messages.map(m => m.data));
+    
+    // Acknowledge entire batch at once (efficient!)
+    await client.ack(messages);  // Pass array for batch ack
+  } catch (error) {
+    // Mark entire batch as failed
+    await client.ack(messages, false, { error: error.message });
+  }
+}
 ```
 
 #### 4. Acknowledge Messages
@@ -313,7 +366,7 @@ for await (const message of client.take('tasks', {
 #### Batch Processing
 
 ```javascript
-// Accumulate and process in batches
+// Method 1: Manual batching with take()
 const batch = [];
 for await (const message of client.take('analytics', { batch: 100 })) {
   batch.push(message);
@@ -328,6 +381,15 @@ for await (const message of client.take('analytics', { batch: 100 })) {
     batch.length = 0;
   }
 }
+
+// Method 2: Direct batch processing with takeBatch() (RECOMMENDED)
+for await (const messages of client.takeBatch('analytics', { batch: 1000 })) {
+  // messages is already an array!
+  await processBatch(messages.map(m => m.data));
+  
+  // Single batch acknowledgment (much faster!)
+  await client.ack(messages);
+}
 ```
 
 #### Parallel Processing with Partitions
@@ -881,6 +943,190 @@ await client.queue('time-sensitive', {
 
 ---
 
+## 🚀 Cursor-Based Consumption Strategy
+
+Queen uses a **cursor-based consumption model** for optimal performance at scale, providing O(batch_size) constant-time operations regardless of queue depth.
+
+### How It Works
+
+Traditional message queues scan through all messages to find pending ones, leading to performance degradation as messages accumulate. Queen's cursor-based approach maintains a position marker (cursor) for each consumer, allowing direct access to the next batch of messages.
+
+**Partition Cursors:**
+
+Each partition maintains a cursor position per consumer group:
+- `last_consumed_created_at`: Timestamp of last consumed message
+- `last_consumed_id`: UUID of last consumed message (tie-breaker for same timestamp)
+- `total_messages_consumed`: Running count of consumed messages
+
+The cursor always moves **forward** in time, ensuring strict FIFO ordering.
+
+### The takeBatch Method
+
+Queen provides two consumption methods:
+
+**1. `take()` - Individual message iterator:**
+```javascript
+// Processes messages one at a time
+for await (const message of client.take('orders', { batch: 1000 })) {
+  await processOrder(message.data);
+  await client.ack(message);
+}
+```
+
+**2. `takeBatch()` - Array iterator (HIGH PERFORMANCE):**
+```javascript
+// Yields arrays of messages - achieves 100k+ msg/s throughput
+for await (const messages of client.takeBatch('orders', { batch: 1000 })) {
+  // messages is an array of up to 1000 message objects
+  await processBatch(messages.map(m => m.data));
+  
+  // Batch acknowledge - single DB transaction for all messages
+  await client.ack(messages);
+}
+```
+
+**Under the hood**, both methods use cursor-based batch retrieval:
+
+```sql
+-- Cursor-based query (simplified)
+SELECT * FROM messages 
+WHERE partition_id = $1 
+  AND id > $2::uuid  -- Start after last cursor position
+ORDER BY created_at ASC, id ASC
+LIMIT $3  -- Batch size
+FOR UPDATE SKIP LOCKED
+```
+
+**Key characteristics:**
+1. **Constant-time**: Performance stays consistent whether you've consumed 0% or 99% of messages
+2. **FIFO guarantee**: Messages always returned in creation order
+3. **Lock-free scanning**: `SKIP LOCKED` prevents contention between consumers
+4. **Efficient**: No table scans - direct cursor-based access using UUIDv7 (time-ordered)
+
+**Performance tip:** Use `takeBatch()` with large batch sizes (1,000-10,000) for maximum throughput. The server fetches messages in batches regardless, but `takeBatch()` gives you the array directly, allowing bulk processing and batch acknowledgment in a single operation.
+
+### Batch Acknowledgment Semantics
+
+Queen handles batch acknowledgments intelligently:
+
+**Partial Success** (some messages succeed, some fail):
+```javascript
+// Batch: 10,000 messages
+// Success: 9,999 messages
+// Failed: 1 message
+
+// Behavior:
+// ✅ Cursor advances past all 10,000 messages
+// ✅ Failed message moved to Dead Letter Queue
+// ✅ Next take() starts from message 10,001
+// ✅ FIFO maintained, no redelivery of successful messages
+```
+
+**Total Batch Failure** (all messages fail):
+```javascript
+// Batch: 10,000 messages
+// Success: 0 messages
+// Failed: 10,000 messages
+
+// Behavior:
+// ❌ Cursor DOES NOT advance
+// ❌ Messages NOT moved to DLQ
+// ✅ Lease released
+// ✅ Next take() gets SAME batch (retry)
+// ✅ Allows recovery from transient failures
+```
+
+This design handles transient failures (network issues, service outages) gracefully while preventing poison messages from blocking the queue.
+
+### Performance Comparison
+
+| Operation | Traditional Approach | Cursor Approach | Improvement |
+|-----------|---------------------|-----------------|-------------|
+| Pop @ 0% consumed | O(partition_size) | O(batch_size) | Same |
+| Pop @ 50% consumed | O(partition_size) | O(batch_size) | **10-100x faster** |
+| Pop @ 99% consumed | O(partition_size) | O(batch_size) | **100-1000x faster** |
+
+**Real-world benchmark** (1M messages):
+```
+Traditional:
+  Early batches: 300ms per pop
+  Late batches:  3500ms per pop (10x degradation)
+  
+Cursor-based:
+  Early batches: 150ms per pop
+  Late batches:  200ms per pop (constant!)
+```
+
+### Dead Letter Queue
+
+Individual message failures are moved to the Dead Letter Queue for inspection and manual intervention:
+
+```javascript
+// Monitor DLQ
+const response = await fetch('http://localhost:6632/api/v1/analytics/dlq');
+const dlqMessages = await response.json();
+
+// Inspect failed messages
+for (const msg of dlqMessages) {
+  console.log(`Failed: ${msg.error_message}`);
+  
+  // After fixing issue, can re-push if needed
+  await client.push(msg.queue, fixedPayload);
+}
+```
+
+**DLQ Query:**
+```sql
+SELECT * FROM queen.dead_letter_queue
+WHERE consumer_group = 'my-group'
+ORDER BY failed_at DESC
+LIMIT 100;
+```
+
+### Batch Size Guidelines
+
+Choose batch sizes based on your workload:
+
+**Smaller batches (100-1,000):**
+- ✅ Faster individual batch processing
+- ✅ Less impact if entire batch fails
+- ✅ Lower memory footprint
+- ❌ More network round-trips
+
+**Larger batches (5,000-10,000):**
+- ✅ Higher throughput (100,000+ msg/sec achievable)
+- ✅ Fewer network round-trips
+- ✅ Better database efficiency
+- ❌ More messages retry if entire batch fails
+- ❌ Higher memory usage
+
+**Recommendation:** Start with 1,000-2,000 for balanced performance. Increase to 5,000-10,000 for maximum throughput with reliable processing.
+
+### Monitoring Cursor Progress
+
+Track consumption progress via SQL:
+
+```sql
+-- View cursor positions
+SELECT 
+  p.name as partition,
+  pc.consumer_group,
+  pc.total_messages_consumed,
+  pc.total_batches_consumed,
+  pc.last_consumed_at,
+  EXTRACT(EPOCH FROM (NOW() - pc.last_consumed_at)) as seconds_since_last_consume
+FROM queen.partition_cursors pc
+JOIN queen.partitions p ON p.id = pc.partition_id
+ORDER BY pc.last_consumed_at DESC;
+
+-- Monitor DLQ
+SELECT COUNT(*) as failed_count, consumer_group
+FROM queen.dead_letter_queue
+GROUP BY consumer_group;
+```
+
+---
+
 ## 🔌 HTTP API Reference
 
 Base URL: `http://localhost:6632/api/v1`
@@ -1067,6 +1313,12 @@ POST /api/v1/messages/{transactionId}/dlq
 DELETE /api/v1/queues/{queue}/clear
 ```
 
+**Delete queue:**
+```
+DELETE /api/v1/resources/queues/{queue}
+```
+_Note: Deletes the queue and all its partitions, messages, and related data._
+
 ### System Health
 
 **Health check:**
@@ -1108,10 +1360,12 @@ See [API.md](API.md) for complete API documentation.
 
 Queen includes a comprehensive web dashboard for monitoring and management.
 
+[Dashboard](/assets/dashboard-01.png)
+
 ### Access
 
 1. Start the server: `npm start`
-2. Open browser: `http://localhost:6632`
+2. Open browser: `http://localhost:4000`
 3. WebSocket connection provides real-time updates
 
 ### Features
@@ -1496,7 +1750,7 @@ Promise.all([
 await emitEvents();
 ```
 
-### Example 4: Batch Processing
+### Example 4: Batch Processing (High Throughput)
 
 ```javascript
 import { Queen } from 'queen-mq';
@@ -1515,7 +1769,7 @@ await client.queue('data-processing', {
 // Producer: Send data
 async function sendData() {
   const records = [];
-  for (let i = 0; i < 1000; i++) {
+  for (let i = 0; i < 100000; i++) {
     records.push({ id: i, value: Math.random() });
   }
   
@@ -1523,45 +1777,34 @@ async function sendData() {
   await client.push('data-processing/analytics', records);
 }
 
-// Consumer: Batch processor
+// Consumer: HIGH PERFORMANCE batch processor using takeBatch()
 async function batchProcessor() {
-  const BATCH_SIZE = 100;
-  const batch = [];
+  const BATCH_SIZE = 5000;  // Large batches for 100k+ msg/s throughput
   
-  for await (const message of client.take('data-processing/analytics', {
+  // takeBatch() yields arrays directly - no manual batching needed!
+  for await (const messages of client.takeBatch('data-processing/analytics', {
     batch: BATCH_SIZE,
     wait: true,
     timeout: 30000
   })) {
-    batch.push(message);
-    
-    // Process when batch is full
-    if (batch.length >= BATCH_SIZE) {
-      try {
-        console.log(`Processing batch of ${batch.length} records`);
-        
-        // Extract data
-        const records = batch.map(m => m.data);
-        
-        // Bulk process
-        await bulkInsertToDatabase(records);
-        
-        // Acknowledge all
-        for (const msg of batch) {
-          await client.ack(msg);
-        }
-        
-        console.log(`✓ Batch complete`);
-        batch.length = 0;
-      } catch (error) {
-        console.error('Batch processing failed:', error);
-        
-        // Mark all as failed
-        for (const msg of batch) {
-          await client.ack(msg, false);
-        }
-        batch.length = 0;
-      }
+    try {
+      console.log(`Processing batch of ${messages.length} records`);
+      
+      // Extract data
+      const records = messages.map(m => m.data);
+      
+      // Bulk process (single DB operation)
+      await bulkInsertToDatabase(records);
+      
+      // Batch acknowledge (single DB transaction!)
+      await client.ack(messages);
+      
+      console.log(`✓ Batch complete in single transaction`);
+    } catch (error) {
+      console.error('Batch processing failed:', error);
+      
+      // Mark entire batch as failed (single transaction)
+      await client.ack(messages, false, { error: error.message });
     }
   }
 }
@@ -1569,6 +1812,12 @@ async function batchProcessor() {
 // Run
 await sendData();
 await batchProcessor();
+
+// Performance characteristics:
+// - Batch size 5000: ~100,000 messages/second
+// - Single DB transaction per batch (fetch + ack)
+// - Constant memory usage
+// - No performance degradation as queue grows
 ```
 
 ### Example 5: Scheduled Jobs
@@ -1886,33 +2135,34 @@ Apache License 2.0 - see [LICENSE.md](LICENSE.md) for details.
 
 ## 📈 Performance
 
-**Benchmarks** (PostgreSQL 16, Node.js 22):
-- **Throughput**: 10,000+ messages/second
+**Benchmarks** (PostgreSQL 16, Node.js 22, cursor-based consumption):
+- **Throughput**: 100,000+ messages/second with batch operations
 - **Latency**: < 10ms for immediate pop operations
+- **Constant-time consumption**: O(batch_size) regardless of queue depth
 - **Concurrent Connections**: 1,000+ long polling connections
 - **Database**: Optimized with proper indexing and connection pooling
 
-**Optimization Features:**
+**Cursor-Based Architecture Benefits:**
+- **No performance degradation**: Consistent speed whether queue has 1K or 1B messages
+- **Predictable latency**: 150-200ms per batch throughout entire queue lifecycle
+- **Efficient batch processing**: Direct cursor access eliminates table scans
+- **Scalable to billions**: UUIDv7-based cursor positioning
+
+**Additional Optimization Features:**
 - Connection pooling with configurable size
 - Resource caching for queue/partition lookups
-- Batch operations for bulk inserts/updates
+- Batch operations for bulk inserts/updates (up to 10,000 messages per batch)
 - Optimized SQL queries with proper indexes
 - Event-driven architecture for minimal polling overhead
 - Long polling for real-time message delivery
+- SKIP LOCKED for lock-free concurrent consumption
 
 ---
 
 ## 🎯 Roadmap
 
-- [ ] **Horizontal Scaling**: Better support for multiple server instances
 - [ ] **Message Scheduling**: Cron-like scheduling for recurring jobs
-- [ ] **Priority Lanes**: Dynamic priority adjustment based on load
-- [ ] **Metrics Export**: Prometheus/Grafana integration
-- [ ] **Admin API**: REST API for queue management
 - [ ] **Client Libraries**: Python, Go, Java clients
-- [ ] **Message Tracing**: Distributed tracing integration
-- [ ] **Queue Templates**: Pre-configured queue patterns
-- [ ] **GraphQL API**: Alternative to REST API
 - [ ] **Kubernetes Operator**: Native K8s support
 
 ---
@@ -1921,6 +2171,6 @@ Apache License 2.0 - see [LICENSE.md](LICENSE.md) for details.
 
 **Queen Message Queue System** - Built for performance, reliability, and developer happiness 🚀
 
-Made with ❤️ by [Smartpricing](https://github.com/smartpricing)
+Made with ❤️ by [Smartness](https://github.com/smartpricing)
 
 </div>