queen-mq 0.1.0 → 0.1.1

package/README.md

@@ -1,40 +1,117 @@
-# Queen - High-Performance Message Queue System
+# Queen - PostgreSQL-backed Message Queue System
 
-A modern, high-performance message queue system built with PostgreSQL and uWebSockets.js, featuring priority-based processing, advanced scheduling, and real-time monitoring.
+<div align="center">
+
+**A modern, high-performance message queue system built on PostgreSQL**
+
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE.md)
+[![Node](https://img.shields.io/badge/node-%3E%3D22.0.0-brightgreen.svg)](https://nodejs.org/)
+
+[Quick Start](#-quick-start) • [Client Examples](#-client-examples) • [Server Setup](#-server-setup) • [Core Concepts](#-core-concepts) • [API Reference](#-http-api-reference) • [Dashboard](#-dashboard)
+
+</div>
 
 ![Queen Dashboard](assets/dashboard.png)
 
-## 🚀 Features
+---
+
+## 🎯 Introduction
+
+**Queen** is a production-ready message queue system that combines the reliability of PostgreSQL with the performance of modern async architectures. Built with uWebSockets.js for blazing-fast HTTP handling and designed for real-world workloads.
+
+### Why Queen?
 
-- **🏗️ Flexible Architecture**: Queues → Partitions → Messages with optional namespace/task grouping
-- **⚡ Priority Processing**: Queue and partition-level priorities with FIFO within partitions
-- **🕒 Advanced Scheduling**: Delayed processing and window buffering
-- **🔄 Reliable Processing**: Lease-based processing with automatic retry and dead letter queues
-- **📊 Real-time Monitoring**: WebSocket dashboard with live metrics and analytics
-- **🔒 Message Guarantees**: ACID transactions, idempotency, and no message loss
-- **🚄 High Performance**: 10,000+ messages/second with sub-10ms latency
-- **🌐 Long Polling**: Event-driven optimization for real-time message consumption
-- **📦 Batch Operations**: Efficient bulk message processing with individual and batch consumer modes
-- **🛠️ Client SDK**: Full-featured JavaScript client with retry logic and helpers
+**🚀 Developer-First API**
+- **4 methods, infinite patterns**: `queue()`, `push()`, `take()`, `ack()`
+- **Async iteration**: Process messages with familiar `for await` syntax
+- **Smart addressing**: `orders/urgent@workers` - queue, partition, and consumer group in one
+
+**⚡ Production-Ready Performance**
+- **10,000+ msg/sec** throughput with sub-10ms latency
+- **Long polling** for event-driven, real-time message delivery
+- **Partition locking** prevents duplicate processing across consumers
+- **Connection pooling** and optimized batch operations
+
+**🏗️ Flexible Architecture**
+- **Queue Mode**: Competitive consumption (traditional work queue)
+- **Bus Mode**: Pub/sub with consumer groups (event streaming)
+- **Mixed Mode**: Combine both patterns in the same system
+- **Partitions**: FIFO ordering with parallel processing
+
+**🔒 Enterprise Features**
+- **AES-256-GCM Encryption**: Protect sensitive data at rest
+- **Message Retention**: Automatic cleanup policies
+- **Message Eviction**: SLA enforcement for time-sensitive tasks
+- **Dead Letter Queue**: Handle failed messages gracefully
+
+**📊 Built-in Observability**
+- **Real-time Dashboard**: WebSocket-powered monitoring
+- **Rich Analytics**: Throughput, lag, queue depth metrics
+- **Message Browser**: Search, inspect, and retry messages
+- **System Health**: Database, memory, and performance metrics
+
+### Use Cases
+
+- **Task Queues**: Background jobs, email sending, data processing
+- **Event Streaming**: Audit logs, analytics, multi-service event handling
+- **Workflow Orchestration**: Multi-stage pipelines, saga patterns
+- **Rate Limiting**: Throttle and batch time-sensitive operations
+- **Priority Processing**: Handle urgent tasks before routine ones
+
+---
 
 ## 📋 Table of Contents
 
-- [Quick Start](#quick-start)
-- [Architecture](#architecture)
-- [Core Concepts](#core-concepts)
-- [API Reference](#api-reference)
-- [Client SDK](#client-sdk)
-- [Dashboard](#dashboard)
-- [Examples](#examples)
-- [Performance](#performance)
-- [Configuration](#configuration)
+- [First Queue](#-first-queue)
+- [Quick Start](#-quick-start)
+- [Client Examples](#-client-examples)
+- [Server Setup](#-server-setup)
+- [Core Concepts](#-core-concepts)
+- [HTTP API Reference](#-http-api-reference)
+- [Dashboard](#-dashboard)
+- [Configuration](#-configuration)
+- [Full Examples](#-full-examples)
+- [Testing](#-testing)
+- [Contributing](#-contributing)
+
+---
+
+## First Queue
+
+```javascript
+import { Queen } from 'queen-mq';
+
+const client = new Queen({
+  baseUrls: ['http://localhost:6632']
+});
+
+// Configure queue
+await client.queue('tasks', {
+  leaseTime: 300,
+  retryLimit: 3
+});
+
+// Push a message
+await client.push('tasks', { 
+  action: 'send-email',
+  to: 'user@example.com' 
+});
+
+// Process messages
+for await (const message of client.take('tasks')) {
+  console.log('Processing:', message.data);
+  await client.ack(message);
+}
+```
+
+That's it! You now have a working message queue system.
 
 ## 🏃 Quick Start
 
 ### Prerequisites
 
-- Node.js 22+
-- PostgreSQL 12+
+- **Node.js 22+**
+- **PostgreSQL 12+**
 
 ### Installation
 
@@ -47,556 +124,788 @@ cd queen
 nvm use 22
 npm install
 
-# Set up environment (optional)
+# Initialize database schema
+node init-db.js
+```
+
+### Set Environment (Optional)
+
+```bash
+# Database connection
 export PG_USER=postgres
 export PG_HOST=localhost
 export PG_DB=postgres
 export PG_PASSWORD=postgres
 export PG_PORT=5432
+
+# Enable encryption (optional)
+export QUEEN_ENCRYPTION_KEY=$(openssl rand -hex 32)
 ```
 
-### Database Setup
+### Start the Server
 
 ```bash
-# Initialize the database schema
-node init-db.js
+npm start
+# Server starts on http://localhost:6632
 ```
 
-### Start the Server
+---
 
-```bash
-# Optional: Enable encryption
-export QUEEN_ENCRYPTION_KEY=$(openssl rand -hex 32)
+## 💻 Client Examples
 
-# Start the Queen server
-npm start
-# Or use the startup script
-./start.sh
+The Queen client provides a minimalist API with just 4 methods that compose into any messaging pattern you need.
 
-# Server starts on http://localhost:6632
+### Installation
+
+```bash
+npm install queen-mq
 ```
 
 ### Basic Usage
 
+#### 1. Configure a Queue
+
 ```javascript
-import { createQueenClient } from '@dev.smartpricing/queen'
+import { Queen } from 'queen-mq';
+
+const client = new Queen({
+  baseUrls: ['http://localhost:6632'],
+  timeout: 30000,
+  retryAttempts: 3
+});
 
-const client = createQueenClient({
-  baseUrl: 'http://localhost:6632'
+// Configure with options
+await client.queue('orders', {
+  priority: 10,          // Higher priority queues processed first
+  leaseTime: 600,        // 10 minutes to process each message
+  retryLimit: 3,         // Retry up to 3 times
+  delayedProcessing: 0,  // No delay (immediate processing)
 });
 
-// Push a message
-await client.push({
-  items: [{
-    queue: 'email-queue',
-    partition: 'urgent',
-    payload: { to: 'user@example.com', subject: 'Hello!' }
-  }]
+// Configure with namespace and task for grouping
+await client.queue('order-processing', {
+  priority: 10
+}, {
+  namespace: 'ecommerce',
+  task: 'checkout'
 });
+```
+
+#### 2. Push Messages
 
-// Pop and process messages
-const result = await client.pop({
-  queue: 'email-queue',
-  batch: 10,
-  wait: true
+```javascript
+// Single message
+await client.push('orders', { 
+  orderId: 12345, 
+  amount: 99.99 
 });
 
-for (const message of result.messages) {
-  console.log('Processing:', message.data);
-  await client.ack(message.transactionId, 'completed');
+// To a specific partition
+await client.push('orders/urgent', { 
+  orderId: 12346, 
+  amount: 999.99,
+  priority: 'high'
+});
+
+// Batch messages
+await client.push('orders', [
+  { orderId: 12347, amount: 49.99 },
+  { orderId: 12348, amount: 79.99 },
+  { orderId: 12349, amount: 29.99 }
+]);
+
+// With message properties
+await client.push('orders', {
+  orderId: 12350,
+  amount: 199.99
+}, {
+  transactionId: 'txn-12350',  // For idempotency
+  traceId: '550e8400-e29b-41d4-a716-446655440000'  // Valid UUID for tracing
+});
+```
+
+#### 3. Take Messages (Async Iterator)
+
+```javascript
+// Process continuously with long polling
+for await (const message of client.take('orders', { 
+  wait: true,      // Enable long polling
+  timeout: 30000,  // 30 second timeout
+  batch: 10        // Fetch up to 10 at once
+})) {
+  try {
+    await processOrder(message.data);
+    await client.ack(message);  // Success
+  } catch (error) {
+    await client.ack(message, false, { error: error.message });  // Failure
+  }
+}
+
+// Process limited messages
+for await (const message of client.take('orders', { limit: 100 })) {
+  await processOrder(message.data);
+  await client.ack(message);
+}
+
+// Take from specific partition
+for await (const message of client.take('orders/urgent')) {
+  await processUrgentOrder(message.data);
+  await client.ack(message);
 }
 ```
 
-## 🏗️ Architecture
+#### 4. Acknowledge Messages
+
+```javascript
+// Acknowledge success
+await client.ack(message);
+// or
+await client.ack(message, true);
+
+// Acknowledge failure (will retry based on retryLimit)
+await client.ack(message, false);
 
-### System Overview
+// Acknowledge with error context
+await client.ack(message, false, { 
+  error: 'Payment gateway timeout' 
+});
+
+// Acknowledge using transaction ID
+await client.ack('4dfb0478-655b-4c91-bcd9-b7acacf0400f', true);
 
+// Request explicit retry
+await client.ack(message, 'retry');
 ```
-┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
-│   Client SDK    │───▶│   Queen Server   │───▶│   PostgreSQL    │
-└─────────────────┘    └──────────────────┘    └─────────────────┘
-                              │
-                              ▼
-                       ┌──────────────────┐
-                       │   Dashboard UI   │
-                       └──────────────────┘
+
+### Address Notation
+
+Queen uses a simple addressing scheme that encodes queue, partition, and consumer group:
+
+```javascript
+// Basic addresses
+'orders'                    // Queue (Default partition)
+'orders/urgent'             // Queue with specific partition
+'orders@workers'            // Queue with consumer group (bus mode)
+'orders/urgent@workers'     // Full address: queue + partition + group
+
+// Namespace/task filtering (cross-queue consumption)
+'namespace:ecommerce'                      // All queues in namespace
+'task:checkout'                            // All queues with task
+'namespace:ecommerce/task:checkout'        // Combined filter
+'namespace:ecommerce/task:checkout@audit'  // With consumer group
 ```
 
-### Data Model
+### Consumer Patterns
+
+#### Continuous Processing (Long Polling)
 
+```javascript
+// Efficient real-time processing
+for await (const message of client.take('tasks', {
+  wait: true,      // Long polling - waits for messages
+  timeout: 30000   // Server timeout
+})) {
+  await processTask(message.data);
+  await client.ack(message);
+}
 ```
-Queues (with optional namespace/task grouping)
-  └── Partitions (FIFO ordering, priority-based selection)
-       └── Messages (lease-based processing)
+
+#### Batch Processing
+
+```javascript
+// Accumulate and process in batches
+const batch = [];
+for await (const message of client.take('analytics', { batch: 100 })) {
+  batch.push(message);
+  
+  if (batch.length >= 100) {
+    await processBatch(batch.map(m => m.data));
+    
+    // Acknowledge all
+    for (const msg of batch) {
+      await client.ack(msg);
+    }
+    batch.length = 0;
+  }
+}
 ```
 
-**Database Schema:**
-- `queen.queues` - Top-level message containers with optional grouping
-- `queen.partitions` - Subdivisions within queues where FIFO is maintained
-- `queen.messages` - Individual messages with processing state
+#### Parallel Processing with Partitions
 
-### Key Components
+```javascript
+// Create workers for parallel processing
+const partitions = ['worker-1', 'worker-2', 'worker-3', 'worker-4'];
 
-- **uWebSockets.js Server**: High-performance HTTP/WebSocket server
-- **Queue Manager**: Core message processing logic with optimizations
-- **Resource Cache**: In-memory caching for queue/partition lookups
-- **Event Manager**: Real-time notifications for long polling
-- **WebSocket Server**: Live dashboard updates and monitoring
+// Distribute messages across partitions
+for (let i = 0; i < messages.length; i++) {
+  const partition = partitions[i % partitions.length];
+  await client.push(`tasks/${partition}`, messages[i]);
+}
 
-## 💡 Core Concepts
+// Each worker processes its own partition (in parallel)
+async function worker(partition) {
+  for await (const msg of client.take(`tasks/${partition}`)) {
+    await processTask(msg.data);
+    await client.ack(msg);
+  }
+}
 
-### Queues and Partitions
+// Start all workers
+await Promise.all(partitions.map(p => worker(p)));
+```
 
-**Queues** are the top-level organizational units. Each queue automatically gets a "Default" partition, and you can create additional partitions for different processing priorities or logical separation.
+#### Consumer Groups (Bus Mode)
 
 ```javascript
-// Messages go to "Default" partition if not specified
-await client.push({
-  items: [{ queue: 'orders', payload: { orderId: 123 } }]
-});
+// Multiple services process the same messages independently
 
-// Explicit partition specification
-await client.push({
-  items: [{ 
-    queue: 'orders', 
-    partition: 'high-priority',
-    payload: { orderId: 456, urgent: true } 
-  }]
-});
+// Analytics service
+for await (const event of client.take('events@analytics')) {
+  await updateAnalytics(event.data);
+  await client.ack(event, true, { group: 'analytics' });
+}
+
+// Monitoring service (gets same messages)
+for await (const event of client.take('events@monitoring')) {
+  await checkThresholds(event.data);
+  await client.ack(event, true, { group: 'monitoring' });
+}
+
+// Audit service (also gets same messages)
+for await (const event of client.take('events@audit')) {
+  await logToAudit(event.data);
+  await client.ack(event, true, { group: 'audit' });
+}
 ```
 
-### Priority Processing
+#### Subscription Modes
+
+```javascript
+// Start from all existing messages (replay)
+for await (const event of client.take('events@replay-service', {
+  subscriptionMode: 'all'
+})) {
+  await replayEvent(event.data);
+  await client.ack(event);
+}
+
+// Start from new messages only (real-time)
+for await (const event of client.take('events@realtime', {
+  subscriptionMode: 'new'
+})) {
+  await processEvent(event.data);
+  await client.ack(event);
+}
 
-The system supports two levels of priority:
+// Start from specific timestamp
+for await (const event of client.take('events@historical', {
+  subscriptionMode: 'from',
+  subscriptionFrom: '2024-01-01T00:00:00Z'
+})) {
+  await processHistoricalEvent(event.data);
+  await client.ack(event);
+}
+```
 
-1. **Queue Priority**: Higher priority queues are processed first
-2. **FIFO Within Partitions**: Messages within the same partition are always processed in order
-3. **Partitions**: Partitions are now simple FIFO containers - all configuration is at the queue level
+#### Error Handling
 
 ```javascript
-// Configure queue with priority
-await client.configure({
-  queue: 'orders',
-  options: { priority: 10 } // Higher number = higher priority
-});
+// Robust error handling with retries
+for await (const message of client.take('critical-tasks')) {
+  let retries = 3;
+  
+  while (retries > 0) {
+    try {
+      await processTask(message.data);
+      await client.ack(message);
+      break;
+    } catch (error) {
+      retries--;
+      if (retries === 0) {
+        console.error('Task failed after retries:', error);
+        await client.ack(message, false, { error: error.message });
+      } else {
+        await new Promise(r => setTimeout(r, 1000 * (4 - retries)));
+      }
+    }
+  }
+}
 ```
 
-### Message Lifecycle
+#### Graceful Shutdown
+
+```javascript
+let running = true;
+
+process.on('SIGTERM', () => {
+  console.log('Shutting down gracefully...');
+  running = false;
+});
 
+for await (const message of client.take('orders')) {
+  if (!running) break;
+  
+  await processOrder(message.data);
+  await client.ack(message);
+}
+
+await client.close();
+console.log('Shutdown complete');
 ```
-pending → processing → completed/failed → (retry) → dead_letter
+
+---
+
+## 🖥️ Server Setup
+
+### Single Server
+
+```bash
+# Start the server
+npm start
+
+# Or with custom configuration
+PORT=6632 \
+DB_POOL_SIZE=20 \
+QUEEN_ENCRYPTION_KEY=$(openssl rand -hex 32) \
+npm start
 ```
 
-1. **Pending**: Message is queued and waiting to be processed
-2. **Processing**: Message is leased to a worker (with timeout)
-3. **Completed**: Message was successfully processed
-4. **Failed**: Message processing failed (may retry based on configuration)
-5. **Dead Letter**: Message exceeded retry limits
+### Multi-Server (Load Balanced)
 
-### Lease-Based Processing
+Queen supports running multiple servers for high availability and load distribution:
+
+```bash
+# Server 1
+PORT=6632 WORKER_ID=server-1 npm start
+
+# Server 2
+PORT=6633 WORKER_ID=server-2 npm start
+
+# Server 3
+PORT=6634 WORKER_ID=server-3 npm start
+```
 
-Messages are "leased" to workers for a specific duration. If not acknowledged within the lease time, they automatically return to pending status for retry.
+Client configuration:
 
 ```javascript
-// Configure lease time (default: 300 seconds)
-await client.configure({
-  queue: 'long-tasks',
-  options: { leaseTime: 600 } // 10 minutes
+const client = new Queen({
+  baseUrls: [
+    'http://localhost:6632',
+    'http://localhost:6633',
+    'http://localhost:6634'
+  ],
+  loadBalancingStrategy: 'ROUND_ROBIN',  // or 'RANDOM', 'LEAST_CONNECTIONS'
+  enableFailover: true
 });
 ```
 
-## 🔐 Advanced Concepts
+### Docker Deployment
 
-### Partition Locking
+```dockerfile
+FROM node:22-alpine
+
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --production
 
-Partition locking is a critical mechanism that ensures message processing isolation and prevents duplicate processing. When a consumer retrieves messages from a partition, that partition becomes "locked" to that consumer for the duration of the lease.
+COPY . .
 
-#### How Partition Locking Works
+EXPOSE 6632
+CMD ["node", "src/server.js"]
+```
 
-1. **Lock Acquisition**: When a consumer calls `pop()`, the system:
-   - Checks for available messages in unlocked partitions
-   - Acquires a lease on the partition(s) containing those messages
-   - Records the lease with an expiration time based on the queue's `leaseTime`
+```yaml
+# docker-compose.yml
+version: '3.8'
 
-2. **Lock Duration**: The partition remains locked until:
-   - The consumer acknowledges all messages (releases the lock)
-   - The lease expires (automatic release after `leaseTime` seconds)
-   - The consumer explicitly releases the partition
+services:
+  postgres:
+    image: postgres:16
+    environment:
+      POSTGRES_DB: queen
+      POSTGRES_USER: queen
+      POSTGRES_PASSWORD: queen
+    volumes:
+      - postgres_data:/var/lib/postgresql/data
+    ports:
+      - "5432:5432"
 
-3. **Lock Scope**: 
-   - In **Queue Mode**: Each consumer gets a unique session, preventing any other consumer from accessing the same partition
-   - In **Bus Mode**: Locks are per consumer group, allowing different groups to process the same messages independently
+  queen:
+    build: .
+    ports:
+      - "6632:6632"
+    environment:
+      PG_HOST: postgres
+      PG_DB: queen
+      PG_USER: queen
+      PG_PASSWORD: queen
+      DB_POOL_SIZE: 20
+      QUEEN_ENCRYPTION_KEY: ${QUEEN_ENCRYPTION_KEY}
+    depends_on:
+      - postgres
 
-```javascript
-// Example: Two consumers in queue mode
-const consumer1 = await client.pop({ queue: 'orders' }); 
-// Consumer 1 gets messages from partition A and locks it
+volumes:
+  postgres_data:
+```
+
+### Environment Variables
+
+See the [Configuration](#-configuration) section for a complete list of environment variables.
 
-const consumer2 = await client.pop({ queue: 'orders' }); 
-// Consumer 2 gets messages from partition B (A is locked)
+### Database Schema
 
-// After Consumer 1 acknowledges:
-await client.ack(consumer1.messages[0].transactionId, 'completed');
-// Partition A is now unlocked and available
+The database schema is automatically created when you run:
+
+```bash
+node init-db.js
 ```
 
-### FIFO Ordering Guarantees
+This creates:
+- `queen.queues` - Top-level message containers
+- `queen.partitions` - Subdivisions within queues (FIFO ordering)
+- `queen.messages` - Individual messages with processing state
 
-Queen provides strong FIFO (First-In-First-Out) ordering guarantees **within each partition**. This means:
+---
 
-#### Partition-Level FIFO
+## 💡 Core Concepts
 
-Messages within the same partition are always processed in the exact order they were received:
+### Architecture
 
-```javascript
-// These messages will be processed in order 1, 2, 3
-await client.push({
-  items: [
-    { queue: 'tasks', partition: 'user-123', payload: { step: 1 } },
-    { queue: 'tasks', partition: 'user-123', payload: { step: 2 } },
-    { queue: 'tasks', partition: 'user-123', payload: { step: 3 } }
-  ]
-});
+Queen uses a two-tier architecture:
 
-// Consumer will always receive them in order 1, 2, 3
-const result = await client.pop({ queue: 'tasks', partition: 'user-123' });
+```
+Queues (optional namespace/task grouping)
+  └── Partitions (FIFO ordering, parallel processing)
+       └── Messages (lease-based processing)
 ```
 
-#### Cross-Partition Ordering
+**Key Principles:**
+- **Configuration at queue level**: All settings (priority, lease time, retries) apply to the entire queue
+- **FIFO within partitions**: Messages in the same partition are always processed in order
+- **Partition locking**: Prevents duplicate processing across consumers
+- **Lease-based processing**: Messages automatically return to pending if not acknowledged
 
-Messages in different partitions can be processed in parallel and have no ordering guarantees relative to each other:
+### Queues and Partitions
+
+**Queues** are top-level organizational units. Each queue automatically gets a "Default" partition, and you can create additional partitions for logical separation or parallel processing.
 
 ```javascript
-// These can be processed in any order relative to each other
-await client.push({
-  items: [
-    { queue: 'tasks', partition: 'user-123', payload: { data: 'A' } },
-    { queue: 'tasks', partition: 'user-456', payload: { data: 'B' } }
-  ]
-});
+// Messages go to "Default" partition
+await client.push('orders', { orderId: 123 });
+
+// Push to specific partition
+await client.push('orders/high-priority', { orderId: 456 });
+
+// Take from specific partition
+for await (const order of client.take('orders/high-priority')) {
+  await processUrgentOrder(order.data);
+  await client.ack(order);
+}
 ```
 
-#### Use Cases for Partitioning
+**Partitions enable:**
+- **Parallel processing**: Different consumers can process different partitions simultaneously
+- **Ordered processing**: FIFO guarantees within each partition
+- **Logical separation**: Different priorities, teams, or workflow stages
+- **Resource isolation**: Lock contention is per-partition
 
-- **Per-User Processing**: Use user ID as partition to ensure all user operations are processed in order
-- **Per-Resource Processing**: Use resource ID to maintain operation order for specific resources
-- **Priority Lanes**: Use different partitions for different priority levels
+### Message Lifecycle
 
-### Consumer Groups (Bus Mode)
+```
+pending → processing → completed/failed → (retry) → dead_letter
+```
 
-Consumer groups enable pub-sub messaging patterns where multiple independent consumers can process the same messages. This is ideal for scenarios like event streaming, audit logging, and analytics.
+1. **Pending**: Message queued, waiting to be processed
+2. **Processing**: Leased to a worker (with timeout)
+3. **Completed**: Successfully processed
+4. **Failed**: Processing failed (may retry based on `retryLimit`)
+5. **Dead Letter**: Exceeded retry limits
 
-#### How Consumer Groups Work
+### Partition Locking
 
-1. **Independent Processing**: Each consumer group maintains its own:
-   - Message status tracking
-   - Partition leases
-   - Retry counters
-   - Processing state
+**Partition locking ensures message processing isolation** - when a consumer retrieves messages from a partition, that partition is locked to prevent other consumers from accessing it until:
 
-2. **Message Visibility**: All consumer groups see all messages, but each group tracks which messages it has processed independently
+- The consumer acknowledges all messages (releases lock)
+- The lease expires (automatic release)
+- The consumer explicitly releases the partition
 
-3. **Partition Locking per Group**: Within a consumer group, partition locking still applies to prevent duplicate processing
+**Lock Scope:**
+- **Queue Mode**: Each consumer session is unique - locks prevent any other consumer from accessing the partition
+- **Bus Mode**: Locks are per consumer group - different groups can process the same partition independently
 
 ```javascript
-// Analytics service (Group A)
-const analyticsResult = await client.pop({
-  queue: 'events',
-  consumerGroup: 'analytics-service'
-});
+// Example: Partition locking in action
 
-// Audit service (Group B) - gets the same messages
-const auditResult = await client.pop({
-  queue: 'events',
-  consumerGroup: 'audit-service'
-});
+// Consumer 1 takes from partition A (locks it)
+for await (const msg of client.take('orders', { limit: 5 })) {
+  // Processing partition A - no other consumer can access it
+  await client.ack(msg);
+  // Partition A unlocked after all 5 messages acknowledged
+}
 
-// Billing service (Group C) - also gets the same messages
-const billingResult = await client.pop({
-  queue: 'events',
-  consumerGroup: 'billing-service'
-});
+// Consumer 2 gets messages from partition B (A was locked)
+for await (const msg of client.take('orders', { limit: 5 })) {
+  // Processing partition B instead
+  await client.ack(msg);
+}
 ```
 
-#### Consumer Group Subscription Modes
+### FIFO Ordering
 
-When a consumer group is created, it can specify when to start consuming messages:
+Queen provides **strong FIFO guarantees within each partition**:
 
 ```javascript
-// Start from all existing messages
-await client.pop({
-  queue: 'events',
-  consumerGroup: 'replay-service',
-  subscriptionMode: 'all'
-});
-
-// Start from messages created after joining
-await client.pop({
-  queue: 'events',
-  consumerGroup: 'realtime-service',
-  subscriptionMode: 'new'
-});
+// These messages will be processed in order 1, 2, 3
+await client.push('tasks/user-123', [
+  { step: 1, action: 'create' },
+  { step: 2, action: 'update' },
+  { step: 3, action: 'complete' }
+]);
 
-// Start from a specific timestamp
-await client.pop({
-  queue: 'events',
-  consumerGroup: 'batch-processor',
-  subscriptionFrom: '2024-01-01T00:00:00Z'
-});
+// Consumer will always receive them in order
+for await (const task of client.take('tasks/user-123')) {
+  console.log(task.data.step);  // Prints: 1, then 2, then 3
+  await client.ack(task);
+}
 ```
 
-### Namespace and Task Filtering
+**Use cases:**
+- **Per-user operations**: Use user ID as partition for ordered processing
+- **Per-resource operations**: Use resource ID to maintain operation order
+- **Workflow stages**: Use partition to represent different stages
 
-Queen supports cross-queue message consumption through namespace and task filtering, with full partition locking support:
-
-#### Namespace-Based Routing
+### Consumer Groups (Bus Mode)
 
-Group related queues under a namespace and consume from all of them:
+Consumer groups enable **pub-sub messaging** where multiple independent consumers process the same messages:
 
 ```javascript
-// Configure multiple queues with the same namespace
-await client.configure({
-  queue: 'orders-processing',
-  namespace: 'ecommerce',
-  task: 'process',
-  options: { leaseTime: 30 }
-});
+// Push once
+await client.push('events', { type: 'order.created', orderId: 123 });
+
+// Multiple services consume independently
+// Service 1: Analytics
+for await (const event of client.take('events@analytics')) {
+  await updateAnalytics(event.data);
+  await client.ack(event);
+}
 
-await client.configure({
-  queue: 'inventory-updates',
-  namespace: 'ecommerce',
-  task: 'update',
-  options: { leaseTime: 30 }
-});
+// Service 2: Notification (gets same message)
+for await (const event of client.take('events@notification')) {
+  await sendNotification(event.data);
+  await client.ack(event);
+}
 
-// Consume from all queues in the namespace
-const messages = await client.pop({
-  namespace: 'ecommerce'
-}, { batch: 10 });
-// Gets messages from both queues, with partition locking across all
+// Service 3: Audit (also gets same message)
+for await (const event of client.take('events@audit')) {
+  await logEvent(event.data);
+  await client.ack(event);
+}
 ```
 
-#### Task-Based Routing
+**Each consumer group maintains:**
+- Independent message status tracking
+- Separate partition leases
+- Individual retry counters
+- Isolated processing state
 
-Filter messages by specific tasks across namespaces:
+### Queue Mode vs Bus Mode
 
+**Queue Mode** (default - competitive consumption):
 ```javascript
-// Consume only 'process' tasks from the ecommerce namespace
-const messages = await client.pop({
-  namespace: 'ecommerce',
-  task: 'process'
-}, { batch: 5 });
-```
+// Without consumer group - messages distributed
+await client.push('tasks', { id: 1 });
+await client.push('tasks', { id: 2 });
 
-#### Partition Locking with Filters
+// Worker 1 gets message 1
+for await (const msg of client.take('tasks')) { }
 
-When using namespace/task filtering:
-- The system locks all partitions from which messages are retrieved
-- Different consumers cannot access the same partitions until locks are released
-- Consumer groups maintain independent locks
+// Worker 2 gets message 2 (different message)
+for await (const msg of client.take('tasks')) { }
+```
 
+**Bus Mode** (pub-sub with consumer groups):
 ```javascript
-// Consumer 1: Gets messages and locks partitions A, B, C
-const result1 = await client.pop({ namespace: 'ecommerce' });
+// With consumer groups - all groups see all messages
+await client.push('events', { id: 1 });
 
-// Consumer 2: Gets messages from different partitions D, E (A, B, C are locked)
-const result2 = await client.pop({ namespace: 'ecommerce' });
+// Group 1 gets message 1
+for await (const msg of client.take('events@group1')) { }
 
-// No partition overlap between consumers
+// Group 2 also gets message 1 (same message)
+for await (const msg of client.take('events@group2')) { }
 ```
 
-### Concurrency Control
+**Mixed Mode** (combine both):
+```javascript
+// Competitive workers process jobs
+for await (const job of client.take('jobs')) {
+  await processJob(job.data);
+  await client.ack(job);
+}
 
-Queen provides several mechanisms for controlling concurrent message processing:
+// Monitoring sees all jobs (bus mode)
+for await (const job of client.take('jobs@monitoring')) {
+  await monitorJob(job.data);
+  await client.ack(job);
+}
+```
 
-#### 1. Partition-Based Concurrency
+### Priority Processing
 
-Control parallelism by the number of partitions:
+Configure priority at the **queue level**:
 
 ```javascript
-// Create multiple partitions for parallel processing
-const partitions = ['worker-1', 'worker-2', 'worker-3', 'worker-4'];
+await client.queue('urgent-orders', { priority: 100 });
+await client.queue('normal-orders', { priority: 50 });
+await client.queue('batch-jobs', { priority: 10 });
 
-// Distribute messages across partitions
-await client.push({
-  items: messages.map((msg, i) => ({
-    queue: 'tasks',
-    partition: partitions[i % partitions.length],
-    payload: msg
-  }))
+// Urgent orders processed first, then normal, then batch
+```
+
+### Lease-Based Processing
+
+Messages are "leased" to workers for a specific duration. If not acknowledged within the lease time, they automatically return to pending status:
+
+```javascript
+// Configure lease time
+await client.queue('long-tasks', { 
+  leaseTime: 600  // 10 minutes to process
 });
 
-// Each worker processes one partition
-const worker1 = await client.pop({ queue: 'tasks', partition: 'worker-1' });
-const worker2 = await client.pop({ queue: 'tasks', partition: 'worker-2' });
-// Workers process in parallel without interference
+// If worker crashes or takes too long:
+// - After 10 minutes, lease expires
+// - Message returns to pending
+// - Another worker can pick it up
 ```
 
-#### 2. Lease-Based Concurrency
+### Delayed Processing
 
-Automatic concurrency control through lease timeouts:
+Schedule messages for future processing:
 
 ```javascript
-// Configure short leases for quick tasks
-await client.configure({
-  queue: 'quick-tasks',
-  options: { 
-    leaseTime: 30,  // 30 seconds per message
-    retryLimit: 3   // Retry up to 3 times
-  }
+await client.queue('scheduled-jobs', {
+  delayedProcessing: 3600  // 1 hour delay
 });
 
-// Long-running tasks need longer leases
-await client.configure({
-  queue: 'heavy-processing',
-  options: { 
-    leaseTime: 600,  // 10 minutes per message
-    retryLimit: 1    // Retry only once
-  }
+await client.push('scheduled-jobs', { 
+  reportType: 'daily-sales' 
 });
+// Message won't be available for processing until 1 hour later
 ```
 
-#### 3. Batch Size Control
+### Window Buffering
 
-Limit concurrent processing per consumer:
+Batch messages within a time window:
 
 ```javascript
-// Each consumer processes max 5 messages at a time
-const batch = await client.pop({ 
-  queue: 'tasks' 
-}, { 
-  batch: 5  // Limit to 5 concurrent messages
+await client.queue('analytics', {
+  windowBuffer: 60  // Wait 60 seconds to accumulate messages
 });
 
-// Process batch
-for (const message of batch.messages) {
-  await processMessage(message);
-  await client.ack(message.transactionId, 'completed');
-}
+// Messages held for 60 seconds to allow efficient batching
 ```
 
-### Message Visibility and Isolation
+### Retry and Dead Letter Queue
 
-#### Queue Mode (Default)
+```javascript
+await client.queue('payments', {
+  retryLimit: 3,              // Retry up to 3 times
+  dlqAfterMaxRetries: true    // Move to DLQ after max retries
+});
 
-In queue mode, messages are consumed competitively - once a consumer gets a message, no other consumer can see it:
+// Failed messages automatically retry
+await client.ack(message, false);  // Will retry if retries < 3
 
-```javascript
-// Without consumer group - competitive consumption
-const consumer1 = await client.pop({ queue: 'tasks' });
-const consumer2 = await client.pop({ queue: 'tasks' });
-// Each consumer gets different messages
+// After 3 failures, message moves to dead_letter status
 ```
 
-#### Bus Mode (Consumer Groups)
+### Enterprise Features
+
+#### 1. Encryption (AES-256-GCM)
 
-In bus mode, all consumer groups see all messages:
+```bash
+# Generate encryption key
+export QUEEN_ENCRYPTION_KEY=$(openssl rand -hex 32)
+```
 
 ```javascript
-// With consumer groups - broadcast consumption
-const service1 = await client.pop({ 
-  queue: 'events',
-  consumerGroup: 'service-1'
+await client.queue('sensitive-data', {
+  encryptionEnabled: true
 });
 
-const service2 = await client.pop({ 
-  queue: 'events',
-  consumerGroup: 'service-2'
-});
-// Both services get the same messages
+// Messages encrypted at rest in database
+await client.push('sensitive-data', { ssn: '123-45-6789' });
 ```
 
-#### Mixed Mode
-
-You can combine both patterns in the same system:
+#### 2. Message Retention
 
 ```javascript
-// Competitive workers for processing
-const worker = await client.pop({ queue: 'jobs' });
-
-// Broadcast to monitoring services
-const monitor = await client.pop({ 
-  queue: 'jobs',
-  consumerGroup: 'monitoring'
+await client.queue('temp-queue', {
+  retentionSeconds: 3600,           // Delete pending after 1 hour
+  completedRetentionSeconds: 300,   // Delete completed after 5 minutes
+  retentionEnabled: true
 });
+```
+
+#### 3. Message Eviction (SLA Enforcement)
 
-const analytics = await client.pop({ 
-  queue: 'jobs',
-  consumerGroup: 'analytics'
+```javascript
+await client.queue('time-sensitive', {
+  maxWaitTimeSeconds: 60  // Evict messages older than 1 minute
 });
 ```
 
 ### Best Practices
 
-#### 1. Partition Strategy
+**1. Partition Strategy**
+- Use user IDs for per-user ordering
+- Use resource IDs for per-resource ordering
+- Use round-robin for load distribution
+- Keep partition counts manageable (10-100s, not 1000s)
 
-- **User-based**: Use user IDs as partitions for per-user ordering
-- **Resource-based**: Use resource IDs for ordered operations on resources
-- **Round-robin**: Use rotating partition names for load distribution
-- **Priority-based**: Use separate partitions for different priority levels
+**2. Lease Management**
+- Set lease time slightly longer than expected processing time
+- Handle timeouts gracefully
+- Acknowledge messages as soon as processing completes
 
-#### 2. Consumer Group Design
+**3. Consumer Group Design**
+- One clear purpose per consumer group
+- Design groups to be independent
+- Ensure operations are idempotent
 
-- **Single Responsibility**: Each consumer group should have one clear purpose
-- **Independent Processing**: Design groups to be independent of each other
-- **Idempotent Operations**: Ensure operations can be safely retried
+**4. Error Handling**
+- Always wrap processing in try-catch
+- Provide meaningful error messages in ack
+- Use retry limits appropriately
+- Monitor dead letter queue
 
-#### 3. Lease Management
+---
 
-- **Right-size Leases**: Set lease times slightly longer than expected processing time
-- **Handle Timeouts**: Implement proper timeout handling and retries
-- **Release Early**: Acknowledge messages as soon as processing completes
+## 🔌 HTTP API Reference
 
-#### 4. Error Handling
-
-```javascript
-try {
-  const messages = await client.pop({ queue: 'tasks' });
-  
-  for (const message of messages.messages) {
-    try {
-      await processMessage(message);
-      await client.ack(message.transactionId, 'completed');
-    } catch (error) {
-      // Log error but don't ack - message will retry
-      console.error('Processing failed:', error);
-      await client.ack(message.transactionId, 'failed', error.message);
-    }
-  }
-} catch (error) {
-  console.error('Pop failed:', error);
-}
-```
-
-## 🔌 API Reference
-
-### Base URL
-```
-http://localhost:6632/api/v1
-```
+Base URL: `http://localhost:6632/api/v1`
 
 ### Push Messages
 
 **Endpoint:** `POST /api/v1/push`
 
-```javascript
+**Request:**
+```json
 {
   "items": [
     {
-      "queue": "email-queue",           // Required
-      "partition": "urgent",            // Optional (defaults to "Default")
-      "payload": {                      // Required: message data
-        "to": "user@example.com",
-        "subject": "Hello"
-      },
-      "transactionId": "uuid-here"      // Optional: for idempotency
+      "queue": "orders",
+      "partition": "urgent",
+      "payload": { "orderId": 123, "amount": 99.99 },
+      "transactionId": "optional-idempotency-key",
+      "traceId": "550e8400-e29b-41d4-a716-446655440000"
     }
   ]
 }
 ```
 
 **Response:**
-```javascript
+```json
 {
   "messages": [
     {
@@ -610,35 +919,40 @@ http://localhost:6632/api/v1
 
 ### Pop Messages
 
-**From Specific Partition:**
+**From specific partition:**
 ```
 GET /api/v1/pop/queue/{queue}/partition/{partition}?wait=true&timeout=30000&batch=10
 ```
 
-**From Any Partition in Queue:**
+**From any partition in queue:**
 ```
 GET /api/v1/pop/queue/{queue}?wait=true&timeout=30000&batch=10
 ```
 
-**With Namespace/Task Filter:**
+**With namespace/task filter:**
+```
+GET /api/v1/pop?namespace=ecommerce&task=checkout&wait=true&timeout=30000&batch=10
+```
+
+**With consumer group (bus mode):**
 ```
-GET /api/v1/pop?namespace=my-app&task=emails&wait=true&timeout=30000&batch=10
+GET /api/v1/pop/queue/{queue}?consumerGroup=analytics&subscriptionMode=all&wait=true&timeout=30000
 ```
 
 **Response:**
-```javascript
+```json
 {
   "messages": [
     {
       "id": "018e63b7-6165-453f-88ae-56effa177605",
       "transactionId": "4dfb0478-655b-4c91-bcd9-b7acacf0400f",
-      "queue": "email-queue",
+      "queue": "orders",
       "partition": "urgent",
-      "data": { "to": "user@example.com", "subject": "Hello" },
+      "data": { "orderId": 123, "amount": 99.99 },
       "retryCount": 0,
       "priority": 10,
-      "createdAt": "2023-10-08T12:00:00.000Z",
-      "options": { "leaseTime": 300 }
+      "createdAt": "2024-10-08T12:00:00.000Z",
+      "options": { "leaseTime": 300, "retryLimit": 3 }
     }
   ]
 }
@@ -646,850 +960,967 @@ GET /api/v1/pop?namespace=my-app&task=emails&wait=true&timeout=30000&batch=10
 
 ### Acknowledge Messages
 
-**Single Acknowledgment:**
-```javascript
+**Single:**
+```json
 POST /api/v1/ack
 {
-  "transactionId": "uuid",
-  "status": "completed",        // "completed" or "failed"
-  "error": "optional error"     // Required if status is "failed"
+  "transactionId": "4dfb0478-655b-4c91-bcd9-b7acacf0400f",
+  "status": "completed",
+  "consumerGroup": "analytics",
+  "error": null
 }
 ```
 
-**Batch Acknowledgment:**
-```javascript
+**Batch:**
+```json
 POST /api/v1/ack/batch
 {
   "acknowledgments": [
-    { "transactionId": "uuid1", "status": "completed" },
-    { "transactionId": "uuid2", "status": "failed", "error": "Processing error" }
+    { "transactionId": "uuid-1", "status": "completed" },
+    { "transactionId": "uuid-2", "status": "failed", "error": "Processing error" }
   ]
 }
 ```
 
-### Queue Configuration
+### Configure Queue
 
-```javascript
+```json
 POST /api/v1/configure
 {
-  "queue": "email-queue",
-  "partition": "urgent",        // Optional (defaults to "Default")
+  "queue": "orders",
+  "namespace": "ecommerce",
+  "task": "checkout",
   "options": {
-    "leaseTime": 600,           // Seconds before lease expires
-    "retryLimit": 5,            // Max retry attempts
-    "priority": 10,             // Partition priority (higher = first)
-    "delayedProcessing": 60,    // Delay before message becomes available
-    "windowBuffer": 30          // Buffer messages for batch processing
+    "leaseTime": 600,
+    "retryLimit": 5,
+    "priority": 10,
+    "maxSize": 10000,
+    "ttl": 3600,
+    "dlqAfterMaxRetries": true,
+    "delayedProcessing": 0,
+    "windowBuffer": 0,
+    "retentionSeconds": 0,
+    "completedRetentionSeconds": 0,
+    "retentionEnabled": false,
+    "encryptionEnabled": false,
+    "maxWaitTimeSeconds": 0
   }
 }
 ```
 
 ### Analytics
 
-```javascript
-// Get all queues overview
-GET /api/v1/analytics/queues
+**Queue statistics:**
+```
+GET /api/v1/analytics/queue/{queue}
+```
 
-// Get queue statistics
-GET /api/v1/analytics/queue/{queueName}
+**All queues overview:**
+```
+GET /api/v1/analytics/queues
+```
 
-// Get namespace statistics
-GET /api/v1/analytics?namespace={namespace}
+**Queue depths:**
+```
+GET /api/v1/analytics/queue-depths
+```
 
-// Get throughput metrics
+**Throughput metrics:**
+```
 GET /api/v1/analytics/throughput
+```
 
-// Get queue depths
-GET /api/v1/analytics/queue-depths
+**Queue lag analysis:**
+```
+GET /api/v1/analytics/queue-lag?queue=orders
 ```
 
-## 📱 Client SDK
+### Message Management
 
-### Installation
+**List messages:**
+```
+GET /api/v1/messages?queue=orders&status=pending&limit=100
+```
 
-```javascript
-import { createQueenClient } from './src/client/queenClient.js';
+**Get single message:**
+```
+GET /api/v1/messages/{transactionId}
+```
 
-const client = createQueenClient({
-  baseUrl: 'http://localhost:6632',
-  timeout: 30000,
-  retryAttempts: 3,
-  retryDelay: 1000
-});
+**Delete message:**
+```
+DELETE /api/v1/messages/{transactionId}
 ```
 
-### Basic Operations
+**Retry failed message:**
+```
+POST /api/v1/messages/{transactionId}/retry
+```
 
-```javascript
-// Configure a queue
-await client.configure({
-  queue: 'orders',
-  options: {
-    priority: 10,
-    leaseTime: 600,
-    retryLimit: 3
-  }
-});
+**Move to dead letter queue:**
+```
+POST /api/v1/messages/{transactionId}/dlq
+```
 
-// Push single message
-await client.push({
-  items: [{
-    queue: 'orders',
-    partition: 'high-priority',
-    payload: { orderId: 123, amount: 99.99 }
-  }]
-});
+**Clear queue:**
+```
+DELETE /api/v1/queues/{queue}/clear
+```
 
-// Push batch of messages
-await client.push({
-  items: [
-    { queue: 'orders', payload: { orderId: 124 } },
-    { queue: 'orders', payload: { orderId: 125 } },
-    { queue: 'orders', payload: { orderId: 126 } }
-  ]
-});
+### System Health
 
-// Pop messages with long polling
-const result = await client.pop({
-  queue: 'orders',
-  wait: true,
-  timeout: 30000,
-  batch: 10
-});
+**Health check:**
+```
+GET /health
+```
 
-// Process messages
-for (const message of result.messages) {
-  try {
-    await processOrder(message.data);
-    await client.ack(message.transactionId, 'completed');
-  } catch (error) {
-    await client.ack(message.transactionId, 'failed', error.message);
-  }
-}
+**Detailed metrics:**
+```
+GET /metrics
 ```
 
-### Consumer Pattern
+### WebSocket (Real-time Updates)
 
-The SDK provides a convenient consumer helper for continuous message processing with two modes:
+**Connect:**
+```javascript
+const ws = new WebSocket('ws://localhost:6632/ws/dashboard');
 
-#### Individual Message Processing
+ws.onmessage = (event) => {
+  const { event: eventType, data } = JSON.parse(event.data);
+  // Handle events: message.pushed, message.completed, queue.depth, etc.
+};
+```
 
-Process messages one by one (default behavior):
+**Events:**
+- `message.pushed` - New message added
+- `message.processing` - Message being processed
+- `message.completed` - Message completed
+- `message.failed` - Message failed
+- `queue.created` - New queue created
+- `queue.depth` - Queue depth update (every 5s)
+- `system.stats` - System statistics (every 10s)
 
-```javascript
-const stopConsumer = client.consume({
-  queue: 'orders',
-  partition: 'high-priority',
-  handler: async (message) => {
-    console.log('Processing order:', message.data.orderId);
-    await processOrder(message.data);
-    // Message is automatically acknowledged on success
-  },
-  options: {
-    batch: 5,
-    wait: true,
-    timeout: 30000,
-    stopOnError: false
-  }
-});
+See [API.md](API.md) for complete API documentation.
 
-// Stop the consumer when needed
-// stopConsumer();
-```
+---
 
-#### Batch Message Processing
+## 📊 Dashboard
 
-Process entire batches of messages at once for better performance:
+Queen includes a comprehensive web dashboard for monitoring and management.
 
-```javascript
-const stopConsumer = client.consume({
-  queue: 'orders',
-  partition: 'high-priority',
-  handlerBatch: async (messages) => {
-    console.log(`Processing batch of ${messages.length} orders`);
-    
-    // Process all messages in parallel
-    await Promise.all(messages.map(async (message) => {
-      console.log('Processing order:', message.data.orderId);
-      await processOrder(message.data);
-    }));
-    
-    // Or process sequentially if needed
-    // for (const message of messages) {
-    //   await processOrder(message.data);
-    // }
-    
-    // All messages are automatically batch-acknowledged on success
-  },
-  options: {
-    batch: 10,  // Larger batches for better throughput
-    wait: true,
-    timeout: 30000,
-    stopOnError: false
-  }
-});
+### Access
+
+1. Start the server: `npm start`
+2. Open browser: `http://localhost:6632`
+3. WebSocket connection provides real-time updates
+
+### Features
+
+**System Overview**
+- Real-time metrics: total messages, processing rate, system health
+- Queue summary with pending/processing/completed counts
+- Performance indicators: throughput, latency, error rates
+
+**Queue Management**
+- Queue list with status and message counts
+- Partition view with priority indicators
+- Message browser with search and filter
+- Retry and DLQ management
+
+**Real-time Monitoring**
+- Live updates via WebSocket
+- Throughput charts (messages per second over time)
+- Queue depth graphs with trend analysis
+- Lag monitoring (processing time and backlog)
+
+**Analytics Dashboard**
+- Performance metrics per queue
+- Historical trends and patterns
+- System health monitoring
+- Database connections and memory usage
+
+**Message Browser**
+- Search by queue, partition, status, time range
+- View full payload and metadata
+- Manually retry failed messages
+- Dead letter queue management
+
+### Dashboard Development
+
+The dashboard is built with Vue.js and located in the `dashboard/` directory:
+
+```bash
+cd dashboard
+npm install
+npm run dev    # Development mode
+npm run build  # Production build
 ```
 
-**Key Benefits of Batch Processing:**
-- **Higher Throughput**: Process multiple messages simultaneously
-- **Efficient Acknowledgments**: Single batch ACK instead of individual ACKs
-- **Atomic Processing**: Either the entire batch succeeds or fails together
-- **Reduced Network Overhead**: Fewer round trips to the server
+---
 
-**Important Notes:**
-- Use either `handler` OR `handlerBatch`, not both
-- In batch mode, if processing fails, all messages in the batch are marked as failed
-- Batch size is controlled by the `batch` option (default: 1)
+## ⚙️ Configuration
 
-### Advanced Features
+All configuration uses environment variables with sensible defaults. Configuration is centralized in `src/config.js`.
 
-```javascript
-// Pop with namespace filter (cross-queue priority)
-const result = await client.pop({
-  namespace: 'ecommerce',
-  batch: 10,
-  wait: true
-});
+### Server Configuration
 
-// Batch acknowledgment
-await client.ackBatch([
-  { transactionId: 'uuid1', status: 'completed' },
-  { transactionId: 'uuid2', status: 'failed', error: 'Invalid data' }
-]);
+```bash
+PORT=6632                      # Server port (default: 6632)
+HOST=0.0.0.0                   # Server host (default: 0.0.0.0)
+WORKER_ID=worker-1             # Worker identifier
+APP_NAME=queen-mq              # Application name
 
-// Message management
-const messages = await client.messages.list({
-  queue: 'orders',
-  status: 'failed',
-  limit: 100
-});
+# CORS
+CORS_MAX_AGE=86400
+CORS_ALLOWED_ORIGINS=*
+CORS_ALLOWED_METHODS=GET,POST,PUT,DELETE,OPTIONS
+CORS_ALLOWED_HEADERS=Content-Type,Authorization
+```
+
+### Database Configuration
+
+```bash
+# Connection
+PG_USER=postgres
+PG_HOST=localhost
+PG_DB=postgres
+PG_PASSWORD=postgres
+PG_PORT=5432
 
-await client.messages.retry('transaction-id');
-await client.messages.moveToDLQ('transaction-id');
+# Connection pool
+DB_POOL_SIZE=20                # Max connections
+DB_IDLE_TIMEOUT=30000          # Idle timeout (ms)
+DB_CONNECTION_TIMEOUT=2000     # Connection timeout (ms)
+DB_STATEMENT_TIMEOUT=30000     # Statement timeout (ms)
+DB_QUERY_TIMEOUT=30000         # Query timeout (ms)
+DB_MAX_RETRIES=3               # Max retry attempts
 ```
 
-## 📊 Dashboard
+### Queue Processing
 
-The Queen system includes a comprehensive web dashboard for monitoring and management.
-
-### Accessing the Dashboard
-
-1. **Start the server**: `npm start`
-2. **Open dashboard**: Navigate to `http://localhost:6632` in your browser
-3. **WebSocket connection**: The dashboard connects via WebSocket for real-time updates
-
-### Dashboard Features
-
-#### 1. **System Overview**
-- **Real-time Metrics**: Total messages, processing rate, system health
-- **Queue Summary**: Active queues, pending messages, processing status
-- **Performance Indicators**: Throughput, latency, error rates
-
-#### 2. **Queue Management**
-- **Queue List**: All queues with current status and message counts
-- **Partition View**: Partitions within each queue with priority indicators
-- **Message Counts**: Pending, processing, completed, failed, and dead letter counts
-- **Priority Visualization**: Color-coded priority levels
-
-#### 3. **Real-time Monitoring**
-- **Live Updates**: WebSocket-powered real-time data updates
-- **Throughput Charts**: Messages per second over time
-- **Queue Depth Graphs**: Pending message counts with trend analysis
-- **Lag Monitoring**: Processing time and queue lag metrics
-
-#### 4. **Message Browser**
-- **Message Search**: Filter by queue, partition, status, or time range
-- **Message Details**: Full payload, metadata, and processing history
-- **Retry Management**: Manually retry failed messages
-- **Dead Letter Queue**: View and manage messages that exceeded retry limits
-
-#### 5. **Analytics Dashboard**
-- **Performance Metrics**: Detailed throughput and latency statistics
-- **Queue Analytics**: Per-queue performance and usage patterns
-- **Historical Data**: Trends and patterns over time
-- **System Health**: Database connections, memory usage, error rates
-
-#### 6. **Configuration Management**
-- **Queue Configuration**: View and modify queue settings
-- **Partition Settings**: Priority, lease time, retry limits
-- **System Settings**: Global configuration options
-
-### Dashboard Components
-
-The dashboard is built with Vue.js and includes:
-
-```
-dashboard/
-├── src/
-│   ├── components/
-│   │   ├── charts/           # Chart components
-│   │   │   ├── QueueDepthChart.vue
-│   │   │   ├── QueueLagChart.vue
-│   │   │   └── ThroughputChart.vue
-│   │   ├── cards/            # Metric cards
-│   │   │   └── MetricCard.vue
-│   │   ├── common/           # Shared components
-│   │   │   └── ActivityFeed.vue
-│   │   └── layout/           # Layout components
-│   │       ├── AppHeader.vue
-│   │       ├── AppLayout.vue
-│   │       └── AppSidebar.vue
-│   ├── views/                # Main pages
-│   │   ├── Dashboard.vue     # System overview
-│   │   ├── Queues.vue        # Queue management
-│   │   ├── QueueDetail.vue   # Individual queue details
-│   │   ├── Messages.vue      # Message browser
-│   │   └── Analytics.vue     # Analytics dashboard
-│   └── services/
-│       ├── api.js            # API client
-│       └── websocket.js      # WebSocket connection
-```
-
-### WebSocket API
-
-The dashboard connects via WebSocket for real-time updates:
+```bash
+# Pop defaults
+DEFAULT_TIMEOUT=30000          # Default pop timeout (ms)
+MAX_TIMEOUT=60000              # Maximum pop timeout (ms)
+DEFAULT_BATCH_SIZE=1           # Default batch size
+BATCH_INSERT_SIZE=1000         # Batch size for bulk inserts
 
-```javascript
-// Connect to dashboard WebSocket
-const ws = new WebSocket('ws://localhost:6632/ws/dashboard');
+# Long polling
+QUEUE_POLL_INTERVAL=100        # Poll interval (ms)
+QUEUE_POLL_INTERVAL_FILTERED=1000  # Poll interval for filtered pops (ms)
 
-// Receive real-time updates
-ws.onmessage = (event) => {
-  const { event: eventType, data } = JSON.parse(event.data);
-  
-  switch (eventType) {
-    case 'queue.depth.updated':
-      updateQueueDepth(data);
-      break;
-    case 'message.processed':
-      updateThroughput(data);
-      break;
-    case 'system.stats':
-      updateSystemStats(data);
-      break;
-  }
-};
+# Queue defaults
+DEFAULT_LEASE_TIME=300         # Lease time (seconds)
+DEFAULT_RETRY_LIMIT=3          # Retry limit
+DEFAULT_RETRY_DELAY=1000       # Retry delay (ms)
+DEFAULT_MAX_SIZE=10000         # Max queue size
+DEFAULT_TTL=3600               # TTL (seconds)
+DEFAULT_PRIORITY=0             # Priority
+DEFAULT_DELAYED_PROCESSING=0   # Delayed processing (seconds)
+DEFAULT_WINDOW_BUFFER=0        # Window buffer (seconds)
 ```
 
-## 📚 Examples
+### Background Jobs
 
-### Basic Email Queue
+```bash
+LEASE_RECLAIM_INTERVAL=5000    # Lease reclamation (ms)
+RETENTION_INTERVAL=300000      # Retention checks (ms)
+RETENTION_BATCH_SIZE=1000      # Retention batch size
+PARTITION_CLEANUP_DAYS=7       # Days before cleaning empty partitions
+EVICTION_INTERVAL=60000        # Eviction checks (ms)
+EVICTION_BATCH_SIZE=1000       # Eviction batch size
+```
 
-```javascript
-// Configure email queue with priority
-await client.configure({
-  queue: 'emails-urgent',
-  options: { priority: 10, leaseTime: 300 }
-});
+### WebSocket
 
-await client.configure({
-  queue: 'emails-normal',
-  options: { priority: 5, leaseTime: 300 }
-});
+```bash
+WS_COMPRESSION=0               # Compression level
+WS_MAX_PAYLOAD_LENGTH=16384    # Max payload (bytes)
+WS_IDLE_TIMEOUT=60             # Idle timeout (seconds)
+WS_MAX_CONNECTIONS=1000        # Max connections
+WS_HEARTBEAT_INTERVAL=30000    # Heartbeat (ms)
+```
 
-// Send urgent email
-await client.push({
-  items: [{
-    queue: 'emails',
-    partition: 'urgent',
-    payload: {
-      to: 'admin@company.com',
-      subject: 'System Alert',
-      body: 'Critical system issue detected'
-    }
-  }]
-});
+### Encryption
 
-// Process emails (urgent emails processed first)
-const result = await client.pop({
-  queue: 'emails',
-  batch: 10,
-  wait: true
-});
+```bash
+# Generate key: openssl rand -hex 32
+QUEEN_ENCRYPTION_KEY=<64-hex-chars>  # AES-256-GCM encryption key
 ```
 
-### Delayed Job Processing
+### Client SDK
 
-```javascript
-// Configure queue with delayed processing
-await client.configure({
-  queue: 'scheduled-jobs',
-  options: {
-    delayedProcessing: 3600, // 1 hour delay
-    priority: 5
-  }
-});
+```bash
+QUEEN_BASE_URL=http://localhost:6632
+CLIENT_RETRY_ATTEMPTS=3
+CLIENT_RETRY_DELAY=1000
+CLIENT_RETRY_BACKOFF=2
+CLIENT_POOL_SIZE=10
+CLIENT_REQUEST_TIMEOUT=30000
+```
 
-// Schedule a job for later processing
-await client.push({
-  items: [{
-    queue: 'scheduled-jobs',
-    partition: 'daily-reports',
-    payload: {
-      reportType: 'daily-sales',
-      date: '2023-10-08',
-      recipients: ['manager@company.com']
-    }
-  }]
-});
+### Queue Options
 
-// Job will not be available for processing until 1 hour later
+```javascript
+{
+  // Processing
+  leaseTime: 300,              // Seconds before lease expires
+  retryLimit: 3,               // Max retry attempts
+  priority: 0,                 // Queue priority (higher = first)
+  delayedProcessing: 0,        // Delay in seconds
+  windowBuffer: 0,             // Buffer time for batching
+  dlqAfterMaxRetries: true,    // Move to DLQ after max retries
+  
+  // Encryption (Queue-level)
+  encryptionEnabled: false,    // Enable AES-256-GCM encryption
+  
+  // Retention (Partition-level)
+  retentionSeconds: 0,                // Delete pending messages after X seconds
+  completedRetentionSeconds: 0,       // Delete completed/failed after X seconds
+  partitionRetentionSeconds: 0,       // Delete empty partitions after X seconds
+  retentionEnabled: false,            // Enable retention
+  
+  // Eviction (Queue-level)
+  maxWaitTimeSeconds: 0        // Evict messages older than X seconds
+}
 ```
 
-### Batch Processing with Window Buffer
+---
+
+## 📚 Full Examples
+
+### Example 1: Email Queue with Priority
 
 ```javascript
-// Configure for batch processing
-await client.configure({
-  queue: 'analytics',
-  options: {
-    windowBuffer: 60,  // Wait 60 seconds to batch messages
-    priority: 3
-  }
+import { Queen } from 'queen-mq';
+
+const client = new Queen({
+  baseUrls: ['http://localhost:6632']
 });
 
-// Send multiple events
-for (let i = 0; i < 100; i++) {
-  await client.push({
-    items: [{
-      queue: 'analytics',
-      partition: 'events',
-      payload: { userId: i, action: 'page_view', timestamp: Date.now() }
-    }]
-  });
-}
+// Configure queues with different priorities
+await client.queue('emails-urgent', { 
+  priority: 10, 
+  leaseTime: 300,
+  retryLimit: 5
+});
 
-// Messages will be held for 60 seconds to allow batching
-// Then all messages become available at once for efficient processing
-```
+await client.queue('emails-normal', { 
+  priority: 5, 
+  leaseTime: 300,
+  retryLimit: 3
+});
 
-### Multi-Queue Processing with Priorities
+// Producer: Send emails
+async function sendEmails() {
+  // Urgent email
+  await client.push('emails-urgent', {
+    to: 'admin@company.com',
+    subject: 'Critical Alert',
+    body: 'System issue detected',
+    timestamp: Date.now()
+  });
 
-```javascript
-// Set up multiple queues with different priorities
-const queues = [
-  { name: 'critical-alerts', priority: 100 },
-  { name: 'user-notifications', priority: 50 },
-  { name: 'background-tasks', priority: 10 }
-];
-
-for (const queue of queues) {
-  await client.configure({
-    queue: queue.name,
-    options: { priority: queue.priority }
+  // Normal email
+  await client.push('emails-normal', {
+    to: 'user@example.com',
+    subject: 'Welcome',
+    body: 'Thanks for signing up',
+    timestamp: Date.now()
   });
 }
 
-// Consumer that processes all queues by priority
-const stopConsumer = client.consume({
-  namespace: 'my-app', // Process all queues in namespace by priority
-  handler: async (message) => {
-    console.log(`Processing ${message.queue}: ${message.data.type}`);
-    await processMessage(message);
-  },
-  options: { batch: 5, wait: true }
-});
+// Consumer: Process emails
+async function processEmails() {
+  // Urgent emails processed first (higher priority)
+  for await (const email of client.take('emails-urgent', { 
+    wait: true,
+    timeout: 30000
+  })) {
+    try {
+      console.log('Sending urgent email:', email.data.to);
+      await sendEmail(email.data);
+      await client.ack(email);
+    } catch (error) {
+      console.error('Failed to send email:', error);
+      await client.ack(email, false, { error: error.message });
+    }
+  }
+}
+
+// Send batch of emails
+await sendEmails();
+
+// Start processing
+processEmails().catch(console.error);
 ```
 
-### High-Throughput Batch Processing
+### Example 2: Task Pipeline
 
 ```javascript
-// Configure queue for high-throughput batch processing
-await client.configure({
-  queue: 'data-processing',
-  options: {
-    priority: 5,
-    leaseTime: 600, // 10 minutes for batch processing
-    windowBuffer: 30 // Buffer messages for 30 seconds
-  }
+import { Queen } from 'queen-mq';
+
+const client = new Queen({
+  baseUrls: ['http://localhost:6632']
 });
 
-// High-performance batch consumer
-const stopConsumer = client.consume({
-  queue: 'data-processing',
-  partition: 'analytics',
-  handlerBatch: async (messages) => {
-    const startTime = Date.now();
-    console.log(`Processing batch of ${messages.length} analytics events`);
-    
+// Configure pipeline stages
+await client.queue('stage-1-validate', { priority: 10 });
+await client.queue('stage-2-process', { priority: 9 });
+await client.queue('stage-3-finalize', { priority: 8 });
+
+// Stage 1: Validate
+async function validateStage() {
+  for await (const msg of client.take('stage-1-validate', { wait: true })) {
     try {
-      // Extract all payloads for batch processing
-      const events = messages.map(msg => ({
-        id: msg.transactionId,
-        ...msg.data
-      }));
+      const validated = await validate(msg.data);
+      await client.ack(msg);
       
-      // Process entire batch efficiently
-      await processAnalyticsBatch(events);
-      
-      const processingTime = Date.now() - startTime;
-      console.log(`✅ Batch processed in ${processingTime}ms`);
+      // Pass to next stage
+      await client.push('stage-2-process', validated);
+    } catch (error) {
+      await client.ack(msg, false, { error: error.message });
+    }
+  }
+}
+
+// Stage 2: Process
+async function processStage() {
+  for await (const msg of client.take('stage-2-process', { wait: true })) {
+    try {
+      const processed = await process(msg.data);
+      await client.ack(msg);
       
+      // Pass to next stage
+      await client.push('stage-3-finalize', processed);
     } catch (error) {
-      console.error('Batch processing failed:', error);
-      throw error; // Will mark all messages as failed
+      await client.ack(msg, false, { error: error.message });
     }
-  },
-  options: {
-    batch: 50,     // Process up to 50 messages at once
-    wait: true,    // Use long polling
-    timeout: 30000,
-    stopOnError: false
   }
-});
+}
 
-async function processAnalyticsBatch(events) {
-  // Example: Bulk insert to database
-  await database.analytics.insertMany(events);
-  
-  // Example: Send to external analytics service
-  await analyticsService.sendBatch(events);
-  
-  // Example: Update aggregated metrics
-  await updateMetrics(events);
+// Stage 3: Finalize
+async function finalizeStage() {
+  for await (const msg of client.take('stage-3-finalize', { wait: true })) {
+    try {
+      await finalize(msg.data);
+      await client.ack(msg);
+      console.log('Pipeline complete:', msg.data.id);
+    } catch (error) {
+      await client.ack(msg, false, { error: error.message });
+    }
+  }
 }
-```
 
-## ⚡ Performance
+// Start pipeline
+Promise.all([
+  validateStage(),
+  processStage(),
+  finalizeStage()
+]);
 
-### Benchmarks
+// Add work to pipeline
+await client.push('stage-1-validate', { id: 1, data: 'raw data' });
+```
 
-- **Throughput**: 10,000+ messages/second
-- **Latency**: < 10ms for immediate pop operations
-- **Concurrent Connections**: 1,000+ long polling connections
-- **Database**: Optimized for PostgreSQL with proper indexing
+### Example 3: Event Streaming (Bus Mode)
 
-### Optimization Features
+```javascript
+import { Queen } from 'queen-mq';
 
-- **Connection Pooling**: Efficient database connection management
-- **Resource Caching**: In-memory cache for queue/partition lookups
-- **Batch Operations**: Bulk insert/update for high throughput
-- **Optimized Queries**: Carefully crafted SQL with proper indexes
-- **Event-Driven Architecture**: Minimal polling overhead
+const client = new Queen({
+  baseUrls: ['http://localhost:6632']
+});
 
-### Performance Tuning
+// Configure event queue
+await client.queue('events', { 
+  priority: 10,
+  leaseTime: 60
+});
 
-```javascript
-// Environment variables for performance tuning
-export DB_POOL_SIZE=20              // Database connection pool size
-export DB_IDLE_TIMEOUT=30000        // Connection idle timeout
-export DB_CONNECTION_TIMEOUT=2000   // Connection establishment timeout
-```
+// Producer: Emit events
+async function emitEvents() {
+  await client.push('events', {
+    type: 'order.created',
+    orderId: 12345,
+    userId: 789,
+    amount: 99.99,
+    timestamp: Date.now()
+  });
+}
 
-## 🔒 Enterprise Features
+// Consumer 1: Analytics Service
+async function analyticsService() {
+  for await (const event of client.take('events@analytics', {
+    subscriptionMode: 'all',  // Replay all messages
+    wait: true
+  })) {
+    console.log('[Analytics] Processing event:', event.data.type);
+    await updateAnalytics(event.data);
+    await client.ack(event, true, { group: 'analytics' });
+  }
+}
 
-Queen includes three powerful enterprise features for production deployments:
+// Consumer 2: Notification Service
+async function notificationService() {
+  for await (const event of client.take('events@notifications', {
+    subscriptionMode: 'new',  // Only new messages
+    wait: true
+  })) {
+    console.log('[Notifications] Processing event:', event.data.type);
+    await sendNotification(event.data);
+    await client.ack(event, true, { group: 'notifications' });
+  }
+}
 
-### 1. Encryption
-Protect sensitive data with AES-256-GCM encryption at the queue level.
+// Consumer 3: Audit Service
+async function auditService() {
+  for await (const event of client.take('events@audit', {
+    subscriptionMode: 'all',  // Log everything
+    wait: true
+  })) {
+    console.log('[Audit] Logging event:', event.data.type);
+    await logToAudit(event.data);
+    await client.ack(event, true, { group: 'audit' });
+  }
+}
 
-**Setup:**
-```bash
-# Set encryption key (64 hex characters = 32 bytes)
-export QUEEN_ENCRYPTION_KEY=$(openssl rand -hex 32)
+// Start all services (they all see the same events)
+Promise.all([
+  analyticsService(),
+  notificationService(),
+  auditService()
+]);
+
+// Emit events
+await emitEvents();
 ```
 
-**Configuration:**
+### Example 4: Batch Processing
+
 ```javascript
-await client.configure({
-  queue: 'sensitive-data',
-  options: {
-    encryptionEnabled: true
-  }
+import { Queen } from 'queen-mq';
+
+const client = new Queen({
+  baseUrls: ['http://localhost:6632']
 });
-```
 
-### 2. Message Retention
-Automatically clean up old messages to prevent storage bloat.
+// Configure for batch processing
+await client.queue('data-processing', {
+  priority: 5,
+  leaseTime: 600,        // 10 minutes for batch
+  windowBuffer: 30       // Buffer for 30 seconds
+});
 
-**Configuration:**
-```javascript
-await client.configure({
-  queue: 'temp-queue',
-  options: {
-    retentionSeconds: 3600,          // Delete pending after 1 hour
-    completedRetentionSeconds: 300,  // Delete completed after 5 minutes
-    retentionEnabled: true
+// Producer: Send data
+async function sendData() {
+  const records = [];
+  for (let i = 0; i < 1000; i++) {
+    records.push({ id: i, value: Math.random() });
   }
-});
-```
+  
+  // Push in batches
+  await client.push('data-processing/analytics', records);
+}
 
-**Environment:**
-```bash
-export RETENTION_INTERVAL=300000  # Cleanup interval in milliseconds
+// Consumer: Batch processor
+async function batchProcessor() {
+  const BATCH_SIZE = 100;
+  const batch = [];
+  
+  for await (const message of client.take('data-processing/analytics', {
+    batch: BATCH_SIZE,
+    wait: true,
+    timeout: 30000
+  })) {
+    batch.push(message);
+    
+    // Process when batch is full
+    if (batch.length >= BATCH_SIZE) {
+      try {
+        console.log(`Processing batch of ${batch.length} records`);
+        
+        // Extract data
+        const records = batch.map(m => m.data);
+        
+        // Bulk process
+        await bulkInsertToDatabase(records);
+        
+        // Acknowledge all
+        for (const msg of batch) {
+          await client.ack(msg);
+        }
+        
+        console.log(`✓ Batch complete`);
+        batch.length = 0;
+      } catch (error) {
+        console.error('Batch processing failed:', error);
+        
+        // Mark all as failed
+        for (const msg of batch) {
+          await client.ack(msg, false);
+        }
+        batch.length = 0;
+      }
+    }
+  }
+}
+
+// Run
+await sendData();
+await batchProcessor();
 ```
 
-### 3. Message Eviction
-Enforce SLAs by automatically evicting messages that wait too long.
+### Example 5: Scheduled Jobs
 
-**Configuration:**
 ```javascript
-await client.configure({
-  queue: 'time-sensitive',
-  options: {
-    maxWaitTimeSeconds: 60  // Evict messages older than 1 minute
-  }
+import { Queen } from 'queen-mq';
+
+const client = new Queen({
+  baseUrls: ['http://localhost:6632']
 });
-```
 
-**Environment:**
-```bash
-export EVICTION_INTERVAL=60000  # Check interval in milliseconds
+// Configure with delayed processing
+await client.queue('scheduled-jobs', {
+  delayedProcessing: 3600,  // 1 hour delay
+  priority: 5
+});
+
+// Schedule a job
+async function scheduleReport() {
+  await client.push('scheduled-jobs/daily-reports', {
+    reportType: 'daily-sales',
+    date: new Date().toISOString().split('T')[0],
+    recipients: ['manager@company.com'],
+    scheduledAt: Date.now()
+  });
+  
+  console.log('Report scheduled for processing in 1 hour');
+}
+
+// Process scheduled jobs
+async function processScheduledJobs() {
+  for await (const job of client.take('scheduled-jobs/daily-reports', {
+    wait: true
+  })) {
+    try {
+      console.log('Generating report:', job.data.reportType);
+      await generateReport(job.data);
+      await client.ack(job);
+    } catch (error) {
+      await client.ack(job, false, { error: error.message });
+    }
+  }
+}
+
+await scheduleReport();
+processScheduledJobs().catch(console.error);
 ```
 
-### Combined Example
+### Example 6: Rate Limiting
+
 ```javascript
-await client.configure({
-  queue: 'production-queue',
-  options: {
-    // Encryption
-    encryptionEnabled: true,
-    
-    // Retention
-    retentionSeconds: 86400,
-    completedRetentionSeconds: 3600,
-    retentionEnabled: true,
-    
-    // Eviction
-    maxWaitTimeSeconds: 600,
+import { Queen } from 'queen-mq';
+
+const client = new Queen({
+  baseUrls: ['http://localhost:6632']
+});
+
+await client.queue('api-calls', { 
+  priority: 5,
+  leaseTime: 60
+});
+
+// Producer: Queue API calls
+async function queueApiCalls(calls) {
+  await client.push('api-calls', calls);
+}
+
+// Consumer: Rate-limited processor (10 calls per second max)
+async function rateLimitedProcessor() {
+  const RATE_LIMIT = 10;  // calls per second
+  const INTERVAL = 1000;  // 1 second
+  
+  let callsThisInterval = 0;
+  let intervalStart = Date.now();
+  
+  for await (const call of client.take('api-calls', { wait: true })) {
+    // Check if we need to wait
+    if (callsThisInterval >= RATE_LIMIT) {
+      const elapsed = Date.now() - intervalStart;
+      if (elapsed < INTERVAL) {
+        await new Promise(r => setTimeout(r, INTERVAL - elapsed));
+      }
+      callsThisInterval = 0;
+      intervalStart = Date.now();
+    }
     
-    // Standard options
-    priority: 10,
-    leaseTime: 300
+    try {
+      await makeApiCall(call.data);
+      await client.ack(call);
+      callsThisInterval++;
+    } catch (error) {
+      await client.ack(call, false, { error: error.message });
+    }
   }
-});
+}
+
+// Generate calls
+const calls = Array.from({ length: 100 }, (_, i) => ({
+  id: i,
+  endpoint: '/api/data',
+  method: 'GET'
+}));
+
+await queueApiCalls(calls);
+rateLimitedProcessor().catch(console.error);
 ```
 
-## ⚙️ Configuration
+### Example 7: Enterprise Features
 
-### Environment Variables
+```javascript
+import { Queen } from 'queen-mq';
 
-All configuration values have sensible defaults and can be overridden using environment variables. Configuration is centralized in `src/config.js`.
+const client = new Queen({
+  baseUrls: ['http://localhost:6632']
+});
 
-#### Server Configuration
+// Configure with all enterprise features
+await client.queue('production-queue', {
+  // Encryption
+  encryptionEnabled: true,
+  
+  // Retention
+  retentionSeconds: 86400,              // Delete pending after 24 hours
+  completedRetentionSeconds: 3600,      // Delete completed after 1 hour
+  retentionEnabled: true,
+  
+  // Eviction (SLA enforcement)
+  maxWaitTimeSeconds: 600,              // Evict messages older than 10 minutes
+  
+  // Standard options
+  priority: 10,
+  leaseTime: 300,
+  retryLimit: 3,
+  dlqAfterMaxRetries: true
+});
 
-```bash
-# Server basics
-PORT=6632                      # Server port (default: 6632)
-HOST=0.0.0.0                   # Server host (default: 0.0.0.0)
-WORKER_ID=worker-1             # Worker identifier (default: worker-${process.pid})
-APP_NAME=queen-uws             # Application name for database connections
+// Push sensitive data (will be encrypted)
+await client.push('production-queue', {
+  userId: 123,
+  creditCard: '4111-1111-1111-1111',
+  amount: 99.99,
+  timestamp: Date.now()
+});
 
-# CORS settings
-CORS_MAX_AGE=86400             # CORS max age in seconds (default: 86400 = 24 hours)
-CORS_ALLOWED_ORIGINS=*         # Allowed origins (default: *)
-CORS_ALLOWED_METHODS=GET,POST,PUT,DELETE,OPTIONS  # Allowed methods
-CORS_ALLOWED_HEADERS=Content-Type,Authorization   # Allowed headers
+// Process (data decrypted automatically)
+for await (const message of client.take('production-queue', { wait: true })) {
+  console.log('Processing encrypted data:', message.data.userId);
+  await processPayment(message.data);
+  await client.ack(message);
+}
 ```
 
-#### Database Configuration
+---
 
-```bash
-# Connection settings
-PG_USER=postgres               # PostgreSQL user (default: postgres)
-PG_HOST=localhost              # PostgreSQL host (default: localhost)
-PG_DB=postgres                 # PostgreSQL database (default: postgres)
-PG_PASSWORD=postgres           # PostgreSQL password (default: postgres)
-PG_PORT=5432                   # PostgreSQL port (default: 5432)
+## 🧪 Testing
 
-# Connection pool settings
-DB_POOL_SIZE=20                # Max pool size (default: 20)
-DB_IDLE_TIMEOUT=30000          # Idle connection timeout in ms (default: 30000)
-DB_CONNECTION_TIMEOUT=2000     # Connection timeout in ms (default: 2000)
-DB_STATEMENT_TIMEOUT=30000     # Statement timeout in ms (default: 30000)
-DB_QUERY_TIMEOUT=30000         # Query timeout in ms (default: 30000)
-DB_MAX_RETRIES=3               # Max retry attempts for queries (default: 3)
-```
+Queen includes a comprehensive test suite covering all features.
 
-#### Queue Processing Configuration
+### Run Tests
 
 ```bash
-# Pop operation defaults
-DEFAULT_TIMEOUT=30000          # Default pop timeout in ms (default: 30000)
-MAX_TIMEOUT=60000              # Maximum pop timeout in ms (default: 60000)
-DEFAULT_BATCH_SIZE=1           # Default batch size for pop (default: 1)
-BATCH_INSERT_SIZE=1000         # Batch size for bulk inserts (default: 1000)
+# Start the server first
+npm start
 
-# Long polling
-QUEUE_POLL_INTERVAL=100        # Poll interval in ms (default: 100)
-QUEUE_POLL_INTERVAL_FILTERED=1000  # Poll interval for filtered pops (default: 1000)
+# Run all tests
+node src/test/test-new.js
 
-# Queue defaults
-DEFAULT_LEASE_TIME=300         # Default lease time in seconds (default: 300 = 5 minutes)
-DEFAULT_RETRY_LIMIT=3          # Default retry limit (default: 3)
-DEFAULT_RETRY_DELAY=1000       # Default retry delay in ms (default: 1000)
-DEFAULT_MAX_SIZE=10000         # Default max queue size (default: 10000)
-DEFAULT_TTL=3600               # Default TTL in seconds (default: 3600 = 1 hour)
-DEFAULT_PRIORITY=0             # Default queue priority (default: 0)
-DEFAULT_DELAYED_PROCESSING=0   # Default delayed processing in seconds (default: 0)
-DEFAULT_WINDOW_BUFFER=0        # Default window buffer in seconds (default: 0)
+# Run specific test categories
+node src/test/test-new.js core        # Core features
+node src/test/test-new.js partition   # Partition locking
+node src/test/test-new.js enterprise  # Enterprise features
+node src/test/test-new.js bus         # Bus mode
+node src/test/test-new.js edge        # Edge cases
+node src/test/test-new.js advanced    # Advanced patterns
 
-# Dead Letter Queue
-DEFAULT_DLQ_ENABLED=false      # Enable DLQ by default (default: false)
-DEFAULT_DLQ_AFTER_MAX_RETRIES=false  # Move to DLQ after max retries (default: false)
+# Show help
+node src/test/test-new.js help
+```
 
-# Retention
-DEFAULT_RETENTION_SECONDS=0    # Default retention for all messages (default: 0 = disabled)
-DEFAULT_COMPLETED_RETENTION_SECONDS=0  # Retention for completed messages (default: 0)
-DEFAULT_RETENTION_ENABLED=false  # Enable retention by default (default: false)
+### Test Coverage
 
-# Eviction
-DEFAULT_MAX_WAIT_TIME_SECONDS=0  # Max wait time before eviction (default: 0 = disabled)
-```
+The test suite verifies:
 
-#### Background Jobs Configuration
+**Core Features:**
+- Queue creation and configuration
+- Single and batch message push
+- Message take and acknowledgment
+- Delayed processing
+- Partition FIFO ordering
+
+**Partition Locking:**
+- Lock acquisition and release
+- Bus mode partition locking
+- Specific partition locking
+- Namespace/task filtering with locking
+
+**Enterprise Features:**
+- AES-256-GCM encryption
+- Message retention policies
+- Message eviction
+- Combined enterprise features
+
+**Bus Mode:**
+- Consumer groups
+- Subscription modes (all, new, from)
+- Consumer group isolation
+- Mixed mode (queue + bus)
+
+**Edge Cases:**
+- Empty and null payloads
+- Very large payloads
+- Concurrent operations
+- Lease expiration
+- SQL injection prevention
+- XSS prevention
+
+**Advanced Patterns:**
+- Multi-stage pipelines
+- Fan-out/fan-in
+- Priority scenarios
+- Dead letter queue
+- Circuit breaker
+- Message deduplication
+- Event sourcing
 
-```bash
-# Job intervals
-LEASE_RECLAIM_INTERVAL=5000    # Lease reclamation interval in ms (default: 5000)
-RETENTION_INTERVAL=300000      # Retention check interval in ms (default: 300000 = 5 minutes)
-RETENTION_BATCH_SIZE=1000      # Retention batch size (default: 1000)
-PARTITION_CLEANUP_DAYS=7       # Days before cleaning empty partitions (default: 7)
-EVICTION_INTERVAL=60000        # Eviction check interval in ms (default: 60000 = 1 minute)
-EVICTION_BATCH_SIZE=1000       # Eviction batch size (default: 1000)
+### Test Results
 
-# WebSocket updates
-QUEUE_DEPTH_UPDATE_INTERVAL=5000    # Queue depth update interval (default: 5000)
-SYSTEM_STATS_UPDATE_INTERVAL=10000  # System stats update interval (default: 10000)
+Example output:
 ```
+🚀 Starting Queen Message Queue Test Suite
+   Using the new minimalist Queen client interface
+================================================================================
 
-#### WebSocket Configuration
+📦 CORE FEATURES
+----------------------------------------
+✅ Queue Creation Policy
+✅ Single Message Push
+✅ Batch Message Push
+✅ Queue Configuration
+✅ Take and Acknowledgment
+✅ Delayed Processing
+✅ Partition FIFO Ordering
 
-```bash
-# WebSocket settings
-WS_COMPRESSION=0               # Compression level (default: 0 = disabled)
-WS_MAX_PAYLOAD_LENGTH=16384    # Max payload length in bytes (default: 16384 = 16KB)
-WS_IDLE_TIMEOUT=60             # Idle timeout in seconds (default: 60)
-WS_MAX_CONNECTIONS=1000        # Max concurrent connections (default: 1000)
-WS_HEARTBEAT_INTERVAL=30000    # Heartbeat interval in ms (default: 30000)
+🔒 PARTITION LOCKING
+----------------------------------------
+✅ Partition Locking
+✅ Bus Partition Locking
+✅ Specific Partition Locking
+✅ Namespace Task Filtering
+✅ Namespace Task Bus Mode
+
+📈 Test Summary
+================================================================================
+Total: 42 | Passed: 42 | Failed: 0 | Duration: 45.2s
 ```
 
-#### Encryption Configuration
+---
 
-```bash
-# Encryption settings
-QUEEN_ENCRYPTION_KEY=<64-hex>  # 32-byte key as 64 hex characters
-                                # Generate with: openssl rand -hex 32
-                                # Required for encryption features
-```
+## 🤝 Contributing
 
-#### Client SDK Configuration
+We welcome contributions! Here's how to get started:
 
-```bash
-# Client defaults
-QUEEN_BASE_URL=http://localhost:6632  # Default server URL
-CLIENT_RETRY_ATTEMPTS=3        # Default retry attempts (default: 3)
-CLIENT_RETRY_DELAY=1000        # Default retry delay in ms (default: 1000)
-CLIENT_RETRY_BACKOFF=2         # Retry backoff multiplier (default: 2)
-CLIENT_POOL_SIZE=10            # Client connection pool size (default: 10)
-CLIENT_REQUEST_TIMEOUT=30000   # Request timeout in ms (default: 30000)
-```
+1. **Fork the repository**
+2. **Create a feature branch**: `git checkout -b feature/amazing-feature`
+3. **Make your changes**
+4. **Run the test suite**: `node src/test/test-new.js`
+5. **Commit your changes**: `git commit -m 'Add amazing feature'`
+6. **Push to the branch**: `git push origin feature/amazing-feature`
+7. **Open a Pull Request**
 
-#### API Configuration
+### Development Setup
 
 ```bash
-# Pagination
-API_DEFAULT_LIMIT=100          # Default page size (default: 100)
-API_MAX_LIMIT=1000             # Maximum page size (default: 1000)
-API_DEFAULT_OFFSET=0           # Default offset (default: 0)
-```
+# Clone your fork
+git clone https://github.com/your-username/queen
+cd queen
 
-#### Analytics Configuration
+# Install dependencies
+nvm use 22
+npm install
 
-```bash
-# Analytics settings
-ANALYTICS_RECENT_HOURS=24      # Hours to consider for recent stats (default: 24)
-ANALYTICS_MIN_COMPLETED=5      # Min completed messages for stats (default: 5)
-RECENT_MESSAGE_WINDOW=60       # Recent message window in seconds (default: 60)
-RELATED_MESSAGE_WINDOW=3600    # Related message window in seconds (default: 3600)
-MAX_RELATED_MESSAGES=10        # Max related messages to return (default: 10)
-```
+# Initialize database
+node init-db.js
 
-#### Monitoring Configuration
+# Start server
+npm start
 
-```bash
-# Performance monitoring
-ENABLE_REQUEST_COUNTING=true   # Enable request counting (default: true)
-ENABLE_MESSAGE_COUNTING=true   # Enable message counting (default: true)
-METRICS_ENDPOINT_ENABLED=true  # Enable /metrics endpoint (default: true)
-HEALTH_CHECK_ENABLED=true      # Enable /health endpoint (default: true)
+# Run tests
+node src/test/test-new.js
 ```
 
-#### Logging Configuration
+### Code Style
 
-```bash
-# Logging settings
-ENABLE_LOGGING=true            # Enable logging (default: true)
-LOG_LEVEL=info                 # Log level (default: info)
-LOG_FORMAT=json                # Log format (default: json)
-LOG_TIMESTAMP=true             # Include timestamps (default: true)
-```
+- Use ES6+ features
+- Follow existing code style
+- Add comments for complex logic
+- Write tests for new features
 
-### Queue Options
+---
 
-```javascript
-{
-  // Standard Options
-  "leaseTime": 300,           // Seconds before message lease expires
-  "retryLimit": 3,            // Maximum retry attempts
-  "priority": 0,              // Queue/partition priority (higher = first)
-  "delayedProcessing": 0,     // Delay in seconds before message is available
-  "windowBuffer": 0,          // Buffer time in seconds for batching
-  "dlqAfterMaxRetries": true, // Move to dead letter queue after max retries
-  
-  // Encryption (Queue-level)
-  "encryptionEnabled": false,  // Enable AES-256-GCM encryption for this queue
-  
-  // Retention (Partition-level)
-  "retentionSeconds": 0,              // Delete pending messages after X seconds (0 = disabled)
-  "completedRetentionSeconds": 0,     // Delete completed/failed messages after X seconds
-  "partitionRetentionSeconds": 0,     // Delete empty partitions after X seconds
-  "retentionEnabled": false,          // Enable retention for this partition
-  
-  // Eviction (Queue-level)
-  "maxWaitTimeSeconds": 0     // Evict messages older than X seconds (0 = disabled)
-}
-```
+## 📄 License
 
-## 🧪 Testing
+Apache License 2.0 - see [LICENSE.md](LICENSE.md) for details.
 
-### Run Core Feature Tests
+---
 
-```bash
-# Start the server
-npm start
+## 🔗 Links
 
-# Run comprehensive test suite
-node src/test/core-features-test.js
+- **Repository**: [github.com/smartpricing/queen](https://github.com/smartpricing/queen)
+- **Documentation**: See `docs/` directory
+- **Issues**: [GitHub Issues](https://github.com/smartpricing/queen/issues)
+- **API Reference**: [API.md](API.md)
 
-# Run full test suite (more detailed)
-node src/test/comprehensive-test.js
-```
+---
 
-### Test Results
+## 📈 Performance
 
-The test suite verifies:
-- ✅ Single and batch message push
-- ✅ Queue configuration and options
-- ✅ Pop operations (specific partition and queue-level)
-- ✅ Delayed processing (2+ second delays)
-- ✅ Partition priority ordering
-- ✅ Consumer pattern with automatic acknowledgment
-- ✅ Message acknowledgment and retry logic
-- ✅ FIFO ordering within partitions
+**Benchmarks** (PostgreSQL 16, Node.js 22):
+- **Throughput**: 10,000+ messages/second
+- **Latency**: < 10ms for immediate pop operations
+- **Concurrent Connections**: 1,000+ long polling connections
+- **Database**: Optimized with proper indexing and connection pooling
 
-## 🤝 Contributing
+**Optimization Features:**
+- Connection pooling with configurable size
+- Resource caching for queue/partition lookups
+- Batch operations for bulk inserts/updates
+- Optimized SQL queries with proper indexes
+- Event-driven architecture for minimal polling overhead
+- Long polling for real-time message delivery
 
-1. Fork the repository
-2. Create a feature branch
-3. Make your changes
-4. Run the test suite
-5. Submit a pull request
+---
 
-## 📄 License
+## 🎯 Roadmap
 
-MIT License - see LICENSE file for details.
+- [ ] **Horizontal Scaling**: Better support for multiple server instances
+- [ ] **Message Scheduling**: Cron-like scheduling for recurring jobs
+- [ ] **Priority Lanes**: Dynamic priority adjustment based on load
+- [ ] **Metrics Export**: Prometheus/Grafana integration
+- [ ] **Admin API**: REST API for queue management
+- [ ] **Client Libraries**: Python, Go, Java clients
+- [ ] **Message Tracing**: Distributed tracing integration
+- [ ] **Queue Templates**: Pre-configured queue patterns
+- [ ] **GraphQL API**: Alternative to REST API
+- [ ] **Kubernetes Operator**: Native K8s support
 
 ---
 
-**Queen Message Queue System** - Built for performance, reliability, and scalability. 🚀
+<div align="center">
+
+**Queen Message Queue System** - Built for performance, reliability, and developer happiness 🚀
+
+Made with ❤️ by [Smartpricing](https://github.com/smartpricing)
+
+</div>