queen-mq 0.6.3 → 0.7.4

package/README.md

@@ -1,459 +1,632 @@
-# Queen MQ - PostgreSQL-backed C++ Message Queue
+# Queen MQ - JavaScript Client
 
 <div align="center">
 
-**A modern, performant message queue system built on PostgreSQL**
+**Modern, high-performance message queue client for Node.js**
 
+[![npm](https://img.shields.io/npm/v/queen-mq.svg)](https://www.npmjs.com/package/queen-mq)
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE.md)
 [![Node](https://img.shields.io/badge/node-%3E%3D22.0.0-brightgreen.svg)](https://nodejs.org/)
-[![C++](https://img.shields.io/badge/C%2B%2B-17-blue.svg)](https://en.cppreference.com/w/cpp/17)
 
-[Quick Start](#one-single-example) • [Complete V2 Guide](client-js/client-v2/README.md) • [Webapp](#webapp) • [Server Setup](#install-the-server-and-configure-it) • [HTTP API](#raw-http-api)
-
-<p align="center">
-  <img src="assets/queen-logo.svg" alt="Queen Logo" width="120" />
-</p>
+[Quick Start](#quick-start) • [Complete Guide](client-v2/README.md) • [Examples](#examples) • [API Reference](#api-reference)
 
 </div>
 
 ---
 
-Why "Queen"? Because years ago, when I first read the word "queue" in my mind, I read it as "queen".
+## What is Queen MQ?
+
+Queen MQ is a PostgreSQL-backed message queue system with a powerful feature set:
+
+- **FIFO Partitions** - Unlimited ordered partitions within queues
+- **Consumer Groups** - Kafka-style consumer groups for scalability
+- **Flexible Semantics** - Exactly-once, at-least-once, and at-most-once delivery
+- **Transactions** - Atomic operations across push and ack
+- **High Performance** - 200K+ messages/sec with proper batching
+- **Subscription Modes** - Process from beginning, new messages only, or from timestamp
+- **Dead Letter Queue** - Automatic failure handling and monitoring
+- **Message Tracing** - Debug distributed workflows with trace timelines
+- **Client-Side Buffering** - 10x-100x throughput boost for high-volume pushes
+- **Real-time Streaming** - Windowed aggregation and processing
+
+This client provides a fluent, promise-based API for Node.js applications.
 
 ---
 
-[QUICKSTART](docs/QUICKSTART.md)
+## Installation
 
-Latest server production version is **0.6.3**.
+```bash
+npm install queen-mq
+```
+
+**Requirements:** Node.js 22+
 
 ---
 
-## Introduction
-
-QueenMQ is a queue system written in C++ and backed by PostgreSQL, born from the need to manage many FIFO partitions for Smartchat with solid guarantees around delivery and failure handling. During the initial development, I realized that with a few simple additions to the original design, I could build a very powerful and flexible queue system. This project is almost entirely written by AI, with my supervision—only the test files (or a good part of them) are manually written.
-
-Here are the main features:
-- Unlimited FIFO partitions within queues
-- Queue semantics like RabbitMQ
-- Consumer groups over queues like Kafka
-- Allows for all the patterns you need, from simple queues to complex workflows and request/response patterns
-- QoS levels: Exactly-once delivery (with transactionId), at-least-once delivery, and at-most-once delivery
-- Subscription modes for replay (new messages only or from a specific timestamp) and message history control
-- Transactions between operations (push and ack mainly) for atomicity
-- Dead letter queue for failure handling
-- Lease renewal for long-running tasks
-- Message tracing for debugging workflows
-- Encryption of messages at DB level
-- Automatic message retention and cleanup - Configurable per-queue retention policies
-- Streaming capabilities for real-time aggregation and processing of messages
-- A nice webapp for monitoring and managing the system
-- Maintenace mode that allows to continue pushing messages even when the database is down, and to drain the messages to the database when the maintenance mode is disabled
-
-The system consists of a PostgreSQL database, a replicated server that can be scaled horizontally (though it won't be the bottleneck), and a client library for interacting with the server. All client-server communication happens over HTTP, and the client library are available for JavaScript and C++ (Other languages like Python are planned). There's also a modern Vue 3 web app for monitoring and managing the system.
-
-With proper batching, the system can handle +200k **messages** per second (not req/s) on modest hardware.
-
-Main documentation:
-- [Client Guide JS](client-js/client-v2/README.md)
-- [Client Guide C++](client-cpp/README.md)
-- [Server Guide](server/README.md)
-- [Streaming Guide](docs/STREAMING_USAGE.md)
-- [API Reference](server/API.md)
-- [Message Retention & Cleanup](docs/RETENTION.md)
-- [Webapp](webapp/README.md)
-- [Expose the Webapp behind a proxy](proxy/README.md)
-
-## Concepts
-
-Although the system is designed to be simple to use (not really), there are some concepts that are important to understand to use the system effectively.
+## Quick Start
+
+```javascript
+import { Queen } from 'queen-mq'
+
+// Connect to Queen server
+const queen = new Queen('http://localhost:6632')
+
+// Create a queue
+await queen.queue('tasks').create()
+
+// Push messages
+await queen.queue('tasks').push([
+  { data: { task: 'send-email', to: 'alice@example.com' } }
+])
+
+// Consume messages
+await queen.queue('tasks').consume(async (message) => {
+  console.log('Processing:', message.data)
+  // Auto-ack on success, auto-retry on error
+})
+```
+
+---
+
+## Core Concepts
 
 ### Queues
 
-Queues are a way to organize your messages into logical groups. Each queue is like a container for messages. You can have as many queues as you want, and you can configure each queue with different settings, like the lease time, retry limit, encryption, retention, priority, delayed processing, window buffer and retention.
+Logical containers for messages with configurable settings:
+- **Lease time** - How long a consumer has to process a message
+- **Retry limit** - Number of retry attempts before DLQ
+- **Priority** - Queue priority for multi-queue consumers
+- **Encryption** - Message payload encryption at rest
+- **Retention** - Automatic cleanup policies
+
+```javascript
+await queen.queue('orders')
+  .config({
+    leaseTime: 300,        // 5 minutes
+    retryLimit: 3,
+    priority: 5,
+    encryptionEnabled: false
+  })
+  .create()
+```
 
 ### Partitions
 
-Partitions are a way to organize your messages into logical ordered groups. Each partition is like a separate queue inside a queue, and messages in the same partition are guaranteed to be processed in order. Only one consumer/client can acquire the lock for a partition at a time. You can have as many partitions as you want, and you can have as many consumers/clients as you want. If the lease in not acknowledged in time, the message is released and can be acquired by another consumer/client. Use renewLease() to renew the lease during long-running tasks. Each different consumer group can acquire the lock for a partition independently, so you can have as many consumer groups as you want.
+Ordered lanes within a queue. Messages in the same partition are processed sequentially:
 
-### Default partition and consumer group
+```javascript
+// All messages for user-123 are processed in order
+await queen.queue('user-events')
+  .partition('user-123')
+  .push([
+    { data: { event: 'login' } },
+    { data: { event: 'view-page' } },
+    { data: { event: 'logout' } }
+  ])
+```
 
-The default partition (if not specified) is `Default`. If you don't specify a consumer group, the messages are processed in queue mode, that is creating a default consumer group named `__QUEUE_MODE__`. Each queue can be used at the same time by multiple consumer groups, and each consumer group can process messages from multiple partitions.
+**Use cases:**
+- Per-user ordering
+- Per-tenant isolation
+- Sharding for parallelism
 
-### Consumer groups
+### Consumer Groups
 
-Consumer groups are a way to process messages for different purposes. Each consumer group is like a separate queue, and messages in the same consumer group partition are guaranteed to be processed in order. Each consumer group tracks its own position in the queue. You can start a consumer group from the beginning of the queue, from a specific timestamp, or only new messages.
+Multiple consumers sharing work, with independent progress tracking:
 
-### Subscription modes
+```javascript
+// Worker 1 & 2 share the load
+await queen.queue('emails')
+  .group('processors')
+  .consume(async (message) => {
+    await sendEmail(message.data)
+  })
+
+// Separate group processes same messages independently
+await queen.queue('emails')
+  .group('analytics')
+  .consume(async (message) => {
+    await logMetrics(message.data)
+  })
+```
 
-Subscription modes are a way to control the message history that is processed. You can choose to process all messages (including historical ones), only new messages, or messages from a specific timestamp. Subscription modes are only available when using consumer groups.
+### Subscription Modes
 
-**Server Default:** By default, new consumer groups process all historical messages. You can change this server-wide:
+Control whether consumer groups process historical messages:
 
-```bash
-export DEFAULT_SUBSCRIPTION_MODE="new"  # Skip historical messages by default
-./bin/queen-server
+```javascript
+// Default: Process ALL messages (including backlog)
+await queen.queue('events')
+  .group('batch-analytics')
+  .consume(async (message) => { /* all messages */ })
+
+// Skip history, only new messages
+await queen.queue('events')
+  .group('realtime-monitor')
+  .subscriptionMode('new')
+  .consume(async (message) => { /* new only */ })
+
+// Start from specific timestamp
+await queen.queue('events')
+  .group('replay')
+  .subscriptionFrom('2025-10-28T10:00:00.000Z')
+  .consume(async (message) => { /* from timestamp */ })
 ```
 
-This is useful for real-time systems where only new messages matter, or to prevent accidental processing of large backlogs. Clients can still override with `.subscriptionMode()` if needed.
+---
 
-### Long polling (waiting for messages)
+## Connection Options
 
-The client works with the pull model for pop operations, meaning that you need to explicitly request messages from the queue. Pop and consume mehtods can "wait" server side for messages to be available. When the method is called with wait=true, the method will block until messages are available or the timeout is reached. When the timeout is reached, the method returns an empty array. Long polling is a very efficient way to wait for messages, and it is the recommended way to consume messages.
+### Single Server
 
-### Lease renewal
+```javascript
+const queen = new Queen('http://localhost:6632')
+```
 
-Lease renewal is a way to keep the lock for a partition or consumer group alive. You can use lease renewal to prevent the lock from expiring and being acquired by another consumer/client.
+### Multiple Servers (High Availability)
 
-### Ack and Nack
+```javascript
+const queen = new Queen([
+  'http://server1:6632',
+  'http://server2:6632'
+])
+```
 
-Ack and Nack are the way to acknowledge or not a message. Ack means that the message has been processed successfully, and Nack means that the message has not been processed successfully. Ack/Nack requires the partitionId, the transactionId and the leaseId to be specified. If the ack is coming from a consumer group, the consumer group name is also required. This logic is handled automatically by the client library, you don't need to worry about it. If the message has already been acknowledged, the ack will be ignored. Based on the configuration, the message can be automatically acknowledged or manually acknowledged.
+### Full Configuration
 
-### Transactions
+```javascript
+const queen = new Queen({
+  urls: ['http://server1:6632', 'http://server2:6632'],
+  timeoutMillis: 30000,
+  retryAttempts: 3,
+  loadBalancingStrategy: 'round-robin',  // or 'session'
+  enableFailover: true
+})
+```
 
-Transactions are a way to ensure that a group of operations are atomic. You can use transactions to ensure that a group of operations are executed together, and if one of the operations fails, the entire transaction is rolled back. Transactions are useful to achieve the exactly-once guarantee. You can process a message on a queue, and forward to the next queue only if the consumer lease is still valid, concatenating ack and push into a single transaction.
+---
 
-### Dead letter queue
+## Basic Usage Patterns
 
-The dead letter queue is a way to handle messages that are not processed successfully. When a message is not processed successfully, it is moved to the dead letter queue. Messages goes in the DLQ when they are NACK after the retry limit is reached. You configure the retry limit for each queue at queue config.
+### Push Messages
 
-## Comparison with RabbitMQ, Kafka, and NATS
+```javascript
+// Simple push
+await queen.queue('tasks').push([
+  { data: { job: 'resize-image', imageId: 123 } }
+])
 
-For users familiar with existing message queue systems, here's how Queen's semantics and usage patterns compare:
+// With partition
+await queen.queue('tasks')
+  .partition('tenant-456')
+  .push([{ data: { action: 'process' } }])
+
+// With custom transaction ID (for exactly-once)
+await queen.queue('tasks').push([
+  {
+    transactionId: 'unique-id-123',
+    data: { value: 42 }
+  }
+])
+```
 
-### Conceptual Mapping
+### Consume Messages (Long-Running Workers)
 
-| Queen Concept | RabbitMQ Equivalent | Kafka Equivalent | NATS Equivalent |
-|---------------|---------------------|------------------|-----------------|
-| Queue | Queue | Topic | Stream (JetStream) |
-| Partition | N/A (queues are single-consumer by default) | Partition | N/A |
-| Consumer Group | Competing Consumers pattern | Consumer Group | Queue Group |
-| Queue Mode (no group) | Exclusive consumer | N/A (always uses groups) | Single subscriber |
-| Lease | Message TTL / Visibility timeout | N/A (commit-based) | Ack wait / nak delay |
-| Ack/Nack | Ack/Nack | Commit offset | Ack/Nak |
-| Transaction | Publisher confirms + consumer acks | Transactional producer/consumer | N/A |
-| Dead Letter Queue | Dead Letter Exchange | N/A (manual) | N/A (manual) |
+```javascript
+// Runs forever, processes messages as they arrive
+await queen.queue('tasks')
+  .concurrency(10)        // 10 parallel workers
+  .batch(20)              // Fetch 20 at a time
+  .consume(async (message) => {
+    await processTask(message.data)
+    // Auto-ack on success, auto-retry on error
+  })
+
+// Process with limit and stop
+await queen.queue('tasks')
+  .limit(100)
+  .consume(async (message) => {
+    await processTask(message.data)
+  })
+```
 
+### Pop Messages (On-Demand Processing)
 
-## One single example
+```javascript
+// Grab messages manually
+const messages = await queen.queue('tasks')
+  .batch(10)
+  .wait(true)  // Long polling
+  .pop()
+
+// Manual acknowledgment
+for (const message of messages) {
+  try {
+    await processMessage(message.data)
+    await queen.ack(message, true)  // Success
+  } catch (error) {
+    await queen.ack(message, false)  // Retry
+  }
+}
+```
 
-> 📖 **[Complete Guide: client-js/client-v2/README.md](client-js/client-v2/README.md)** - Full tutorial with all features!
+### Transactions (Atomic Operations)
 
 ```javascript
-import { Queen } from 'queen-mq'
+// Pop from queue A
+const messages = await queen.queue('input').pop()
 
-// Connect to Queen
-const queen = new Queen('http://localhost:6632')
+// Atomically: ack input AND push output
+await queen.transaction()
+  .ack(messages[0])
+  .queue('output')
+  .push([{ data: processedResult }])
+  .commit()
 
-// Create a queue
-await queen
-.queue('critical-task')
-.config({
-  leaseTime: 10, // 10 seconds to process the messages (seconds)
-})
-.create()
+// If commit fails, nothing happens - message stays in input queue
+```
 
-// Push messages
-await queen
-.queue('critical-task')
-.partition('tenant-123')
-.push([
-  { 
-    transactionId: 'my-id', // This is autogenerated, but you can set your own. Unique per queue partition.
-    data: { id: 123, description: 'Critical task' }  // The message payload
-  },
-])
-.onSuccess(async (messages) => { // Not mandatory
-  console.log('Messages pushed successfully:', messages)
-})
-.onDuplicate(async (messages) => { // Not mandatory, triggered when a message with the same transactionId is pushed
-  console.warn('Duplicate transaction IDs detected')
-})
-.onError(async (messages, error) => {  // Without callbacks, push throws an error if some messages are not pushed
-  console.error('Error pushing messages:', error)
-})
+### Client-Side Buffering (High Throughput)
 
-// Consume messages
-await queen
-.queue('critical-task')
-// The consumer group name, without this, the messages are processed in queue mode
-.group('processor-consumer-group') 
-// 10 parallel workers
-.concurrency(10)
-// I want to manually ack/nack messages 
-.autoAck(false) 
-// 10 messages per batch, prefetch them
-.batch(10) 
-// Auto-renew the lease for the messages every 2 seconds
-.renewLease(true, 2000) 
-// Process each message individually
-.each() 
-// Do your work here
-.consume(async (message) => {
-  console.log('Processing:', message.data) 
-})
-// Ack the messages if you processed them successfully
-.onSuccess(async (message) => {
-  await queen
-  .transaction()
-  .queue('critical-task-next')
-  .partition('XXX')
-  .push([{ data: { message: 'Final', count: 3 } }])
-  .ack(message)
-  .commit()  
-})
-// Nack the messages if you failed to process them
-.onError(async (message, error) => {
-  console.error('Error processing messages:', error)
-  await queen.ack(message, false)
-})
+```javascript
+// Buffer messages locally, batch to server
+for (let i = 0; i < 10000; i++) {
+  await queen.queue('events')
+    .buffer({ messageCount: 500, timeMillis: 1000 })
+    .push([{ data: { id: i } }])
+}
+
+// Flush remaining buffered messages
+await queen.flushAllBuffers()
+
+// Result: 10x-100x faster than individual pushes
 ```
 
-## Webapp
+### Dead Letter Queue
 
-A modern Vue 3 web interface for managing and monitoring Queen MQ. 
+```javascript
+// Enable DLQ on queue
+await queen.queue('risky')
+  .config({ retryLimit: 3, dlqAfterMaxRetries: true })
+  .create()
+
+// Query failed messages
+const dlq = await queen.queue('risky')
+  .dlq()
+  .limit(10)
+  .get()
+
+console.log(`Found ${dlq.total} failed messages`)
+for (const msg of dlq.messages) {
+  console.log('Error:', msg.errorMessage)
+}
+```
 
-![Queen MQ Dashboard](./assets/dashboard.png)
+### Message Tracing
 
-![Queen MQ Queues](./assets/queues.png) 
+```javascript
+await queen.queue('orders').consume(async (msg) => {
+  const orderId = msg.data.orderId
+  
+  // Record trace with name for cross-service correlation
+  await msg.trace({
+    traceName: `order-${orderId}`,
+    eventType: 'info',
+    data: { text: 'Order processing started' }
+  })
+  
+  await processOrder(msg.data)
+  
+  await msg.trace({
+    traceName: `order-${orderId}`,
+    eventType: 'processing',
+    data: { 
+      text: 'Order completed',
+      total: msg.data.total
+    }
+  })
+})
 
-![Queen MQ Messages](./assets/messages.png) 
+// View traces in webapp: Traces → Search "order-12345"
+```
 
-![Queen MQ Traces](./assets/traces.png) 
+---
 
-![Queen MQ Analytics](./assets/analytics.png) 
+## Examples
 
-![Queen MQ System Metrics](./assets/systemmetrics.png) 
+### Complete Pipeline with Consumer Groups
 
-**Features:**
-- 📊 Real-time dashboard with system metrics
-- 📈 Message throughput visualization
-- 🔍 Queue management and monitoring
-- 👥 Consumer group tracking
-- 💬 Message browser with trace timeline
-- 🔎 **Trace explorer for debugging distributed workflows**
-- 📉 Analytics and insights
-- 🌓 Dark/light theme support
+```javascript
+import { Queen } from 'queen-mq'
 
-**Quick Start:**
-```bash
-cd webapp
-npm install
-npm run dev
+const queen = new Queen('http://localhost:6632')
+
+// Stage 1: Ingest with buffering
+async function ingestEvents() {
+  for (let i = 0; i < 10000; i++) {
+    await queen.queue('raw-events')
+      .partition(`user-${i % 100}`)
+      .buffer({ messageCount: 500, timeMillis: 1000 })
+      .push([{ data: { userId: i % 100, event: 'page_view' } }])
+  }
+  await queen.flushAllBuffers()
+}
+
+// Stage 2: Process with transactions
+async function processEvents() {
+  await queen.queue('raw-events')
+    .group('processors')
+    .concurrency(5)
+    .batch(10)
+    .autoAck(false)
+    .consume(async (messages) => {
+      const results = messages.map(m => process(m.data))
+      
+      // Atomic: ack all inputs, push all outputs
+      const txn = queen.transaction()
+      for (const msg of messages) txn.ack(msg)
+      txn.queue('processed-events').push(results.map(r => ({ data: r })))
+      await txn.commit()
+    })
+}
+
+// Stage 3: Separate analytics consumer (fan-out)
+async function analytics() {
+  await queen.queue('raw-events')
+    .group('analytics')
+    .subscriptionMode('new')  // Skip backlog
+    .consume(async (message) => {
+      await logMetrics(message.data)
+    })
+}
+
+await ingestEvents()
+await Promise.all([processEvents(), analytics()])
 ```
 
-The dashboard will be available at `http://localhost:4000` or at `http://localhost:6632` directly from the server.
+### Long-Running Tasks with Lease Renewal
 
-See [webapp/README.md](webapp/README.md) for more details.
+```javascript
+await queen.queue('video-processing')
+  .renewLease(true, 60000)  // Renew every 60 seconds
+  .consume(async (message) => {
+    // Can take hours - lease keeps renewing automatically
+    await processVideo(message.data)
+  })
+```
 
-## Architecture
+### Error Handling with Callbacks
 
-Queen uses a high-performance **acceptor/worker pattern** with uWebSockets, featuring a **fully asynchronous, non-blocking PostgreSQL architecture** for maximum throughput and minimal latency.
+```javascript
+await queen.queue('tasks')
+  .autoAck(false)
+  .consume(async (message) => {
+    return await riskyOperation(message.data)
+  })
+  .onSuccess(async (message, result) => {
+    console.log('Success:', result)
+    await queen.ack(message, true)
+  })
+  .onError(async (message, error) => {
+    console.error('Failed:', error.message)
+    
+    // Custom retry logic
+    if (error.message.includes('temporary')) {
+      await queen.ack(message, false)  // Retry
+    } else {
+      await queen.ack(message, 'failed', { error: error.message })
+    }
+  })
+```
 
-### Core Components
+---
 
-**Network Layer:**
-- **UWS Acceptor**: Single thread listening on port 6632, distributes connections round-robin to workers
-- **UWS Workers**: Configurable event loop threads (default: 10) handling HTTP routes and WebSocket connections
+## API Reference
 
-**Database Layer:**
-- **AsyncDbPool**: Non-blocking PostgreSQL connection pool (142 connections) using libpq async API
-  - Socket-based I/O with `select()` for non-blocking operations
-  - RAII-based resource management with automatic connection cleanup
-  - Connection health monitoring and automatic reset
-  - Thread-safe with mutex/condition variable synchronization
-- **AsyncQueueManager**: Event-loop-based queue operations
-  - Direct execution in worker threads for PUSH, POP, ACK, and TRANSACTION operations
-  - Batch processing with dynamic sizing
-  - Encryption support with status checks
-  - Automatic failover to file buffer when database unavailable
+### Queue Operations
 
-**Background Services:**
-- **Poll Workers**: 4 dedicated threads for long-polling operations
-  - Non-blocking I/O with exponential backoff (100ms→2000ms)
-  - Intention registry for efficient request grouping
-  - Rate-limited queries to prevent database overload
-- **Background Pool**: 8 connections for metrics, retention, eviction, and stream management
+```javascript
+// Create
+await queen.queue('my-queue').create()
+await queen.queue('my-queue').config({ priority: 5 }).create()
 
-### Request Flow
+// Delete
+await queen.queue('my-queue').delete()
 
-**Standard Operations (PUSH/POP/ACK/TRANSACTION):**
+// Get info
+const info = await queen.getQueueInfo('my-queue')
 ```
-Client Request
-    ↓
-Acceptor (port 6632)
-    ↓
-Worker (event loop) → AsyncQueueManager → AsyncDbPool → PostgreSQL
-    ↓                  (non-blocking)      (socket I/O)
-Response sent immediately
+
+### Push
+
+```javascript
+await queen.queue('q').push([{ data: { value: 1 } }])
+await queen.queue('q').partition('p1').push([{ data: { value: 1 } }])
+await queen.queue('q').buffer({ messageCount: 100, timeMillis: 1000 }).push([...])
 ```
 
-**Long-Polling Operations (wait=true):**
+### Pop
+
+```javascript
+const msgs = await queen.queue('q').pop()
+const msgs = await queen.queue('q').batch(10).pop()
+const msgs = await queen.queue('q').batch(10).wait(true).pop()
 ```
-Client Request
-    ↓
-Worker registers intention in Registry
-    ↓
-Poll Worker (50ms interval)
-    ↓
-Non-blocking query via AsyncDbPool
-    ↓
-Messages distributed to waiting clients
+
+### Consume
+
+```javascript
+await queen.queue('q').consume(async (msg) => { /* process */ })
+await queen.queue('q').limit(10).consume(async (msg) => { /* process */ })
+await queen.queue('q').concurrency(5).consume(async (msg) => { /* 5 workers */ })
+await queen.queue('q').group('my-group').consume(async (msg) => { /* consumer group */ })
 ```
 
-### Performance Characteristics
+### Acknowledgment
 
-**Latency:**
-- **POP (immediate)**: 10-50ms
-- **ACK**: 10-50ms
-- **TRANSACTION**: 50-200ms
-- **Long-polling**: Configurable (50ms-2000ms backoff)
+```javascript
+await queen.ack(message, true)   // Success
+await queen.ack(message, false)  // Retry
+await queen.ack(message, false, { error: 'reason' })
+await queen.ack([msg1, msg2], true)  // Batch ack
+```
 
-**Throughput:**
-- **Peak**: 148,000+ msg/s
-- **Sustained**: 130,000+ msg/s
-- **Batch push**: 5,000-8,000 msg/s (with batches of 100)
+### Transactions
 
-**Resource Usage:**
-- **Database connections**: 150 total (142 async + 8 background)
-- **Threads**: 14 total (10 workers + 4 poll workers)
-- **Memory**: ~80MB for thread stacks + connection overhead
+```javascript
+await queen.transaction()
+  .ack(message)
+  .queue('output')
+  .push([{ data: { result: 'processed' } }])
+  .commit()
+```
 
-### Scalability
+### Lease Renewal
 
-The event-driven architecture enables:
-- ✅ Unlimited concurrent requests (limited only by connection pool)
-- ✅ Horizontal scaling (multiple server instances)
-- ✅ Efficient resource utilization (non-blocking I/O)
-- ✅ Low latency under high load
-- ✅ Automatic load distribution across workers
+```javascript
+await queen.renew(message)
+await queen.renew([msg1, msg2, msg3])
+await queen.queue('q').renewLease(true, 60000).consume(async (msg) => { /* auto-renew */ })
+```
 
-**📚 Technical Documentation:**
-- [Server Architecture Guide](server/README.md) - Complete server setup and configuration
-- [Architecture Diagrams](assets/architecture.svg) - Visual architecture overview
+### Buffering
 
-### PostgreSQL Failover
+```javascript
+await queen.flushAllBuffers()
+await queen.queue('q').flushBuffer()
+const stats = queen.getBufferStats()
+```
 
-Queen automatically buffers messages to disk when PostgreSQL is unavailable - **zero message loss**:
+### Dead Letter Queue
 
-- Normal pushes go directly to PostgreSQL (FIFO preserved)
-- If PostgreSQL is down, messages buffered to file (macOS: `/tmp/queen`, Linux: `/var/lib/queen/buffers`)
-- Automatic replay when PostgreSQL recovers
-- Survives server crashes and restarts
-- Directory auto-created on first run
+```javascript
+const dlq = await queen.queue('q').dlq().limit(10).get()
+const dlq = await queen.queue('q').dlq('consumer-group').limit(10).get()
+const dlq = await queen.queue('q').dlq().from('2025-01-01').to('2025-01-31').get()
+```
 
-**No configuration needed** - failover is automatic!
+### Shutdown
 
-**Custom directory:**
-```bash
-FILE_BUFFER_DIR=/custom/path ./bin/queen-server
+```javascript
+await queen.close()  // Flush buffers and close connections
 ```
 
-## Performance Benchmarks
+---
+
+## Configuration Defaults
 
-**Preliminary results from C++ client benchmark** (detailed benchmarks on dedicated hardware coming soon)
+### Client Defaults
 
-### Test Environment
-- **Hardware:** Apple M4 Air (all components on same machine, 10 processors available)
-- **Server:** 1 server, 4 workers, 95 total DB connections/threads
-- **Database:** PostgreSQL in Docker
-- **Client:** C++ benchmark tool (`benchmark/bin/benchmark`)
+```javascript
+{
+  timeoutMillis: 30000,
+  retryAttempts: 3,
+  retryDelayMillis: 1000,
+  loadBalancingStrategy: 'round-robin',
+  enableFailover: true
+}
+```
 
-### Results
+### Queue Defaults
 
-All tests run with: `--threads 10 --partitions 10 --mode single-queue`
+```javascript
+{
+  leaseTime: 300,         // 5 minutes
+  retryLimit: 3,
+  priority: 0,
+  delayedProcessing: 0,
+  windowBuffer: 0,
+  maxSize: 0,            // Unlimited
+  retentionSeconds: 0,   // Keep forever
+  encryptionEnabled: false
+}
+```
 
-| Test | Mode | Messages | Batch Size | Throughput | Bandwidth |
-|------|------|----------|------------|------------|-----------|
-| **T1** | Producer | 10,000 | 1 | 785 msg/sec | 0.22 MB/sec |
-| **T1** | Consumer | 10,000 | 1 | 456 msg/sec | 0.23 MB/sec |
-| **T2** | Producer | 10,000 | 10 | 7,677 msg/sec | 2.20 MB/sec |
-| **T2** | Consumer | 10,000 | 10 | 4,989 msg/sec | 2.53 MB/sec |
-| **T3** | Producer | 10,000 | 100 | 39,065 msg/sec | 11.18 MB/sec |
-| **T3** | Consumer | 10,000 | 100 | 30,079 msg/sec | 15.26 MB/sec |
-| **T4** | Producer | 10,000 | 1,000 | 85,862 msg/sec | 24.57 MB/sec |
-| **T4** | Consumer | 10,000 | 1,000 | **488,650 msg/sec** | **247.87 MB/sec** |
-| **T5** | Producer | 100,000 | 1,000 | **90,601 msg/sec** | **25.92 MB/sec** |
-| **T5** | Consumer | 100,000 | 1,000 | 84,530 msg/sec | 42.96 MB/sec |
+### Consume Defaults
 
-### Key Observations
+```javascript
+{
+  concurrency: 1,
+  batch: 1,
+  autoAck: true,
+  wait: true,            // Long polling
+  timeoutMillis: 30000,
+  limit: null,           // Run forever
+  renewLease: false
+}
+```
 
-- ✅ **Batch size matters:** Larger batches (1,000) dramatically improve throughput
-- ✅ **Consumer performance:** Peaks at 488K msg/sec with batch size 1,000 (247 MB/sec bandwidth)
-- ✅ **Producer peak:** 90K msg/sec with batch size 1,000 on 100K messages
-- ⚠️ **Small batches:** Performance drops significantly with batch=1 (lock contention)
-- 📈 **Scalability:** Performance improves with larger message volumes (T4 vs T5)
+---
 
-**Note:** All timing metrics are based on processing time (excludes idle time) and measure actual message processing (first message → last message).
+## Logging
 
-### Run Your Own Benchmarks
+Enable detailed logging for debugging:
 
 ```bash
-cd benchmark
-make
-
-# Producer
-./bin/benchmark producer --threads 10 --count 1000000 --batch 1000 --partitions 100 --mode single-queue
+export QUEEN_CLIENT_LOG=true
+node your-app.js
+```
 
-# Consumer
-./bin/benchmark consumer --threads 10 --batch 1000 --partitions 100 --mode single-queue
+Example output:
+```
+[2025-10-28T10:30:45.123Z] [INFO] [Queen.constructor] {"status":"initialized","urls":1}
+[2025-10-28T10:30:45.234Z] [INFO] [QueueBuilder.push] {"queue":"tasks","partition":"Default","count":5}
 ```
 
-See [benchmark/README.md](benchmark/README.md) for detailed usage.
+---
 
-## Install the server and configure it
+## Best Practices
 
-### Quick Start
+1. ✅ **Use `consume()` for workers** - Simpler API, handles retries automatically
+2. ✅ **Use `pop()` for control** - When you need precise control over acking
+3. ✅ **Buffer for speed** - Always use buffering when pushing many messages
+4. ✅ **Partitions for order** - Use partitions when message order matters
+5. ✅ **Consumer groups for scale** - Run multiple workers in the same group
+6. ✅ **Transactions for consistency** - Use transactions for atomic operations
+7. ✅ **Enable DLQ** - Always enable DLQ in production
+8. ✅ **Renew long leases** - Use auto-renewal for long-running tasks
+9. ✅ **Graceful shutdown** - Always call `queen.close()` before exiting
+10. ✅ **Monitor DLQ** - Regularly check for failed messages
 
-```sh
-cd server
-make clean
-make deps
-make build-only
-DB_POOL_SIZE=50 ./bin/queen-server
-```
+---
 
-**📖 Complete Build & Tuning Guide:** [server/README.md](server/README.md)
+## TypeScript Support
 
-Includes:
-- Build instructions and optimization
-- Performance tuning (worker threads, database pool)
-- Production deployment (systemd, Docker, load balancing)
-- Troubleshooting common issues
-- Benchmarking guides
+Full TypeScript definitions included:
 
-### Environment Variables
+```typescript
+import { Queen, Message, QueueConfig } from 'queen-mq'
 
-[The full list of environment variables is here](server/ENV_VARIABLES.md)
+const queen: Queen = new Queen('http://localhost:6632')
 
-### With Docker
-```sh
-./build.sh
+interface OrderData {
+  orderId: number
+  amount: number
+}
+
+const messages: Message<OrderData>[] = await queen.queue('orders').pop()
 ```
 
-### Running on k8s
+---
 
-[Running in k8s](server/k8s-example.yaml)
+## Documentation
 
-## 🔌 Raw HTTP API
+- **[Complete V2 Guide](client-v2/README.md)** - Full tutorial with all features (94 test examples)
+- **[HTTP API Reference](https://github.com/smartpricing/queen/blob/master/server/API.md)** - Raw HTTP endpoints
+- **[Server Guide](https://github.com/smartpricing/queen/blob/master/server/README.md)** - Server setup and configuration
+- **[Architecture Guide](https://github.com/smartpricing/queen/blob/master/documentation/ARCHITECTURE.md)** - Deep dive into internals
 
-You can use Queen directly from HTTP without the JS client.
+---
 
-[Here the complete list of API endpoints](server/API.md)
+## Support
 
-## ⚠️ Known Issues & Roadmap
+- **GitHub:** [smartpricing/queen](https://github.com/smartpricing/queen)
+- **Issues:** [GitHub Issues](https://github.com/smartpricing/queen/issues)
+- **LinkedIn:** [Smartness](https://www.linkedin.com/company/smartness-com/)
 
-### Server Startup Timing (Critical)
-**Issue:** Worker initialization timeout (30s → 3600s) now matches file buffer recovery timeout. This is a temporary fix.
+---
 
-**Better Solution Needed:**
-- Make recovery non-blocking while preserving FIFO ordering guarantees
-- Implement progressive readiness with memory-buffered queue during recovery
-- Add configurable recovery timeout with graceful degradation
-- See: `server/src/services/file_buffer.cpp:212` (MAX_STARTUP_RECOVERY_SECONDS)
-- See: `server/src/acceptor_server.cpp:1876` (worker initialization timeout)
+## License
 
+Apache 2.0 - See [LICENSE.md](../LICENSE.md)
 
-### Other TODO Items
-- Proper concurrency on clients 
-- Check client failover
-- Py client