queen-mq 0.2.23 → 0.3.1

package/README.md

@@ -7,7 +7,7 @@
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE.md)
 [![Node](https://img.shields.io/badge/node-%3E%3D22.0.0-brightgreen.svg)](https://nodejs.org/)
 
-[Quick Start](#js-client-usage) • [Examples](#-examples) • [Webapp](#webapp) • [Server Setup](#install-server-and-configure-it) • [HTTP API](#raw-http-api)
+[Quick Start](#js-client-usage-v2) • [Complete V2 Guide](client-js/client-v2/README.md) • [Examples](#-examples) • [Webapp](#webapp) • [Server Setup](#install-server-and-configure-it) • [HTTP API](#raw-http-api)
 
 <p align="center">
   <img src="assets/queen-logo-rose.svg" alt="Queen Logo" width="120" />
@@ -17,163 +17,173 @@
 
 ---
 
-## Introduction
-
-QueenMQ is a queue system written in C++ and backed by Postgres. Supports queues and consumer groups.
-
-## JS Client usage
-
-```js
-import { Queen } from 'queen-mq'
-
-const client = new Queen({ 
-    baseUrls: ['http://localhost:6632'],
-    timeout: 30000,
-    retryAttempts: 3
-});
-
-const queue = 'html-processing'
-
-// Create a queue
-await client.queue(queue, { leaseTime: 30 });
+Why "Queen"? Because years ago, when I first read the word "queue" in my mind, I read it as "queen".
 
-// Push some data, specifyng the partition
-await client.push(`${queue}/customer-1828`, [ { id: 1 } ]); 
-
-// Consume data with iterators
-for await (const msg of client.take(queue, { limit: 1 })) {
-    console.log(msg.data.id)
-    await client.ack(msg) //  OR await client.ack(msg, false) for nack
-}
-
-// Consume data with iterators, getting the entire batch
-for await (const messages of client.takeBatch(queue, { limit: 1, batch: 10, wait: true, timeout: 2000 })) {
-    const newMex = messages.map(x => x.data.id * 2)
-    await client.ack(messages) //  OR await client.ack(messages, false) for nack
-}
-
-// Consume data with a consumer group
-for await (const msg of client.take(`${queue}@analytics-data`, { limit: 2, batch: 2 })) {
-    // Do your computation and than ack with consumer group
-    await client.ack(msg, true, { group: 'analytics-data' });
-}
-
-// Consume data from any partition of the queue, continusly
-// This "pipeline" is useful for doing exactly one processing
-await client 
-.pipeline(queue)
-.withAutoRenewal({ 
-  interval: 5000  // Renew lease every 5 seconds
-})    
-.withConcurrency(5) // Five parallel promises
-.take(10, {
-  wait: true, // Use long polling
-  timeout: 30000 // Long polling length in millisconds
-})
-.processBatch(async (messages) => {
-    return messages.map(x => x.data.id * 2);
-})
-.atomically((tx, originalMessages, processedMessages) => { // ack and push are transactional inside atomically
-    tx.ack(originalMessages);
-    tx.push('another-queue', processedMessages); 
-})
-.repeat({ continuous: true })
-.execute();
-```
-
-## 📚 Examples
+---
 
-### Basic Usage
-- **[Basic Queue Operations](examples/01-basic-usage.js)** - Create queue, push, take, and ack messages
-- **[Batch Operations](examples/02-batch-operations.js)** - Push and consume messages in batches
+## Introduction
 
-### Advanced Features
-- **[Queue Configuration](examples/06-queue-configuration.js)** - Configure maxSize, windowBuffer, delay, retryLimit, and priority
-- **[Delayed Processing](examples/04-delayed-processing.js)** - Process messages after a delay
-- **[Window Buffer](examples/05-window-buffer.js)** - Delay message availability after push
+QueenMQ is a queue system written in C++ and backed by PostgreSQL, born from the need to manage many FIFO partitions for Smartchat with solid guarantees around delivery and failure handling. During the initial development, I realized that with a few simple additions to the original design, I could build a very powerful and flexible queue system. This project is almost entirely written by AI, with my supervision—only the test files (or a good part of them) are manually written.
 
-### Filtering & Routing
-- **[Namespace & Task Filtering](examples/07-namespace-task-filtering.js)** - Route and filter messages by namespace and task
-- **[Consumer Groups](examples/08-consumer-groups.js)** - Multiple consumer groups processing same messages
+Here are the main features:
+- Unlimited FIFO partitions within queues
+- Queue semantics like RabbitMQ
+- Consumer groups over queues like Kafka
+- QoS levels: Exactly-once delivery (with transactionId), at-least-once delivery, and at-most-once delivery
+- Subscription modes for replay (new messages only or from a specific timestamp) and message history control
+- Transactions between operations (push and ack mainly) for atomicity
+- Dead letter queue for failure handling
+- Lease renewal for long-running tasks
+- Message tracing for debugging workflows
+- Encryption of messages at DB level
 
-### Pipelines
-- **[Transactional Pipelines](examples/03-transactional-pipeline.js)** - Atomic processing with ack and push in a transaction
+The system consists of a PostgreSQL database, a replicated server that can be scaled horizontally (though it won't be the bottleneck), and a client library for interacting with the server. All client-server communication happens over HTTP, and the client library is written in JavaScript (support for other languages is planned). There's also a modern Vue 3 web app for monitoring and managing the system.
 
-### Event Streaming (QoS 0)
-- **[Event Streaming](examples/09-event-streaming.js)** - At-most-once delivery with buffering and auto-ack
+With proper batching, the system can handle +100k **messages** per second (not req/s) on modest hardware.
 
-## QoS 0: At-Most-Once Event Streaming
+Main documentation:
+- [Client Guide](client-js/client-v2/README.md)
+- [Server Guide](server/README.md)
+- [API Reference](API.md)
+- [Webapp](webapp/README.md)
 
-For high-throughput event streams, Queen supports **at-most-once delivery** with server-side buffering and auto-acknowledgment.
 
-### Server-Side Buffering
+## JS Client Usage
 
-Batch events on the server for 10-100x reduction in database writes:
+> 📖 **[Complete Guide: client-js/client-v2/README.md](client-js/client-v2/README.md)** - Full tutorial with all features!
 
 ```javascript
-// Push with buffering (QoS 0)
-await client.push('metrics', { cpu: 45, memory: 67 }, {
-  bufferMs: 100,      // Server batches for 100ms
-  bufferMax: 100      // Or until 100 events
-});
-
-// Result: 1000 events = ~10 DB writes (instead of 1000)
-```
+import { Queen } from 'queen-mq'
 
-### Auto-Acknowledgment
+// Connect to Queen
+const queen = new Queen('http://localhost:6632')
 
-Skip manual ack for fire-and-forget consumption:
+// Create a queue
+await queen.queue('tasks').create()
+
+// Push messages
+await queen.queue('tasks').push([
+  { data: { job: 'send-email', to: 'alice@example.com' } },
+  { data: { job: 'process-image', id: 123 } }
+])
+
+// Consume messages (auto-ack on success, auto-nack on error)
+await queen.queue('tasks').consume(async (message) => {
+  console.log('Processing:', message.data)
+  // If this succeeds → message completed ✅
+  // If this throws → message retried 🔄
+})
 
-```javascript
-// Consume with auto-ack
-for await (const msg of client.take('metrics', { autoAck: true })) {
-  updateDashboard(msg.data);
-  // No ack() needed - automatically acknowledged!
+// Pop messages manually
+const messages = await queen.queue('tasks').batch(10).pop()
+for (const msg of messages) {
+  try {
+    await processMessage(msg.data)
+    await queen.ack(msg, true)  // Success
+  } catch (error) {
+    await queen.ack(msg, false)  // Retry
+  }
 }
-```
-
-### Fan-Out Pattern (Consumer Groups)
-
-Combine buffering + auto-ack + consumer groups for pub/sub:
-
-```javascript
-// Publisher (buffered)
-await client.push('events', { action: 'login' }, { bufferMs: 100 });
 
-// Multiple subscribers (each group gets all messages)
-for await (const e of client.take('events@dashboard', { autoAck: true })) {
-  updateUI(e.data);
-}
+// Partitions (for ordering)
+await queen
+  .queue('user-events')
+  .partition('user-123')
+  .push([{ data: { event: 'login' } }])
+
+// Consumer groups (for scaling)
+await queen
+  .queue('emails')
+  .group('processors')
+  .concurrency(5)
+  .consume(async (message) => {
+    await sendEmail(message.data)
+  })
+
+// Subscription modes (control message history)
+// Skip historical messages, only process new ones
+await queen
+  .queue('events')
+  .group('realtime-monitor')
+  .subscriptionMode('new')
+  .consume(async (message) => {
+    console.log('New event only:', message.data)
+  })
+
+// Start from a specific timestamp
+const timestamp = '2025-10-28T10:00:00.000Z'
+await queen
+  .queue('events')
+  .group('replay-from-timestamp')
+  .subscriptionFrom(timestamp)
+  .consume(async (message) => {
+    console.log('Replaying from 10am:', message.data)
+  })
+
+// Transactions (atomic ack + push)
+const [msg] = await queen.queue('input').pop()
+await queen
+  .transaction()
+  .ack(msg)
+  .queue('output')
+  .push([{ data: { processed: true } }])
+  .commit()
+
+// Client-side buffering (for high throughput)
+await queen
+  .queue('logs')
+  .buffer({ messageCount: 100, timeMillis: 1000 })
+  .push([{ data: { level: 'info', message: 'Server started' } }])
+
+// Message tracing (for debugging and monitoring)
+await queen.queue('orders').consume(async (msg) => {
+  await msg.trace({
+    traceName: ['tenant-acme', 'order-flow-123'],
+    data: { text: 'Processing started', orderId: msg.data.id }
+  })
+  
+  // Process order...
+  
+  await msg.trace({
+    traceName: ['tenant-acme', 'order-flow-123'],
+    data: { text: 'Order completed' }
+  })
+})
 
-for await (const e of client.take('events@analytics', { autoAck: true })) {
-  trackEvent(e.data);
-}
+// Graceful shutdown
+await queen.close()
 ```
 
-### PostgreSQL Failover
+**Key Features:**
+- ✅ Fluent, chainable API
+- ✅ Auto-acknowledgment (or manual control)
+- ✅ Partitions for ordered processing
+- ✅ Consumer groups for scaling
+- ✅ **Subscription modes (new messages only or from timestamp)**
+- ✅ Transactions for atomicity
+- ✅ Client-side buffering for speed
+- ✅ Dead letter queue for failures
+- ✅ Lease renewal for long tasks
+- ✅ **Message tracing for debugging workflows**
+- ✅ Graceful shutdown with buffer flush
 
-Queen automatically buffers messages to disk when PostgreSQL is unavailable - **zero message loss**:
-
-- Normal pushes go directly to PostgreSQL (FIFO preserved)
-- If PostgreSQL is down, messages buffered to file (macOS: `/tmp/queen`, Linux: `/var/lib/queen/buffers`)
-- Automatic replay when PostgreSQL recovers
-- Survives server crashes and restarts
-- Directory auto-created on first run
+## 📚 Examples
 
-**No configuration needed** - failover is automatic!
+### Client
 
-**Custom directory:**
-```bash
-FILE_BUFFER_DIR=/custom/path ./bin/queen-server
-```
+See the **[Complete V2 Guide](client-js/client-v2/README.md)** with 14 parts covering everything from basics to advanced features:
+- Queue creation, push, and consume
+- Partitions and consumer groups
+- **Subscription modes (new messages, timestamps)**
+- Transactions and buffering
+- Dead letter queues
+- Lease renewal
+- Message tracing
+- Complete real-world pipeline example
+- And much more!
 
-### When to Use
+**Test Files** (94 working examples): [client-js/test-v2/](client-js/test-v2/)
 
-| Feature | Use For | Don't Use For |
-|---------|---------|---------------|
-| **Buffering** | Metrics, logs, analytics, UI updates | Critical tasks, payments |
-| **Auto-Ack** | Fire-and-forget events, notifications | Tasks requiring retry logic |
-| **Failover** | Everything (automatic) | N/A - always beneficial |
 
 ## Architecture
 
@@ -206,6 +216,23 @@ Queen uses a high-performance **acceptor/worker pattern** with uWebSockets, comb
 
 This architecture provides high concurrency, efficient connection pooling, and minimal latency for both immediate and long-polling requests.
 
+### PostgreSQL Failover
+
+Queen automatically buffers messages to disk when PostgreSQL is unavailable - **zero message loss**:
+
+- Normal pushes go directly to PostgreSQL (FIFO preserved)
+- If PostgreSQL is down, messages buffered to file (macOS: `/tmp/queen`, Linux: `/var/lib/queen/buffers`)
+- Automatic replay when PostgreSQL recovers
+- Survives server crashes and restarts
+- Directory auto-created on first run
+
+**No configuration needed** - failover is automatic!
+
+**Custom directory:**
+```bash
+FILE_BUFFER_DIR=/custom/path ./bin/queen-server
+```
+
 ## Webapp
 
 A modern Vue 3 web interface for managing and monitoring Queen MQ.
@@ -215,7 +242,8 @@ A modern Vue 3 web interface for managing and monitoring Queen MQ.
 - 📈 Message throughput visualization
 - 🔍 Queue management and monitoring
 - 👥 Consumer group tracking
-- 💬 Message browser
+- 💬 Message browser with trace timeline
+- 🔎 **Trace explorer for debugging distributed workflows**
 - 📉 Analytics and insights
 - 🌓 Dark/light theme support
 
@@ -285,7 +313,5 @@ You can use Queen directly from HTTP without the JS client.
 
 ### Other TODO Items
 - retention jobs
-- reconsume
-- new client
 - auth
 - streaming engine

package/client-js/client-v2/LOGGING.md

@@ -0,0 +1,240 @@
+# Queen Client V2 - Logging Documentation
+
+## Overview
+
+The Queen Client V2 includes comprehensive operation logging that captures every significant action performed by the client. Logging is **disabled by default** and can be enabled via the `QUEEN_CLIENT_LOG` environment variable.
+
+## Enabling Logging
+
+```bash
+export QUEEN_CLIENT_LOG=true
+node your-app.js
+```
+
+Or inline:
+```bash
+QUEEN_CLIENT_LOG=true node your-app.js
+```
+
+## Log Format
+
+All logs follow this format:
+```
+[TIMESTAMP] [LEVEL] [OPERATION] DETAILS
+```
+
+- **TIMESTAMP**: ISO 8601 format (e.g., `2025-10-28T10:30:45.123Z`)
+- **LEVEL**: `INFO`, `WARN`, or `ERROR`
+- **OPERATION**: Component and method (e.g., `Queen.push`, `HttpClient.request`)
+- **DETAILS**: JSON object with relevant context
+
+### Example Log Output
+
+```
+[2025-10-28T10:30:45.123Z] [INFO] [Queen.constructor] {"status":"initialized","urls":1}
+[2025-10-28T10:30:45.234Z] [INFO] [QueueBuilder.push] {"queue":"tasks","partition":"Default","count":5,"buffered":true}
+[2025-10-28T10:30:45.456Z] [INFO] [BufferManager.addMessage] {"queueAddress":"tasks/Default","messageCount":5}
+[2025-10-28T10:30:46.789Z] [INFO] [HttpClient.request] {"method":"POST","url":"http://localhost:6632/api/v1/push","hasBody":true,"timeout":30000}
+[2025-10-28T10:30:46.890Z] [INFO] [HttpClient.response] {"method":"POST","url":"http://localhost:6632/api/v1/push","status":200}
+[2025-10-28T10:30:47.123Z] [INFO] [BufferManager.flushBuffer] {"queueAddress":"tasks/Default","status":"success","messagesSent":5}
+```
+
+## Logged Operations
+
+### Queen (Main Client)
+
+| Operation | Logged Details |
+|-----------|---------------|
+| `Queen.constructor` | Configuration summary, URL count |
+| `Queen.ack` | Batch/single, message count, status, context |
+| `Queen.renew` | Lease ID count, success/failure per lease |
+| `Queen.flushAllBuffers` | Start and completion status |
+| `Queen.getBufferStats` | Buffer statistics |
+| `Queen.close` | Shutdown phases, errors |
+
+### QueueBuilder (Queue Operations)
+
+| Operation | Logged Details |
+|-----------|---------------|
+| `QueueBuilder.create` | Queue name, namespace, task |
+| `QueueBuilder.delete` | Queue name |
+| `QueueBuilder.push` | Queue, partition, count, buffered flag |
+| `QueueBuilder.pop` | Queue, partition, batch size, wait mode, result count |
+| `QueueBuilder.flushBuffer` | Queue address |
+| `QueueBuilder.dlq` | Queue, consumer group, partition |
+
+### HttpClient (Network Operations)
+
+| Operation | Logged Details |
+|-----------|---------------|
+| `HttpClient.constructor` | Configuration (timeout, retries, failover) |
+| `HttpClient.request` | Method, URL, body presence, timeout |
+| `HttpClient.response` | Method, URL, HTTP status |
+| `HttpClient.retry` | Attempt number, delay, error message |
+| `HttpClient.failover` | Server count, attempted URLs, failures |
+
+### BufferManager (Client-Side Buffering)
+
+| Operation | Logged Details |
+|-----------|---------------|
+| `BufferManager.createBuffer` | Queue address, buffer options |
+| `BufferManager.addMessage` | Queue address, current message count |
+| `BufferManager.flushBuffer` | Queue address, message count, status |
+| `BufferManager.flushAllBuffers` | Buffer count, pending flushes |
+| `BufferManager.getStats` | Active buffers, total messages, oldest age |
+| `BufferManager.cleanup` | Buffer count being cleaned |
+
+### ConsumerManager (Message Consumption)
+
+| Operation | Logged Details |
+|-----------|---------------|
+| `ConsumerManager.start` | Queue, concurrency, batch size, auto-ack |
+| `ConsumerManager.worker` | Worker ID, lifecycle events, processed count |
+| `ConsumerManager.processMessage` | Transaction ID, ack/nack status, errors |
+| `ConsumerManager.processBatch` | Message count, ack/nack status, errors |
+
+### TransactionBuilder (Atomic Operations)
+
+| Operation | Logged Details |
+|-----------|---------------|
+| `TransactionBuilder.ack` | Message count, status |
+| `TransactionBuilder.queue.push` | Queue name, item count |
+| `TransactionBuilder.commit` | Operation count, required leases, success/failure |
+
+### Other Builders
+
+| Operation | Logged Details |
+|-----------|---------------|
+| `OperationBuilder.execute` | Method, path, success/failure |
+| `PushBuilder.execute` | Queue, partition, count, buffered flag, results |
+| `DLQBuilder.get` | Queue, consumer group, limit, offset, results |
+
+## Use Cases
+
+### 1. Debugging
+
+Use logging to trace through client operations and identify where issues occur:
+
+```bash
+QUEEN_CLIENT_LOG=true node my-app.js 2>&1 | grep ERROR
+```
+
+### 2. Performance Analysis
+
+Monitor HTTP request/response times and buffer flush patterns:
+
+```bash
+QUEEN_CLIENT_LOG=true node my-app.js 2>&1 | grep HttpClient
+```
+
+### 3. Audit Trail
+
+Create an audit log of all queue operations:
+
+```bash
+QUEEN_CLIENT_LOG=true node my-app.js 2>&1 | tee audit.log
+```
+
+### 4. Development
+
+Enable logging during development to understand client behavior:
+
+```javascript
+// In .env file
+QUEEN_CLIENT_LOG=true
+```
+
+### 5. Production Troubleshooting
+
+Temporarily enable logging in production to diagnose issues:
+
+```bash
+# Enable for one process
+QUEEN_CLIENT_LOG=true pm2 restart my-app --update-env
+
+# Disable after troubleshooting
+pm2 restart my-app
+```
+
+## Performance Impact
+
+- **Disabled (default)**: Zero performance impact - all logging calls are no-ops
+- **Enabled**: Minimal impact - logging is asynchronous and uses `console.log/warn/error`
+
+## Log Levels
+
+- **INFO**: Normal operations (most logs)
+- **WARN**: Recoverable issues (retries, failover, network errors)
+- **ERROR**: Operation failures (push failed, ack failed, etc.)
+
+## Filtering Logs
+
+### By Component
+```bash
+QUEEN_CLIENT_LOG=true node app.js 2>&1 | grep "HttpClient"
+QUEEN_CLIENT_LOG=true node app.js 2>&1 | grep "BufferManager"
+QUEEN_CLIENT_LOG=true node app.js 2>&1 | grep "ConsumerManager"
+```
+
+### By Level
+```bash
+QUEEN_CLIENT_LOG=true node app.js 2>&1 | grep "ERROR"
+QUEEN_CLIENT_LOG=true node app.js 2>&1 | grep "WARN"
+```
+
+### By Operation
+```bash
+QUEEN_CLIENT_LOG=true node app.js 2>&1 | grep "push"
+QUEEN_CLIENT_LOG=true node app.js 2>&1 | grep "ack"
+QUEEN_CLIENT_LOG=true node app.js 2>&1 | grep "pop"
+```
+
+## Integration with Log Aggregation
+
+The structured JSON format makes it easy to integrate with log aggregation tools:
+
+### Winston
+```javascript
+import winston from 'winston'
+
+// Redirect console.log to Winston
+console.log = winston.info
+console.error = winston.error
+console.warn = winston.warn
+```
+
+### Pino
+```javascript
+import pino from 'pino'
+const logger = pino()
+
+console.log = (msg) => logger.info(msg)
+console.error = (msg) => logger.error(msg)
+console.warn = (msg) => logger.warn(msg)
+```
+
+### Datadog, CloudWatch, etc.
+
+The ISO 8601 timestamps and JSON format are compatible with most log aggregation services.
+
+## Best Practices
+
+1. **Disable in Production**: Only enable when needed for troubleshooting
+2. **Use Log Rotation**: If keeping logs enabled, use log rotation to manage disk space
+3. **Filter Sensitive Data**: Payloads are NOT logged - only metadata
+4. **Monitor Log Volume**: High-throughput applications generate many logs when enabled
+5. **Use Structured Search**: Leverage JSON format for precise filtering
+
+## Implementation Details
+
+The logging system is implemented in `utils/logger.js` and imported by all major components:
+
+- `Queen.js` - Main client operations
+- `HttpClient.js` - Network layer
+- `BufferManager.js` - Client-side buffering
+- `ConsumerManager.js` - Message consumption
+- `QueueBuilder.js` - Queue operations
+- `TransactionBuilder.js` - Atomic transactions
+
+All logging calls check the `QUEEN_CLIENT_LOG` environment variable and are no-ops when disabled, ensuring zero performance impact by default.
+