queen-mq 0.1.0 → 0.1.2

package/QUERY_ANALSYS.md

@@ -1,72 +0,0 @@
-### Query Analysis and Improvement Plan
-
-This document outlines the planned changes to improve SQL safety, correctness, FIFO guarantees, duplicate-avoidance, and performance.
-
-## Goals
-- Enforce per-partition FIFO even when no partition is specified.
-- Prevent duplicate message selection across workers.
-- Harden transactions against injection and long blocks.
-- Reduce planner anti-patterns; improve query efficiency.
-- Align retention/eviction with schema v2 semantics.
-
-## Changes and Why
-
-1) Security and transaction hardening
-- What: Whitelist transaction isolation levels in `withTransaction` (no dynamic interpolation).
-- Why: Avoids SQL injection risk and invalid isolation values.
-
-- What: Set `statement_timeout` and `lock_timeout` via `SET LOCAL` at transaction start.
-- Why: Prevents long blocking operations and stuck workers under contention.
-
-- What: Add retry for serialization failures (SQLSTATE 40001) and deadlocks (40P01) in the query wrapper.
-- Why: These are expected under high contention; bounded retry improves resiliency.
-
-2) Strict FIFO when partition is unspecified
-- What: Partition-first selection in queue-mode and namespace/task pop paths; lock the chosen partition, then fetch only from that partition ordered by `created_at ASC`.
-- Why: Guarantees FIFO per partition and eliminates mixed-partition batches.
-
-- What: Add fairness heuristic for choosing the partition (e.g., partition with the oldest pending message or least recent `last_activity`).
-- Why: Prevents starvation and balances throughput without breaking FIFO.
-
-3) Availability predicate simplification
-- What: Replace `LEFT JOIN messages_status ... WHERE (ms.id IS NULL OR ms.status IN (...))` with `NOT EXISTS`/UNION.
-- Why: Removes OR conditions that block index use; produces better plans.
-
-- What: Precompute per-queue time thresholds (delayed_processing, max_wait_time, window_buffer) in a small CTE.
-- Why: Avoids non-sargable predicates involving NOW() and enables index range scans on `created_at`.
-
-4) Pop query shaping
-- What: Eliminate correlated subselects in `RETURNING` by projecting needed columns in the CTE and returning them directly.
-- Why: Avoids per-row subqueries; simpler and faster.
-
-- What: Keep `FOR UPDATE OF m SKIP LOCKED` in queue-mode pops.
-- Why: Ensures a message is only claimed by one worker in competing consumer mode.
-
-5) Bus-mode (consumer groups)
-- What: Use `ON CONFLICT DO NOTHING` (or update) when inserting into `messages_status`.
-- Why: Handles races on `(message_id, consumer_group)` without surfacing errors; maintains single-delivery per group.
-
-6) API pagination
-- What: Implement keyset (cursor) pagination for `listMessages` while preserving offset for backward compatibility.
-- Why: OFFSET is O(n) at high pages; keyset keeps scans bounded and stable using `(created_at, id)`.
-
-7) Retention and eviction
-- What: Align `retentionService` with schema v2: delete only when configured to do so; rely on cascade for statuses.
-- Why: Prevents unintended loss of unprocessed messages; keeps behavior explicit.
-
-- What: Keep eviction as status transitions (e.g., `evicted`) and log counts to retention history.
-- Why: Avoids destructive deletes while signaling expired messages.
-
-8) Tests
-- What: Add/adjust tests for single-partition FIFO (when partition unspecified), no duplicates under concurrency (queue/bus), keyset pagination, and retention/eviction behavior.
-- Why: Prevent regressions and validate concurrency semantics.
-
-## Rollout Order
-1. Transaction hardening (whitelist isolation, timeouts) and retry policy.
-2. Partition-first pop logic (queue-mode, namespace/task) with fairness.
-3. Availability predicate rewrite and time-threshold precompute.
-4. Pop `RETURNING` cleanup (remove correlated subselects).
-5. Bus-mode conflict handling with ON CONFLICT.
-6. Keyset pagination for `listMessages`.
-7. Retention adjustments.
-8. Tests and docs.

package/QUEUE_BUS.md

@@ -1,334 +0,0 @@
-# Queue/Bus Mode Implementation Plan
-
-## Overview
-Implement dynamic queue/bus mode based on consumer group parameter, allowing the same queue to serve both competing consumers (queue mode) and broadcast to multiple consumer groups (bus mode).
-
-## Core Concept
-- **Queue Mode**: No `consumerGroup` specified → competing consumers (traditional queue behavior)
-- **Bus Mode**: `consumerGroup` specified → each consumer group gets all messages (pub/sub behavior)
-- **Mixed Mode**: Same queue can serve both patterns simultaneously
-
-## Architecture Changes
-
-### 1. Database Schema Changes (`src/database/schema-v2.sql`)
-
-#### 1.1 Modify Messages Table
-- Remove all status-related fields from `queen.messages`
-- Keep only: `id`, `transaction_id`, `partition_id`, `payload`, `created_at`, `is_encrypted`
-- Keep `transaction_id` in messages table (set during push, immutable)
-
-#### 1.2 Create Messages Status Table
-```sql
-CREATE TABLE queen.messages_status (
-    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-    message_id UUID REFERENCES queen.messages(id) ON DELETE CASCADE,
-    consumer_group VARCHAR(255),  -- NULL for queue mode
-    status VARCHAR(20) DEFAULT 'pending',
-    worker_id VARCHAR(255),
-    locked_at TIMESTAMP,
-    completed_at TIMESTAMP,
-    failed_at TIMESTAMP,
-    error_message TEXT,
-    retry_count INTEGER DEFAULT 0,
-    lease_expires_at TIMESTAMP,
-    processing_at TIMESTAMP,
-    created_at TIMESTAMP DEFAULT NOW(),
-    UNIQUE(message_id, consumer_group)
-);
-```
-
-#### 1.3 Consumer Group Registry (Optional)
-```sql
-CREATE TABLE queen.consumer_groups (
-    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-    queue_id UUID REFERENCES queen.queues(id) ON DELETE CASCADE,
-    name VARCHAR(255) NOT NULL,
-    created_at TIMESTAMP DEFAULT NOW(),
-    subscription_start_from TIMESTAMP,  -- NULL = consume all, timestamp = consume from this point
-    active BOOLEAN DEFAULT true,
-    UNIQUE(queue_id, name)
-);
-```
-
-#### 1.3 Create Optimized Indexes
-- Index for queue mode lookups (consumer_group IS NULL)
-- Index for bus mode lookups (consumer_group IS NOT NULL)
-- Composite indexes for efficient pop operations
-- Indexes for lease expiration checks
-
-### 2. Consumer Group Subscription Strategies
-
-#### Option 1: Consume All Messages (Default)
-- New consumer groups start from the beginning
-- Process all existing messages in the queue
-- Good for: Analytics, audit logs, data replication
-
-#### Option 2: Consume From Subscription Time
-- New consumer groups only see messages created after subscription
-- Skip all historical messages
-- Good for: Real-time notifications, monitoring
-
-#### Option 3: Consume From Specific Time
-- Consumer group specifies a start timestamp
-- Process messages from that point forward
-- Good for: Replay scenarios, recovery
-
-#### Implementation Approach:
-```javascript
-// Option 1: Consume all (default)
-client.pop({ 
-  queue: 'events', 
-  consumerGroup: 'analytics' 
-});
-
-// Option 2: From subscription time
-client.pop({ 
-  queue: 'events', 
-  consumerGroup: 'notifications',
-  subscriptionMode: 'new'  // Only new messages
-});
-
-// Option 3: From specific time
-client.pop({ 
-  queue: 'events', 
-  consumerGroup: 'replay',
-  subscriptionFrom: '2024-01-01T00:00:00Z'
-});
-```
-
-#### Database Implementation:
-```sql
--- When popping with a new consumer group:
--- 1. Check if consumer group exists in registry
--- 2. If not, create it with subscription preferences
--- 3. Only return messages based on subscription_start_from
-
--- For "new only" mode:
-INSERT INTO queen.consumer_groups (queue_id, name, subscription_start_from)
-VALUES ($1, $2, NOW());
-
--- For "all messages" mode:
-INSERT INTO queen.consumer_groups (queue_id, name, subscription_start_from)
-VALUES ($1, $2, NULL);  -- NULL means consume all
-
--- Pop query considers subscription time:
-SELECT m.* FROM queen.messages m
-WHERE m.partition_id = $1
-  AND NOT EXISTS (
-    SELECT 1 FROM queen.messages_status ms
-    WHERE ms.message_id = m.id 
-      AND ms.consumer_group = $2
-  )
-  AND m.created_at >= COALESCE(
-    (SELECT subscription_start_from 
-     FROM queen.consumer_groups 
-     WHERE queue_id = $3 AND name = $2),
-    '1970-01-01'::timestamp  -- If NULL, consume all
-  )
-ORDER BY m.created_at ASC
-LIMIT $4;
-```
-
-### 3. Backend Changes
-
-#### 3.1 Queue Manager (`src/managers/queueManagerOptimized.js`)
-
-##### Push Messages
-- Keep existing behavior: generate transaction_id during push
-- Only insert into messages table (no status entry)
-- Transaction_id remains in messages table (immutable)
-
-##### Pop Messages
-- **Queue Mode Logic** (no consumer group):
-  ```javascript
-  // 1. Find messages without status OR with pending status where consumer_group IS NULL
-  // 2. Create/update status entry with consumer_group = NULL
-  // 3. Mark as processing with lease
-  // 4. Return messages
-  ```
-
-- **Bus Mode Logic** (with consumer group):
-  ```javascript
-  // 1. Find messages without status entry for this specific consumer_group
-  // 2. Create status entry for this consumer_group
-  // 3. Mark as processing with lease
-  // 4. Return messages
-  ```
-
-##### Acknowledge Messages
-- Update status for specific consumer_group (or NULL)
-- In bus mode, message stays available for other consumer groups
-- In queue mode, message is done after single acknowledgment
-
-##### Reclaim Leases
-- Check lease expiration per consumer_group
-- Reset to pending for that specific consumer_group only
-
-#### 3.2 Pop Route (`src/routes/pop.js`)
-- Accept optional `consumerGroup` parameter
-- Pass through to queue manager
-
-#### 3.3 Server Routes (`src/server.js`)
-- Update pop endpoint to accept consumer_group query parameter
-- Ensure backward compatibility (no consumer_group = queue mode)
-
-### 4. Client SDK Changes (`src/client/queenClient.js`)
-
-#### 4.1 Pop Method
-```javascript
-pop({ queue, partition, consumerGroup, wait, timeout, batch })
-```
-
-#### 4.2 Consume Method
-```javascript
-consume({ queue, partition, consumerGroup, handler, options })
-```
-
-#### 4.3 Documentation
-- Add consumer group examples
-- Document queue vs bus mode behavior
-
-### 5. Testing (`src/test/test.js`)
-
-#### 5.1 Queue Mode Tests
-- Test competing consumers (no consumer group)
-- Verify only one consumer gets each message
-- Test acknowledgment completes message
-
-#### 5.2 Bus Mode Tests
-- Test multiple consumer groups
-- Verify all groups get all messages
-- Test independent acknowledgment per group
-- Test consumer group isolation
-
-#### 5.3 Mixed Mode Tests
-- Test same queue with both modes simultaneously
-- Verify queue mode consumers compete
-- Verify bus mode consumers all receive messages
-
-#### 5.4 Edge Cases
-- Test consumer group with special characters
-- Test very long consumer group names
-- Test lease expiration per consumer group
-- Test retry logic per consumer group
-
-### 6. Examples
-
-#### 6.1 Create Bus Mode Example (`examples/bus-mode.js`)
-- Demonstrate multiple consumer groups
-- Show independent progress tracking
-- Compare with queue mode
-
-#### 6.2 Create Mixed Mode Example (`examples/mixed-mode.js`)
-- Same queue, different consumption patterns
-- Workers + Analytics + Notifications
-
-### 7. Documentation Updates
-
-#### 7.1 README.md
-- Add Bus Mode section
-- Document consumer groups
-- Add comparison with Kafka
-- Update API reference
-
-#### 7.2 API.md
-- Document consumer_group parameter
-- Add bus mode examples
-
-## Implementation Steps
-
-### Phase 1: Database Schema
-1. Backup existing schema
-2. Modify `schema-v2.sql` with new structure
-3. Drop and recreate database:
-   ```bash
-   nvm use 22 && node init-db.js
-   ```
-
-### Phase 2: Backend Core
-1. Update queue manager push logic
-2. Implement dual-mode pop logic
-3. Update acknowledgment logic
-4. Update lease reclaim logic
-5. Test with basic operations
-
-### Phase 3: API Layer
-1. Update pop route to accept consumer_group
-2. Update server endpoints
-3. Ensure backward compatibility
-4. Test API endpoints
-
-### Phase 4: Client SDK
-1. Add consumerGroup parameter to pop
-2. Update consume method
-3. Add TypeScript types (if applicable)
-4. Test client operations
-
-### Phase 5: Testing
-1. Write comprehensive test suite
-2. Test queue mode (backward compatibility)
-3. Test bus mode (new functionality)
-4. Test mixed mode scenarios
-5. Performance testing
-
-### Phase 6: Examples & Documentation
-1. Create example scripts
-2. Update README
-3. Update API documentation
-4. Create migration guide
-
-## Testing Commands
-
-```bash
-# Reinitialize database
-nvm use 22 && node init-db.js
-
-# Start server
-nvm use 22 && npm start
-
-# Run tests
-nvm use 22 && node src/test/test.js
-
-# Test bus mode
-nvm use 22 && node examples/bus-mode.js
-
-# Test mixed mode
-nvm use 22 && node examples/mixed-mode.js
-```
-
-## Performance Considerations
-
-### Optimizations
-1. Use partial indexes for NULL/NOT NULL consumer_group
-2. Batch status insertions for bus mode
-3. Consider materialized views for message counts
-4. Add connection pooling per consumer group
-
-### Monitoring
-1. Track consumer group lag
-2. Monitor status table growth
-3. Add metrics for queue vs bus mode usage
-4. Dashboard updates for consumer groups
-
-## Rollback Plan
-1. Keep backup of original schema
-2. Maintain backward compatibility
-3. Feature flag for bus mode (optional)
-4. Gradual rollout strategy
-
-## Success Criteria
-- [ ] All existing tests pass (backward compatibility)
-- [ ] Queue mode works as before
-- [ ] Bus mode delivers to all consumer groups
-- [ ] Mixed mode works correctly
-- [ ] Performance remains acceptable
-- [ ] Documentation is complete
-- [ ] Examples demonstrate all patterns
-
-## Notes
-- Consumer group = NULL is special case for queue mode
-- Transaction ID moves to status table
-- Messages table becomes immutable after insert
-- Status table handles all mutable state
-- Consider TTL for old status entries
-- Add cleanup job for completed messages in bus mode

package/V2-PLAN.md

@@ -1,236 +0,0 @@
-# Queen V2 Architecture Migration Plan
-
-## Overview
-Restructuring the queue system from a rigid 3-tier hierarchy (Namespaces → Tasks → Queues) to a flexible 2-tier system (Queues → Partitions) with optional grouping fields.
-
-## New Data Model
-
-### Core Structure
-```
-Queues (with optional namespace/task grouping)
-  └── Partitions (FIFO ordering)
-       └── Messages
-```
-
-### Key Changes
-1. **Remove** `namespaces` table
-2. **Rename** `tasks` table → `queues` table
-3. **Rename** `queues` table → `partitions` table
-4. **Add** optional `namespace` and `task` fields to `queues` table for grouping
-5. **Default Partition**: Each queue automatically gets a "Default" partition
-
-## Database Schema Changes
-
-### 1. Update schema.sql
-- Drop the `namespaces` table
-- Rename and restructure tables:
-  ```sql
-  -- New queues table (was tasks)
-  CREATE TABLE queen.queues (
-    id UUID PRIMARY KEY,
-    name VARCHAR(255) UNIQUE NOT NULL,
-    namespace VARCHAR(255), -- optional grouping
-    task VARCHAR(255),      -- optional grouping
-    created_at TIMESTAMP
-  );
-  
-  -- New partitions table (was queues)
-  CREATE TABLE queen.partitions (
-    id UUID PRIMARY KEY,
-    queue_id UUID REFERENCES queen.queues(id),
-    name VARCHAR(255) NOT NULL DEFAULT 'Default',
-    priority INTEGER DEFAULT 0,
-    options JSONB,
-    created_at TIMESTAMP,
-    UNIQUE(queue_id, name)
-  );
-  
-  -- Messages table stays similar, references partitions
-  CREATE TABLE queen.messages (
-    id UUID PRIMARY KEY,
-    partition_id UUID REFERENCES queen.partitions(id),
-    -- rest stays the same
-  );
-  ```
-
-### 2. Update Indexes
-- Adjust all indexes to work with new structure
-- Add indexes for optional namespace/task fields on queues table
-- Maintain performance-critical indexes on messages table
-
-## API Route Changes
-
-### Push Routes
-**Current**: `POST /api/v1/push`
-```json
-{
-  "items": [{
-    "ns": "namespace",
-    "task": "task", 
-    "queue": "queue",
-    "payload": {}
-  }]
-}
-```
-
-**New**: `POST /api/v1/push`
-```json
-{
-  "items": [{
-    "queue": "queue",
-    "partition": "partition", // optional, defaults to "Default"
-    "payload": {}
-  }]
-}
-```
-
-### Pop Routes
-**Current**:
-- `/api/v1/pop/ns/:ns/task/:task/queue/:queue`
-- `/api/v1/pop/ns/:ns/task/:task`
-- `/api/v1/pop/ns/:ns`
-
-**New**:
-- `/api/v1/pop/queue/:queue/partition/:partition` - Pop from specific partition
-- `/api/v1/pop/queue/:queue` - Pop from any available partition in queue
-- `/api/v1/pop?namespace=:ns` - Pop from any queue in namespace (optional filter)
-- `/api/v1/pop?task=:task` - Pop from any queue in task (optional filter)
-
-### Configure Routes
-**Current**: `POST /api/v1/configure`
-```json
-{
-  "ns": "namespace",
-  "task": "task",
-  "queue": "queue",
-  "options": {}
-}
-```
-
-**New**: `POST /api/v1/configure`
-```json
-{
-  "queue": "queue",
-  "partition": "partition", // optional, defaults to "Default"
-  "options": {}
-}
-```
-
-### Analytics Routes
-**Current**:
-- `/api/v1/analytics/ns/:ns`
-- `/api/v1/analytics/ns/:ns/task/:task`
-
-**New**:
-- `/api/v1/analytics/queue/:queue`
-- `/api/v1/analytics?namespace=:ns` - Filter by namespace
-- `/api/v1/analytics?task=:task` - Filter by task
-
-## Code Changes Required
-
-### 1. Database Layer (`src/database/`)
-- [ ] Rewrite `schema.sql` with new structure
-- [ ] Update `connection.js` initialization if needed
-
-### 2. Manager Classes (`src/managers/`)
-- [ ] Update `queueManagerOptimized.js`:
-  - [ ] Change `ensureResources()` to handle queue → partition structure
-  - [ ] Update `pushMessages()` to default to "Default" partition
-  - [ ] Modify `popMessages()` for new partition logic
-  - [ ] Adjust `getQueueStats()` for new structure
-- [ ] Update `resourceCache.js`:
-  - [ ] Change cache key format from `ns:task:queue` to `queue:partition`
-- [ ] Update `eventManager.js`:
-  - [ ] Adjust event paths from `ns/task/queue` to `queue/partition`
-
-### 3. Route Handlers (`src/routes/`)
-- [ ] Update `push.js`:
-  - [ ] Remove ns/task requirements
-  - [ ] Add partition defaulting logic
-- [ ] Update `pop.js`:
-  - [ ] New route parameter structure
-  - [ ] Implement "any available partition" logic
-- [ ] Update `configure.js`:
-  - [ ] Work with queue/partition instead of ns/task/queue
-- [ ] Update `analytics.js`:
-  - [ ] New aggregation logic based on optional namespace/task fields
-- [ ] Update `messages.js`:
-  - [ ] Adjust queries for new structure
-
-### 4. Server (`src/server.js`)
-- [ ] Update all route definitions
-- [ ] Adjust URL patterns
-- [ ] Update WebSocket event paths
-
-### 5. Client Library (`src/client/`)
-- [ ] Update `queenClient.js` with new API structure
-- [ ] Adjust method signatures
-
-### 6. WebSocket (`src/websocket/`)
-- [ ] Update event naming and paths
-- [ ] Adjust queue depth updates
-
-## Implementation Order
-
-1. **Database Schema**
-   - Create new `schema-v2.sql`
-   - Test schema independently
-
-2. **Core Manager Updates**
-   - Update `queueManagerOptimized.js` with new logic
-   - Update resource caching
-   - Update event manager
-
-3. **Route Handlers**
-   - Update each route file for new structure
-   - Maintain backwards compatibility temporarily if needed
-
-4. **Server Integration**
-   - Update `server.js` with new routes
-   - Test each endpoint
-
-5. **Client Updates**
-   - Update client library
-   - Update examples
-
-6. **Frontend Updates**
-   - Update API calls in frontend
-   - Adjust data structures
-
-## Partition Selection Logic
-
-When popping from a queue (without specifying partition):
-1. Select the partition with the oldest pending message
-2. Use `ORDER BY created_at ASC` across partitions
-3. Maintain FIFO within each partition
-4. Use `FOR UPDATE SKIP LOCKED` for concurrency
-
-## Default Partition Behavior
-
-1. **Auto-creation**: When a queue is created, automatically create a "Default" partition
-2. **Push behavior**: If no partition specified, use "Default"
-3. **Pop behavior**: Include "Default" when popping from queue level
-
-## Node
-Always use "nvm use 22 && your node.js command"
-
-## Testing Checklist
-
-- [ ] Push to queue with no partition (should use Default)
-- [ ] Push to specific partition
-- [ ] Pop from specific partition
-- [ ] Pop from queue (any partition)
-- [ ] Queue configuration
-- [ ] Partition configuration
-- [ ] Analytics with namespace filtering
-- [ ] Analytics with task filtering
-- [ ] Message acknowledgment
-- [ ] Batch operations
-- [ ] WebSocket events
-- [ ] Long polling
-- [ ] Lease expiration
-- [ ] Performance with multiple partitions
-
-## Migration Notes
-
-- No data migration needed (starting fresh)

package/dashboard/.vscode/extensions.json

@@ -1,3 +0,0 @@
-{
-  "recommendations": ["Vue.volar"]
-}

package/dashboard/README.md

@@ -1,5 +0,0 @@
-# Vue 3 + Vite
-
-This template should help get you started developing with Vue 3 in Vite. The template uses Vue 3 `<script setup>` SFCs, check out the [script setup docs](https://v3.vuejs.org/api/sfc-script-setup.html#sfc-script-setup) to learn more.
-
-Learn more about IDE Support for Vue in the [Vue Docs Scaling up Guide](https://vuejs.org/guide/scaling-up/tooling.html#ide-support).

package/dashboard/index.html

@@ -1,14 +0,0 @@
-<!doctype html>
-<html lang="en">
-  <head>
-    <meta charset="UTF-8" />
-    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
-    <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
-    <title>Queen Dashboard - Message Queue Management</title>
-    <meta name="description" content="Real-time monitoring and management dashboard for Queen Message Queue System">
-  </head>
-  <body>
-    <div id="app"></div>
-    <script type="module" src="/src/main.js"></script>
-  </body>
-</html>