queen-mq 0.1.0 → 0.1.1

package/src/test/MIGRATION_ISSUES.md

@@ -0,0 +1,174 @@
+# Test Migration Issues and Fixes
+
+## Summary
+
+When migrating tests from the old `queenClient` interface to the new minimalist `Queen` client interface, we discovered client bugs that needed fixing. The system is transactional - ACKs immediately release partition locks when awaited.
+
+## Client Bugs Fixed
+
+### 1. Null Payload Handling in `push()`
+**Issue**: `typeof null === 'object'` is `true` in JavaScript, causing the code to try to access properties on `null` payloads.
+
+**Location**: `src/client/client.js` line 153
+
+**Fix**:
+```javascript
+// Before
+const isMessageObject = typeof item === 'object' && 
+  (item._payload || item._transactionId || item._traceId);
+
+// After  
+const isMessageObject = typeof item === 'object' && item !== null &&
+  (item._payload || item._transactionId || item._traceId);
+```
+
+**Why**: Tests correctly validate edge cases including null payloads. The client must handle them gracefully.
+
+### 2. Null Handling in `ack()`
+**Issue**: Same null-checking issue when extracting transaction ID from message object.
+
+**Location**: `src/client/client.js` line 330
+
+**Fix**:
+```javascript
+// Before
+const transactionId = typeof message === 'object' ? 
+  (message.transactionId || message.id) : 
+  message;
+
+// After
+const transactionId = typeof message === 'object' && message !== null ? 
+  (message.transactionId || message.id) : 
+  message;
+```
+
+### 3. Consumer Group Parsing for Namespace/Task Addresses
+**Issue**: The `#parseAddress()` method didn't extract consumer groups from namespace/task patterns like `namespace:billing@group-a`.
+
+**Location**: `src/client/client.js` line 77-101
+
+**Fix**: Extract `@group` portion before parsing namespace/task segments:
+```javascript
+#parseAddress(address) {
+  if (address.includes('namespace:') || address.includes('task:')) {
+    const parts = {};
+    
+    // Extract consumer group if present (after @)
+    let workingAddress = address;
+    const atIndex = address.lastIndexOf('@');
+    if (atIndex > 0) {
+      parts.consumerGroup = address.substring(atIndex + 1);
+      workingAddress = address.substring(0, atIndex);
+    }
+    
+    const segments = workingAddress.split('/');
+    for (const segment of segments) {
+      if (segment.startsWith('namespace:')) {
+        parts.namespace = segment.substring(10);
+      } else if (segment.startsWith('task:')) {
+        parts.task = segment.substring(5);
+      }
+    }
+    return parts;
+  }
+  // ...
+}
+```
+
+### 4. Defensive Filtering of Null Messages
+**Issue**: Server might return null values in the `messages` array response.
+
+**Location**: `src/client/client.js` in `take()` method
+
+**Fix**: Filter out null/undefined messages when yielding:
+```javascript
+// Yield each message (filter out any null/undefined values)
+for (const message of result.messages) {
+  if (message) { // Skip null/undefined messages
+    yield message;
+    count++;
+    if (limit && count >= limit) return;
+  }
+}
+```
+
+## Test Fixes
+
+### 1. Consumer Group Isolation Test
+**Problem**: Test was ACKing all messages as completed for both groups, but the test expects group2 to have some failed messages to verify isolation.
+
+**Solution**: Actually fail the first 2 messages for group2:
+```javascript
+// Fail first 2 messages for group2, complete the rest
+for (let i = 0; i < group2Messages.length; i++) {
+  if (i < 2) {
+    await client.ack(group2Messages[i], false, { 
+      error: 'Test failure', 
+      group: 'group2' 
+    });
+  } else {
+    await client.ack(group2Messages[i], true, { group: 'group2' });
+  }
+}
+```
+
+### 2. Empty and Null Payloads Test
+**Problem**: If server returns null message objects in the array, the test would crash.
+
+**Solution**: Add defensive check (though client now filters these):
+```javascript
+for await (const msg of client.take(queue, { limit: 10 })) {
+  if (!msg) {
+    throw new Error('Received null/undefined message from take()');
+  }
+  messages.push(msg);
+  await client.ack(msg);
+}
+```
+
+## Key Learnings
+
+### System is Transactional ✅
+
+The Queen system is **transactional**. When you `await client.ack(msg)`:
+- The ACK completes atomically
+- The partition lock is released immediately
+- No "timing issues" or delays needed
+
+### Correct Pattern
+```javascript
+// ✅ This works correctly:
+for await (const msg of client.take(queue, { limit: 10 })) {
+  await client.ack(msg); // Partition unlocked when this completes
+}
+
+// Next consumer can immediately access the partition:
+for await (const msg of client.take(queue, { limit: 10 })) {
+  // Gets messages right away
+}
+```
+
+### Edge Case Handling
+
+1. **Null payloads are valid**: Tests should validate that the system handles `null`, `undefined`, empty objects, and empty strings as payloads.
+
+2. **Client must be defensive**: Check for `typeof x === 'object' && x !== null` when you need to access properties.
+
+3. **Server responses should be filtered**: If server returns null messages in an array, filter them out client-side.
+
+### Test Design Principles
+
+1. **Test real edge cases**: null, undefined, empty strings, large payloads, etc.
+
+2. **Don't add artificial delays**: The system is transactional. If you need a delay, something is wrong.
+
+3. **Verify isolation**: When testing consumer groups, actually test that different groups can have different statuses for the same messages.
+
+4. **Use appropriate batch sizes**: Match batch size to expected message count for efficient testing.
+
+## Final Results
+
+After fixing client bugs and test issues:
+- 33/33 tests should pass
+- No timing sensitivity
+- Clean, transactional behavior throughout

package/src/test/README.md

@@ -0,0 +1,203 @@
+# Queen Message Queue Test Suite
+
+This directory contains the comprehensive test suite for the Queen Message Queue system, using the new minimalist `Queen` client interface.
+
+## Test Structure
+
+The test suite is organized into focused, modular files:
+
+### Core Files
+
+- **`test-new.js`** - Main test runner that orchestrates all tests
+- **`utils.js`** - Shared utilities (logging, database helpers, result tracking)
+
+### Test Categories
+
+#### 1. Core Features (`core-tests.js`)
+- Queue creation policy
+- Single/batch message push
+- Queue configuration
+- Take and acknowledgment
+- Delayed processing
+- FIFO ordering within partitions
+
+#### 2. Partition Locking (`partition-locking-tests.js`)
+- Partition locking in queue mode
+- Partition locking in bus mode
+- Specific partition requests with locking
+- Namespace/task filtering with partition locking
+
+#### 3. Enterprise Features (`enterprise-tests.js`)
+- Message encryption/decryption
+- Retention policies (pending & completed messages)
+- Message eviction
+- Combined enterprise features
+- Enterprise error handling
+
+#### 4. Bus Mode Features (`bus-mode-tests.js`)
+- Consumer groups
+- Mixed mode (queue + bus)
+- Subscription modes (all vs new messages)
+- Consumer group isolation
+
+#### 5. Edge Cases (`edge-case-tests.js`)
+- Empty and null payloads
+- Very large payloads
+- Concurrent push/take operations
+- Retry limit exhaustion
+- Lease expiration and redelivery
+- SQL injection prevention
+- XSS prevention
+
+#### 6. Advanced Patterns (`advanced-pattern-tests.js`)
+- Multi-stage pipeline workflow
+- Fan-out/fan-in pattern
+- Dead letter queue pattern
+- Circuit breaker pattern
+- Message deduplication
+
+## New vs Old Interface
+
+### Old Interface (test.js)
+```javascript
+import { createQueenClient } from '../client/queenClient.js';
+
+const client = createQueenClient({ baseUrls: [...] });
+
+// Configure
+await client.configure({ queue: 'myqueue', options: {} });
+
+// Push
+await client.push({ items: [{ queue: 'myqueue', partition: 'Default', payload: {...} }] });
+
+// Pop
+const result = await client.pop({ queue: 'myqueue', batch: 10 });
+
+// Ack
+await client.ack(transactionId, 'completed', null, consumerGroup);
+```
+
+### New Interface (test-new.js)
+```javascript
+import { Queen } from '../client/client.js';
+
+const client = new Queen({ baseUrls: [...] });
+
+// Configure
+await client.queue('myqueue', {}, { namespace, task });
+
+// Push
+await client.push('myqueue/partition', payload);
+
+// Take (async iterator)
+for await (const msg of client.take('myqueue/partition@group', { limit: 10 })) {
+  // Process message
+  await client.ack(msg, true, { group: 'mygroup' });
+}
+```
+
+## Key Differences
+
+1. **Address Format**: The new interface uses a unified address format:
+   - `"queue"` - Simple queue
+   - `"queue/partition"` - Specific partition
+   - `"queue@group"` - Consumer group
+   - `"queue/partition@group"` - Partition + group
+   - `"namespace:name"` - Namespace filter
+   - `"task:name"` - Task filter
+   - `"namespace:billing/task:process"` - Combined filters
+
+2. **Async Iterator**: `pop()` → `take()` using async iteration
+   ```javascript
+   for await (const msg of client.take(address, options)) {
+     // Handle message
+   }
+   ```
+
+3. **Simplified Methods**: 4 core methods instead of many:
+   - `client.queue(name, options, metadata)` - Configure
+   - `client.push(address, payload, options)` - Send
+   - `client.take(address, options)` - Receive (async iterator)
+   - `client.ack(message, status, context)` - Acknowledge
+
+4. **Message Properties**: Can be in payload or options:
+   ```javascript
+   // In options
+   await client.push('queue', { data }, { transactionId: 'txn-1' });
+   
+   // In payload
+   await client.push('queue', { data, transactionId: 'txn-1' });
+   ```
+
+## Running Tests
+
+```bash
+# Make sure Queen servers are running first
+# And database is accessible
+
+cd /Users/alice/Work/queen
+
+# Run all tests
+nvm use 22 && node src/test/test-new.js
+
+# Run specific test category
+node src/test/test-new.js core        # Core features only
+node src/test/test-new.js partition   # Partition locking tests only
+node src/test/test-new.js enterprise  # Enterprise features only
+node src/test/test-new.js bus         # Bus mode tests only
+node src/test/test-new.js edge        # Edge cases only
+node src/test/test-new.js advanced    # Advanced patterns only
+
+# Show help
+node src/test/test-new.js help
+```
+
+### Available Test Categories
+
+- **`core`** - Core features (queue creation, push, take, ack, delayed processing, FIFO)
+- **`partition`** (or `locking`) - Partition locking in queue and bus modes
+- **`enterprise`** - Enterprise features (encryption, retention, eviction)
+- **`bus`** - Bus mode features (consumer groups, mixed mode, subscription modes, isolation)
+- **`edge`** - Edge cases (null payloads, large payloads, concurrency, SQL injection, XSS)
+- **`advanced`** (or `pattern`) - Advanced patterns (pipeline, fan-out/fan-in, DLQ, circuit breaker, deduplication)
+
+## Test Configuration
+
+Tests use environment variables for configuration:
+- `PG_HOST` - PostgreSQL host (default: localhost)
+- `PG_PORT` - PostgreSQL port (default: 5432)
+- `PG_DB` - Database name (default: postgres)
+- `PG_USER` - Database user (default: postgres)
+- `PG_PASSWORD` - Database password (default: postgres)
+- `QUEEN_ENCRYPTION_KEY` - Optional encryption key for encryption tests
+
+## Adding New Tests
+
+1. Create a test function in the appropriate category file:
+   ```javascript
+   export async function testMyFeature(client) {
+     startTest('My Feature Test', 'category');
+     try {
+       // Test code here
+       passTest('Feature works correctly');
+     } catch (error) {
+       failTest(error);
+     }
+   }
+   ```
+
+2. Import and add to `test-new.js`:
+   ```javascript
+   import { testMyFeature } from './category-tests.js';
+   
+   // In runTests():
+   await runTest(() => testMyFeature(client));
+   ```
+
+## Notes
+
+- All tests automatically clean up test data before and after running
+- Tests use the `test-*`, `edge-*`, `pattern-*`, and `workflow-*` queue name prefixes
+- The test suite supports both single-server and multi-server configurations
+- Some enterprise tests (encryption, retention) are skipped if not configured
+