queen-mq 0.1.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/API.md +862 -752
- package/AUTH.md +2044 -0
- package/LICENSE.md +202 -0
- package/README.md +1705 -1051
- package/WEBAPP.md +1889 -0
- package/assets/dashboard-01.png +0 -0
- package/assets/queen-logo-blue.svg +210 -0
- package/assets/queen-logo-cyan.svg +210 -0
- package/assets/queen-logo-indigo.svg +210 -0
- package/assets/queen-logo-orange.svg +210 -0
- package/assets/queen-logo-pink.svg +210 -0
- package/assets/queen-logo-purple.svg +210 -0
- package/assets/queen-logo-rose.svg +239 -0
- package/assets/queen-logo.svg +263 -0
- package/examples/batch-processing.js +58 -0
- package/examples/test-complete-client.js +260 -0
- package/examples/test-dashboard-api.js +200 -0
- package/examples/test-traceid.js +147 -0
- package/package.json +17 -4
- package/server.log +1 -0
- package/src/benchmark/consumer.js +207 -0
- package/src/benchmark/consumer_multi.js +216 -0
- package/src/benchmark/producer.js +75 -0
- package/src/benchmark/producer_multi.js +115 -0
- package/src/client/client.js +300 -31
- package/src/client/queenClient.js +5 -0
- package/src/cluster-server.js +242 -0
- package/src/config.js +19 -5
- package/src/database/connection.js +42 -16
- package/src/database/poolManager.js +7 -0
- package/src/database/schema-v2.sql +194 -130
- package/src/managers/queueManagerOptimized.js +823 -933
- package/src/managers/systemEventManager.js +8 -3
- package/src/routes/messages.js +127 -57
- package/src/routes/pop.js +27 -43
- package/src/routes/resources.js +61 -27
- package/src/routes/status.js +1037 -0
- package/src/server.js +308 -272
- package/src/services/evictionService.js +57 -28
- package/src/services/retentionService.js +44 -11
- package/src/test/MIGRATION_ISSUES.md +174 -0
- package/src/test/README.md +203 -0
- package/src/test/advanced-pattern-tests.js +1137 -0
- package/src/test/bus-mode-tests.js +361 -0
- package/src/test/core-tests.js +342 -0
- package/src/test/edge-case-tests.js +561 -0
- package/src/test/enterprise-tests.js +637 -0
- package/src/test/partition-locking-tests.js +545 -0
- package/src/test/test-new.js +278 -0
- package/src/test/test.js +6 -3
- package/src/test/utils.js +169 -0
- package/src/utils/streaming.js +231 -0
- package/src/utils/uuid.js +2 -2
- package/src/websocket/wsServer.js +10 -3
- package/test-keepalive-v2.sh +22 -0
- package/webapp/COLOR_GUIDE.md +118 -0
- package/webapp/README.md +143 -0
- package/webapp/index.html +14 -0
- package/webapp/package-lock.json +3184 -0
- package/webapp/package.json +25 -0
- package/webapp/postcss.config.js +7 -0
- package/webapp/public/assets/queen-logo-blue.svg +210 -0
- package/webapp/public/assets/queen-logo-cyan.svg +210 -0
- package/webapp/public/assets/queen-logo-indigo.svg +210 -0
- package/webapp/public/assets/queen-logo-orange.svg +210 -0
- package/webapp/public/assets/queen-logo-pink.svg +210 -0
- package/webapp/public/assets/queen-logo-purple.svg +210 -0
- package/webapp/public/assets/queen-logo-rose.svg +239 -0
- package/webapp/public/assets/queen-logo.svg +263 -0
- package/webapp/src/App.vue +19 -0
- package/webapp/src/api/analytics.js +10 -0
- package/webapp/src/api/client.js +29 -0
- package/webapp/src/api/consumers.js +52 -0
- package/webapp/src/api/health.js +7 -0
- package/webapp/src/api/messages.js +26 -0
- package/webapp/src/api/queues.js +14 -0
- package/webapp/src/api/resources.js +8 -0
- package/webapp/src/assets/styles/main.css +357 -0
- package/webapp/src/components/analytics/AnalyticsFilters.vue +87 -0
- package/webapp/src/components/analytics/AnalyticsMetrics.vue +57 -0
- package/webapp/src/components/analytics/MessageDistributionChart.vue +111 -0
- package/webapp/src/components/analytics/MessageFlowChart.vue +173 -0
- package/webapp/src/components/analytics/TimeRangeSelector.vue +27 -0
- package/webapp/src/components/analytics/TopQueuesChart.vue +132 -0
- package/webapp/src/components/common/ConfirmDialog.vue +56 -0
- package/webapp/src/components/common/LoadingSpinner.vue +6 -0
- package/webapp/src/components/common/MetricCard.vue +43 -0
- package/webapp/src/components/common/StatusBadge.vue +45 -0
- package/webapp/src/components/dashboard/MessageStatusCard.vue +50 -0
- package/webapp/src/components/dashboard/PerformanceCard.vue +38 -0
- package/webapp/src/components/dashboard/ThroughputChart.vue +182 -0
- package/webapp/src/components/dashboard/TopQueuesTable.vue +53 -0
- package/webapp/src/components/layout/AppLayout.vue +110 -0
- package/webapp/src/components/layout/AppSidebar.vue +304 -0
- package/webapp/src/components/messages/MessageDetailPanel.vue +242 -0
- package/webapp/src/components/messages/MessageFilters.vue +114 -0
- package/webapp/src/components/queue-detail/PartitionList.vue +79 -0
- package/webapp/src/components/queue-detail/PushMessageModal.vue +175 -0
- package/webapp/src/components/queue-detail/QueueConfig.vue +63 -0
- package/webapp/src/components/queue-detail/QueueDetailHeader.vue +53 -0
- package/webapp/src/components/queue-detail/RecentMessages.vue +76 -0
- package/webapp/src/components/queues/CreateQueueModal.vue +193 -0
- package/webapp/src/components/queues/QueueFilters.vue +90 -0
- package/webapp/src/composables/useApi.js +34 -0
- package/webapp/src/composables/useTheme.js +36 -0
- package/webapp/src/main.js +11 -0
- package/webapp/src/router/index.js +42 -0
- package/webapp/src/utils/colors.js +96 -0
- package/webapp/src/utils/formatters.js +49 -0
- package/webapp/src/views/Analytics.vue +377 -0
- package/webapp/src/views/ConsumerGroups.vue +433 -0
- package/webapp/src/views/Dashboard.vue +418 -0
- package/webapp/src/views/Messages.vue +361 -0
- package/webapp/src/views/QueueDetail.vue +582 -0
- package/webapp/src/views/Queues.vue +496 -0
- package/webapp/tailwind.config.js +25 -0
- package/webapp/vite.config.js +10 -0
- package/CACHE.md +0 -519
- package/DASHBOARD-V3.md +0 -478
- package/DASHBOARD.md +0 -382
- package/MOD_QUEUE.md +0 -453
- package/PARTITION_LOCKING_DESIGN.md +0 -989
- package/PLAN.md +0 -707
- package/QUERY_ANALSYS.md +0 -72
- package/QUEUE_BUS.md +0 -334
- package/V2-PLAN.md +0 -236
- package/dashboard/.vscode/extensions.json +0 -3
- package/dashboard/README.md +0 -5
- package/dashboard/index.html +0 -14
- package/dashboard/package-lock.json +0 -1458
- package/dashboard/package.json +0 -25
- package/dashboard/public/vite.svg +0 -1
- package/dashboard/src/App.vue +0 -29
- package/dashboard/src/assets/styles/main.css +0 -908
- package/dashboard/src/assets/vue.svg +0 -1
- package/dashboard/src/components/cards/MetricCard.vue +0 -298
- package/dashboard/src/components/charts/QueueDepthChart.vue +0 -276
- package/dashboard/src/components/charts/QueueLagChart.vue +0 -436
- package/dashboard/src/components/charts/ThroughputChart.vue +0 -302
- package/dashboard/src/components/common/ActivityFeed.vue +0 -251
- package/dashboard/src/components/layout/AppHeader.vue +0 -208
- package/dashboard/src/components/layout/AppLayout.vue +0 -88
- package/dashboard/src/components/layout/AppSidebar.vue +0 -261
- package/dashboard/src/main.js +0 -44
- package/dashboard/src/router.js +0 -54
- package/dashboard/src/services/api.js +0 -187
- package/dashboard/src/services/websocket.js +0 -167
- package/dashboard/src/utils/constants.js +0 -56
- package/dashboard/src/utils/helpers.js +0 -118
- package/dashboard/src/views/Analytics.vue +0 -912
- package/dashboard/src/views/Dashboard.vue +0 -906
- package/dashboard/src/views/Messages.vue +0 -437
- package/dashboard/src/views/QueueDetail.vue +0 -501
- package/dashboard/src/views/Queues.vue +0 -333
- package/dashboard/vite.config.js +0 -30
- package/debug-namespace.js +0 -110
- package/docs/long-polling.md +0 -159
- package/docs/multi-server-cache-solutions.md +0 -185
- package/docs/performance-tuning.md +0 -222
- package/src/routes/analytics.js +0 -812
package/PLAN.md
DELETED
|
@@ -1,707 +0,0 @@
|
|
|
1
|
-
# Queen Message Queue System - Implementation Plan
|
|
2
|
-
|
|
3
|
-
## Overview
|
|
4
|
-
High-performance message queue system with hierarchical organization (namespace → task → queue → message) backed by PostgreSQL and uWebSockets.js, using functional programming patterns.
|
|
5
|
-
|
|
6
|
-
## Architecture
|
|
7
|
-
|
|
8
|
-
```
|
|
9
|
-
Client SDK → HTTP → uWS Routes → Queue Manager → PostgreSQL
|
|
10
|
-
↓
|
|
11
|
-
Long Polling + Events
|
|
12
|
-
```
|
|
13
|
-
|
|
14
|
-
## Project Structure
|
|
15
|
-
|
|
16
|
-
```
|
|
17
|
-
/src/
|
|
18
|
-
├── server.js # Main uWS server setup (HTTP + WebSocket)
|
|
19
|
-
├── config/
|
|
20
|
-
│ ├── database.js # PostgreSQL connection config
|
|
21
|
-
│ └── server.js # Server configuration
|
|
22
|
-
├── database/
|
|
23
|
-
│ ├── schema.sql # Database schema (queen schema)
|
|
24
|
-
│ ├── migrations/ # Schema migrations
|
|
25
|
-
│ └── connection.js # DB connection pool
|
|
26
|
-
├── managers/
|
|
27
|
-
│ ├── queueManager.js # Core queue processing logic (factory)
|
|
28
|
-
│ ├── eventManager.js # Event emitter for long polling (factory)
|
|
29
|
-
│ ├── resourceCache.js # In-memory resource cache (factory)
|
|
30
|
-
│ └── transactionManager.js # Transaction ID deduplication (factory)
|
|
31
|
-
├── routes/
|
|
32
|
-
│ ├── configure.js # POST /api/v1/configure
|
|
33
|
-
│ ├── push.js # POST /api/v1/push
|
|
34
|
-
│ ├── pop.js # GET /api/v1/pop variants
|
|
35
|
-
│ └── analytics.js # Analytics endpoints
|
|
36
|
-
├── websocket/
|
|
37
|
-
│ ├── wsServer.js # WebSocket server for dashboard events
|
|
38
|
-
│ ├── handlers.js # WebSocket message handlers
|
|
39
|
-
│ └── broadcaster.js # Event broadcasting to connected clients
|
|
40
|
-
├── services/
|
|
41
|
-
│ ├── namespaceService.js # Namespace operations (factory)
|
|
42
|
-
│ ├── taskService.js # Task operations (factory)
|
|
43
|
-
│ ├── queueService.js # Queue operations (factory)
|
|
44
|
-
│ └── messageService.js # Message operations (factory)
|
|
45
|
-
├── utils/
|
|
46
|
-
│ ├── uuid.js # UUID v4 generation
|
|
47
|
-
│ ├── validation.js # Request validation
|
|
48
|
-
│ ├── errors.js # Error handling
|
|
49
|
-
│ └── functional.js # Functional utilities (pipe, compose)
|
|
50
|
-
├── middleware/
|
|
51
|
-
│ ├── cors.js # CORS handling
|
|
52
|
-
│ ├── logging.js # Request logging
|
|
53
|
-
│ └── rateLimit.js # Rate limiting
|
|
54
|
-
└── client/ # Client SDK
|
|
55
|
-
├── index.js # Client SDK entry point
|
|
56
|
-
├── queenClient.js # Main client factory function
|
|
57
|
-
├── api/
|
|
58
|
-
│ ├── configure.js # Configure API calls
|
|
59
|
-
│ ├── push.js # Push API calls
|
|
60
|
-
│ ├── pop.js # Pop API calls with long polling
|
|
61
|
-
│ └── analytics.js # Analytics API calls
|
|
62
|
-
└── utils/
|
|
63
|
-
├── http.js # HTTP client wrapper
|
|
64
|
-
└── retry.js # Retry logic for failed requests
|
|
65
|
-
```
|
|
66
|
-
|
|
67
|
-
## Database Schema
|
|
68
|
-
|
|
69
|
-
### Tables Hierarchy
|
|
70
|
-
```
|
|
71
|
-
namespaces (ns)
|
|
72
|
-
└── tasks
|
|
73
|
-
└── queues
|
|
74
|
-
└── messages
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
### Schema Design
|
|
78
|
-
```sql
|
|
79
|
-
-- Create queen schema
|
|
80
|
-
CREATE SCHEMA IF NOT EXISTS queen;
|
|
81
|
-
|
|
82
|
-
-- Namespaces table
|
|
83
|
-
CREATE TABLE queen.namespaces (
|
|
84
|
-
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
85
|
-
name VARCHAR(255) UNIQUE NOT NULL,
|
|
86
|
-
created_at TIMESTAMP DEFAULT NOW(),
|
|
87
|
-
updated_at TIMESTAMP DEFAULT NOW()
|
|
88
|
-
);
|
|
89
|
-
|
|
90
|
-
-- Tasks table
|
|
91
|
-
CREATE TABLE queen.tasks (
|
|
92
|
-
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
93
|
-
namespace_id UUID REFERENCES queen.namespaces(id) ON DELETE CASCADE,
|
|
94
|
-
name VARCHAR(255) NOT NULL,
|
|
95
|
-
created_at TIMESTAMP DEFAULT NOW(),
|
|
96
|
-
updated_at TIMESTAMP DEFAULT NOW(),
|
|
97
|
-
UNIQUE(namespace_id, name)
|
|
98
|
-
);
|
|
99
|
-
|
|
100
|
-
-- Queues table
|
|
101
|
-
CREATE TABLE queen.queues (
|
|
102
|
-
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
103
|
-
task_id UUID REFERENCES queen.tasks(id) ON DELETE CASCADE,
|
|
104
|
-
name VARCHAR(255) NOT NULL,
|
|
105
|
-
priority INTEGER DEFAULT 0, -- Queue priority (higher = processed first)
|
|
106
|
-
options JSONB DEFAULT '{}', -- Includes leaseTime, delayedProcessing, windowBuffer
|
|
107
|
-
created_at TIMESTAMP DEFAULT NOW(),
|
|
108
|
-
updated_at TIMESTAMP DEFAULT NOW(),
|
|
109
|
-
UNIQUE(task_id, name)
|
|
110
|
-
);
|
|
111
|
-
|
|
112
|
-
-- Messages table
|
|
113
|
-
CREATE TABLE queen.messages (
|
|
114
|
-
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
115
|
-
transaction_id UUID UNIQUE NOT NULL, -- For idempotency
|
|
116
|
-
queue_id UUID REFERENCES queen.queues(id) ON DELETE CASCADE,
|
|
117
|
-
payload JSONB NOT NULL,
|
|
118
|
-
status VARCHAR(20) DEFAULT 'pending',
|
|
119
|
-
worker_id VARCHAR(255),
|
|
120
|
-
created_at TIMESTAMP DEFAULT NOW(),
|
|
121
|
-
locked_at TIMESTAMP,
|
|
122
|
-
completed_at TIMESTAMP,
|
|
123
|
-
failed_at TIMESTAMP,
|
|
124
|
-
error_message TEXT,
|
|
125
|
-
retry_count INTEGER DEFAULT 0,
|
|
126
|
-
lease_expires_at TIMESTAMP -- For lease-based processing
|
|
127
|
-
);
|
|
128
|
-
|
|
129
|
-
-- Transaction tracking for idempotency
|
|
130
|
-
CREATE TABLE queen.transactions (
|
|
131
|
-
transaction_id UUID PRIMARY KEY,
|
|
132
|
-
message_id UUID REFERENCES queen.messages(id),
|
|
133
|
-
operation VARCHAR(20) NOT NULL, -- 'push' or 'pop'
|
|
134
|
-
created_at TIMESTAMP DEFAULT NOW(),
|
|
135
|
-
expires_at TIMESTAMP DEFAULT NOW() + INTERVAL '24 hours'
|
|
136
|
-
);
|
|
137
|
-
|
|
138
|
-
-- Queue processing coordination
|
|
139
|
-
CREATE TABLE queen.queue_processors (
|
|
140
|
-
queue_path VARCHAR(500) PRIMARY KEY, -- ns/task/queue format
|
|
141
|
-
worker_id VARCHAR(255) NOT NULL,
|
|
142
|
-
claimed_at TIMESTAMP DEFAULT NOW(),
|
|
143
|
-
last_activity TIMESTAMP DEFAULT NOW(),
|
|
144
|
-
messages_processed INTEGER DEFAULT 0
|
|
145
|
-
);
|
|
146
|
-
|
|
147
|
-
-- Analytics/metrics table
|
|
148
|
-
CREATE TABLE queen.queue_metrics (
|
|
149
|
-
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
150
|
-
queue_id UUID REFERENCES queen.queues(id),
|
|
151
|
-
metric_type VARCHAR(50), -- 'depth', 'throughput', 'latency'
|
|
152
|
-
value NUMERIC,
|
|
153
|
-
timestamp TIMESTAMP DEFAULT NOW()
|
|
154
|
-
);
|
|
155
|
-
|
|
156
|
-
-- WebSocket connections tracking
|
|
157
|
-
CREATE TABLE queen.ws_connections (
|
|
158
|
-
connection_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
159
|
-
client_id VARCHAR(255),
|
|
160
|
-
connected_at TIMESTAMP DEFAULT NOW(),
|
|
161
|
-
last_ping TIMESTAMP DEFAULT NOW(),
|
|
162
|
-
metadata JSONB DEFAULT '{}'
|
|
163
|
-
);
|
|
164
|
-
```
|
|
165
|
-
|
|
166
|
-
### Indexes
|
|
167
|
-
```sql
|
|
168
|
-
-- Performance indexes
|
|
169
|
-
CREATE INDEX idx_messages_queue_status_created ON queen.messages(queue_id, status, created_at);
|
|
170
|
-
CREATE INDEX idx_messages_status_created ON queen.messages(status, created_at);
|
|
171
|
-
CREATE INDEX idx_messages_transaction_id ON queen.messages(transaction_id);
|
|
172
|
-
CREATE INDEX idx_messages_lease_expires ON queen.messages(lease_expires_at) WHERE status = 'processing';
|
|
173
|
-
CREATE INDEX idx_transactions_expires ON queen.transactions(expires_at);
|
|
174
|
-
CREATE INDEX idx_queue_processors_activity ON queen.queue_processors(last_activity);
|
|
175
|
-
CREATE INDEX idx_namespaces_name ON queen.namespaces(name);
|
|
176
|
-
CREATE INDEX idx_tasks_namespace_name ON queen.tasks(namespace_id, name);
|
|
177
|
-
CREATE INDEX idx_queues_task_name ON queen.queues(task_id, name);
|
|
178
|
-
CREATE INDEX idx_queues_priority ON queen.queues(priority DESC);
|
|
179
|
-
CREATE INDEX idx_ws_connections_last_ping ON queen.ws_connections(last_ping);
|
|
180
|
-
```
|
|
181
|
-
|
|
182
|
-
## API Endpoints
|
|
183
|
-
|
|
184
|
-
### Configuration
|
|
185
|
-
```
|
|
186
|
-
POST /api/v1/configure
|
|
187
|
-
Body: {
|
|
188
|
-
"ns": "email-service",
|
|
189
|
-
"task": "send-notifications",
|
|
190
|
-
"queue": "high-priority",
|
|
191
|
-
"options": {
|
|
192
|
-
"leaseTime": 300, // Seconds a message is leased to worker
|
|
193
|
-
"maxSize": 10000,
|
|
194
|
-
"ttl": 3600,
|
|
195
|
-
"retryLimit": 3,
|
|
196
|
-
"deadLetterQueue": true,
|
|
197
|
-
"delayedProcessing": 60, // Don't process messages until 60s old
|
|
198
|
-
"windowBuffer": 30, // Wait until last message is 30s old
|
|
199
|
-
"priority": 10 // Higher priority queues processed first
|
|
200
|
-
}
|
|
201
|
-
}
|
|
202
|
-
Response: 201 Created
|
|
203
|
-
```
|
|
204
|
-
|
|
205
|
-
### Push Messages
|
|
206
|
-
```
|
|
207
|
-
POST /api/v1/push
|
|
208
|
-
Body: {
|
|
209
|
-
"items": [
|
|
210
|
-
{
|
|
211
|
-
"ns": "email-service",
|
|
212
|
-
"task": "send-notifications",
|
|
213
|
-
"queue": "high-priority",
|
|
214
|
-
"payload": { "to": "user@example.com", "template": "welcome" },
|
|
215
|
-
"transactionId": "550e8400-e29b-41d4-a716-446655440000" // Optional, auto-generated if not provided
|
|
216
|
-
}
|
|
217
|
-
],
|
|
218
|
-
"config": {
|
|
219
|
-
"batchMode": true,
|
|
220
|
-
"autoCreate": true // Auto-create missing ns/task/queue
|
|
221
|
-
}
|
|
222
|
-
}
|
|
223
|
-
Response: 201 Created
|
|
224
|
-
{
|
|
225
|
-
"messages": [
|
|
226
|
-
{
|
|
227
|
-
"id": "msg-uuid",
|
|
228
|
-
"transactionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
229
|
-
"status": "queued"
|
|
230
|
-
}
|
|
231
|
-
]
|
|
232
|
-
}
|
|
233
|
-
|
|
234
|
-
Note:
|
|
235
|
-
- Resources (ns/task/queue) are auto-created if they don't exist
|
|
236
|
-
- transactionId ensures idempotency - duplicate pushes with same ID are ignored
|
|
237
|
-
```
|
|
238
|
-
|
|
239
|
-
### Pop Messages (Long Polling)
|
|
240
|
-
```
|
|
241
|
-
GET /api/v1/pop/ns/email-service?timeout=30000&wait=true
|
|
242
|
-
GET /api/v1/pop/ns/email-service/task/send-notifications?timeout=30000&wait=true
|
|
243
|
-
GET /api/v1/pop/ns/email-service/task/send-notifications/queue/high-priority?timeout=30000&wait=true
|
|
244
|
-
|
|
245
|
-
Query Parameters:
|
|
246
|
-
- timeout: Max wait time in ms (default: 30000, max: 60000)
|
|
247
|
-
- wait: Enable long polling (default: false)
|
|
248
|
-
- batch: Number of messages to return (default: 1, max: 100)
|
|
249
|
-
|
|
250
|
-
Response: 200 OK
|
|
251
|
-
{
|
|
252
|
-
"messages": [
|
|
253
|
-
{
|
|
254
|
-
"id": "msg-uuid",
|
|
255
|
-
"transactionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
256
|
-
"payload": { "to": "user@example.com", "template": "welcome" },
|
|
257
|
-
"queue": "email-service/send-notifications/high-priority",
|
|
258
|
-
"createdAt": "2025-10-07T12:00:00Z"
|
|
259
|
-
}
|
|
260
|
-
]
|
|
261
|
-
}
|
|
262
|
-
or 204 No Content on timeout
|
|
263
|
-
|
|
264
|
-
Note:
|
|
265
|
-
- Batch returns messages from the same scope, respecting queue priorities
|
|
266
|
-
- Each message includes its transactionId for acknowledgment
|
|
267
|
-
- Messages are marked as 'processing' with lease time
|
|
268
|
-
```
|
|
269
|
-
|
|
270
|
-
### Analytics
|
|
271
|
-
```
|
|
272
|
-
GET /api/v1/analytics/queues # All queue stats
|
|
273
|
-
GET /api/v1/analytics/ns/:nsId # Namespace stats
|
|
274
|
-
GET /api/v1/analytics/ns/:nsId/task/:taskId # Task stats
|
|
275
|
-
GET /api/v1/analytics/queue-depths # Current queue depths
|
|
276
|
-
GET /api/v1/analytics/throughput # Messages/second metrics
|
|
277
|
-
```
|
|
278
|
-
|
|
279
|
-
### Message Acknowledgment
|
|
280
|
-
```
|
|
281
|
-
POST /api/v1/ack
|
|
282
|
-
Body: {
|
|
283
|
-
"transactionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
284
|
-
"status": "completed" // or "failed"
|
|
285
|
-
"error": "Optional error message if failed"
|
|
286
|
-
}
|
|
287
|
-
Response: 200 OK
|
|
288
|
-
{
|
|
289
|
-
"transactionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
290
|
-
"status": "completed",
|
|
291
|
-
"completedAt": "2025-10-07T12:00:00Z"
|
|
292
|
-
}
|
|
293
|
-
|
|
294
|
-
Note:
|
|
295
|
-
- Must ACK within leaseTime or message returns to pending
|
|
296
|
-
- Failed messages may be retried based on queue configuration
|
|
297
|
-
```
|
|
298
|
-
|
|
299
|
-
### Batch Acknowledgment
|
|
300
|
-
```
|
|
301
|
-
POST /api/v1/ack/batch
|
|
302
|
-
Body: {
|
|
303
|
-
"acknowledgments": [
|
|
304
|
-
{
|
|
305
|
-
"transactionId": "550e8400-e29b-41d4-a716-446655440000",
|
|
306
|
-
"status": "completed"
|
|
307
|
-
},
|
|
308
|
-
{
|
|
309
|
-
"transactionId": "660e8400-e29b-41d4-a716-446655440001",
|
|
310
|
-
"status": "failed",
|
|
311
|
-
"error": "Connection timeout"
|
|
312
|
-
}
|
|
313
|
-
]
|
|
314
|
-
}
|
|
315
|
-
Response: 200 OK
|
|
316
|
-
{
|
|
317
|
-
"processed": 2,
|
|
318
|
-
"results": [
|
|
319
|
-
{ "transactionId": "550e8400...", "status": "completed" },
|
|
320
|
-
{ "transactionId": "660e8400...", "status": "failed", "retryScheduled": true }
|
|
321
|
-
]
|
|
322
|
-
}
|
|
323
|
-
```
|
|
324
|
-
|
|
325
|
-
### WebSocket Events (Dashboard)
|
|
326
|
-
```
|
|
327
|
-
WS /ws/dashboard
|
|
328
|
-
|
|
329
|
-
Events emitted to connected dashboard clients:
|
|
330
|
-
- message.pushed: { queue, transactionId, timestamp }
|
|
331
|
-
- message.processing: { queue, transactionId, workerId, timestamp }
|
|
332
|
-
- message.completed: { queue, transactionId, timestamp }
|
|
333
|
-
- message.failed: { queue, transactionId, error, timestamp }
|
|
334
|
-
- queue.created: { ns, task, queue, timestamp }
|
|
335
|
-
- queue.depth: { queue, depth, timestamp }
|
|
336
|
-
- worker.connected: { workerId, timestamp }
|
|
337
|
-
- worker.disconnected: { workerId, timestamp }
|
|
338
|
-
```
|
|
339
|
-
|
|
340
|
-
## Core Components (Functional Style)
|
|
341
|
-
|
|
342
|
-
### queueManager Factory
|
|
343
|
-
```javascript
|
|
344
|
-
// Factory function returns queue processing functions
|
|
345
|
-
export const createQueueManager = (db, eventManager, cache) => ({
|
|
346
|
-
getNextMessage: async (scope) => { /* ... */ },
|
|
347
|
-
processQueues: async () => { /* ... */ },
|
|
348
|
-
handleDelayedProcessing: async () => { /* ... */ },
|
|
349
|
-
handleWindowBuffer: async () => { /* ... */ },
|
|
350
|
-
checkLeaseExpiry: async () => { /* ... */ }
|
|
351
|
-
})
|
|
352
|
-
```
|
|
353
|
-
|
|
354
|
-
### eventManager Factory
|
|
355
|
-
```javascript
|
|
356
|
-
// Factory for event coordination
|
|
357
|
-
export const createEventManager = () => ({
|
|
358
|
-
emit: (event, data) => { /* ... */ },
|
|
359
|
-
on: (event, handler) => { /* ... */ },
|
|
360
|
-
once: (event, handler) => { /* ... */ },
|
|
361
|
-
removeListener: (event, handler) => { /* ... */ }
|
|
362
|
-
})
|
|
363
|
-
```
|
|
364
|
-
|
|
365
|
-
### resourceCache Factory
|
|
366
|
-
```javascript
|
|
367
|
-
// In-memory cache for resource existence
|
|
368
|
-
export const createResourceCache = () => ({
|
|
369
|
-
checkResource: async (ns, task, queue) => { /* ... */ },
|
|
370
|
-
cacheResource: (ns, task, queue) => { /* ... */ },
|
|
371
|
-
invalidate: (ns, task, queue) => { /* ... */ }
|
|
372
|
-
})
|
|
373
|
-
```
|
|
374
|
-
|
|
375
|
-
### Route Handlers (Functional)
|
|
376
|
-
- Pure functions for request handling
|
|
377
|
-
- Composable middleware using pipe/compose
|
|
378
|
-
- Immutable request/response transformations
|
|
379
|
-
- Functional error handling with Either/Result patterns
|
|
380
|
-
|
|
381
|
-
## Implementation Phases
|
|
382
|
-
|
|
383
|
-
### Phase 1: Core Infrastructure (Week 1)
|
|
384
|
-
- [ ] Database schema and migrations
|
|
385
|
-
- [ ] Basic uWS server setup with routing
|
|
386
|
-
- [ ] PostgreSQL connection pool
|
|
387
|
-
- [ ] Basic CRUD operations for ns/task/queue/message
|
|
388
|
-
- [ ] UUID generation and validation utilities
|
|
389
|
-
|
|
390
|
-
### Phase 2: Queue Processing (Week 2)
|
|
391
|
-
- [ ] OptimalQueueManager implementation
|
|
392
|
-
- [ ] Background message processing loop
|
|
393
|
-
- [ ] Queue coordination and locking
|
|
394
|
-
- [ ] Message status transitions (pending → processing → completed/failed)
|
|
395
|
-
- [ ] Worker heartbeat and cleanup mechanisms
|
|
396
|
-
|
|
397
|
-
### Phase 3: HTTP API (Week 3)
|
|
398
|
-
- [ ] Configure endpoint (create ns/task/queue)
|
|
399
|
-
- [ ] Push endpoint (insert messages)
|
|
400
|
-
- [ ] Basic pop endpoint (immediate fetch)
|
|
401
|
-
- [ ] Request validation and error handling
|
|
402
|
-
- [ ] API response formatting
|
|
403
|
-
|
|
404
|
-
### Phase 4: Long Polling & Advanced Features (Week 4)
|
|
405
|
-
- [ ] EventManager for real-time coordination
|
|
406
|
-
- [ ] Long polling implementation in pop routes
|
|
407
|
-
- [ ] Timeout handling and cleanup
|
|
408
|
-
- [ ] Connection management for concurrent requests
|
|
409
|
-
- [ ] Delayed processing queue support
|
|
410
|
-
- [ ] Window buffer queue support
|
|
411
|
-
- [ ] Lease-based message processing
|
|
412
|
-
- [ ] Performance testing and optimization
|
|
413
|
-
|
|
414
|
-
### Phase 5: Analytics & Monitoring (Week 5)
|
|
415
|
-
- [ ] Queue depth tracking
|
|
416
|
-
- [ ] Throughput metrics collection
|
|
417
|
-
- [ ] Processing latency measurements
|
|
418
|
-
- [ ] Analytics API endpoints
|
|
419
|
-
- [ ] Basic dashboard/monitoring
|
|
420
|
-
|
|
421
|
-
### Phase 6: Client SDK & Production Features (Week 6)
|
|
422
|
-
- [ ] JavaScript client SDK implementation
|
|
423
|
-
- [ ] Client connection pooling
|
|
424
|
-
- [ ] Client-side retry logic
|
|
425
|
-
- [ ] Dead letter queues
|
|
426
|
-
- [ ] Message retry logic
|
|
427
|
-
- [ ] Queue size limits and backpressure
|
|
428
|
-
- [ ] Rate limiting
|
|
429
|
-
- [ ] Comprehensive error handling
|
|
430
|
-
- [ ] Performance benchmarking
|
|
431
|
-
|
|
432
|
-
## Configuration
|
|
433
|
-
|
|
434
|
-
### Environment Variables
|
|
435
|
-
```bash
|
|
436
|
-
# Database
|
|
437
|
-
PG_USER=postgres
|
|
438
|
-
PG_HOST=localhost
|
|
439
|
-
PG_DB=postgres
|
|
440
|
-
PG_PASSWORD=postgres
|
|
441
|
-
PG_PORT=5432
|
|
442
|
-
DB_POOL_SIZE=20
|
|
443
|
-
DB_TIMEOUT=30000
|
|
444
|
-
|
|
445
|
-
# Server
|
|
446
|
-
PORT=3000
|
|
447
|
-
HOST=0.0.0.0
|
|
448
|
-
WORKER_ID=worker-${HOSTNAME}-${PID}
|
|
449
|
-
|
|
450
|
-
# Queue Processing
|
|
451
|
-
QUEUE_POLL_INTERVAL=100
|
|
452
|
-
MAX_BATCH_SIZE=100
|
|
453
|
-
WORKER_HEARTBEAT_INTERVAL=10000
|
|
454
|
-
STALE_WORKER_TIMEOUT=30000
|
|
455
|
-
|
|
456
|
-
# Long Polling
|
|
457
|
-
DEFAULT_TIMEOUT=30000
|
|
458
|
-
MAX_TIMEOUT=60000
|
|
459
|
-
MAX_CONCURRENT_POLLS=1000
|
|
460
|
-
|
|
461
|
-
# Analytics
|
|
462
|
-
METRICS_COLLECTION_INTERVAL=5000
|
|
463
|
-
METRICS_RETENTION_DAYS=30
|
|
464
|
-
|
|
465
|
-
# WebSocket
|
|
466
|
-
WS_HEARTBEAT_INTERVAL=30000
|
|
467
|
-
WS_MAX_CONNECTIONS=1000
|
|
468
|
-
```
|
|
469
|
-
|
|
470
|
-
### Queue Options Schema
|
|
471
|
-
```javascript
|
|
472
|
-
{
|
|
473
|
-
leaseTime: 300, // Seconds a message is leased to worker
|
|
474
|
-
maxSize: 10000, // Max messages in queue
|
|
475
|
-
ttl: 3600, // Message TTL in seconds
|
|
476
|
-
retryLimit: 3, // Max retry attempts
|
|
477
|
-
retryDelay: 1000, // Delay between retries (ms)
|
|
478
|
-
deadLetterQueue: true, // Enable DLQ
|
|
479
|
-
priority: 0, // Queue priority (higher = processed first)
|
|
480
|
-
delayedProcessing: 0, // Seconds to wait before processing (0 = immediate)
|
|
481
|
-
windowBuffer: 0 // Seconds to wait after last message before processing batch
|
|
482
|
-
}
|
|
483
|
-
```
|
|
484
|
-
|
|
485
|
-
## Performance Targets
|
|
486
|
-
|
|
487
|
-
- **Throughput**: 10,000+ messages/second
|
|
488
|
-
- **Latency**: < 10ms for immediate pop, < 100ms for long polling response
|
|
489
|
-
- **Concurrency**: 1,000+ concurrent long polling connections
|
|
490
|
-
- **Memory**: < 512MB for 1M queued messages
|
|
491
|
-
- **CPU**: < 50% on 4-core system at target throughput
|
|
492
|
-
|
|
493
|
-
## Testing Strategy
|
|
494
|
-
|
|
495
|
-
### Unit Tests
|
|
496
|
-
- Database operations (CRUD)
|
|
497
|
-
- Queue processing logic
|
|
498
|
-
- Event management
|
|
499
|
-
- Utility functions
|
|
500
|
-
|
|
501
|
-
### Integration Tests
|
|
502
|
-
- End-to-end API workflows
|
|
503
|
-
- Long polling behavior
|
|
504
|
-
- Multi-worker coordination
|
|
505
|
-
- Database transaction handling
|
|
506
|
-
|
|
507
|
-
### Performance Tests
|
|
508
|
-
- Load testing with artillery/k6
|
|
509
|
-
- Concurrent connection limits
|
|
510
|
-
- Memory leak detection
|
|
511
|
-
- Database performance under load
|
|
512
|
-
|
|
513
|
-
### Chaos Tests
|
|
514
|
-
- Worker failure scenarios
|
|
515
|
-
- Database connection loss
|
|
516
|
-
- Network partitions
|
|
517
|
-
- High contention situations
|
|
518
|
-
|
|
519
|
-
## Deployment Considerations
|
|
520
|
-
|
|
521
|
-
### Docker Setup
|
|
522
|
-
- Multi-stage build for production
|
|
523
|
-
- PostgreSQL container for development
|
|
524
|
-
- Health checks and graceful shutdown
|
|
525
|
-
- Resource limits and monitoring
|
|
526
|
-
|
|
527
|
-
### Production Deployment
|
|
528
|
-
- Horizontal scaling with load balancer
|
|
529
|
-
- Database connection pooling
|
|
530
|
-
- Monitoring and alerting
|
|
531
|
-
- Backup and recovery procedures
|
|
532
|
-
|
|
533
|
-
## Client SDK Design
|
|
534
|
-
|
|
535
|
-
### Installation
|
|
536
|
-
```javascript
|
|
537
|
-
npm install queen-client
|
|
538
|
-
```
|
|
539
|
-
|
|
540
|
-
### Basic Usage
|
|
541
|
-
```javascript
|
|
542
|
-
import { createQueenClient } from 'queen-client'
|
|
543
|
-
|
|
544
|
-
const client = createQueenClient({
|
|
545
|
-
baseUrl: 'http://localhost:3000',
|
|
546
|
-
timeout: 30000,
|
|
547
|
-
retryAttempts: 3
|
|
548
|
-
})
|
|
549
|
-
|
|
550
|
-
// Configure queue
|
|
551
|
-
await client.configure({
|
|
552
|
-
ns: 'email-service',
|
|
553
|
-
task: 'notifications',
|
|
554
|
-
queue: 'high-priority',
|
|
555
|
-
options: {
|
|
556
|
-
leaseTime: 300,
|
|
557
|
-
delayedProcessing: 60,
|
|
558
|
-
windowBuffer: 30,
|
|
559
|
-
priority: 10
|
|
560
|
-
}
|
|
561
|
-
})
|
|
562
|
-
|
|
563
|
-
// Push messages
|
|
564
|
-
const { messageIds } = await client.push({
|
|
565
|
-
items: [{
|
|
566
|
-
ns: 'email-service',
|
|
567
|
-
task: 'notifications',
|
|
568
|
-
queue: 'high-priority',
|
|
569
|
-
payload: { to: 'user@example.com', template: 'welcome' }
|
|
570
|
-
}]
|
|
571
|
-
})
|
|
572
|
-
|
|
573
|
-
// Pop with long polling
|
|
574
|
-
const message = await client.pop({
|
|
575
|
-
ns: 'email-service',
|
|
576
|
-
task: 'notifications',
|
|
577
|
-
queue: 'high-priority',
|
|
578
|
-
wait: true,
|
|
579
|
-
timeout: 30000
|
|
580
|
-
})
|
|
581
|
-
|
|
582
|
-
// Process message
|
|
583
|
-
try {
|
|
584
|
-
await processMessage(message)
|
|
585
|
-
// Acknowledge successful processing
|
|
586
|
-
await client.ack(message.transactionId, 'completed')
|
|
587
|
-
} catch (error) {
|
|
588
|
-
// Acknowledge failure (message may be retried)
|
|
589
|
-
await client.ack(message.transactionId, 'failed', error.message)
|
|
590
|
-
}
|
|
591
|
-
```
|
|
592
|
-
|
|
593
|
-
### Client Features
|
|
594
|
-
- Automatic retry with exponential backoff
|
|
595
|
-
- Connection pooling for HTTP/2
|
|
596
|
-
- Long polling support with automatic reconnection
|
|
597
|
-
- Batch operations for push/pop
|
|
598
|
-
- Promise-based API with async/await support
|
|
599
|
-
- Event emitter for real-time updates
|
|
600
|
-
- Built-in request/response validation
|
|
601
|
-
|
|
602
|
-
## Advanced Queue Processing
|
|
603
|
-
|
|
604
|
-
### Delayed Processing
|
|
605
|
-
Messages in queues with `delayedProcessing` are not eligible for processing until they've aged for the specified duration:
|
|
606
|
-
```sql
|
|
607
|
-
-- Only select messages older than delayedProcessing seconds
|
|
608
|
-
WHERE created_at <= NOW() - INTERVAL '60 seconds'
|
|
609
|
-
```
|
|
610
|
-
|
|
611
|
-
### Window Buffer
|
|
612
|
-
Queues with `windowBuffer` wait until the newest message in the queue is at least X seconds old before processing any messages:
|
|
613
|
-
```sql
|
|
614
|
-
-- Check if newest message is old enough
|
|
615
|
-
SELECT MAX(created_at) < NOW() - INTERVAL '30 seconds'
|
|
616
|
-
FROM messages
|
|
617
|
-
WHERE queue_id = ? AND status = 'pending'
|
|
618
|
-
```
|
|
619
|
-
|
|
620
|
-
### Lease-Based Processing
|
|
621
|
-
Messages are leased to workers for a specific duration. If not acknowledged within the lease time, they become available again:
|
|
622
|
-
```sql
|
|
623
|
-
-- Reclaim expired leases (runs periodically)
|
|
624
|
-
UPDATE queen.messages
|
|
625
|
-
SET status = 'pending',
|
|
626
|
-
worker_id = NULL,
|
|
627
|
-
lease_expires_at = NULL,
|
|
628
|
-
retry_count = retry_count + 1
|
|
629
|
-
WHERE status = 'processing'
|
|
630
|
-
AND lease_expires_at < NOW()
|
|
631
|
-
AND retry_count < (
|
|
632
|
-
SELECT (options->>'retryLimit')::int
|
|
633
|
-
FROM queen.queues
|
|
634
|
-
WHERE id = queue_id
|
|
635
|
-
);
|
|
636
|
-
|
|
637
|
-
-- Move to DLQ if retry limit exceeded
|
|
638
|
-
UPDATE queen.messages
|
|
639
|
-
SET status = 'dead_letter'
|
|
640
|
-
WHERE status = 'processing'
|
|
641
|
-
AND lease_expires_at < NOW()
|
|
642
|
-
AND retry_count >= (
|
|
643
|
-
SELECT (options->>'retryLimit')::int
|
|
644
|
-
FROM queen.queues
|
|
645
|
-
WHERE id = queue_id
|
|
646
|
-
);
|
|
647
|
-
```
|
|
648
|
-
|
|
649
|
-
### Queue Priority Processing
|
|
650
|
-
Higher priority queues are processed first:
|
|
651
|
-
```sql
|
|
652
|
-
-- Select next message respecting queue priorities
|
|
653
|
-
WITH prioritized_queues AS (
|
|
654
|
-
SELECT q.id, q.priority
|
|
655
|
-
FROM queues q
|
|
656
|
-
JOIN messages m ON m.queue_id = q.id
|
|
657
|
-
WHERE m.status = 'pending'
|
|
658
|
-
ORDER BY q.priority DESC, m.created_at
|
|
659
|
-
LIMIT 1
|
|
660
|
-
)
|
|
661
|
-
SELECT m.* FROM messages m
|
|
662
|
-
JOIN prioritized_queues pq ON m.queue_id = pq.id
|
|
663
|
-
WHERE m.status = 'pending'
|
|
664
|
-
ORDER BY m.created_at
|
|
665
|
-
LIMIT 1
|
|
666
|
-
FOR UPDATE SKIP LOCKED
|
|
667
|
-
```
|
|
668
|
-
|
|
669
|
-
## Message Durability & Ordering Guarantees
|
|
670
|
-
|
|
671
|
-
### Message Durability Guarantees
|
|
672
|
-
1. **Transactional Writes**: All message inserts are within PostgreSQL transactions
|
|
673
|
-
2. **Idempotency**: Transaction IDs prevent duplicate message insertion
|
|
674
|
-
3. **Lease-Based Processing**: Messages have lease timeouts - if not acknowledged, they return to pending
|
|
675
|
-
4. **Retry Logic**: Failed messages can be retried with configurable limits
|
|
676
|
-
5. **Dead Letter Queue**: Messages exceeding retry limits go to DLQ (not lost)
|
|
677
|
-
6. **WAL & Replication**: PostgreSQL WAL ensures durability even on crashes
|
|
678
|
-
|
|
679
|
-
### Message Ordering Guarantees
|
|
680
|
-
1. **FIFO Within Queue**: Messages are strictly ordered by `created_at` within each queue
|
|
681
|
-
2. **FOR UPDATE SKIP LOCKED**: Ensures only one worker processes a message
|
|
682
|
-
3. **Queue-Level Coordination**: Optional queue locking prevents concurrent processing
|
|
683
|
-
4. **Atomic Status Updates**: Message status changes are atomic operations
|
|
684
|
-
5. **No Out-of-Order Processing**: Delayed processing and window buffer maintain order
|
|
685
|
-
|
|
686
|
-
### Failure Scenarios Handled
|
|
687
|
-
- **Worker Crash**: Lease expires, message returns to pending
|
|
688
|
-
- **Database Connection Loss**: Transactions rollback, no partial state
|
|
689
|
-
- **Network Partition**: Client retries with same transactionId (idempotent)
|
|
690
|
-
- **Server Restart**: All pending messages preserved in PostgreSQL
|
|
691
|
-
- **Duplicate Push**: Transaction ID prevents duplicate insertion
|
|
692
|
-
- **Missing ACK**: Lease timeout returns message to queue for retry
|
|
693
|
-
- **Explicit NACK**: Client sends failed status, message retried per configuration
|
|
694
|
-
|
|
695
|
-
## Summary of Key Decisions
|
|
696
|
-
|
|
697
|
-
1. **Auto-creation**: Resources (ns/task/queue) are auto-created on push if they don't exist, with efficient in-memory caching
|
|
698
|
-
2. **Batch behavior**: Batch returns N messages from the same scope, respecting queue priorities
|
|
699
|
-
3. **No global pop**: Removed `GET /api/v1/pop` - must specify at least namespace
|
|
700
|
-
4. **Queue priority**: Queues have priority (not messages) to maintain FIFO within each queue
|
|
701
|
-
5. **No authentication**: Skipped for initial implementation
|
|
702
|
-
6. **Functional style**: Factory functions instead of classes throughout
|
|
703
|
-
7. **Advanced features**: Support for delayed processing, window buffering, and lease-based message handling
|
|
704
|
-
8. **Transaction IDs**: Every message has a transactionId for idempotency and deduplication
|
|
705
|
-
9. **WebSocket Support**: Real-time events for dashboard monitoring
|
|
706
|
-
10. **Client SDK in /src/client**: JavaScript client with long polling and retry support
|
|
707
|
-
11. **Queen Schema**: All PostgreSQL tables under `queen` schema for isolation
|