queen-mq 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (160) hide show
  1. package/API.md +862 -752
  2. package/AUTH.md +2044 -0
  3. package/LICENSE.md +202 -0
  4. package/README.md +1705 -1051
  5. package/WEBAPP.md +1889 -0
  6. package/assets/dashboard-01.png +0 -0
  7. package/assets/queen-logo-blue.svg +210 -0
  8. package/assets/queen-logo-cyan.svg +210 -0
  9. package/assets/queen-logo-indigo.svg +210 -0
  10. package/assets/queen-logo-orange.svg +210 -0
  11. package/assets/queen-logo-pink.svg +210 -0
  12. package/assets/queen-logo-purple.svg +210 -0
  13. package/assets/queen-logo-rose.svg +239 -0
  14. package/assets/queen-logo.svg +263 -0
  15. package/examples/batch-processing.js +58 -0
  16. package/examples/test-complete-client.js +260 -0
  17. package/examples/test-dashboard-api.js +200 -0
  18. package/examples/test-traceid.js +147 -0
  19. package/package.json +17 -4
  20. package/server.log +1 -0
  21. package/src/benchmark/consumer.js +207 -0
  22. package/src/benchmark/consumer_multi.js +216 -0
  23. package/src/benchmark/producer.js +75 -0
  24. package/src/benchmark/producer_multi.js +115 -0
  25. package/src/client/client.js +300 -31
  26. package/src/client/queenClient.js +5 -0
  27. package/src/cluster-server.js +242 -0
  28. package/src/config.js +19 -5
  29. package/src/database/connection.js +42 -16
  30. package/src/database/poolManager.js +7 -0
  31. package/src/database/schema-v2.sql +194 -130
  32. package/src/managers/queueManagerOptimized.js +823 -933
  33. package/src/managers/systemEventManager.js +8 -3
  34. package/src/routes/messages.js +127 -57
  35. package/src/routes/pop.js +27 -43
  36. package/src/routes/resources.js +61 -27
  37. package/src/routes/status.js +1037 -0
  38. package/src/server.js +308 -272
  39. package/src/services/evictionService.js +57 -28
  40. package/src/services/retentionService.js +44 -11
  41. package/src/test/MIGRATION_ISSUES.md +174 -0
  42. package/src/test/README.md +203 -0
  43. package/src/test/advanced-pattern-tests.js +1137 -0
  44. package/src/test/bus-mode-tests.js +361 -0
  45. package/src/test/core-tests.js +342 -0
  46. package/src/test/edge-case-tests.js +561 -0
  47. package/src/test/enterprise-tests.js +637 -0
  48. package/src/test/partition-locking-tests.js +545 -0
  49. package/src/test/test-new.js +278 -0
  50. package/src/test/test.js +6 -3
  51. package/src/test/utils.js +169 -0
  52. package/src/utils/streaming.js +231 -0
  53. package/src/utils/uuid.js +2 -2
  54. package/src/websocket/wsServer.js +10 -3
  55. package/test-keepalive-v2.sh +22 -0
  56. package/webapp/COLOR_GUIDE.md +118 -0
  57. package/webapp/README.md +143 -0
  58. package/webapp/index.html +14 -0
  59. package/webapp/package-lock.json +3184 -0
  60. package/webapp/package.json +25 -0
  61. package/webapp/postcss.config.js +7 -0
  62. package/webapp/public/assets/queen-logo-blue.svg +210 -0
  63. package/webapp/public/assets/queen-logo-cyan.svg +210 -0
  64. package/webapp/public/assets/queen-logo-indigo.svg +210 -0
  65. package/webapp/public/assets/queen-logo-orange.svg +210 -0
  66. package/webapp/public/assets/queen-logo-pink.svg +210 -0
  67. package/webapp/public/assets/queen-logo-purple.svg +210 -0
  68. package/webapp/public/assets/queen-logo-rose.svg +239 -0
  69. package/webapp/public/assets/queen-logo.svg +263 -0
  70. package/webapp/src/App.vue +19 -0
  71. package/webapp/src/api/analytics.js +10 -0
  72. package/webapp/src/api/client.js +29 -0
  73. package/webapp/src/api/consumers.js +52 -0
  74. package/webapp/src/api/health.js +7 -0
  75. package/webapp/src/api/messages.js +26 -0
  76. package/webapp/src/api/queues.js +14 -0
  77. package/webapp/src/api/resources.js +8 -0
  78. package/webapp/src/assets/styles/main.css +357 -0
  79. package/webapp/src/components/analytics/AnalyticsFilters.vue +87 -0
  80. package/webapp/src/components/analytics/AnalyticsMetrics.vue +57 -0
  81. package/webapp/src/components/analytics/MessageDistributionChart.vue +111 -0
  82. package/webapp/src/components/analytics/MessageFlowChart.vue +173 -0
  83. package/webapp/src/components/analytics/TimeRangeSelector.vue +27 -0
  84. package/webapp/src/components/analytics/TopQueuesChart.vue +132 -0
  85. package/webapp/src/components/common/ConfirmDialog.vue +56 -0
  86. package/webapp/src/components/common/LoadingSpinner.vue +6 -0
  87. package/webapp/src/components/common/MetricCard.vue +43 -0
  88. package/webapp/src/components/common/StatusBadge.vue +45 -0
  89. package/webapp/src/components/dashboard/MessageStatusCard.vue +50 -0
  90. package/webapp/src/components/dashboard/PerformanceCard.vue +38 -0
  91. package/webapp/src/components/dashboard/ThroughputChart.vue +182 -0
  92. package/webapp/src/components/dashboard/TopQueuesTable.vue +53 -0
  93. package/webapp/src/components/layout/AppLayout.vue +110 -0
  94. package/webapp/src/components/layout/AppSidebar.vue +304 -0
  95. package/webapp/src/components/messages/MessageDetailPanel.vue +242 -0
  96. package/webapp/src/components/messages/MessageFilters.vue +114 -0
  97. package/webapp/src/components/queue-detail/PartitionList.vue +79 -0
  98. package/webapp/src/components/queue-detail/PushMessageModal.vue +175 -0
  99. package/webapp/src/components/queue-detail/QueueConfig.vue +63 -0
  100. package/webapp/src/components/queue-detail/QueueDetailHeader.vue +53 -0
  101. package/webapp/src/components/queue-detail/RecentMessages.vue +76 -0
  102. package/webapp/src/components/queues/CreateQueueModal.vue +193 -0
  103. package/webapp/src/components/queues/QueueFilters.vue +90 -0
  104. package/webapp/src/composables/useApi.js +34 -0
  105. package/webapp/src/composables/useTheme.js +36 -0
  106. package/webapp/src/main.js +11 -0
  107. package/webapp/src/router/index.js +42 -0
  108. package/webapp/src/utils/colors.js +96 -0
  109. package/webapp/src/utils/formatters.js +49 -0
  110. package/webapp/src/views/Analytics.vue +377 -0
  111. package/webapp/src/views/ConsumerGroups.vue +433 -0
  112. package/webapp/src/views/Dashboard.vue +418 -0
  113. package/webapp/src/views/Messages.vue +361 -0
  114. package/webapp/src/views/QueueDetail.vue +582 -0
  115. package/webapp/src/views/Queues.vue +496 -0
  116. package/webapp/tailwind.config.js +25 -0
  117. package/webapp/vite.config.js +10 -0
  118. package/CACHE.md +0 -519
  119. package/DASHBOARD-V3.md +0 -478
  120. package/DASHBOARD.md +0 -382
  121. package/MOD_QUEUE.md +0 -453
  122. package/PARTITION_LOCKING_DESIGN.md +0 -989
  123. package/PLAN.md +0 -707
  124. package/QUERY_ANALSYS.md +0 -72
  125. package/QUEUE_BUS.md +0 -334
  126. package/V2-PLAN.md +0 -236
  127. package/dashboard/.vscode/extensions.json +0 -3
  128. package/dashboard/README.md +0 -5
  129. package/dashboard/index.html +0 -14
  130. package/dashboard/package-lock.json +0 -1458
  131. package/dashboard/package.json +0 -25
  132. package/dashboard/public/vite.svg +0 -1
  133. package/dashboard/src/App.vue +0 -29
  134. package/dashboard/src/assets/styles/main.css +0 -908
  135. package/dashboard/src/assets/vue.svg +0 -1
  136. package/dashboard/src/components/cards/MetricCard.vue +0 -298
  137. package/dashboard/src/components/charts/QueueDepthChart.vue +0 -276
  138. package/dashboard/src/components/charts/QueueLagChart.vue +0 -436
  139. package/dashboard/src/components/charts/ThroughputChart.vue +0 -302
  140. package/dashboard/src/components/common/ActivityFeed.vue +0 -251
  141. package/dashboard/src/components/layout/AppHeader.vue +0 -208
  142. package/dashboard/src/components/layout/AppLayout.vue +0 -88
  143. package/dashboard/src/components/layout/AppSidebar.vue +0 -261
  144. package/dashboard/src/main.js +0 -44
  145. package/dashboard/src/router.js +0 -54
  146. package/dashboard/src/services/api.js +0 -187
  147. package/dashboard/src/services/websocket.js +0 -167
  148. package/dashboard/src/utils/constants.js +0 -56
  149. package/dashboard/src/utils/helpers.js +0 -118
  150. package/dashboard/src/views/Analytics.vue +0 -912
  151. package/dashboard/src/views/Dashboard.vue +0 -906
  152. package/dashboard/src/views/Messages.vue +0 -437
  153. package/dashboard/src/views/QueueDetail.vue +0 -501
  154. package/dashboard/src/views/Queues.vue +0 -333
  155. package/dashboard/vite.config.js +0 -30
  156. package/debug-namespace.js +0 -110
  157. package/docs/long-polling.md +0 -159
  158. package/docs/multi-server-cache-solutions.md +0 -185
  159. package/docs/performance-tuning.md +0 -222
  160. package/src/routes/analytics.js +0 -812
package/PLAN.md DELETED
@@ -1,707 +0,0 @@
1
- # Queen Message Queue System - Implementation Plan
2
-
3
- ## Overview
4
- High-performance message queue system with hierarchical organization (namespace → task → queue → message) backed by PostgreSQL and uWebSockets.js, using functional programming patterns.
5
-
6
- ## Architecture
7
-
8
- ```
9
- Client SDK → HTTP → uWS Routes → Queue Manager → PostgreSQL
10
- ↓
11
- Long Polling + Events
12
- ```
13
-
14
- ## Project Structure
15
-
16
- ```
17
- /src/
18
- ├── server.js # Main uWS server setup (HTTP + WebSocket)
19
- ├── config/
20
- │ ├── database.js # PostgreSQL connection config
21
- │ └── server.js # Server configuration
22
- ├── database/
23
- │ ├── schema.sql # Database schema (queen schema)
24
- │ ├── migrations/ # Schema migrations
25
- │ └── connection.js # DB connection pool
26
- ├── managers/
27
- │ ├── queueManager.js # Core queue processing logic (factory)
28
- │ ├── eventManager.js # Event emitter for long polling (factory)
29
- │ ├── resourceCache.js # In-memory resource cache (factory)
30
- │ └── transactionManager.js # Transaction ID deduplication (factory)
31
- ├── routes/
32
- │ ├── configure.js # POST /api/v1/configure
33
- │ ├── push.js # POST /api/v1/push
34
- │ ├── pop.js # GET /api/v1/pop variants
35
- │ └── analytics.js # Analytics endpoints
36
- ├── websocket/
37
- │ ├── wsServer.js # WebSocket server for dashboard events
38
- │ ├── handlers.js # WebSocket message handlers
39
- │ └── broadcaster.js # Event broadcasting to connected clients
40
- ├── services/
41
- │ ├── namespaceService.js # Namespace operations (factory)
42
- │ ├── taskService.js # Task operations (factory)
43
- │ ├── queueService.js # Queue operations (factory)
44
- │ └── messageService.js # Message operations (factory)
45
- ├── utils/
46
- │ ├── uuid.js # UUID v4 generation
47
- │ ├── validation.js # Request validation
48
- │ ├── errors.js # Error handling
49
- │ └── functional.js # Functional utilities (pipe, compose)
50
- ├── middleware/
51
- │ ├── cors.js # CORS handling
52
- │ ├── logging.js # Request logging
53
- │ └── rateLimit.js # Rate limiting
54
- └── client/ # Client SDK
55
- ├── index.js # Client SDK entry point
56
- ├── queenClient.js # Main client factory function
57
- ├── api/
58
- │ ├── configure.js # Configure API calls
59
- │ ├── push.js # Push API calls
60
- │ ├── pop.js # Pop API calls with long polling
61
- │ └── analytics.js # Analytics API calls
62
- └── utils/
63
- ├── http.js # HTTP client wrapper
64
- └── retry.js # Retry logic for failed requests
65
- ```
66
-
67
- ## Database Schema
68
-
69
- ### Tables Hierarchy
70
- ```
71
- namespaces (ns)
72
- └── tasks
73
- └── queues
74
- └── messages
75
- ```
76
-
77
- ### Schema Design
78
- ```sql
79
- -- Create queen schema
80
- CREATE SCHEMA IF NOT EXISTS queen;
81
-
82
- -- Namespaces table
83
- CREATE TABLE queen.namespaces (
84
- id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
85
- name VARCHAR(255) UNIQUE NOT NULL,
86
- created_at TIMESTAMP DEFAULT NOW(),
87
- updated_at TIMESTAMP DEFAULT NOW()
88
- );
89
-
90
- -- Tasks table
91
- CREATE TABLE queen.tasks (
92
- id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
93
- namespace_id UUID REFERENCES queen.namespaces(id) ON DELETE CASCADE,
94
- name VARCHAR(255) NOT NULL,
95
- created_at TIMESTAMP DEFAULT NOW(),
96
- updated_at TIMESTAMP DEFAULT NOW(),
97
- UNIQUE(namespace_id, name)
98
- );
99
-
100
- -- Queues table
101
- CREATE TABLE queen.queues (
102
- id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
103
- task_id UUID REFERENCES queen.tasks(id) ON DELETE CASCADE,
104
- name VARCHAR(255) NOT NULL,
105
- priority INTEGER DEFAULT 0, -- Queue priority (higher = processed first)
106
- options JSONB DEFAULT '{}', -- Includes leaseTime, delayedProcessing, windowBuffer
107
- created_at TIMESTAMP DEFAULT NOW(),
108
- updated_at TIMESTAMP DEFAULT NOW(),
109
- UNIQUE(task_id, name)
110
- );
111
-
112
- -- Messages table
113
- CREATE TABLE queen.messages (
114
- id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
115
- transaction_id UUID UNIQUE NOT NULL, -- For idempotency
116
- queue_id UUID REFERENCES queen.queues(id) ON DELETE CASCADE,
117
- payload JSONB NOT NULL,
118
- status VARCHAR(20) DEFAULT 'pending',
119
- worker_id VARCHAR(255),
120
- created_at TIMESTAMP DEFAULT NOW(),
121
- locked_at TIMESTAMP,
122
- completed_at TIMESTAMP,
123
- failed_at TIMESTAMP,
124
- error_message TEXT,
125
- retry_count INTEGER DEFAULT 0,
126
- lease_expires_at TIMESTAMP -- For lease-based processing
127
- );
128
-
129
- -- Transaction tracking for idempotency
130
- CREATE TABLE queen.transactions (
131
- transaction_id UUID PRIMARY KEY,
132
- message_id UUID REFERENCES queen.messages(id),
133
- operation VARCHAR(20) NOT NULL, -- 'push' or 'pop'
134
- created_at TIMESTAMP DEFAULT NOW(),
135
- expires_at TIMESTAMP DEFAULT NOW() + INTERVAL '24 hours'
136
- );
137
-
138
- -- Queue processing coordination
139
- CREATE TABLE queen.queue_processors (
140
- queue_path VARCHAR(500) PRIMARY KEY, -- ns/task/queue format
141
- worker_id VARCHAR(255) NOT NULL,
142
- claimed_at TIMESTAMP DEFAULT NOW(),
143
- last_activity TIMESTAMP DEFAULT NOW(),
144
- messages_processed INTEGER DEFAULT 0
145
- );
146
-
147
- -- Analytics/metrics table
148
- CREATE TABLE queen.queue_metrics (
149
- id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
150
- queue_id UUID REFERENCES queen.queues(id),
151
- metric_type VARCHAR(50), -- 'depth', 'throughput', 'latency'
152
- value NUMERIC,
153
- timestamp TIMESTAMP DEFAULT NOW()
154
- );
155
-
156
- -- WebSocket connections tracking
157
- CREATE TABLE queen.ws_connections (
158
- connection_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
159
- client_id VARCHAR(255),
160
- connected_at TIMESTAMP DEFAULT NOW(),
161
- last_ping TIMESTAMP DEFAULT NOW(),
162
- metadata JSONB DEFAULT '{}'
163
- );
164
- ```
165
-
166
- ### Indexes
167
- ```sql
168
- -- Performance indexes
169
- CREATE INDEX idx_messages_queue_status_created ON queen.messages(queue_id, status, created_at);
170
- CREATE INDEX idx_messages_status_created ON queen.messages(status, created_at);
171
- CREATE INDEX idx_messages_transaction_id ON queen.messages(transaction_id);
172
- CREATE INDEX idx_messages_lease_expires ON queen.messages(lease_expires_at) WHERE status = 'processing';
173
- CREATE INDEX idx_transactions_expires ON queen.transactions(expires_at);
174
- CREATE INDEX idx_queue_processors_activity ON queen.queue_processors(last_activity);
175
- CREATE INDEX idx_namespaces_name ON queen.namespaces(name);
176
- CREATE INDEX idx_tasks_namespace_name ON queen.tasks(namespace_id, name);
177
- CREATE INDEX idx_queues_task_name ON queen.queues(task_id, name);
178
- CREATE INDEX idx_queues_priority ON queen.queues(priority DESC);
179
- CREATE INDEX idx_ws_connections_last_ping ON queen.ws_connections(last_ping);
180
- ```
181
-
182
- ## API Endpoints
183
-
184
- ### Configuration
185
- ```
186
- POST /api/v1/configure
187
- Body: {
188
- "ns": "email-service",
189
- "task": "send-notifications",
190
- "queue": "high-priority",
191
- "options": {
192
- "leaseTime": 300, // Seconds a message is leased to worker
193
- "maxSize": 10000,
194
- "ttl": 3600,
195
- "retryLimit": 3,
196
- "deadLetterQueue": true,
197
- "delayedProcessing": 60, // Don't process messages until 60s old
198
- "windowBuffer": 30, // Wait until last message is 30s old
199
- "priority": 10 // Higher priority queues processed first
200
- }
201
- }
202
- Response: 201 Created
203
- ```
204
-
205
- ### Push Messages
206
- ```
207
- POST /api/v1/push
208
- Body: {
209
- "items": [
210
- {
211
- "ns": "email-service",
212
- "task": "send-notifications",
213
- "queue": "high-priority",
214
- "payload": { "to": "user@example.com", "template": "welcome" },
215
- "transactionId": "550e8400-e29b-41d4-a716-446655440000" // Optional, auto-generated if not provided
216
- }
217
- ],
218
- "config": {
219
- "batchMode": true,
220
- "autoCreate": true // Auto-create missing ns/task/queue
221
- }
222
- }
223
- Response: 201 Created
224
- {
225
- "messages": [
226
- {
227
- "id": "msg-uuid",
228
- "transactionId": "550e8400-e29b-41d4-a716-446655440000",
229
- "status": "queued"
230
- }
231
- ]
232
- }
233
-
234
- Note:
235
- - Resources (ns/task/queue) are auto-created if they don't exist
236
- - transactionId ensures idempotency - duplicate pushes with same ID are ignored
237
- ```
238
-
239
- ### Pop Messages (Long Polling)
240
- ```
241
- GET /api/v1/pop/ns/email-service?timeout=30000&wait=true
242
- GET /api/v1/pop/ns/email-service/task/send-notifications?timeout=30000&wait=true
243
- GET /api/v1/pop/ns/email-service/task/send-notifications/queue/high-priority?timeout=30000&wait=true
244
-
245
- Query Parameters:
246
- - timeout: Max wait time in ms (default: 30000, max: 60000)
247
- - wait: Enable long polling (default: false)
248
- - batch: Number of messages to return (default: 1, max: 100)
249
-
250
- Response: 200 OK
251
- {
252
- "messages": [
253
- {
254
- "id": "msg-uuid",
255
- "transactionId": "550e8400-e29b-41d4-a716-446655440000",
256
- "payload": { "to": "user@example.com", "template": "welcome" },
257
- "queue": "email-service/send-notifications/high-priority",
258
- "createdAt": "2025-10-07T12:00:00Z"
259
- }
260
- ]
261
- }
262
- or 204 No Content on timeout
263
-
264
- Note:
265
- - Batch returns messages from the same scope, respecting queue priorities
266
- - Each message includes its transactionId for acknowledgment
267
- - Messages are marked as 'processing' with lease time
268
- ```
269
-
270
- ### Analytics
271
- ```
272
- GET /api/v1/analytics/queues # All queue stats
273
- GET /api/v1/analytics/ns/:nsId # Namespace stats
274
- GET /api/v1/analytics/ns/:nsId/task/:taskId # Task stats
275
- GET /api/v1/analytics/queue-depths # Current queue depths
276
- GET /api/v1/analytics/throughput # Messages/second metrics
277
- ```
278
-
279
- ### Message Acknowledgment
280
- ```
281
- POST /api/v1/ack
282
- Body: {
283
- "transactionId": "550e8400-e29b-41d4-a716-446655440000",
284
- "status": "completed" // or "failed"
285
- "error": "Optional error message if failed"
286
- }
287
- Response: 200 OK
288
- {
289
- "transactionId": "550e8400-e29b-41d4-a716-446655440000",
290
- "status": "completed",
291
- "completedAt": "2025-10-07T12:00:00Z"
292
- }
293
-
294
- Note:
295
- - Must ACK within leaseTime or message returns to pending
296
- - Failed messages may be retried based on queue configuration
297
- ```
298
-
299
- ### Batch Acknowledgment
300
- ```
301
- POST /api/v1/ack/batch
302
- Body: {
303
- "acknowledgments": [
304
- {
305
- "transactionId": "550e8400-e29b-41d4-a716-446655440000",
306
- "status": "completed"
307
- },
308
- {
309
- "transactionId": "660e8400-e29b-41d4-a716-446655440001",
310
- "status": "failed",
311
- "error": "Connection timeout"
312
- }
313
- ]
314
- }
315
- Response: 200 OK
316
- {
317
- "processed": 2,
318
- "results": [
319
- { "transactionId": "550e8400...", "status": "completed" },
320
- { "transactionId": "660e8400...", "status": "failed", "retryScheduled": true }
321
- ]
322
- }
323
- ```
324
-
325
- ### WebSocket Events (Dashboard)
326
- ```
327
- WS /ws/dashboard
328
-
329
- Events emitted to connected dashboard clients:
330
- - message.pushed: { queue, transactionId, timestamp }
331
- - message.processing: { queue, transactionId, workerId, timestamp }
332
- - message.completed: { queue, transactionId, timestamp }
333
- - message.failed: { queue, transactionId, error, timestamp }
334
- - queue.created: { ns, task, queue, timestamp }
335
- - queue.depth: { queue, depth, timestamp }
336
- - worker.connected: { workerId, timestamp }
337
- - worker.disconnected: { workerId, timestamp }
338
- ```
339
-
340
- ## Core Components (Functional Style)
341
-
342
- ### queueManager Factory
343
- ```javascript
344
- // Factory function returns queue processing functions
345
- export const createQueueManager = (db, eventManager, cache) => ({
346
- getNextMessage: async (scope) => { /* ... */ },
347
- processQueues: async () => { /* ... */ },
348
- handleDelayedProcessing: async () => { /* ... */ },
349
- handleWindowBuffer: async () => { /* ... */ },
350
- checkLeaseExpiry: async () => { /* ... */ }
351
- })
352
- ```
353
-
354
- ### eventManager Factory
355
- ```javascript
356
- // Factory for event coordination
357
- export const createEventManager = () => ({
358
- emit: (event, data) => { /* ... */ },
359
- on: (event, handler) => { /* ... */ },
360
- once: (event, handler) => { /* ... */ },
361
- removeListener: (event, handler) => { /* ... */ }
362
- })
363
- ```
364
-
365
- ### resourceCache Factory
366
- ```javascript
367
- // In-memory cache for resource existence
368
- export const createResourceCache = () => ({
369
- checkResource: async (ns, task, queue) => { /* ... */ },
370
- cacheResource: (ns, task, queue) => { /* ... */ },
371
- invalidate: (ns, task, queue) => { /* ... */ }
372
- })
373
- ```
374
-
375
- ### Route Handlers (Functional)
376
- - Pure functions for request handling
377
- - Composable middleware using pipe/compose
378
- - Immutable request/response transformations
379
- - Functional error handling with Either/Result patterns
380
-
381
- ## Implementation Phases
382
-
383
- ### Phase 1: Core Infrastructure (Week 1)
384
- - [ ] Database schema and migrations
385
- - [ ] Basic uWS server setup with routing
386
- - [ ] PostgreSQL connection pool
387
- - [ ] Basic CRUD operations for ns/task/queue/message
388
- - [ ] UUID generation and validation utilities
389
-
390
- ### Phase 2: Queue Processing (Week 2)
391
- - [ ] OptimalQueueManager implementation
392
- - [ ] Background message processing loop
393
- - [ ] Queue coordination and locking
394
- - [ ] Message status transitions (pending → processing → completed/failed)
395
- - [ ] Worker heartbeat and cleanup mechanisms
396
-
397
- ### Phase 3: HTTP API (Week 3)
398
- - [ ] Configure endpoint (create ns/task/queue)
399
- - [ ] Push endpoint (insert messages)
400
- - [ ] Basic pop endpoint (immediate fetch)
401
- - [ ] Request validation and error handling
402
- - [ ] API response formatting
403
-
404
- ### Phase 4: Long Polling & Advanced Features (Week 4)
405
- - [ ] EventManager for real-time coordination
406
- - [ ] Long polling implementation in pop routes
407
- - [ ] Timeout handling and cleanup
408
- - [ ] Connection management for concurrent requests
409
- - [ ] Delayed processing queue support
410
- - [ ] Window buffer queue support
411
- - [ ] Lease-based message processing
412
- - [ ] Performance testing and optimization
413
-
414
- ### Phase 5: Analytics & Monitoring (Week 5)
415
- - [ ] Queue depth tracking
416
- - [ ] Throughput metrics collection
417
- - [ ] Processing latency measurements
418
- - [ ] Analytics API endpoints
419
- - [ ] Basic dashboard/monitoring
420
-
421
- ### Phase 6: Client SDK & Production Features (Week 6)
422
- - [ ] JavaScript client SDK implementation
423
- - [ ] Client connection pooling
424
- - [ ] Client-side retry logic
425
- - [ ] Dead letter queues
426
- - [ ] Message retry logic
427
- - [ ] Queue size limits and backpressure
428
- - [ ] Rate limiting
429
- - [ ] Comprehensive error handling
430
- - [ ] Performance benchmarking
431
-
432
- ## Configuration
433
-
434
- ### Environment Variables
435
- ```bash
436
- # Database
437
- PG_USER=postgres
438
- PG_HOST=localhost
439
- PG_DB=postgres
440
- PG_PASSWORD=postgres
441
- PG_PORT=5432
442
- DB_POOL_SIZE=20
443
- DB_TIMEOUT=30000
444
-
445
- # Server
446
- PORT=3000
447
- HOST=0.0.0.0
448
- WORKER_ID=worker-${HOSTNAME}-${PID}
449
-
450
- # Queue Processing
451
- QUEUE_POLL_INTERVAL=100
452
- MAX_BATCH_SIZE=100
453
- WORKER_HEARTBEAT_INTERVAL=10000
454
- STALE_WORKER_TIMEOUT=30000
455
-
456
- # Long Polling
457
- DEFAULT_TIMEOUT=30000
458
- MAX_TIMEOUT=60000
459
- MAX_CONCURRENT_POLLS=1000
460
-
461
- # Analytics
462
- METRICS_COLLECTION_INTERVAL=5000
463
- METRICS_RETENTION_DAYS=30
464
-
465
- # WebSocket
466
- WS_HEARTBEAT_INTERVAL=30000
467
- WS_MAX_CONNECTIONS=1000
468
- ```
469
-
470
- ### Queue Options Schema
471
- ```javascript
472
- {
473
- leaseTime: 300, // Seconds a message is leased to worker
474
- maxSize: 10000, // Max messages in queue
475
- ttl: 3600, // Message TTL in seconds
476
- retryLimit: 3, // Max retry attempts
477
- retryDelay: 1000, // Delay between retries (ms)
478
- deadLetterQueue: true, // Enable DLQ
479
- priority: 0, // Queue priority (higher = processed first)
480
- delayedProcessing: 0, // Seconds to wait before processing (0 = immediate)
481
- windowBuffer: 0 // Seconds to wait after last message before processing batch
482
- }
483
- ```
484
-
485
- ## Performance Targets
486
-
487
- - **Throughput**: 10,000+ messages/second
488
- - **Latency**: < 10ms for immediate pop, < 100ms for long polling response
489
- - **Concurrency**: 1,000+ concurrent long polling connections
490
- - **Memory**: < 512MB for 1M queued messages
491
- - **CPU**: < 50% on 4-core system at target throughput
492
-
493
- ## Testing Strategy
494
-
495
- ### Unit Tests
496
- - Database operations (CRUD)
497
- - Queue processing logic
498
- - Event management
499
- - Utility functions
500
-
501
- ### Integration Tests
502
- - End-to-end API workflows
503
- - Long polling behavior
504
- - Multi-worker coordination
505
- - Database transaction handling
506
-
507
- ### Performance Tests
508
- - Load testing with artillery/k6
509
- - Concurrent connection limits
510
- - Memory leak detection
511
- - Database performance under load
512
-
513
- ### Chaos Tests
514
- - Worker failure scenarios
515
- - Database connection loss
516
- - Network partitions
517
- - High contention situations
518
-
519
- ## Deployment Considerations
520
-
521
- ### Docker Setup
522
- - Multi-stage build for production
523
- - PostgreSQL container for development
524
- - Health checks and graceful shutdown
525
- - Resource limits and monitoring
526
-
527
- ### Production Deployment
528
- - Horizontal scaling with load balancer
529
- - Database connection pooling
530
- - Monitoring and alerting
531
- - Backup and recovery procedures
532
-
533
- ## Client SDK Design
534
-
535
- ### Installation
536
- ```javascript
537
- npm install queen-client
538
- ```
539
-
540
- ### Basic Usage
541
- ```javascript
542
- import { createQueenClient } from 'queen-client'
543
-
544
- const client = createQueenClient({
545
- baseUrl: 'http://localhost:3000',
546
- timeout: 30000,
547
- retryAttempts: 3
548
- })
549
-
550
- // Configure queue
551
- await client.configure({
552
- ns: 'email-service',
553
- task: 'notifications',
554
- queue: 'high-priority',
555
- options: {
556
- leaseTime: 300,
557
- delayedProcessing: 60,
558
- windowBuffer: 30,
559
- priority: 10
560
- }
561
- })
562
-
563
- // Push messages
564
- const { messageIds } = await client.push({
565
- items: [{
566
- ns: 'email-service',
567
- task: 'notifications',
568
- queue: 'high-priority',
569
- payload: { to: 'user@example.com', template: 'welcome' }
570
- }]
571
- })
572
-
573
- // Pop with long polling
574
- const message = await client.pop({
575
- ns: 'email-service',
576
- task: 'notifications',
577
- queue: 'high-priority',
578
- wait: true,
579
- timeout: 30000
580
- })
581
-
582
- // Process message
583
- try {
584
- await processMessage(message)
585
- // Acknowledge successful processing
586
- await client.ack(message.transactionId, 'completed')
587
- } catch (error) {
588
- // Acknowledge failure (message may be retried)
589
- await client.ack(message.transactionId, 'failed', error.message)
590
- }
591
- ```
592
-
593
- ### Client Features
594
- - Automatic retry with exponential backoff
595
- - Connection pooling for HTTP/2
596
- - Long polling support with automatic reconnection
597
- - Batch operations for push/pop
598
- - Promise-based API with async/await support
599
- - Event emitter for real-time updates
600
- - Built-in request/response validation
601
-
602
- ## Advanced Queue Processing
603
-
604
- ### Delayed Processing
605
- Messages in queues with `delayedProcessing` are not eligible for processing until they've aged for the specified duration:
606
- ```sql
607
- -- Only select messages older than delayedProcessing seconds
608
- WHERE created_at <= NOW() - INTERVAL '60 seconds'
609
- ```
610
-
611
- ### Window Buffer
612
- Queues with `windowBuffer` wait until the newest message in the queue is at least X seconds old before processing any messages:
613
- ```sql
614
- -- Check if newest message is old enough
615
- SELECT MAX(created_at) < NOW() - INTERVAL '30 seconds'
616
- FROM messages
617
- WHERE queue_id = ? AND status = 'pending'
618
- ```
619
-
620
- ### Lease-Based Processing
621
- Messages are leased to workers for a specific duration. If not acknowledged within the lease time, they become available again:
622
- ```sql
623
- -- Reclaim expired leases (runs periodically)
624
- UPDATE queen.messages
625
- SET status = 'pending',
626
- worker_id = NULL,
627
- lease_expires_at = NULL,
628
- retry_count = retry_count + 1
629
- WHERE status = 'processing'
630
- AND lease_expires_at < NOW()
631
- AND retry_count < (
632
- SELECT (options->>'retryLimit')::int
633
- FROM queen.queues
634
- WHERE id = queue_id
635
- );
636
-
637
- -- Move to DLQ if retry limit exceeded
638
- UPDATE queen.messages
639
- SET status = 'dead_letter'
640
- WHERE status = 'processing'
641
- AND lease_expires_at < NOW()
642
- AND retry_count >= (
643
- SELECT (options->>'retryLimit')::int
644
- FROM queen.queues
645
- WHERE id = queue_id
646
- );
647
- ```
648
-
649
- ### Queue Priority Processing
650
- Higher priority queues are processed first:
651
- ```sql
652
- -- Select next message respecting queue priorities
653
- WITH prioritized_queues AS (
654
- SELECT q.id, q.priority
655
- FROM queues q
656
- JOIN messages m ON m.queue_id = q.id
657
- WHERE m.status = 'pending'
658
- ORDER BY q.priority DESC, m.created_at
659
- LIMIT 1
660
- )
661
- SELECT m.* FROM messages m
662
- JOIN prioritized_queues pq ON m.queue_id = pq.id
663
- WHERE m.status = 'pending'
664
- ORDER BY m.created_at
665
- LIMIT 1
666
- FOR UPDATE SKIP LOCKED
667
- ```
668
-
669
- ## Message Durability & Ordering Guarantees
670
-
671
- ### Message Durability Guarantees
672
- 1. **Transactional Writes**: All message inserts are within PostgreSQL transactions
673
- 2. **Idempotency**: Transaction IDs prevent duplicate message insertion
674
- 3. **Lease-Based Processing**: Messages have lease timeouts - if not acknowledged, they return to pending
675
- 4. **Retry Logic**: Failed messages can be retried with configurable limits
676
- 5. **Dead Letter Queue**: Messages exceeding retry limits go to DLQ (not lost)
677
- 6. **WAL & Replication**: PostgreSQL WAL ensures durability even on crashes
678
-
679
- ### Message Ordering Guarantees
680
- 1. **FIFO Within Queue**: Messages are strictly ordered by `created_at` within each queue
681
- 2. **FOR UPDATE SKIP LOCKED**: Ensures only one worker processes a message
682
- 3. **Queue-Level Coordination**: Optional queue locking prevents concurrent processing
683
- 4. **Atomic Status Updates**: Message status changes are atomic operations
684
- 5. **No Out-of-Order Processing**: Delayed processing and window buffer maintain order
685
-
686
- ### Failure Scenarios Handled
687
- - **Worker Crash**: Lease expires, message returns to pending
688
- - **Database Connection Loss**: Transactions rollback, no partial state
689
- - **Network Partition**: Client retries with same transactionId (idempotent)
690
- - **Server Restart**: All pending messages preserved in PostgreSQL
691
- - **Duplicate Push**: Transaction ID prevents duplicate insertion
692
- - **Missing ACK**: Lease timeout returns message to queue for retry
693
- - **Explicit NACK**: Client sends failed status, message retried per configuration
694
-
695
- ## Summary of Key Decisions
696
-
697
- 1. **Auto-creation**: Resources (ns/task/queue) are auto-created on push if they don't exist, with efficient in-memory caching
698
- 2. **Batch behavior**: Batch returns N messages from the same scope, respecting queue priorities
699
- 3. **No global pop**: Removed `GET /api/v1/pop` - must specify at least namespace
700
- 4. **Queue priority**: Queues have priority (not messages) to maintain FIFO within each queue
701
- 5. **No authentication**: Skipped for initial implementation
702
- 6. **Functional style**: Factory functions instead of classes throughout
703
- 7. **Advanced features**: Support for delayed processing, window buffering, and lease-based message handling
704
- 8. **Transaction IDs**: Every message has a transactionId for idempotency and deduplication
705
- 9. **WebSocket Support**: Real-time events for dashboard monitoring
706
- 10. **Client SDK in /src/client**: JavaScript client with long polling and retry support
707
- 11. **Queen Schema**: All PostgreSQL tables under `queen` schema for isolation