@boopkit/cli 0.1.0

package/profiles/web-app/reference/message-queue-patterns.ref.md

@@ -0,0 +1,247 @@
+# Message Queue Patterns
+
+Architectural guidance for RabbitMQ consumer patterns in worker processes. These patterns are not scaffolded by `boop init` — they document proven approaches for when your project adds async processing. Covers consumer modes, retry/DLQ, buffered batch ACK, and backpressure via prefetch.
+
+## Infrastructure
+
+- **Provider**: RabbitMQ via Heroku CLOUDAMQP
+- **Library**: `amqplib` (v0.10.x)
+- **Service**: `src/services/message_queue.service.ts`
+- **Queue naming**: `{QUEUE_PREFIX}/{queue-name}` — `QUEUE_PREFIX` is user-specific in locahost (e.g., `mike`), empty in dev, staging, production
+
+## Publishing
+
+`MessageQueueService.publish()` uses a **persistent AMQP connection** that is lazy-initialized on first call and reused across all subsequent calls within the same process.
+
+```typescript
+const { message_queue: MessageQueueService } = services;
+await MessageQueueService.publish('batch-ready', [message1, message2]);
+```
+
+### Connection lifecycle
+
+- **Lazy init**: No connection is opened until the first `publish()` call
+- **Reuse**: The same connection and channel are reused for all subsequent publishes
+- **Auto-reconnect**: If the connection drops (`error` or `close` event), the cached reference is cleared and the next `publish()` call opens a fresh connection
+- **Error recovery**: If `publish()` catches an error (e.g., channel closed), it resets connection state and returns `false` — the next call will reconnect
+- **Shutdown**: The publish connection is tracked in `listenerConnections` and closed by the coordinated SIGINT handler alongside listener connections
+
+### Why persistent (not per-call)
+
+The original implementation opened a new AMQP connection for every `publish()` call, then immediately closed it. This works for infrequent publishing but fails under rapid iteration (e.g., `ch-export-push` publishing 177 batch-ready messages in seconds). CloudAMQP's shared plan rate-limits connection churn, causing `ECONNREFUSED` errors.
+
+### Callers
+
+| Caller | Queue | Frequency |
+|--------|-------|-----------|
+| Scheduler jobs | various | 1-2 per 10-min cycle |
+
+## Consumer Modes
+
+The `MessageQueueService.listen()` function supports two modes controlled by `opts.manualAck`:
+
+### Auto-ACK Mode (default)
+
+```typescript
+MessageQueueService.listen('queue-name', handler);
+```
+
+- Handler signature: `async handler(msg) → boolean`
+- Returns `true` → message ACKed immediately
+- Returns `false` → sent to retry queue with exponential backoff
+- Throws → NACKed to DLQ after max retries
+- **Use case**: Simple request-response consumers (e.g., entity replication, long running jobs)
+
+### Manual-ACK Mode
+
+```typescript
+MessageQueueService.listen('queue-name', handler, { manualAck: true, prefetch: N });
+```
+
+- Handler signature: `async handler(msg, ack, nack)`
+- `ack()` and `nack()` are guarded — only the first call takes effect
+- Handler is responsible for calling `ack()` or `nack()` (can be deferred)
+- If handler throws without settling → message moved to DLQ
+- **Use case**: Buffered/batch consumers where ACK is deferred until a batch operation completes
+
+## Retry & Dead Letter Queue (DLQ)
+
+Every queue gets automatic retry and DLQ infrastructure:
+
+| Queue | Purpose |
+|-------|---------|
+| `{name}` | Main queue — consumer listens here |
+| `{name}.retry` | Retry queue — messages wait here with TTL before redelivery to main queue |
+| `{name}.failed` | DLQ — messages that exceed max retries land here |
+
+**Retry behavior**:
+- Exponential backoff: `min(BASE * 2^retryCount, MAX)` — default BASE=4s, MAX=60s
+- Max retries: 5 (configurable via `MAX_MESSAGE_RETRIES` env var)
+- Retry count tracked in `x-retry-count` message header
+- After max retries → NACK without requeue → routed to DLQ via dead letter exchange
+
+## Buffered Batch ACK Pattern
+
+For consumers that buffer events and flush in batches (e.g., the S3 batch writer), ACK is deferred until the batch operation completes. This pattern combines manual-ACK mode with prefetch-based backpressure.
+
+### How it works
+
+```
+RabbitMQ           Consumer (worker)              S3 Batch Writer
+   │                    │                              │
+   │── deliver msg 1 ──>│                              │
+   │── deliver msg 2 ──>│── add(event, messageRef) ──>│ buffer
+   │── deliver msg 3 ──>│── add(event, messageRef) ──>│ buffer
+   │── deliver msg 4 ──>│── add(event, messageRef) ──>│ buffer
+   │── deliver msg 5 ──>│── add(event, messageRef) ──>│ buffer ──> threshold!
+   │                    │                              │
+   │                    │                    flush to S3 (JSONL.gz)
+   │                    │                              │
+   │                    │                     S3 upload success
+   │                    │                              │
+   │<─── ACK msg 1 ────│<──── ack(ref1) ─────────────│
+   │<─── ACK msg 2 ────│<──── ack(ref2) ─────────────│
+   │<─── ACK msg 3 ────│<──── ack(ref3) ─────────────│
+   │<─── ACK msg 4 ────│<──── ack(ref4) ─────────────│
+   │<─── ACK msg 5 ────│<──── ack(ref5) ─────────────│
+   │                    │                              │
+   │── deliver msg 6 ──>│         (next batch)         │
+   ...
+```
+
+### MessageRef interface
+
+The consumer wraps the MQ channel and message into a `MessageRef` passed to the batch writer:
+
+```typescript
+interface MessageRef {
+    channel: { ack: () => void; nack: () => void };
+    message: any;  // original amqplib message (unused, kept for debugging)
+}
+```
+
+The batch writer stores message refs alongside buffered events. On flush:
+- **Success**: calls `ref.channel.ack()` for each message in the batch
+- **Failure**: calls `ref.channel.nack()` for each message → triggers retry
+
+### Implementation
+
+```typescript
+// In access_logs.service.ts — onLogRequest handler
+S3BatchWriterService.add(logEntry, {
+    channel: { ack: () => ack(), nack: () => nack() },
+    message: msg
+});
+
+// In s3_batch_writer.service.ts — flushToS3
+// After successful S3 upload:
+for (const ref of snap.messageRefs) {
+    ref.channel.ack();  // ACK each buffered message
+}
+// On upload failure:
+for (const ref of snap.messageRefs) {
+    ref.channel.nack();  // NACK → retry with backoff
+}
+```
+
+## Prefetch & Backpressure
+
+### The prefetch-batch alignment rule
+
+**When using buffered batch ACK, `prefetch` must equal the batch size (`MAX_EVENTS`).**
+
+| Prefetch | Behavior | Problem |
+|----------|----------|---------|
+| `1` | RabbitMQ delivers 1 message, waits for ACK before sending next | **Deadlock**: batch never fills, message sits unACKed forever (until age timer) |
+| `= MAX_EVENTS` | RabbitMQ delivers exactly N messages, batch fills, flush, ACK, next N delivered | **Correct**: clean lockstep between delivery and batch processing |
+| `0` (unlimited) | RabbitMQ delivers all available messages immediately | **Risky**: buffer grows unbounded; concurrent flush triggers may race |
+
+### How prefetch creates lockstep
+
+With `prefetch = MAX_EVENTS = 5`:
+
+1. RabbitMQ delivers 5 messages (consumer's unACKed limit)
+2. All 5 buffered → threshold hit → flush → S3 upload → ACK all 5
+3. RabbitMQ sees 5 ACKs → delivers next 5
+4. Repeat
+
+If fewer than `MAX_EVENTS` messages exist in the queue, the age-check timer (default 30s interval, 10min max age) eventually flushes the partial batch.
+
+### Configuration
+
+```typescript
+// In access_logs.service.ts — setupQueueListener()
+const prefetch = S3BatchWriterService.config.MAX_EVENTS;
+await MessageQueueService.listen('access-logs/logRequest', onLogRequest, {
+    manualAck: true,
+    prefetch
+});
+```
+
+**Environment variables**:
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `S3_BATCH_MAX_EVENTS` | `10000` | Events per batch (also sets prefetch) |
+| `S3_BATCH_MAX_BYTES` | `10485760` (10MB) | Byte size threshold |
+| `S3_BATCH_MAX_AGE_MS` | `600000` (10min) | Maximum age before flush |
+| `S3_BATCH_TIMER_INTERVAL_MS` | `30000` (30s) | Age-check timer interval |
+
+## Known Issues
+
+## Service Init Ordering
+
+Services are loaded and initialized alphabetically by `loadServices()` in `src/services/index.ts`. This creates an ordering dependency:
+
+- `access_logs` (a) initializes before `s3_batch_writer` (s)
+- If `access_logs` subscribes to a queue during `init()`, messages arrive before the batch writer's flush callback is wired
+
+**Solution**: Queue subscription is deferred to `setupQueueListener()`, called by the worker **after** all services are initialized:
+
+```
+loadServices()          →  all service.init() in alphabetical order
+setupQueueListeners()   →  AccessLogsService.setupQueueListener() first
+                           then other queue listeners
+```
+
+This ensures the S3 batch writer is fully ready (flush callback, timer, shutdown handlers) before any messages are consumed.
+
+## Shutdown Behavior
+
+### Known issue: Shutdown flush ACK race
+
+During SIGINT/SIGTERM, both the MQ service and batch writer register independent signal handlers:
+- **MQ handler**: closes the AMQP connection immediately
+- **Batch writer handler**: flushes remaining events to S3, then tries to ACK
+
+The MQ connection closes before the batch writer can ACK. The S3 upload succeeds but ACKs fail with `Channel closed`.
+
+**Impact**: Low — data is safe in S3, unACKed messages are redelivered on next start, `event_id`-based dedup in v2's ReplacingMergeTree handles duplicates.
+
+**Fix**: Requires coordinated shutdown ordering (batch writer flush+ACK → MQ close → process exit).
+
+## Queue Listeners Registration
+
+Queue listeners are registered in `src/worker.ts`:
+
+```typescript
+// Access-logs listener — set up first, after all services init
+await AccessLogsService.setupQueueListener();
+
+// Other listeners — auto-ACK mode
+const listeners = [
+    { queue: 'clickhouse-entity-replication', handler: ... },
+    { queue: 'web_log_uploaded', handler: ... },
+    { queue: 's3-loader', handler: ... },
+    { queue: 's3-loader-watchdog', handler: ... },
+];
+```
+
+## References
+
+These file paths follow the conventions described above. They represent the expected locations when you implement async processing in your project:
+
+- `src/services/message_queue.service.ts` — MQ service (publish, listen, retry/DLQ)
+- `src/services/access_logs.service.ts` — S3 batch consumer, `setupQueueListener()`
+- `src/services/s3_batch_writer.service.ts` — batch buffer, flush, ACK/NACK
+- `src/worker.ts` — consumer registration
+- [Architecture](./architecture.ref.md) — system overview