elevenlabs-webhook-nodejs 1.0.0

package/docs/architecture/16-BACKGROUND-JOBS.md

@@ -0,0 +1,469 @@
+# Background Jobs
+
+> BullMQ queues and scheduled tasks
+> Last updated: 2026-04-02
+
+---
+
+## 1. Why BullMQ
+
+The current system uses `setInterval`, `setTimeout`, and `node-cron` for all background work. These are in-memory primitives -- they are lost on process restart, cannot be observed, and offer no retry or concurrency control.
+
+BullMQ solves every one of these problems:
+
+| Requirement | setInterval/setTimeout | BullMQ |
+|-------------|----------------------|--------|
+| Survives restart | No -- lost on crash or deploy | Yes -- jobs persisted in Redis |
+| Retries on failure | Manual | Built-in with backoff strategy |
+| Concurrency control | None | Per-queue concurrency limit |
+| Observable | No visibility | Bull Board UI, job events, completion callbacks |
+| Deduplication | Manual Map tracking | Built-in via `jobId` |
+| Delayed execution | setTimeout (in-memory) | Delayed jobs (persisted) |
+| Cron schedules | node-cron (in-memory) | Repeatable jobs (persisted) |
+| Multi-instance safe | No -- each instance runs its own timers | Yes -- Redis-backed, single processor per job |
+
+---
+
+## 2. Complete Queue Inventory
+
+The system defines 18 queues across 6 functional areas. Each queue has a dedicated processor class in the owning module.
+
+### 2.1 WhatsApp Queues (8 queues)
+
+| # | Queue Name | Purpose | Concurrency | Trigger | Replaces |
+|---|-----------|---------|-------------|---------|----------|
+| 1 | `whatsapp:grace` | Grace period expiry -- activates AI after waiting period | 5 | Delayed job on new session (default 180s) | `setTimeout` in `sessionTimers` Map |
+| 2 | `whatsapp:message-buffer` | Message debounce -- combines rapid messages before sending to AI | 10 | Delayed job on each message (10-50s random) | `setTimeout` in `messageBufferTimers` Map |
+| 3 | `whatsapp:session-cleanup` | Sweep inactive sessions -- resolve sessions with no activity for 2h | 1 | Repeatable every 5 min | `setInterval` 5 min |
+| 4 | `whatsapp:cost-sweep` | Fetch ElevenLabs conversation costs for active sessions | 1 | Repeatable every 10 min | `setInterval` 10 min |
+| 5 | `whatsapp:operator-expiry` | Expire unanswered operator requests and notify AI | 1 | Repeatable every 15 min | `setInterval` 15 min |
+| 6 | `whatsapp:profile-update` | Update `patient.patient_ai_summary` (encrypted with `PATIENT_SUMMARY_ENCRYPTION_KEY`) after session resolution | 3 | Fire-and-forget on session resolve | Detached `async` call |
+| 7 | `whatsapp:session-labels` | Generate sentiment and category labels for resolved sessions | 3 | Fire-and-forget on session resolve | Detached `async` call |
+| 8 | `whatsapp:message-cleanup` | Delete messages older than `clinic.clinic_message_retention_days` (default 7) and associated MinIO media files | 1 | Repeatable daily at 03:00 | New feature |
+
+### 2.2 ElevenLabs Queues (2 queues)
+
+| # | Queue Name | Purpose | Concurrency | Trigger | Replaces |
+|---|-----------|---------|-------------|---------|----------|
+| 9 | `elevenlabs:webhook` | Process post-call webhooks (transcript, summary, notification) | 5 | On webhook POST from ElevenLabs | Background `async` in route handler |
+| 10 | `elevenlabs:tools` | Process tool call webhooks during voice calls | 5 | On tool webhook POST from ElevenLabs | Inline processing in route handler |
+
+### 2.3 Billing Queues (2 queues)
+
+| # | Queue Name | Purpose | Concurrency | Trigger | Replaces |
+|---|-----------|---------|-------------|---------|----------|
+| 11 | `billing:snapshot` | Snapshot expired billing periods for usage reporting | 1 | Repeatable daily at 00:05 Istanbul time | `node-cron` daily |
+| 12 | `billing:alerts` | Send billing usage alerts (75%, 90%, 100% thresholds) | 1 | Repeatable daily at 09:00 Istanbul time | `node-cron` daily |
+
+### 2.4 Appointment Queues (3 queues)
+
+| # | Queue Name | Purpose | Concurrency | Trigger | Replaces |
+|---|-----------|---------|-------------|---------|----------|
+| 13 | `appointment-validation:dispatch` | Dispatch batch validation calls to ElevenLabs | 3 | On batch submit, processes entries sequentially | Async function with sleep loop |
+| 14 | `appointment-validation:sync` | Sync batch call statuses from ElevenLabs | 1 | Repeatable every 60s | `setInterval` 60s |
+| 15 | `appointment:reminder` | Send appointment reminders via WhatsApp (Meta template message). SMS via NetGSM is a stub (deferred). | 3 | Delayed job (24h + 2h before appointment) | New feature (P0) |
+
+### 2.5 Infrastructure Queues (2 queues)
+
+| # | Queue Name | Purpose | Concurrency | Trigger | Replaces |
+|---|-----------|---------|-------------|---------|----------|
+| 16 | `inbound-call:schedule-sync` | Sync phone number schedules with ElevenLabs | 1 | Repeatable every 15 min | `node-cron` every 15 min |
+| 17 | `log:cleanup` | Delete application logs older than retention period | 1 | Repeatable daily at 03:00 | MongoDB TTL index |
+
+### 2.6 Integration Queues (1 queue)
+
+| # | Queue Name | Purpose | Concurrency | Trigger | Replaces |
+|---|-----------|---------|-------------|---------|----------|
+| 18 | `webhook:outgoing` | Deliver tenant outgoing webhooks with retry | 5 | On business events (appointment, lead, call, etc.) | New feature (P1) |
+
+---
+
+## 3. Repeatable Jobs (Cron Schedules)
+
+Repeatable jobs replace all `node-cron` and `setInterval` patterns. They are registered once at module initialization and persist in Redis -- surviving restarts without re-registration.
+
+```typescript
+// Registration in module onModuleInit or queue module setup:
+
+// Billing
+await billingSnapshotQueue.add('run', {}, {
+  repeat: { pattern: '5 0 * * *', tz: 'Europe/Istanbul' },  // 00:05 daily
+});
+
+await billingAlertQueue.add('run', {}, {
+  repeat: { pattern: '0 9 * * *', tz: 'Europe/Istanbul' },  // 09:00 daily
+});
+
+// Appointment validation
+await appointmentSyncQueue.add('run', {}, {
+  repeat: { every: 60_000 },  // every 60 seconds
+});
+
+// Phone schedule sync
+await inboundScheduleQueue.add('run', {}, {
+  repeat: { pattern: '*/15 * * * *' },  // every 15 minutes
+});
+
+// WhatsApp sweeps
+await sessionCleanupQueue.add('run', {}, {
+  repeat: { every: 300_000 },  // every 5 minutes
+});
+
+await costSweepQueue.add('run', {}, {
+  repeat: { every: 600_000 },  // every 10 minutes
+});
+
+await operatorExpiryQueue.add('run', {}, {
+  repeat: { every: 900_000 },  // every 15 minutes
+});
+
+// Message cleanup (7-day retention)
+await messageCleanupQueue.add('run', {}, {
+  repeat: { pattern: '0 3 * * *' },  // daily at 03:00
+});
+
+// Log cleanup
+await logCleanupQueue.add('run', {}, {
+  repeat: { pattern: '0 3 * * *' },  // daily at 03:00
+});
+```
+
+### Complete Schedule Reference
+
+| Queue | Schedule | Timezone | Description |
+|-------|----------|----------|-------------|
+| `billing:snapshot` | `5 0 * * *` (00:05 daily) | Europe/Istanbul | Snapshot expired billing periods |
+| `billing:alerts` | `0 9 * * *` (09:00 daily) | Europe/Istanbul | Check and send usage alerts |
+| `appointment-validation:sync` | Every 60s | -- | Sync call statuses from ElevenLabs |
+| `inbound-call:schedule-sync` | `*/15 * * * *` (every 15 min) | -- | Sync phone schedules with ElevenLabs |
+| `whatsapp:session-cleanup` | Every 5 min | -- | Resolve inactive sessions (2h threshold) |
+| `whatsapp:cost-sweep` | Every 10 min | -- | Fetch ElevenLabs conversation costs |
+| `whatsapp:operator-expiry` | Every 15 min | -- | Expire unanswered operator requests |
+| `whatsapp:message-cleanup` | `0 3 * * *` (daily 03:00) | -- | Delete messages older than `clinic_message_retention_days` (default 7) + associated MinIO media |
+| `log:cleanup` | `0 3 * * *` (daily 03:00) | -- | Delete logs older than retention period (default 30 days) |
+
+---
+
+## 4. Delayed Jobs
+
+Delayed jobs execute once after a specified delay. They replace all `setTimeout` calls that were lost on restart.
+
+### 4.1 Grace Period Timer (`whatsapp:grace`)
+
+When a new WhatsApp session is created, a delayed job starts the grace period. If no human responds within the grace period, the processor activates the AI agent.
+
+```typescript
+await graceQueue.add(
+  'grace-expired',
+  { sessionId, chatId, tenantId, clinicId },
+  {
+    delay: gracePeriodSeconds * 1000,  // default 180s, configurable per clinic
+    jobId: `grace:${sessionId}`,       // deduplicate -- one timer per session
+  },
+);
+
+// Cancel on human response:
+await graceQueue.remove(`grace:${sessionId}`);
+```
+
+### 4.2 Message Buffer Timer (`whatsapp:message-buffer`)
+
+When a patient sends a message during an active session, a delayed job debounces rapid messages. The delay is randomized (10-50 seconds) to simulate human-like response timing.
+
+```typescript
+// On each incoming message: remove old job, create new one
+await messageBufferQueue.remove(`buffer:${sessionId}`);
+await messageBufferQueue.add(
+  'buffer-expired',
+  { sessionId, chatId, tenantId },
+  {
+    delay: randomDelay(10_000, 50_000),
+    jobId: `buffer:${sessionId}`,      // replace on new message
+  },
+);
+```
+
+When the buffer expires, the processor collects all pending messages and sends them to ElevenLabs as a combined batch.
+
+### 4.3 Appointment Reminder (`appointment:reminder`)
+
+When an appointment is booked, two delayed jobs are created for reminders (default: 24 hours and 2 hours before):
+
+```typescript
+for (const reminder of reminderSchedule) {
+  const delay = appointmentTime - (reminder.hours_before * 3600_000) - Date.now();
+  if (delay > 0) {
+    await reminderQueue.add(
+      'send',
+      {
+        appointmentId,
+        tenantId,
+        clinicId,
+        patientPhone,
+        patientName,
+        startTime,
+        template: reminder.message_template,
+        channels: reminder.channels,  // ['whatsapp'] or ['whatsapp', 'sms']
+      },
+      {
+        delay,
+        jobId: `reminder:${appointmentId}:${reminder.hours_before}h`,
+      },
+    );
+  }
+}
+
+// On cancellation: remove all scheduled reminders
+await reminderQueue.remove(`reminder:${appointmentId}:24h`);
+await reminderQueue.remove(`reminder:${appointmentId}:2h`);
+```
+
+The processor sends reminders via WhatsApp (Meta template message) and optionally SMS (NetGSM -- currently a stub, deferred), based on the clinic's configured channels.
+
+---
+
+## 5. Fire-and-Forget Jobs
+
+These jobs are added to a queue and processed independently. The caller does not wait for results.
+
+### 5.1 Profile Update (`whatsapp:profile-update`)
+
+Added when a session resolves. The processor collects all past sessions for the contact and generates an AI summary.
+
+```typescript
+// On session resolution:
+await profileUpdateQueue.add('update', {
+  tenantId,
+  contactPhone,
+  contactId,
+});
+```
+
+Concurrency: 3. Multiple profile updates can run in parallel since they operate on different contacts.
+
+### 5.2 Session Labels (`whatsapp:session-labels`)
+
+Added when a session resolves. The processor generates sentiment and category labels from the conversation.
+
+```typescript
+await sessionLabelsQueue.add('generate', {
+  tenantId,
+  sessionId,
+  chatId,
+});
+```
+
+Concurrency: 3. Independent of profile updates.
+
+### 5.3 ElevenLabs Webhook (`elevenlabs:webhook`)
+
+Post-call webhook processing. The route handler enqueues the payload immediately and returns 200 to ElevenLabs. The processor handles transcript retrieval, summary generation, notification sending, and billing.
+
+```typescript
+// In webhook controller:
+await elevenlabsWebhookQueue.add('process', {
+  conversationId,
+  agentId,
+  rawPayload,
+});
+return { status: 'queued' };
+```
+
+Concurrency: 5. Multiple calls can complete simultaneously.
+
+### 5.4 Outgoing Webhooks (`webhook:outgoing`)
+
+Delivers tenant-configured outgoing webhooks to external systems (EMR, HIS, etc.):
+
+```typescript
+await outgoingWebhookQueue.add('deliver', {
+  tenantId,
+  webhookId,
+  event: 'appointment.created',
+  payload: { appointmentId, patientName, startTime },
+});
+```
+
+The processor signs the payload with HMAC-SHA256 and delivers via HTTP POST. See Section 6 for retry strategy.
+
+---
+
+## 6. Error Handling
+
+### 6.1 Retry Strategy
+
+Each queue has a configured retry strategy with exponential backoff:
+
+```typescript
+// Default retry config applied to all queues:
+const defaultJobOptions: JobsOptions = {
+  attempts: 3,
+  backoff: {
+    type: 'exponential',
+    delay: 5_000,  // 5s, 10s, 20s
+  },
+  removeOnComplete: { count: 1000 },  // Keep last 1000 completed jobs
+  removeOnFail: { count: 5000 },      // Keep last 5000 failed jobs
+};
+```
+
+Queue-specific overrides:
+
+| Queue | Max Attempts | Backoff | Rationale |
+|-------|-------------|---------|-----------|
+| `whatsapp:grace` | 1 | None | Grace period is time-sensitive -- retry makes no sense |
+| `whatsapp:message-buffer` | 1 | None | Same -- debounce is time-sensitive |
+| `elevenlabs:webhook` | 3 | Exponential 5s | ElevenLabs might be temporarily unavailable |
+| `elevenlabs:tools` | 1 | None | Tool calls are synchronous -- caller is waiting |
+| `webhook:outgoing` | 3 | Exponential 10s | External systems may be temporarily down |
+| `appointment:reminder` | 2 | Fixed 30s | Important to deliver, but not worth many retries |
+| `billing:snapshot` | 3 | Exponential 60s | Critical for billing accuracy |
+| All sweep jobs | 1 | None | They run frequently -- next run will catch anything missed |
+
+### 6.2 Dead Letter Handling
+
+Jobs that exhaust all retry attempts are kept in the failed set (up to 5000 per queue). The system does **not** use a separate dead letter queue -- failed jobs remain in their original queue's failed set for inspection.
+
+Failed jobs trigger:
+
+1. **Error log** -- `error` level log with queue name, job ID, error message, and attempt count
+2. **Alert for critical queues** -- `billing:snapshot` and `elevenlabs:webhook` failures emit a `notification` WebSocket event to the admin dashboard
+
+```typescript
+// In processor base class or queue event listener:
+@OnQueueFailed()
+async onFailed(job: Job, error: Error) {
+  this.logger.error({
+    msg: `Job failed after ${job.attemptsMade} attempts`,
+    queue: job.queueName,
+    jobId: job.id,
+    data: job.data,
+    error: error.message,
+    stack: error.stack,
+  });
+
+  if (['billing:snapshot', 'elevenlabs:webhook'].includes(job.queueName)) {
+    this.eventEmitter.emit('notification', {
+      tenantId: job.data.tenantId,
+      type: 'system_error',
+      title: `Background job failed: ${job.queueName}`,
+      body: error.message,
+    });
+  }
+}
+```
+
+### 6.3 Stalled Job Detection
+
+BullMQ automatically detects stalled jobs (processor crashed mid-execution). Stalled jobs are retried up to `maxStalledCount` (default 1). The stall check interval is 30 seconds.
+
+---
+
+## 7. Queue Module Setup
+
+All queues are registered in a central `QueueModule` that is imported by the root `AppModule`. Each feature module registers its own processors.
+
+```
+src/
+  queue/
+    queue.module.ts          # Registers all 18 queues with BullMQ
+    queue.config.ts          # Queue names, default job options, repeat configs
+
+  whatsapp/
+    processors/
+      grace.processor.ts              # whatsapp:grace
+      message-buffer.processor.ts     # whatsapp:message-buffer
+      session-cleanup.processor.ts    # whatsapp:session-cleanup
+      cost-sweep.processor.ts         # whatsapp:cost-sweep
+      operator-expiry.processor.ts    # whatsapp:operator-expiry
+      profile-update.processor.ts     # whatsapp:profile-update
+      session-labels.processor.ts     # whatsapp:session-labels
+      message-cleanup.processor.ts    # whatsapp:message-cleanup
+
+  webhooks/
+    processors/
+      elevenlabs-webhook.processor.ts # elevenlabs:webhook
+
+  billing/
+    processors/
+      snapshot.processor.ts           # billing:snapshot
+      alerts.processor.ts             # billing:alerts
+
+  appointment-validation/
+    processors/
+      dispatch.processor.ts           # appointment-validation:dispatch
+      sync.processor.ts               # appointment-validation:sync
+
+  phone-numbers/
+    processors/
+      schedule-sync.processor.ts      # inbound-call:schedule-sync
+
+  logging/
+    processors/
+      log-cleanup.processor.ts        # log:cleanup
+```
+
+### Queue Registration
+
+```typescript
+// src/queue/queue.module.ts
+
+@Module({
+  imports: [
+    BullModule.forRoot({
+      connection: { url: process.env.REDIS_URL },
+    }),
+
+    // Register each queue
+    BullModule.registerQueue(
+      { name: 'whatsapp:grace' },
+      { name: 'whatsapp:message-buffer' },
+      { name: 'whatsapp:session-cleanup' },
+      { name: 'whatsapp:cost-sweep' },
+      { name: 'whatsapp:operator-expiry' },
+      { name: 'whatsapp:profile-update' },
+      { name: 'whatsapp:session-labels' },
+      { name: 'whatsapp:message-cleanup' },
+      { name: 'elevenlabs:webhook' },
+      { name: 'elevenlabs:tools' },
+      { name: 'billing:snapshot' },
+      { name: 'billing:alerts' },
+      { name: 'appointment-validation:dispatch' },
+      { name: 'appointment-validation:sync' },
+      { name: 'appointment:reminder' },
+      { name: 'inbound-call:schedule-sync' },
+      { name: 'log:cleanup' },
+      { name: 'webhook:outgoing' },
+    ),
+  ],
+  exports: [BullModule],
+})
+export class QueueModule {}
+```
+
+---
+
+## 8. Monitoring
+
+BullMQ queues are observable via:
+
+1. **Bull Board** -- web UI for inspecting queues, jobs, and failures. Available at `/admin/queues` (superadmin only).
+2. **Job events** -- `completed`, `failed`, `stalled` events logged at appropriate levels.
+3. **Redis** -- queue metadata stored in Redis, queryable for health checks.
+4. **System health endpoint** -- `GET /admin/system/health` includes queue depth and failed job counts.
+
+---
+
+## 9. Migration Checklist
+
+When migrating from the current system to BullMQ:
+
+| Current Pattern | BullMQ Replacement | Migration Step |
+|----------------|-------------------|----------------|
+| `setInterval(fn, 300_000)` in service constructor | Repeatable job with `every: 300_000` | Remove setInterval, register repeatable in module init |
+| `setTimeout(fn, delay)` stored in Map | Delayed job with `jobId` for deduplication | Remove Map + setTimeout, add queue job |
+| `node-cron.schedule('5 0 * * *', fn)` | Repeatable job with `pattern: '5 0 * * *'` | Remove node-cron, register repeatable in module init |
+| Fire-and-forget `async` call (no await) | `queue.add()` | Replace detached async with queue add |
+| Inline webhook processing | `queue.add()` in controller, processor handles logic | Move logic to processor class |
+
+Every `setInterval`, `setTimeout` stored in a Map, and `node-cron` call in the current codebase must be identified and converted. There should be zero in-memory timers in the NestJS rewrite.