elevenlabs-webhook-nodejs 1.0.0

package/docs/architecture/05-OPERATOR-WORKFLOW.md

@@ -0,0 +1,444 @@
+# 05 - Operator-in-the-Loop (Doctor) Workflow
+
+> Last updated: 2026-04-02
+
+> The operator workflow enables a human doctor/specialist to review patient media (photos, videos, documents)
+> and respond with pricing or medical guidance. The AI acts as intermediary -- forwarding media to the doctor,
+> waiting for a response, and then reformulating the doctor's reply for the patient.
+
+---
+
+## 1. Overview
+
+```
+Patient             AI Agent           Doctor (Operator)
+  |                    |                     |
+  |-- sends photos --->|                     |
+  |                    |                     |
+  |                    |-- AI calls          |
+  |                    |   forward_to_       |
+  |                    |   operator tool     |
+  |                    |                     |
+  |                    |-- forwards media -->|
+  |                    |   with REF code     |
+  |                    |                     |
+  |<-- "please wait" --|                     |
+  |                    |                     |
+  |    ... doctor reviews photos ...         |
+  |                    |                     |
+  |                    |<-- reply with REF --|
+  |                    |    + price/advice   |
+  |                    |                     |
+  |                    |-- AI reformulates   |
+  |                    |   for patient       |
+  |                    |                     |
+  |<-- formatted msg --|                     |
+```
+
+---
+
+## 2. Forward to Operator Flow
+
+### Trigger
+
+The ElevenLabs AI agent calls the `forward_to_operator` client tool when it determines the patient needs a doctor's review (typically after the patient sends photos).
+
+### Step-by-Step
+
+```
+1. AI calls forward_to_operator tool
+       |
+       v
+2. Load session, chat, tenant
+   Verify operator_workflow_enabled and operator_phone configured
+       |
+       v
+3. Find ALL media messages from patient in this session
+   Filter: sender='contact', media_type IN ('imageMessage','videoMessage','documentMessage')
+   Sort: oldest first
+       |
+       v
+4. Generate REF code
+   Format: {prefix}-{4 alphanumeric}  (e.g., R-AB2X, K-9F3M)
+   Regex:  /\b([A-Z]-[A-Z0-9]{4})\b/
+   Prefix: from clinic.clinic_operator_prefix directly (default "R")
+       |
+       v
+5. Create OperatorRequest record
+   Fields: tenant_id, session_id, chat_id, ref_code, status='pending',
+           forwarded_media_types[], forwarded_count, patient_message_ids[],
+           forwarded_to_phone, expires_at
+       |
+       v
+6. Set session.awaiting_operator = true
+       |
+       v
+7. Forward each media to doctor's WhatsApp
+   First media caption:
+     "REF: {code}
+      Hasta: {phone}
+      Hasta adi: {name}
+      Fotograf: 1/{total}
+      Not: {optional AI note}"
+   Subsequent media captions: "REF: {code} ({n}/{total})"
+   500ms delay between each send (WhatsApp rate limiting)
+       |
+       v
+8. Track success/failure counts per media
+   Update OperatorRequest.failed_count if any forwarding failed
+       |
+       v
+9. Return tool result to AI:
+   { ref_code, forwarded: true, total_media, forwarded_ok, failed }
+       |
+       v
+10. AI tells patient to wait for the doctor's response
+```
+
+### Sequence Diagram: Forward Flow
+
+```
+Patient        WhatsApp Agent       Meta Business API        Doctor
+  |                 |                       |                    |
+  |-- photo 1 ----->|                       |                    |
+  |-- photo 2 ----->|                       |                    |
+  |-- "check pls" ->|                       |                    |
+  |                 |                       |                    |
+  |                 |<-- client_tool_call --|                    |
+  |                 |    forward_to_        |                    |
+  |                 |    operator           |                    |
+  |                 |                       |                    |
+  |                 |  1. Find 2 media msgs |                    |
+  |                 |  2. Gen REF: R-AB2X   |                    |
+  |                 |  3. Create OpRequest  |                    |
+  |                 |  4. awaiting=true     |                    |
+  |                 |                       |                    |
+  |                 |-- sendMedia --------->|                    |
+  |                 |   photo1 + caption    |--- photo1+REF ---->|
+  |                 |   (MinIO signed URL)  |                    |
+  |                 |     (500ms delay)     |                    |
+  |                 |                       |                    |
+  |                 |-- sendMedia --------->|                    |
+  |                 |   photo2 + caption    |--- photo2+REF ---->|
+  |                 |   (MinIO signed URL)  |                    |
+  |                 |                       |                    |
+  |                 |-- client_tool_result->|                    |
+  |                 |   {forwarded: true}   |                    |
+  |                 |                       |                    |
+  |                 |<-- agent_response ----|                    |
+  |<-- "Lutfen      |                       |                    |
+  |    bekleyin..." |                       |                    |
+```
+
+### No Media Case
+
+If the patient has not sent any forwardable media in the session, the system sends a text-only message to the doctor with the REF code, patient phone, and any AI note. The media count is 0.
+
+---
+
+## 3. Doctor Reply Flow
+
+### Trigger
+
+The doctor sends a WhatsApp message containing (or quoting) a REF code. This message arrives at the clinic's WhatsApp number and is processed by the webhook handler.
+
+### Step-by-Step
+
+```
+1. Webhook receives message
+       |
+       v
+2. Operator detection
+   - Normalize sender phone (strip non-digits)
+   - Check against known operator phones:
+     a. clinic_operator_phone (operator workflow)
+     b. conversation_forwarding_phone (text forwarding variant)
+   - Also check is_sender=true with valid REF code (operator using same device)
+       |
+       v
+3. Extract REF code
+   Search order:
+     a. Message text:  /\b([A-Z]-[A-Z0-9]{4})\b/
+     b. Quoted text:   payload.quoted_text (doctor replies to the forwarded message)
+       |
+       v
+4. Look up OperatorRequest
+   Query: { ref_code, tenant_id, status IN ('pending','responded') }
+   Sort: newest first (allows follow-up replies)
+       |
+       v
+5. Update OperatorRequest
+   - status = 'responded'
+   - operator_response = message text
+   - responded_at = now
+   - If media attached: response_media_type, response_message_id
+       |
+       v
+6. If doctor sent media (document/image):
+   Forward it directly to the patient via Meta Business API
+       |
+       v
+7. injectOperatorResponse()
+   - Find session and chat
+   - Check if WS connection is still alive
+   - If dead: recreate connection (isReconnect=true) with full chat context
+   - Send contextual_update:
+     "Estetisyen yanit verdi (Ref: {code}): {response}. Bu bilgiyi hastaya
+      dogal ve sicak bir sekilde ilet."
+   - Send user_message: "Estetisyenden yanit geldi."
+   - Clear session.awaiting_operator = false
+       |
+       v
+8. Send confirmation to doctor:
+   "check_mark Yanit alindi. Ref: {code} -- Hastaya iletilecek."
+   or (if media was also forwarded):
+   "check_mark Yanit ve belge hastaya iletildi. Ref: {code}"
+       |
+       v
+9. AI reformulates the doctor's response for the patient
+   (natural, warm tone -- not a verbatim relay)
+```
+
+### Sequence Diagram: Reply Flow
+
+```
+Doctor          Webhook Handler       WhatsAppAgentService       ElevenLabs       Patient
+  |                   |                       |                      |               |
+  |-- "R-AB2X         |                       |                      |               |
+  |   5000TL" ------->|                       |                      |               |
+  |                   |                       |                      |               |
+  |                   | 1. Detect operator    |                      |               |
+  |                   | 2. Extract REF code   |                      |               |
+  |                   | 3. Find OpRequest     |                      |               |
+  |                   | 4. Update: responded  |                      |               |
+  |                   |                       |                      |               |
+  |                   |--- injectOperator --->|                      |               |
+  |                   |    Response()         |                      |               |
+  |                   |                       |                      |               |
+  |                   |                       | WS alive? Yes        |               |
+  |                   |                       |-- contextual_update->|               |
+  |                   |                       |   "Doctor said..."   |               |
+  |                   |                       |-- user_message ----->|               |
+  |                   |                       |   "Yanit geldi"      |               |
+  |                   |                       |                      |               |
+  |                   |                       | awaiting=false       |               |
+  |                   |                       |                      |               |
+  |                   |                       |<-- agent_response ---|               |
+  |                   |                       |    (reformulated)    |               |
+  |                   |                       |                      |               |
+  |                   |                       |-- sendMessage() ----+----- msg ----->|
+  |                   |                       |                      |               |
+  |<-- confirmation --|                       |                      |               |
+  |   "Yanit alindi"  |                       |                      |               |
+```
+
+### WS Reconnection During Reply
+
+If the ElevenLabs WebSocket connection died while waiting for the doctor (which can happen if the doctor takes hours to respond):
+
+1. `injectOperatorResponse()` detects the dead connection
+2. Creates a fresh connection with `isReconnect: true`
+3. Sends full chat context via `contextual_update` (includes the operator request timeline events)
+4. Then sends the doctor's response as `contextual_update`
+5. Sends `user_message` to trigger AI to generate a patient-facing response
+
+### Follow-Up Replies
+
+The doctor can send additional messages with the same REF code after the initial reply:
+- The system detects `status === 'responded'` (already responded) and handles it as a follow-up
+- If the follow-up contains media (document, image), it is forwarded directly to the patient
+- A `contextual_update` notifies the AI about the additional document
+- Confirmation is sent to the doctor
+
+---
+
+## 4. Expiry Flow
+
+Operator requests expire if the doctor does not respond within the configured timeout.
+
+### Configuration
+
+- **Default timeout:** 24 hours (configurable per clinic via `clinic.clinic_operator_timeout_hours` -- read directly, no tenant fallback)
+- **Sweep interval:** Every 15 minutes (BullMQ `whatsapp:operator-expiry` repeatable job)
+
+### Step-by-Step
+
+```
+1. BullMQ sweep fires every 15 minutes
+       |
+       v
+2. Find OperatorRequests where:
+   status = 'pending' AND expires_at < now
+       |
+       v
+3. For each expired request:
+   a. Update status to 'expired'
+   b. Clear session.awaiting_operator = false
+   c. If WS connection is alive:
+      - Send contextual_update:
+        "Doktordan yanit alinamadi (Ref: {code} suresi doldu).
+         Hastaya durumu bildir ve daha sonra donus yapilacagini soyle."
+      - Send user_message: "Doktordan henuz yanit gelmedi."
+      - AI generates a message informing the patient
+       |
+       v
+4. If WS connection is not alive:
+   The expiry is recorded but no AI notification is sent.
+   The patient will be informed when the next message triggers a new session.
+```
+
+---
+
+## 5. Forward Conversation (Text-Only Variant)
+
+A separate tool (`forward_conversation_to_operator`) forwards the conversation text (not media) to a designated phone. This uses the `conversation_forwarding` feature flag and `conversation_forwarding_phone` setting.
+
+The flow is similar but simpler:
+1. AI calls `forward_conversation_to_operator` with a summary and patient messages
+2. System sends a text message to the forwarding phone with the REF code and content
+3. Doctor can reply with the same REF code mechanism
+
+---
+
+## 6. Send Example Photo
+
+A pre-configured guide image (e.g., camera angle instructions, posing guidance) that the AI can send to the patient before they take their photos.
+
+### Flow
+
+1. AI calls `send_example_photo` tool
+2. System loads `operator_example_photo_base64` from tenant settings (data URL)
+3. Strips the `data:image/...;base64,` prefix
+4. Sends via Meta Business API (`sendImage`) with optional caption from `operator_example_photo_caption`
+5. Pre-creates a `WhatsAppMessage` record for deduplication (so the echo webhook does not double-store it)
+
+### Configuration
+
+| Setting | Type | Description |
+|---------|------|-------------|
+| `operator_example_photo_enabled` | boolean | Feature toggle |
+| `operator_example_photo_base64` | string | Data URL of the guide image |
+| `operator_example_photo_caption` | string | Optional caption text |
+
+**Target migration:** The base64 data URL moves from the database to MinIO (`{tenantId}/operator-photos/example-photo.{ext}`), with only the MinIO key stored in the database.
+
+---
+
+## 7. Media Handling
+
+### Architecture
+
+| Media Stage | Storage | Access |
+|-------------|---------|--------|
+| Incoming media | MinIO `{clinicId}/whatsapp-media/{chatId}/{messageId}` | Signed URLs (1hr TTL) |
+| Forwarded media | MinIO `{clinicId}/operator-photos/forwarded/{requestId}/{n}.{ext}` | Signed URLs |
+| Example photo | MinIO `{clinicId}/operator-photos/example-photo.{ext}` | Signed URLs |
+
+### Forwarding Flow
+
+When `forward_to_operator` is called:
+1. Look up media messages in the session (stored in MinIO on receipt from Meta webhook)
+2. Generate a signed URL for each media file from MinIO
+3. Send to the doctor's WhatsApp via Meta Business API `send media` endpoint with the signed URL
+4. 500ms delay between each send (WhatsApp rate limiting)
+
+---
+
+## 8. REF Code System
+
+### Format
+
+```
+{prefix}-{4 alphanumeric characters}
+Pattern:  /\b([A-Z]-[A-Z0-9]{4})\b/
+Examples: R-AB2X, K-9F3M, D-00A1
+```
+
+- **Prefix:** Single uppercase letter, configurable per clinic (`clinic.clinic_operator_prefix`, default `"R"`)
+- **Characters:** Uppercase letters and digits, 4 characters
+- **Scope:** Unique per tenant (not globally unique)
+
+### Generation
+
+```typescript
+async generateRefCode(tenantId: string, prefix: string): Promise<string> {
+  // Generate random 4-char alphanumeric, check for collision within tenant
+  // Retry if collision (extremely unlikely with 36^4 = 1.6M combinations per prefix)
+}
+```
+
+### Matching
+
+REF codes are extracted from:
+1. The message text itself (doctor types the code in their reply)
+2. The `quoted_text` field (doctor uses WhatsApp's reply/quote feature on the forwarded message)
+
+The regex `/\b([A-Z]-[A-Z0-9]{4})\b/` is applied to both.
+
+---
+
+## 9. OperatorRequest Model
+
+### Schema (PostgreSQL Target)
+
+```
+Table: operator_requests
+Prefix: opreq_
+
+Key columns:
+  opreq_id                UUID PK
+  opreq_tenant_id         FK -> tenants
+  opreq_clinic_id         FK -> clinics (NOT NULL -- required)
+  opreq_session_id        FK -> whatsapp_sessions
+  opreq_chat_id           FK -> whatsapp_chats
+  opreq_ref_code          String
+  opreq_request_type      String (default: 'general')  -- not just photo_review; media is optional
+  opreq_request_message   String? (encrypted: AES-256-GCM via MESSAGE_ENCRYPTION_KEY)
+  opreq_status            Enum: pending | responded | expired | cancelled
+  opreq_forwarded_count   Int
+  opreq_failed_count      Int
+  opreq_forwarded_to      String (doctor's phone)
+  opreq_operator_response String? (encrypted: AES-256-GCM via MESSAGE_ENCRYPTION_KEY)
+  opreq_responded_at      DateTime?
+  opreq_expires_at        DateTime
+
+Indexes:
+  @@index([opreq_session_id, opreq_status])
+  @@index([opreq_status, opreq_expires_at])  -- for expiry sweep
+  @@unique([opreq_tenant_id, opreq_ref_code]) -- per-tenant uniqueness
+```
+
+**Encryption:** `opreq_request_message` and `opreq_operator_response` are encrypted at rest using AES-256-GCM with `MESSAGE_ENCRYPTION_KEY`. Decrypted on read only.
+
+### Status Transitions
+
+```
+pending -----> responded   (doctor replied with REF code)
+pending -----> expired     (timeout reached, no reply)
+pending -----> cancelled   (manual cancellation)
+```
+
+### Related Tables
+
+- `OperatorRequestMedia` (target): separate table for `forwarded_media_types[]` and `patient_message_ids[]` (currently embedded arrays in MongoDB)
+
+---
+
+## 10. Configuration Reference
+
+### Clinic-Level Settings (Self-Contained)
+
+All operator configuration is read directly from the clinic record. No tenant-level fallback.
+
+| Setting | Type | Default | Description |
+|---------|------|---------|-------------|
+| `clinic_operator_phone` | string | null | Doctor's WhatsApp number for this clinic |
+| `clinic_operator_prefix` | string | `"R"` | REF code prefix letter |
+| `clinic_operator_timeout_hours` | int | 24 | Hours before operator request expires |
+
+### Feature Flags
+
+| Flag | Required For |
+|------|-------------|
+| `operator_workflow` | `forward_to_operator`, `send_example_photo` tools |
+| `conversation_forwarding` | `forward_conversation_to_operator` tool |