elevenlabs-webhook-nodejs 1.0.0

package/SYSTEM_STATUS.md

@@ -0,0 +1,332 @@
+# Tektip VoiceAI SaaS Platform — System Status
+
+## Overview
+
+Multi-tenant SaaS platform that powers AI voice agents over phone lines and WhatsApp. Receives call data from ElevenLabs via webhooks, processes transcripts with AI (OpenAI/Gemini), manages WhatsApp AI chat sessions via Unipile + ElevenLabs WebSocket, and provides a full management dashboard.
+
+**Codebase**: ~17,000+ lines backend TypeScript + ~5,000+ lines frontend TypeScript/React
+**Status**: MVP feature-complete (Phases 0–6 done), WhatsApp Agent system with session-based architecture, not yet production-hardened
+
+---
+
+## Tech Stack
+
+| Layer | Technology |
+|-------|-----------|
+| Runtime | Node.js + TypeScript 5.9 |
+| Backend | Fastify 5 |
+| Database | MongoDB (Mongoose 8) |
+| Auth | JWT dual-token (access 15min + refresh 7d) with bcrypt |
+| AI | OpenAI / Google Gemini (configurable via `AI_PROVIDER` env) |
+| WhatsApp Agent | Unipile (message relay) + ElevenLabs WebSocket (text-only AI) |
+| Notifications | Twilio WhatsApp |
+| Frontend | React 19 + Vite 7 + Tailwind CSS v4 + shadcn/ui |
+| UI Components | shadcn/ui (18 components), Radix UI primitives, Lucide icons |
+
+---
+
+## Architecture
+
+```
+┌─────────────────┐     ┌──────────────────┐     ┌─────────────┐
+│  ElevenLabs     │────▶│  Fastify API     │────▶│  MongoDB    │
+│  (Webhooks +    │◀───▶│  (Backend)       │     │             │
+│   WebSocket)    │     └──────────────────┘     └─────────────┘
+└─────────────────┘           │  │  ▲
+                              │  │  │
+                              ▼  │  │
+                        ┌────────┤  │
+                        │        ▼  │
+                  ┌─────┴──┐  ┌────┴────┐  ┌──────────┐
+                  │ OpenAI │  │ React   │  │ Unipile  │
+                  │ Gemini │  │ SPA     │  │ WhatsApp │
+                  └────────┘  └─────────┘  └──────────┘
+```
+
+### Backend Modules (`src/modules/`)
+
+| Module | Files | Description |
+|--------|-------|-------------|
+| `auth` | model, routes, service, refresh-token model | Register, login, logout, refresh, `/auth/me` |
+| `tenants` | model, routes | Tenant CRUD, settings, plan management |
+| `users` | routes | User list, invite, delete (per tenant) |
+| `agents` | model, routes | AI agent CRUD, toggle active/inactive |
+| `calls` | model, routes | Call list (paginated + filtered), stats aggregation, detail |
+| `surveys` | model, routes | Survey CRUD, question builder, duplicate |
+| `campaigns` | model, routes, service | Campaign CRUD, state machine, phone list, results + CSV export |
+| `billing` | model, routes, service | Usage summary, event history, call charge logging |
+| `whatsapp` | models (chat, session, message), routes | WhatsApp Agent: chat management, session lifecycle, takeover/release/close |
+| `webhooks` | routes, service | ElevenLabs post-call webhook + Unipile WhatsApp message webhook |
+| `admin` | routes | Super admin: tenant management, feature toggle, system health, usage |
+
+### Plugins (`src/plugins/`)
+
+| Plugin | Purpose |
+|--------|---------|
+| `cors.ts` | CORS configuration |
+| `jwt.ts` | JWT verification + `fastify.authenticate` decorator |
+| `rawBody.ts` | Raw body for webhook signature verification |
+| `rbac.ts` | `requireRole()` + `requireFeature()` middleware. Superadmin bypasses role checks. |
+
+### Frontend Pages (18 total)
+
+| Page | Route | Description |
+|------|-------|-------------|
+| Login | `/login` | Card-based login form |
+| Register | `/register` | Registration with tenant creation |
+| Layout | (shell) | Collapsible sidebar, dropdown user menu, Toaster |
+| Dashboard | `/dashboard` | 8 stat cards + recent calls table |
+| Calls | `/calls` | Paginated table with direction/status filters |
+| CallDetail | `/calls/:id` | Tabs: summary, survey responses, transcript |
+| Settings | `/settings` | Tabs: company info + user management with invite dialog |
+| Billing | `/billing` | Usage stats + transaction history table |
+| Surveys | `/surveys` | Survey list with create dialog, row actions (copy/delete) |
+| SurveyDetail | `/surveys/:id` | Question builder (add/edit/remove), status management |
+| SurveyPreview | `/surveys/:id/preview` | Interactive question-by-question preview |
+| Campaigns | `/campaigns` | Campaign list with status filter + progress bars |
+| CampaignNew | `/campaigns/new` | 3-step wizard: info → schedule → phone list |
+| CampaignDetail | `/campaigns/:id` | Stats, state machine actions, phone list preview |
+| CampaignResults | `/campaigns/:id/results` | Dynamic columns per survey question + CSV export |
+| AdminTenants | `/admin/tenants` | Tenant list with search, create dialog |
+| AdminTenantDetail | `/admin/tenants/:id` | Edit tenant, feature checkboxes, user/agent tables |
+| AdminSystem | `/admin/system` | System health stats + per-tenant usage table |
+| WhatsApp | `/whatsapp`, `/whatsapp/:chatId` | Two-panel WhatsApp Web-style layout: chat list (left) + chat detail (right). Both routes render the same component. |
+
+### Shared Components (`web/src/components/`)
+
+| Component | Usage |
+|-----------|-------|
+| `StatCard` | Dashboard, Billing, CampaignDetail, AdminTenantDetail, AdminSystem |
+| `CallStatusBadge` | Dashboard, Calls, CallDetail, CampaignDetail, CampaignResults |
+| `CampaignStatusBadge` | Campaigns, CampaignDetail |
+| `SurveyStatusBadge` | Surveys |
+| `DirectionBadge` | Dashboard, Calls, CallDetail |
+| `BillingTypeBadge` | Billing |
+| `PageHeader` | All pages |
+| `PageLoader` / `TableLoader` | All pages (skeleton loading) |
+| `EmptyState` | All table pages |
+| `Pagination` | Calls, Campaigns, Billing, AdminTenants |
+| `ChatModeBadge` | WhatsApp (AI/Human/Closed) |
+| `SessionStatusBadge` | WhatsApp (shows "Bekleniyor" for waiting) |
+
+#### WhatsApp Components (`web/src/components/whatsapp/`)
+
+| Component | Purpose |
+|-----------|---------|
+| `ChatList` | Left panel: searchable list of chats with unread indicators |
+| `ChatListItem` | Single chat row in the list (avatar, name, last message preview) |
+| `ChatDetail` | Right panel: full chat view with header, messages, and input |
+| `ChatHeader` | Top bar showing contact info, session status, takeover/release actions |
+| `MessageStream` | Scrollable message area with auto-scroll and session dividers |
+| `MessageBubble` | Individual message bubble (contact/ai/human styling) |
+| `SessionDivider` | Visual separator between sessions with status label |
+| `MessageInput` | Text input with send button, disabled during AI/closed modes |
+| `SessionHistory` | Expandable list of past sessions for a chat |
+| `SettingsDialog` | WhatsApp agent configuration dialog (buffer time, model, etc.) |
+
+---
+
+## Data Models
+
+| Model | Key Fields |
+|-------|-----------|
+| **Tenant** | name, plan (free/starter/pro/premium), enabled_features[], settings. `settings.whatsapp_agent.agent_id` is ObjectId reference to Agent (not raw ElevenLabs string). |
+| **User** | email, password (bcrypt), name, role (superadmin/admin/manager/viewer), tenant_id |
+| **RefreshToken** | user_id, token, expires_at |
+| **Agent** | tenant_id, elevenlabs_agent_id, phone_number, agent_template, direction, config, post_call_processing |
+| **Call** | tenant_id, agent_id, campaign_id, direction, caller/callee_number, duration, status, transcript, llm_summary, survey_responses[] |
+| **Survey** | tenant_id, name, description, questions[], status (draft/active/archived) |
+| **Campaign** | tenant_id, name, survey_id, agent_id, status (draft→scheduled→running→paused→completed), phone_list[], schedule, stats |
+| **BillingEvent** | tenant_id, type (call_charge/subscription/top_up/refund), amount, currency, call_id, description |
+| **WhatsAppChat** | tenant_id, unipile_chat_id, contact_name, contact_phone, is_closed, active_session_id, last_message_at, message_count |
+| **WhatsAppSession** | chat_id, tenant_id, status (waiting/active/resolved), resolved_by (human/ai/timeout), grace_deadline, taken_over_by, elevenlabs_conversation_id, message_count |
+| **WhatsAppMessage** | chat_id, session_id, tenant_id, sender (contact/ai/human), text, sent_via_unipile |
+
+### RBAC
+
+| Role | Access |
+|------|--------|
+| `superadmin` | Everything (bypasses role checks) + admin panel |
+| `admin` | Full tenant access, user management |
+| `manager` | Read + write (create surveys, campaigns, etc.) |
+| `viewer` | Read-only |
+
+### Feature Gating
+
+Features are toggled per tenant via `enabled_features[]`:
+- `inbound_agent` — Inbound call handling
+- `outbound_campaigns` — Campaign management
+- `survey_builder` — Survey creation/management
+- `whatsapp_notifications` — WhatsApp post-call notifications
+- `whatsapp_agent` — WhatsApp AI chat agent (session-based)
+
+Checked at runtime via `requireFeature()` middleware (queries DB, not stale JWT).
+
+---
+
+## Webhook Pipelines
+
+### ElevenLabs Post-Call Webhook
+```
+POST /webhooks/elevenlabs
+  → Verify HMAC-SHA256 signature (webhookValidator)
+  → Respond 200 immediately
+  → Background processing:
+    1. Look up Agent by elevenlabs_agent_id
+    2. Format transcript
+    3. Save Call record to MongoDB
+    4. Run post-call strategy:
+       - "summarize" → AI summary
+       - "extract_survey" → AI survey extraction
+       - "summarize_and_extract" → both
+       - "none" → skip
+    5. Send WhatsApp notification (if configured)
+    6. Log billing charge
+```
+
+### Unipile WhatsApp Webhook
+```
+POST /webhooks/unipile
+  → Validate shared secret header (optional)
+  → Respond 200 immediately
+  → Background processing:
+    1. Find tenant by unipile_account_id
+    2. Validate phone number (reject group chats)
+    3. Check blacklist
+    4. Find/create WhatsAppChat
+    5. Look up or create WhatsAppSession
+    6. Route by session state:
+       - No session → create session (waiting), start grace timer
+       - Waiting → store message (timer running)
+       - Active (AI) → forward to WhatsAppAgentService
+       - Active (human) → store message only
+    7. Self-sent messages → resolve waiting sessions (preserving session_id on the message), detect human takeover
+```
+
+---
+
+## Environment Variables
+
+| Variable | Purpose |
+|----------|---------|
+| `PORT` | Server port |
+| `MONGODB_URI` | MongoDB connection string |
+| `JWT_SECRET` | Access token signing |
+| `JWT_REFRESH_SECRET` | Refresh token signing |
+| `JWT_ACCESS_EXPIRY` | Access token TTL (e.g. "15m") |
+| `JWT_REFRESH_EXPIRY` | Refresh token TTL (e.g. "7d") |
+| `ELEVENLABS_WEBHOOK_SECRET` | Webhook signature verification |
+| `AI_PROVIDER` | "openai" or "gemini" |
+| `OPENAI_API_KEY` / `OPENAI_MODEL` | OpenAI config |
+| `GEMINI_API_KEY` / `GEMINI_MODEL` | Gemini config |
+| `SUMMARY_LANGUAGE` / `SUMMARY_MAX_TOKENS` | AI summary settings |
+| `TWILIO_ACCOUNT_SID` / `TWILIO_AUTH_TOKEN` / `TWILIO_WHATSAPP_NUMBER` | Twilio WhatsApp |
+| `UNIPILE_API_KEY` / `UNIPILE_DSN` | Unipile WhatsApp API |
+| `UNIPILE_WEBHOOK_SECRET` | Unipile webhook validation |
+| `ELEVENLABS_API_KEY` | ElevenLabs SDK (agents, signed URLs) |
+| `ADMIN_PASSWORD` | Legacy admin panel |
+
+---
+
+## Scripts
+
+| Script | Command | Purpose |
+|--------|---------|---------|
+| `seed.ts` | `tsx scripts/seed.ts` | Populate mock data (agents, calls, billing, surveys, campaigns) |
+| `promote.ts` | `tsx scripts/promote.ts` | Promote first user to superadmin |
+| `addRecipient.ts` | `npm run add:recipient` | Add WhatsApp recipient |
+| `listRecipients.ts` | `npm run list:recipients` | List WhatsApp recipients |
+| `retrigger.ts` | `npm run retrigger` | Re-process a webhook payload |
+| `testSend.ts` | `npm run test:send` | Send test WhatsApp message |
+| `migrate-sessions.ts` | `tsx scripts/migrate-sessions.ts` | Migrate WhatsApp chats to session-based architecture (supports `--dry-run`) |
+| `migrate-agent-ref.ts` | `tsx scripts/migrate-agent-ref.ts` | Migrate tenant WhatsApp agent_id from raw string to Agent ObjectId reference (supports `--dry-run`) |
+
+---
+
+## What's Done (Phases 0–6)
+
+- **Phase 0 — Foundation**: Fastify + TS rewrite, MongoDB models, config, error handling, logging (Winston)
+- **Phase 1 — Core Backend**: JWT dual-token auth, RBAC middleware, feature gating, tenant/user management
+- **Phase 2 — Webhook Pipeline**: ElevenLabs webhook receiver, AI summary (OpenAI + Gemini), WhatsApp notifications, call storage
+- **Phase 3 — Survey & Campaign Engine**: Survey builder with 4 question types + branch logic, campaign state machine, phone list management, results aggregation + CSV export
+- **Phase 4 — Billing**: Per-call billing events, usage summary aggregation, event history with pagination
+- **Phase 5 — React Dashboard**: 17-page SPA with auth, all CRUD operations, 3-step campaign wizard, survey preview
+- **Phase 6 — Admin Panel**: Super admin routes, tenant management, feature toggles, system health dashboard, per-tenant usage
+- **UI Overhaul**: Full shadcn/ui migration — Card/Table/Dialog/Select/Badge/Skeleton/Tabs/Progress components, collapsible sidebar with tooltips, Toaster notifications
+- **WhatsApp AI Agent**: Full Unipile integration (QR connection, send/receive), ElevenLabs text-only WebSocket for AI responses, grace period pattern, session-based architecture (WhatsAppSession model), human takeover/release with "immediate response" option, blacklist management, stuck session sweeper, startup recovery
+- **WebSocket reconnection** with exponential backoff (3 retries)
+- **Session inactivity auto-resolve** (30min timeout)
+- **Message buffer/debounce** (20s before AI responds)
+- **Agent model integration** — tenant references Agent by ObjectId, checks `is_active`
+- **WhatsApp UI redesign** — two-panel layout (WhatsApp Web-style) with session tabs and dividers
+
+---
+
+## What's NOT Done / Known Gaps
+
+### Recently Resolved
+
+- ~~No error recovery for WebSocket~~ — Fixed: exponential backoff reconnection with 3 retries
+- ~~WhatsApp UI needs redesign~~ — Fixed: two-panel WhatsApp Web-style layout with component-based architecture
+
+### High Priority (Production Blockers)
+
+1. **No automated tests** — Zero unit/integration/e2e tests exist
+2. **No input validation** — Routes accept unvalidated body payloads (Zod is installed but unused)
+3. **No rate limiting** — API endpoints have no rate limiting
+4. **No HTTPS/TLS** — Assumes reverse proxy handles it
+5. **No password reset flow** — No forgot password / email verification
+6. **No audit logging** — No record of who changed what
+7. **Campaign execution engine** — The campaign state machine exists but there's **no actual dialing engine** (no cron/worker to trigger outbound calls via ElevenLabs API). Campaigns can be created and managed, but calls aren't automatically initiated.
+
+### Medium Priority
+
+8. **No file upload for phone lists** — Only textarea paste (no CSV file upload)
+9. **No real-time updates** — No WebSocket/SSE for live campaign progress
+10. **No pagination on some endpoints** — Surveys list has no pagination
+11. **No search/filter on calls by phone number** — Only direction + status filters
+12. **No dark mode toggle** — CSS variables exist for dark mode but no UI toggle
+13. **No mobile responsive design** — Sidebar doesn't collapse to hamburger on mobile
+14. **No i18n framework** — Turkish strings are hardcoded throughout
+15. **Agent management UI is minimal** — Can list/toggle agents but can't create them from the dashboard (only via admin API)
+
+### Low Priority / Nice-to-Have
+
+16. **No data export beyond campaign results** — No call export, no billing export
+17. **No dashboard date range filters** — Stats are always all-time
+18. **No user profile/password change** — Users can't update their own info
+19. **No email notifications** — Only WhatsApp
+20. **No webhook retry/dead letter queue** — Failed webhooks are logged but not retried
+21. **Code splitting** — Single 541KB JS bundle (warning in build output)
+22. **Legacy routes still registered** — `/legacy-admin` routes from old Express codebase
+
+---
+
+## Running the System
+
+```bash
+# Backend (from root)
+npm run dev          # tsx watch src/index.ts
+
+# Frontend (from web/)
+cd web && npm run dev  # Vite dev server on :5173
+
+# Seed mock data
+tsx scripts/seed.ts
+
+# Promote user to superadmin
+tsx scripts/promote.ts
+
+# Production build
+cd web && npm run build
+npm run build && npm start
+```
+
+**Dev URLs**:
+- Frontend: http://localhost:5173
+- API: http://localhost:5173/api (proxied by Vite)
+- Direct API: http://localhost:3000
+
+---
+
+*Last updated: February 2026*

package/backup_script/main.py

@@ -0,0 +1,146 @@
+#!/usr/bin/env python3
+"""backup_r2.py — MongoDB backup to Cloudflare R2 with Slack notifications."""
+
+import os
+import subprocess
+import json
+import ssl
+import urllib.request
+from datetime import datetime, timedelta
+from pathlib import Path
+
+import boto3
+import certifi
+from dotenv import load_dotenv
+
+# Load .env from parent directory
+load_dotenv(Path(__file__).resolve().parent.parent / ".env")
+
+# ── Config from .env ──
+MONGODB_URI = os.getenv("MONGODB_URI", "mongodb://localhost:27017/asistan724")
+MONGO_USER = os.getenv("MONGO_USER")
+MONGO_PASS = os.getenv("MONGO_PASS")
+BACKUP_DIR = os.getenv("BACKUP_DIR", "/tmp/backups")
+RETENTION_DAYS = int(os.getenv("RETENTION_DAYS", "7"))
+
+R2 = {
+    "bucket": os.getenv("R2_BUCKET"),
+    "endpoint": os.getenv("R2_ENDPOINT"),
+    "access_key": os.getenv("R2_ACCESS_KEY"),
+    "secret_key": os.getenv("R2_SECRET_KEY"),
+}
+
+SLACK_BOT_TOKEN = os.getenv("SLACK_BOT_TOKEN")
+SLACK_CHANNEL = os.getenv("SLACK_CHANNEL")
+
+
+def slack_notify(status: str, message: str):
+    emoji = "✅" if status == "success" else "❌"
+    color = "good" if status == "success" else "danger"
+
+    payload = json.dumps({
+        "channel": SLACK_CHANNEL,
+        "attachments": [{
+            "color": color,
+            "text": f"{emoji} *Backup {status}* — {datetime.now():%Y-%m-%d %H:%M}\n{message}"
+        }]
+    }).encode()
+
+    try:
+        req = urllib.request.Request(
+            "https://slack.com/api/chat.postMessage",
+            data=payload,
+            headers={
+                "Content-Type": "application/json",
+                "Authorization": f"Bearer {SLACK_BOT_TOKEN}"
+            }
+        )
+        context = ssl.create_default_context(cafile=certifi.where())
+        urllib.request.urlopen(req, context=context)
+    except Exception as e:
+        print(f"Slack notification failed: {e}")
+
+
+def get_r2_client():
+    return boto3.client(
+        "s3",
+        endpoint_url=R2["endpoint"],
+        aws_access_key_id=R2["access_key"],
+        aws_secret_access_key=R2["secret_key"],
+    )
+
+
+def dump_mongodb(output_path: str):
+    """Run mongodump and produce a gzipped archive."""
+    print("Dumping MongoDB...")
+    cmd = ["mongodump", "--uri", MONGODB_URI, "--gzip", "--archive"]
+
+    if MONGO_USER and MONGO_PASS:
+        cmd += ["--username", MONGO_USER, "--password", MONGO_PASS,
+                "--authenticationDatabase", "admin"]
+
+    result = subprocess.run(cmd, capture_output=True)
+
+    if result.returncode != 0:
+        raise Exception(f"mongodump failed: {result.stderr.decode()}")
+
+    with open(output_path, "wb") as f:
+        f.write(result.stdout)
+
+
+def upload_to_r2(client, local_path: str, r2_key: str):
+    print(f"Uploading {r2_key}...")
+    client.upload_file(local_path, R2["bucket"], r2_key)
+
+
+def cleanup_old_backups(client):
+    print("Cleaning old backups...")
+    cutoff = datetime.now() - timedelta(days=RETENTION_DAYS)
+
+    paginator = client.get_paginator("list_objects_v2")
+    for page in paginator.paginate(Bucket=R2["bucket"], Prefix="portal-asistan-7-24/mongo_"):
+        for obj in page.get("Contents", []):
+            if obj["LastModified"].replace(tzinfo=None) < cutoff:
+                print(f"  Deleting {obj['Key']}")
+                client.delete_object(Bucket=R2["bucket"], Key=obj["Key"])
+
+
+def file_size_human(path: str) -> str:
+    size = os.path.getsize(path)
+    for unit in ["B", "KB", "MB", "GB"]:
+        if size < 1024:
+            return f"{size:.1f}{unit}"
+        size /= 1024
+    return f"{size:.1f}TB"
+
+
+def main():
+    date_str = datetime.now().strftime("%Y-%m-%d_%H-%M")
+    backup_dir = Path(BACKUP_DIR)
+    backup_dir.mkdir(parents=True, exist_ok=True)
+    hostname = os.uname().nodename
+
+    db_file = str(backup_dir / f"mongo_{date_str}.gz")
+
+    try:
+        client = get_r2_client()
+
+        dump_mongodb(db_file)
+        db_size = file_size_human(db_file)
+        upload_to_r2(client, db_file, f"portal-asistan-7-24/mongo_{date_str}.gz")
+
+        cleanup_old_backups(client)
+        slack_notify("success", f"DB: {db_size}\nServer: {hostname}")
+        print("Backup complete!")
+
+    except Exception as e:
+        slack_notify("error", f"Failed: {e}\nServer: {hostname}")
+        print(f"Backup failed: {e}")
+
+    finally:
+        if os.path.exists(db_file):
+            os.remove(db_file)
+
+
+if __name__ == "__main__":
+    main()

package/baileys/.dockerignore

@@ -0,0 +1,7 @@
+node_modules
+data
+auth_state
+baileys.log
+index.js
+*.md
+.env

package/baileys/Dockerfile

@@ -0,0 +1,13 @@
+FROM node:22-alpine
+
+WORKDIR /app
+
+COPY package.json package-lock.json* ./
+RUN npm ci --omit=dev
+
+COPY server.js worker.js ./
+COPY src ./src
+
+EXPOSE 3001
+
+CMD ["node", "server.js"]