elevenlabs-webhook-nodejs 1.0.0

package/docs/architecture/13-LOGGING-AUDIT.md

@@ -0,0 +1,549 @@
+# 13. Logging and Audit Trail
+
+> Application logging for debugging/monitoring and audit logging for compliance/accountability.
+> Last updated: 2026-04-02
+
+---
+
+## Application Logging (Pino)
+
+### Why Pino
+
+- **Native to Fastify**: NestJS Fastify adapter uses Pino natively -- zero additional overhead.
+- **5x faster than Winston**: Matters at scale. Pino avoids synchronous string formatting.
+- **Structured JSON by default**: Every log entry is a parseable JSON object.
+- **`pino-pretty` for dev**: Human-readable console output during development.
+- **Async write**: Never blocks the event loop.
+
+The current system uses Winston with a custom MongoDB transport. The migration replaces this with Pino + PostgreSQL.
+
+### Architecture
+
+```
+NestJS App
+  |
+  +-- Pino Logger (via @nestjs/platform-fastify built-in)
+  |     +-- Console transport (pretty-print in dev, JSON in prod)
+  |     +-- PostgreSQL transport (async, batched -- replaces MongoDB)
+  |     +-- File transport (optional, for log rotation/archival)
+  |
+  +-- Request Logger (Fastify hook -- auto on every request)
+  |     +-- method, url, status, duration, userId, tenantId, clinicId, requestId
+  |
+  +-- Correlation ID (X-Request-Id header or auto-generated UUID)
+        +-- Propagated to all service calls, queue jobs, external API calls
+```
+
+### Log Levels
+
+| Level | When to Use | Example |
+|-------|------------|---------|
+| `fatal` | Process cannot continue | Database connection lost, unrecoverable state |
+| `error` | Operation failed, needs attention | ElevenLabs WS disconnect, Google Calendar API 500, payment failed |
+| `warn` | Unexpected but handled | Retry triggered, rate limit approaching, token refresh failed then succeeded |
+| `info` | Business events | Session created, appointment booked, call completed, batch submitted |
+| `debug` | Developer troubleshooting | WS message received, tool call params, queue job started |
+| `trace` | Extreme detail | Raw HTTP payloads, full WS frames (dev only) |
+
+**Production:** `info` and above go to PostgreSQL. `debug` and `trace` stay in console/file only -- they are too noisy for the database.
+
+**Development:** `debug` and above to console (via `pino-pretty`).
+
+### Structured Log Format
+
+Every log entry includes auto-injected context. No string interpolation -- all data is structured.
+
+```typescript
+{
+  // Auto-injected by Pino
+  level: 'info',
+  time: 1711700000000,
+  pid: 12345,
+  hostname: 'api-1',
+
+  // Injected by request context (AsyncLocalStorage)
+  requestId: 'req_abc123',        // Correlation ID
+  tenantId: 'tenant_xyz',         // null for platform routes
+  clinicId: 'clinic_456',         // null if not clinic-scoped
+  userId: 'user_789',             // null for unauthenticated
+
+  // Log-specific
+  msg: 'Appointment booked',
+  service: 'calendar.tools',
+  data: {
+    appointmentId: '...',
+    patientPhone: '+905551234567',
+    startTime: '2026-04-15T09:00:00Z',
+  },
+}
+```
+
+### AsyncLocalStorage for Request Context Propagation
+
+A correlation ID and user context are propagated across the entire request lifecycle without passing parameters through every function call. This is achieved via Node.js `AsyncLocalStorage`.
+
+```typescript
+@Injectable()
+export class RequestContextInterceptor implements NestInterceptor {
+  constructor(private readonly als: AsyncLocalStorage<RequestContext>) {}
+
+  intercept(context: ExecutionContext, next: CallHandler) {
+    const request = context.switchToHttp().getRequest();
+    const ctx: RequestContext = {
+      requestId: request.headers['x-request-id'] || randomUUID(),
+      tenantId: request.user?.tenantId ?? null,
+      clinicId: null,  // Set later by ClinicAccessGuard
+      userId: request.user?.sub ?? null,
+    };
+
+    return new Observable((subscriber) => {
+      this.als.run(ctx, () => {
+        next.handle().subscribe(subscriber);
+      });
+    });
+  }
+}
+```
+
+Usage anywhere in the codebase:
+
+```typescript
+const ctx = this.als.getStore();
+this.logger.info({ ...ctx, msg: 'Appointment booked', data: { appointmentId } });
+```
+
+The correlation ID (`requestId`) follows a request through service calls, queue jobs, and external API calls, making it possible to trace an entire flow from a single ID.
+
+### PostgreSQL Log Storage
+
+Only `info` and above go to PostgreSQL. The `logs` table serves the admin dashboard.
+
+```prisma
+model Log {
+  log_id         String   @id @default(uuid())
+  log_level      String                        // fatal, error, warn, info
+  log_message    String
+  log_service    String   @default("api")      // api, whatsapp-agent, billing, etc.
+  log_meta       Json     @default("{}")       // Structured data payload
+  log_request_id String?                       // Correlation ID
+  log_tenant_id  String?
+  log_clinic_id  String?
+  log_user_id    String?
+  log_timestamp  DateTime @default(now())
+
+  @@index([log_level])
+  @@index([log_timestamp])
+  @@index([log_level, log_timestamp(sort: Desc)])
+  @@index([log_tenant_id, log_timestamp(sort: Desc)])
+  @@index([log_request_id])
+  @@index([log_service, log_timestamp(sort: Desc)])
+  @@map("logs")
+}
+```
+
+### Async Batched PostgreSQL Writer
+
+Logs are never written synchronously to PostgreSQL. The writer buffers entries and flushes in batches to avoid per-log write overhead.
+
+```typescript
+@Injectable()
+export class PgLogTransport {
+  private buffer: LogEntry[] = [];
+  private readonly MAX_BUFFER = 100;
+  private readonly FLUSH_INTERVAL = 5_000; // 5 seconds
+
+  constructor(private readonly prisma: PrismaService) {
+    setInterval(() => this.flush(), this.FLUSH_INTERVAL);
+  }
+
+  write(entry: LogEntry) {
+    this.buffer.push(entry);
+    if (this.buffer.length >= this.MAX_BUFFER) {
+      this.flush();
+    }
+  }
+
+  private async flush() {
+    if (this.buffer.length === 0) return;
+    const batch = this.buffer.splice(0);
+    await this.prisma.log.createMany({ data: batch }).catch(() => {
+      // Never let logging failure crash the app
+    });
+  }
+}
+```
+
+**Flush triggers:**
+
+- Buffer reaches 100 entries.
+- 5-second interval timer fires.
+- Whichever comes first.
+
+A failed flush is silently swallowed. Application stability always takes priority over log persistence.
+
+### Log Cleanup
+
+BullMQ repeatable job `log:cleanup` runs daily at 03:00 (Europe/Istanbul). Deletes logs older than the configured retention period.
+
+```typescript
+const retentionDays = tenant?.settings?.log_retention_days ?? 30;
+await this.prisma.log.deleteMany({
+  where: {
+    tenantId,
+    timestamp: { lt: subDays(new Date(), retentionDays) },
+  },
+});
+```
+
+Default retention: **30 days**. Per-tenant override possible via `tenant.settings.log_retention_days`.
+
+### Log Endpoints
+
+| Method | Path | Permission | Description |
+|--------|------|-----------|-------------|
+| GET | `/admin/logs` | `audit:read` | Paginated, filterable by level, service, tenant, clinic, requestId, date range |
+| GET | `/admin/logs/stats` | `audit:read` | Count by level (cached 30s in Redis) |
+| DELETE | `/admin/logs` | superadmin only | Delete logs older than specified date |
+| GET | `/admin/logs/request/:requestId` | `audit:read` | All logs for a single request (trace a full flow end-to-end) |
+
+### Business Event Catalog (30+ Events)
+
+| Service | Event | Level | Data |
+|---------|-------|-------|------|
+| **Auth** | Login success | info | userId, email, ip |
+| **Auth** | Login failure | warn | email, ip, reason |
+| **Auth** | Token refreshed / revoked | info | userId |
+| **WhatsApp** | Session created | info | chatId, sessionId |
+| **WhatsApp** | Session resolved | info | chatId, sessionId, resolvedBy |
+| **WhatsApp** | AI response sent | info | chatId, messageLength, durationMs |
+| **WhatsApp** | WS connection opened | info | chatId, agentId |
+| **WhatsApp** | WS connection closed | info | chatId, agentId, reason |
+| **WhatsApp** | Message buffer triggered | debug | sessionId, messageCount |
+| **Tools** | Tool executed | info | toolName, channel, durationMs, isError, toollog_clinic_id |
+| **Operator** | Request created | info | refCode, sessionId |
+| **Operator** | Request responded | info | refCode, sessionId |
+| **Operator** | Request expired | info | refCode, sessionId |
+| **Calendar** | Slot reserved | info | appointmentId, patientPhone, startTime |
+| **Calendar** | Slot cancelled | info | appointmentId, patientPhone |
+| **ElevenLabs** | Webhook received | info | conversationId, agentId, status |
+| **ElevenLabs** | Post-call summary generated | info | callId, model, tokenUsage |
+| **Billing** | Charge logged | info | tenantId, callId, amount |
+| **Billing** | Alert sent | warn | tenantId, alertType, threshold |
+| **Campaigns** | Call dispatched | info | campaignId, entryId |
+| **Campaigns** | Call completed | info | campaignId, entryId, status |
+| **Validation** | Batch submitted | info | batchId, totalEntries |
+| **Validation** | Batch completed | info | batchId, completedCount |
+| **Bridge** | Message sent | info | provider, chatId |
+| **Bridge** | Message failed | error | provider, chatId, error |
+| **Meta** | Webhook received | info | wabaId, messageType |
+| **NetGSM** | SMS sent | info | phones, jobId |
+| **NetGSM** | SMS failed | error | phones, error |
+| **Payments** | Payment link created | info | paymentId, amount |
+| **Payments** | Payment completed | info | paymentId, amount, status |
+| **WhatsApp** | Message cleanup executed | info | jobName: `whatsapp:message-cleanup`, deletedCount, retentionDays |
+| **Patient** | Patient created | info | patientId, phone, clinicId |
+| **Patient** | Patient updated | info | patientId, changes |
+| **Patient** | Patient merged | info | primaryPatientId, mergedPatientId |
+| **Queue** | Job failed after retries | error | queue, jobId, error, attempts |
+
+---
+
+## Audit Logging
+
+### Purpose
+
+Track **who did what, when, and what changed**. Audit logs are separate from application logs. They answer questions like: "Who changed the grace period from 180 to 300, and when?"
+
+Audit logs serve three purposes:
+
+- **Compliance**: Regulatory requirements (KVKK) demand traceability of data access and changes.
+- **Accountability**: Clinic owners can see what their staff did and when.
+- **Forensics**: Investigating incidents after the fact.
+
+### Key Properties
+
+- **Immutable**: Audit logs are append-only. They are never updated.
+- **Never auto-deleted**: Unlike application logs, audit logs are not subject to cleanup jobs. Retention is permanent (or years per regulation).
+- **Synchronous writes**: Audit events are written to PostgreSQL synchronously. They must not be lost, even at the cost of a small latency increase.
+- **Real-time push**: New audit events are emitted via WebSocket to the dashboard for live monitoring.
+
+### Architecture
+
+```
+User Action (HTTP request)
+  |
+  +-- AuditInterceptor (NestJS interceptor on mutating endpoints)
+  |     +-- Captures: before state -> after state -> diff
+  |
+  +-- EventEmitter (for non-HTTP actions)
+        +-- e.g., AI booking appointment, operator request expiry
+
+  |
+  v
+
+AuditService.log(event)
+  |
+  +-- Write to PostgreSQL (audit_logs table) -- synchronous, never batched
+  +-- Emit WebSocket event to dashboard (real-time audit feed)
+```
+
+### Prisma Model
+
+```prisma
+model AuditLog {
+  id         String   @id @default(uuid())
+  tenantId   String?                        // null for platform actions (sales creating tenant)
+  clinicId   String?                        // null for tenant-level actions
+  userId     String?                        // null for system actions (cron, AI)
+  userEmail  String?                        // Denormalized -- survives user deletion
+  userRole   String?                        // Role at time of action
+  action     String                         // e.g., 'tenant.settings.updated'
+  entity     String                         // e.g., 'Tenant', 'Appointment'
+  entityId   String                         // ID of affected record
+  changes    Json?                          // {field: {old: x, new: y}} -- null for non-update actions
+  metadata   Json?                          // Extra context: {ip, userAgent, requestId, reason}
+  createdAt  DateTime @default(now())
+
+  // NO updatedAt -- audit logs are immutable
+
+  @@index([tenantId, createdAt(sort: Desc)])
+  @@index([clinicId, createdAt(sort: Desc)])
+  @@index([entity, entityId])
+  @@index([userId, createdAt(sort: Desc)])
+  @@index([action, createdAt(sort: Desc)])
+  @@map("audit_logs")
+}
+```
+
+**Note:** `userEmail` and `userRole` are denormalized (copied at write time). This ensures the audit trail remains accurate even if the user is later deleted or their role changes.
+
+### AuditService
+
+```typescript
+@Injectable()
+export class AuditService {
+  constructor(
+    private readonly prisma: PrismaService,
+    private readonly eventEmitter: EventEmitter2,
+  ) {}
+
+  async log(event: AuditEvent): Promise<void> {
+    const entry = await this.prisma.auditLog.create({
+      data: {
+        tenantId: event.tenantId,
+        clinicId: event.clinicId,
+        userId: event.userId,
+        userEmail: event.userEmail,
+        userRole: event.userRole,
+        action: event.action,
+        entity: event.entity,
+        entityId: event.entityId,
+        changes: event.changes ?? undefined,
+        metadata: {
+          ip: event.ip,
+          userAgent: event.userAgent,
+          requestId: event.requestId,
+          ...(event.reason && { reason: event.reason }),
+        },
+      },
+    });
+
+    // Push to dashboard in real-time
+    this.eventEmitter.emit('audit.created', entry);
+  }
+
+  static diff(
+    before: Record<string, any>,
+    after: Record<string, any>,
+  ): Record<string, { old: any; new: any }> | null {
+    const changes: Record<string, { old: any; new: any }> = {};
+    for (const key of new Set([...Object.keys(before), ...Object.keys(after)])) {
+      if (JSON.stringify(before[key]) !== JSON.stringify(after[key])) {
+        changes[key] = { old: before[key], new: after[key] };
+      }
+    }
+    return Object.keys(changes).length > 0 ? changes : null;
+  }
+}
+```
+
+The `diff()` method compares two object snapshots and produces a `{ field: { old, new } }` change record. It uses JSON serialization for deep comparison.
+
+### @Auditable() Decorator
+
+The `@Auditable()` decorator auto-wraps controller methods with before/after capture. It eliminates the need to manually call `AuditService.log()` on every mutating endpoint.
+
+```typescript
+export function Auditable(action: string, entity: string) {
+  return applyDecorators(
+    SetMetadata('audit:action', action),
+    SetMetadata('audit:entity', entity),
+    UseInterceptors(AuditInterceptor),
+  );
+}
+
+// Usage:
+@Put(':id')
+@Auditable('clinic.settings.updated', 'Clinic')
+async updateClinic(@Param('id') id: string, @Body() dto: UpdateClinicDto) {
+  return this.clinicsService.update(id, dto);
+}
+```
+
+**AuditInterceptor behavior:**
+
+1. **Before**: Fetch current state of the entity (by ID from route params).
+2. **Execute**: Run the controller handler.
+3. **After**: Fetch new state, compute diff via `AuditService.diff()`.
+4. **Write**: Create audit log entry with the changes.
+
+### Complete Action Catalog (30+ Actions)
+
+**Auth Actions:**
+
+| Action | Trigger |
+|--------|---------|
+| `user.login` | Successful login |
+| `user.login.failed` | Failed login attempt (track brute force) |
+| `user.logout` | Logout |
+| `user.password.changed` | Password change |
+| `user.password.reset` | Password reset via token |
+| `user.2fa.enabled` | 2FA enabled |
+| `user.2fa.disabled` | 2FA disabled |
+
+**User Management:**
+
+| Action | Trigger |
+|--------|---------|
+| `user.created` | New user invited |
+| `user.updated` | Name, role, or active status changed |
+| `user.deleted` | User deactivated/removed |
+| `user.clinics.updated` | Clinic assignments changed |
+
+**Tenant and Clinic:**
+
+| Action | Trigger |
+|--------|---------|
+| `tenant.created` | New tenant onboarded (by sales/superadmin) |
+| `tenant.settings.updated` | Global settings changed |
+| `tenant.features.toggled` | Feature flags changed |
+| `tenant.deactivated` | Tenant deactivated |
+| `tenant.reactivated` | Tenant reactivated |
+| `clinic.created` | New clinic created |
+| `clinic.updated` | Clinic settings changed |
+| `clinic.deleted` | Clinic deleted |
+
+**Phone Numbers:**
+
+| Action | Trigger |
+|--------|---------|
+| `phone.created` | Phone number added |
+| `phone.updated` | Phone number settings changed |
+| `phone.deleted` | Phone number removed |
+| `phone.whatsapp.connected` | WhatsApp connection established |
+| `phone.whatsapp.disconnected` | WhatsApp disconnected |
+
+**WhatsApp:**
+
+| Action | Trigger |
+|--------|---------|
+| `session.taken_over` | Human takes over from AI |
+| `session.released` | Human releases back to AI |
+| `session.resolved.manual` | Manual resolution |
+| `chat.message.sent.human` | Human sent message in live chat |
+| `blacklist.added` | Number blacklisted |
+| `blacklist.removed` | Number removed from blacklist |
+
+**Calendar:**
+
+| Action | Trigger |
+|--------|---------|
+| `calendar.connected` | Google Calendar OAuth completed |
+| `calendar.disconnected` | Google Calendar disconnected |
+| `appointment.created` | Appointment booked (by AI or human) |
+| `appointment.cancelled` | Appointment cancelled (by AI or human) |
+| `appointment.rescheduled` | Appointment rescheduled |
+
+**Campaigns and Validation:**
+
+| Action | Trigger |
+|--------|---------|
+| `campaign.created` | New outbound campaign created |
+| `campaign.started` | Campaign started |
+| `campaign.stopped` | Campaign stopped |
+| `batch.created` | Validation batch created |
+| `batch.submitted` | Validation batch submitted for calling |
+| `batch.cancelled` | Validation batch cancelled |
+
+**Patient and Clinical:**
+
+| Action | Trigger |
+|--------|---------|
+| `patient.created` | New patient record created |
+| `patient.updated` | Patient info changed |
+| `patient.merged` | Duplicate patient records merged |
+| `treatment_plan.created` | New treatment plan created for patient |
+| `treatment_plan.completed` | Treatment plan marked as completed |
+| `examination.created` | New examination record created |
+
+**Compliance (KVKK):**
+
+| Action | Trigger |
+|--------|---------|
+| `kvkk.export.requested` | Data export requested |
+| `kvkk.deletion.requested` | Data deletion requested |
+| `kvkk.deletion.completed` | Data deletion executed |
+
+**Platform (Sales/Superadmin):**
+
+| Action | Trigger |
+|--------|---------|
+| `platform.tenant.created` | Sales created new tenant |
+| `platform.tenant.plan.changed` | Plan changed by sales |
+| `platform.tenant.deactivated` | Tenant deactivated by sales |
+
+---
+
+## Key Differences: Application Logs vs. Audit Logs
+
+| | Application Logs | Audit Logs |
+|---|---|---|
+| **Purpose** | Debugging, monitoring, alerting | Compliance, accountability, forensics |
+| **Who reads** | Developers, ops | Clinic owners, compliance officers |
+| **Retention** | 30 days (configurable per tenant) | Permanent (or years per regulation) |
+| **Write mode** | Async, batched (buffer 100, flush every 5s) | Synchronous -- must not be lost |
+| **Content** | What happened technically | Who did what to which record, with before/after diff |
+| **Deletable** | Yes (BullMQ cleanup job) | No (immutable, append-only) |
+| **Filterable by** | level, service, requestId, date | user, action, entity, tenant, date range |
+| **Stored in** | `logs` table | `audit_logs` table |
+| **Real-time push** | No | Yes (WebSocket to dashboard) |
+| **Includes diff** | No | Yes (`changes` field with `{old, new}` per field) |
+| **User identity** | Optional (`userId` from context) | Required (denormalized `userEmail`, `userRole`) |
+
+---
+
+## Endpoints
+
+### Application Log Endpoints
+
+| Method | Path | Permission | Description |
+|--------|------|-----------|-------------|
+| GET | `/admin/logs` | `audit:read` | Paginated, filterable by level, service, tenant, clinic, requestId, date range |
+| GET | `/admin/logs/stats` | `audit:read` | Count by level (cached 30s in Redis) |
+| DELETE | `/admin/logs` | superadmin only | Delete logs older than specified date |
+| GET | `/admin/logs/request/:requestId` | `audit:read` | All logs for a single request (end-to-end trace) |
+
+### Audit Log Endpoints
+
+| Method | Path | Permission | Description |
+|--------|------|-----------|-------------|
+| GET | `/audit-logs` | `audit:read` | Tenant-scoped audit trail (owner/admin sees their tenant) |
+| GET | `/audit-logs/entity/:entity/:entityId` | `audit:read` | Full history of a specific record |
+| GET | `/audit-logs/user/:userId` | `audit:read` | All actions by a specific user |
+| GET | `/platform/audit-logs` | superadmin/sales | Cross-tenant audit trail |
+
+**Query parameters (both endpoints):** `action`, `entity`, `userId`, `clinicId`, `from`, `to`, `page`, `limit`
+
+### Audit Log Archival
+
+Audit logs are never auto-deleted. For very large tenants, old audit logs (older than 2 years) can be archived to MinIO as compressed JSON and purged from PostgreSQL. This is a manual superadmin operation, not an automatic process.