elevenlabs-webhook-nodejs 1.0.0

package/docs/architecture/18-FILE-STORAGE.md

@@ -0,0 +1,410 @@
+# File Storage (MinIO)
+
+> S3-compatible object storage -- per-tenant, server-side encrypted, signed URLs only
+> Last updated: 2026-04-02
+
+---
+
+## 1. Why MinIO
+
+MinIO is a self-hosted S3-compatible object storage server. It replaces all binary data currently stored in MongoDB (base64 in documents) and in-memory caches.
+
+| Requirement | Current Solution | Problem | MinIO |
+|-------------|-----------------|---------|-------|
+| KVKK compliance (data in Turkey) | MongoDB (local) | Binary in DB is inefficient | Self-hosted, data stays on-premises |
+| Per-tenant isolation | None | Anyone with DB access sees all data | Bucket paths prefixed with `{tenantId}/` |
+| Server-side encryption | None | Plain base64 in database | SSE-S3 (AES-256) managed by MinIO |
+| Access control | None | Direct field access | Signed URLs with 1-hour TTL, never public |
+| Large files | Base64 in MongoDB docs (16MB BSON limit) | Size limits, bloated documents | No practical limit per object |
+| Meta webhook media | Download from Meta API on receipt | Ephemeral Meta URLs expire | Persistent, independent of process lifecycle |
+| Cost | Included in MongoDB | Wastes DB resources on binary | Dedicated storage, cheap disk |
+
+---
+
+## 2. Environment Variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `MINIO_ENDPOINT` | MinIO hostname | `minio` (Docker) or `localhost` |
+| `MINIO_PORT` | MinIO port | `9000` |
+| `MINIO_ACCESS_KEY` | Access key | `minioadmin` |
+| `MINIO_SECRET_KEY` | Secret key | `minioadmin` |
+| `MINIO_USE_SSL` | Enable TLS | `false` (Docker internal), `true` (production) |
+| `MINIO_BUCKET` | Default bucket name | `portal-asistan` |
+
+In production, MinIO runs inside the Docker Compose network (`asistan724-net`) with no port binding to the host. Only the NestJS application can reach it.
+
+---
+
+## 3. Bucket Structure
+
+A single bucket (`portal-asistan`) with tenant-prefixed paths:
+
+```
+portal-asistan/
+  {tenantId}/
+    patients/{patientId}/
+      documents/{documentId}.{ext}           # Patient documents (uploaded by staff)
+      photos/{photoSetId}/{n}.{ext}          # Patient photo sets (before/after, exam photos)
+    operator-photos/
+      example-photo.{ext}                    # Operator example photo (camera angle guide)
+      forwarded/{requestId}/{n}.{ext}        # Media forwarded to doctor per operator request
+    whatsapp-media/
+      {chatId}/{messageId}.{ext}             # Incoming media from patients (downloaded from Meta on webhook receipt)
+      audio/{messageId}.{ext}                # Voice messages (downloaded for STT)
+    exports/
+      leads/{timestamp}.csv                  # Lead export files
+      outbound-campaigns/{campaignId}.csv    # Campaign result exports
+      validations/{batchId}.csv              # Appointment validation exports
+      kvkk/{phone}_{timestamp}.zip           # KVKK data export packages
+    recordings/
+      {conversationId}.{ext}                 # Call recordings (mirrored from ElevenLabs)
+```
+
+### Path Conventions
+
+- All paths start with `{tenantId}/` for tenant isolation
+- Subdirectories group by functional area
+- Files are named with their source ID (messageId, requestId, conversationId) for traceability
+- Extensions are preserved from the original file
+
+### Tenant Isolation
+
+MinIO itself does not enforce per-tenant access. Isolation is enforced at the application layer:
+
+1. `MinioService` requires `tenantId` as the first parameter on all operations
+2. All object keys are constructed as `${tenantId}/...` inside the service
+3. Signed URLs are generated per-object -- a URL for tenant A's file cannot access tenant B's files
+4. The `EncryptionService` is not used for MinIO -- SSE-S3 handles encryption transparently
+
+---
+
+## 4. Signed URLs
+
+All file access goes through pre-signed URLs. No MinIO object is ever publicly accessible.
+
+### How Signed URLs Work
+
+1. The application calls `minioClient.presignedGetObject(bucket, key, ttlSeconds)`
+2. MinIO returns a URL with embedded authentication parameters (signature, expiry)
+3. The URL is valid for the specified TTL (default 1 hour)
+4. After TTL expires, the URL returns 403 Forbidden
+5. The URL is specific to one object -- it cannot be used to access other files
+
+### TTL Policy
+
+| Use Case | TTL | Rationale |
+|----------|-----|-----------|
+| Dashboard media display | 1 hour | Covers a normal browsing session |
+| API response (media URL) | 1 hour | Frontend caches URL briefly |
+| KVKK data export download | 1 hour | Admin downloads immediately after generation |
+| CSV export download | 1 hour | User downloads from export UI |
+
+### Never Public
+
+```typescript
+// WRONG -- never do this:
+minioClient.setBucketPolicy(bucket, publicReadPolicy);
+
+// RIGHT -- always signed:
+const url = await minioClient.presignedGetObject(bucket, key, 3600);
+```
+
+The MinIO bucket policy is set to private. No anonymous access is possible.
+
+---
+
+## 5. What Moved to MinIO
+
+### 5.1 Operator Example Photo
+
+**Current:** `tenant.settings.operator_example_photo_base64` -- a data URL (base64-encoded image) stored directly in the MongoDB tenant document. Can be several megabytes, bloating the document.
+
+**Target:** MinIO object at `{tenantId}/operator-photos/example-photo.{ext}`. The database stores only the MinIO key (`clinic_settings.operator_example_photo_key`).
+
+```typescript
+// Upload (when tenant configures example photo):
+const key = `${tenantId}/operator-photos/example-photo.png`;
+await this.minioService.putObject(key, buffer, 'image/png');
+await this.prisma.clinic.update({
+  where: { clinic_id: clinicId },
+  data: { clinic_settings: { ...settings, operator_example_photo_key: key } },
+});
+
+// Use (when AI tool sends example photo to patient):
+const url = await this.minioService.presignedGetObject(key);
+await this.metaWhatsAppService.sendImage(phoneNumberId, recipientPhone, url, caption);
+```
+
+### 5.2 Meta Webhook Media
+
+**Current:** N/A (Baileys used in-memory `messageCache` Map -- lost on restart, consumed RAM).
+
+**Target:** When a Meta webhook delivers a media message, the media is downloaded from the Meta CDN immediately and stored in MinIO at `{tenantId}/whatsapp-media/{chatId}/{messageId}.{ext}`. The MinIO key is saved on the `WhatsAppMessage` record. Subsequent access (forwarding, display) uses signed URLs.
+
+```typescript
+// In Meta webhook handler, on media message received:
+const mediaUrl = await this.metaWhatsAppService.getMediaUrl(mediaId, accessToken);
+const response = await fetch(mediaUrl, { headers: { Authorization: `Bearer ${accessToken}` } });
+const buffer = Buffer.from(await response.arrayBuffer());
+const key = `${tenantId}/whatsapp-media/${chatId}/${messageId}.jpg`;
+await this.minioService.putObject(key, buffer, 'image/jpeg');
+
+// On forward or display:
+const url = await this.minioService.presignedGetObject(key, 3600);
+```
+
+Media is stored persistently in MinIO on receipt. There is no in-memory cache.
+
+### 5.3 CSV Exports
+
+**Current:** Generated on-the-fly and streamed directly to the HTTP response. No persistence -- if the request fails, the user must regenerate.
+
+**Target:** Generated, stored in MinIO at `{tenantId}/exports/...`, and served via signed URL. Auto-deleted after 24 hours by a cleanup job.
+
+```typescript
+// Generate and store:
+const csvBuffer = await this.generateLeadsCsv(tenantId, filters);
+const key = `${tenantId}/exports/leads/${Date.now()}.csv`;
+await this.minioService.putObject(key, csvBuffer, 'text/csv');
+const url = await this.minioService.presignedGetObject(key);
+
+// Return URL to frontend:
+return { downloadUrl: url };
+
+// Cleanup (daily job):
+// Delete all objects in {tenantId}/exports/ older than 24 hours
+```
+
+### 5.4 Call Recordings
+
+**Current:** `Call.recording_url` points to an external ElevenLabs URL. Ephemeral -- ElevenLabs may delete recordings after a retention period.
+
+**Target:** Optionally mirrored to MinIO at `{tenantId}/recordings/{conversationId}.{ext}`. The `CallDetail.detail_recording_key` stores the MinIO key. If mirrored, the signed URL from MinIO is served instead of the external URL.
+
+```typescript
+// In post-call webhook processor:
+if (recordingUrl && tenant.settings.mirror_recordings) {
+  const response = await fetch(recordingUrl);
+  const buffer = Buffer.from(await response.arrayBuffer());
+  const key = `${tenantId}/recordings/${conversationId}.mp3`;
+  await this.minioService.putObject(key, buffer, 'audio/mpeg');
+  await this.prisma.callDetail.update({
+    where: { detail_call_id: callId },
+    data: { detail_recording_key: key },
+  });
+}
+```
+
+---
+
+## 6. NestJS MinioService
+
+```typescript
+// src/integrations/minio/minio.service.ts
+
+@Injectable()
+export class MinioService implements OnModuleInit {
+  private client: Client;
+  private readonly bucket: string;
+
+  constructor(private readonly configService: ConfigService) {
+    this.bucket = configService.get('MINIO_BUCKET') ?? 'portal-asistan';
+    this.client = new Client({
+      endPoint: configService.get('MINIO_ENDPOINT'),
+      port: parseInt(configService.get('MINIO_PORT') ?? '9000'),
+      useSSL: configService.get('MINIO_USE_SSL') === 'true',
+      accessKey: configService.get('MINIO_ACCESS_KEY'),
+      secretKey: configService.get('MINIO_SECRET_KEY'),
+    });
+  }
+
+  async onModuleInit() {
+    // Ensure bucket exists on startup
+    const exists = await this.client.bucketExists(this.bucket);
+    if (!exists) {
+      await this.client.makeBucket(this.bucket);
+    }
+  }
+
+  /**
+   * Upload an object.
+   * Key must start with tenantId/ -- enforced by caller.
+   */
+  async putObject(
+    key: string,
+    data: Buffer | Readable,
+    contentType: string,
+  ): Promise<void> {
+    await this.client.putObject(this.bucket, key, data, {
+      'Content-Type': contentType,
+    });
+  }
+
+  /**
+   * Generate a pre-signed GET URL.
+   * Default TTL: 1 hour (3600 seconds).
+   */
+  async presignedGetObject(key: string, ttlSeconds = 3600): Promise<string> {
+    return this.client.presignedGetObject(this.bucket, key, ttlSeconds);
+  }
+
+  /**
+   * Delete an object.
+   */
+  async removeObject(key: string): Promise<void> {
+    await this.client.removeObject(this.bucket, key);
+  }
+
+  /**
+   * Delete all objects under a prefix.
+   * Used for tenant data deletion (KVKK) and export cleanup.
+   */
+  async removeObjects(prefix: string): Promise<void> {
+    const objects: string[] = [];
+    const stream = this.client.listObjects(this.bucket, prefix, true);
+    for await (const obj of stream) {
+      objects.push(obj.name);
+    }
+    if (objects.length > 0) {
+      await this.client.removeObjects(this.bucket, objects);
+    }
+  }
+
+  /**
+   * Check if an object exists.
+   */
+  async objectExists(key: string): Promise<boolean> {
+    try {
+      await this.client.statObject(this.bucket, key);
+      return true;
+    } catch {
+      return false;
+    }
+  }
+
+  /**
+   * Get object as Buffer.
+   * Use sparingly -- prefer signed URLs for serving to clients.
+   */
+  async getObject(key: string): Promise<Buffer> {
+    const stream = await this.client.getObject(this.bucket, key);
+    const chunks: Buffer[] = [];
+    for await (const chunk of stream) {
+      chunks.push(chunk as Buffer);
+    }
+    return Buffer.concat(chunks);
+  }
+}
+```
+
+Uses the `minio` npm package (officially `minio-js`), which is S3-compatible. If the project later migrates to AWS S3 or another S3-compatible provider, only the `MinioService` constructor changes -- all callers remain the same.
+
+---
+
+## 7. Module Structure
+
+```
+src/integrations/minio/
+  minio.module.ts     # MinioModule -- provides and exports MinioService
+  minio.service.ts    # S3-compatible file operations
+```
+
+`MinioModule` is a global integration module. It is imported by any module that needs file storage:
+
+- **WhatsApp module** -- incoming media storage (Meta webhook media), example photo retrieval
+- **Calls module** -- recording mirroring
+- **Webhooks module** -- Meta webhook media download
+- **Admin module** -- KVKK data export, CSV exports
+- **Outbound Campaigns** -- result CSV export
+- **Appointment Validation** -- result CSV export
+- **Patients module** -- patient documents and photo sets
+
+---
+
+## 8. Server-Side Encryption (SSE-S3)
+
+MinIO is configured with server-side encryption using its built-in key management:
+
+```yaml
+# docker-compose.yml (MinIO service)
+minio:
+  image: minio/minio
+  environment:
+    MINIO_ROOT_USER: ${MINIO_ACCESS_KEY}
+    MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY}
+    MINIO_KMS_KES_ENDPOINT: ""          # Use built-in encryption
+    MINIO_KMS_KES_KEY_NAME: "default"
+  command: server /data --console-address ":9001"
+  volumes:
+    - minio-data:/data
+```
+
+SSE-S3 encrypts objects at rest transparently. The application does not need to handle encryption/decryption -- MinIO manages it. Objects are encrypted on disk and decrypted automatically when accessed via the API.
+
+This satisfies KVKK requirements for data at rest encryption of patient media (medical photos, voice recordings, documents).
+
+---
+
+## 9. Cleanup Jobs
+
+### 9.1 Export Cleanup
+
+CSV exports and KVKK data packages are temporary. A daily cleanup job removes files older than 24 hours:
+
+```typescript
+// In log:cleanup processor (or dedicated export:cleanup):
+const prefix = `${tenantId}/exports/`;
+const stream = this.minioService.client.listObjects(bucket, prefix, true);
+for await (const obj of stream) {
+  const age = Date.now() - obj.lastModified.getTime();
+  if (age > 24 * 60 * 60 * 1000) {  // 24 hours
+    await this.minioService.removeObject(obj.name);
+  }
+}
+```
+
+### 9.2 Tenant Data Deletion (KVKK)
+
+When a KVKK data deletion request is executed, all MinIO objects for the patient are removed:
+
+```typescript
+// Delete all media for a specific chat:
+await this.minioService.removeObjects(`${tenantId}/whatsapp-media/${chatId}/`);
+
+// Delete all operator photos for specific requests:
+for (const requestId of requestIds) {
+  await this.minioService.removeObjects(`${tenantId}/operator-photos/forwarded/${requestId}/`);
+}
+```
+
+### 9.3 Tenant Hard Delete
+
+When a superadmin hard-deletes a tenant, all MinIO objects under the tenant prefix are removed:
+
+```typescript
+await this.minioService.removeObjects(`${tenantId}/`);
+```
+
+---
+
+## 10. Migration Notes
+
+### What Changes from Current System
+
+| Data | Current Storage | Target Storage | DB Field |
+|------|----------------|---------------|----------|
+| Operator example photo | `base64` string in MongoDB tenant doc | MinIO `{tenantId}/operator-photos/example-photo.{ext}` | `clinic_settings.operator_example_photo_key` (MinIO key) |
+| Meta webhook media | N/A (Baileys used in-memory cache) | MinIO `{tenantId}/whatsapp-media/{chatId}/{messageId}.{ext}` -- downloaded from Meta CDN on receipt | `msg_media_key` on `WhatsAppMessage` |
+| Patient documents | N/A | MinIO `{tenantId}/patients/{patientId}/documents/{documentId}.{ext}` | `document_file_key` on `PatientDocument` |
+| Patient photos | N/A | MinIO `{tenantId}/patients/{patientId}/photos/{photoSetId}/{n}.{ext}` | `photo_file_key` on `PatientPhoto` |
+| CSV exports | Generated on-the-fly, not persisted | MinIO `{tenantId}/exports/...` | Returned as signed URL |
+| Call recordings | External ElevenLabs URL | Optionally mirrored to MinIO `{tenantId}/recordings/...` | `detail_recording_key` on `CallDetail` |
+
+### Architecture Rule
+
+**No base64 in database.** This is a non-negotiable rule in the NestJS rewrite. All binary data (images, audio, video, documents, exports) goes to MinIO. The database stores only the MinIO object key (a string).
+
+### Dependencies
+
+MinIO is set up in Phase 1 (Config, Database, Redis, MinIO) and is available from the start. All subsequent phases use it as needed.