@seeka-labs/cli-apps 3.7.0 → 3.8.2

package/dist/ai-context/common/guides/data-flow.md

@@ -0,0 +1,455 @@
+# Data Flow in Seeka Apps
+
+This guide explains how data flows through Seeka apps, from receiving webhooks to sending data to external platforms.
+
+## Overview
+
+```
+┌──────────────┐    Webhook     ┌──────────────┐    Queue      ┌──────────────┐
+│              │ ────────────▶  │              │ ────────────▶ │              │
+│  Seeka Core  │                │   Webhook    │               │    Queue     │
+│              │                │   Handler    │               │   Handler    │
+└──────────────┘                └──────────────┘               └──────────────┘
+                                       │                              │
+                                       │ Sync                         │ Async
+                                       ▼                              ▼
+                                ┌──────────────┐               ┌──────────────┐
+                                │  Installation │               │   Service    │
+                                │   Storage    │               │    Layer     │
+                                └──────────────┘               └──────────────┘
+                                                                      │
+                                                                      ▼
+                                                               ┌──────────────┐
+                                                               │   External   │
+                                                               │   Platform   │
+                                                               └──────────────┘
+```
+
+## Webhook Types and Data Flow
+
+### 1. App Installation Flow
+
+```
+Seeka Core                    Your App
+    │                            │
+    │  AppInstalled webhook      │
+    │ ─────────────────────────▶ │
+    │                            │
+    │                            │ 1. Validate webhook signature
+    │                            │ 2. Validate installation settings
+    │                            │ 3. Store installation context
+    │                            │ 4. (Optional) Initialize external account
+    │                            │
+    │  200 OK / 400 Error        │
+    │ ◀───────────────────────── │
+```
+
+**Payload Structure:**
+```typescript
+interface SeekaAppInstalledWebhookPayload {
+    type: 'AppInstalled';
+    isTest: boolean;
+    context: {
+        applicationInstallId: string;
+        organisationId: string;
+        organisationBrandId: string;
+    };
+    content: {
+        grantedPermissions: string[];
+        installationSettings: Record<string, unknown>;
+    };
+}
+```
+
+**Handler Pattern:**
+```typescript
+const onInstallation = async (
+    payload: SeekaAppInstalledWebhookPayload,
+    logger: Logger
+): Promise<string | null> => {
+    // 1. Validate settings
+    const error = await validateInstallationSettings(
+        payload.content.installationSettings,
+        logger
+    );
+    if (error) return error;
+
+    // 2. Store installation
+    await createOrUpdateInstallation({
+        applicationInstallId: payload.context.applicationInstallId,
+        organisationId: payload.context.organisationId,
+        organisationBrandId: payload.context.organisationBrandId,
+        installedAt: new Date().toISOString(),
+        installationState: {
+            grantedPermissions: payload.content.grantedPermissions,
+            installationSettings: payload.content.installationSettings
+        }
+    }, logger);
+
+    return null; // Success
+};
+```
+
+### 2. Activity Accepted Flow
+
+This is the most common flow - receiving activity data from Seeka.
+
+```
+Seeka Core                    Your App                      External Platform
+    │                            │                                │
+    │  ActivityAccepted webhook  │                                │
+    │ ─────────────────────────▶ │                                │
+    │                            │                                │
+    │                            │ 1. Validate signature          │
+    │                            │ 2. Get installation            │
+    │                            │ 3. Queue for processing        │
+    │                            │                                │
+    │  204 No Content            │                                │
+    │ ◀───────────────────────── │                                │
+    │                            │                                │
+    │                            │ [Queue Handler]                │
+    │                            │ 4. Transform data              │
+    │                            │ 5. Send to platform ──────────▶│
+    │                            │                                │
+    │                            │ 6. Handle response  ◀──────────│
+```
+
+**Payload Structure:**
+```typescript
+interface SeekaActivityAcceptedWebhookPayload {
+    type: 'ActivityAccepted';
+    isTest: boolean;
+    context: {
+        applicationInstallId: string;
+        organisationId: string;
+        organisationBrandId: string;
+    };
+    content: {
+        activity: {
+            activityId: string;
+            activityName: string;
+            activityNameCustom?: string;
+            timestamp: string;
+            properties?: Record<string, unknown>;
+        };
+        person?: {
+            personId: string;
+            email?: string;
+            emailHashed?: string;
+            phone?: string;
+            phoneHashed?: string;
+        };
+        source?: {
+            loc: string;
+            ref?: string;
+        };
+    };
+}
+```
+
+**Synchronous vs Asynchronous Processing:**
+
+| Approach | When to Use | Pros | Cons |
+|----------|-------------|------|------|
+| Synchronous | Simple, fast operations | Immediate feedback | Blocks webhook response |
+| Asynchronous | External API calls, complex processing | Reliable, scalable | More complex |
+
+**Recommended: Asynchronous Pattern**
+```typescript
+// In webhook handler - queue the work
+const handleSeekaActivity = async (
+    activity: SeekaActivityAcceptedWebhookPayload,
+    installation: AppInstallContext,
+    context: InvocationContext,
+    logger: Logger
+) => {
+    await triggerBackgroundJob(queueNames.trackActivity, {
+        payload: activity.content,
+        organisationBrandId: installation.organisationBrandId,
+        organisationId: installation.organisationId,
+        applicationInstallId: installation.applicationInstallId,
+        causationId: context.invocationId,
+        correlationId: context.invocationId
+    }, logger);
+};
+
+// In queue handler - do the actual work
+const processActivity = async (item: TrackActivityQueueItem, logger: Logger) => {
+    const installation = await getInstallation(item.applicationInstallId, true, logger);
+
+    // Transform data
+    const platformEvent = transformToPlatformFormat(item.payload);
+
+    // Send to external platform
+    await externalPlatformClient.sendEvent(platformEvent);
+};
+```
+
+### 3. Identity Changed Flow
+
+Triggered when user identity data is updated.
+
+```typescript
+interface SeekaIdentityChangedWebhookPayload {
+    type: 'IdentityChanged';
+    context: {
+        applicationInstallId: string;
+        organisationId: string;
+        organisationBrandId: string;
+    };
+    content: {
+        personId: string;
+        changes: {
+            field: string;
+            oldValue?: unknown;
+            newValue?: unknown;
+        }[];
+    };
+}
+```
+
+### 4. Browser SDK Plugin Flow
+
+Returns configuration for browser-side plugins.
+
+```
+Browser                       Your App
+    │                            │
+    │  BrowserSdkPlugin request  │
+    │ ─────────────────────────▶ │
+    │                            │
+    │                            │ 1. Get installation
+    │                            │ 2. Build plugin config
+    │                            │ 3. Load plugin code
+    │                            │
+    │  Plugin config + code      │
+    │ ◀───────────────────────── │
+```
+
+**Important:** Never expose sensitive data (API keys, secrets) in browser plugin config!
+
+```typescript
+// Safe plugin config
+const pluginConfig = {
+    appId: process.env.SEEKA_APP_ID,
+    appInstallId: installation.applicationInstallId,
+    appUrl: process.env.SELF_HOST_BASEURL,
+    // Only include non-sensitive settings
+    installationSettings: {
+        trackingId: installation.installationState?.installationSettings?.trackingId
+    }
+};
+```
+
+## Data Storage
+
+### Installation Context Storage
+
+Installations are stored using the SDK's storage utilities:
+
+```typescript
+import {
+    createOrUpdateInstallation,
+    getInstallation,
+    tryGetInstallation,
+    deleteInstallation
+} from '@seeka-labs/sdk-apps-server-host';
+
+// Create or update
+await createOrUpdateInstallation(installationData, logger);
+
+// Get (throws if not found)
+const installation = await getInstallation(installationId, decrypt, logger);
+
+// Try get (returns null if not found)
+const installation = await tryGetInstallation(installationId, decrypt, logger);
+
+// Delete
+await deleteInstallation(installationId, logger);
+```
+
+### Sensitive Data Encryption
+
+Sensitive data is automatically encrypted when stored:
+
+```typescript
+// The SDK handles encryption/decryption automatically
+// Just ensure DATA_ENCRYPTION_KEY is set in environment
+
+// When storing - data is encrypted
+await createOrUpdateInstallation({
+    ...installation,
+    installationState: {
+        installationSettings: {
+            apiKey: 'sensitive-api-key'  // Will be encrypted
+        }
+    }
+}, logger);
+
+// When retrieving with decrypt=true - data is decrypted
+const installation = await getInstallation(id, true, logger);
+console.log(installation.installationState.installationSettings.apiKey);
+// Output: 'sensitive-api-key' (decrypted)
+```
+
+## Queue-Based Processing
+
+### Queue Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     Azure Storage Queue                      │
+├─────────────────────────────────────────────────────────────┤
+│                                                              │
+│  track-activity-queue                                        │
+│  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐                           │
+│  │ Msg │ │ Msg │ │ Msg │ │ Msg │ ...                        │
+│  └─────┘ └─────┘ └─────┘ └─────┘                           │
+│                                                              │
+└─────────────────────────────────────────────────────────────┘
+                          │
+                          │ Triggers
+                          ▼
+              ┌─────────────────────┐
+              │  Queue Handler      │
+              │  (Azure Function)   │
+              └─────────────────────┘
+```
+
+### Queue Message Structure
+
+```typescript
+interface TrackActivityQueueItem {
+    payload: SeekaActivityAcceptedWebhookPayload['content'];
+    organisationBrandId: string;
+    organisationId: string;
+    applicationInstallId: string;
+    causationId: string;
+    correlationId: string;
+}
+```
+
+### Triggering Background Jobs
+
+```typescript
+import { triggerBackgroundJob } from '@seeka-labs/sdk-apps-server-host';
+
+await triggerBackgroundJob(
+    queueNames.trackActivity,
+    {
+        payload: activity.content,
+        applicationInstallId: installation.applicationInstallId,
+        // ... other fields
+    },
+    logger
+);
+```
+
+## External Platform Integration
+
+### Request/Response Flow
+
+```typescript
+const sendToPlatform = async (
+    event: PlatformEvent,
+    installation: AppInstallContext,
+    logger: Logger
+): Promise<void> => {
+    const settings = installation.installationState?.installationSettings;
+
+    logger.info('Sending event to platform', {
+        eventName: event.event_name,
+        installationId: installation.applicationInstallId
+    });
+
+    const response = await fetch('https://api.platform.com/events', {
+        method: 'POST',
+        headers: {
+            'Authorization': `Bearer ${settings.apiKey}`,
+            'Content-Type': 'application/json'
+        },
+        body: JSON.stringify(event)
+    });
+
+    if (!response.ok) {
+        const errorBody = await response.text();
+        logger.error('Platform API error', {
+            status: response.status,
+            body: errorBody
+        });
+        throw new Error(`Platform API error: ${response.status}`);
+    }
+
+    logger.info('Event sent successfully');
+};
+```
+
+### Handling Platform Responses
+
+```typescript
+// Success responses
+if (response.status === 200 || response.status === 201) {
+    // Event accepted
+}
+
+// Partial success (batch)
+if (response.status === 207) {
+    const result = await response.json();
+    // Check individual item statuses
+}
+
+// Rate limiting
+if (response.status === 429) {
+    const retryAfter = response.headers.get('Retry-After');
+    // Implement backoff
+}
+
+// Authentication errors
+if (response.status === 401 || response.status === 403) {
+    // Token may be expired - refresh and retry
+}
+```
+
+## Error Handling in Data Flow
+
+### Webhook Handler Errors
+
+```typescript
+try {
+    await processWebhook(body, logger);
+    return { status: 204 };
+} catch (error) {
+    logger.error('Webhook processing failed', { error });
+
+    // Return appropriate status
+    if (error instanceof ValidationError) {
+        return { status: 400, jsonBody: { error: error.message } };
+    }
+
+    return { status: 500, jsonBody: { error: 'Internal error' } };
+}
+```
+
+### Queue Handler Errors
+
+```typescript
+// Queue handlers should throw to trigger retry
+const processQueueItem = async (item: QueueItem, logger: Logger) => {
+    try {
+        await doWork(item);
+    } catch (error) {
+        if (isRetryable(error)) {
+            throw error;  // Will be retried
+        }
+
+        // Log and swallow non-retryable errors
+        logger.error('Permanent failure', { error, item });
+    }
+};
+```
+
+## See Also
+
+- `getting-started.md` - Initial setup guide
+- `common-patterns.md` - Implementation patterns
+- `../architecture/v3-structure.md` - Template structure