@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/04-REFERENCE/architecture/architecture-06-state-management.md

@@ -1,653 +1,653 @@
-# State Management Architecture
-
-This document explains the distributed state management system in the FC Connect SDK, which prevents duplicate file processing and enables coordination across multiple workflow executions.
-
-## Overview
-
-The SDK provides a **distributed state management system** built on key-value (KV) storage that enables:
-
-- File deduplication across workflow runs
-- Job ID caching for reuse
-- Batch count tracking for job strategies
-- Timestamp-based windowing
-- Distributed locking with stale detection
-
-```mermaid
-graph TB
-    subgraph "State Management"
-        STATE_SERVICE[StateService]
-        KV_ADAPTER[KV Adapter Interface]
-    end
-
-    subgraph "Storage Implementations"
-        VERSORI_KV[VersoriKVAdapter]
-        MEMORY_KV[InMemoryKVAdapter]
-        CUSTOM_KV[Custom KV Adapter]
-    end
-
-    subgraph "Use Cases"
-        DEDUP[File Deduplication]
-        JOB_CACHE[Job ID Caching]
-        BATCH_COUNT[Batch Counting]
-        LOCK[Distributed Locking]
-    end
-
-    STATE_SERVICE --> KV_ADAPTER
-    KV_ADAPTER --> VERSORI_KV
-    KV_ADAPTER --> MEMORY_KV
-    KV_ADAPTER --> CUSTOM_KV
-
-    STATE_SERVICE --> DEDUP
-    STATE_SERVICE --> JOB_CACHE
-    STATE_SERVICE --> BATCH_COUNT
-    STATE_SERVICE --> LOCK
-
-    style STATE_SERVICE fill:#e3f2fd
-    style VERSORI_KV fill:#e8f5e9
-    style DEDUP fill:#fff3e0
-```
-
-## State Object Structure
-
-The SDK stores state in structured objects:
-
-```typescript
-// Sync state (for workflow tracking)
-interface SyncState {
-  lastProcessedTimestamp?: string;
-  lastProcessedFile?: string;
-  lastProcessedCount?: number;
-  isInitialized: boolean;
-  lastSyncResult?: 'success' | 'failure' | 'partial';
-}
-
-// Daily job information (for DAILY strategy)
-interface DailyJob {
-  jobId: string;
-  createdAt: string;
-  expiresAt: string;
-  batchCount?: number;
-}
-
-// Processed file metadata
-interface ProcessedFile {
-  fileName: string;
-  lastModified: string;
-  recordCount?: number;
-  processedAt?: string;
-}
-```
-
-## File Deduplication Flow
-
-```mermaid
-sequenceDiagram
-    participant Workflow
-    participant StateService
-    participant KVStorage
-    participant S3
-
-    Workflow->>S3: List files
-    S3-->>Workflow: [file1.csv, file2.csv, file3.csv]
-
-    loop For Each File
-        Workflow->>StateService: isFileProcessed(kv, 'file1.csv', workflowId)
-        StateService->>KVStorage: get(file key)
-        KVStorage-->>StateService: File status
-
-        StateService->>StateService: Check file key
-
-        alt File Already Processed
-            StateService-->>Workflow: true (skip file)
-        else File Not Processed
-            StateService-->>Workflow: false (process file)
-
-            Workflow->>Workflow: Process file1.csv
-
-            Workflow->>StateService: updateSyncState(kv, processedFiles, workflowId)
-            StateService->>StateService: Add to processedFiles array
-            StateService->>KVStorage: set(state key, updatedState)
-            KVStorage-->>StateService: Confirmation
-            StateService-->>Workflow: Marked as processed
-        end
-    end
-```
-
-**Example Code:**
-
-```typescript
-import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-// ✅ CORRECT: Access openKv from Versori context
-// import { openKv } from '@versori/run'; // ❌ WRONG - Not a direct export
-
-// In Versori workflow handler:
-const { openKv } = ctx;
-const kv = openKv(':project:');
-const kvAdapter = new VersoriKVAdapter(kv);
-const stateService = new StateService(logger); // logger is optional
-
-// IMPORTANT: All StateService methods require 'kv' as first parameter
-
-// Check if file processed (note: kv is first parameter)
-const processed = await stateService.isFileProcessed(
-  kvAdapter,
-  'inventory-2025-01-15.csv',
-  'my-ingestion'
-);
-
-if (!processed) {
-  // Process file
-  const result = await processFile();
-
-  // Update sync state with processed file metadata
-  await stateService.updateSyncState(
-    kvAdapter,
-    [
-      {
-        fileName: 'inventory-2025-01-15.csv',
-        lastModified: new Date().toISOString(),
-        recordCount: 1500,
-      },
-    ],
-    'my-ingestion'
-  );
-}
-```
-
-## Job Caching for DAILY Strategy
-
-The DAILY job strategy reuses the same job for all files processed in a day:
-
-```mermaid
-flowchart TD
-    START[Start Processing] --> CHECK{Daily Job<br/>Cached?}
-
-    CHECK -->|Yes| VALIDATE{Job Still<br/>Valid?}
-    CHECK -->|No| CREATE_NEW[Create New Job]
-
-    VALIDATE -->|Valid| USE_CACHED[Use Cached Job ID]
-    VALIDATE -->|Expired/Closed| CREATE_NEW
-
-    CREATE_NEW --> CACHE[Cache Job ID in State]
-    CACHE --> USE_NEW[Use New Job]
-
-    USE_CACHED --> SEND[Send Batches]
-    USE_NEW --> SEND
-
-    SEND --> DONE[Complete]
-
-    style CHECK fill:#e3f2fd
-    style CREATE_NEW fill:#fff3e0
-    style USE_CACHED fill:#e8f5e9
-```
-
-**State Flow:**
-
-```mermaid
-sequenceDiagram
-    participant Orch as Orchestrator
-    participant State as StateService
-    participant KV as KVStorage
-    participant Fluent as FluentAPI
-
-    Orch->>State: getDailyJob(KV, 'ingestion')
-    State->>KV: get(jobKey)
-    KV-->>State: { jobId, expiresAt }
-
-    alt Job Exists and Valid
-        State-->>Orch: jobId
-    else Missing or Expired
-        Orch->>Fluent: Create new job
-        Fluent-->>Orch: job-456
-        Orch->>State: setDailyJob(KV, 'ingestion', 'job-456', 24)
-        State->>KV: set(jobKey, job)
-    end
-```
-
-**Example:**
-
-```typescript
-const stateService = new StateService(logger);
-
-// Get cached daily job (kv is first parameter)
-let dailyJob = await stateService.getDailyJob(kvAdapter, 'inventory-ingestion');
-let jobId = dailyJob?.jobId;
-
-if (!jobId) {
-  // Create new job
-  const job = await fluentClient.createJob({
-    name: `inventory-sync-${new Date().toISOString().split('T')[0]}`,
-  });
-
-  // Cache for reuse (kv is first parameter)
-  await stateService.setDailyJob(kvAdapter, 'inventory-ingestion', job.id, 24);
-  jobId = job.id;
-}
-
-// Use job for all batches today
-await fluentClient.sendBatch(jobId, { entityType: 'INVENTORY', action: 'UPSERT', entities: data });
-```
-
-## Batch Counting for BATCHES_PER_JOB Strategy
-
-Track batches per job to create a new job after N batches:
-
-```mermaid
-flowchart TD
-    START[Start Batch] --> GET_COUNT[Get Current Batch Count]
-    GET_COUNT --> CHECK{Count ≥<br/>Max Batches?}
-
-    CHECK -->|Yes| CREATE[Create New Job]
-    CHECK -->|No| USE[Use Current Job]
-
-    CREATE --> RESET[Reset Batch Count = 0]
-    RESET --> INCREMENT
-
-    USE --> INCREMENT[Increment Batch Count]
-    INCREMENT --> UPDATE[Update State]
-    UPDATE --> SEND[Send Batch]
-    SEND --> DONE[Complete]
-
-    style CHECK fill:#e3f2fd
-    style CREATE fill:#fff3e0
-    style INCREMENT fill:#e8f5e9
-```
-
-**Example:**
-
-```typescript
-const MAX_BATCHES_PER_JOB = 10;
-
-// Read current state (example tracking via your own key)
-const dailyJob = await stateService.getDailyJob(kvAdapter, 'inventory-ingestion');
-const currentCount = dailyJob?.batchCount ?? 0;
-
-if (currentCount >= MAX_BATCHES_PER_JOB) {
-  const job = await fluentClient.createJob({ name: 'inventory-sync' });
-  await stateService.setDailyJob(kvAdapter, 'inventory-ingestion', job.id, 24);
-}
-
-await fluentClient.sendBatch(dailyJob?.jobId || '', {
-  entityType: 'INVENTORY',
-  action: 'UPSERT',
-  entities: data,
-});
-```
-
-## Distributed Locking
-
-Prevent concurrent workflow executions from processing the same files:
-
-```mermaid
-sequenceDiagram
-    participant W1 as Workflow 1
-    participant W2 as Workflow 2
-    participant State as StateService
-    participant KV as KVStorage
-
-    par Concurrent Execution
-        W1->>State: acquireLock('file1.csv', kv, 15)
-        W2->>State: acquireLock('file1.csv', kv, 15)
-    end
-
-    State->>KV: get('lock:file1.csv')
-    KV-->>State: null (no lock)
-
-    State->>KV: set('lock:file1.csv', { owner: 'W1', timestamp })
-    KV-->>State: Lock acquired
-
-    State-->>W1: Lock acquired (process file)
-    State-->>W2: Lock already held (skip file)
-
-    W1->>W1: Process file1.csv
-    W1->>State: releaseLock('file1.csv', kv)
-    State->>KV: delete('lock:file1.csv')
-    KV-->>State: Lock released
-```
-
-**Stale Lock Detection:**
-
-If a workflow crashes while holding a lock, the lock becomes stale:
-
-```mermaid
-flowchart TD
-    TRY[Try Acquire Lock] --> EXISTS{Lock<br/>Exists?}
-
-    EXISTS -->|No| ACQUIRE[Acquire Lock]
-    EXISTS -->|Yes| CHECK_TIME{Lock Age > 30 min?}
-
-    CHECK_TIME -->|Yes| STALE[Lock is Stale]
-    CHECK_TIME -->|No| HELD[Lock Held by Other Process]
-
-    STALE --> OVERRIDE[Override Stale Lock]
-    OVERRIDE --> ACQUIRE
-
-    ACQUIRE --> SUCCESS[Lock Acquired]
-    HELD --> FAIL[Lock Failed]
-
-    style ACQUIRE fill:#e8f5e9
-    style STALE fill:#fff3e0
-    style FAIL fill:#ffebee
-```
-
-**Example:**
-
-```typescript
-const stateService = new StateService(logger);
-
-// Acquire lock (kv is first parameter)
-const acquired = await stateService.acquireLock('file1.csv', kvAdapter, 15);
-
-if (acquired) {
-  try {
-    // Process file
-    await processFile();
-  } finally {
-    // Always release lock
-    await stateService.releaseLock('file1.csv', kvAdapter);
-  }
-}
-```
-
-## State Cleanup and Maintenance
-
-```mermaid
-flowchart TD
-    START[Maintenance Job] --> GET[Get Current State]
-    GET --> FILTER[Filter Processed Files]
-
-    FILTER --> OLD{Files Older<br/>Than 30 Days?}
-
-    OLD -->|Yes| REMOVE[Remove from Array]
-    OLD -->|No| KEEP[Keep in Array]
-
-    REMOVE --> UPDATE[Update State]
-    KEEP --> UPDATE
-
-    UPDATE --> CHECK_JOB{Daily Job<br/>Still Valid?}
-
-    CHECK_JOB -->|Expired| CLEAR_JOB[Clear dailyJobId]
-    CHECK_JOB -->|Valid| SKIP
-
-    CLEAR_JOB --> SAVE[Save State]
-    SKIP --> SAVE
-
-    SAVE --> DONE[Complete]
-
-    style REMOVE fill:#fff3e0
-    style CLEAR_JOB fill:#fff3e0
-    style SAVE fill:#e8f5e9
-```
-
-**Example Cleanup:**
-
-```typescript
-async function cleanupOldEntries() {
-  // Get sync state (kv is first parameter)
-  const state = await stateService.getSyncState(kvAdapter, 'my-workflow');
-
-  // Note: File tracking uses individual file keys, not an array
-  // This example shows conceptual cleanup - actual implementation
-  // would need to list and delete individual file keys
-
-  logger.info('Current sync state', {
-    lastProcessedFile: state.lastProcessedFile,
-    lastProcessedTimestamp: state.lastProcessedTimestamp,
-    totalRecords: state.lastProcessedCount,
-  });
-}
-```
-
-## KV Adapter Interface
-
-The SDK defines a KV adapter interface for pluggable storage:
-
-```mermaid
-classDiagram
-    class KVStorageAdapter {
-        <<interface>>
-        +get(key: string) Promise~any~
-        +set(key: string, value: any) Promise~void~
-        +delete(key: string) Promise~void~
-        +list(prefix?: string) Promise~string[]~
-    }
-
-    class VersoriKVAdapter {
-        -kv: KV
-        +get(key: string) Promise~any~
-        +set(key: string, value: any) Promise~void~
-        +delete(key: string) Promise~void~
-        +list(prefix?: string) Promise~string[]~
-    }
-
-    class InMemoryKVAdapter {
-        -storage: Map
-        +get(key: string) Promise~any~
-        +set(key: string, value: any) Promise~void~
-        +delete(key: string) Promise~void~
-        +list(prefix?: string) Promise~string[]~
-    }
-
-    class CustomKVAdapter {
-        -client: any
-        +get(key: string) Promise~any~
-        +set(key: string, value: any) Promise~void~
-        +delete(key: string) Promise~void~
-        +list(prefix?: string) Promise~string[]~
-    }
-
-    KVStorageAdapter <|.. VersoriKVAdapter
-    KVStorageAdapter <|.. InMemoryKVAdapter
-    KVStorageAdapter <|.. CustomKVAdapter
-```
-
-**Implementing Custom Adapter:**
-
-```typescript
-import { KVStorageAdapter } from '@fluentcommerce/fc-connect-sdk';
-
-class RedisKVAdapter implements KVStorageAdapter {
-  constructor(private redis: RedisClient) {}
-
-  async get(key: string): Promise<any> {
-    const value = await this.redis.get(key);
-    return value ? JSON.parse(value) : null;
-  }
-
-  async set(key: string, value: any): Promise<void> {
-    await this.redis.set(key, JSON.stringify(value));
-  }
-
-  async delete(key: string): Promise<void> {
-    await this.redis.del(key);
-  }
-
-  async list(prefix?: string): Promise<string[]> {
-    const pattern = prefix ? `${prefix}*` : '*';
-    return await this.redis.keys(pattern);
-  }
-}
-
-// Use custom adapter
-const kvAdapter = new RedisKVAdapter(redisClient);
-const stateService = new StateService(logger);
-```
-
-## State Versioning
-
-Handle state schema evolution gracefully:
-
-```mermaid
-flowchart TD
-    READ[Read State from KV] --> CHECK{Has Version?}
-
-    CHECK -->|No| V1[Assume Version 1]
-    CHECK -->|Yes| VERSION{Which<br/>Version?}
-
-    VERSION -->|v1| MIGRATE_V2[Migrate v1 → v2]
-    VERSION -->|v2| MIGRATE_V3[Migrate v2 → v3]
-    VERSION -->|v3| CURRENT[Use Current Schema]
-
-    MIGRATE_V2 --> MIGRATE_V3
-    V1 --> MIGRATE_V2
-
-    MIGRATE_V3 --> CURRENT
-    CURRENT --> USE[Use State]
-
-    USE --> WRITE[Write with Current Version]
-
-    style MIGRATE_V2 fill:#fff3e0
-    style MIGRATE_V3 fill:#fff3e0
-    style CURRENT fill:#e8f5e9
-```
-
-**Example Migration:**
-
-```typescript
-interface StateV1 {
-  processedFiles: string[]; // Just file keys
-}
-
-interface StateV2 {
-  version: 2;
-  processedFiles: ProcessedFileEntry[]; // Full metadata
-}
-
-async function migrateState(state: any): Promise<StateV2> {
-  if (!state.version) {
-    // Migrate v1 to v2
-    return {
-      version: 2,
-      processedFiles: state.processedFiles.map((key: string) => ({
-        key,
-        timestamp: new Date().toISOString(),
-      })),
-    };
-  }
-
-  return state;
-}
-```
-
-## Performance Considerations
-
-```mermaid
-graph TB
-    PERF[Performance Optimizations]
-
-    PERF --> BATCH_OPS[Batch KV Operations]
-    PERF --> CACHE[In-memory Caching]
-    PERF --> COMPRESS[Compress Large States]
-    PERF --> PRUNE[Prune Old Entries]
-
-    BATCH_OPS --> READ[Batch Reads]
-    BATCH_OPS --> WRITE[Batch Writes]
-
-    CACHE --> TTL[Time-based TTL]
-    CACHE --> INVALIDATE[Smart Invalidation]
-
-    COMPRESS --> GZIP[GZIP Compression]
-    COMPRESS --> THRESHOLD[Only if > 1MB]
-
-    PRUNE --> SCHEDULE[Scheduled Cleanup]
-    PRUNE --> LIMIT[Max 10k Entries]
-
-    style PERF fill:#e3f2fd
-    style BATCH_OPS fill:#fff3e0
-    style CACHE fill:#f3e5f5
-    style COMPRESS fill:#e8f5e9
-    style PRUNE fill:#e1f5ff
-```
-
-## Best Practices
-
-### 1. Namespace State Keys
-
-```typescript
-// Good - namespaced with logger
-const stateService = new StateService(logger); // logger is optional
-const namespace = 'inventory-ingestion';
-
-// Use workflowId parameter to scope state
-await stateService.isFileProcessed(kvAdapter, 'file.csv', namespace);
-```
-
-### 2. Handle State Corruption
-
-```typescript
-// Implement your own backup/fallback strategy using your KV implementation
-```
-
-### 3. Monitor State Size
-
-```typescript
-// Implement size checks using your KV implementation if listing is supported
-```
-
-### 4. Test with Mocks
-
-```typescript
-import { StateService } from '@fluentcommerce/fc-connect-sdk';
-
-describe('Ingestion with State', () => {
-  it('should skip processed files', async () => {
-    // Mock KV adapter for testing
-    const mockKV = {
-      get: jest.fn(),
-      set: jest.fn(),
-      delete: jest.fn(),
-      atomic: jest.fn()
-    };
-
-    const stateService = new StateService(); // no logger for tests
-
-    // Pre-populate state
-    await stateService.updateSyncState(mockKV, [{
-      fileName: 'file1.csv',
-      lastModified: new Date().toISOString(),
-    }]);
-
-    // Test deduplication (kv is first parameter)
-    const processed = await stateService.isFileProcessed(mockKV, 'file1.csv');
-    expect(processed).toBe(true);
-  });
-});
-```
-
-## Monitoring and Debugging
-
-### State Inspection
-
-```typescript
-async function inspectState() {
-  // Get sync state (kv is first parameter)
-  const state = await stateService.getSyncState(kvAdapter, 'my-workflow');
-
-  console.log('State Summary:');
-  console.log('- Last processed file:', state.lastProcessedFile);
-  console.log('- Last processed timestamp:', state.lastProcessedTimestamp);
-  console.log('- Total records:', state.lastProcessedCount);
-  console.log('- Initialized:', state.isInitialized);
-  console.log('- Last sync result:', state.lastSyncResult);
-}
-```
-
-### State Reset (Emergency)
-
-```typescript
-async function resetState() {
-  logger.warn('Resetting state - all processed files will be forgotten');
-
-  // Clear state for workflow (kv is first parameter)
-  await stateService.clearState(kvAdapter, 'my-workflow');
-
-  logger.info('State reset complete');
-}
-```
-
-## Next Steps
-
-- [Overview](./architecture-01-overview.md) - Architecture overview
-- [Client Architecture](./architecture-02-client-architecture.md) - Client layer
-- [Data Flow](./architecture-03-data-flow.md) - Data processing patterns
-- [Getting Started](../../00-START-HERE/getting-started.md) - Start building
+# State Management Architecture
+
+This document explains the distributed state management system in the FC Connect SDK, which prevents duplicate file processing and enables coordination across multiple workflow executions.
+
+## Overview
+
+The SDK provides a **distributed state management system** built on key-value (KV) storage that enables:
+
+- File deduplication across workflow runs
+- Job ID caching for reuse
+- Batch count tracking for job strategies
+- Timestamp-based windowing
+- Distributed locking with stale detection
+
+```mermaid
+graph TB
+    subgraph "State Management"
+        STATE_SERVICE[StateService]
+        KV_ADAPTER[KV Adapter Interface]
+    end
+
+    subgraph "Storage Implementations"
+        VERSORI_KV[VersoriKVAdapter]
+        MEMORY_KV[InMemoryKVAdapter]
+        CUSTOM_KV[Custom KV Adapter]
+    end
+
+    subgraph "Use Cases"
+        DEDUP[File Deduplication]
+        JOB_CACHE[Job ID Caching]
+        BATCH_COUNT[Batch Counting]
+        LOCK[Distributed Locking]
+    end
+
+    STATE_SERVICE --> KV_ADAPTER
+    KV_ADAPTER --> VERSORI_KV
+    KV_ADAPTER --> MEMORY_KV
+    KV_ADAPTER --> CUSTOM_KV
+
+    STATE_SERVICE --> DEDUP
+    STATE_SERVICE --> JOB_CACHE
+    STATE_SERVICE --> BATCH_COUNT
+    STATE_SERVICE --> LOCK
+
+    style STATE_SERVICE fill:#e3f2fd
+    style VERSORI_KV fill:#e8f5e9
+    style DEDUP fill:#fff3e0
+```
+
+## State Object Structure
+
+The SDK stores state in structured objects:
+
+```typescript
+// Sync state (for workflow tracking)
+interface SyncState {
+  lastProcessedTimestamp?: string;
+  lastProcessedFile?: string;
+  lastProcessedCount?: number;
+  isInitialized: boolean;
+  lastSyncResult?: 'success' | 'failure' | 'partial';
+}
+
+// Daily job information (for DAILY strategy)
+interface DailyJob {
+  jobId: string;
+  createdAt: string;
+  expiresAt: string;
+  batchCount?: number;
+}
+
+// Processed file metadata
+interface ProcessedFile {
+  fileName: string;
+  lastModified: string;
+  recordCount?: number;
+  processedAt?: string;
+}
+```
+
+## File Deduplication Flow
+
+```mermaid
+sequenceDiagram
+    participant Workflow
+    participant StateService
+    participant KVStorage
+    participant S3
+
+    Workflow->>S3: List files
+    S3-->>Workflow: [file1.csv, file2.csv, file3.csv]
+
+    loop For Each File
+        Workflow->>StateService: isFileProcessed(kv, 'file1.csv', workflowId)
+        StateService->>KVStorage: get(file key)
+        KVStorage-->>StateService: File status
+
+        StateService->>StateService: Check file key
+
+        alt File Already Processed
+            StateService-->>Workflow: true (skip file)
+        else File Not Processed
+            StateService-->>Workflow: false (process file)
+
+            Workflow->>Workflow: Process file1.csv
+
+            Workflow->>StateService: updateSyncState(kv, processedFiles, workflowId)
+            StateService->>StateService: Add to processedFiles array
+            StateService->>KVStorage: set(state key, updatedState)
+            KVStorage-->>StateService: Confirmation
+            StateService-->>Workflow: Marked as processed
+        end
+    end
+```
+
+**Example Code:**
+
+```typescript
+import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+// ✅ CORRECT: Access openKv from Versori context
+// import { openKv } from '@versori/run'; // ❌ WRONG - Not a direct export
+
+// In Versori workflow handler:
+const { openKv } = ctx;
+const kv = openKv(':project:');
+const kvAdapter = new VersoriKVAdapter(kv);
+const stateService = new StateService(logger); // logger is optional
+
+// IMPORTANT: All StateService methods require 'kv' as first parameter
+
+// Check if file processed (note: kv is first parameter)
+const processed = await stateService.isFileProcessed(
+  kvAdapter,
+  'inventory-2025-01-15.csv',
+  'my-ingestion'
+);
+
+if (!processed) {
+  // Process file
+  const result = await processFile();
+
+  // Update sync state with processed file metadata
+  await stateService.updateSyncState(
+    kvAdapter,
+    [
+      {
+        fileName: 'inventory-2025-01-15.csv',
+        lastModified: new Date().toISOString(),
+        recordCount: 1500,
+      },
+    ],
+    'my-ingestion'
+  );
+}
+```
+
+## Job Caching for DAILY Strategy
+
+The DAILY job strategy reuses the same job for all files processed in a day:
+
+```mermaid
+flowchart TD
+    START[Start Processing] --> CHECK{Daily Job<br/>Cached?}
+
+    CHECK -->|Yes| VALIDATE{Job Still<br/>Valid?}
+    CHECK -->|No| CREATE_NEW[Create New Job]
+
+    VALIDATE -->|Valid| USE_CACHED[Use Cached Job ID]
+    VALIDATE -->|Expired/Closed| CREATE_NEW
+
+    CREATE_NEW --> CACHE[Cache Job ID in State]
+    CACHE --> USE_NEW[Use New Job]
+
+    USE_CACHED --> SEND[Send Batches]
+    USE_NEW --> SEND
+
+    SEND --> DONE[Complete]
+
+    style CHECK fill:#e3f2fd
+    style CREATE_NEW fill:#fff3e0
+    style USE_CACHED fill:#e8f5e9
+```
+
+**State Flow:**
+
+```mermaid
+sequenceDiagram
+    participant Orch as Orchestrator
+    participant State as StateService
+    participant KV as KVStorage
+    participant Fluent as FluentAPI
+
+    Orch->>State: getDailyJob(KV, 'ingestion')
+    State->>KV: get(jobKey)
+    KV-->>State: { jobId, expiresAt }
+
+    alt Job Exists and Valid
+        State-->>Orch: jobId
+    else Missing or Expired
+        Orch->>Fluent: Create new job
+        Fluent-->>Orch: job-456
+        Orch->>State: setDailyJob(KV, 'ingestion', 'job-456', 24)
+        State->>KV: set(jobKey, job)
+    end
+```
+
+**Example:**
+
+```typescript
+const stateService = new StateService(logger);
+
+// Get cached daily job (kv is first parameter)
+let dailyJob = await stateService.getDailyJob(kvAdapter, 'inventory-ingestion');
+let jobId = dailyJob?.jobId;
+
+if (!jobId) {
+  // Create new job
+  const job = await fluentClient.createJob({
+    name: `inventory-sync-${new Date().toISOString().split('T')[0]}`,
+  });
+
+  // Cache for reuse (kv is first parameter)
+  await stateService.setDailyJob(kvAdapter, 'inventory-ingestion', job.id, 24);
+  jobId = job.id;
+}
+
+// Use job for all batches today
+await fluentClient.sendBatch(jobId, { entityType: 'INVENTORY', action: 'UPSERT', entities: data });
+```
+
+## Batch Counting for BATCHES_PER_JOB Strategy
+
+Track batches per job to create a new job after N batches:
+
+```mermaid
+flowchart TD
+    START[Start Batch] --> GET_COUNT[Get Current Batch Count]
+    GET_COUNT --> CHECK{Count ≥<br/>Max Batches?}
+
+    CHECK -->|Yes| CREATE[Create New Job]
+    CHECK -->|No| USE[Use Current Job]
+
+    CREATE --> RESET[Reset Batch Count = 0]
+    RESET --> INCREMENT
+
+    USE --> INCREMENT[Increment Batch Count]
+    INCREMENT --> UPDATE[Update State]
+    UPDATE --> SEND[Send Batch]
+    SEND --> DONE[Complete]
+
+    style CHECK fill:#e3f2fd
+    style CREATE fill:#fff3e0
+    style INCREMENT fill:#e8f5e9
+```
+
+**Example:**
+
+```typescript
+const MAX_BATCHES_PER_JOB = 10;
+
+// Read current state (example tracking via your own key)
+const dailyJob = await stateService.getDailyJob(kvAdapter, 'inventory-ingestion');
+const currentCount = dailyJob?.batchCount ?? 0;
+
+if (currentCount >= MAX_BATCHES_PER_JOB) {
+  const job = await fluentClient.createJob({ name: 'inventory-sync' });
+  await stateService.setDailyJob(kvAdapter, 'inventory-ingestion', job.id, 24);
+}
+
+await fluentClient.sendBatch(dailyJob?.jobId || '', {
+  entityType: 'INVENTORY',
+  action: 'UPSERT',
+  entities: data,
+});
+```
+
+## Distributed Locking
+
+Prevent concurrent workflow executions from processing the same files:
+
+```mermaid
+sequenceDiagram
+    participant W1 as Workflow 1
+    participant W2 as Workflow 2
+    participant State as StateService
+    participant KV as KVStorage
+
+    par Concurrent Execution
+        W1->>State: acquireLock('file1.csv', kv, 15)
+        W2->>State: acquireLock('file1.csv', kv, 15)
+    end
+
+    State->>KV: get('lock:file1.csv')
+    KV-->>State: null (no lock)
+
+    State->>KV: set('lock:file1.csv', { owner: 'W1', timestamp })
+    KV-->>State: Lock acquired
+
+    State-->>W1: Lock acquired (process file)
+    State-->>W2: Lock already held (skip file)
+
+    W1->>W1: Process file1.csv
+    W1->>State: releaseLock('file1.csv', kv)
+    State->>KV: delete('lock:file1.csv')
+    KV-->>State: Lock released
+```
+
+**Stale Lock Detection:**
+
+If a workflow crashes while holding a lock, the lock becomes stale:
+
+```mermaid
+flowchart TD
+    TRY[Try Acquire Lock] --> EXISTS{Lock<br/>Exists?}
+
+    EXISTS -->|No| ACQUIRE[Acquire Lock]
+    EXISTS -->|Yes| CHECK_TIME{Lock Age > 30 min?}
+
+    CHECK_TIME -->|Yes| STALE[Lock is Stale]
+    CHECK_TIME -->|No| HELD[Lock Held by Other Process]
+
+    STALE --> OVERRIDE[Override Stale Lock]
+    OVERRIDE --> ACQUIRE
+
+    ACQUIRE --> SUCCESS[Lock Acquired]
+    HELD --> FAIL[Lock Failed]
+
+    style ACQUIRE fill:#e8f5e9
+    style STALE fill:#fff3e0
+    style FAIL fill:#ffebee
+```
+
+**Example:**
+
+```typescript
+const stateService = new StateService(logger);
+
+// Acquire lock (kv is first parameter)
+const acquired = await stateService.acquireLock('file1.csv', kvAdapter, 15);
+
+if (acquired) {
+  try {
+    // Process file
+    await processFile();
+  } finally {
+    // Always release lock
+    await stateService.releaseLock('file1.csv', kvAdapter);
+  }
+}
+```
+
+## State Cleanup and Maintenance
+
+```mermaid
+flowchart TD
+    START[Maintenance Job] --> GET[Get Current State]
+    GET --> FILTER[Filter Processed Files]
+
+    FILTER --> OLD{Files Older<br/>Than 30 Days?}
+
+    OLD -->|Yes| REMOVE[Remove from Array]
+    OLD -->|No| KEEP[Keep in Array]
+
+    REMOVE --> UPDATE[Update State]
+    KEEP --> UPDATE
+
+    UPDATE --> CHECK_JOB{Daily Job<br/>Still Valid?}
+
+    CHECK_JOB -->|Expired| CLEAR_JOB[Clear dailyJobId]
+    CHECK_JOB -->|Valid| SKIP
+
+    CLEAR_JOB --> SAVE[Save State]
+    SKIP --> SAVE
+
+    SAVE --> DONE[Complete]
+
+    style REMOVE fill:#fff3e0
+    style CLEAR_JOB fill:#fff3e0
+    style SAVE fill:#e8f5e9
+```
+
+**Example Cleanup:**
+
+```typescript
+async function cleanupOldEntries() {
+  // Get sync state (kv is first parameter)
+  const state = await stateService.getSyncState(kvAdapter, 'my-workflow');
+
+  // Note: File tracking uses individual file keys, not an array
+  // This example shows conceptual cleanup - actual implementation
+  // would need to list and delete individual file keys
+
+  logger.info('Current sync state', {
+    lastProcessedFile: state.lastProcessedFile,
+    lastProcessedTimestamp: state.lastProcessedTimestamp,
+    totalRecords: state.lastProcessedCount,
+  });
+}
+```
+
+## KV Adapter Interface
+
+The SDK defines a KV adapter interface for pluggable storage:
+
+```mermaid
+classDiagram
+    class KVStorageAdapter {
+        <<interface>>
+        +get(key: string) Promise~any~
+        +set(key: string, value: any) Promise~void~
+        +delete(key: string) Promise~void~
+        +list(prefix?: string) Promise~string[]~
+    }
+
+    class VersoriKVAdapter {
+        -kv: KV
+        +get(key: string) Promise~any~
+        +set(key: string, value: any) Promise~void~
+        +delete(key: string) Promise~void~
+        +list(prefix?: string) Promise~string[]~
+    }
+
+    class InMemoryKVAdapter {
+        -storage: Map
+        +get(key: string) Promise~any~
+        +set(key: string, value: any) Promise~void~
+        +delete(key: string) Promise~void~
+        +list(prefix?: string) Promise~string[]~
+    }
+
+    class CustomKVAdapter {
+        -client: any
+        +get(key: string) Promise~any~
+        +set(key: string, value: any) Promise~void~
+        +delete(key: string) Promise~void~
+        +list(prefix?: string) Promise~string[]~
+    }
+
+    KVStorageAdapter <|.. VersoriKVAdapter
+    KVStorageAdapter <|.. InMemoryKVAdapter
+    KVStorageAdapter <|.. CustomKVAdapter
+```
+
+**Implementing Custom Adapter:**
+
+```typescript
+import { KVStorageAdapter } from '@fluentcommerce/fc-connect-sdk';
+
+class RedisKVAdapter implements KVStorageAdapter {
+  constructor(private redis: RedisClient) {}
+
+  async get(key: string): Promise<any> {
+    const value = await this.redis.get(key);
+    return value ? JSON.parse(value) : null;
+  }
+
+  async set(key: string, value: any): Promise<void> {
+    await this.redis.set(key, JSON.stringify(value));
+  }
+
+  async delete(key: string): Promise<void> {
+    await this.redis.del(key);
+  }
+
+  async list(prefix?: string): Promise<string[]> {
+    const pattern = prefix ? `${prefix}*` : '*';
+    return await this.redis.keys(pattern);
+  }
+}
+
+// Use custom adapter
+const kvAdapter = new RedisKVAdapter(redisClient);
+const stateService = new StateService(logger);
+```
+
+## State Versioning
+
+Handle state schema evolution gracefully:
+
+```mermaid
+flowchart TD
+    READ[Read State from KV] --> CHECK{Has Version?}
+
+    CHECK -->|No| V1[Assume Version 1]
+    CHECK -->|Yes| VERSION{Which<br/>Version?}
+
+    VERSION -->|v1| MIGRATE_V2[Migrate v1 → v2]
+    VERSION -->|v2| MIGRATE_V3[Migrate v2 → v3]
+    VERSION -->|v3| CURRENT[Use Current Schema]
+
+    MIGRATE_V2 --> MIGRATE_V3
+    V1 --> MIGRATE_V2
+
+    MIGRATE_V3 --> CURRENT
+    CURRENT --> USE[Use State]
+
+    USE --> WRITE[Write with Current Version]
+
+    style MIGRATE_V2 fill:#fff3e0
+    style MIGRATE_V3 fill:#fff3e0
+    style CURRENT fill:#e8f5e9
+```
+
+**Example Migration:**
+
+```typescript
+interface StateV1 {
+  processedFiles: string[]; // Just file keys
+}
+
+interface StateV2 {
+  version: 2;
+  processedFiles: ProcessedFileEntry[]; // Full metadata
+}
+
+async function migrateState(state: any): Promise<StateV2> {
+  if (!state.version) {
+    // Migrate v1 to v2
+    return {
+      version: 2,
+      processedFiles: state.processedFiles.map((key: string) => ({
+        key,
+        timestamp: new Date().toISOString(),
+      })),
+    };
+  }
+
+  return state;
+}
+```
+
+## Performance Considerations
+
+```mermaid
+graph TB
+    PERF[Performance Optimizations]
+
+    PERF --> BATCH_OPS[Batch KV Operations]
+    PERF --> CACHE[In-memory Caching]
+    PERF --> COMPRESS[Compress Large States]
+    PERF --> PRUNE[Prune Old Entries]
+
+    BATCH_OPS --> READ[Batch Reads]
+    BATCH_OPS --> WRITE[Batch Writes]
+
+    CACHE --> TTL[Time-based TTL]
+    CACHE --> INVALIDATE[Smart Invalidation]
+
+    COMPRESS --> GZIP[GZIP Compression]
+    COMPRESS --> THRESHOLD[Only if > 1MB]
+
+    PRUNE --> SCHEDULE[Scheduled Cleanup]
+    PRUNE --> LIMIT[Max 10k Entries]
+
+    style PERF fill:#e3f2fd
+    style BATCH_OPS fill:#fff3e0
+    style CACHE fill:#f3e5f5
+    style COMPRESS fill:#e8f5e9
+    style PRUNE fill:#e1f5ff
+```
+
+## Best Practices
+
+### 1. Namespace State Keys
+
+```typescript
+// Good - namespaced with logger
+const stateService = new StateService(logger); // logger is optional
+const namespace = 'inventory-ingestion';
+
+// Use workflowId parameter to scope state
+await stateService.isFileProcessed(kvAdapter, 'file.csv', namespace);
+```
+
+### 2. Handle State Corruption
+
+```typescript
+// Implement your own backup/fallback strategy using your KV implementation
+```
+
+### 3. Monitor State Size
+
+```typescript
+// Implement size checks using your KV implementation if listing is supported
+```
+
+### 4. Test with Mocks
+
+```typescript
+import { StateService } from '@fluentcommerce/fc-connect-sdk';
+
+describe('Ingestion with State', () => {
+  it('should skip processed files', async () => {
+    // Mock KV adapter for testing
+    const mockKV = {
+      get: jest.fn(),
+      set: jest.fn(),
+      delete: jest.fn(),
+      atomic: jest.fn()
+    };
+
+    const stateService = new StateService(); // no logger for tests
+
+    // Pre-populate state
+    await stateService.updateSyncState(mockKV, [{
+      fileName: 'file1.csv',
+      lastModified: new Date().toISOString(),
+    }]);
+
+    // Test deduplication (kv is first parameter)
+    const processed = await stateService.isFileProcessed(mockKV, 'file1.csv');
+    expect(processed).toBe(true);
+  });
+});
+```
+
+## Monitoring and Debugging
+
+### State Inspection
+
+```typescript
+async function inspectState() {
+  // Get sync state (kv is first parameter)
+  const state = await stateService.getSyncState(kvAdapter, 'my-workflow');
+
+  console.log('State Summary:');
+  console.log('- Last processed file:', state.lastProcessedFile);
+  console.log('- Last processed timestamp:', state.lastProcessedTimestamp);
+  console.log('- Total records:', state.lastProcessedCount);
+  console.log('- Initialized:', state.isInitialized);
+  console.log('- Last sync result:', state.lastSyncResult);
+}
+```
+
+### State Reset (Emergency)
+
+```typescript
+async function resetState() {
+  logger.warn('Resetting state - all processed files will be forgotten');
+
+  // Clear state for workflow (kv is first parameter)
+  await stateService.clearState(kvAdapter, 'my-workflow');
+
+  logger.info('State reset complete');
+}
+```
+
+## Next Steps
+
+- [Overview](./architecture-01-overview.md) - Architecture overview
+- [Client Architecture](./architecture-02-client-architecture.md) - Client layer
+- [Data Flow](./architecture-03-data-flow.md) - Data processing patterns
+- [Getting Started](../../00-START-HERE/getting-started.md) - Start building