@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md

@@ -1,1037 +1,1037 @@
-# Module 7: State Management
-
-[← Back to Ingestion Guide](../ingestion-readme.md)
-
-**Module 7 of 9** | **Level**: Advanced | **Time**: 20 minutes
-
----
-
-## Overview
-
-This module covers state management for ingestion workflows, focusing on duplicate prevention, file tracking, and distributed locking. Learn how to build idempotent, production-ready ingestion pipelines using the SDK's `StateService`, `JobTracker`, and `VersoriFileTracker`.
-
-## Learning Objectives
-
-By the end of this module, you will:
-
-- ✅ Understand why state management is critical for ingestion workflows
-- ✅ Prevent duplicate file processing with file tracking
-- ✅ Use distributed locking for concurrent safety
-- ✅ Implement idempotent operations that can be safely retried
-- ✅ Store and retrieve sync state for incremental processing
-- ✅ Handle state storage in Versori and standalone environments
-
----
-
-## Why State Management is Critical
-
-### The Duplicate Processing Problem
-
-**Without state management:**
-
-```typescript
-// ❌ WRONG - Reprocessing files creates duplicates
-async function ingestFromS3Naive() {
-  const files = await s3.listFiles({ prefix: 'data/' });
-
-  for (const file of files) {
-    const data = await s3.downloadFile(file.path);
-    const records = await parser.parse(data);
-
-    // Problem: Every run processes ALL files again
-    await sendToBatchAPI(records);
-  }
-}
-
-// Results:
-// - Run 1: Processes 100 files ✅
-// - Run 2: Processes same 100 files again ❌ DUPLICATES
-// - Run 3: Processes same 100 files again ❌ MORE DUPLICATES
-```
-
-**With state management:**
-
-```typescript
-// ✅ CORRECT - Only process new files
-async function ingestFromS3WithState(state: StateService, kv: KVStore) {
-  const files = await s3.listFiles({ prefix: 'data/' });
-
-  for (const file of files) {
-    // Check if already processed
-    if (await state.isFileProcessed(kv, file.path)) {
-      console.log(`Skipping processed file: ${file.path}`);
-      continue;
-    }
-
-    const data = await s3.downloadFile(file.path);
-    const records = await parser.parse(data);
-
-    await sendToBatchAPI(records);
-
-    // Mark as processed
-    await state.updateSyncState(kv, [
-      {
-        fileName: file.path,
-        lastModified: file.lastModified,
-        recordCount: records.length,
-      },
-    ]);
-  }
-}
-
-// Results:
-// - Run 1: Processes 100 new files ✅
-// - Run 2: Skips 100 processed files, processes 10 new files ✅
-// - Run 3: Skips 110 processed files, processes 5 new files ✅
-```
-
-### Business Impact
-
-| Scenario                 | Without State               | With State                          |
-| ------------------------ | --------------------------- | ----------------------------------- |
-| **Daily sync rerun**     | Duplicates entire inventory | Processes only new files            |
-| **Retry failed batch**   | Reprocesses all batches     | Retries only failed batch           |
-| **Concurrent execution** | Race conditions, duplicates | Distributed locks prevent conflicts |
-| **Incremental updates**  | Must track externally       | Automatic high-watermark tracking   |
-
----
-
-## StateService Overview
-
-The `StateService` provides comprehensive state management with three core capabilities:
-
-### 1. File Processing Tracking
-
-Track which files have been processed to prevent duplicates.
-
-### 2. Distributed Locking
-
-Ensure only one process handles a file at a time (concurrent safety).
-
-### 3. Sync State Management
-
-Store high-watermark timestamps for incremental processing.
-
-### Architecture
-
-```mermaid
-graph TD
-    A[Ingestion Workflow] --> B{Check State}
-    B --> C[StateService]
-    C --> D[KV Store]
-
-    B -->|File Processed| E[Skip File]
-    B -->|New File| F[Acquire Lock]
-
-    F -->|Lock Acquired| G[Process File]
-    F -->|Lock Held| H[Wait/Skip]
-
-    G --> I[Send to Batch API]
-    I --> J[Mark as Processed]
-    J --> K[Release Lock]
-```
-
----
-
-## File Tracking for Duplicate Prevention
-
-### File Tracking with StateService (Recommended)
-
-**Use `StateService`** for production workflows with sync state and metadata tracking:
-
-```typescript
-import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-// ✅ CORRECT: Access openKv from Versori context
-// import { openKv } from '@versori/run'; // ❌ WRONG - Not a direct export
-
-async function ingestWithFileTracking(
-  files: string[],
-  ctx: any
-): Promise<void> {
-  const { log, openKv } = ctx;
-  const logger = toStructuredLogger(log, {
-    service: 'inventory-ingestion'
-  });
-
-  const stateService = new StateService(logger);
-  const kv = new VersoriKVAdapter(openKv(':project:'));
-  let processedCount = 0;
-  let skippedCount = 0;
-
-  for (const fileName of files) {
-    // Check if file already processed
-    if (await state.isFileProcessed(kv, fileName)) {
-      log.info(`Skipping processed file: ${fileName}`);
-      skippedCount++;
-      continue;
-    }
-
-    // Process file
-    log.info(`Processing new file: ${fileName}`);
-    const records = await processFile(fileName);
-    await sendToBatchAPI(records);
-
-    // Mark as processed with metadata
-    await state.updateSyncState(kv, [
-      {
-        fileName,
-        lastModified: new Date().toISOString(),
-        recordCount: records.length,
-      },
-    ]);
-
-    processedCount++;
-  }
-
-  log.info(`Processed: ${processedCount}, Skipped: ${skippedCount}`);
-}
-```
-
-### Tracking with Metadata
-
-Store rich metadata for auditing and troubleshooting:
-
-```typescript
-interface FileProcessingMetadata {
-  fileName: string;
-  lastModified: string;
-  recordCount: number;
-  processedAt: string;
-  jobId?: string;
-  batchIds?: string[];
-  fileSize?: number;
-  checksumMD5?: string;
-  source?: string;
-}
-
-async function trackFileWithMetadata(
-  stateService: StateService,
-  kv: KVStore,
-  fileName: string,
-  metadata: FileProcessingMetadata
-): Promise<void> {
-  await stateService.updateSyncState(kv, [metadata]);
-
-  console.log('File tracked:', {
-    file: fileName,
-    records: metadata.recordCount,
-    jobId: metadata.jobId,
-  });
-}
-
-// Usage
-await trackFileWithMetadata(stateService, kv, 'inventory-2025-01-19.csv', {
-  fileName: 'inventory-2025-01-19.csv',
-  lastModified: '2025-01-19T08:00:00Z',
-  recordCount: 5000,
-  processedAt: new Date().toISOString(),
-  jobId: 'JOB-12345',
-  batchIds: ['BATCH-001', 'BATCH-002'],
-  fileSize: 1024000,
-  checksumMD5: 'a1b2c3d4e5f6',
-  source: 'S3',
-});
-```
-
----
-
-## Distributed Locking for Concurrent Safety
-
-### The Concurrency Problem
-
-**Without locking:**
-
-```typescript
-// ❌ WRONG - Race condition when multiple processes run
-// Process A: Checks file not processed → Starts processing
-// Process B: Checks file not processed → Starts processing
-// Result: File processed twice simultaneously
-```
-
-**With locking:**
-
-```typescript
-// ✅ CORRECT - Only one process acquires lock
-// Process A: Acquires lock → Processes file → Releases lock
-// Process B: Lock held → Waits or skips → Tries later
-// Result: File processed exactly once
-```
-
-### Acquire and Release Lock Pattern
-
-```typescript
-import { StateService, KVStore } from '@fluentcommerce/fc-connect-sdk';
-
-async function processFileWithLock(
-  fileName: string,
-  state: StateService,
-  kv: KVStore
-): Promise<void> {
-  const lockName = `file:${fileName}`;
-  const lockTimeoutMinutes = 15;
-
-  try {
-    // Attempt to acquire lock
-    const lockAcquired = await state.acquireLock(lockName, kv, lockTimeoutMinutes);
-
-    if (!lockAcquired) {
-      console.log(`Lock held by another process, skipping: ${fileName}`);
-      return;
-    }
-
-    console.log(`Lock acquired for: ${fileName}`);
-
-    // Check if already processed (double-check after acquiring lock)
-    if (await state.isFileProcessed(kv, fileName)) {
-      console.log(`File already processed: ${fileName}`);
-      return;
-    }
-
-    // Process file
-    const records = await processFile(fileName);
-    await sendToBatchAPI(records);
-
-    // Mark as processed
-    await state.updateSyncState(kv, [
-      {
-        fileName,
-        lastModified: new Date().toISOString(),
-        recordCount: records.length,
-      },
-    ]);
-
-    console.log(`Successfully processed: ${fileName}`);
-  } finally {
-    // ALWAYS release lock in finally block
-    await state.releaseLock(lockName, kv);
-    console.log(`Lock released for: ${fileName}`);
-  }
-}
-```
-
-### Lock Timeout and Stale Lock Detection
-
-The SDK automatically handles stale locks:
-
-```typescript
-// Scenario: Process crashes while holding lock
-// 1. Process A acquires lock with 15-minute timeout
-// 2. Process A crashes at minute 10 (lock still held)
-// 3. Minute 16: Lock expires automatically
-// 4. Process B acquires stale lock (overrides expired lock)
-
-async function processWithStaleLockHandling(
-  fileName: string,
-  state: StateService,
-  kv: KVStore
-): Promise<void> {
-  const lockTimeoutMinutes = 15;
-
-  // SDK automatically detects and overrides stale locks
-  const lockAcquired = await state.acquireLock(`file:${fileName}`, kv, lockTimeoutMinutes);
-
-  if (!lockAcquired) {
-    // Lock is held by active process (not stale)
-    console.log('Lock actively held by another process');
-    return;
-  }
-
-  // Lock acquired (either fresh or stale was overridden)
-  try {
-    await processFile(fileName);
-  } finally {
-    await state.releaseLock(`file:${fileName}`, kv);
-  }
-}
-```
-
----
-
-## Idempotent Processing
-
-**Idempotency** means operations can be safely repeated without changing the result beyond the initial application.
-
-### Idempotent Ingestion Pattern
-
-```typescript
-async function idempotentIngestion(
-  fileName: string,
-  state: StateService,
-  kv: KVStore
-): Promise<{ success: boolean; message: string }> {
-  const lockName = `ingestion:${fileName}`;
-
-  try {
-    // 1. Acquire lock (prevents concurrent processing)
-    const lockAcquired = await state.acquireLock(lockName, kv, 15);
-    if (!lockAcquired) {
-      return {
-        success: true,
-        message: 'File locked by another process, will retry later',
-      };
-    }
-
-    // 2. Check if already processed (prevents duplicates)
-    if (await state.isFileProcessed(kv, fileName)) {
-      return {
-        success: true,
-        message: 'File already processed, skipping',
-      };
-    }
-
-    // 3. Process file
-    const records = await processFile(fileName);
-
-    // 4. Send to Batch API (UPSERT is idempotent)
-    const job = await client.createJob({
-      name: `Import ${fileName}`,
-      retailerId: process.env.FLUENT_RETAILER_ID!,
-    });
-
-    await client.sendBatch(job.id, {
-      action: 'UPSERT', // ✅ UPSERT is idempotent
-      entityType: 'INVENTORY',
-      source: 'STATE_MANAGEMENT_EXAMPLE',
-      event: 'INVENTORY_UPDATE',
-      entities: records,
-    });
-
-    // 5. Mark as processed (atomic operation)
-    await state.updateSyncState(kv, [
-      {
-        fileName,
-        lastModified: new Date().toISOString(),
-        recordCount: records.length,
-        processedAt: new Date().toISOString(),
-      },
-    ]);
-
-    return {
-      success: true,
-      message: `Processed ${records.length} records`,
-    };
-  } catch (error) {
-    // DO NOT mark as processed on error
-    console.error(`Processing failed for ${fileName}:`, error);
-    return {
-      success: false,
-      message: (error as Error).message,
-    };
-  } finally {
-    // Always release lock
-    await state.releaseLock(lockName, kv);
-  }
-}
-
-// Safe to retry - will skip if already completed
-await idempotentIngestion('inventory.csv', state, kv);
-await idempotentIngestion('inventory.csv', state, kv); // ✅ Skips, no duplicates
-await idempotentIngestion('inventory.csv', state, kv); // ✅ Skips, no duplicates
-```
-
-### Why This is Idempotent
-
-| Step                 | Idempotency Guarantee                          |
-| -------------------- | ---------------------------------------------- |
-| **Lock acquisition** | If locked, skip (safe to retry)                |
-| **Processed check**  | If processed, skip (safe to retry)             |
-| **Batch API UPSERT** | UPSERT is idempotent (same data = same result) |
-| **State update**     | Atomic operation (safe to retry)               |
-
----
-
-## Sync State for Incremental Processing
-
-Track the last processed timestamp to enable incremental updates:
-
-### Get and Update Sync State
-
-```typescript
-import { StateService, SyncState } from '@fluentcommerce/fc-connect-sdk';
-
-async function incrementalSync(
-  state: StateService,
-  kv: KVStore,
-  workflowId: string
-): Promise<void> {
-  // Get last sync state
-  const syncState: SyncState = await state.getSyncState(kv, workflowId);
-
-  if (syncState.isInitialized) {
-    console.log('Last sync:', syncState.lastProcessedTimestamp);
-    console.log('Last file:', syncState.lastProcessedFile);
-  } else {
-    console.log('First sync (no prior state)');
-  }
-
-  // Get files modified after last sync
-  const newFiles = await getFilesModifiedAfter(syncState.lastProcessedTimestamp);
-
-  if (newFiles.length === 0) {
-    console.log('No new files to process');
-    return;
-  }
-
-  const processedFiles = [];
-
-  for (const file of newFiles) {
-    const records = await processFile(file.name);
-    await sendToBatchAPI(records);
-
-    processedFiles.push({
-      fileName: file.name,
-      lastModified: file.lastModified,
-      recordCount: records.length,
-    });
-  }
-
-  // Update sync state with all processed files
-  await state.updateSyncState(kv, processedFiles, workflowId);
-
-  console.log(`Synced ${processedFiles.length} new files`);
-}
-```
-
-### Daily Job Management with State
-
-Reuse a single job for all batches processed in a day:
-
-```typescript
-import { StateService, DailyJob } from '@fluentcommerce/fc-connect-sdk';
-
-async function processWithDailyJob(
-  files: string[],
-  state: StateService,
-  kv: KVStore,
-  client: FluentClient
-): Promise<void> {
-  const workflowId = 'inventory-ingestion';
-
-  // Check for existing daily job
-  let dailyJob: DailyJob | null = await state.getDailyJob(kv, workflowId);
-
-  if (!dailyJob || new Date(dailyJob.expiresAt) < new Date()) {
-    // Create new daily job
-    const job = await client.createJob({
-      name: `Daily Inventory Sync - ${new Date().toISOString().split('T')[0]}`,
-      retailerId: process.env.FLUENT_RETAILER_ID!,
-    });
-
-    // Store daily job (expires in 24 hours)
-    await state.setDailyJob(kv, workflowId, job.id, 24);
-
-    dailyJob = {
-      jobId: job.id,
-      createdAt: new Date().toISOString(),
-      expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString(),
-    };
-
-    console.log(`Created daily job: ${job.id}`);
-  } else {
-    console.log(`Reusing daily job: ${dailyJob.jobId}`);
-  }
-
-  // Process all files with same job
-  for (const fileName of files) {
-    if (await state.isFileProcessed(kv, fileName, workflowId)) {
-      continue;
-    }
-
-    const records = await processFile(fileName);
-
-    await client.sendBatch(dailyJob.jobId, {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      source: 'DAILY_JOB_REUSE',
-      event: 'INVENTORY_UPDATE',
-      entities: records,
-    });
-
-    await state.updateSyncState(
-      kv,
-      [
-        {
-          fileName,
-          lastModified: new Date().toISOString(),
-          recordCount: records.length,
-        },
-      ],
-      workflowId
-    );
-  }
-}
-```
-
----
-
-## State Storage Adapters
-
-### Versori Platform (Built-in KV)
-
-```typescript
-import { StateService } from '@fluentcommerce/fc-connect-sdk';
-import { workflow } from '@versori/run';
-
-export const inventoryIngestion = workflow('inventory-ingestion', async ctx => {
-  const { log, openKv } = ctx;
-
-  // Open Versori KV store
-  const kvAdapter = new VersoriKVAdapter(openKv(':project:'));
-
-  // Create logger and state service
-  const logger = toStructuredLogger(log, {
-    service: 'inventory-ingestion'
-  });
-  const stateService = new StateService(logger);
-
-  // Use state methods with kvAdapter parameter
-  const isProcessed = await stateService.isFileProcessed(kvAdapter, 'inventory.csv');
-
-  if (!isProcessed) {
-    // Process file
-    await processFile('inventory.csv');
-
-    // Mark as processed
-    await stateService.updateSyncState(kvAdapter, [
-      {
-        fileName: 'inventory.csv',
-        lastModified: new Date().toISOString(),
-        recordCount: 1000,
-      },
-    ]);
-  }
-});
-```
-
-### Node.js Standalone (Custom KV)
-
-```typescript
-import { StateService, KVStore } from '@fluentcommerce/fc-connect-sdk';
-
-// Implement KVStore interface with Redis, DynamoDB, etc.
-class RedisKVStore implements KVStore {
-  constructor(private redis: RedisClient) {}
-
-  async get(keys: string[]): Promise<{ value: unknown } | null> {
-    const key = keys.join(':');
-    const value = await this.redis.get(key);
-    return value ? { value: JSON.parse(value) } : null;
-  }
-
-  async set(keys: string[], value: unknown): Promise<void> {
-    const key = keys.join(':');
-    await this.redis.set(key, JSON.stringify(value));
-  }
-
-  async delete(keys: string[]): Promise<void> {
-    const key = keys.join(':');
-    await this.redis.del(key);
-  }
-
-  atomic() {
-    // Implement atomic operations with Redis transactions
-    const operations: Array<{ type: string; key: string[]; value?: unknown }> = [];
-
-    return {
-      check: (entries: Array<{ key: string[]; versionstamp: string | null }>) => {
-        // Redis WATCH for optimistic locking
-        entries.forEach(e => this.redis.watch(e.key.join(':')));
-      },
-      set: (key: string[], value: unknown) => {
-        operations.push({ type: 'set', key, value });
-      },
-      commit: async (): Promise<boolean> => {
-        const multi = this.redis.multi();
-        operations.forEach(op => {
-          if (op.type === 'set') {
-            multi.set(op.key.join(':'), JSON.stringify(op.value));
-          }
-        });
-        const results = await multi.exec();
-        return results !== null;
-      },
-    };
-  }
-}
-
-// Usage
-const logger = toStructuredLogger(createConsoleLogger(), {
-  service: 'inventory-ingestion',
-  correlationId: generateCorrelationId()
-});
-
-const redisKV = new RedisKVStore(redisClient);
-const stateService = new StateService(logger);
-
-await stateService.isFileProcessed(redisKV, 'inventory.csv');
-```
-
----
-
-## Complete Production Example
-
-Full ingestion workflow with state management, locking, and error handling:
-
-```typescript
-import {
-  createClient,
-  FluentClient,
-  StateService,
-  KVStore,
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  BatchAction,
-  EntityType,
-} from '@fluentcommerce/fc-connect-sdk';
-
-interface IngestionConfig {
-  s3: {
-    bucket: string;
-    prefix: string;
-  };
-  fluent: {
-    baseUrl: string;
-    clientId: string;
-    clientSecret: string;
-    retailerId: string;
-  };
-  workflow: {
-    id: string;
-    lockTimeoutMinutes: number;
-  };
-}
-
-async function productionIngestionWithState(
-  config: IngestionConfig,
-  kv: KVStore
-): Promise<{
-  success: boolean;
-  filesProcessed: number;
-  filesSkipped: number;
-  errors: string[];
-}> {
-  const logger = toStructuredLogger(createConsoleLogger(), {
-    service: 'inventory-ingestion',
-    correlationId: generateCorrelationId()
-  });
-
-  const stateService = new StateService(logger);
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_CSV',
-      connectionId: 's3-state-example',
-      name: 'S3 State Example',
-      s3Config: {
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-        region: process.env.AWS_REGION!,
-        bucket: 'inventory-bucket',
-      },
-    },
-    logger
-  );
-
-  const client = await createClient({
-    config: {
-      baseUrl: config.fluent.baseUrl,
-      clientId: config.fluent.clientId,
-      clientSecret: config.fluent.clientSecret,
-      retailerId: config.fluent.retailerId,
-    }
-  });
-
-  const parser = new CSVParserService();
-  const mapper = new UniversalMapper({
-    fields: {
-      ref: {
-        source: 'sku',
-        required: true,
-        comment: '✅ InventoryPositionInput.ref (String! required)',
-      },
-      productRef: {
-        source: 'sku',
-        required: true,
-        comment: '✅ InventoryPositionInput.productRef (String! required)',
-      },
-      locationRef: {
-        source: 'warehouse',
-        required: true,
-        comment: '✅ InventoryPositionInput.locationRef (String! required)',
-      },
-      qty: {
-        source: 'quantity',
-        resolver: 'sdk.parseInt',
-        required: true,
-        comment: '✅ InventoryPositionInput.qty (Int! required)',
-      },
-      status: {
-        source: 'status',
-        resolver: 'sdk.uppercase',
-        defaultValue: 'AVAILABLE',
-        comment: '✅ InventoryPositionInput.status (String optional)',
-      },
-    },
-  });
-
-  let filesProcessed = 0;
-  let filesSkipped = 0;
-  const errors: string[] = [];
-
-  try {
-    // List files from S3
-    const files = await s3.listFiles({ prefix: config.s3.prefix });
-    console.log(`Found ${files.length} files in S3`);
-
-    // Get or create daily job
-    let dailyJob = await state.getDailyJob(kv, config.workflow.id);
-
-    if (!dailyJob) {
-      const job = await client.createJob({
-        name: `Daily Sync - ${new Date().toISOString().split('T')[0]}`,
-        retailerId: config.fluent.retailerId,
-      });
-
-      await state.setDailyJob(kv, config.workflow.id, job.id, 24);
-      dailyJob = {
-        jobId: job.id,
-        createdAt: new Date().toISOString(),
-        expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString(),
-      };
-    }
-
-    console.log(`Using job: ${dailyJob.jobId}`);
-
-    for (const file of files) {
-      const lockName = `file:${file.path}`;
-
-      try {
-        // 1. Acquire lock
-        const lockAcquired = await state.acquireLock(
-          lockName,
-          kv,
-          config.workflow.lockTimeoutMinutes
-        );
-
-        if (!lockAcquired) {
-          console.log(`File locked by another process: ${file.path}`);
-          filesSkipped++;
-          continue;
-        }
-
-        // 2. Check if already processed
-        if (await state.isFileProcessed(kv, file.path, config.workflow.id)) {
-          console.log(`File already processed: ${file.path}`);
-          filesSkipped++;
-          await state.releaseLock(lockName, kv);
-          continue;
-        }
-
-        // 3. Download and parse file
-        const fileContent = await s3.downloadFile(file.path);
-        const records = await parser.parse(fileContent);
-
-        console.log(`Parsed ${records.length} records from ${file.path}`);
-
-        // 4. Transform data
-        const mappingResult = await mapper.map(records);
-
-        if (!mappingResult.success) {
-          throw new Error(`Mapping failed: ${JSON.stringify(mappingResult.errors)}`);
-        }
-
-        // 5. Send to Batch API
-        const batch = await client.sendBatch(dailyJob.jobId, {
-          action: 'UPSERT',
-          entityType: 'INVENTORY',
-          source: 'COMPLETE_WORKFLOW',
-          event: 'INVENTORY_UPDATE',
-          entities: mappingResult.data,
-        });
-
-        console.log(`Batch sent: ${batch.id}`);
-
-        // 6. Mark as processed
-        await state.updateSyncState(
-          kv,
-          [
-            {
-              fileName: file.path,
-              lastModified: file.lastModified || new Date().toISOString(),
-              recordCount: mappingResult.data.length,
-              processedAt: new Date().toISOString(),
-            },
-          ],
-          config.workflow.id
-        );
-
-        filesProcessed++;
-
-        // 7. Release lock
-        await state.releaseLock(lockName, kv);
-      } catch (error) {
-        const errorMsg = `Failed to process ${file.path}: ${(error as Error).message}`;
-        console.error(errorMsg);
-        errors.push(errorMsg);
-
-        // Release lock on error
-        await state.releaseLock(lockName, kv);
-      }
-    }
-
-    return {
-      success: errors.length === 0,
-      filesProcessed,
-      filesSkipped,
-      errors,
-    };
-  } catch (error) {
-    console.error('Ingestion failed:', error);
-    throw error;
-  }
-}
-
-// Usage
-const result = await productionIngestionWithState(
-  {
-    s3: {
-      bucket: 'inventory-bucket',
-      prefix: 'data/',
-    },
-    fluent: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!,
-    },
-    workflow: {
-      id: 'inventory-ingestion',
-      lockTimeoutMinutes: 15,
-    },
-  },
-  kv
-);
-
-console.log('Ingestion result:', result);
-// Output:
-// {
-//   success: true,
-//   filesProcessed: 10,
-//   filesSkipped: 5,
-//   errors: []
-// }
-```
-
----
-
-## Troubleshooting
-
-### Problem: Files Reprocessed Every Run
-
-**Symptoms:**
-
-- Same files processed repeatedly
-- Duplicate inventory positions
-
-**Solution:**
-
-```typescript
-// Check if state is being updated correctly
-const syncState = await state.getSyncState(kv, workflowId);
-console.log('Sync state:', syncState);
-
-// Verify file is marked as processed
-const isProcessed = await state.isFileProcessed(kv, fileName, workflowId);
-console.log(`File ${fileName} processed:`, isProcessed);
-
-// Ensure updateSyncState is called AFTER successful processing
-await state.updateSyncState(kv, [{ fileName, lastModified, recordCount }], workflowId);
-```
-
-### Problem: Lock Never Released
-
-**Symptoms:**
-
-- Files stuck in "locked" state
-- Processing stops
-
-**Solution:**
-
-```typescript
-// ALWAYS use try-finally to release locks
-try {
-  const lockAcquired = await state.acquireLock(lockName, kv, 15);
-  if (lockAcquired) {
-    await processFile();
-  }
-} finally {
-  // ✅ CRITICAL: Release lock even if processing fails
-  await state.releaseLock(lockName, kv);
-}
-```
-
-### Problem: Stale Locks Blocking Processing
-
-**Symptoms:**
-
-- Locks from crashed processes
-- Processing stuck indefinitely
-
-**Solution:**
-
-```typescript
-// SDK automatically overrides stale locks based on timeout
-// If lock expired (past expiresAt), next acquireLock succeeds
-
-// To manually clear stale locks (if needed):
-await state.releaseLock(lockName, kv); // Force release
-```
-
-### Problem: State Lost Between Runs
-
-**Symptoms:**
-
-- First-time sync every run
-- `isInitialized: false` always
-
-**Solution:**
-
-```typescript
-// Verify KV store persistence
-const testKey = ['test', 'persistence'];
-await kv.set(testKey, { value: 'test' });
-const retrieved = await kv.get(testKey);
-console.log('KV persistence test:', retrieved);
-
-// Ensure workflowId is consistent
-const WORKFLOW_ID = 'inventory-ingestion'; // ✅ Use constant
-await state.updateSyncState(kv, files, WORKFLOW_ID);
-```
-
----
-
-## Key Takeaways
-
-- 🎯 **Always use state management** - Prevents duplicates and enables idempotency
-- 🎯 **Acquire locks before processing** - Prevents concurrent conflicts
-- 🎯 **Release locks in finally blocks** - Ensures cleanup even on errors
-- 🎯 **Track file metadata** - Rich metadata enables auditing and troubleshooting
-- 🎯 **Use workflow IDs** - Scope state to specific workflows
-- 🎯 **Handle stale locks** - SDK automatically overrides expired locks
-
----
-
-## Next Steps
-
-Continue to [Module 8: Performance Optimization](./02-core-guides-ingestion-08-performance-optimization.md) to learn how to optimize batch sizing, parallel processing, and job strategies for large-scale ingestion.
-
----
-
-[← Previous: Batch API](./02-core-guides-ingestion-06-batch-api.md) | [Back to Guide](../ingestion-readme.md) | [Next: Performance Optimization →](./02-core-guides-ingestion-08-performance-optimization.md)
-
-## Related Documentation
-
-- [StateService API](../../api-reference/modules/api-reference-05-services.md#state-service) - Complete API documentation
-- [Versori KV Integration](../../../04-REFERENCE/platforms/versori/#key-value-storage) - Platform-specific KV usage
-- [Error Handling Guide](../../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md) - Retry patterns and error recovery
-- [Best Practices](./02-core-guides-ingestion-09-best-practices.md) - Production patterns and monitoring
+# Module 7: State Management
+
+[← Back to Ingestion Guide](../ingestion-readme.md)
+
+**Module 7 of 9** | **Level**: Advanced | **Time**: 20 minutes
+
+---
+
+## Overview
+
+This module covers state management for ingestion workflows, focusing on duplicate prevention, file tracking, and distributed locking. Learn how to build idempotent, production-ready ingestion pipelines using the SDK's `StateService`, `JobTracker`, and `VersoriFileTracker`.
+
+## Learning Objectives
+
+By the end of this module, you will:
+
+- ✅ Understand why state management is critical for ingestion workflows
+- ✅ Prevent duplicate file processing with file tracking
+- ✅ Use distributed locking for concurrent safety
+- ✅ Implement idempotent operations that can be safely retried
+- ✅ Store and retrieve sync state for incremental processing
+- ✅ Handle state storage in Versori and standalone environments
+
+---
+
+## Why State Management is Critical
+
+### The Duplicate Processing Problem
+
+**Without state management:**
+
+```typescript
+// ❌ WRONG - Reprocessing files creates duplicates
+async function ingestFromS3Naive() {
+  const files = await s3.listFiles({ prefix: 'data/' });
+
+  for (const file of files) {
+    const data = await s3.downloadFile(file.path);
+    const records = await parser.parse(data);
+
+    // Problem: Every run processes ALL files again
+    await sendToBatchAPI(records);
+  }
+}
+
+// Results:
+// - Run 1: Processes 100 files ✅
+// - Run 2: Processes same 100 files again ❌ DUPLICATES
+// - Run 3: Processes same 100 files again ❌ MORE DUPLICATES
+```
+
+**With state management:**
+
+```typescript
+// ✅ CORRECT - Only process new files
+async function ingestFromS3WithState(state: StateService, kv: KVStore) {
+  const files = await s3.listFiles({ prefix: 'data/' });
+
+  for (const file of files) {
+    // Check if already processed
+    if (await state.isFileProcessed(kv, file.path)) {
+      console.log(`Skipping processed file: ${file.path}`);
+      continue;
+    }
+
+    const data = await s3.downloadFile(file.path);
+    const records = await parser.parse(data);
+
+    await sendToBatchAPI(records);
+
+    // Mark as processed
+    await state.updateSyncState(kv, [
+      {
+        fileName: file.path,
+        lastModified: file.lastModified,
+        recordCount: records.length,
+      },
+    ]);
+  }
+}
+
+// Results:
+// - Run 1: Processes 100 new files ✅
+// - Run 2: Skips 100 processed files, processes 10 new files ✅
+// - Run 3: Skips 110 processed files, processes 5 new files ✅
+```
+
+### Business Impact
+
+| Scenario                 | Without State               | With State                          |
+| ------------------------ | --------------------------- | ----------------------------------- |
+| **Daily sync rerun**     | Duplicates entire inventory | Processes only new files            |
+| **Retry failed batch**   | Reprocesses all batches     | Retries only failed batch           |
+| **Concurrent execution** | Race conditions, duplicates | Distributed locks prevent conflicts |
+| **Incremental updates**  | Must track externally       | Automatic high-watermark tracking   |
+
+---
+
+## StateService Overview
+
+The `StateService` provides comprehensive state management with three core capabilities:
+
+### 1. File Processing Tracking
+
+Track which files have been processed to prevent duplicates.
+
+### 2. Distributed Locking
+
+Ensure only one process handles a file at a time (concurrent safety).
+
+### 3. Sync State Management
+
+Store high-watermark timestamps for incremental processing.
+
+### Architecture
+
+```mermaid
+graph TD
+    A[Ingestion Workflow] --> B{Check State}
+    B --> C[StateService]
+    C --> D[KV Store]
+
+    B -->|File Processed| E[Skip File]
+    B -->|New File| F[Acquire Lock]
+
+    F -->|Lock Acquired| G[Process File]
+    F -->|Lock Held| H[Wait/Skip]
+
+    G --> I[Send to Batch API]
+    I --> J[Mark as Processed]
+    J --> K[Release Lock]
+```
+
+---
+
+## File Tracking for Duplicate Prevention
+
+### File Tracking with StateService (Recommended)
+
+**Use `StateService`** for production workflows with sync state and metadata tracking:
+
+```typescript
+import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+// ✅ CORRECT: Access openKv from Versori context
+// import { openKv } from '@versori/run'; // ❌ WRONG - Not a direct export
+
+async function ingestWithFileTracking(
+  files: string[],
+  ctx: any
+): Promise<void> {
+  const { log, openKv } = ctx;
+  const logger = toStructuredLogger(log, {
+    service: 'inventory-ingestion'
+  });
+
+  const stateService = new StateService(logger);
+  const kv = new VersoriKVAdapter(openKv(':project:'));
+  let processedCount = 0;
+  let skippedCount = 0;
+
+  for (const fileName of files) {
+    // Check if file already processed
+    if (await state.isFileProcessed(kv, fileName)) {
+      log.info(`Skipping processed file: ${fileName}`);
+      skippedCount++;
+      continue;
+    }
+
+    // Process file
+    log.info(`Processing new file: ${fileName}`);
+    const records = await processFile(fileName);
+    await sendToBatchAPI(records);
+
+    // Mark as processed with metadata
+    await state.updateSyncState(kv, [
+      {
+        fileName,
+        lastModified: new Date().toISOString(),
+        recordCount: records.length,
+      },
+    ]);
+
+    processedCount++;
+  }
+
+  log.info(`Processed: ${processedCount}, Skipped: ${skippedCount}`);
+}
+```
+
+### Tracking with Metadata
+
+Store rich metadata for auditing and troubleshooting:
+
+```typescript
+interface FileProcessingMetadata {
+  fileName: string;
+  lastModified: string;
+  recordCount: number;
+  processedAt: string;
+  jobId?: string;
+  batchIds?: string[];
+  fileSize?: number;
+  checksumMD5?: string;
+  source?: string;
+}
+
+async function trackFileWithMetadata(
+  stateService: StateService,
+  kv: KVStore,
+  fileName: string,
+  metadata: FileProcessingMetadata
+): Promise<void> {
+  await stateService.updateSyncState(kv, [metadata]);
+
+  console.log('File tracked:', {
+    file: fileName,
+    records: metadata.recordCount,
+    jobId: metadata.jobId,
+  });
+}
+
+// Usage
+await trackFileWithMetadata(stateService, kv, 'inventory-2025-01-19.csv', {
+  fileName: 'inventory-2025-01-19.csv',
+  lastModified: '2025-01-19T08:00:00Z',
+  recordCount: 5000,
+  processedAt: new Date().toISOString(),
+  jobId: 'JOB-12345',
+  batchIds: ['BATCH-001', 'BATCH-002'],
+  fileSize: 1024000,
+  checksumMD5: 'a1b2c3d4e5f6',
+  source: 'S3',
+});
+```
+
+---
+
+## Distributed Locking for Concurrent Safety
+
+### The Concurrency Problem
+
+**Without locking:**
+
+```typescript
+// ❌ WRONG - Race condition when multiple processes run
+// Process A: Checks file not processed → Starts processing
+// Process B: Checks file not processed → Starts processing
+// Result: File processed twice simultaneously
+```
+
+**With locking:**
+
+```typescript
+// ✅ CORRECT - Only one process acquires lock
+// Process A: Acquires lock → Processes file → Releases lock
+// Process B: Lock held → Waits or skips → Tries later
+// Result: File processed exactly once
+```
+
+### Acquire and Release Lock Pattern
+
+```typescript
+import { StateService, KVStore } from '@fluentcommerce/fc-connect-sdk';
+
+async function processFileWithLock(
+  fileName: string,
+  state: StateService,
+  kv: KVStore
+): Promise<void> {
+  const lockName = `file:${fileName}`;
+  const lockTimeoutMinutes = 15;
+
+  try {
+    // Attempt to acquire lock
+    const lockAcquired = await state.acquireLock(lockName, kv, lockTimeoutMinutes);
+
+    if (!lockAcquired) {
+      console.log(`Lock held by another process, skipping: ${fileName}`);
+      return;
+    }
+
+    console.log(`Lock acquired for: ${fileName}`);
+
+    // Check if already processed (double-check after acquiring lock)
+    if (await state.isFileProcessed(kv, fileName)) {
+      console.log(`File already processed: ${fileName}`);
+      return;
+    }
+
+    // Process file
+    const records = await processFile(fileName);
+    await sendToBatchAPI(records);
+
+    // Mark as processed
+    await state.updateSyncState(kv, [
+      {
+        fileName,
+        lastModified: new Date().toISOString(),
+        recordCount: records.length,
+      },
+    ]);
+
+    console.log(`Successfully processed: ${fileName}`);
+  } finally {
+    // ALWAYS release lock in finally block
+    await state.releaseLock(lockName, kv);
+    console.log(`Lock released for: ${fileName}`);
+  }
+}
+```
+
+### Lock Timeout and Stale Lock Detection
+
+The SDK automatically handles stale locks:
+
+```typescript
+// Scenario: Process crashes while holding lock
+// 1. Process A acquires lock with 15-minute timeout
+// 2. Process A crashes at minute 10 (lock still held)
+// 3. Minute 16: Lock expires automatically
+// 4. Process B acquires stale lock (overrides expired lock)
+
+async function processWithStaleLockHandling(
+  fileName: string,
+  state: StateService,
+  kv: KVStore
+): Promise<void> {
+  const lockTimeoutMinutes = 15;
+
+  // SDK automatically detects and overrides stale locks
+  const lockAcquired = await state.acquireLock(`file:${fileName}`, kv, lockTimeoutMinutes);
+
+  if (!lockAcquired) {
+    // Lock is held by active process (not stale)
+    console.log('Lock actively held by another process');
+    return;
+  }
+
+  // Lock acquired (either fresh or stale was overridden)
+  try {
+    await processFile(fileName);
+  } finally {
+    await state.releaseLock(`file:${fileName}`, kv);
+  }
+}
+```
+
+---
+
+## Idempotent Processing
+
+**Idempotency** means operations can be safely repeated without changing the result beyond the initial application.
+
+### Idempotent Ingestion Pattern
+
+```typescript
+async function idempotentIngestion(
+  fileName: string,
+  state: StateService,
+  kv: KVStore
+): Promise<{ success: boolean; message: string }> {
+  const lockName = `ingestion:${fileName}`;
+
+  try {
+    // 1. Acquire lock (prevents concurrent processing)
+    const lockAcquired = await state.acquireLock(lockName, kv, 15);
+    if (!lockAcquired) {
+      return {
+        success: true,
+        message: 'File locked by another process, will retry later',
+      };
+    }
+
+    // 2. Check if already processed (prevents duplicates)
+    if (await state.isFileProcessed(kv, fileName)) {
+      return {
+        success: true,
+        message: 'File already processed, skipping',
+      };
+    }
+
+    // 3. Process file
+    const records = await processFile(fileName);
+
+    // 4. Send to Batch API (UPSERT is idempotent)
+    const job = await client.createJob({
+      name: `Import ${fileName}`,
+      retailerId: process.env.FLUENT_RETAILER_ID!,
+    });
+
+    await client.sendBatch(job.id, {
+      action: 'UPSERT', // ✅ UPSERT is idempotent
+      entityType: 'INVENTORY',
+      source: 'STATE_MANAGEMENT_EXAMPLE',
+      event: 'INVENTORY_UPDATE',
+      entities: records,
+    });
+
+    // 5. Mark as processed (atomic operation)
+    await state.updateSyncState(kv, [
+      {
+        fileName,
+        lastModified: new Date().toISOString(),
+        recordCount: records.length,
+        processedAt: new Date().toISOString(),
+      },
+    ]);
+
+    return {
+      success: true,
+      message: `Processed ${records.length} records`,
+    };
+  } catch (error) {
+    // DO NOT mark as processed on error
+    console.error(`Processing failed for ${fileName}:`, error);
+    return {
+      success: false,
+      message: (error as Error).message,
+    };
+  } finally {
+    // Always release lock
+    await state.releaseLock(lockName, kv);
+  }
+}
+
+// Safe to retry - will skip if already completed
+await idempotentIngestion('inventory.csv', state, kv);
+await idempotentIngestion('inventory.csv', state, kv); // ✅ Skips, no duplicates
+await idempotentIngestion('inventory.csv', state, kv); // ✅ Skips, no duplicates
+```
+
+### Why This is Idempotent
+
+| Step                 | Idempotency Guarantee                          |
+| -------------------- | ---------------------------------------------- |
+| **Lock acquisition** | If locked, skip (safe to retry)                |
+| **Processed check**  | If processed, skip (safe to retry)             |
+| **Batch API UPSERT** | UPSERT is idempotent (same data = same result) |
+| **State update**     | Atomic operation (safe to retry)               |
+
+---
+
+## Sync State for Incremental Processing
+
+Track the last processed timestamp to enable incremental updates:
+
+### Get and Update Sync State
+
+```typescript
+import { StateService, SyncState } from '@fluentcommerce/fc-connect-sdk';
+
+async function incrementalSync(
+  state: StateService,
+  kv: KVStore,
+  workflowId: string
+): Promise<void> {
+  // Get last sync state
+  const syncState: SyncState = await state.getSyncState(kv, workflowId);
+
+  if (syncState.isInitialized) {
+    console.log('Last sync:', syncState.lastProcessedTimestamp);
+    console.log('Last file:', syncState.lastProcessedFile);
+  } else {
+    console.log('First sync (no prior state)');
+  }
+
+  // Get files modified after last sync
+  const newFiles = await getFilesModifiedAfter(syncState.lastProcessedTimestamp);
+
+  if (newFiles.length === 0) {
+    console.log('No new files to process');
+    return;
+  }
+
+  const processedFiles = [];
+
+  for (const file of newFiles) {
+    const records = await processFile(file.name);
+    await sendToBatchAPI(records);
+
+    processedFiles.push({
+      fileName: file.name,
+      lastModified: file.lastModified,
+      recordCount: records.length,
+    });
+  }
+
+  // Update sync state with all processed files
+  await state.updateSyncState(kv, processedFiles, workflowId);
+
+  console.log(`Synced ${processedFiles.length} new files`);
+}
+```
+
+### Daily Job Management with State
+
+Reuse a single job for all batches processed in a day:
+
+```typescript
+import { StateService, DailyJob } from '@fluentcommerce/fc-connect-sdk';
+
+async function processWithDailyJob(
+  files: string[],
+  state: StateService,
+  kv: KVStore,
+  client: FluentClient
+): Promise<void> {
+  const workflowId = 'inventory-ingestion';
+
+  // Check for existing daily job
+  let dailyJob: DailyJob | null = await state.getDailyJob(kv, workflowId);
+
+  if (!dailyJob || new Date(dailyJob.expiresAt) < new Date()) {
+    // Create new daily job
+    const job = await client.createJob({
+      name: `Daily Inventory Sync - ${new Date().toISOString().split('T')[0]}`,
+      retailerId: process.env.FLUENT_RETAILER_ID!,
+    });
+
+    // Store daily job (expires in 24 hours)
+    await state.setDailyJob(kv, workflowId, job.id, 24);
+
+    dailyJob = {
+      jobId: job.id,
+      createdAt: new Date().toISOString(),
+      expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString(),
+    };
+
+    console.log(`Created daily job: ${job.id}`);
+  } else {
+    console.log(`Reusing daily job: ${dailyJob.jobId}`);
+  }
+
+  // Process all files with same job
+  for (const fileName of files) {
+    if (await state.isFileProcessed(kv, fileName, workflowId)) {
+      continue;
+    }
+
+    const records = await processFile(fileName);
+
+    await client.sendBatch(dailyJob.jobId, {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      source: 'DAILY_JOB_REUSE',
+      event: 'INVENTORY_UPDATE',
+      entities: records,
+    });
+
+    await state.updateSyncState(
+      kv,
+      [
+        {
+          fileName,
+          lastModified: new Date().toISOString(),
+          recordCount: records.length,
+        },
+      ],
+      workflowId
+    );
+  }
+}
+```
+
+---
+
+## State Storage Adapters
+
+### Versori Platform (Built-in KV)
+
+```typescript
+import { StateService } from '@fluentcommerce/fc-connect-sdk';
+import { workflow } from '@versori/run';
+
+export const inventoryIngestion = workflow('inventory-ingestion', async ctx => {
+  const { log, openKv } = ctx;
+
+  // Open Versori KV store
+  const kvAdapter = new VersoriKVAdapter(openKv(':project:'));
+
+  // Create logger and state service
+  const logger = toStructuredLogger(log, {
+    service: 'inventory-ingestion'
+  });
+  const stateService = new StateService(logger);
+
+  // Use state methods with kvAdapter parameter
+  const isProcessed = await stateService.isFileProcessed(kvAdapter, 'inventory.csv');
+
+  if (!isProcessed) {
+    // Process file
+    await processFile('inventory.csv');
+
+    // Mark as processed
+    await stateService.updateSyncState(kvAdapter, [
+      {
+        fileName: 'inventory.csv',
+        lastModified: new Date().toISOString(),
+        recordCount: 1000,
+      },
+    ]);
+  }
+});
+```
+
+### Node.js Standalone (Custom KV)
+
+```typescript
+import { StateService, KVStore } from '@fluentcommerce/fc-connect-sdk';
+
+// Implement KVStore interface with Redis, DynamoDB, etc.
+class RedisKVStore implements KVStore {
+  constructor(private redis: RedisClient) {}
+
+  async get(keys: string[]): Promise<{ value: unknown } | null> {
+    const key = keys.join(':');
+    const value = await this.redis.get(key);
+    return value ? { value: JSON.parse(value) } : null;
+  }
+
+  async set(keys: string[], value: unknown): Promise<void> {
+    const key = keys.join(':');
+    await this.redis.set(key, JSON.stringify(value));
+  }
+
+  async delete(keys: string[]): Promise<void> {
+    const key = keys.join(':');
+    await this.redis.del(key);
+  }
+
+  atomic() {
+    // Implement atomic operations with Redis transactions
+    const operations: Array<{ type: string; key: string[]; value?: unknown }> = [];
+
+    return {
+      check: (entries: Array<{ key: string[]; versionstamp: string | null }>) => {
+        // Redis WATCH for optimistic locking
+        entries.forEach(e => this.redis.watch(e.key.join(':')));
+      },
+      set: (key: string[], value: unknown) => {
+        operations.push({ type: 'set', key, value });
+      },
+      commit: async (): Promise<boolean> => {
+        const multi = this.redis.multi();
+        operations.forEach(op => {
+          if (op.type === 'set') {
+            multi.set(op.key.join(':'), JSON.stringify(op.value));
+          }
+        });
+        const results = await multi.exec();
+        return results !== null;
+      },
+    };
+  }
+}
+
+// Usage
+const logger = toStructuredLogger(createConsoleLogger(), {
+  service: 'inventory-ingestion',
+  correlationId: generateCorrelationId()
+});
+
+const redisKV = new RedisKVStore(redisClient);
+const stateService = new StateService(logger);
+
+await stateService.isFileProcessed(redisKV, 'inventory.csv');
+```
+
+---
+
+## Complete Production Example
+
+Full ingestion workflow with state management, locking, and error handling:
+
+```typescript
+import {
+  createClient,
+  FluentClient,
+  StateService,
+  KVStore,
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  BatchAction,
+  EntityType,
+} from '@fluentcommerce/fc-connect-sdk';
+
+interface IngestionConfig {
+  s3: {
+    bucket: string;
+    prefix: string;
+  };
+  fluent: {
+    baseUrl: string;
+    clientId: string;
+    clientSecret: string;
+    retailerId: string;
+  };
+  workflow: {
+    id: string;
+    lockTimeoutMinutes: number;
+  };
+}
+
+async function productionIngestionWithState(
+  config: IngestionConfig,
+  kv: KVStore
+): Promise<{
+  success: boolean;
+  filesProcessed: number;
+  filesSkipped: number;
+  errors: string[];
+}> {
+  const logger = toStructuredLogger(createConsoleLogger(), {
+    service: 'inventory-ingestion',
+    correlationId: generateCorrelationId()
+  });
+
+  const stateService = new StateService(logger);
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_CSV',
+      connectionId: 's3-state-example',
+      name: 'S3 State Example',
+      s3Config: {
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+        region: process.env.AWS_REGION!,
+        bucket: 'inventory-bucket',
+      },
+    },
+    logger
+  );
+
+  const client = await createClient({
+    config: {
+      baseUrl: config.fluent.baseUrl,
+      clientId: config.fluent.clientId,
+      clientSecret: config.fluent.clientSecret,
+      retailerId: config.fluent.retailerId,
+    }
+  });
+
+  const parser = new CSVParserService();
+  const mapper = new UniversalMapper({
+    fields: {
+      ref: {
+        source: 'sku',
+        required: true,
+        comment: '✅ InventoryPositionInput.ref (String! required)',
+      },
+      productRef: {
+        source: 'sku',
+        required: true,
+        comment: '✅ InventoryPositionInput.productRef (String! required)',
+      },
+      locationRef: {
+        source: 'warehouse',
+        required: true,
+        comment: '✅ InventoryPositionInput.locationRef (String! required)',
+      },
+      qty: {
+        source: 'quantity',
+        resolver: 'sdk.parseInt',
+        required: true,
+        comment: '✅ InventoryPositionInput.qty (Int! required)',
+      },
+      status: {
+        source: 'status',
+        resolver: 'sdk.uppercase',
+        defaultValue: 'AVAILABLE',
+        comment: '✅ InventoryPositionInput.status (String optional)',
+      },
+    },
+  });
+
+  let filesProcessed = 0;
+  let filesSkipped = 0;
+  const errors: string[] = [];
+
+  try {
+    // List files from S3
+    const files = await s3.listFiles({ prefix: config.s3.prefix });
+    console.log(`Found ${files.length} files in S3`);
+
+    // Get or create daily job
+    let dailyJob = await state.getDailyJob(kv, config.workflow.id);
+
+    if (!dailyJob) {
+      const job = await client.createJob({
+        name: `Daily Sync - ${new Date().toISOString().split('T')[0]}`,
+        retailerId: config.fluent.retailerId,
+      });
+
+      await state.setDailyJob(kv, config.workflow.id, job.id, 24);
+      dailyJob = {
+        jobId: job.id,
+        createdAt: new Date().toISOString(),
+        expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString(),
+      };
+    }
+
+    console.log(`Using job: ${dailyJob.jobId}`);
+
+    for (const file of files) {
+      const lockName = `file:${file.path}`;
+
+      try {
+        // 1. Acquire lock
+        const lockAcquired = await state.acquireLock(
+          lockName,
+          kv,
+          config.workflow.lockTimeoutMinutes
+        );
+
+        if (!lockAcquired) {
+          console.log(`File locked by another process: ${file.path}`);
+          filesSkipped++;
+          continue;
+        }
+
+        // 2. Check if already processed
+        if (await state.isFileProcessed(kv, file.path, config.workflow.id)) {
+          console.log(`File already processed: ${file.path}`);
+          filesSkipped++;
+          await state.releaseLock(lockName, kv);
+          continue;
+        }
+
+        // 3. Download and parse file
+        const fileContent = await s3.downloadFile(file.path);
+        const records = await parser.parse(fileContent);
+
+        console.log(`Parsed ${records.length} records from ${file.path}`);
+
+        // 4. Transform data
+        const mappingResult = await mapper.map(records);
+
+        if (!mappingResult.success) {
+          throw new Error(`Mapping failed: ${JSON.stringify(mappingResult.errors)}`);
+        }
+
+        // 5. Send to Batch API
+        const batch = await client.sendBatch(dailyJob.jobId, {
+          action: 'UPSERT',
+          entityType: 'INVENTORY',
+          source: 'COMPLETE_WORKFLOW',
+          event: 'INVENTORY_UPDATE',
+          entities: mappingResult.data,
+        });
+
+        console.log(`Batch sent: ${batch.id}`);
+
+        // 6. Mark as processed
+        await state.updateSyncState(
+          kv,
+          [
+            {
+              fileName: file.path,
+              lastModified: file.lastModified || new Date().toISOString(),
+              recordCount: mappingResult.data.length,
+              processedAt: new Date().toISOString(),
+            },
+          ],
+          config.workflow.id
+        );
+
+        filesProcessed++;
+
+        // 7. Release lock
+        await state.releaseLock(lockName, kv);
+      } catch (error) {
+        const errorMsg = `Failed to process ${file.path}: ${(error as Error).message}`;
+        console.error(errorMsg);
+        errors.push(errorMsg);
+
+        // Release lock on error
+        await state.releaseLock(lockName, kv);
+      }
+    }
+
+    return {
+      success: errors.length === 0,
+      filesProcessed,
+      filesSkipped,
+      errors,
+    };
+  } catch (error) {
+    console.error('Ingestion failed:', error);
+    throw error;
+  }
+}
+
+// Usage
+const result = await productionIngestionWithState(
+  {
+    s3: {
+      bucket: 'inventory-bucket',
+      prefix: 'data/',
+    },
+    fluent: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!,
+    },
+    workflow: {
+      id: 'inventory-ingestion',
+      lockTimeoutMinutes: 15,
+    },
+  },
+  kv
+);
+
+console.log('Ingestion result:', result);
+// Output:
+// {
+//   success: true,
+//   filesProcessed: 10,
+//   filesSkipped: 5,
+//   errors: []
+// }
+```
+
+---
+
+## Troubleshooting
+
+### Problem: Files Reprocessed Every Run
+
+**Symptoms:**
+
+- Same files processed repeatedly
+- Duplicate inventory positions
+
+**Solution:**
+
+```typescript
+// Check if state is being updated correctly
+const syncState = await state.getSyncState(kv, workflowId);
+console.log('Sync state:', syncState);
+
+// Verify file is marked as processed
+const isProcessed = await state.isFileProcessed(kv, fileName, workflowId);
+console.log(`File ${fileName} processed:`, isProcessed);
+
+// Ensure updateSyncState is called AFTER successful processing
+await state.updateSyncState(kv, [{ fileName, lastModified, recordCount }], workflowId);
+```
+
+### Problem: Lock Never Released
+
+**Symptoms:**
+
+- Files stuck in "locked" state
+- Processing stops
+
+**Solution:**
+
+```typescript
+// ALWAYS use try-finally to release locks
+try {
+  const lockAcquired = await state.acquireLock(lockName, kv, 15);
+  if (lockAcquired) {
+    await processFile();
+  }
+} finally {
+  // ✅ CRITICAL: Release lock even if processing fails
+  await state.releaseLock(lockName, kv);
+}
+```
+
+### Problem: Stale Locks Blocking Processing
+
+**Symptoms:**
+
+- Locks from crashed processes
+- Processing stuck indefinitely
+
+**Solution:**
+
+```typescript
+// SDK automatically overrides stale locks based on timeout
+// If lock expired (past expiresAt), next acquireLock succeeds
+
+// To manually clear stale locks (if needed):
+await state.releaseLock(lockName, kv); // Force release
+```
+
+### Problem: State Lost Between Runs
+
+**Symptoms:**
+
+- First-time sync every run
+- `isInitialized: false` always
+
+**Solution:**
+
+```typescript
+// Verify KV store persistence
+const testKey = ['test', 'persistence'];
+await kv.set(testKey, { value: 'test' });
+const retrieved = await kv.get(testKey);
+console.log('KV persistence test:', retrieved);
+
+// Ensure workflowId is consistent
+const WORKFLOW_ID = 'inventory-ingestion'; // ✅ Use constant
+await state.updateSyncState(kv, files, WORKFLOW_ID);
+```
+
+---
+
+## Key Takeaways
+
+- 🎯 **Always use state management** - Prevents duplicates and enables idempotency
+- 🎯 **Acquire locks before processing** - Prevents concurrent conflicts
+- 🎯 **Release locks in finally blocks** - Ensures cleanup even on errors
+- 🎯 **Track file metadata** - Rich metadata enables auditing and troubleshooting
+- 🎯 **Use workflow IDs** - Scope state to specific workflows
+- 🎯 **Handle stale locks** - SDK automatically overrides expired locks
+
+---
+
+## Next Steps
+
+Continue to [Module 8: Performance Optimization](./02-core-guides-ingestion-08-performance-optimization.md) to learn how to optimize batch sizing, parallel processing, and job strategies for large-scale ingestion.
+
+---
+
+[← Previous: Batch API](./02-core-guides-ingestion-06-batch-api.md) | [Back to Guide](../ingestion-readme.md) | [Next: Performance Optimization →](./02-core-guides-ingestion-08-performance-optimization.md)
+
+## Related Documentation
+
+- [StateService API](../../api-reference/modules/api-reference-05-services.md#state-service) - Complete API documentation
+- [Versori KV Integration](../../../04-REFERENCE/platforms/versori/#key-value-storage) - Platform-specific KV usage
+- [Error Handling Guide](../../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md) - Retry patterns and error recovery
+- [Best Practices](./02-core-guides-ingestion-09-best-practices.md) - Production patterns and monitoring