@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-09-best-practices.md

@@ -1,1893 +1,1893 @@
-# Module 9: Best Practices
-
-[← Back to Ingestion Guide](../ingestion-readme.md)
-
-**Module 9 of 9** | **Level**: All Levels | **Time**: 30 minutes
-
----
-
-## Overview
-
-This module covers production-ready patterns for data ingestion workflows. Learn comprehensive error handling strategies, monitoring approaches, security best practices, scheduling patterns, data quality validation, and complete production deployment patterns.
-
-New in this version:
-
-- Preflight validation before runs (PreflightValidator)
-- Job lifecycle tracking (JobTracker)
-- Partial batch failure recovery (PartialBatchRecovery)
-- Versori file-level deduplication (VersoriFileTracker)
-
-## Learning Objectives
-
-By the end of this module, you will:
-
-- ✅ Implement robust error handling with retry logic and circuit breakers
-- ✅ Set up comprehensive monitoring, logging, and alerting
-- ✅ Apply security best practices for credentials and data
-- ✅ Design effective scheduling and trigger strategies
-- ✅ Validate data quality before ingestion
-- ✅ Build production-ready ingestion pipelines
-- ✅ Troubleshoot common issues effectively
-- ✅ Follow deployment checklists for success
-
----
-
-## Error Handling
-
-### Retry Strategy with Exponential Backoff
-
-```typescript
-async function retryWithBackoff<T>(
-  operation: () => Promise<T>,
-  maxRetries: number = 3
-): Promise<T> {
-  let lastError: Error;
-  let delay = 1000; // Start with 1 second
-
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      return await operation();
-    } catch (error) {
-      lastError = error;
-
-      if (!isRetryable(error) || attempt === maxRetries) {
-        throw error;
-      }
-
-      console.log(\`Retry \${attempt}/\${maxRetries} after \${delay}ms\`);
-      await sleep(delay);
-      delay *= 2; // Exponential backoff
-    }
-  }
-
-  throw lastError;
-}
-
-function isRetryable(error: any): boolean {
-  const retryableCodes = [
-    'RATE_LIMIT_ERROR',
-    'NETWORK_ERROR',
-    'TIMEOUT_ERROR',
-    'SERVICE_UNAVAILABLE'
-  ];
-
-  return (
-    retryableCodes.includes(error.code) ||
-    error.message.includes('timeout') ||
-    error.message.includes('ECONNRESET')
-  );
-}
-```
-
-### SDK-Specific Error Handling
-
-The SDK throws specific error types that require different handling strategies:
-
-```typescript
-import {
-  FileParsingError,
-  MappingError,
-  BatchAPIError,
-  StateError,
-  createConsoleLogger,
-  toStructuredLogger,
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function handleSDKErrors(fileKey: string) {
-  try {
-    await processFile(fileKey);
-  } catch (error) {
-    if (error instanceof FileParsingError) {
-      // File format issues - log and skip
-      console.error(`Invalid file format: ${fileKey}`, {
-        line: error.line,
-        column: error.column,
-        message: error.message,
-      });
-
-      // Move to error bucket for manual review
-      await moveToErrorBucket(fileKey, error);
-      return { status: 'skipped', reason: 'parsing_error' };
-    } else if (error instanceof MappingError) {
-      // Field mapping failures - log details
-      console.error(`Field mapping failed: ${fileKey}`, {
-        invalidFields: error.invalidFields,
-        recordIndex: error.recordIndex,
-        errors: error.errors,
-      });
-
-      // Save error report
-      await saveErrorReport(fileKey, error);
-      return { status: 'failed', reason: 'mapping_error' };
-    } else if (error instanceof BatchAPIError) {
-      // Batch API rejections - check if retryable
-      if (error.isRetryable) {
-        console.warn(`Retryable batch error: ${error.code}`);
-        throw error; // Will be caught by retry logic
-      } else {
-        console.error(`Permanent batch error: ${error.code}`, {
-          jobId: error.jobId,
-          batchId: error.batchId,
-          details: error.details,
-        });
-
-        // Save to dead letter queue
-        await saveToDLQ(fileKey, error);
-        return { status: 'failed', reason: 'batch_api_error' };
-      }
-    } else if (error instanceof StateError) {
-      // State management issues - critical
-      console.error(`State management error: ${error.message}`);
-
-      // Alert operations team
-      await sendAlert('CRITICAL: State management failure', error);
-      throw error; // Don't continue if state is broken
-    } else {
-      // Unknown error - log and alert
-      console.error(`Unexpected error processing ${fileKey}:`, error);
-      await sendAlert('Unknown ingestion error', error);
-      throw error;
-    }
-  }
-}
-```
-
-### Circuit Breaker Pattern
-
-Prevent cascading failures when external services are down:
-
-```typescript
-interface CircuitBreakerConfig {
-  failureThreshold: number;
-  resetTimeoutMs: number;
-  monitoringWindowMs: number;
-}
-
-enum CircuitState {
-  CLOSED = 'CLOSED', // Normal operation
-  OPEN = 'OPEN', // Blocking requests
-  HALF_OPEN = 'HALF_OPEN', // Testing recovery
-}
-
-class CircuitBreaker {
-  private state: CircuitState = CircuitState.CLOSED;
-  private failureCount: number = 0;
-  private lastFailureTime: number = 0;
-  private failures: number[] = [];
-
-  constructor(private config: CircuitBreakerConfig) {}
-
-  async execute<T>(operation: () => Promise<T>): Promise<T> {
-    // Check if circuit should reset
-    if (this.shouldAttemptReset()) {
-      this.state = CircuitState.HALF_OPEN;
-    }
-
-    // Block if circuit is open
-    if (this.state === CircuitState.OPEN) {
-      throw new Error(
-        `Circuit breaker OPEN. Last failure: ${new Date(this.lastFailureTime).toISOString()}`
-      );
-    }
-
-    try {
-      const result = await operation();
-
-      // Success - reset if in half-open state
-      if (this.state === CircuitState.HALF_OPEN) {
-        console.log('Circuit breaker: Service recovered, closing circuit');
-        this.reset();
-      }
-
-      return result;
-    } catch (error) {
-      this.recordFailure();
-
-      // Open circuit if threshold exceeded
-      if (this.failureCount >= this.config.failureThreshold) {
-        this.state = CircuitState.OPEN;
-        this.lastFailureTime = Date.now();
-        console.error(`Circuit breaker OPENED after ${this.failureCount} failures`);
-      }
-
-      throw error;
-    }
-  }
-
-  private recordFailure(): void {
-    const now = Date.now();
-    this.failures.push(now);
-
-    // Remove old failures outside monitoring window
-    this.failures = this.failures.filter(time => now - time < this.config.monitoringWindowMs);
-
-    this.failureCount = this.failures.length;
-  }
-
-  private shouldAttemptReset(): boolean {
-    if (this.state !== CircuitState.OPEN) {
-      return false;
-    }
-
-    const timeSinceLastFailure = Date.now() - this.lastFailureTime;
-    return timeSinceLastFailure >= this.config.resetTimeoutMs;
-  }
-
-  private reset(): void {
-    this.state = CircuitState.CLOSED;
-    this.failureCount = 0;
-    this.failures = [];
-  }
-
-  getState(): CircuitState {
-    return this.state;
-  }
-}
-
-// Usage
-const fluentAPICircuit = new CircuitBreaker({
-  failureThreshold: 5, // Open after 5 failures
-  resetTimeoutMs: 60000, // Try again after 1 minute
-  monitoringWindowMs: 120000, // Track failures over 2 minutes
-});
-
-async function processWithCircuitBreaker(fileKey: string) {
-  try {
-    await fluentAPICircuit.execute(async () => {
-      return await processFile(fileKey);
-    });
-  } catch (error) {
-    if (error.message.includes('Circuit breaker OPEN')) {
-      console.log('Service temporarily unavailable, requeueing file');
-      await requeueFile(fileKey);
-    } else {
-      throw error;
-    }
-  }
-}
-```
-
-### Dead Letter Queue Pattern
-
-Store failed records for later processing:
-
-```typescript
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-
-interface DLQRecord {
-  originalFile: string;
-  failedAt: string;
-  errorType: string;
-  errorMessage: string;
-  recordData: any;
-  retryCount: number;
-  metadata: Record<string, any>;
-}
-
-class DeadLetterQueue {
-  constructor(
-    private s3: S3DataSource,
-    private dlqBucket: string,
-    private dlqPrefix: string = 'dlq/'
-  ) {}
-
-  /**
-   * Save failed record to DLQ
-   */
-  async save(record: DLQRecord): Promise<void> {
-    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-    const key = `${this.dlqPrefix}${record.errorType}/${timestamp}-${record.originalFile}`;
-
-    await this.s3.uploadFile(key, JSON.stringify(record, null, 2));
-
-    console.log(`Saved to DLQ: ${key}`);
-  }
-
-  /**
-   * Retry processing records from DLQ
-   */
-  async retryAll(
-    processor: (record: any) => Promise<void>
-  ): Promise<{ successful: number; failed: number }> {
-    const files = await this.s3.listFiles({ prefix: this.dlqPrefix });
-    let successful = 0;
-    let failed = 0;
-
-    for (const file of files) {
-      try {
-        const content = await this.s3.downloadFile(file.path);
-        const dlqRecord: DLQRecord = JSON.parse(content);
-
-        // Attempt reprocessing
-        await processor(dlqRecord.recordData);
-
-        // Success - delete from DLQ (Note: deleteObject method may not exist - use uploadFile with empty content or check SDK)
-        // await this.s3.deleteObject(this.dlqBucket, file.path);
-        successful++;
-      } catch (error) {
-        console.error(`DLQ retry failed for ${file.path}:`, error);
-        failed++;
-
-        // Update retry count
-        await this.incrementRetryCount(file.path);
-      }
-    }
-
-    return { successful, failed };
-  }
-
-  /**
-   * Increment retry count in DLQ record
-   */
-  private async incrementRetryCount(key: string): Promise<void> {
-    try {
-      const content = await this.s3.downloadFile(key);
-      const record: DLQRecord = JSON.parse(content);
-
-      record.retryCount = (record.retryCount || 0) + 1;
-      record.metadata.lastRetryAttempt = new Date().toISOString();
-
-      await this.s3.uploadFile(key, JSON.stringify(record, null, 2));
-    } catch (error) {
-      console.error(`Failed to update retry count for ${key}:`, error);
-    }
-  }
-}
-
-// Usage
-const dlq = new DeadLetterQueue(s3DataSource, 'my-dlq-bucket');
-
-async function processFileWithDLQ(fileKey: string) {
-  try {
-    const records = await parseFile(fileKey);
-
-    for (const [index, record] of records.entries()) {
-      try {
-        await processRecord(record);
-      } catch (error) {
-        // Save failed record to DLQ
-        await dlq.save({
-          originalFile: fileKey,
-          failedAt: new Date().toISOString(),
-          errorType: error.constructor.name,
-          errorMessage: error.message,
-          recordData: record,
-          retryCount: 0,
-          metadata: {
-            recordIndex: index,
-            totalRecords: records.length,
-          },
-        });
-      }
-    }
-  } catch (error) {
-    console.error(`Complete file failure: ${fileKey}`, error);
-    throw error;
-  }
-}
-```
-
-### Graceful Degradation
-
-```typescript
-async function processFileWithFallback(fileKey: string) {
-  try {
-    // Primary processing path
-    await processFileNormally(fileKey);
-  } catch (error) {
-    console.error(\`Primary processing failed: \${error.message}\`);
-
-    try {
-      // Fallback: Process with reduced batch size
-      await processFileWithSmallerBatches(fileKey);
-    } catch (fallbackError) {
-      console.error(\`Fallback processing failed: \${fallbackError.message}\`);
-
-      // Last resort: Save to dead letter queue
-      await saveToDeadLetterQueue(fileKey, error);
-      throw new Error(\`File processing failed completely: \${fileKey}\`);
-    }
-  }
-}
-```
-
-## Validation Best Practices
-
-### Pre-Flight Validation
-
-```typescript
-async function validateBeforeIngestion(records: any[]): Promise<ValidationResult> {
-  const errors = [];
-  const warnings = [];
-
-  // Schema validation
-  for (const [index, record] of records.entries()) {
-    // Required fields
-    if (!record.ref) {
-      errors.push(\`Row \${index}: Missing required field 'ref'\`);
-    }
-
-    if (!record.productRef) {
-      errors.push(\`Row \${index}: Missing required field 'productRef'\`);
-    }
-
-    if (!record.locationRef) {
-      errors.push(\`Row \${index}: Missing required field 'locationRef'\`);
-    }
-
-    if (record.qty === undefined || record.qty === null) {
-      errors.push(\`Row \${index}: Missing required field 'qty'\`);
-    }
-
-    // Type validation
-    if (typeof record.qty !== 'number') {
-      errors.push(\`Row \${index}: 'qty' must be a number\`);
-    }
-
-    // Business rules
-    if (record.qty < 0) {
-      errors.push(\`Row \${index}: Quantity cannot be negative\`);
-    }
-
-    if (record.qty > 1000000) {
-      warnings.push(\`Row \${index}: Unusually large quantity\`);
-    }
-  }
-
-  return {
-    isValid: errors.length === 0,
-    errors,
-    warnings
-  };
-}
-
-// Usage
-const validation = await validateBeforeIngestion(records);
-
-if (!validation.isValid) {
-  console.error('Validation errors:', validation.errors);
-  throw new Error('Data validation failed');
-}
-
-if (validation.warnings.length > 0) {
-  console.warn('Validation warnings:', validation.warnings);
-}
-```
-
-## Scheduling and Triggers
-
-###Cron Scheduling Patterns
-
-Different scheduling strategies for different ingestion scenarios:
-
-```typescript
-// Daily midnight sync (00:00 UTC)
-const dailyMidnightCron = '0 0 * * *';
-
-// Every 6 hours
-const every6HoursCron = '0 */6 * * *';
-
-// Every hour during business hours (9 AM - 5 PM, Mon-Fri)
-const businessHoursCron = '0 9-17 * * 1-5';
-
-// Every 15 minutes
-const every15MinutesCron = '*/15 * * * *';
-
-// Weekly Sunday at 2 AM
-const weeklySundayCron = '0 2 * * 0';
-```
-
-**Recommended schedules by use case:**
-
-| Use Case                | Schedule      | Cron Expression  | Rationale                    |
-| ----------------------- | ------------- | ---------------- | ---------------------------- |
-| **WMS Daily Sync**      | Once daily    | `0 2 * * *`      | Process overnight updates    |
-| **3PL Cycle Counts**    | Every 6 hours | `0 */6 * * *`    | Keep inventory fresh         |
-| **Real-time Updates**   | Every 15 min  | `*/15 * * * *`   | Near real-time sync          |
-| **Weekly Full Sync**    | Sunday 2 AM   | `0 2 * * 0`      | Comprehensive reconciliation |
-| **Business Hours Only** | 9 AM - 5 PM   | `0 9-17 * * 1-5` | During operational hours     |
-
-### Trigger Mechanisms
-
-Different ways to trigger ingestion workflows:
-
-#### 1. S3 Event Triggers
-
-```typescript
-// Versori workflow triggered by S3 event
-export const s3TriggeredIngestion = webhook('s3-inventory-upload', {
-  response: { mode: 'sync' },
-})
-  .then(
-    fn('parse-s3-event', async ({ data }) => {
-      const event = JSON.parse(data);
-      const bucket = event.Records[0].s3.bucket.name;
-      const key = decodeURIComponent(event.Records[0].s3.object.key);
-
-      return { bucket, key };
-    })
-  )
-  .then(
-    fn('process-file', async ({ data, connections, log, openKv }) => {
-      const client = await createClient({ connections, log, openKv });
-
-      // Initialize logger and state management
-      const logger = toStructuredLogger(log, {
-        service: 'ingestion',
-        correlationId: generateCorrelationId()
-      });
-
-      const stateService = new StateService(logger);
-      const kvAdapter = new VersoriKVAdapter(openKv());
-
-      const s3 = new S3DataSource(
-        {
-          type: 'S3_CSV',
-          connectionId: 's3-triggered',
-          name: 'S3 Triggered Ingestion',
-          s3Config: { region: 'us-east-1', bucket: data.bucket },
-        },
-        logger
-      );
-
-      // Check if already processed
-      if (await stateService.isFileProcessed(kvAdapter, data.key)) {
-        log.info(`File already processed: ${data.key}`);
-        return { status: 'skipped', reason: 'already_processed' };
-      }
-
-      // Process file
-      await processInventoryFile(client, s3, data.bucket, data.key);
-
-      // Mark as processed
-      await stateService.updateSyncState(kvAdapter, [{
-        fileName: data.key,
-        lastModified: new Date().toISOString(),
-      }]);
-
-      return { status: 'success', file: data.key };
-    })
-  )
-  .catch(({ data, log }) => {
-    log.error('S3 triggered ingestion failed:', data);
-    return { status: 'error', error: data };
-  });
-```
-
-#### 2. SFTP Polling
-
-```typescript
-import { SftpDataSource, StateService, VersoriKVAdapter, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function pollSftpDirectory(kv: any) {
-  const logger = toStructuredLogger(createConsoleLogger(), {
-    service: 'ingestion',
-    correlationId: generateCorrelationId()
-  });
-
-  const stateService = new StateService(logger);
-  const kvAdapter = new VersoriKVAdapter(kv);
-
-  const sftp = new SftpDataSource({
-    type: 'SFTP_CSV',
-    connectionId: 'sftp-polling',
-    name: 'SFTP Polling',
-    settings: {
-      host: process.env.SFTP_HOST!,
-      port: 22,
-      username: process.env.SFTP_USER!,
-      privateKey: fs.readFileSync('/path/to/private/key'),
-    }
-  }, logger);
-
-  const remoteDir = '/inbound/inventory/';
-  const files = await sftp.listFiles(remoteDir);
-
-  for (const file of files) {
-    if (await stateService.isFileProcessed(kvAdapter, file.name)) {
-      continue; // Skip already processed
-    }
-
-    // Download and process
-    const localPath = `/tmp/${file.name}`;
-    await sftp.downloadFile(file.path, localPath); // Use file.path (full path) for download
-
-    await processLocalFile(localPath);
-
-    // Mark as processed
-    await stateService.updateSyncState(kvAdapter, [{
-      fileName: file.name,
-      lastModified: new Date().toISOString(),
-    }]);
-
-    // Optionally move to processed folder
-    await sftp.moveFile(`${remoteDir}${file.name}`, `/processed/${file.name}`);
-  }
-}
-```
-
-#### 3. Webhook Triggers
-
-```typescript
-// Manual or external system triggered ingestion
-export const manualIngestion = webhook('manual-trigger', {
-  response: { mode: 'sync' },
-})
-  .then(
-    fn('validate-request', ({ data }) => {
-      const { bucket, prefix, batchSize = 2000 } = JSON.parse(data);
-
-      if (!bucket || !prefix) {
-        throw new Error('Missing required parameters: bucket, prefix');
-      }
-
-      return { bucket, prefix, batchSize };
-    })
-  )
-  .then(
-    fn('run-ingestion', async ({ data, connections, log }) => {
-      const client = await createClient(ctx); // Auto-detects Versori context
-      const s3 = new S3DataSource(
-        {
-          type: 'S3_CSV',
-          s3Config: { region: 'us-east-1' },
-        },
-        log
-      );
-
-      const files = await s3.listFiles({ prefix: data.prefix });
-
-      log.info(`Found ${files.length} files to process`);
-
-      const results = await processFilesInParallel(client, s3, data.bucket, files, {
-        batchSize: data.batchSize,
-      });
-
-      return {
-        filesProcessed: results.successful,
-        filesFailed: results.failed,
-        totalRecords: results.totalRecords,
-      };
-    })
-  );
-```
-
-### Backfill Strategies
-
-Handle missed files or catch-up processing:
-
-```typescript
-async function backfillMissedFiles(startDate: Date, endDate: Date, kv: any): Promise<void> {
-  const logger = toStructuredLogger(createConsoleLogger(), {
-    service: 'backfill',
-    correlationId: generateCorrelationId()
-  });
-
-  const stateService = new StateService(logger);
-  const kvAdapter = new VersoriKVAdapter(kv);
-
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_CSV',
-      connectionId: 's3-backfill',
-      name: 'S3 Backfill',
-      s3Config: s3Config,
-    },
-    logger
-  );
-
-  console.log(`Backfilling files from ${startDate} to ${endDate}`);
-
-  // List all files in date range
-  const allFiles = await s3.listFiles({ prefix: 'data/' });
-
-  const filesInRange = allFiles.filter(file => {
-    const fileDate = extractDateFromKey(file.path);
-    return fileDate >= startDate && fileDate <= endDate;
-  });
-
-  console.log(`Found ${filesInRange.length} files in date range`);
-
-  // Check processing status
-  const unprocessedFiles = [];
-  for (const file of filesInRange) {
-    if (!(await stateService.isFileProcessed(kvAdapter, file.path))) {
-      unprocessedFiles.push(file);
-    }
-  }
-
-  console.log(`${unprocessedFiles.length} files need processing`);
-
-  // Process with lower concurrency to avoid overwhelming system
-  await processFilesInParallel(unprocessedFiles, 2); // Only 2 concurrent
-}
-
-function extractDateFromKey(key: string): Date {
-  // Extract date from key like "data/inventory-2025-01-15.csv"
-  const match = key.match(/(\d{4}-\d{2}-\d{2})/);
-  return match ? new Date(match[1]) : new Date(0);
-}
-```
-
-## Data Quality and Validation
-
-### Pre-Ingestion Data Quality Checks
-
-```typescript
-interface DataQualityRule {
-  name: string;
-  validate: (record: any) => { isValid: boolean; error?: string };
-  severity: 'error' | 'warning';
-}
-
-class DataQualityValidator {
-  private rules: DataQualityRule[] = [];
-
-  addRule(rule: DataQualityRule): void {
-    this.rules.push(rule);
-  }
-
-  validate(records: any[]): {
-    passed: any[];
-    failed: Array<{ record: any; errors: string[] }>;
-    warnings: Array<{ record: any; warnings: string[] }>;
-  } {
-    const passed: any[] = [];
-    const failed: Array<{ record: any; errors: string[] }> = [];
-    const warnings: Array<{ record: any; warnings: string[] }> = [];
-
-    for (const record of records) {
-      const errors: string[] = [];
-      const warns: string[] = [];
-
-      for (const rule of this.rules) {
-        const result = rule.validate(record);
-
-        if (!result.isValid) {
-          if (rule.severity === 'error') {
-            errors.push(`${rule.name}: ${result.error}`);
-          } else {
-            warns.push(`${rule.name}: ${result.error}`);
-          }
-        }
-      }
-
-      if (errors.length > 0) {
-        failed.push({ record, errors });
-      } else {
-        passed.push(record);
-
-        if (warns.length > 0) {
-          warnings.push({ record, warnings: warns });
-        }
-      }
-    }
-
-    return { passed, failed, warnings };
-  }
-}
-
-// Define quality rules
-const qualityValidator = new DataQualityValidator();
-
-qualityValidator.addRule({
-  name: 'Required Fields',
-  severity: 'error',
-  validate: record => {
-    const required = ['sku', 'warehouse', 'quantity'];
-    const missing = required.filter(field => !record[field]);
-
-    if (missing.length > 0) {
-      return {
-        isValid: false,
-        error: `Missing required fields: ${missing.join(', ')}`,
-      };
-    }
-
-    return { isValid: true };
-  },
-});
-
-qualityValidator.addRule({
-  name: 'Quantity Range',
-  severity: 'error',
-  validate: record => {
-    const qty = parseInt(record.quantity, 10);
-
-    if (isNaN(qty) || qty < 0) {
-      return {
-        isValid: false,
-        error: `Invalid quantity: ${record.quantity}`,
-      };
-    }
-
-    return { isValid: true };
-  },
-});
-
-qualityValidator.addRule({
-  name: 'SKU Format',
-  severity: 'error',
-  validate: record => {
-    const skuPattern = /^SKU-[A-Z0-9]{2,10}$/;
-
-    if (!skuPattern.test(record.sku)) {
-      return {
-        isValid: false,
-        error: `Invalid SKU format: ${record.sku}`,
-      };
-    }
-
-    return { isValid: true };
-  },
-});
-
-qualityValidator.addRule({
-  name: 'Suspiciously Large Quantity',
-  severity: 'warning',
-  validate: record => {
-    const qty = parseInt(record.quantity, 10);
-
-    if (qty > 100000) {
-      return {
-        isValid: false,
-        error: `Unusually large quantity: ${qty}`,
-      };
-    }
-
-    return { isValid: true };
-  },
-});
-
-// Usage
-const records = await parseCSV(fileContent);
-const validation = qualityValidator.validate(records);
-
-console.log(`Passed: ${validation.passed.length}`);
-console.log(`Failed: ${validation.failed.length}`);
-console.log(`Warnings: ${validation.warnings.length}`);
-
-// Log failures
-if (validation.failed.length > 0) {
-  console.error('Data quality failures:');
-  validation.failed.forEach(({ record, errors }) => {
-    console.error(`  SKU ${record.sku}:`, errors);
-  });
-
-  // Save failed records for review
-  await saveDataQualityReport(validation.failed);
-}
-
-// Proceed with passed records only
-await processRecords(validation.passed);
-```
-
-### Reconciliation Reports
-
-```typescript
-interface ReconciliationReport {
-  fileKey: string;
-  processedAt: string;
-  totalRecords: number;
-  successfulRecords: number;
-  failedRecords: number;
-  duplicateRecords: number;
-  dataQualityIssues: number;
-  batchesCreated: number;
-  processingTimeMs: number;
-}
-
-async function generateReconciliationReport(
-  fileKey: string,
-  results: ProcessingResult
-): Promise<ReconciliationReport> {
-  return {
-    fileKey,
-    processedAt: new Date().toISOString(),
-    totalRecords: results.totalRecords,
-    successfulRecords: results.successful,
-    failedRecords: results.failed,
-    duplicateRecords: results.duplicates || 0,
-    dataQualityIssues: results.qualityIssues || 0,
-    batchesCreated: results.batches,
-    processingTimeMs: results.processingTime,
-  };
-}
-
-// Save report to S3
-async function saveReconciliationReport(report: ReconciliationReport): Promise<void> {
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_JSON',
-      connectionId: 's3-reports',
-      name: 'S3 Reports',
-      s3Config: s3Config,
-    },
-    logger
-  );
-  const reportKey = `reports/reconciliation/${report.fileKey}-${Date.now()}.json`;
-
-  await s3.uploadFile(reportKey, JSON.stringify(report, null, 2));
-
-  console.log(`Reconciliation report saved: ${reportKey}`);
-}
-```
-
-## Security Best Practices
-
-### Credential Management
-
-```typescript
-// ✅ CORRECT - Use environment variables
-const config = {
-  fluent: {
-    baseUrl: process.env.FLUENT_BASE_URL!,
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-  s3: {
-    region: process.env.AWS_REGION!,
-    credentials: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    },
-  },
-};
-
-// ❌ WRONG - Never hardcode credentials
-const badConfig = {
-  fluent: {
-    clientId: 'hardcoded-client-id', // ❌ Don't do this
-    clientSecret: 'hardcoded-secret', // ❌ Security risk
-  },
-};
-```
-
-### Secrets Manager Integration
-
-```typescript
-import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
-
-async function loadSecretsFromAWS(): Promise<any> {
-  const client = new SecretsManagerClient({ region: 'us-east-1' });
-
-  const command = new GetSecretValueCommand({
-    SecretId: 'fluent-commerce/production',
-  });
-
-  const response = await client.send(command);
-  return JSON.parse(response.SecretString!);
-}
-
-// Usage
-const secrets = await loadSecretsFromAWS();
-
-const client = await createClient({
-  config: {
-    baseUrl: secrets.FLUENT_BASE_URL,
-    clientId: secrets.FLUENT_CLIENT_ID,
-    clientSecret: secrets.FLUENT_CLIENT_SECRET,
-    retailerId: secrets.FLUENT_RETAILER_ID,
-  }
-});
-```
-
-### Data Encryption
-
-```typescript
-// Encrypt sensitive data before storing
-import * as crypto from 'crypto';
-
-function encryptSensitiveData(data: string, key: string): string {
-  const iv = crypto.randomBytes(16);
-  const cipher = crypto.createCipheriv('aes-256-cbc', Buffer.from(key), iv);
-
-  let encrypted = cipher.update(data, 'utf8', 'hex');
-  encrypted += cipher.final('hex');
-
-  return iv.toString('hex') + ':' + encrypted;
-}
-
-function decryptSensitiveData(encrypted: string, key: string): string {
-  const parts = encrypted.split(':');
-  const iv = Buffer.from(parts[0], 'hex');
-  const encryptedData = parts[1];
-
-  const decipher = crypto.createDecipheriv('aes-256-cbc', Buffer.from(key), iv);
-
-  let decrypted = decipher.update(encryptedData, 'hex', 'utf8');
-  decrypted += decipher.final('utf8');
-
-  return decrypted;
-}
-```
-
-### Audit Logging
-
-```typescript
-interface AuditLog {
-  timestamp: string;
-  operation: string;
-  user: string;
-  resource: string;
-  status: 'success' | 'failure';
-  details: Record<string, any>;
-}
-
-async function logAuditEvent(event: Omit<AuditLog, 'timestamp'>): Promise<void> {
-  const auditLog: AuditLog = {
-    timestamp: new Date().toISOString(),
-    ...event,
-  };
-
-  // Log to CloudWatch, Datadog, or custom logging service
-  console.log('[AUDIT]', JSON.stringify(auditLog));
-
-  // Optionally save to S3 for long-term retention
-  await saveAuditLog(auditLog);
-}
-
-// Usage
-await logAuditEvent({
-  operation: 'INVENTORY_INGESTION',
-  user: process.env.USER || 'system',
-  resource: `s3://bucket/file.csv`,
-  status: 'success',
-  details: {
-    recordsProcessed: 1000,
-    jobId: 'job-123',
-    duration: 5000,
-  },
-});
-```
-
-## Monitoring and Alerting
-
-### Metrics Tracking
-
-```typescript
-interface IngestionMetrics {
-  timestamp: string;
-  filesProcessed: number;
-  recordsIngested: number;
-  failedRecords: number;
-  processingTimeMs: number;
-  avgRecordsPerSecond: number;
-  errorRate: number;
-}
-
-class MetricsCollector {
-  private metrics: IngestionMetrics[] = [];
-
-  recordIngestion(result: IngestionResult): void {
-    const metrics: IngestionMetrics = {
-      timestamp: new Date().toISOString(),
-      filesProcessed: result.filesProcessed,
-      recordsIngested: result.recordsIngested,
-      failedRecords: result.failedRecords,
-      processingTimeMs: result.processingTimeMs,
-      avgRecordsPerSecond: result.recordsIngested / (result.processingTimeMs / 1000),
-      errorRate: result.failedRecords / result.recordsIngested
-    };
-
-    this.metrics.push(metrics);
-
-    // Alert on high error rate
-    if (metrics.errorRate > 0.05) { // > 5% error rate
-      this.sendAlert('High error rate detected', metrics);
-    }
-
-    // Alert on slow processing
-    if (metrics.avgRecordsPerSecond < 10) {
-      this.sendAlert('Slow processing speed', metrics);
-    }
-  }
-
-  private sendAlert(message: string, metrics: IngestionMetrics): void {
-    console.error(\`[ALERT] \${message}\`, metrics);
-    // Send to monitoring service (e.g., Datadog, New Relic)
-  }
-}
-```
-
-### Health Checks
-
-```typescript
-async function performHealthCheck(): Promise<HealthCheckResult> {
-  const checks = [];
-
-  // Check Fluent API connectivity
-  try {
-    await client.graphql({
-      query: '{ __typename }'
-    });
-    checks.push({ name: 'Fluent API', status: 'healthy' });
-  } catch (error) {
-    checks.push({ name: 'Fluent API', status: 'unhealthy', error: error.message });
-  }
-
-  // Check S3 connectivity
-  try {
-    await s3.listFiles({ prefix: 'health-check/', maxKeys: 1 });
-    checks.push({ name: 'S3', status: 'healthy' });
-  } catch (error) {
-    checks.push({ name: 'S3', status: 'unhealthy', error: error.message });
-  }
-
-  // Check state storage
-  try {
-    await state.set('health-check', Date.now());
-    await state.get('health-check');
-    checks.push({ name: 'State Storage', status: 'healthy' });
-  } catch (error) {
-    checks.push({ name: 'State Storage', status: 'unhealthy', error: error.message });
-  }
-
-  const allHealthy = checks.every(c => c.status === 'healthy');
-
-  return {
-    status: allHealthy ? 'healthy' : 'degraded',
-    checks,
-    timestamp: new Date().toISOString(),
-  };
-}
-```
-
-## Complete Production Pipeline Example
-
-Putting all best practices together into a production-ready ingestion pipeline:
-
-```typescript
-import {
-  createClient,
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  StateService,
-  VersoriKVAdapter,
-  FluentClient,
-  FileParsingError,
-  MappingError,
-  BatchAPIError,
-} from '@fluentcommerce/fc-connect-sdk';
-import * as winston from 'winston';
-
-/**
- * Production-grade ingestion pipeline with all best practices
- * - Error handling with retry and circuit breaker
- * - Comprehensive monitoring and logging
- * - Security best practices
- * - Data quality validation
- * - State management
- */
-class ProductionIngestionPipeline {
-  private client: FluentClient;
-  private s3: S3DataSource;
-  private parser: CSVParserService;
-  private mapper: UniversalMapper;
-  private state: StateService;
-  private kv: KVStore;
-  private logger: winston.Logger;
-  private circuitBreaker: CircuitBreaker;
-  private dlq: DeadLetterQueue;
-  private metrics: MetricsCollector;
-  private qualityValidator: DataQualityValidator;
-
-  constructor(private config: ProductionConfig) {}
-
-  async initialize(): Promise<void> {
-    // Setup structured logging
-    this.logger = winston.createLogger({
-      level: 'info',
-      format: winston.format.combine(winston.format.timestamp(), winston.format.json()),
-      transports: [
-        new winston.transports.Console(),
-        new winston.transports.File({ filename: 'ingestion-error.log', level: 'error' }),
-        new winston.transports.File({ filename: 'ingestion-combined.log' }),
-      ],
-    });
-
-    this.logger.info('Initializing production ingestion pipeline');
-
-    // Initialize Fluent client
-    this.client = await createClient({ config: this.config.fluent });
-
-    // Initialize data sources
-    this.s3 = new S3DataSource(
-      {
-        type: 'S3_CSV',
-        connectionId: 's3-production',
-        name: 'S3 Production',
-        s3Config: this.config.s3,
-      },
-      this.logger
-    );
-    this.parser = new CSVParserService();
-
-    // Initialize field mapper with custom resolvers
-    this.mapper = new UniversalMapper(this.config.mapping, {
-      customResolvers: this.config.customResolvers || {},
-    });
-
-    // Initialize state management
-    const logger = toStructuredLogger(this.logger, {
-      service: 'production-pipeline',
-      correlationId: generateCorrelationId()
-    });
-
-    this.kv = new VersoriKVAdapter(this.config.kv);
-    this.stateService = new StateService(logger);
-
-    // Initialize circuit breaker
-    this.circuitBreaker = new CircuitBreaker({
-      failureThreshold: 5,
-      resetTimeoutMs: 60000,
-      monitoringWindowMs: 120000,
-    });
-
-    // Initialize dead letter queue
-    this.dlq = new DeadLetterQueue(this.s3, this.config.dlqBucket, 'dlq/');
-
-    // Initialize metrics collector
-    this.metrics = new MetricsCollector();
-
-    // Initialize data quality validator
-    this.qualityValidator = this.createQualityValidator();
-
-    this.logger.info('Pipeline initialization complete');
-  }
-
-  /**
-   * Run the complete ingestion pipeline
-   */
-  async run(bucket: string, prefix: string): Promise<ExecutionReport> {
-    const startTime = Date.now();
-
-    try {
-      this.logger.info('Starting ingestion pipeline', { bucket, prefix });
-
-      // Health check before starting
-      const healthCheck = await this.performHealthCheck();
-      if (healthCheck.status !== 'healthy') {
-        throw new Error(`System unhealthy: ${JSON.stringify(healthCheck.checks)}`);
-      }
-
-      // List files
-      const allFiles = await this.s3.listFiles({ prefix });
-      this.logger.info(`Found ${allFiles.length} total files`);
-
-      // Filter unprocessed files
-      const unprocessedFiles = await this.filterUnprocessedFiles(allFiles);
-      this.logger.info(`${unprocessedFiles.length} files to process`);
-
-      if (unprocessedFiles.length === 0) {
-        return this.createReport(startTime, { filesProcessed: 0 });
-      }
-
-      // Process files with circuit breaker
-      const results = await this.circuitBreaker.execute(async () => {
-        return await this.processFilesWithRetry(bucket, unprocessedFiles);
-      });
-
-      // Generate report
-      const report = this.createReport(startTime, results);
-
-      // Log metrics
-      this.metrics.recordIngestion({
-        filesProcessed: results.filesProcessed,
-        recordsIngested: results.recordsIngested,
-        failedRecords: results.failedRecords,
-        processingTimeMs: report.totalTimeMs,
-        avgRecordsPerSecond: report.throughput,
-        errorRate: results.failedRecords / (results.recordsIngested || 1),
-      });
-
-      this.logger.info('Pipeline execution complete', report);
-
-      return report;
-    } catch (error) {
-      this.logger.error('Pipeline execution failed', {
-        error: error.message,
-        stack: error.stack,
-      });
-
-      await this.sendAlert('CRITICAL: Ingestion pipeline failure', error);
-      throw error;
-    }
-  }
-
-  /**
-   * Process files with retry logic
-   */
-  private async processFilesWithRetry(
-    bucket: string,
-    files: Array<{ key: string }>
-  ): Promise<ProcessingResults> {
-    const results: ProcessingResults = {
-      filesProcessed: 0,
-      filesFailed: 0,
-      recordsIngested: 0,
-      failedRecords: 0,
-      details: [],
-    };
-
-    for (const file of files) {
-      const maxRetries = 3;
-      let attempt = 0;
-      let success = false;
-
-      while (attempt < maxRetries && !success) {
-        attempt++;
-
-        try {
-          const fileResult = await this.processFile(bucket, file.path);
-
-          results.filesProcessed++;
-          results.recordsIngested += fileResult.recordsProcessed;
-          results.details.push({
-            file: file.path,
-            status: 'success',
-            records: fileResult.recordsProcessed,
-          });
-
-          success = true;
-
-          this.logger.info(`File processed successfully`, {
-            file: file.path,
-            records: fileResult.recordsProcessed,
-            attempt,
-          });
-        } catch (error) {
-          this.logger.warn(`File processing attempt ${attempt} failed`, {
-            file: file.path,
-            error: error.message,
-          });
-
-          if (attempt < maxRetries && this.isRetryable(error)) {
-            const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
-            await new Promise(resolve => setTimeout(resolve, delay));
-          } else {
-            results.filesFailed++;
-            results.details.push({
-              file: file.path,
-              status: 'failed',
-              error: error.message,
-            });
-
-            this.logger.error(`File processing failed permanently`, {
-              file: file.path,
-              attempts: attempt,
-              error: error.message,
-            });
-          }
-        }
-      }
-    }
-
-    return results;
-  }
-
-  /**
-   * Process a single file with all validations
-   */
-  private async processFile(bucket: string, fileKey: string): Promise<FileProcessingResult> {
-    this.logger.info(`Processing file: ${fileKey}`);
-
-    try {
-      // Read file
-      const fileContent = await this.s3.downloadFile(fileKey);
-
-      // Parse CSV
-      const records = await this.parser.parse(fileContent);
-      this.logger.debug(`Parsed ${records.length} records from ${fileKey}`);
-
-      // Data quality validation
-      const validation = this.qualityValidator.validate(records);
-
-      if (validation.failed.length > 0) {
-        this.logger.warn(`Data quality issues found`, {
-          file: fileKey,
-          failed: validation.failed.length,
-          warnings: validation.warnings.length,
-        });
-
-        // Save failed records to DLQ
-        for (const { record, errors } of validation.failed) {
-          await this.dlq.save({
-            originalFile: fileKey,
-            failedAt: new Date().toISOString(),
-            errorType: 'DATA_QUALITY_ERROR',
-            errorMessage: errors.join('; '),
-            recordData: record,
-            retryCount: 0,
-            metadata: {},
-          });
-        }
-      }
-
-      // Proceed with passed records only
-      if (validation.passed.length === 0) {
-        throw new Error('No valid records after quality validation');
-      }
-
-      // Field mapping
-      const mappingResult = await this.mapper.map(validation.passed);
-
-      if (!mappingResult.success) {
-        throw new MappingError(`Field mapping failed: ${mappingResult.errors.join(', ')}`);
-      }
-
-      // Create job
-      const job = await this.client.createJob({
-        name: `Production Import - ${fileKey}`,
-        retailerId: this.config.fluent.retailerId,
-        metadata: {
-          fileName: fileKey,
-          recordCount: mappingResult.data.length,
-          pipeline: 'production',
-        },
-      });
-
-      // Send to Batch API
-      await this.client.sendBatch(job.id, {
-        action: 'UPSERT',
-        entityType: 'INVENTORY',
-        entities: mappingResult.data,
-      });
-
-      // Mark as processed
-      await this.state.markFileProcessed(fileKey, {
-        jobId: job.id,
-        recordCount: mappingResult.data.length,
-        timestamp: new Date().toISOString(),
-      });
-
-      // Audit log
-      await this.logAuditEvent({
-        operation: 'INVENTORY_INGESTION',
-        user: 'system',
-        resource: `s3://${bucket}/${fileKey}`,
-        status: 'success',
-        details: {
-          recordsProcessed: mappingResult.data.length,
-          jobId: job.id,
-        },
-      });
-
-      return {
-        recordsProcessed: mappingResult.data.length,
-        jobId: job.id,
-      };
-    } catch (error) {
-      // Audit log failure
-      await this.logAuditEvent({
-        operation: 'INVENTORY_INGESTION',
-        user: 'system',
-        resource: `s3://${bucket}/${fileKey}`,
-        status: 'failure',
-        details: {
-          error: error.message,
-        },
-      });
-
-      throw error;
-    }
-  }
-
-  /**
-   * Check if error is retryable
-   */
-  private isRetryable(error: any): boolean {
-    if (error instanceof FileParsingError) {
-      return false; // File format errors are not retryable
-    }
-
-    if (error instanceof BatchAPIError) {
-      return error.isRetryable;
-    }
-
-    // Network errors are retryable
-    return (
-      error.code === 'NETWORK_ERROR' ||
-      error.message.includes('timeout') ||
-      error.message.includes('ECONNRESET')
-    );
-  }
-
-  /**
-   * Filter unprocessed files
-   */
-  private async filterUnprocessedFiles(
-    files: FileMetadata[]
-  ): Promise<FileMetadata[]> {
-    const unprocessed = [];
-
-    for (const file of files) {
-      if (!(await this.state.isFileProcessed(this.kv, file.path))) {
-        unprocessed.push(file);
-      }
-    }
-
-    return unprocessed;
-  }
-
-  /**
-   * Create data quality validator
-   */
-  private createQualityValidator(): DataQualityValidator {
-    const validator = new DataQualityValidator();
-
-    // Add validation rules based on configuration
-    validator.addRule({
-      name: 'Required Fields',
-      severity: 'error',
-      validate: record => {
-        const required = this.config.requiredFields || [];
-        const missing = required.filter(field => !record[field]);
-
-        if (missing.length > 0) {
-          return {
-            isValid: false,
-            error: `Missing required fields: ${missing.join(', ')}`,
-          };
-        }
-
-        return { isValid: true };
-      },
-    });
-
-    return validator;
-  }
-
-  /**
-   * Perform system health check
-   */
-  private async performHealthCheck(): Promise<HealthCheckResult> {
-    const checks = [];
-
-    // Check Fluent API
-    try {
-      await this.client.graphql({
-        query: '{ __typename }'
-      });
-      checks.push({ name: 'Fluent API', status: 'healthy' });
-    } catch (error) {
-      checks.push({ name: 'Fluent API', status: 'unhealthy', error: error.message });
-    }
-
-    // Check S3
-    try {
-      await this.s3.listObjects(this.config.s3Bucket, 'health-check/');
-      checks.push({ name: 'S3', status: 'healthy' });
-    } catch (error) {
-      checks.push({ name: 'S3', status: 'unhealthy', error: error.message });
-    }
-
-    const allHealthy = checks.every(c => c.status === 'healthy');
-
-    return {
-      status: allHealthy ? 'healthy' : 'degraded',
-      checks,
-      timestamp: new Date().toISOString(),
-    };
-  }
-
-  /**
-   * Create execution report
-   */
-  private createReport(startTime: number, results: Partial<ProcessingResults>): ExecutionReport {
-    const totalTimeMs = Date.now() - startTime;
-
-    return {
-      startTime: new Date(startTime).toISOString(),
-      endTime: new Date().toISOString(),
-      totalTimeMs,
-      filesProcessed: results.filesProcessed || 0,
-      filesFailed: results.filesFailed || 0,
-      recordsIngested: results.recordsIngested || 0,
-      failedRecords: results.failedRecords || 0,
-      throughput: (results.recordsIngested || 0) / (totalTimeMs / 1000),
-      successRate:
-        ((results.filesProcessed || 0) /
-          ((results.filesProcessed || 0) + (results.filesFailed || 0))) *
-        100,
-    };
-  }
-
-  /**
-   * Log audit event
-   */
-  private async logAuditEvent(event: Omit<AuditLog, 'timestamp'>): Promise<void> {
-    const auditLog: AuditLog = {
-      timestamp: new Date().toISOString(),
-      ...event,
-    };
-
-    this.logger.info('[AUDIT]', auditLog);
-
-    // Optionally save to S3 for compliance
-    // await this.s3.putObject(auditBucket, auditKey, JSON.stringify(auditLog));
-  }
-
-  /**
-   * Send alert to monitoring service
-   */
-  private async sendAlert(message: string, error: any): Promise<void> {
-    this.logger.error('[ALERT]', { message, error: error.message });
-
-    // Integration with alerting service (PagerDuty, Slack, etc.)
-    // await alertService.send({ message, error });
-  }
-}
-
-// Usage
-const pipeline = new ProductionIngestionPipeline({
-  fluent: {
-    baseUrl: process.env.FLUENT_BASE_URL!,
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-  s3: {
-    region: process.env.AWS_REGION!,
-    credentials: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    },
-  },
-  s3Bucket: 'inventory-bucket',
-  dlqBucket: 'inventory-dlq-bucket',
-  mapping: {
-    fields: {
-      ref: { source: 'sku', required: true },
-      productRef: { source: 'product_id', required: true },
-      locationRef: { source: 'warehouse_code', required: true },
-      qty: { source: 'quantity', resolver: 'sdk.parseInt', required: true },
-      type: { source: 'inventory_type', default: 'ON_HAND' },
-      status: { source: 'status', default: 'AVAILABLE' },
-    },
-  },
-  requiredFields: ['sku', 'warehouse', 'quantity'],
-  kv: openKv(),
-});
-
-await pipeline.initialize();
-const report = await pipeline.run('inventory-bucket', 'data/');
-
-console.log('Ingestion complete:', report);
-```
-
-## Common Errors and Solutions
-
-### Error: JOB_EXPIRED
-
-**Cause:** Job exceeded 24-hour lifetime
-
-**Solution:**
-
-```typescript
-async function handleExpiredJob(originalJobId: string, remainingBatches: any[]) {
-  // Create new job
-  const newJob = await client.createJob({
-    name: \`Recovery - \${new Date().toISOString()}\`,
-    retailerId: process.env.FLUENT_RETAILER_ID!,
-    metadata: {
-      originalJobId,
-      reason: 'JOB_EXPIRED'
-    }
-  });
-
-  // Resend remaining batches
-  for (const batch of remainingBatches) {
-    await client.sendBatch(newJob.id, batch);
-  }
-
-  return newJob.id;
-}
-```
-
-### Error: VALIDATION_ERROR
-
-**Cause:** Invalid field values or missing required fields
-
-**Solution:**
-
-```typescript
-async function handleValidationError(records: any[], error: any) {
-  // Parse validation errors
-  const invalidRecords = extractInvalidRecords(error);
-
-  // Filter out invalid records
-  const validRecords = records.filter(r => !invalidRecords.includes(r));
-
-  // Log invalid records
-  console.error('Invalid records:', invalidRecords);
-  await saveToErrorLog(invalidRecords);
-
-  // Retry with valid records only
-  return validRecords;
-}
-```
-
-### Error: RATE_LIMIT_ERROR
-
-**Cause:** Too many requests to Fluent API
-
-**Solution:**
-
-```typescript
-class RateLimitHandler {
-  async handleRateLimit(operation: () => Promise<any>): Promise<any> {
-    try {
-      return await operation();
-    } catch (error) {
-      if (error.code === 'RATE_LIMIT_ERROR') {
-        const retryAfter = error.retryAfter || 60000; // Default 60s
-        console.log(\`Rate limited. Waiting \${retryAfter}ms\`);
-        await sleep(retryAfter);
-        return await operation(); // Retry once
-      }
-      throw error;
-    }
-  }
-}
-```
-
-## Production Checklist
-
-### Before Deployment
-
-- [ ] Schema validated against Fluent Commerce API
-- [ ] Field mappings tested with sample data
-- [ ] Error handling implemented for all operations
-- [ ] State management configured and tested
-- [ ] Monitoring and alerting configured
-- [ ] Health checks implemented
-- [ ] Rate limiting configured
-- [ ] Dead letter queue configured
-- [ ] Rollback plan documented
-
-### During Deployment
-
-- [ ] Start with small batch sizes (100-500)
-- [ ] Monitor error rates closely
-- [ ] Test with sample files first
-- [ ] Gradually increase batch sizes
-- [ ] Verify state management works correctly
-
-### After Deployment
-
-- [ ] Monitor ingestion metrics daily
-- [ ] Review error logs regularly
-- [ ] Track processing times
-- [ ] Optimize based on metrics
-- [ ] Document common issues and solutions
-
-## Debugging Tips
-
-### Enable Debug Logging
-
-```typescript
-const client = await createClient({
-  config: {
-    ...config
-  },
-  logger: {
-    debug: (...args) => console.log('[DEBUG]', ...args),
-    info: (...args) => console.log('[INFO]', ...args),
-    warn: (...args) => console.warn('[WARN]', ...args),
-    error: (...args) => console.error('[ERROR]', ...args),
-  },
-});
-```
-
-### Trace File Processing
-
-```typescript
-async function traceFileProcessing(fileKey: string) {
-  console.log(\`[TRACE] Starting processing: \${fileKey}\`);
-
-  try {
-    console.log('[TRACE] Reading file from S3');
-    const data = await s3.downloadFile(fileKey);
-    console.log(\`[TRACE] File size: \${data.length} bytes\`);
-
-    console.log('[TRACE] Parsing CSV');
-    const records = await parser.parse(data);
-    console.log(\`[TRACE] Parsed \${records.length} records\`);
-
-    console.log('[TRACE] Mapping fields');
-    const result = await mapper.map(records);
-    console.log(\`[TRACE] Mapped \${result.data.length} valid records\`);
-
-    console.log('[TRACE] Creating job');
-    const job = await client.createJob({ name: \`Import - \${fileKey}\` });
-    console.log(\`[TRACE] Job created: \${job.id}\`);
-
-    console.log('[TRACE] Sending batch');
-    await client.sendBatch(job.id, { entities: result.data });
-    console.log('[TRACE] Batch sent successfully');
-
-  } catch (error) {
-    console.error(\`[TRACE] Error at: \${error.message}\`);
-    throw error;
-  }
-}
-```
-
-## Key Takeaways
-
-- 🎯 **Always validate** - Check schema and data before sending
-- 🎯 **Handle errors gracefully** - Retry transient failures, log permanent ones
-- 🎯 **Monitor everything** - Track metrics, set up alerts
-- 🎯 **Use state management** - Prevent duplicates, track history
-- 🎯 **Test thoroughly** - Start small, scale gradually
-- 🎯 **Document** - Keep runbooks for common issues
-
-## Congratulations!
-
-You've completed the Data Ingestion Learning Path! You now know:
-
-- ✅ Core ingestion concepts and architecture
-- ✅ How to read from multiple data sources
-- ✅ How to parse CSV, Parquet, XML, and JSON
-- ✅ How to transform data with UniversalMapper
-- ✅ How to use the Batch API effectively
-- ✅ How to implement state management
-- ✅ How to optimize performance
-- ✅ How to build production-ready ingestion workflows
-
-## Next Steps
-
-Congratulations! You've completed the Data Ingestion Learning Path and mastered production-ready ingestion patterns.
-
-**Continue Learning:**
-
-- 📖 [Data Extraction Guide](../../extraction/) - Learn reverse workflows (Fluent → S3/Parquet/CSV)
-- 📖 [Resolver Development Guide](../../mapping/resolvers/mapping-resolvers-readme.md) - Build custom data transformations
-- 📖 [Universal Mapping Guide](../../mapping/mapping-readme.md) - Master field mapping patterns
-- 📖 [Error Handling Guide](../../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md) - Deep dive into error patterns
-- 📖 [Versori Platform Integration](../../../04-REFERENCE/platforms/versori/) - Deploy to production
-
-**Real-World Examples:**
-
-- 🔍 [Complete Connector Examples](../../../01-TEMPLATES/versori/workflows/readme.md) - Production Versori connectors
-- 📋 [Use Case Library](../../../01-TEMPLATES/readme.md) - Business-specific implementations
-- 🛠️ [CLI Tools](../../../../bin/) - Code generation and validation utilities
-
-**Resources:**
-
-- 📚 [Complete API Reference](../../api-reference/api-reference-readme.md)
-- 📖 [Quick Reference Cheat Sheet](../ingestion-quick-reference.md)
-- 🐛 [Troubleshooting Guide](../../../00-START-HERE/troubleshooting-quick-reference.md)
-
----
-
-[← Back to Ingestion Guide](../ingestion-readme.md) | [Previous: Module 8 - Performance Optimization](./02-core-guides-ingestion-08-performance-optimization.md)
-
-**Need Help?**
-
-- 📧 Email: support@fluentcommerce.com
-- 📚 Documentation: [FC Connect SDK Docs](../../../)
-- 🔗 GitHub: Report issues or contribute
+# Module 9: Best Practices
+
+[← Back to Ingestion Guide](../ingestion-readme.md)
+
+**Module 9 of 9** | **Level**: All Levels | **Time**: 30 minutes
+
+---
+
+## Overview
+
+This module covers production-ready patterns for data ingestion workflows. Learn comprehensive error handling strategies, monitoring approaches, security best practices, scheduling patterns, data quality validation, and complete production deployment patterns.
+
+New in this version:
+
+- Preflight validation before runs (PreflightValidator)
+- Job lifecycle tracking (JobTracker)
+- Partial batch failure recovery (PartialBatchRecovery)
+- Versori file-level deduplication (VersoriFileTracker)
+
+## Learning Objectives
+
+By the end of this module, you will:
+
+- ✅ Implement robust error handling with retry logic and circuit breakers
+- ✅ Set up comprehensive monitoring, logging, and alerting
+- ✅ Apply security best practices for credentials and data
+- ✅ Design effective scheduling and trigger strategies
+- ✅ Validate data quality before ingestion
+- ✅ Build production-ready ingestion pipelines
+- ✅ Troubleshoot common issues effectively
+- ✅ Follow deployment checklists for success
+
+---
+
+## Error Handling
+
+### Retry Strategy with Exponential Backoff
+
+```typescript
+async function retryWithBackoff<T>(
+  operation: () => Promise<T>,
+  maxRetries: number = 3
+): Promise<T> {
+  let lastError: Error;
+  let delay = 1000; // Start with 1 second
+
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await operation();
+    } catch (error) {
+      lastError = error;
+
+      if (!isRetryable(error) || attempt === maxRetries) {
+        throw error;
+      }
+
+      console.log(\`Retry \${attempt}/\${maxRetries} after \${delay}ms\`);
+      await sleep(delay);
+      delay *= 2; // Exponential backoff
+    }
+  }
+
+  throw lastError;
+}
+
+function isRetryable(error: any): boolean {
+  const retryableCodes = [
+    'RATE_LIMIT_ERROR',
+    'NETWORK_ERROR',
+    'TIMEOUT_ERROR',
+    'SERVICE_UNAVAILABLE'
+  ];
+
+  return (
+    retryableCodes.includes(error.code) ||
+    error.message.includes('timeout') ||
+    error.message.includes('ECONNRESET')
+  );
+}
+```
+
+### SDK-Specific Error Handling
+
+The SDK throws specific error types that require different handling strategies:
+
+```typescript
+import {
+  FileParsingError,
+  MappingError,
+  BatchAPIError,
+  StateError,
+  createConsoleLogger,
+  toStructuredLogger,
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function handleSDKErrors(fileKey: string) {
+  try {
+    await processFile(fileKey);
+  } catch (error) {
+    if (error instanceof FileParsingError) {
+      // File format issues - log and skip
+      console.error(`Invalid file format: ${fileKey}`, {
+        line: error.line,
+        column: error.column,
+        message: error.message,
+      });
+
+      // Move to error bucket for manual review
+      await moveToErrorBucket(fileKey, error);
+      return { status: 'skipped', reason: 'parsing_error' };
+    } else if (error instanceof MappingError) {
+      // Field mapping failures - log details
+      console.error(`Field mapping failed: ${fileKey}`, {
+        invalidFields: error.invalidFields,
+        recordIndex: error.recordIndex,
+        errors: error.errors,
+      });
+
+      // Save error report
+      await saveErrorReport(fileKey, error);
+      return { status: 'failed', reason: 'mapping_error' };
+    } else if (error instanceof BatchAPIError) {
+      // Batch API rejections - check if retryable
+      if (error.isRetryable) {
+        console.warn(`Retryable batch error: ${error.code}`);
+        throw error; // Will be caught by retry logic
+      } else {
+        console.error(`Permanent batch error: ${error.code}`, {
+          jobId: error.jobId,
+          batchId: error.batchId,
+          details: error.details,
+        });
+
+        // Save to dead letter queue
+        await saveToDLQ(fileKey, error);
+        return { status: 'failed', reason: 'batch_api_error' };
+      }
+    } else if (error instanceof StateError) {
+      // State management issues - critical
+      console.error(`State management error: ${error.message}`);
+
+      // Alert operations team
+      await sendAlert('CRITICAL: State management failure', error);
+      throw error; // Don't continue if state is broken
+    } else {
+      // Unknown error - log and alert
+      console.error(`Unexpected error processing ${fileKey}:`, error);
+      await sendAlert('Unknown ingestion error', error);
+      throw error;
+    }
+  }
+}
+```
+
+### Circuit Breaker Pattern
+
+Prevent cascading failures when external services are down:
+
+```typescript
+interface CircuitBreakerConfig {
+  failureThreshold: number;
+  resetTimeoutMs: number;
+  monitoringWindowMs: number;
+}
+
+enum CircuitState {
+  CLOSED = 'CLOSED', // Normal operation
+  OPEN = 'OPEN', // Blocking requests
+  HALF_OPEN = 'HALF_OPEN', // Testing recovery
+}
+
+class CircuitBreaker {
+  private state: CircuitState = CircuitState.CLOSED;
+  private failureCount: number = 0;
+  private lastFailureTime: number = 0;
+  private failures: number[] = [];
+
+  constructor(private config: CircuitBreakerConfig) {}
+
+  async execute<T>(operation: () => Promise<T>): Promise<T> {
+    // Check if circuit should reset
+    if (this.shouldAttemptReset()) {
+      this.state = CircuitState.HALF_OPEN;
+    }
+
+    // Block if circuit is open
+    if (this.state === CircuitState.OPEN) {
+      throw new Error(
+        `Circuit breaker OPEN. Last failure: ${new Date(this.lastFailureTime).toISOString()}`
+      );
+    }
+
+    try {
+      const result = await operation();
+
+      // Success - reset if in half-open state
+      if (this.state === CircuitState.HALF_OPEN) {
+        console.log('Circuit breaker: Service recovered, closing circuit');
+        this.reset();
+      }
+
+      return result;
+    } catch (error) {
+      this.recordFailure();
+
+      // Open circuit if threshold exceeded
+      if (this.failureCount >= this.config.failureThreshold) {
+        this.state = CircuitState.OPEN;
+        this.lastFailureTime = Date.now();
+        console.error(`Circuit breaker OPENED after ${this.failureCount} failures`);
+      }
+
+      throw error;
+    }
+  }
+
+  private recordFailure(): void {
+    const now = Date.now();
+    this.failures.push(now);
+
+    // Remove old failures outside monitoring window
+    this.failures = this.failures.filter(time => now - time < this.config.monitoringWindowMs);
+
+    this.failureCount = this.failures.length;
+  }
+
+  private shouldAttemptReset(): boolean {
+    if (this.state !== CircuitState.OPEN) {
+      return false;
+    }
+
+    const timeSinceLastFailure = Date.now() - this.lastFailureTime;
+    return timeSinceLastFailure >= this.config.resetTimeoutMs;
+  }
+
+  private reset(): void {
+    this.state = CircuitState.CLOSED;
+    this.failureCount = 0;
+    this.failures = [];
+  }
+
+  getState(): CircuitState {
+    return this.state;
+  }
+}
+
+// Usage
+const fluentAPICircuit = new CircuitBreaker({
+  failureThreshold: 5, // Open after 5 failures
+  resetTimeoutMs: 60000, // Try again after 1 minute
+  monitoringWindowMs: 120000, // Track failures over 2 minutes
+});
+
+async function processWithCircuitBreaker(fileKey: string) {
+  try {
+    await fluentAPICircuit.execute(async () => {
+      return await processFile(fileKey);
+    });
+  } catch (error) {
+    if (error.message.includes('Circuit breaker OPEN')) {
+      console.log('Service temporarily unavailable, requeueing file');
+      await requeueFile(fileKey);
+    } else {
+      throw error;
+    }
+  }
+}
+```
+
+### Dead Letter Queue Pattern
+
+Store failed records for later processing:
+
+```typescript
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+
+interface DLQRecord {
+  originalFile: string;
+  failedAt: string;
+  errorType: string;
+  errorMessage: string;
+  recordData: any;
+  retryCount: number;
+  metadata: Record<string, any>;
+}
+
+class DeadLetterQueue {
+  constructor(
+    private s3: S3DataSource,
+    private dlqBucket: string,
+    private dlqPrefix: string = 'dlq/'
+  ) {}
+
+  /**
+   * Save failed record to DLQ
+   */
+  async save(record: DLQRecord): Promise<void> {
+    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+    const key = `${this.dlqPrefix}${record.errorType}/${timestamp}-${record.originalFile}`;
+
+    await this.s3.uploadFile(key, JSON.stringify(record, null, 2));
+
+    console.log(`Saved to DLQ: ${key}`);
+  }
+
+  /**
+   * Retry processing records from DLQ
+   */
+  async retryAll(
+    processor: (record: any) => Promise<void>
+  ): Promise<{ successful: number; failed: number }> {
+    const files = await this.s3.listFiles({ prefix: this.dlqPrefix });
+    let successful = 0;
+    let failed = 0;
+
+    for (const file of files) {
+      try {
+        const content = await this.s3.downloadFile(file.path);
+        const dlqRecord: DLQRecord = JSON.parse(content);
+
+        // Attempt reprocessing
+        await processor(dlqRecord.recordData);
+
+        // Success - delete from DLQ (Note: deleteObject method may not exist - use uploadFile with empty content or check SDK)
+        // await this.s3.deleteObject(this.dlqBucket, file.path);
+        successful++;
+      } catch (error) {
+        console.error(`DLQ retry failed for ${file.path}:`, error);
+        failed++;
+
+        // Update retry count
+        await this.incrementRetryCount(file.path);
+      }
+    }
+
+    return { successful, failed };
+  }
+
+  /**
+   * Increment retry count in DLQ record
+   */
+  private async incrementRetryCount(key: string): Promise<void> {
+    try {
+      const content = await this.s3.downloadFile(key);
+      const record: DLQRecord = JSON.parse(content);
+
+      record.retryCount = (record.retryCount || 0) + 1;
+      record.metadata.lastRetryAttempt = new Date().toISOString();
+
+      await this.s3.uploadFile(key, JSON.stringify(record, null, 2));
+    } catch (error) {
+      console.error(`Failed to update retry count for ${key}:`, error);
+    }
+  }
+}
+
+// Usage
+const dlq = new DeadLetterQueue(s3DataSource, 'my-dlq-bucket');
+
+async function processFileWithDLQ(fileKey: string) {
+  try {
+    const records = await parseFile(fileKey);
+
+    for (const [index, record] of records.entries()) {
+      try {
+        await processRecord(record);
+      } catch (error) {
+        // Save failed record to DLQ
+        await dlq.save({
+          originalFile: fileKey,
+          failedAt: new Date().toISOString(),
+          errorType: error.constructor.name,
+          errorMessage: error.message,
+          recordData: record,
+          retryCount: 0,
+          metadata: {
+            recordIndex: index,
+            totalRecords: records.length,
+          },
+        });
+      }
+    }
+  } catch (error) {
+    console.error(`Complete file failure: ${fileKey}`, error);
+    throw error;
+  }
+}
+```
+
+### Graceful Degradation
+
+```typescript
+async function processFileWithFallback(fileKey: string) {
+  try {
+    // Primary processing path
+    await processFileNormally(fileKey);
+  } catch (error) {
+    console.error(\`Primary processing failed: \${error.message}\`);
+
+    try {
+      // Fallback: Process with reduced batch size
+      await processFileWithSmallerBatches(fileKey);
+    } catch (fallbackError) {
+      console.error(\`Fallback processing failed: \${fallbackError.message}\`);
+
+      // Last resort: Save to dead letter queue
+      await saveToDeadLetterQueue(fileKey, error);
+      throw new Error(\`File processing failed completely: \${fileKey}\`);
+    }
+  }
+}
+```
+
+## Validation Best Practices
+
+### Pre-Flight Validation
+
+```typescript
+async function validateBeforeIngestion(records: any[]): Promise<ValidationResult> {
+  const errors = [];
+  const warnings = [];
+
+  // Schema validation
+  for (const [index, record] of records.entries()) {
+    // Required fields
+    if (!record.ref) {
+      errors.push(\`Row \${index}: Missing required field 'ref'\`);
+    }
+
+    if (!record.productRef) {
+      errors.push(\`Row \${index}: Missing required field 'productRef'\`);
+    }
+
+    if (!record.locationRef) {
+      errors.push(\`Row \${index}: Missing required field 'locationRef'\`);
+    }
+
+    if (record.qty === undefined || record.qty === null) {
+      errors.push(\`Row \${index}: Missing required field 'qty'\`);
+    }
+
+    // Type validation
+    if (typeof record.qty !== 'number') {
+      errors.push(\`Row \${index}: 'qty' must be a number\`);
+    }
+
+    // Business rules
+    if (record.qty < 0) {
+      errors.push(\`Row \${index}: Quantity cannot be negative\`);
+    }
+
+    if (record.qty > 1000000) {
+      warnings.push(\`Row \${index}: Unusually large quantity\`);
+    }
+  }
+
+  return {
+    isValid: errors.length === 0,
+    errors,
+    warnings
+  };
+}
+
+// Usage
+const validation = await validateBeforeIngestion(records);
+
+if (!validation.isValid) {
+  console.error('Validation errors:', validation.errors);
+  throw new Error('Data validation failed');
+}
+
+if (validation.warnings.length > 0) {
+  console.warn('Validation warnings:', validation.warnings);
+}
+```
+
+## Scheduling and Triggers
+
+###Cron Scheduling Patterns
+
+Different scheduling strategies for different ingestion scenarios:
+
+```typescript
+// Daily midnight sync (00:00 UTC)
+const dailyMidnightCron = '0 0 * * *';
+
+// Every 6 hours
+const every6HoursCron = '0 */6 * * *';
+
+// Every hour during business hours (9 AM - 5 PM, Mon-Fri)
+const businessHoursCron = '0 9-17 * * 1-5';
+
+// Every 15 minutes
+const every15MinutesCron = '*/15 * * * *';
+
+// Weekly Sunday at 2 AM
+const weeklySundayCron = '0 2 * * 0';
+```
+
+**Recommended schedules by use case:**
+
+| Use Case                | Schedule      | Cron Expression  | Rationale                    |
+| ----------------------- | ------------- | ---------------- | ---------------------------- |
+| **WMS Daily Sync**      | Once daily    | `0 2 * * *`      | Process overnight updates    |
+| **3PL Cycle Counts**    | Every 6 hours | `0 */6 * * *`    | Keep inventory fresh         |
+| **Real-time Updates**   | Every 15 min  | `*/15 * * * *`   | Near real-time sync          |
+| **Weekly Full Sync**    | Sunday 2 AM   | `0 2 * * 0`      | Comprehensive reconciliation |
+| **Business Hours Only** | 9 AM - 5 PM   | `0 9-17 * * 1-5` | During operational hours     |
+
+### Trigger Mechanisms
+
+Different ways to trigger ingestion workflows:
+
+#### 1. S3 Event Triggers
+
+```typescript
+// Versori workflow triggered by S3 event
+export const s3TriggeredIngestion = webhook('s3-inventory-upload', {
+  response: { mode: 'sync' },
+})
+  .then(
+    fn('parse-s3-event', async ({ data }) => {
+      const event = JSON.parse(data);
+      const bucket = event.Records[0].s3.bucket.name;
+      const key = decodeURIComponent(event.Records[0].s3.object.key);
+
+      return { bucket, key };
+    })
+  )
+  .then(
+    fn('process-file', async ({ data, connections, log, openKv }) => {
+      const client = await createClient({ connections, log, openKv });
+
+      // Initialize logger and state management
+      const logger = toStructuredLogger(log, {
+        service: 'ingestion',
+        correlationId: generateCorrelationId()
+      });
+
+      const stateService = new StateService(logger);
+      const kvAdapter = new VersoriKVAdapter(openKv());
+
+      const s3 = new S3DataSource(
+        {
+          type: 'S3_CSV',
+          connectionId: 's3-triggered',
+          name: 'S3 Triggered Ingestion',
+          s3Config: { region: 'us-east-1', bucket: data.bucket },
+        },
+        logger
+      );
+
+      // Check if already processed
+      if (await stateService.isFileProcessed(kvAdapter, data.key)) {
+        log.info(`File already processed: ${data.key}`);
+        return { status: 'skipped', reason: 'already_processed' };
+      }
+
+      // Process file
+      await processInventoryFile(client, s3, data.bucket, data.key);
+
+      // Mark as processed
+      await stateService.updateSyncState(kvAdapter, [{
+        fileName: data.key,
+        lastModified: new Date().toISOString(),
+      }]);
+
+      return { status: 'success', file: data.key };
+    })
+  )
+  .catch(({ data, log }) => {
+    log.error('S3 triggered ingestion failed:', data);
+    return { status: 'error', error: data };
+  });
+```
+
+#### 2. SFTP Polling
+
+```typescript
+import { SftpDataSource, StateService, VersoriKVAdapter, createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function pollSftpDirectory(kv: any) {
+  const logger = toStructuredLogger(createConsoleLogger(), {
+    service: 'ingestion',
+    correlationId: generateCorrelationId()
+  });
+
+  const stateService = new StateService(logger);
+  const kvAdapter = new VersoriKVAdapter(kv);
+
+  const sftp = new SftpDataSource({
+    type: 'SFTP_CSV',
+    connectionId: 'sftp-polling',
+    name: 'SFTP Polling',
+    settings: {
+      host: process.env.SFTP_HOST!,
+      port: 22,
+      username: process.env.SFTP_USER!,
+      privateKey: fs.readFileSync('/path/to/private/key'),
+    }
+  }, logger);
+
+  const remoteDir = '/inbound/inventory/';
+  const files = await sftp.listFiles(remoteDir);
+
+  for (const file of files) {
+    if (await stateService.isFileProcessed(kvAdapter, file.name)) {
+      continue; // Skip already processed
+    }
+
+    // Download and process
+    const localPath = `/tmp/${file.name}`;
+    await sftp.downloadFile(file.path, localPath); // Use file.path (full path) for download
+
+    await processLocalFile(localPath);
+
+    // Mark as processed
+    await stateService.updateSyncState(kvAdapter, [{
+      fileName: file.name,
+      lastModified: new Date().toISOString(),
+    }]);
+
+    // Optionally move to processed folder
+    await sftp.moveFile(`${remoteDir}${file.name}`, `/processed/${file.name}`);
+  }
+}
+```
+
+#### 3. Webhook Triggers
+
+```typescript
+// Manual or external system triggered ingestion
+export const manualIngestion = webhook('manual-trigger', {
+  response: { mode: 'sync' },
+})
+  .then(
+    fn('validate-request', ({ data }) => {
+      const { bucket, prefix, batchSize = 2000 } = JSON.parse(data);
+
+      if (!bucket || !prefix) {
+        throw new Error('Missing required parameters: bucket, prefix');
+      }
+
+      return { bucket, prefix, batchSize };
+    })
+  )
+  .then(
+    fn('run-ingestion', async ({ data, connections, log }) => {
+      const client = await createClient(ctx); // Auto-detects Versori context
+      const s3 = new S3DataSource(
+        {
+          type: 'S3_CSV',
+          s3Config: { region: 'us-east-1' },
+        },
+        log
+      );
+
+      const files = await s3.listFiles({ prefix: data.prefix });
+
+      log.info(`Found ${files.length} files to process`);
+
+      const results = await processFilesInParallel(client, s3, data.bucket, files, {
+        batchSize: data.batchSize,
+      });
+
+      return {
+        filesProcessed: results.successful,
+        filesFailed: results.failed,
+        totalRecords: results.totalRecords,
+      };
+    })
+  );
+```
+
+### Backfill Strategies
+
+Handle missed files or catch-up processing:
+
+```typescript
+async function backfillMissedFiles(startDate: Date, endDate: Date, kv: any): Promise<void> {
+  const logger = toStructuredLogger(createConsoleLogger(), {
+    service: 'backfill',
+    correlationId: generateCorrelationId()
+  });
+
+  const stateService = new StateService(logger);
+  const kvAdapter = new VersoriKVAdapter(kv);
+
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_CSV',
+      connectionId: 's3-backfill',
+      name: 'S3 Backfill',
+      s3Config: s3Config,
+    },
+    logger
+  );
+
+  console.log(`Backfilling files from ${startDate} to ${endDate}`);
+
+  // List all files in date range
+  const allFiles = await s3.listFiles({ prefix: 'data/' });
+
+  const filesInRange = allFiles.filter(file => {
+    const fileDate = extractDateFromKey(file.path);
+    return fileDate >= startDate && fileDate <= endDate;
+  });
+
+  console.log(`Found ${filesInRange.length} files in date range`);
+
+  // Check processing status
+  const unprocessedFiles = [];
+  for (const file of filesInRange) {
+    if (!(await stateService.isFileProcessed(kvAdapter, file.path))) {
+      unprocessedFiles.push(file);
+    }
+  }
+
+  console.log(`${unprocessedFiles.length} files need processing`);
+
+  // Process with lower concurrency to avoid overwhelming system
+  await processFilesInParallel(unprocessedFiles, 2); // Only 2 concurrent
+}
+
+function extractDateFromKey(key: string): Date {
+  // Extract date from key like "data/inventory-2025-01-15.csv"
+  const match = key.match(/(\d{4}-\d{2}-\d{2})/);
+  return match ? new Date(match[1]) : new Date(0);
+}
+```
+
+## Data Quality and Validation
+
+### Pre-Ingestion Data Quality Checks
+
+```typescript
+interface DataQualityRule {
+  name: string;
+  validate: (record: any) => { isValid: boolean; error?: string };
+  severity: 'error' | 'warning';
+}
+
+class DataQualityValidator {
+  private rules: DataQualityRule[] = [];
+
+  addRule(rule: DataQualityRule): void {
+    this.rules.push(rule);
+  }
+
+  validate(records: any[]): {
+    passed: any[];
+    failed: Array<{ record: any; errors: string[] }>;
+    warnings: Array<{ record: any; warnings: string[] }>;
+  } {
+    const passed: any[] = [];
+    const failed: Array<{ record: any; errors: string[] }> = [];
+    const warnings: Array<{ record: any; warnings: string[] }> = [];
+
+    for (const record of records) {
+      const errors: string[] = [];
+      const warns: string[] = [];
+
+      for (const rule of this.rules) {
+        const result = rule.validate(record);
+
+        if (!result.isValid) {
+          if (rule.severity === 'error') {
+            errors.push(`${rule.name}: ${result.error}`);
+          } else {
+            warns.push(`${rule.name}: ${result.error}`);
+          }
+        }
+      }
+
+      if (errors.length > 0) {
+        failed.push({ record, errors });
+      } else {
+        passed.push(record);
+
+        if (warns.length > 0) {
+          warnings.push({ record, warnings: warns });
+        }
+      }
+    }
+
+    return { passed, failed, warnings };
+  }
+}
+
+// Define quality rules
+const qualityValidator = new DataQualityValidator();
+
+qualityValidator.addRule({
+  name: 'Required Fields',
+  severity: 'error',
+  validate: record => {
+    const required = ['sku', 'warehouse', 'quantity'];
+    const missing = required.filter(field => !record[field]);
+
+    if (missing.length > 0) {
+      return {
+        isValid: false,
+        error: `Missing required fields: ${missing.join(', ')}`,
+      };
+    }
+
+    return { isValid: true };
+  },
+});
+
+qualityValidator.addRule({
+  name: 'Quantity Range',
+  severity: 'error',
+  validate: record => {
+    const qty = parseInt(record.quantity, 10);
+
+    if (isNaN(qty) || qty < 0) {
+      return {
+        isValid: false,
+        error: `Invalid quantity: ${record.quantity}`,
+      };
+    }
+
+    return { isValid: true };
+  },
+});
+
+qualityValidator.addRule({
+  name: 'SKU Format',
+  severity: 'error',
+  validate: record => {
+    const skuPattern = /^SKU-[A-Z0-9]{2,10}$/;
+
+    if (!skuPattern.test(record.sku)) {
+      return {
+        isValid: false,
+        error: `Invalid SKU format: ${record.sku}`,
+      };
+    }
+
+    return { isValid: true };
+  },
+});
+
+qualityValidator.addRule({
+  name: 'Suspiciously Large Quantity',
+  severity: 'warning',
+  validate: record => {
+    const qty = parseInt(record.quantity, 10);
+
+    if (qty > 100000) {
+      return {
+        isValid: false,
+        error: `Unusually large quantity: ${qty}`,
+      };
+    }
+
+    return { isValid: true };
+  },
+});
+
+// Usage
+const records = await parseCSV(fileContent);
+const validation = qualityValidator.validate(records);
+
+console.log(`Passed: ${validation.passed.length}`);
+console.log(`Failed: ${validation.failed.length}`);
+console.log(`Warnings: ${validation.warnings.length}`);
+
+// Log failures
+if (validation.failed.length > 0) {
+  console.error('Data quality failures:');
+  validation.failed.forEach(({ record, errors }) => {
+    console.error(`  SKU ${record.sku}:`, errors);
+  });
+
+  // Save failed records for review
+  await saveDataQualityReport(validation.failed);
+}
+
+// Proceed with passed records only
+await processRecords(validation.passed);
+```
+
+### Reconciliation Reports
+
+```typescript
+interface ReconciliationReport {
+  fileKey: string;
+  processedAt: string;
+  totalRecords: number;
+  successfulRecords: number;
+  failedRecords: number;
+  duplicateRecords: number;
+  dataQualityIssues: number;
+  batchesCreated: number;
+  processingTimeMs: number;
+}
+
+async function generateReconciliationReport(
+  fileKey: string,
+  results: ProcessingResult
+): Promise<ReconciliationReport> {
+  return {
+    fileKey,
+    processedAt: new Date().toISOString(),
+    totalRecords: results.totalRecords,
+    successfulRecords: results.successful,
+    failedRecords: results.failed,
+    duplicateRecords: results.duplicates || 0,
+    dataQualityIssues: results.qualityIssues || 0,
+    batchesCreated: results.batches,
+    processingTimeMs: results.processingTime,
+  };
+}
+
+// Save report to S3
+async function saveReconciliationReport(report: ReconciliationReport): Promise<void> {
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_JSON',
+      connectionId: 's3-reports',
+      name: 'S3 Reports',
+      s3Config: s3Config,
+    },
+    logger
+  );
+  const reportKey = `reports/reconciliation/${report.fileKey}-${Date.now()}.json`;
+
+  await s3.uploadFile(reportKey, JSON.stringify(report, null, 2));
+
+  console.log(`Reconciliation report saved: ${reportKey}`);
+}
+```
+
+## Security Best Practices
+
+### Credential Management
+
+```typescript
+// ✅ CORRECT - Use environment variables
+const config = {
+  fluent: {
+    baseUrl: process.env.FLUENT_BASE_URL!,
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+  s3: {
+    region: process.env.AWS_REGION!,
+    credentials: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    },
+  },
+};
+
+// ❌ WRONG - Never hardcode credentials
+const badConfig = {
+  fluent: {
+    clientId: 'hardcoded-client-id', // ❌ Don't do this
+    clientSecret: 'hardcoded-secret', // ❌ Security risk
+  },
+};
+```
+
+### Secrets Manager Integration
+
+```typescript
+import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
+
+async function loadSecretsFromAWS(): Promise<any> {
+  const client = new SecretsManagerClient({ region: 'us-east-1' });
+
+  const command = new GetSecretValueCommand({
+    SecretId: 'fluent-commerce/production',
+  });
+
+  const response = await client.send(command);
+  return JSON.parse(response.SecretString!);
+}
+
+// Usage
+const secrets = await loadSecretsFromAWS();
+
+const client = await createClient({
+  config: {
+    baseUrl: secrets.FLUENT_BASE_URL,
+    clientId: secrets.FLUENT_CLIENT_ID,
+    clientSecret: secrets.FLUENT_CLIENT_SECRET,
+    retailerId: secrets.FLUENT_RETAILER_ID,
+  }
+});
+```
+
+### Data Encryption
+
+```typescript
+// Encrypt sensitive data before storing
+import * as crypto from 'crypto';
+
+function encryptSensitiveData(data: string, key: string): string {
+  const iv = crypto.randomBytes(16);
+  const cipher = crypto.createCipheriv('aes-256-cbc', Buffer.from(key), iv);
+
+  let encrypted = cipher.update(data, 'utf8', 'hex');
+  encrypted += cipher.final('hex');
+
+  return iv.toString('hex') + ':' + encrypted;
+}
+
+function decryptSensitiveData(encrypted: string, key: string): string {
+  const parts = encrypted.split(':');
+  const iv = Buffer.from(parts[0], 'hex');
+  const encryptedData = parts[1];
+
+  const decipher = crypto.createDecipheriv('aes-256-cbc', Buffer.from(key), iv);
+
+  let decrypted = decipher.update(encryptedData, 'hex', 'utf8');
+  decrypted += decipher.final('utf8');
+
+  return decrypted;
+}
+```
+
+### Audit Logging
+
+```typescript
+interface AuditLog {
+  timestamp: string;
+  operation: string;
+  user: string;
+  resource: string;
+  status: 'success' | 'failure';
+  details: Record<string, any>;
+}
+
+async function logAuditEvent(event: Omit<AuditLog, 'timestamp'>): Promise<void> {
+  const auditLog: AuditLog = {
+    timestamp: new Date().toISOString(),
+    ...event,
+  };
+
+  // Log to CloudWatch, Datadog, or custom logging service
+  console.log('[AUDIT]', JSON.stringify(auditLog));
+
+  // Optionally save to S3 for long-term retention
+  await saveAuditLog(auditLog);
+}
+
+// Usage
+await logAuditEvent({
+  operation: 'INVENTORY_INGESTION',
+  user: process.env.USER || 'system',
+  resource: `s3://bucket/file.csv`,
+  status: 'success',
+  details: {
+    recordsProcessed: 1000,
+    jobId: 'job-123',
+    duration: 5000,
+  },
+});
+```
+
+## Monitoring and Alerting
+
+### Metrics Tracking
+
+```typescript
+interface IngestionMetrics {
+  timestamp: string;
+  filesProcessed: number;
+  recordsIngested: number;
+  failedRecords: number;
+  processingTimeMs: number;
+  avgRecordsPerSecond: number;
+  errorRate: number;
+}
+
+class MetricsCollector {
+  private metrics: IngestionMetrics[] = [];
+
+  recordIngestion(result: IngestionResult): void {
+    const metrics: IngestionMetrics = {
+      timestamp: new Date().toISOString(),
+      filesProcessed: result.filesProcessed,
+      recordsIngested: result.recordsIngested,
+      failedRecords: result.failedRecords,
+      processingTimeMs: result.processingTimeMs,
+      avgRecordsPerSecond: result.recordsIngested / (result.processingTimeMs / 1000),
+      errorRate: result.failedRecords / result.recordsIngested
+    };
+
+    this.metrics.push(metrics);
+
+    // Alert on high error rate
+    if (metrics.errorRate > 0.05) { // > 5% error rate
+      this.sendAlert('High error rate detected', metrics);
+    }
+
+    // Alert on slow processing
+    if (metrics.avgRecordsPerSecond < 10) {
+      this.sendAlert('Slow processing speed', metrics);
+    }
+  }
+
+  private sendAlert(message: string, metrics: IngestionMetrics): void {
+    console.error(\`[ALERT] \${message}\`, metrics);
+    // Send to monitoring service (e.g., Datadog, New Relic)
+  }
+}
+```
+
+### Health Checks
+
+```typescript
+async function performHealthCheck(): Promise<HealthCheckResult> {
+  const checks = [];
+
+  // Check Fluent API connectivity
+  try {
+    await client.graphql({
+      query: '{ __typename }'
+    });
+    checks.push({ name: 'Fluent API', status: 'healthy' });
+  } catch (error) {
+    checks.push({ name: 'Fluent API', status: 'unhealthy', error: error.message });
+  }
+
+  // Check S3 connectivity
+  try {
+    await s3.listFiles({ prefix: 'health-check/', maxKeys: 1 });
+    checks.push({ name: 'S3', status: 'healthy' });
+  } catch (error) {
+    checks.push({ name: 'S3', status: 'unhealthy', error: error.message });
+  }
+
+  // Check state storage
+  try {
+    await state.set('health-check', Date.now());
+    await state.get('health-check');
+    checks.push({ name: 'State Storage', status: 'healthy' });
+  } catch (error) {
+    checks.push({ name: 'State Storage', status: 'unhealthy', error: error.message });
+  }
+
+  const allHealthy = checks.every(c => c.status === 'healthy');
+
+  return {
+    status: allHealthy ? 'healthy' : 'degraded',
+    checks,
+    timestamp: new Date().toISOString(),
+  };
+}
+```
+
+## Complete Production Pipeline Example
+
+Putting all best practices together into a production-ready ingestion pipeline:
+
+```typescript
+import {
+  createClient,
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  StateService,
+  VersoriKVAdapter,
+  FluentClient,
+  FileParsingError,
+  MappingError,
+  BatchAPIError,
+} from '@fluentcommerce/fc-connect-sdk';
+import * as winston from 'winston';
+
+/**
+ * Production-grade ingestion pipeline with all best practices
+ * - Error handling with retry and circuit breaker
+ * - Comprehensive monitoring and logging
+ * - Security best practices
+ * - Data quality validation
+ * - State management
+ */
+class ProductionIngestionPipeline {
+  private client: FluentClient;
+  private s3: S3DataSource;
+  private parser: CSVParserService;
+  private mapper: UniversalMapper;
+  private state: StateService;
+  private kv: KVStore;
+  private logger: winston.Logger;
+  private circuitBreaker: CircuitBreaker;
+  private dlq: DeadLetterQueue;
+  private metrics: MetricsCollector;
+  private qualityValidator: DataQualityValidator;
+
+  constructor(private config: ProductionConfig) {}
+
+  async initialize(): Promise<void> {
+    // Setup structured logging
+    this.logger = winston.createLogger({
+      level: 'info',
+      format: winston.format.combine(winston.format.timestamp(), winston.format.json()),
+      transports: [
+        new winston.transports.Console(),
+        new winston.transports.File({ filename: 'ingestion-error.log', level: 'error' }),
+        new winston.transports.File({ filename: 'ingestion-combined.log' }),
+      ],
+    });
+
+    this.logger.info('Initializing production ingestion pipeline');
+
+    // Initialize Fluent client
+    this.client = await createClient({ config: this.config.fluent });
+
+    // Initialize data sources
+    this.s3 = new S3DataSource(
+      {
+        type: 'S3_CSV',
+        connectionId: 's3-production',
+        name: 'S3 Production',
+        s3Config: this.config.s3,
+      },
+      this.logger
+    );
+    this.parser = new CSVParserService();
+
+    // Initialize field mapper with custom resolvers
+    this.mapper = new UniversalMapper(this.config.mapping, {
+      customResolvers: this.config.customResolvers || {},
+    });
+
+    // Initialize state management
+    const logger = toStructuredLogger(this.logger, {
+      service: 'production-pipeline',
+      correlationId: generateCorrelationId()
+    });
+
+    this.kv = new VersoriKVAdapter(this.config.kv);
+    this.stateService = new StateService(logger);
+
+    // Initialize circuit breaker
+    this.circuitBreaker = new CircuitBreaker({
+      failureThreshold: 5,
+      resetTimeoutMs: 60000,
+      monitoringWindowMs: 120000,
+    });
+
+    // Initialize dead letter queue
+    this.dlq = new DeadLetterQueue(this.s3, this.config.dlqBucket, 'dlq/');
+
+    // Initialize metrics collector
+    this.metrics = new MetricsCollector();
+
+    // Initialize data quality validator
+    this.qualityValidator = this.createQualityValidator();
+
+    this.logger.info('Pipeline initialization complete');
+  }
+
+  /**
+   * Run the complete ingestion pipeline
+   */
+  async run(bucket: string, prefix: string): Promise<ExecutionReport> {
+    const startTime = Date.now();
+
+    try {
+      this.logger.info('Starting ingestion pipeline', { bucket, prefix });
+
+      // Health check before starting
+      const healthCheck = await this.performHealthCheck();
+      if (healthCheck.status !== 'healthy') {
+        throw new Error(`System unhealthy: ${JSON.stringify(healthCheck.checks)}`);
+      }
+
+      // List files
+      const allFiles = await this.s3.listFiles({ prefix });
+      this.logger.info(`Found ${allFiles.length} total files`);
+
+      // Filter unprocessed files
+      const unprocessedFiles = await this.filterUnprocessedFiles(allFiles);
+      this.logger.info(`${unprocessedFiles.length} files to process`);
+
+      if (unprocessedFiles.length === 0) {
+        return this.createReport(startTime, { filesProcessed: 0 });
+      }
+
+      // Process files with circuit breaker
+      const results = await this.circuitBreaker.execute(async () => {
+        return await this.processFilesWithRetry(bucket, unprocessedFiles);
+      });
+
+      // Generate report
+      const report = this.createReport(startTime, results);
+
+      // Log metrics
+      this.metrics.recordIngestion({
+        filesProcessed: results.filesProcessed,
+        recordsIngested: results.recordsIngested,
+        failedRecords: results.failedRecords,
+        processingTimeMs: report.totalTimeMs,
+        avgRecordsPerSecond: report.throughput,
+        errorRate: results.failedRecords / (results.recordsIngested || 1),
+      });
+
+      this.logger.info('Pipeline execution complete', report);
+
+      return report;
+    } catch (error) {
+      this.logger.error('Pipeline execution failed', {
+        error: error.message,
+        stack: error.stack,
+      });
+
+      await this.sendAlert('CRITICAL: Ingestion pipeline failure', error);
+      throw error;
+    }
+  }
+
+  /**
+   * Process files with retry logic
+   */
+  private async processFilesWithRetry(
+    bucket: string,
+    files: Array<{ key: string }>
+  ): Promise<ProcessingResults> {
+    const results: ProcessingResults = {
+      filesProcessed: 0,
+      filesFailed: 0,
+      recordsIngested: 0,
+      failedRecords: 0,
+      details: [],
+    };
+
+    for (const file of files) {
+      const maxRetries = 3;
+      let attempt = 0;
+      let success = false;
+
+      while (attempt < maxRetries && !success) {
+        attempt++;
+
+        try {
+          const fileResult = await this.processFile(bucket, file.path);
+
+          results.filesProcessed++;
+          results.recordsIngested += fileResult.recordsProcessed;
+          results.details.push({
+            file: file.path,
+            status: 'success',
+            records: fileResult.recordsProcessed,
+          });
+
+          success = true;
+
+          this.logger.info(`File processed successfully`, {
+            file: file.path,
+            records: fileResult.recordsProcessed,
+            attempt,
+          });
+        } catch (error) {
+          this.logger.warn(`File processing attempt ${attempt} failed`, {
+            file: file.path,
+            error: error.message,
+          });
+
+          if (attempt < maxRetries && this.isRetryable(error)) {
+            const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
+            await new Promise(resolve => setTimeout(resolve, delay));
+          } else {
+            results.filesFailed++;
+            results.details.push({
+              file: file.path,
+              status: 'failed',
+              error: error.message,
+            });
+
+            this.logger.error(`File processing failed permanently`, {
+              file: file.path,
+              attempts: attempt,
+              error: error.message,
+            });
+          }
+        }
+      }
+    }
+
+    return results;
+  }
+
+  /**
+   * Process a single file with all validations
+   */
+  private async processFile(bucket: string, fileKey: string): Promise<FileProcessingResult> {
+    this.logger.info(`Processing file: ${fileKey}`);
+
+    try {
+      // Read file
+      const fileContent = await this.s3.downloadFile(fileKey);
+
+      // Parse CSV
+      const records = await this.parser.parse(fileContent);
+      this.logger.debug(`Parsed ${records.length} records from ${fileKey}`);
+
+      // Data quality validation
+      const validation = this.qualityValidator.validate(records);
+
+      if (validation.failed.length > 0) {
+        this.logger.warn(`Data quality issues found`, {
+          file: fileKey,
+          failed: validation.failed.length,
+          warnings: validation.warnings.length,
+        });
+
+        // Save failed records to DLQ
+        for (const { record, errors } of validation.failed) {
+          await this.dlq.save({
+            originalFile: fileKey,
+            failedAt: new Date().toISOString(),
+            errorType: 'DATA_QUALITY_ERROR',
+            errorMessage: errors.join('; '),
+            recordData: record,
+            retryCount: 0,
+            metadata: {},
+          });
+        }
+      }
+
+      // Proceed with passed records only
+      if (validation.passed.length === 0) {
+        throw new Error('No valid records after quality validation');
+      }
+
+      // Field mapping
+      const mappingResult = await this.mapper.map(validation.passed);
+
+      if (!mappingResult.success) {
+        throw new MappingError(`Field mapping failed: ${mappingResult.errors.join(', ')}`);
+      }
+
+      // Create job
+      const job = await this.client.createJob({
+        name: `Production Import - ${fileKey}`,
+        retailerId: this.config.fluent.retailerId,
+        metadata: {
+          fileName: fileKey,
+          recordCount: mappingResult.data.length,
+          pipeline: 'production',
+        },
+      });
+
+      // Send to Batch API
+      await this.client.sendBatch(job.id, {
+        action: 'UPSERT',
+        entityType: 'INVENTORY',
+        entities: mappingResult.data,
+      });
+
+      // Mark as processed
+      await this.state.markFileProcessed(fileKey, {
+        jobId: job.id,
+        recordCount: mappingResult.data.length,
+        timestamp: new Date().toISOString(),
+      });
+
+      // Audit log
+      await this.logAuditEvent({
+        operation: 'INVENTORY_INGESTION',
+        user: 'system',
+        resource: `s3://${bucket}/${fileKey}`,
+        status: 'success',
+        details: {
+          recordsProcessed: mappingResult.data.length,
+          jobId: job.id,
+        },
+      });
+
+      return {
+        recordsProcessed: mappingResult.data.length,
+        jobId: job.id,
+      };
+    } catch (error) {
+      // Audit log failure
+      await this.logAuditEvent({
+        operation: 'INVENTORY_INGESTION',
+        user: 'system',
+        resource: `s3://${bucket}/${fileKey}`,
+        status: 'failure',
+        details: {
+          error: error.message,
+        },
+      });
+
+      throw error;
+    }
+  }
+
+  /**
+   * Check if error is retryable
+   */
+  private isRetryable(error: any): boolean {
+    if (error instanceof FileParsingError) {
+      return false; // File format errors are not retryable
+    }
+
+    if (error instanceof BatchAPIError) {
+      return error.isRetryable;
+    }
+
+    // Network errors are retryable
+    return (
+      error.code === 'NETWORK_ERROR' ||
+      error.message.includes('timeout') ||
+      error.message.includes('ECONNRESET')
+    );
+  }
+
+  /**
+   * Filter unprocessed files
+   */
+  private async filterUnprocessedFiles(
+    files: FileMetadata[]
+  ): Promise<FileMetadata[]> {
+    const unprocessed = [];
+
+    for (const file of files) {
+      if (!(await this.state.isFileProcessed(this.kv, file.path))) {
+        unprocessed.push(file);
+      }
+    }
+
+    return unprocessed;
+  }
+
+  /**
+   * Create data quality validator
+   */
+  private createQualityValidator(): DataQualityValidator {
+    const validator = new DataQualityValidator();
+
+    // Add validation rules based on configuration
+    validator.addRule({
+      name: 'Required Fields',
+      severity: 'error',
+      validate: record => {
+        const required = this.config.requiredFields || [];
+        const missing = required.filter(field => !record[field]);
+
+        if (missing.length > 0) {
+          return {
+            isValid: false,
+            error: `Missing required fields: ${missing.join(', ')}`,
+          };
+        }
+
+        return { isValid: true };
+      },
+    });
+
+    return validator;
+  }
+
+  /**
+   * Perform system health check
+   */
+  private async performHealthCheck(): Promise<HealthCheckResult> {
+    const checks = [];
+
+    // Check Fluent API
+    try {
+      await this.client.graphql({
+        query: '{ __typename }'
+      });
+      checks.push({ name: 'Fluent API', status: 'healthy' });
+    } catch (error) {
+      checks.push({ name: 'Fluent API', status: 'unhealthy', error: error.message });
+    }
+
+    // Check S3
+    try {
+      await this.s3.listObjects(this.config.s3Bucket, 'health-check/');
+      checks.push({ name: 'S3', status: 'healthy' });
+    } catch (error) {
+      checks.push({ name: 'S3', status: 'unhealthy', error: error.message });
+    }
+
+    const allHealthy = checks.every(c => c.status === 'healthy');
+
+    return {
+      status: allHealthy ? 'healthy' : 'degraded',
+      checks,
+      timestamp: new Date().toISOString(),
+    };
+  }
+
+  /**
+   * Create execution report
+   */
+  private createReport(startTime: number, results: Partial<ProcessingResults>): ExecutionReport {
+    const totalTimeMs = Date.now() - startTime;
+
+    return {
+      startTime: new Date(startTime).toISOString(),
+      endTime: new Date().toISOString(),
+      totalTimeMs,
+      filesProcessed: results.filesProcessed || 0,
+      filesFailed: results.filesFailed || 0,
+      recordsIngested: results.recordsIngested || 0,
+      failedRecords: results.failedRecords || 0,
+      throughput: (results.recordsIngested || 0) / (totalTimeMs / 1000),
+      successRate:
+        ((results.filesProcessed || 0) /
+          ((results.filesProcessed || 0) + (results.filesFailed || 0))) *
+        100,
+    };
+  }
+
+  /**
+   * Log audit event
+   */
+  private async logAuditEvent(event: Omit<AuditLog, 'timestamp'>): Promise<void> {
+    const auditLog: AuditLog = {
+      timestamp: new Date().toISOString(),
+      ...event,
+    };
+
+    this.logger.info('[AUDIT]', auditLog);
+
+    // Optionally save to S3 for compliance
+    // await this.s3.putObject(auditBucket, auditKey, JSON.stringify(auditLog));
+  }
+
+  /**
+   * Send alert to monitoring service
+   */
+  private async sendAlert(message: string, error: any): Promise<void> {
+    this.logger.error('[ALERT]', { message, error: error.message });
+
+    // Integration with alerting service (PagerDuty, Slack, etc.)
+    // await alertService.send({ message, error });
+  }
+}
+
+// Usage
+const pipeline = new ProductionIngestionPipeline({
+  fluent: {
+    baseUrl: process.env.FLUENT_BASE_URL!,
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+  s3: {
+    region: process.env.AWS_REGION!,
+    credentials: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    },
+  },
+  s3Bucket: 'inventory-bucket',
+  dlqBucket: 'inventory-dlq-bucket',
+  mapping: {
+    fields: {
+      ref: { source: 'sku', required: true },
+      productRef: { source: 'product_id', required: true },
+      locationRef: { source: 'warehouse_code', required: true },
+      qty: { source: 'quantity', resolver: 'sdk.parseInt', required: true },
+      type: { source: 'inventory_type', default: 'ON_HAND' },
+      status: { source: 'status', default: 'AVAILABLE' },
+    },
+  },
+  requiredFields: ['sku', 'warehouse', 'quantity'],
+  kv: openKv(),
+});
+
+await pipeline.initialize();
+const report = await pipeline.run('inventory-bucket', 'data/');
+
+console.log('Ingestion complete:', report);
+```
+
+## Common Errors and Solutions
+
+### Error: JOB_EXPIRED
+
+**Cause:** Job exceeded 24-hour lifetime
+
+**Solution:**
+
+```typescript
+async function handleExpiredJob(originalJobId: string, remainingBatches: any[]) {
+  // Create new job
+  const newJob = await client.createJob({
+    name: \`Recovery - \${new Date().toISOString()}\`,
+    retailerId: process.env.FLUENT_RETAILER_ID!,
+    metadata: {
+      originalJobId,
+      reason: 'JOB_EXPIRED'
+    }
+  });
+
+  // Resend remaining batches
+  for (const batch of remainingBatches) {
+    await client.sendBatch(newJob.id, batch);
+  }
+
+  return newJob.id;
+}
+```
+
+### Error: VALIDATION_ERROR
+
+**Cause:** Invalid field values or missing required fields
+
+**Solution:**
+
+```typescript
+async function handleValidationError(records: any[], error: any) {
+  // Parse validation errors
+  const invalidRecords = extractInvalidRecords(error);
+
+  // Filter out invalid records
+  const validRecords = records.filter(r => !invalidRecords.includes(r));
+
+  // Log invalid records
+  console.error('Invalid records:', invalidRecords);
+  await saveToErrorLog(invalidRecords);
+
+  // Retry with valid records only
+  return validRecords;
+}
+```
+
+### Error: RATE_LIMIT_ERROR
+
+**Cause:** Too many requests to Fluent API
+
+**Solution:**
+
+```typescript
+class RateLimitHandler {
+  async handleRateLimit(operation: () => Promise<any>): Promise<any> {
+    try {
+      return await operation();
+    } catch (error) {
+      if (error.code === 'RATE_LIMIT_ERROR') {
+        const retryAfter = error.retryAfter || 60000; // Default 60s
+        console.log(\`Rate limited. Waiting \${retryAfter}ms\`);
+        await sleep(retryAfter);
+        return await operation(); // Retry once
+      }
+      throw error;
+    }
+  }
+}
+```
+
+## Production Checklist
+
+### Before Deployment
+
+- [ ] Schema validated against Fluent Commerce API
+- [ ] Field mappings tested with sample data
+- [ ] Error handling implemented for all operations
+- [ ] State management configured and tested
+- [ ] Monitoring and alerting configured
+- [ ] Health checks implemented
+- [ ] Rate limiting configured
+- [ ] Dead letter queue configured
+- [ ] Rollback plan documented
+
+### During Deployment
+
+- [ ] Start with small batch sizes (100-500)
+- [ ] Monitor error rates closely
+- [ ] Test with sample files first
+- [ ] Gradually increase batch sizes
+- [ ] Verify state management works correctly
+
+### After Deployment
+
+- [ ] Monitor ingestion metrics daily
+- [ ] Review error logs regularly
+- [ ] Track processing times
+- [ ] Optimize based on metrics
+- [ ] Document common issues and solutions
+
+## Debugging Tips
+
+### Enable Debug Logging
+
+```typescript
+const client = await createClient({
+  config: {
+    ...config
+  },
+  logger: {
+    debug: (...args) => console.log('[DEBUG]', ...args),
+    info: (...args) => console.log('[INFO]', ...args),
+    warn: (...args) => console.warn('[WARN]', ...args),
+    error: (...args) => console.error('[ERROR]', ...args),
+  },
+});
+```
+
+### Trace File Processing
+
+```typescript
+async function traceFileProcessing(fileKey: string) {
+  console.log(\`[TRACE] Starting processing: \${fileKey}\`);
+
+  try {
+    console.log('[TRACE] Reading file from S3');
+    const data = await s3.downloadFile(fileKey);
+    console.log(\`[TRACE] File size: \${data.length} bytes\`);
+
+    console.log('[TRACE] Parsing CSV');
+    const records = await parser.parse(data);
+    console.log(\`[TRACE] Parsed \${records.length} records\`);
+
+    console.log('[TRACE] Mapping fields');
+    const result = await mapper.map(records);
+    console.log(\`[TRACE] Mapped \${result.data.length} valid records\`);
+
+    console.log('[TRACE] Creating job');
+    const job = await client.createJob({ name: \`Import - \${fileKey}\` });
+    console.log(\`[TRACE] Job created: \${job.id}\`);
+
+    console.log('[TRACE] Sending batch');
+    await client.sendBatch(job.id, { entities: result.data });
+    console.log('[TRACE] Batch sent successfully');
+
+  } catch (error) {
+    console.error(\`[TRACE] Error at: \${error.message}\`);
+    throw error;
+  }
+}
+```
+
+## Key Takeaways
+
+- 🎯 **Always validate** - Check schema and data before sending
+- 🎯 **Handle errors gracefully** - Retry transient failures, log permanent ones
+- 🎯 **Monitor everything** - Track metrics, set up alerts
+- 🎯 **Use state management** - Prevent duplicates, track history
+- 🎯 **Test thoroughly** - Start small, scale gradually
+- 🎯 **Document** - Keep runbooks for common issues
+
+## Congratulations!
+
+You've completed the Data Ingestion Learning Path! You now know:
+
+- ✅ Core ingestion concepts and architecture
+- ✅ How to read from multiple data sources
+- ✅ How to parse CSV, Parquet, XML, and JSON
+- ✅ How to transform data with UniversalMapper
+- ✅ How to use the Batch API effectively
+- ✅ How to implement state management
+- ✅ How to optimize performance
+- ✅ How to build production-ready ingestion workflows
+
+## Next Steps
+
+Congratulations! You've completed the Data Ingestion Learning Path and mastered production-ready ingestion patterns.
+
+**Continue Learning:**
+
+- 📖 [Data Extraction Guide](../../extraction/) - Learn reverse workflows (Fluent → S3/Parquet/CSV)
+- 📖 [Resolver Development Guide](../../mapping/resolvers/mapping-resolvers-readme.md) - Build custom data transformations
+- 📖 [Universal Mapping Guide](../../mapping/mapping-readme.md) - Master field mapping patterns
+- 📖 [Error Handling Guide](../../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md) - Deep dive into error patterns
+- 📖 [Versori Platform Integration](../../../04-REFERENCE/platforms/versori/) - Deploy to production
+
+**Real-World Examples:**
+
+- 🔍 [Complete Connector Examples](../../../01-TEMPLATES/versori/workflows/readme.md) - Production Versori connectors
+- 📋 [Use Case Library](../../../01-TEMPLATES/readme.md) - Business-specific implementations
+- 🛠️ [CLI Tools](../../../../bin/) - Code generation and validation utilities
+
+**Resources:**
+
+- 📚 [Complete API Reference](../../api-reference/api-reference-readme.md)
+- 📖 [Quick Reference Cheat Sheet](../ingestion-quick-reference.md)
+- 🐛 [Troubleshooting Guide](../../../00-START-HERE/troubleshooting-quick-reference.md)
+
+---
+
+[← Back to Ingestion Guide](../ingestion-readme.md) | [Previous: Module 8 - Performance Optimization](./02-core-guides-ingestion-08-performance-optimization.md)
+
+**Need Help?**
+
+- 📧 Email: support@fluentcommerce.com
+- 📚 Documentation: [FC Connect SDK Docs](../../../)
+- 🔗 GitHub: Report issues or contribute