@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md

@@ -1,1487 +1,1487 @@
-# File Operations Error Handling Guide
-
-> **Complete guide to handling errors when reading and parsing files from S3 and SFTP**
-> Production-ready patterns for connection errors, file discovery, reading, and parsing failures
-
-## Table of Contents
-
-- [Overview](#overview)
-- [Error Flow Diagram](#error-flow-diagram)
-- [Connection Errors](#connection-errors)
-- [Discovery Errors](#discovery-errors)
-- [Reading Errors](#reading-errors)
-- [Parsing Errors](#parsing-errors)
-- [Complete Examples](#complete-examples)
-- [Retry Strategies](#retry-strategies)
-- [Troubleshooting Flowchart](#troubleshooting-flowchart)
-- [Production Checklist](#production-checklist)
-
----
-
-## Overview
-
-When working with S3 or SFTP data sources, errors can occur at multiple stages:
-
-```
-┌─────────────────┐
-│ 1. Connection   │ ← Credentials, network, timeouts
-├─────────────────┤
-│ 2. Discovery    │ ← File listing, permissions, paths
-├─────────────────┤
-│ 3. Reading      │ ← Download, memory, streaming
-├─────────────────┤
-│ 4. Parsing      │ ← CSV/XML/JSON/Parquet format errors
-└─────────────────┘
-```
-
-This guide covers **all four stages** with production-ready error handling patterns.
-
-### Key Error Types
-
-| Error Class | Stage | Retryable? | Common Causes |
-|-------------|-------|------------|---------------|
-| `FileDiscoveryError` | Connection/Discovery | ✅ Usually | Invalid credentials, network timeout, permissions |
-| `FileParsingError` | Parsing | ❌ No | Malformed XML/CSV/JSON, encoding issues |
-| `IngestionError` (NETWORK_ERROR) | Connection/Reading | ✅ Yes | Network timeout, API unreachable |
-| Memory errors | Reading | ⚠️ Sometimes | File too large, insufficient heap |
-
----
-
-## Error Flow Diagram
-
-```
-┌──────────────────────────────────────────────────────────────┐
-│ START: Read and Parse File from S3/SFTP                      │
-└────────────────────┬─────────────────────────────────────────┘
-                     │
-                     ▼
-         ┌──────────────────────┐
-         │ Create Data Source   │
-         │ (S3/SFTP)            │
-         └──────┬───────────────┘
-                │
-                ▼
-         ┌──────────────────────┐
-         │ Validate Connection  │
-         └──────┬───────────────┘
-                │
-                ├─────────── ❌ FileDiscoveryError
-                │             → Invalid credentials
-                │             → Network timeout
-                │             → Permissions denied
-                │             → RETRY? Yes (if transient)
-                │
-                ▼
-         ┌──────────────────────┐
-         │ List Files           │
-         │ (discovery)          │
-         └──────┬───────────────┘
-                │
-                ├─────────── ❌ FileDiscoveryError
-                │             → Bucket/path not found
-                │             → Access denied
-                │             → RETRY? Usually No (config error)
-                │
-                ▼
-         ┌──────────────────────┐
-         │ Download File        │
-         │ (read content)       │
-         └──────┬───────────────┘
-                │
-                ├─────────── ❌ IngestionError (NETWORK_ERROR)
-                │             → Network timeout
-                │             → Presigned URL expired (S3)
-                │             → SFTP connection lost
-                │             → RETRY? Yes
-                │
-                ├─────────── ❌ Memory Error
-                │             → File too large
-                │             → Insufficient heap
-                │             → RETRY? No (use streaming)
-                │
-                ▼
-         ┌──────────────────────┐
-         │ Parse Content        │
-         │ (CSV/XML/JSON)       │
-         └──────┬───────────────┘
-                │
-                ├─────────── ❌ FileParsingError
-                │             → Malformed XML (missing tags)
-                │             → Invalid JSON (syntax error)
-                │             → CSV column mismatch
-                │             → Encoding error (non-UTF-8)
-                │             → RETRY? No (fix source data)
-                │
-                ▼
-         ┌──────────────────────┐
-         │ ✅ SUCCESS           │
-         │ Parsed data ready    │
-         └──────────────────────┘
-```
-
----
-
-## Connection Errors
-
-### S3 Connection Errors
-
-#### Error 1: Invalid Credentials
-
-**Symptoms:**
-```
-FileDiscoveryError: Access Denied
-  code: FILE_ACCESS_DENIED
-  message: "The AWS Access Key Id you provided does not exist in our records"
-```
-
-**Causes:**
-- Invalid `accessKeyId` or `secretAccessKey`
-- Credentials expired (if using temporary credentials)
-- Wrong AWS region specified
-
-**Solution:**
-```typescript
-import {
-  S3DataSource,
-  FileDiscoveryError,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-async function createS3Source() {
-  try {
-    const s3Source = new S3DataSource(
-      {
-        type: 'S3_CSV',
-        s3Config: {
-          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-          region: process.env.AWS_REGION || 'us-east-1',
-          bucket: process.env.S3_BUCKET,
-        },
-      },
-      logger
-    );
-
-    // ✅ IMPORTANT: Validate connection before proceeding
-    const isValid = await s3Source.validateConnection();
-    if (!isValid) {
-      throw new Error('S3 connection validation failed');
-    }
-
-    return s3Source;
-
-  } catch (error) {
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_ACCESS_DENIED') {
-      logger.error('S3 credentials invalid:', {
-        bucket: process.env.S3_BUCKET,
-        region: process.env.AWS_REGION,
-        // DO NOT log credentials!
-      });
-
-      // Don't retry - configuration error
-      throw new Error('S3 credentials are invalid. Check AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY');
-    }
-    throw error;
-  }
-}
-```
-
-**Debugging Checklist:**
-- [ ] Verify `AWS_ACCESS_KEY_ID` is correct
-- [ ] Verify `AWS_SECRET_ACCESS_KEY` is correct
-- [ ] Check if credentials are expired (temporary credentials)
-- [ ] Verify AWS region matches bucket region
-- [ ] Test credentials with AWS CLI: `aws s3 ls s3://your-bucket`
-
-#### Error 2: Wrong Region
-
-**Symptoms:**
-```
-FileDiscoveryError: The bucket is in this region: us-west-2.
-Please use this region to retry the request
-```
-
-**Solution:**
-```typescript
-// ❌ WRONG: Hardcoded region doesn't match bucket
-const s3Source = new S3DataSource({
-  type: 'S3_CSV',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: 'us-east-1', // ❌ Bucket is in us-west-2
-    bucket: 'my-bucket',
-  },
-}, logger);
-
-// ✅ CORRECT: Use environment variable or bucket's actual region
-const s3Source = new S3DataSource({
-  type: 'S3_CSV',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: process.env.AWS_REGION!, // ✅ Set to 'us-west-2'
-    bucket: 'my-bucket',
-  },
-}, logger);
-```
-
-#### Error 3: Network Timeout (S3)
-
-**Symptoms:**
-```
-IngestionError: Network request timed out
-  code: NETWORK_ERROR
-  isRetryable: true
-```
-
-**Solution with Retry:**
-```typescript
-import { IngestionError, IngestionErrorCode } from '@fluentcommerce/fc-connect-sdk';
-
-async function downloadWithRetry(
-  s3Source: S3DataSource,
-  filePath: string,
-  maxRetries = 3
-) {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      const content = await s3Source.downloadFile(filePath);
-      return content; // ✅ Success
-
-    } catch (error) {
-      if (error instanceof IngestionError &&
-          error.code === IngestionErrorCode.NETWORK_ERROR &&
-          error.isRetryable() &&
-          attempt < maxRetries) {
-
-        const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
-        logger.warn(`S3 network error - retrying in ${delay}ms (attempt ${attempt}/${maxRetries})`);
-        await new Promise(resolve => setTimeout(resolve, delay));
-        continue;
-      }
-
-      throw error; // Not retryable or max retries exceeded
-    }
-  }
-}
-```
-
-### SFTP Connection Errors
-
-#### Error 1: Authentication Failed
-
-**Symptoms:**
-```
-FileDiscoveryError: All configured authentication methods failed
-  code: FILE_ACCESS_DENIED
-```
-
-**Causes:**
-- Invalid SSH private key
-- Wrong username
-- Private key passphrase incorrect
-- Host key verification failed
-
-**Solution:**
-```typescript
-import { SftpDataSource, FileDiscoveryError } from '@fluentcommerce/fc-connect-sdk';
-
-async function createSftpSource() {
-  try {
-    const sftpSource = new SftpDataSource(
-      {
-        type: 'SFTP_CSV',
-        connectionId: 'vendor-sftp',
-        settings: {
-          host: process.env.SFTP_HOST!,
-          port: parseInt(process.env.SFTP_PORT || '22'),
-          username: process.env.SFTP_USERNAME!,
-          privateKey: process.env.SFTP_PRIVATE_KEY!, // ✅ Use SSH key (not password)
-          passphrase: process.env.SFTP_PASSPHRASE, // If key is encrypted
-          remotePath: process.env.SFTP_REMOTE_PATH || '/data',
-        },
-      },
-      logger
-    );
-
-    // ✅ Validate connection
-    const isValid = await sftpSource.validateConnection();
-    if (!isValid) {
-      throw new Error('SFTP connection validation failed');
-    }
-
-    return sftpSource;
-
-  } catch (error) {
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_ACCESS_DENIED') {
-      logger.error('SFTP authentication failed:', {
-        host: process.env.SFTP_HOST,
-        username: process.env.SFTP_USERNAME,
-        // DO NOT log private key or passphrase!
-      });
-
-      // Check common issues
-      throw new Error(
-        'SFTP authentication failed. Check:\n' +
-        '1. SSH private key is correct\n' +
-        '2. Username is correct\n' +
-        '3. Passphrase is correct (if key is encrypted)\n' +
-        '4. Host key is trusted'
-      );
-    }
-    throw error;
-  }
-}
-```
-
-#### Error 2: Connection Timeout (SFTP)
-
-**Symptoms:**
-```
-FileDiscoveryError: Connection timeout after 30000ms
-  code: FILE_DISCOVERY_FAILED
-```
-
-**Causes:**
-- SFTP server unreachable
-- Firewall blocking port 22
-- Network connectivity issues
-- DNS resolution failure
-
-**Solution:**
-```typescript
-async function testSftpConnectivity() {
-  const sftpConfig = {
-    type: 'SFTP_CSV' as const,
-    connectionId: 'test-sftp',
-    settings: {
-      host: process.env.SFTP_HOST!,
-      port: 22,
-      username: process.env.SFTP_USERNAME!,
-      privateKey: process.env.SFTP_PRIVATE_KEY!,
-      remotePath: '/data',
-      timeout: 30000, // 30 seconds
-    },
-  };
-
-  try {
-    const sftpSource = new SftpDataSource(sftpConfig, logger);
-
-    logger.info('Testing SFTP connectivity...');
-    const isValid = await sftpSource.validateConnection();
-
-    if (isValid) {
-      logger.info('✅ SFTP connection successful');
-      return sftpSource;
-    } else {
-      throw new Error('SFTP validation returned false');
-    }
-
-  } catch (error) {
-    if (error instanceof FileDiscoveryError) {
-      logger.error('SFTP connection failed:', {
-        host: sftpConfig.settings.host,
-        port: sftpConfig.settings.port,
-        error: error.message,
-      });
-
-      // Debugging steps
-      console.error('\n🔍 Debugging steps:');
-      console.error(`1. Test DNS: ping ${sftpConfig.settings.host}`);
-      console.error(`2. Test connectivity: telnet ${sftpConfig.settings.host} 22`);
-      console.error('3. Check firewall allows outbound port 22');
-      console.error('4. Verify SFTP server is running');
-    }
-    throw error;
-  }
-}
-```
-
----
-
-## Discovery Errors
-
-### Error 1: File Not Found
-
-**Symptoms:**
-```
-FileDiscoveryError: File not found
-  code: FILE_NOT_FOUND
-  path: "s3://my-bucket/inventory/missing.csv"
-```
-
-**Solution:**
-```typescript
-async function safeFileDownload(
-  dataSource: S3DataSource | SftpDataSource,
-  filePath: string
-) {
-  try {
-    // Check if file exists first
-    const files = await dataSource.listFiles({ prefix: filePath });
-
-    if (files.length === 0) {
-      logger.warn(`File not found: ${filePath}`);
-      return null; // Return null instead of throwing
-    }
-
-    // File exists - download it
-    const content = await dataSource.downloadFile(filePath);
-    return content;
-
-  } catch (error) {
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_NOT_FOUND') {
-      logger.warn('File not found during download:', { path: filePath });
-      return null; // Graceful handling
-    }
-    throw error; // Re-throw other errors
-  }
-}
-```
-
-### Error 2: Access Denied (Permissions)
-
-**Symptoms:**
-```
-FileDiscoveryError: Access Denied
-  code: FILE_ACCESS_DENIED
-  message: "User is not authorized to perform: s3:GetObject"
-```
-
-**IAM Policy Required (S3):**
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Effect": "Allow",
-      "Action": [
-        "s3:ListBucket",
-        "s3:GetObject",
-        "s3:GetObjectVersion"
-      ],
-      "Resource": [
-        "arn:aws:s3:::my-bucket",
-        "arn:aws:s3:::my-bucket/*"
-      ]
-    }
-  ]
-}
-```
-
-**Solution:**
-```typescript
-async function validateS3Permissions(s3Source: S3DataSource) {
-  try {
-    logger.info('Validating S3 permissions...');
-
-    // Test 1: Can list bucket?
-    try {
-      await s3Source.listFiles({ prefix: '' });
-      logger.info('✅ s3:ListBucket permission OK');
-    } catch (error) {
-      logger.error('❌ s3:ListBucket permission DENIED');
-      throw error;
-    }
-
-    // Test 2: Can read objects?
-    const files = await s3Source.listFiles({ prefix: '' });
-    if (files.length > 0) {
-      try {
-        await s3Source.downloadFile(files[0].path);
-        logger.info('✅ s3:GetObject permission OK');
-      } catch (error) {
-        logger.error('❌ s3:GetObject permission DENIED');
-        throw error;
-      }
-    }
-
-    logger.info('✅ All S3 permissions validated');
-
-  } catch (error) {
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_ACCESS_DENIED') {
-      console.error('\n❌ IAM Permissions Required:');
-      console.error('  - s3:ListBucket on arn:aws:s3:::my-bucket');
-      console.error('  - s3:GetObject on arn:aws:s3:::my-bucket/*');
-      console.error('\nAdd this IAM policy to your AWS user/role.');
-    }
-    throw error;
-  }
-}
-```
-
----
-
-## Reading Errors
-
-### Error 1: Out of Memory (Large Files)
-
-**Symptoms:**
-```
-FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
-```
-
-**Causes:**
-- File too large to load in memory
-- Default Node.js heap size (512MB - 2GB)
-- Multiple large files processed concurrently
-
-**Solution 1: Increase Heap Size**
-```bash
-# Temporary fix - increase Node.js heap
-node --max-old-space-size=4096 your-script.js  # 4GB heap
-```
-
-**Solution 2: Stream Large Files (Best Practice)**
-```typescript
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-import { Readable } from 'stream';
-
-async function processLargeFile(s3Source: S3DataSource, filePath: string) {
-  try {
-    // Option 1: Download in binary mode (Buffer)
-    const content = await s3Source.downloadFile(filePath, { encoding: 'binary' });
-
-    // Process in chunks
-    const buffer = content as Buffer;
-    const chunkSize = 1024 * 1024; // 1MB chunks
-
-    for (let offset = 0; offset < buffer.length; offset += chunkSize) {
-      const chunk = buffer.slice(offset, offset + chunkSize);
-      await processChunk(chunk);
-
-      // Allow garbage collection
-      if (global.gc) global.gc();
-    }
-
-  } catch (error) {
-    if (error.message?.includes('heap out of memory')) {
-      logger.error('Out of memory processing large file:', {
-        file: filePath,
-        suggestion: 'Use chunked processing or increase heap size',
-      });
-
-      throw new Error(
-        'File too large for available memory. ' +
-        'Increase heap size with: node --max-old-space-size=4096'
-      );
-    }
-    throw error;
-  }
-}
-
-async function processChunk(chunk: Buffer) {
-  // Process chunk here
-  logger.debug(`Processing chunk: ${chunk.length} bytes`);
-}
-```
-
-### Error 2: Presigned URL Expired (S3)
-
-**Symptoms:**
-```
-FileDiscoveryError: Request has expired
-  code: FILE_ACCESS_DENIED
-  statusCode: 403
-```
-
-**Cause:**
-- Presigned URLs expire after 1 hour (SDK default)
-- Long-running operations exceed URL lifetime
-
-**Solution:**
-```typescript
-async function downloadWithUrlRefresh(s3Source: S3DataSource, filePath: string) {
-  const startTime = Date.now();
-  const URL_LIFETIME_MS = 55 * 60 * 1000; // 55 minutes (buffer before 1 hour expiry)
-
-  try {
-    const content = await s3Source.downloadFile(filePath);
-    return content;
-
-  } catch (error) {
-    const elapsed = Date.now() - startTime;
-
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_ACCESS_DENIED' &&
-        error.message?.includes('expired') &&
-        elapsed > URL_LIFETIME_MS) {
-
-      logger.warn('Presigned URL expired - recreating data source');
-
-      // Recreate S3DataSource (generates new presigned URLs)
-      const newS3Source = new S3DataSource(s3Source.getConfig(), logger);
-
-      // Retry with new URL
-      return await newS3Source.downloadFile(filePath);
-    }
-
-    throw error;
-  }
-}
-```
-
----
-
-## Parsing Errors
-
-### Error 1: Malformed XML
-
-**Symptoms:**
-```
-FileParsingError: Failed to parse XML: Unclosed tag 'customer-email'
-  fileName: "order.xml"
-  lineNumber: 5
-  code: PARSE_ERROR
-```
-
-**Example Malformed XML:**
-```xml
-<?xml version="1.0" encoding="UTF-8"?>
-<order>
-  <order-id>12345</order-id>
-  <customer>
-    <email>test@example.com
-    <!-- ❌ Missing closing tag for <email> and <customer> -->
-```
-
-**Solution:**
-```typescript
-import { XMLParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
-
-async function parseXMLWithValidation(xmlContent: string, fileName: string) {
-  const parser = new XMLParserService();
-
-  try {
-    const parsed = await parser.parse(xmlContent);
-    return parsed;
-
-  } catch (error) {
-    if (error instanceof FileParsingError &&
-        error.code === 'PARSE_ERROR') {
-      logger.error('XML parsing failed:', {
-        fileName: error.fileName || fileName,
-        lineNumber: error.lineNumber,
-        message: error.message,
-        // Show first 200 chars of XML for debugging
-        xmlPreview: xmlContent.substring(0, 200),
-      });
-
-      throw new Error(
-        `Invalid XML in ${fileName}:\n` +
-        `  Line ${error.lineNumber}: ${error.message}\n` +
-        `  Fix: Ensure all XML tags are properly closed`
-      );
-    }
-    throw error;
-  }
-}
-```
-
-**Common XML Errors:**
-| Error | Cause | Fix |
-|-------|-------|-----|
-| `Unclosed tag 'tagname'` | Missing `</tagname>` | Add closing tag |
-| `Unexpected closing tag` | Extra `</tagname>` | Remove duplicate closing tag |
-| `Invalid character` | Special chars not escaped | Use `&amp;`, `&lt;`, `&gt;`, `&quot;`, `&apos;` |
-| `Text not allowed here` | Text outside tags | Wrap text in tags |
-
-### Error 2: Invalid JSON
-
-**Symptoms:**
-```
-FileParsingError: Failed to parse JSON: Unexpected token a in JSON at position 45
-  fileName: "order.json"
-  code: PARSE_ERROR
-```
-
-**Example Invalid JSON:**
-```json
-{
-  "orderId": "12345",
-  "status": active,     // ❌ Unquoted string
-  "items": [
-    {
-      "sku": "ABC-123",
-      "qty": 5,         // ❌ Trailing comma
-    }
-  ],                   // ❌ Trailing comma
-}
-```
-
-**Solution:**
-```typescript
-import { JSONParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
-
-async function parseJSONWithValidation(jsonContent: string, fileName: string) {
-  const parser = new JSONParserService();
-
-  try {
-    const parsed = await parser.parse(jsonContent);
-    return parsed;
-
-  } catch (error) {
-    if (error instanceof FileParsingError &&
-        error.code === 'PARSE_ERROR') {
-
-      // Extract position from error message
-      const positionMatch = error.message.match(/position (\d+)/);
-      const position = positionMatch ? parseInt(positionMatch[1]) : 0;
-
-      // Show context around error
-      const start = Math.max(0, position - 50);
-      const end = Math.min(jsonContent.length, position + 50);
-      const context = jsonContent.substring(start, end);
-
-      logger.error('JSON parsing failed:', {
-        fileName: fileName,
-        position: position,
-        context: context,
-        message: error.message,
-      });
-
-      throw new Error(
-        `Invalid JSON in ${fileName}:\n` +
-        `  ${error.message}\n` +
-        `  Context: ...${context}...`
-      );
-    }
-    throw error;
-  }
-}
-```
-
-**Common JSON Errors:**
-| Error | Cause | Fix |
-|-------|-------|-----|
-| `Unexpected token` | Unquoted string, trailing comma | Quote strings, remove trailing commas |
-| `Unexpected end of JSON` | Missing closing `}` or `]` | Add closing bracket |
-| `Expected property name` | Missing quotes on key | Quote object keys: `"key": value` |
-
-### Error 3: CSV Column Mismatch
-
-**Symptoms:**
-```
-FileParsingError: CSV column count mismatch
-  fileName: "inventory.csv"
-  code: INVALID_CSV_FORMAT
-  expectedColumns: 5
-  actualColumns: 4
-```
-
-**Example Invalid CSV:**
-```csv
-sku,location,quantity,date,status
-SKU-001,LOC-01,100,2024-01-01,ACTIVE
-SKU-002,LOC-02,50,2024-01-02          ← Missing 'status' column
-SKU-003,LOC-03,75,2024-01-03,ACTIVE
-```
-
-**Solution:**
-```typescript
-import { CSVParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
-
-async function parseCSVWithValidation(
-  csvContent: string,
-  fileName: string,
-  expectedColumns?: string[]
-) {
-  const parser = new CSVParserService();
-
-  try {
-    const parsed = await parser.parse(csvContent);
-
-    // Optional: Validate column names
-    if (expectedColumns && parsed.length > 0) {
-      const actualColumns = Object.keys(parsed[0]);
-      const missingColumns = expectedColumns.filter(col => !actualColumns.includes(col));
-
-      if (missingColumns.length > 0) {
-        throw new FileParsingError(
-          `CSV missing required columns: ${missingColumns.join(', ')}`,
-          'INVALID_CSV_FORMAT',
-          { fileName, missingColumns, actualColumns }
-        );
-      }
-    }
-
-    return parsed;
-
-  } catch (error) {
-    if (error instanceof FileParsingError &&
-        error.code === 'INVALID_CSV_FORMAT') {
-      logger.error('CSV parsing failed:', {
-        fileName: fileName,
-        message: error.message,
-        context: error.context,
-      });
-
-      throw new Error(
-        `Invalid CSV in ${fileName}:\n` +
-        `  ${error.message}\n` +
-        `  Fix: Ensure all rows have the same number of columns`
-      );
-    }
-    throw error;
-  }
-}
-
-// Usage
-const expectedColumns = ['sku', 'location', 'quantity', 'date', 'status'];
-const parsed = await parseCSVWithValidation(csvContent, 'inventory.csv', expectedColumns);
-```
-
-### Error 4: Encoding Issues (Non-UTF-8)
-
-**Symptoms:**
-```
-FileParsingError: Invalid UTF-8 sequence
-  code: ENCODING_ERROR
-```
-
-**Cause:**
-- File encoded in ISO-8859-1, Windows-1252, or other non-UTF-8 encoding
-- Special characters corrupted
-
-**Solution:**
-```typescript
-import { XMLParserService } from '@fluentcommerce/fc-connect-sdk';
-import iconv from 'iconv-lite';
-
-async function parseWithEncodingDetection(
-  fileContent: Buffer | string,
-  fileName: string
-) {
-  const parser = new XMLParserService();
-
-  try {
-    // Try UTF-8 first (SDK default)
-    const content = typeof fileContent === 'string' ? fileContent : fileContent.toString('utf-8');
-    return await parser.parse(content);
-
-  } catch (error) {
-    if (error instanceof FileParsingError &&
-        error.code === 'ENCODING_ERROR') {
-
-      logger.warn('UTF-8 parsing failed - trying alternate encodings');
-
-      // Try common alternate encodings
-      const encodings = ['iso-8859-1', 'windows-1252', 'utf-16le'];
-
-      for (const encoding of encodings) {
-        try {
-          const buffer = typeof fileContent === 'string' ?
-            Buffer.from(fileContent, 'binary') : fileContent;
-
-          const converted = iconv.decode(buffer, encoding);
-          const parsed = await parser.parse(converted);
-
-          logger.info(`✅ Successfully parsed with ${encoding} encoding`);
-          return parsed;
-
-        } catch (e) {
-          continue; // Try next encoding
-        }
-      }
-
-      // All encodings failed
-      throw new Error(
-        `Could not parse ${fileName} with any encoding. ` +
-        `Tried: UTF-8, ${encodings.join(', ')}`
-      );
-    }
-    throw error;
-  }
-}
-```
-
-**Note:** The SDK automatically handles UTF-8 BOM (Byte Order Mark) removal, so BOM presence is not an error.
-
----
-
-## Complete Examples
-
-### Example 1: Production-Ready S3 CSV Pipeline
-
-```typescript
-import {
-  S3DataSource,
-  CSVParserService,
-  FileDiscoveryError,
-  FileParsingError,
-  IngestionError,
-  IngestionErrorCode,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-async function processS3InventoryFiles() {
-  let s3Source: S3DataSource | null = null;
-
-  try {
-    // ────────────────────────────────────────────────────────
-    // STAGE 1: Connection
-    // ────────────────────────────────────────────────────────
-    logger.info('Creating S3 data source...');
-
-    s3Source = new S3DataSource(
-      {
-        type: 'S3_CSV',
-        s3Config: {
-          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-          region: process.env.AWS_REGION!,
-          bucket: process.env.S3_BUCKET!,
-        },
-      },
-      logger
-    );
-
-    // Validate connection
-    const isValid = await s3Source.validateConnection();
-    if (!isValid) {
-      throw new Error('S3 connection validation failed');
-    }
-    logger.info('✅ S3 connection validated');
-
-    // ────────────────────────────────────────────────────────
-    // STAGE 2: Discovery
-    // ────────────────────────────────────────────────────────
-    logger.info('Discovering files...');
-
-    const files = await s3Source.listFiles({
-      prefix: 'inventory/',
-      pattern: /\.csv$/i,
-    });
-
-    if (files.length === 0) {
-      logger.warn('No CSV files found in inventory/ folder');
-      return;
-    }
-
-    logger.info(`Found ${files.length} CSV files`);
-
-    // ────────────────────────────────────────────────────────
-    // STAGE 3: Reading & Parsing
-    // ────────────────────────────────────────────────────────
-    const parser = new CSVParserService();
-    const results = [];
-
-    for (const file of files) {
-      try {
-        logger.info(`Processing ${file.name}...`);
-
-        // Download file with retry
-        const content = await downloadWithRetry(s3Source, file.path, 3);
-
-        // Parse CSV
-        const parsed = await parser.parse(content as string);
-
-        logger.info(`✅ Parsed ${parsed.length} records from ${file.name}`);
-        results.push({ file: file.name, records: parsed });
-
-      } catch (error) {
-        if (error instanceof FileParsingError) {
-          logger.error(`❌ Failed to parse ${file.name}:`, {
-            code: error.code,
-            message: error.message,
-            lineNumber: error.lineNumber,
-          });
-          // Continue with next file (don't fail entire batch)
-          continue;
-        }
-        throw error; // Re-throw other errors
-      }
-    }
-
-    logger.info(`✅ Successfully processed ${results.length}/${files.length} files`);
-    return results;
-
-  } catch (error) {
-    // ────────────────────────────────────────────────────────
-    // STAGE 4: Error Handling
-    // ────────────────────────────────────────────────────────
-    if (error instanceof FileDiscoveryError) {
-      logger.error('File discovery error:', {
-        code: error.code,
-        message: error.message,
-        path: error.path,
-      });
-
-      if (error.code === 'FILE_ACCESS_DENIED') {
-        console.error('\n❌ S3 Access Denied:');
-        console.error('  1. Check AWS credentials are correct');
-        console.error('  2. Verify IAM policy allows s3:ListBucket and s3:GetObject');
-        console.error('  3. Confirm bucket exists and region is correct');
-      }
-
-    } else if (error instanceof IngestionError) {
-      logger.error('Ingestion error:', error.toJSON());
-
-      if (error.isRetryable()) {
-        console.error('\n⚠️  Retryable error - consider running again');
-      }
-
-    } else {
-      logger.error('Unexpected error:', error);
-    }
-
-    throw error;
-  }
-}
-
-// Helper: Download with retry (from earlier example)
-async function downloadWithRetry(
-  s3Source: S3DataSource,
-  filePath: string,
-  maxRetries = 3
-) {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      return await s3Source.downloadFile(filePath);
-    } catch (error) {
-      if (error instanceof IngestionError &&
-          error.code === IngestionErrorCode.NETWORK_ERROR &&
-          error.isRetryable() &&
-          attempt < maxRetries) {
-        const delay = Math.pow(2, attempt) * 1000;
-        logger.warn(`Retry attempt ${attempt}/${maxRetries} in ${delay}ms`);
-        await new Promise(resolve => setTimeout(resolve, delay));
-        continue;
-      }
-      throw error;
-    }
-  }
-}
-
-// Run the pipeline
-processS3InventoryFiles()
-  .then(() => logger.info('Pipeline completed successfully'))
-  .catch(error => {
-    logger.error('Pipeline failed:', error);
-    process.exit(1);
-  });
-```
-
-### Example 2: Production-Ready SFTP XML Pipeline
-
-```typescript
-import {
-  SftpDataSource,
-  XMLParserService,
-  FileDiscoveryError,
-  FileParsingError,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-async function processSftpOrderFiles() {
-  let sftpSource: SftpDataSource | null = null;
-
-  try {
-    // ────────────────────────────────────────────────────────
-    // STAGE 1: Connection
-    // ────────────────────────────────────────────────────────
-    logger.info('Connecting to SFTP...');
-
-    sftpSource = new SftpDataSource(
-      {
-        type: 'SFTP_XML',
-        connectionId: 'vendor-sftp',
-        settings: {
-          host: process.env.SFTP_HOST!,
-          port: parseInt(process.env.SFTP_PORT || '22'),
-          username: process.env.SFTP_USERNAME!,
-          privateKey: process.env.SFTP_PRIVATE_KEY!,
-          passphrase: process.env.SFTP_PASSPHRASE,
-          remotePath: '/orders',
-        },
-      },
-      logger
-    );
-
-    // Validate connection
-    const isValid = await sftpSource.validateConnection();
-    if (!isValid) {
-      throw new Error('SFTP connection validation failed');
-    }
-    logger.info('✅ SFTP connection validated');
-
-    // ────────────────────────────────────────────────────────
-    // STAGE 2: Discovery
-    // ────────────────────────────────────────────────────────
-    logger.info('Listing XML files...');
-
-    const files = await sftpSource.listFiles({
-      pattern: /\.xml$/i,
-    });
-
-    if (files.length === 0) {
-      logger.info('No XML files found');
-      return;
-    }
-
-    logger.info(`Found ${files.length} XML files`);
-
-    // ────────────────────────────────────────────────────────
-    // STAGE 3: Reading & Parsing
-    // ────────────────────────────────────────────────────────
-    const parser = new XMLParserService();
-    const results = [];
-
-    for (const file of files) {
-      try {
-        logger.info(`Processing ${file.name}...`);
-
-        // Download file
-        const content = await sftpSource.downloadFile(file.path);
-
-        // Parse XML
-        const parsed = await parser.parse(content as string);
-
-        logger.info(`✅ Parsed XML from ${file.name}`);
-        results.push({ file: file.name, data: parsed });
-
-        // Archive processed file
-        await sftpSource.archiveFile(file.path, `/archive/${file.name}`);
-        logger.info(`Archived ${file.name}`);
-
-      } catch (error) {
-        if (error instanceof FileParsingError) {
-          logger.error(`❌ XML parsing failed for ${file.name}:`, {
-            code: error.code,
-            message: error.message,
-            lineNumber: error.lineNumber,
-          });
-
-          // Move to error folder
-          await sftpSource.archiveFile(file.path, `/errors/${file.name}`);
-          continue; // Process next file
-        }
-        throw error;
-      }
-    }
-
-    logger.info(`✅ Processed ${results.length}/${files.length} files`);
-    return results;
-
-  } catch (error) {
-    if (error instanceof FileDiscoveryError) {
-      logger.error('SFTP error:', {
-        code: error.code,
-        message: error.message,
-      });
-
-      if (error.code === 'FILE_ACCESS_DENIED') {
-        console.error('\n❌ SFTP Authentication Failed:');
-        console.error('  1. Verify SSH private key is correct');
-        console.error('  2. Check username is correct');
-        console.error('  3. Confirm passphrase (if key is encrypted)');
-        console.error('  4. Test connection: ssh username@host');
-      }
-
-      if (error.message?.includes('timeout')) {
-        console.error('\n❌ SFTP Connection Timeout:');
-        console.error('  1. Check SFTP host is reachable');
-        console.error('  2. Verify firewall allows outbound port 22');
-        console.error('  3. Test connectivity: telnet host 22');
-      }
-    }
-
-    throw error;
-  }
-}
-
-// Run the pipeline
-processSftpOrderFiles()
-  .then(() => logger.info('SFTP pipeline completed'))
-  .catch(error => {
-    logger.error('SFTP pipeline failed:', error);
-    process.exit(1);
-  });
-```
-
----
-
-## Retry Strategies
-
-### When to Retry
-
-```typescript
-function shouldRetry(error: unknown): boolean {
-  // 1. Network errors - ALWAYS retry
-  if (error instanceof IngestionError &&
-      error.code === IngestionErrorCode.NETWORK_ERROR) {
-    return true;
-  }
-
-  // 2. Timeout errors - ALWAYS retry
-  if (error instanceof IngestionError &&
-      error.code === IngestionErrorCode.TIMEOUT_ERROR) {
-    return true;
-  }
-
-  // 3. File discovery errors - SOMETIMES retry
-  if (error instanceof FileDiscoveryError) {
-    // Retry if transient (network, timeout)
-    if (error.message?.includes('timeout') ||
-        error.message?.includes('ECONNREFUSED')) {
-      return true;
-    }
-    // Don't retry permanent errors (access denied, not found)
-    return false;
-  }
-
-  // 4. Parsing errors - NEVER retry
-  if (error instanceof FileParsingError) {
-    return false;
-  }
-
-  // 5. Unknown errors - NEVER retry
-  return false;
-}
-```
-
-### Exponential Backoff with Jitter
-
-```typescript
-async function retryWithBackoff<T>(
-  operation: () => Promise<T>,
-  maxRetries = 3,
-  baseDelayMs = 1000
-): Promise<T> {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      return await operation(); // ✅ Success
-
-    } catch (error) {
-      const isLastAttempt = attempt === maxRetries;
-      const shouldRetryError = shouldRetry(error);
-
-      if (!shouldRetryError || isLastAttempt) {
-        throw error; // Give up
-      }
-
-      // Calculate delay with exponential backoff + jitter
-      const exponentialDelay = Math.pow(2, attempt) * baseDelayMs;
-      const jitter = Math.random() * 1000; // Random 0-1000ms
-      const delay = exponentialDelay + jitter;
-
-      logger.warn(`Attempt ${attempt}/${maxRetries} failed - retrying in ${Math.round(delay)}ms`, {
-        error: error instanceof Error ? error.message : String(error),
-      });
-
-      await new Promise(resolve => setTimeout(resolve, delay));
-    }
-  }
-
-  throw new Error('Unreachable'); // TypeScript exhaustiveness check
-}
-
-// Usage
-const content = await retryWithBackoff(
-  () => s3Source.downloadFile('inventory.csv'),
-  3, // max 3 retries
-  1000 // start with 1 second
-);
-```
-
----
-
-## Troubleshooting Flowchart
-
-```
-┌─────────────────────────────────────────────┐
-│ Error occurred during file operation        │
-└────────────────┬────────────────────────────┘
-                 │
-                 ▼
-         ┌───────────────┐
-         │ Error type?   │
-         └───┬───────────┘
-             │
-     ┌───────┼───────┬────────┬──────────┐
-     │       │       │        │          │
-     ▼       ▼       ▼        ▼          ▼
-┌─────────────────────────────────────────────┐
-│ FileDiscoveryError                          │
-├─────────────────────────────────────────────┤
-│ Code: FILE_ACCESS_DENIED                    │
-│ ├─ Check credentials                        │
-│ ├─ Verify IAM/SSH permissions               │
-│ └─ Confirm bucket/host exists               │
-│                                             │
-│ Code: FILE_NOT_FOUND                        │
-│ ├─ Verify file path is correct             │
-│ ├─ Check prefix/pattern                    │
-│ └─ File may have been moved/deleted        │
-│                                             │
-│ Code: FILE_DISCOVERY_FAILED                 │
-│ ├─ Check network connectivity               │
-│ ├─ Test with ping/telnet                   │
-│ └─ Retry with exponential backoff          │
-└─────────────────────────────────────────────┘
-
-┌─────────────────────────────────────────────┐
-│ IngestionError (NETWORK_ERROR)              │
-├─────────────────────────────────────────────┤
-│ Symptoms: Timeout, connection refused      │
-│ ├─ ✅ RETRYABLE: Yes                        │
-│ ├─ Check network connectivity               │
-│ ├─ Verify endpoint is reachable            │
-│ └─ Retry with exponential backoff          │
-└─────────────────────────────────────────────┘
-
-┌─────────────────────────────────────────────┐
-│ FileParsingError                            │
-├─────────────────────────────────────────────┤
-│ Code: PARSE_ERROR                           │
-│ ├─ ❌ NOT RETRYABLE                         │
-│ ├─ Check file format (XML/JSON/CSV valid?) │
-│ ├─ Look at lineNumber in error             │
-│ └─ Fix source data                         │
-│                                             │
-│ Code: ENCODING_ERROR                        │
-│ ├─ File may not be UTF-8                   │
-│ ├─ Try alternate encodings                 │
-│ └─ Convert file to UTF-8                   │
-└─────────────────────────────────────────────┘
-
-┌─────────────────────────────────────────────┐
-│ Memory Error (heap out of memory)           │
-├─────────────────────────────────────────────┤
-│ Symptoms: "JavaScript heap out of memory"  │
-│ ├─ File too large for available memory     │
-│ ├─ Solution 1: Increase heap size          │
-│ │   node --max-old-space-size=4096         │
-│ ├─ Solution 2: Use chunked processing      │
-│ │   Process file in 1MB chunks             │
-│ └─ Solution 3: Stream instead of buffer    │
-└─────────────────────────────────────────────┘
-```
-
----
-
-## Production Checklist
-
-### Pre-Deployment
-
-**Connection Setup:**
-- [ ] S3/SFTP credentials stored in environment variables (not hardcoded)
-- [ ] `validateConnection()` called before processing files
-- [ ] IAM policies grant minimum required permissions (least privilege)
-- [ ] SFTP uses SSH keys (not passwords)
-- [ ] Connection timeout configured appropriately
-
-**Error Handling:**
-- [ ] All file operations wrapped in try-catch blocks
-- [ ] Errors logged with structured data (`error.toJSON()`)
-- [ ] Retry logic implemented for transient errors
-- [ ] Exponential backoff with jitter configured
-- [ ] Max retry limit set (typically 3)
-- [ ] Circuit breaker pattern for repeated failures
-
-**File Processing:**
-- [ ] File existence checked before download
-- [ ] Large files processed in chunks (if > 100MB)
-- [ ] Memory limits configured (`--max-old-space-size`)
-- [ ] Parsed data validated before use
-- [ ] Failed files moved to error folder
-
-**Monitoring:**
-- [ ] Structured logging enabled
-- [ ] Error rates tracked and alerted
-- [ ] File processing metrics collected (count, size, duration)
-- [ ] Presigned URL expiry monitored (S3)
-
-### Post-Deployment
-
-**Validation:**
-- [ ] Successful file processing confirmed
-- [ ] No duplicate processing detected
-- [ ] Error handling working as expected
-- [ ] Retry logic functioning correctly
-- [ ] Memory usage stable
-
-**Monitoring:**
-- [ ] Logs being collected and searchable
-- [ ] Alerts triggering appropriately
-- [ ] Metrics dashboards created
-- [ ] On-call runbooks updated
-
----
-
-## Quick Reference
-
-### Error Type Decision Tree
-
-```
-Does file operation fail?
-  │
-  ├─ Connection/authentication fails
-  │  → FileDiscoveryError (FILE_ACCESS_DENIED)
-  │  → Check credentials, IAM policies, SSH keys
-  │
-  ├─ File not found
-  │  → FileDiscoveryError (FILE_NOT_FOUND)
-  │  → Verify path, check file exists
-  │
-  ├─ Network timeout
-  │  → IngestionError (NETWORK_ERROR)
-  │  → RETRY with exponential backoff
-  │
-  ├─ File downloaded but parsing fails
-  │  → FileParsingError (PARSE_ERROR)
-  │  → DON'T RETRY - fix source data
-  │
-  └─ Out of memory
-     → Memory Error
-     → Increase heap or use chunked processing
-```
-
-### Common Error Codes
-
-| Error Code | Retryable? | Action |
-|------------|------------|--------|
-| `FILE_ACCESS_DENIED` | ❌ No | Fix credentials/permissions |
-| `FILE_NOT_FOUND` | ❌ No | Verify file path |
-| `FILE_DISCOVERY_FAILED` | ✅ Yes | Retry with backoff |
-| `NETWORK_ERROR` | ✅ Yes | Retry with backoff |
-| `TIMEOUT_ERROR` | ✅ Yes | Retry with backoff |
-| `PARSE_ERROR` | ❌ No | Fix source data |
-| `ENCODING_ERROR` | ❌ No | Convert to UTF-8 |
-
----
-
-## Related Documentation
-
-- [Error Handling Guide](../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md) - Core error handling patterns
-- [Data Sources Guide](./data-sources-readme.md) - S3/SFTP configuration and usage
-- [Parsers Guide](../parsers/) - CSV/XML/JSON/Parquet parsing details
-- [Troubleshooting Quick Reference](../../00-START-HERE/troubleshooting-quick-reference.md) - General debugging
-
----
-
-**Need Help?**
-- 📖 Review [Complete Examples](#complete-examples)
-- 🔍 Check [Troubleshooting Flowchart](#troubleshooting-flowchart)
-- 📋 Use [Production Checklist](#production-checklist)
-- 🐛 Report issues at [GitHub](https://github.com/fluentcommerce/fc-connect-sdk/issues)
+# File Operations Error Handling Guide
+
+> **Complete guide to handling errors when reading and parsing files from S3 and SFTP**
+> Production-ready patterns for connection errors, file discovery, reading, and parsing failures
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Error Flow Diagram](#error-flow-diagram)
+- [Connection Errors](#connection-errors)
+- [Discovery Errors](#discovery-errors)
+- [Reading Errors](#reading-errors)
+- [Parsing Errors](#parsing-errors)
+- [Complete Examples](#complete-examples)
+- [Retry Strategies](#retry-strategies)
+- [Troubleshooting Flowchart](#troubleshooting-flowchart)
+- [Production Checklist](#production-checklist)
+
+---
+
+## Overview
+
+When working with S3 or SFTP data sources, errors can occur at multiple stages:
+
+```
+┌─────────────────┐
+│ 1. Connection   │ ← Credentials, network, timeouts
+├─────────────────┤
+│ 2. Discovery    │ ← File listing, permissions, paths
+├─────────────────┤
+│ 3. Reading      │ ← Download, memory, streaming
+├─────────────────┤
+│ 4. Parsing      │ ← CSV/XML/JSON/Parquet format errors
+└─────────────────┘
+```
+
+This guide covers **all four stages** with production-ready error handling patterns.
+
+### Key Error Types
+
+| Error Class | Stage | Retryable? | Common Causes |
+|-------------|-------|------------|---------------|
+| `FileDiscoveryError` | Connection/Discovery | ✅ Usually | Invalid credentials, network timeout, permissions |
+| `FileParsingError` | Parsing | ❌ No | Malformed XML/CSV/JSON, encoding issues |
+| `IngestionError` (NETWORK_ERROR) | Connection/Reading | ✅ Yes | Network timeout, API unreachable |
+| Memory errors | Reading | ⚠️ Sometimes | File too large, insufficient heap |
+
+---
+
+## Error Flow Diagram
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│ START: Read and Parse File from S3/SFTP                      │
+└────────────────────┬─────────────────────────────────────────┘
+                     │
+                     ▼
+         ┌──────────────────────┐
+         │ Create Data Source   │
+         │ (S3/SFTP)            │
+         └──────┬───────────────┘
+                │
+                ▼
+         ┌──────────────────────┐
+         │ Validate Connection  │
+         └──────┬───────────────┘
+                │
+                ├─────────── ❌ FileDiscoveryError
+                │             → Invalid credentials
+                │             → Network timeout
+                │             → Permissions denied
+                │             → RETRY? Yes (if transient)
+                │
+                ▼
+         ┌──────────────────────┐
+         │ List Files           │
+         │ (discovery)          │
+         └──────┬───────────────┘
+                │
+                ├─────────── ❌ FileDiscoveryError
+                │             → Bucket/path not found
+                │             → Access denied
+                │             → RETRY? Usually No (config error)
+                │
+                ▼
+         ┌──────────────────────┐
+         │ Download File        │
+         │ (read content)       │
+         └──────┬───────────────┘
+                │
+                ├─────────── ❌ IngestionError (NETWORK_ERROR)
+                │             → Network timeout
+                │             → Presigned URL expired (S3)
+                │             → SFTP connection lost
+                │             → RETRY? Yes
+                │
+                ├─────────── ❌ Memory Error
+                │             → File too large
+                │             → Insufficient heap
+                │             → RETRY? No (use streaming)
+                │
+                ▼
+         ┌──────────────────────┐
+         │ Parse Content        │
+         │ (CSV/XML/JSON)       │
+         └──────┬───────────────┘
+                │
+                ├─────────── ❌ FileParsingError
+                │             → Malformed XML (missing tags)
+                │             → Invalid JSON (syntax error)
+                │             → CSV column mismatch
+                │             → Encoding error (non-UTF-8)
+                │             → RETRY? No (fix source data)
+                │
+                ▼
+         ┌──────────────────────┐
+         │ ✅ SUCCESS           │
+         │ Parsed data ready    │
+         └──────────────────────┘
+```
+
+---
+
+## Connection Errors
+
+### S3 Connection Errors
+
+#### Error 1: Invalid Credentials
+
+**Symptoms:**
+```
+FileDiscoveryError: Access Denied
+  code: FILE_ACCESS_DENIED
+  message: "The AWS Access Key Id you provided does not exist in our records"
+```
+
+**Causes:**
+- Invalid `accessKeyId` or `secretAccessKey`
+- Credentials expired (if using temporary credentials)
+- Wrong AWS region specified
+
+**Solution:**
+```typescript
+import {
+  S3DataSource,
+  FileDiscoveryError,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+async function createS3Source() {
+  try {
+    const s3Source = new S3DataSource(
+      {
+        type: 'S3_CSV',
+        s3Config: {
+          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+          region: process.env.AWS_REGION || 'us-east-1',
+          bucket: process.env.S3_BUCKET,
+        },
+      },
+      logger
+    );
+
+    // ✅ IMPORTANT: Validate connection before proceeding
+    const isValid = await s3Source.validateConnection();
+    if (!isValid) {
+      throw new Error('S3 connection validation failed');
+    }
+
+    return s3Source;
+
+  } catch (error) {
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_ACCESS_DENIED') {
+      logger.error('S3 credentials invalid:', {
+        bucket: process.env.S3_BUCKET,
+        region: process.env.AWS_REGION,
+        // DO NOT log credentials!
+      });
+
+      // Don't retry - configuration error
+      throw new Error('S3 credentials are invalid. Check AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY');
+    }
+    throw error;
+  }
+}
+```
+
+**Debugging Checklist:**
+- [ ] Verify `AWS_ACCESS_KEY_ID` is correct
+- [ ] Verify `AWS_SECRET_ACCESS_KEY` is correct
+- [ ] Check if credentials are expired (temporary credentials)
+- [ ] Verify AWS region matches bucket region
+- [ ] Test credentials with AWS CLI: `aws s3 ls s3://your-bucket`
+
+#### Error 2: Wrong Region
+
+**Symptoms:**
+```
+FileDiscoveryError: The bucket is in this region: us-west-2.
+Please use this region to retry the request
+```
+
+**Solution:**
+```typescript
+// ❌ WRONG: Hardcoded region doesn't match bucket
+const s3Source = new S3DataSource({
+  type: 'S3_CSV',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: 'us-east-1', // ❌ Bucket is in us-west-2
+    bucket: 'my-bucket',
+  },
+}, logger);
+
+// ✅ CORRECT: Use environment variable or bucket's actual region
+const s3Source = new S3DataSource({
+  type: 'S3_CSV',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: process.env.AWS_REGION!, // ✅ Set to 'us-west-2'
+    bucket: 'my-bucket',
+  },
+}, logger);
+```
+
+#### Error 3: Network Timeout (S3)
+
+**Symptoms:**
+```
+IngestionError: Network request timed out
+  code: NETWORK_ERROR
+  isRetryable: true
+```
+
+**Solution with Retry:**
+```typescript
+import { IngestionError, IngestionErrorCode } from '@fluentcommerce/fc-connect-sdk';
+
+async function downloadWithRetry(
+  s3Source: S3DataSource,
+  filePath: string,
+  maxRetries = 3
+) {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      const content = await s3Source.downloadFile(filePath);
+      return content; // ✅ Success
+
+    } catch (error) {
+      if (error instanceof IngestionError &&
+          error.code === IngestionErrorCode.NETWORK_ERROR &&
+          error.isRetryable() &&
+          attempt < maxRetries) {
+
+        const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
+        logger.warn(`S3 network error - retrying in ${delay}ms (attempt ${attempt}/${maxRetries})`);
+        await new Promise(resolve => setTimeout(resolve, delay));
+        continue;
+      }
+
+      throw error; // Not retryable or max retries exceeded
+    }
+  }
+}
+```
+
+### SFTP Connection Errors
+
+#### Error 1: Authentication Failed
+
+**Symptoms:**
+```
+FileDiscoveryError: All configured authentication methods failed
+  code: FILE_ACCESS_DENIED
+```
+
+**Causes:**
+- Invalid SSH private key
+- Wrong username
+- Private key passphrase incorrect
+- Host key verification failed
+
+**Solution:**
+```typescript
+import { SftpDataSource, FileDiscoveryError } from '@fluentcommerce/fc-connect-sdk';
+
+async function createSftpSource() {
+  try {
+    const sftpSource = new SftpDataSource(
+      {
+        type: 'SFTP_CSV',
+        connectionId: 'vendor-sftp',
+        settings: {
+          host: process.env.SFTP_HOST!,
+          port: parseInt(process.env.SFTP_PORT || '22'),
+          username: process.env.SFTP_USERNAME!,
+          privateKey: process.env.SFTP_PRIVATE_KEY!, // ✅ Use SSH key (not password)
+          passphrase: process.env.SFTP_PASSPHRASE, // If key is encrypted
+          remotePath: process.env.SFTP_REMOTE_PATH || '/data',
+        },
+      },
+      logger
+    );
+
+    // ✅ Validate connection
+    const isValid = await sftpSource.validateConnection();
+    if (!isValid) {
+      throw new Error('SFTP connection validation failed');
+    }
+
+    return sftpSource;
+
+  } catch (error) {
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_ACCESS_DENIED') {
+      logger.error('SFTP authentication failed:', {
+        host: process.env.SFTP_HOST,
+        username: process.env.SFTP_USERNAME,
+        // DO NOT log private key or passphrase!
+      });
+
+      // Check common issues
+      throw new Error(
+        'SFTP authentication failed. Check:\n' +
+        '1. SSH private key is correct\n' +
+        '2. Username is correct\n' +
+        '3. Passphrase is correct (if key is encrypted)\n' +
+        '4. Host key is trusted'
+      );
+    }
+    throw error;
+  }
+}
+```
+
+#### Error 2: Connection Timeout (SFTP)
+
+**Symptoms:**
+```
+FileDiscoveryError: Connection timeout after 30000ms
+  code: FILE_DISCOVERY_FAILED
+```
+
+**Causes:**
+- SFTP server unreachable
+- Firewall blocking port 22
+- Network connectivity issues
+- DNS resolution failure
+
+**Solution:**
+```typescript
+async function testSftpConnectivity() {
+  const sftpConfig = {
+    type: 'SFTP_CSV' as const,
+    connectionId: 'test-sftp',
+    settings: {
+      host: process.env.SFTP_HOST!,
+      port: 22,
+      username: process.env.SFTP_USERNAME!,
+      privateKey: process.env.SFTP_PRIVATE_KEY!,
+      remotePath: '/data',
+      timeout: 30000, // 30 seconds
+    },
+  };
+
+  try {
+    const sftpSource = new SftpDataSource(sftpConfig, logger);
+
+    logger.info('Testing SFTP connectivity...');
+    const isValid = await sftpSource.validateConnection();
+
+    if (isValid) {
+      logger.info('✅ SFTP connection successful');
+      return sftpSource;
+    } else {
+      throw new Error('SFTP validation returned false');
+    }
+
+  } catch (error) {
+    if (error instanceof FileDiscoveryError) {
+      logger.error('SFTP connection failed:', {
+        host: sftpConfig.settings.host,
+        port: sftpConfig.settings.port,
+        error: error.message,
+      });
+
+      // Debugging steps
+      console.error('\n🔍 Debugging steps:');
+      console.error(`1. Test DNS: ping ${sftpConfig.settings.host}`);
+      console.error(`2. Test connectivity: telnet ${sftpConfig.settings.host} 22`);
+      console.error('3. Check firewall allows outbound port 22');
+      console.error('4. Verify SFTP server is running');
+    }
+    throw error;
+  }
+}
+```
+
+---
+
+## Discovery Errors
+
+### Error 1: File Not Found
+
+**Symptoms:**
+```
+FileDiscoveryError: File not found
+  code: FILE_NOT_FOUND
+  path: "s3://my-bucket/inventory/missing.csv"
+```
+
+**Solution:**
+```typescript
+async function safeFileDownload(
+  dataSource: S3DataSource | SftpDataSource,
+  filePath: string
+) {
+  try {
+    // Check if file exists first
+    const files = await dataSource.listFiles({ prefix: filePath });
+
+    if (files.length === 0) {
+      logger.warn(`File not found: ${filePath}`);
+      return null; // Return null instead of throwing
+    }
+
+    // File exists - download it
+    const content = await dataSource.downloadFile(filePath);
+    return content;
+
+  } catch (error) {
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_NOT_FOUND') {
+      logger.warn('File not found during download:', { path: filePath });
+      return null; // Graceful handling
+    }
+    throw error; // Re-throw other errors
+  }
+}
+```
+
+### Error 2: Access Denied (Permissions)
+
+**Symptoms:**
+```
+FileDiscoveryError: Access Denied
+  code: FILE_ACCESS_DENIED
+  message: "User is not authorized to perform: s3:GetObject"
+```
+
+**IAM Policy Required (S3):**
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:ListBucket",
+        "s3:GetObject",
+        "s3:GetObjectVersion"
+      ],
+      "Resource": [
+        "arn:aws:s3:::my-bucket",
+        "arn:aws:s3:::my-bucket/*"
+      ]
+    }
+  ]
+}
+```
+
+**Solution:**
+```typescript
+async function validateS3Permissions(s3Source: S3DataSource) {
+  try {
+    logger.info('Validating S3 permissions...');
+
+    // Test 1: Can list bucket?
+    try {
+      await s3Source.listFiles({ prefix: '' });
+      logger.info('✅ s3:ListBucket permission OK');
+    } catch (error) {
+      logger.error('❌ s3:ListBucket permission DENIED');
+      throw error;
+    }
+
+    // Test 2: Can read objects?
+    const files = await s3Source.listFiles({ prefix: '' });
+    if (files.length > 0) {
+      try {
+        await s3Source.downloadFile(files[0].path);
+        logger.info('✅ s3:GetObject permission OK');
+      } catch (error) {
+        logger.error('❌ s3:GetObject permission DENIED');
+        throw error;
+      }
+    }
+
+    logger.info('✅ All S3 permissions validated');
+
+  } catch (error) {
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_ACCESS_DENIED') {
+      console.error('\n❌ IAM Permissions Required:');
+      console.error('  - s3:ListBucket on arn:aws:s3:::my-bucket');
+      console.error('  - s3:GetObject on arn:aws:s3:::my-bucket/*');
+      console.error('\nAdd this IAM policy to your AWS user/role.');
+    }
+    throw error;
+  }
+}
+```
+
+---
+
+## Reading Errors
+
+### Error 1: Out of Memory (Large Files)
+
+**Symptoms:**
+```
+FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
+```
+
+**Causes:**
+- File too large to load in memory
+- Default Node.js heap size (512MB - 2GB)
+- Multiple large files processed concurrently
+
+**Solution 1: Increase Heap Size**
+```bash
+# Temporary fix - increase Node.js heap
+node --max-old-space-size=4096 your-script.js  # 4GB heap
+```
+
+**Solution 2: Stream Large Files (Best Practice)**
+```typescript
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+import { Readable } from 'stream';
+
+async function processLargeFile(s3Source: S3DataSource, filePath: string) {
+  try {
+    // Option 1: Download in binary mode (Buffer)
+    const content = await s3Source.downloadFile(filePath, { encoding: 'binary' });
+
+    // Process in chunks
+    const buffer = content as Buffer;
+    const chunkSize = 1024 * 1024; // 1MB chunks
+
+    for (let offset = 0; offset < buffer.length; offset += chunkSize) {
+      const chunk = buffer.slice(offset, offset + chunkSize);
+      await processChunk(chunk);
+
+      // Allow garbage collection
+      if (global.gc) global.gc();
+    }
+
+  } catch (error) {
+    if (error.message?.includes('heap out of memory')) {
+      logger.error('Out of memory processing large file:', {
+        file: filePath,
+        suggestion: 'Use chunked processing or increase heap size',
+      });
+
+      throw new Error(
+        'File too large for available memory. ' +
+        'Increase heap size with: node --max-old-space-size=4096'
+      );
+    }
+    throw error;
+  }
+}
+
+async function processChunk(chunk: Buffer) {
+  // Process chunk here
+  logger.debug(`Processing chunk: ${chunk.length} bytes`);
+}
+```
+
+### Error 2: Presigned URL Expired (S3)
+
+**Symptoms:**
+```
+FileDiscoveryError: Request has expired
+  code: FILE_ACCESS_DENIED
+  statusCode: 403
+```
+
+**Cause:**
+- Presigned URLs expire after 1 hour (SDK default)
+- Long-running operations exceed URL lifetime
+
+**Solution:**
+```typescript
+async function downloadWithUrlRefresh(s3Source: S3DataSource, filePath: string) {
+  const startTime = Date.now();
+  const URL_LIFETIME_MS = 55 * 60 * 1000; // 55 minutes (buffer before 1 hour expiry)
+
+  try {
+    const content = await s3Source.downloadFile(filePath);
+    return content;
+
+  } catch (error) {
+    const elapsed = Date.now() - startTime;
+
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_ACCESS_DENIED' &&
+        error.message?.includes('expired') &&
+        elapsed > URL_LIFETIME_MS) {
+
+      logger.warn('Presigned URL expired - recreating data source');
+
+      // Recreate S3DataSource (generates new presigned URLs)
+      const newS3Source = new S3DataSource(s3Source.getConfig(), logger);
+
+      // Retry with new URL
+      return await newS3Source.downloadFile(filePath);
+    }
+
+    throw error;
+  }
+}
+```
+
+---
+
+## Parsing Errors
+
+### Error 1: Malformed XML
+
+**Symptoms:**
+```
+FileParsingError: Failed to parse XML: Unclosed tag 'customer-email'
+  fileName: "order.xml"
+  lineNumber: 5
+  code: PARSE_ERROR
+```
+
+**Example Malformed XML:**
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<order>
+  <order-id>12345</order-id>
+  <customer>
+    <email>test@example.com
+    <!-- ❌ Missing closing tag for <email> and <customer> -->
+```
+
+**Solution:**
+```typescript
+import { XMLParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
+
+async function parseXMLWithValidation(xmlContent: string, fileName: string) {
+  const parser = new XMLParserService();
+
+  try {
+    const parsed = await parser.parse(xmlContent);
+    return parsed;
+
+  } catch (error) {
+    if (error instanceof FileParsingError &&
+        error.code === 'PARSE_ERROR') {
+      logger.error('XML parsing failed:', {
+        fileName: error.fileName || fileName,
+        lineNumber: error.lineNumber,
+        message: error.message,
+        // Show first 200 chars of XML for debugging
+        xmlPreview: xmlContent.substring(0, 200),
+      });
+
+      throw new Error(
+        `Invalid XML in ${fileName}:\n` +
+        `  Line ${error.lineNumber}: ${error.message}\n` +
+        `  Fix: Ensure all XML tags are properly closed`
+      );
+    }
+    throw error;
+  }
+}
+```
+
+**Common XML Errors:**
+| Error | Cause | Fix |
+|-------|-------|-----|
+| `Unclosed tag 'tagname'` | Missing `</tagname>` | Add closing tag |
+| `Unexpected closing tag` | Extra `</tagname>` | Remove duplicate closing tag |
+| `Invalid character` | Special chars not escaped | Use `&amp;`, `&lt;`, `&gt;`, `&quot;`, `&apos;` |
+| `Text not allowed here` | Text outside tags | Wrap text in tags |
+
+### Error 2: Invalid JSON
+
+**Symptoms:**
+```
+FileParsingError: Failed to parse JSON: Unexpected token a in JSON at position 45
+  fileName: "order.json"
+  code: PARSE_ERROR
+```
+
+**Example Invalid JSON:**
+```json
+{
+  "orderId": "12345",
+  "status": active,     // ❌ Unquoted string
+  "items": [
+    {
+      "sku": "ABC-123",
+      "qty": 5,         // ❌ Trailing comma
+    }
+  ],                   // ❌ Trailing comma
+}
+```
+
+**Solution:**
+```typescript
+import { JSONParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
+
+async function parseJSONWithValidation(jsonContent: string, fileName: string) {
+  const parser = new JSONParserService();
+
+  try {
+    const parsed = await parser.parse(jsonContent);
+    return parsed;
+
+  } catch (error) {
+    if (error instanceof FileParsingError &&
+        error.code === 'PARSE_ERROR') {
+
+      // Extract position from error message
+      const positionMatch = error.message.match(/position (\d+)/);
+      const position = positionMatch ? parseInt(positionMatch[1]) : 0;
+
+      // Show context around error
+      const start = Math.max(0, position - 50);
+      const end = Math.min(jsonContent.length, position + 50);
+      const context = jsonContent.substring(start, end);
+
+      logger.error('JSON parsing failed:', {
+        fileName: fileName,
+        position: position,
+        context: context,
+        message: error.message,
+      });
+
+      throw new Error(
+        `Invalid JSON in ${fileName}:\n` +
+        `  ${error.message}\n` +
+        `  Context: ...${context}...`
+      );
+    }
+    throw error;
+  }
+}
+```
+
+**Common JSON Errors:**
+| Error | Cause | Fix |
+|-------|-------|-----|
+| `Unexpected token` | Unquoted string, trailing comma | Quote strings, remove trailing commas |
+| `Unexpected end of JSON` | Missing closing `}` or `]` | Add closing bracket |
+| `Expected property name` | Missing quotes on key | Quote object keys: `"key": value` |
+
+### Error 3: CSV Column Mismatch
+
+**Symptoms:**
+```
+FileParsingError: CSV column count mismatch
+  fileName: "inventory.csv"
+  code: INVALID_CSV_FORMAT
+  expectedColumns: 5
+  actualColumns: 4
+```
+
+**Example Invalid CSV:**
+```csv
+sku,location,quantity,date,status
+SKU-001,LOC-01,100,2024-01-01,ACTIVE
+SKU-002,LOC-02,50,2024-01-02          ← Missing 'status' column
+SKU-003,LOC-03,75,2024-01-03,ACTIVE
+```
+
+**Solution:**
+```typescript
+import { CSVParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
+
+async function parseCSVWithValidation(
+  csvContent: string,
+  fileName: string,
+  expectedColumns?: string[]
+) {
+  const parser = new CSVParserService();
+
+  try {
+    const parsed = await parser.parse(csvContent);
+
+    // Optional: Validate column names
+    if (expectedColumns && parsed.length > 0) {
+      const actualColumns = Object.keys(parsed[0]);
+      const missingColumns = expectedColumns.filter(col => !actualColumns.includes(col));
+
+      if (missingColumns.length > 0) {
+        throw new FileParsingError(
+          `CSV missing required columns: ${missingColumns.join(', ')}`,
+          'INVALID_CSV_FORMAT',
+          { fileName, missingColumns, actualColumns }
+        );
+      }
+    }
+
+    return parsed;
+
+  } catch (error) {
+    if (error instanceof FileParsingError &&
+        error.code === 'INVALID_CSV_FORMAT') {
+      logger.error('CSV parsing failed:', {
+        fileName: fileName,
+        message: error.message,
+        context: error.context,
+      });
+
+      throw new Error(
+        `Invalid CSV in ${fileName}:\n` +
+        `  ${error.message}\n` +
+        `  Fix: Ensure all rows have the same number of columns`
+      );
+    }
+    throw error;
+  }
+}
+
+// Usage
+const expectedColumns = ['sku', 'location', 'quantity', 'date', 'status'];
+const parsed = await parseCSVWithValidation(csvContent, 'inventory.csv', expectedColumns);
+```
+
+### Error 4: Encoding Issues (Non-UTF-8)
+
+**Symptoms:**
+```
+FileParsingError: Invalid UTF-8 sequence
+  code: ENCODING_ERROR
+```
+
+**Cause:**
+- File encoded in ISO-8859-1, Windows-1252, or other non-UTF-8 encoding
+- Special characters corrupted
+
+**Solution:**
+```typescript
+import { XMLParserService } from '@fluentcommerce/fc-connect-sdk';
+import iconv from 'iconv-lite';
+
+async function parseWithEncodingDetection(
+  fileContent: Buffer | string,
+  fileName: string
+) {
+  const parser = new XMLParserService();
+
+  try {
+    // Try UTF-8 first (SDK default)
+    const content = typeof fileContent === 'string' ? fileContent : fileContent.toString('utf-8');
+    return await parser.parse(content);
+
+  } catch (error) {
+    if (error instanceof FileParsingError &&
+        error.code === 'ENCODING_ERROR') {
+
+      logger.warn('UTF-8 parsing failed - trying alternate encodings');
+
+      // Try common alternate encodings
+      const encodings = ['iso-8859-1', 'windows-1252', 'utf-16le'];
+
+      for (const encoding of encodings) {
+        try {
+          const buffer = typeof fileContent === 'string' ?
+            Buffer.from(fileContent, 'binary') : fileContent;
+
+          const converted = iconv.decode(buffer, encoding);
+          const parsed = await parser.parse(converted);
+
+          logger.info(`✅ Successfully parsed with ${encoding} encoding`);
+          return parsed;
+
+        } catch (e) {
+          continue; // Try next encoding
+        }
+      }
+
+      // All encodings failed
+      throw new Error(
+        `Could not parse ${fileName} with any encoding. ` +
+        `Tried: UTF-8, ${encodings.join(', ')}`
+      );
+    }
+    throw error;
+  }
+}
+```
+
+**Note:** The SDK automatically handles UTF-8 BOM (Byte Order Mark) removal, so BOM presence is not an error.
+
+---
+
+## Complete Examples
+
+### Example 1: Production-Ready S3 CSV Pipeline
+
+```typescript
+import {
+  S3DataSource,
+  CSVParserService,
+  FileDiscoveryError,
+  FileParsingError,
+  IngestionError,
+  IngestionErrorCode,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+async function processS3InventoryFiles() {
+  let s3Source: S3DataSource | null = null;
+
+  try {
+    // ────────────────────────────────────────────────────────
+    // STAGE 1: Connection
+    // ────────────────────────────────────────────────────────
+    logger.info('Creating S3 data source...');
+
+    s3Source = new S3DataSource(
+      {
+        type: 'S3_CSV',
+        s3Config: {
+          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+          region: process.env.AWS_REGION!,
+          bucket: process.env.S3_BUCKET!,
+        },
+      },
+      logger
+    );
+
+    // Validate connection
+    const isValid = await s3Source.validateConnection();
+    if (!isValid) {
+      throw new Error('S3 connection validation failed');
+    }
+    logger.info('✅ S3 connection validated');
+
+    // ────────────────────────────────────────────────────────
+    // STAGE 2: Discovery
+    // ────────────────────────────────────────────────────────
+    logger.info('Discovering files...');
+
+    const files = await s3Source.listFiles({
+      prefix: 'inventory/',
+      pattern: /\.csv$/i,
+    });
+
+    if (files.length === 0) {
+      logger.warn('No CSV files found in inventory/ folder');
+      return;
+    }
+
+    logger.info(`Found ${files.length} CSV files`);
+
+    // ────────────────────────────────────────────────────────
+    // STAGE 3: Reading & Parsing
+    // ────────────────────────────────────────────────────────
+    const parser = new CSVParserService();
+    const results = [];
+
+    for (const file of files) {
+      try {
+        logger.info(`Processing ${file.name}...`);
+
+        // Download file with retry
+        const content = await downloadWithRetry(s3Source, file.path, 3);
+
+        // Parse CSV
+        const parsed = await parser.parse(content as string);
+
+        logger.info(`✅ Parsed ${parsed.length} records from ${file.name}`);
+        results.push({ file: file.name, records: parsed });
+
+      } catch (error) {
+        if (error instanceof FileParsingError) {
+          logger.error(`❌ Failed to parse ${file.name}:`, {
+            code: error.code,
+            message: error.message,
+            lineNumber: error.lineNumber,
+          });
+          // Continue with next file (don't fail entire batch)
+          continue;
+        }
+        throw error; // Re-throw other errors
+      }
+    }
+
+    logger.info(`✅ Successfully processed ${results.length}/${files.length} files`);
+    return results;
+
+  } catch (error) {
+    // ────────────────────────────────────────────────────────
+    // STAGE 4: Error Handling
+    // ────────────────────────────────────────────────────────
+    if (error instanceof FileDiscoveryError) {
+      logger.error('File discovery error:', {
+        code: error.code,
+        message: error.message,
+        path: error.path,
+      });
+
+      if (error.code === 'FILE_ACCESS_DENIED') {
+        console.error('\n❌ S3 Access Denied:');
+        console.error('  1. Check AWS credentials are correct');
+        console.error('  2. Verify IAM policy allows s3:ListBucket and s3:GetObject');
+        console.error('  3. Confirm bucket exists and region is correct');
+      }
+
+    } else if (error instanceof IngestionError) {
+      logger.error('Ingestion error:', error.toJSON());
+
+      if (error.isRetryable()) {
+        console.error('\n⚠️  Retryable error - consider running again');
+      }
+
+    } else {
+      logger.error('Unexpected error:', error);
+    }
+
+    throw error;
+  }
+}
+
+// Helper: Download with retry (from earlier example)
+async function downloadWithRetry(
+  s3Source: S3DataSource,
+  filePath: string,
+  maxRetries = 3
+) {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await s3Source.downloadFile(filePath);
+    } catch (error) {
+      if (error instanceof IngestionError &&
+          error.code === IngestionErrorCode.NETWORK_ERROR &&
+          error.isRetryable() &&
+          attempt < maxRetries) {
+        const delay = Math.pow(2, attempt) * 1000;
+        logger.warn(`Retry attempt ${attempt}/${maxRetries} in ${delay}ms`);
+        await new Promise(resolve => setTimeout(resolve, delay));
+        continue;
+      }
+      throw error;
+    }
+  }
+}
+
+// Run the pipeline
+processS3InventoryFiles()
+  .then(() => logger.info('Pipeline completed successfully'))
+  .catch(error => {
+    logger.error('Pipeline failed:', error);
+    process.exit(1);
+  });
+```
+
+### Example 2: Production-Ready SFTP XML Pipeline
+
+```typescript
+import {
+  SftpDataSource,
+  XMLParserService,
+  FileDiscoveryError,
+  FileParsingError,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+async function processSftpOrderFiles() {
+  let sftpSource: SftpDataSource | null = null;
+
+  try {
+    // ────────────────────────────────────────────────────────
+    // STAGE 1: Connection
+    // ────────────────────────────────────────────────────────
+    logger.info('Connecting to SFTP...');
+
+    sftpSource = new SftpDataSource(
+      {
+        type: 'SFTP_XML',
+        connectionId: 'vendor-sftp',
+        settings: {
+          host: process.env.SFTP_HOST!,
+          port: parseInt(process.env.SFTP_PORT || '22'),
+          username: process.env.SFTP_USERNAME!,
+          privateKey: process.env.SFTP_PRIVATE_KEY!,
+          passphrase: process.env.SFTP_PASSPHRASE,
+          remotePath: '/orders',
+        },
+      },
+      logger
+    );
+
+    // Validate connection
+    const isValid = await sftpSource.validateConnection();
+    if (!isValid) {
+      throw new Error('SFTP connection validation failed');
+    }
+    logger.info('✅ SFTP connection validated');
+
+    // ────────────────────────────────────────────────────────
+    // STAGE 2: Discovery
+    // ────────────────────────────────────────────────────────
+    logger.info('Listing XML files...');
+
+    const files = await sftpSource.listFiles({
+      pattern: /\.xml$/i,
+    });
+
+    if (files.length === 0) {
+      logger.info('No XML files found');
+      return;
+    }
+
+    logger.info(`Found ${files.length} XML files`);
+
+    // ────────────────────────────────────────────────────────
+    // STAGE 3: Reading & Parsing
+    // ────────────────────────────────────────────────────────
+    const parser = new XMLParserService();
+    const results = [];
+
+    for (const file of files) {
+      try {
+        logger.info(`Processing ${file.name}...`);
+
+        // Download file
+        const content = await sftpSource.downloadFile(file.path);
+
+        // Parse XML
+        const parsed = await parser.parse(content as string);
+
+        logger.info(`✅ Parsed XML from ${file.name}`);
+        results.push({ file: file.name, data: parsed });
+
+        // Archive processed file
+        await sftpSource.archiveFile(file.path, `/archive/${file.name}`);
+        logger.info(`Archived ${file.name}`);
+
+      } catch (error) {
+        if (error instanceof FileParsingError) {
+          logger.error(`❌ XML parsing failed for ${file.name}:`, {
+            code: error.code,
+            message: error.message,
+            lineNumber: error.lineNumber,
+          });
+
+          // Move to error folder
+          await sftpSource.archiveFile(file.path, `/errors/${file.name}`);
+          continue; // Process next file
+        }
+        throw error;
+      }
+    }
+
+    logger.info(`✅ Processed ${results.length}/${files.length} files`);
+    return results;
+
+  } catch (error) {
+    if (error instanceof FileDiscoveryError) {
+      logger.error('SFTP error:', {
+        code: error.code,
+        message: error.message,
+      });
+
+      if (error.code === 'FILE_ACCESS_DENIED') {
+        console.error('\n❌ SFTP Authentication Failed:');
+        console.error('  1. Verify SSH private key is correct');
+        console.error('  2. Check username is correct');
+        console.error('  3. Confirm passphrase (if key is encrypted)');
+        console.error('  4. Test connection: ssh username@host');
+      }
+
+      if (error.message?.includes('timeout')) {
+        console.error('\n❌ SFTP Connection Timeout:');
+        console.error('  1. Check SFTP host is reachable');
+        console.error('  2. Verify firewall allows outbound port 22');
+        console.error('  3. Test connectivity: telnet host 22');
+      }
+    }
+
+    throw error;
+  }
+}
+
+// Run the pipeline
+processSftpOrderFiles()
+  .then(() => logger.info('SFTP pipeline completed'))
+  .catch(error => {
+    logger.error('SFTP pipeline failed:', error);
+    process.exit(1);
+  });
+```
+
+---
+
+## Retry Strategies
+
+### When to Retry
+
+```typescript
+function shouldRetry(error: unknown): boolean {
+  // 1. Network errors - ALWAYS retry
+  if (error instanceof IngestionError &&
+      error.code === IngestionErrorCode.NETWORK_ERROR) {
+    return true;
+  }
+
+  // 2. Timeout errors - ALWAYS retry
+  if (error instanceof IngestionError &&
+      error.code === IngestionErrorCode.TIMEOUT_ERROR) {
+    return true;
+  }
+
+  // 3. File discovery errors - SOMETIMES retry
+  if (error instanceof FileDiscoveryError) {
+    // Retry if transient (network, timeout)
+    if (error.message?.includes('timeout') ||
+        error.message?.includes('ECONNREFUSED')) {
+      return true;
+    }
+    // Don't retry permanent errors (access denied, not found)
+    return false;
+  }
+
+  // 4. Parsing errors - NEVER retry
+  if (error instanceof FileParsingError) {
+    return false;
+  }
+
+  // 5. Unknown errors - NEVER retry
+  return false;
+}
+```
+
+### Exponential Backoff with Jitter
+
+```typescript
+async function retryWithBackoff<T>(
+  operation: () => Promise<T>,
+  maxRetries = 3,
+  baseDelayMs = 1000
+): Promise<T> {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await operation(); // ✅ Success
+
+    } catch (error) {
+      const isLastAttempt = attempt === maxRetries;
+      const shouldRetryError = shouldRetry(error);
+
+      if (!shouldRetryError || isLastAttempt) {
+        throw error; // Give up
+      }
+
+      // Calculate delay with exponential backoff + jitter
+      const exponentialDelay = Math.pow(2, attempt) * baseDelayMs;
+      const jitter = Math.random() * 1000; // Random 0-1000ms
+      const delay = exponentialDelay + jitter;
+
+      logger.warn(`Attempt ${attempt}/${maxRetries} failed - retrying in ${Math.round(delay)}ms`, {
+        error: error instanceof Error ? error.message : String(error),
+      });
+
+      await new Promise(resolve => setTimeout(resolve, delay));
+    }
+  }
+
+  throw new Error('Unreachable'); // TypeScript exhaustiveness check
+}
+
+// Usage
+const content = await retryWithBackoff(
+  () => s3Source.downloadFile('inventory.csv'),
+  3, // max 3 retries
+  1000 // start with 1 second
+);
+```
+
+---
+
+## Troubleshooting Flowchart
+
+```
+┌─────────────────────────────────────────────┐
+│ Error occurred during file operation        │
+└────────────────┬────────────────────────────┘
+                 │
+                 ▼
+         ┌───────────────┐
+         │ Error type?   │
+         └───┬───────────┘
+             │
+     ┌───────┼───────┬────────┬──────────┐
+     │       │       │        │          │
+     ▼       ▼       ▼        ▼          ▼
+┌─────────────────────────────────────────────┐
+│ FileDiscoveryError                          │
+├─────────────────────────────────────────────┤
+│ Code: FILE_ACCESS_DENIED                    │
+│ ├─ Check credentials                        │
+│ ├─ Verify IAM/SSH permissions               │
+│ └─ Confirm bucket/host exists               │
+│                                             │
+│ Code: FILE_NOT_FOUND                        │
+│ ├─ Verify file path is correct             │
+│ ├─ Check prefix/pattern                    │
+│ └─ File may have been moved/deleted        │
+│                                             │
+│ Code: FILE_DISCOVERY_FAILED                 │
+│ ├─ Check network connectivity               │
+│ ├─ Test with ping/telnet                   │
+│ └─ Retry with exponential backoff          │
+└─────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────┐
+│ IngestionError (NETWORK_ERROR)              │
+├─────────────────────────────────────────────┤
+│ Symptoms: Timeout, connection refused      │
+│ ├─ ✅ RETRYABLE: Yes                        │
+│ ├─ Check network connectivity               │
+│ ├─ Verify endpoint is reachable            │
+│ └─ Retry with exponential backoff          │
+└─────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────┐
+│ FileParsingError                            │
+├─────────────────────────────────────────────┤
+│ Code: PARSE_ERROR                           │
+│ ├─ ❌ NOT RETRYABLE                         │
+│ ├─ Check file format (XML/JSON/CSV valid?) │
+│ ├─ Look at lineNumber in error             │
+│ └─ Fix source data                         │
+│                                             │
+│ Code: ENCODING_ERROR                        │
+│ ├─ File may not be UTF-8                   │
+│ ├─ Try alternate encodings                 │
+│ └─ Convert file to UTF-8                   │
+└─────────────────────────────────────────────┘
+
+┌─────────────────────────────────────────────┐
+│ Memory Error (heap out of memory)           │
+├─────────────────────────────────────────────┤
+│ Symptoms: "JavaScript heap out of memory"  │
+│ ├─ File too large for available memory     │
+│ ├─ Solution 1: Increase heap size          │
+│ │   node --max-old-space-size=4096         │
+│ ├─ Solution 2: Use chunked processing      │
+│ │   Process file in 1MB chunks             │
+│ └─ Solution 3: Stream instead of buffer    │
+└─────────────────────────────────────────────┘
+```
+
+---
+
+## Production Checklist
+
+### Pre-Deployment
+
+**Connection Setup:**
+- [ ] S3/SFTP credentials stored in environment variables (not hardcoded)
+- [ ] `validateConnection()` called before processing files
+- [ ] IAM policies grant minimum required permissions (least privilege)
+- [ ] SFTP uses SSH keys (not passwords)
+- [ ] Connection timeout configured appropriately
+
+**Error Handling:**
+- [ ] All file operations wrapped in try-catch blocks
+- [ ] Errors logged with structured data (`error.toJSON()`)
+- [ ] Retry logic implemented for transient errors
+- [ ] Exponential backoff with jitter configured
+- [ ] Max retry limit set (typically 3)
+- [ ] Circuit breaker pattern for repeated failures
+
+**File Processing:**
+- [ ] File existence checked before download
+- [ ] Large files processed in chunks (if > 100MB)
+- [ ] Memory limits configured (`--max-old-space-size`)
+- [ ] Parsed data validated before use
+- [ ] Failed files moved to error folder
+
+**Monitoring:**
+- [ ] Structured logging enabled
+- [ ] Error rates tracked and alerted
+- [ ] File processing metrics collected (count, size, duration)
+- [ ] Presigned URL expiry monitored (S3)
+
+### Post-Deployment
+
+**Validation:**
+- [ ] Successful file processing confirmed
+- [ ] No duplicate processing detected
+- [ ] Error handling working as expected
+- [ ] Retry logic functioning correctly
+- [ ] Memory usage stable
+
+**Monitoring:**
+- [ ] Logs being collected and searchable
+- [ ] Alerts triggering appropriately
+- [ ] Metrics dashboards created
+- [ ] On-call runbooks updated
+
+---
+
+## Quick Reference
+
+### Error Type Decision Tree
+
+```
+Does file operation fail?
+  │
+  ├─ Connection/authentication fails
+  │  → FileDiscoveryError (FILE_ACCESS_DENIED)
+  │  → Check credentials, IAM policies, SSH keys
+  │
+  ├─ File not found
+  │  → FileDiscoveryError (FILE_NOT_FOUND)
+  │  → Verify path, check file exists
+  │
+  ├─ Network timeout
+  │  → IngestionError (NETWORK_ERROR)
+  │  → RETRY with exponential backoff
+  │
+  ├─ File downloaded but parsing fails
+  │  → FileParsingError (PARSE_ERROR)
+  │  → DON'T RETRY - fix source data
+  │
+  └─ Out of memory
+     → Memory Error
+     → Increase heap or use chunked processing
+```
+
+### Common Error Codes
+
+| Error Code | Retryable? | Action |
+|------------|------------|--------|
+| `FILE_ACCESS_DENIED` | ❌ No | Fix credentials/permissions |
+| `FILE_NOT_FOUND` | ❌ No | Verify file path |
+| `FILE_DISCOVERY_FAILED` | ✅ Yes | Retry with backoff |
+| `NETWORK_ERROR` | ✅ Yes | Retry with backoff |
+| `TIMEOUT_ERROR` | ✅ Yes | Retry with backoff |
+| `PARSE_ERROR` | ❌ No | Fix source data |
+| `ENCODING_ERROR` | ❌ No | Convert to UTF-8 |
+
+---
+
+## Related Documentation
+
+- [Error Handling Guide](../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md) - Core error handling patterns
+- [Data Sources Guide](./data-sources-readme.md) - S3/SFTP configuration and usage
+- [Parsers Guide](../parsers/) - CSV/XML/JSON/Parquet parsing details
+- [Troubleshooting Quick Reference](../../00-START-HERE/troubleshooting-quick-reference.md) - General debugging
+
+---
+
+**Need Help?**
+- 📖 Review [Complete Examples](#complete-examples)
+- 🔍 Check [Troubleshooting Flowchart](#troubleshooting-flowchart)
+- 📋 Use [Production Checklist](#production-checklist)
+- 🐛 Report issues at [GitHub](https://github.com/fluentcommerce/fc-connect-sdk/issues)