npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.53 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (495) hide show

package/docs/02-CORE-GUIDES/data-sources/data-sources-file-operations-error-handling.md CHANGED Viewed

@@ -1,1487 +1,1487 @@
-# File Operations Error Handling Guide
-> **Complete guide to handling errors when reading and parsing files from S3 and SFTP**
-> Production-ready patterns for connection errors, file discovery, reading, and parsing failures
-## Table of Contents
-- [Overview](#overview)
-- [Error Flow Diagram](#error-flow-diagram)
-- [Connection Errors](#connection-errors)
-- [Discovery Errors](#discovery-errors)
-- [Reading Errors](#reading-errors)
-- [Parsing Errors](#parsing-errors)
-- [Complete Examples](#complete-examples)
-- [Retry Strategies](#retry-strategies)
-- [Troubleshooting Flowchart](#troubleshooting-flowchart)
-- [Production Checklist](#production-checklist)
----
-## Overview
-When working with S3 or SFTP data sources, errors can occur at multiple stages:
-```
-┌─────────────────┐
-│ 1. Connection   │ ← Credentials, network, timeouts
-├─────────────────┤
-│ 2. Discovery    │ ← File listing, permissions, paths
-├─────────────────┤
-│ 3. Reading      │ ← Download, memory, streaming
-├─────────────────┤
-│ 4. Parsing      │ ← CSV/XML/JSON/Parquet format errors
-└─────────────────┘
-```
-This guide covers **all four stages** with production-ready error handling patterns.
-### Key Error Types
-| Error Class | Stage | Retryable? | Common Causes |
-|-------------|-------|------------|---------------|
-| `FileDiscoveryError` | Connection/Discovery | ✅ Usually | Invalid credentials, network timeout, permissions |
-| `FileParsingError` | Parsing | ❌ No | Malformed XML/CSV/JSON, encoding issues |
-| `IngestionError` (NETWORK_ERROR) | Connection/Reading | ✅ Yes | Network timeout, API unreachable |
-| Memory errors | Reading | ⚠️ Sometimes | File too large, insufficient heap |
----
-## Error Flow Diagram
-```
-┌──────────────────────────────────────────────────────────────┐
-│ START: Read and Parse File from S3/SFTP                      │
-└────────────────────┬─────────────────────────────────────────┘
-                     │
-                     ▼
-         ┌──────────────────────┐
-         │ Create Data Source   │
-         │ (S3/SFTP)            │
-         └──────┬───────────────┘
-                │
-                ▼
-         ┌──────────────────────┐
-         │ Validate Connection  │
-         └──────┬───────────────┘
-                │
-                ├─────────── ❌ FileDiscoveryError
-                │             → Invalid credentials
-                │             → Network timeout
-                │             → Permissions denied
-                │             → RETRY? Yes (if transient)
-                │
-                ▼
-         ┌──────────────────────┐
-         │ List Files           │
-         │ (discovery)          │
-         └──────┬───────────────┘
-                │
-                ├─────────── ❌ FileDiscoveryError
-                │             → Bucket/path not found
-                │             → Access denied
-                │             → RETRY? Usually No (config error)
-                │
-                ▼
-         ┌──────────────────────┐
-         │ Download File        │
-         │ (read content)       │
-         └──────┬───────────────┘
-                │
-                ├─────────── ❌ IngestionError (NETWORK_ERROR)
-                │             → Network timeout
-                │             → Presigned URL expired (S3)
-                │             → SFTP connection lost
-                │             → RETRY? Yes
-                │
-                ├─────────── ❌ Memory Error
-                │             → File too large
-                │             → Insufficient heap
-                │             → RETRY? No (use streaming)
-                │
-                ▼
-         ┌──────────────────────┐
-         │ Parse Content        │
-         │ (CSV/XML/JSON)       │
-         └──────┬───────────────┘
-                │
-                ├─────────── ❌ FileParsingError
-                │             → Malformed XML (missing tags)
-                │             → Invalid JSON (syntax error)
-                │             → CSV column mismatch
-                │             → Encoding error (non-UTF-8)
-                │             → RETRY? No (fix source data)
-                │
-                ▼
-         ┌──────────────────────┐
-         │ ✅ SUCCESS           │
-         │ Parsed data ready    │
-         └──────────────────────┘
-```
----
-## Connection Errors
-### S3 Connection Errors
-#### Error 1: Invalid Credentials
-**Symptoms:**
-```
-FileDiscoveryError: Access Denied
-  code: FILE_ACCESS_DENIED
-  message: "The AWS Access Key Id you provided does not exist in our records"
-```
-**Causes:**
-- Invalid `accessKeyId` or `secretAccessKey`
-- Credentials expired (if using temporary credentials)
-- Wrong AWS region specified
-**Solution:**
-```typescript
-import {
-  S3DataSource,
-  FileDiscoveryError,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-async function createS3Source() {
-  try {
-    const s3Source = new S3DataSource(
-      {
-        type: 'S3_CSV',
-        s3Config: {
-          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-          region: process.env.AWS_REGION || 'us-east-1',
-          bucket: process.env.S3_BUCKET,
-        },
-      },
-      logger
-    );
-    // ✅ IMPORTANT: Validate connection before proceeding
-    const isValid = await s3Source.validateConnection();
-    if (!isValid) {
-      throw new Error('S3 connection validation failed');
-    }
-    return s3Source;
-  } catch (error) {
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_ACCESS_DENIED') {
-      logger.error('S3 credentials invalid:', {
-        bucket: process.env.S3_BUCKET,
-        region: process.env.AWS_REGION,
-        // DO NOT log credentials!
-      });
-      // Don't retry - configuration error
-      throw new Error('S3 credentials are invalid. Check AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY');
-    }
-    throw error;
-  }
-}
-```
-**Debugging Checklist:**
-- [ ] Verify `AWS_ACCESS_KEY_ID` is correct
-- [ ] Verify `AWS_SECRET_ACCESS_KEY` is correct
-- [ ] Check if credentials are expired (temporary credentials)
-- [ ] Verify AWS region matches bucket region
-- [ ] Test credentials with AWS CLI: `aws s3 ls s3://your-bucket`
-#### Error 2: Wrong Region
-**Symptoms:**
-```
-FileDiscoveryError: The bucket is in this region: us-west-2.
-Please use this region to retry the request
-```
-**Solution:**
-```typescript
-// ❌ WRONG: Hardcoded region doesn't match bucket
-const s3Source = new S3DataSource({
-  type: 'S3_CSV',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: 'us-east-1', // ❌ Bucket is in us-west-2
-    bucket: 'my-bucket',
-  },
-}, logger);
-// ✅ CORRECT: Use environment variable or bucket's actual region
-const s3Source = new S3DataSource({
-  type: 'S3_CSV',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: process.env.AWS_REGION!, // ✅ Set to 'us-west-2'
-    bucket: 'my-bucket',
-  },
-}, logger);
-```
-#### Error 3: Network Timeout (S3)
-**Symptoms:**
-```
-IngestionError: Network request timed out
-  code: NETWORK_ERROR
-  isRetryable: true
-```
-**Solution with Retry:**
-```typescript
-import { IngestionError, IngestionErrorCode } from '@fluentcommerce/fc-connect-sdk';
-async function downloadWithRetry(
-  s3Source: S3DataSource,
-  filePath: string,
-  maxRetries = 3
-) {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      const content = await s3Source.downloadFile(filePath);
-      return content; // ✅ Success
-    } catch (error) {
-      if (error instanceof IngestionError &&
-          error.code === IngestionErrorCode.NETWORK_ERROR &&
-          error.isRetryable() &&
-          attempt < maxRetries) {
-        const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
-        logger.warn(`S3 network error - retrying in ${delay}ms (attempt ${attempt}/${maxRetries})`);
-        await new Promise(resolve => setTimeout(resolve, delay));
-        continue;
-      }
-      throw error; // Not retryable or max retries exceeded
-    }
-  }
-}
-```
-### SFTP Connection Errors
-#### Error 1: Authentication Failed
-**Symptoms:**
-```
-FileDiscoveryError: All configured authentication methods failed
-  code: FILE_ACCESS_DENIED
-```
-**Causes:**
-- Invalid SSH private key
-- Wrong username
-- Private key passphrase incorrect
-- Host key verification failed
-**Solution:**
-```typescript
-import { SftpDataSource, FileDiscoveryError } from '@fluentcommerce/fc-connect-sdk';
-async function createSftpSource() {
-  try {
-    const sftpSource = new SftpDataSource(
-      {
-        type: 'SFTP_CSV',
-        connectionId: 'vendor-sftp',
-        settings: {
-          host: process.env.SFTP_HOST!,
-          port: parseInt(process.env.SFTP_PORT || '22'),
-          username: process.env.SFTP_USERNAME!,
-          privateKey: process.env.SFTP_PRIVATE_KEY!, // ✅ Use SSH key (not password)
-          passphrase: process.env.SFTP_PASSPHRASE, // If key is encrypted
-          remotePath: process.env.SFTP_REMOTE_PATH || '/data',
-        },
-      },
-      logger
-    );
-    // ✅ Validate connection
-    const isValid = await sftpSource.validateConnection();
-    if (!isValid) {
-      throw new Error('SFTP connection validation failed');
-    }
-    return sftpSource;
-  } catch (error) {
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_ACCESS_DENIED') {
-      logger.error('SFTP authentication failed:', {
-        host: process.env.SFTP_HOST,
-        username: process.env.SFTP_USERNAME,
-        // DO NOT log private key or passphrase!
-      });
-      // Check common issues
-      throw new Error(
-        'SFTP authentication failed. Check:\n' +
-        '1. SSH private key is correct\n' +
-        '2. Username is correct\n' +
-        '3. Passphrase is correct (if key is encrypted)\n' +
-        '4. Host key is trusted'
-      );
-    }
-    throw error;
-  }
-}
-```
-#### Error 2: Connection Timeout (SFTP)
-**Symptoms:**
-```
-FileDiscoveryError: Connection timeout after 30000ms
-  code: FILE_DISCOVERY_FAILED
-```
-**Causes:**
-- SFTP server unreachable
-- Firewall blocking port 22
-- Network connectivity issues
-- DNS resolution failure
-**Solution:**
-```typescript
-async function testSftpConnectivity() {
-  const sftpConfig = {
-    type: 'SFTP_CSV' as const,
-    connectionId: 'test-sftp',
-    settings: {
-      host: process.env.SFTP_HOST!,
-      port: 22,
-      username: process.env.SFTP_USERNAME!,
-      privateKey: process.env.SFTP_PRIVATE_KEY!,
-      remotePath: '/data',
-      timeout: 30000, // 30 seconds
-    },
-  };
-  try {
-    const sftpSource = new SftpDataSource(sftpConfig, logger);
-    logger.info('Testing SFTP connectivity...');
-    const isValid = await sftpSource.validateConnection();
-    if (isValid) {
-      logger.info('✅ SFTP connection successful');
-      return sftpSource;
-    } else {
-      throw new Error('SFTP validation returned false');
-    }
-  } catch (error) {
-    if (error instanceof FileDiscoveryError) {
-      logger.error('SFTP connection failed:', {
-        host: sftpConfig.settings.host,
-        port: sftpConfig.settings.port,
-        error: error.message,
-      });
-      // Debugging steps
-      console.error('\n🔍 Debugging steps:');
-      console.error(`1. Test DNS: ping ${sftpConfig.settings.host}`);
-      console.error(`2. Test connectivity: telnet ${sftpConfig.settings.host} 22`);
-      console.error('3. Check firewall allows outbound port 22');
-      console.error('4. Verify SFTP server is running');
-    }
-    throw error;
-  }
-}
-```
----
-## Discovery Errors
-### Error 1: File Not Found
-**Symptoms:**
-```
-FileDiscoveryError: File not found
-  code: FILE_NOT_FOUND
-  path: "s3://my-bucket/inventory/missing.csv"
-```
-**Solution:**
-```typescript
-async function safeFileDownload(
-  dataSource: S3DataSource | SftpDataSource,
-  filePath: string
-) {
-  try {
-    // Check if file exists first
-    const files = await dataSource.listFiles({ prefix: filePath });
-    if (files.length === 0) {
-      logger.warn(`File not found: ${filePath}`);
-      return null; // Return null instead of throwing
-    }
-    // File exists - download it
-    const content = await dataSource.downloadFile(filePath);
-    return content;
-  } catch (error) {
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_NOT_FOUND') {
-      logger.warn('File not found during download:', { path: filePath });
-      return null; // Graceful handling
-    }
-    throw error; // Re-throw other errors
-  }
-}
-```
-### Error 2: Access Denied (Permissions)
-**Symptoms:**
-```
-FileDiscoveryError: Access Denied
-  code: FILE_ACCESS_DENIED
-  message: "User is not authorized to perform: s3:GetObject"
-```
-**IAM Policy Required (S3):**
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Effect": "Allow",
-      "Action": [
-        "s3:ListBucket",
-        "s3:GetObject",
-        "s3:GetObjectVersion"
-      ],
-      "Resource": [
-        "arn:aws:s3:::my-bucket",
-        "arn:aws:s3:::my-bucket/*"
-      ]
-    }
-  ]
-}
-```
-**Solution:**
-```typescript
-async function validateS3Permissions(s3Source: S3DataSource) {
-  try {
-    logger.info('Validating S3 permissions...');
-    // Test 1: Can list bucket?
-    try {
-      await s3Source.listFiles({ prefix: '' });
-      logger.info('✅ s3:ListBucket permission OK');
-    } catch (error) {
-      logger.error('❌ s3:ListBucket permission DENIED');
-      throw error;
-    }
-    // Test 2: Can read objects?
-    const files = await s3Source.listFiles({ prefix: '' });
-    if (files.length > 0) {
-      try {
-        await s3Source.downloadFile(files[0].path);
-        logger.info('✅ s3:GetObject permission OK');
-      } catch (error) {
-        logger.error('❌ s3:GetObject permission DENIED');
-        throw error;
-      }
-    }
-    logger.info('✅ All S3 permissions validated');
-  } catch (error) {
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_ACCESS_DENIED') {
-      console.error('\n❌ IAM Permissions Required:');
-      console.error('  - s3:ListBucket on arn:aws:s3:::my-bucket');
-      console.error('  - s3:GetObject on arn:aws:s3:::my-bucket/*');
-      console.error('\nAdd this IAM policy to your AWS user/role.');
-    }
-    throw error;
-  }
-}
-```
----
-## Reading Errors
-### Error 1: Out of Memory (Large Files)
-**Symptoms:**
-```
-FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
-```
-**Causes:**
-- File too large to load in memory
-- Default Node.js heap size (512MB - 2GB)
-- Multiple large files processed concurrently
-**Solution 1: Increase Heap Size**
-```bash
-# Temporary fix - increase Node.js heap
-node --max-old-space-size=4096 your-script.js  # 4GB heap
-```
-**Solution 2: Stream Large Files (Best Practice)**
-```typescript
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-import { Readable } from 'stream';
-async function processLargeFile(s3Source: S3DataSource, filePath: string) {
-  try {
-    // Option 1: Download in binary mode (Buffer)
-    const content = await s3Source.downloadFile(filePath, { encoding: 'binary' });
-    // Process in chunks
-    const buffer = content as Buffer;
-    const chunkSize = 1024 * 1024; // 1MB chunks
-    for (let offset = 0; offset < buffer.length; offset += chunkSize) {
-      const chunk = buffer.slice(offset, offset + chunkSize);
-      await processChunk(chunk);
-      // Allow garbage collection
-      if (global.gc) global.gc();
-    }
-  } catch (error) {
-    if (error.message?.includes('heap out of memory')) {
-      logger.error('Out of memory processing large file:', {
-        file: filePath,
-        suggestion: 'Use chunked processing or increase heap size',
-      });
-      throw new Error(
-        'File too large for available memory. ' +
-        'Increase heap size with: node --max-old-space-size=4096'
-      );
-    }
-    throw error;
-  }
-}
-async function processChunk(chunk: Buffer) {
-  // Process chunk here
-  logger.debug(`Processing chunk: ${chunk.length} bytes`);
-}
-```
-### Error 2: Presigned URL Expired (S3)
-**Symptoms:**
-```
-FileDiscoveryError: Request has expired
-  code: FILE_ACCESS_DENIED
-  statusCode: 403
-```
-**Cause:**
-- Presigned URLs expire after 1 hour (SDK default)
-- Long-running operations exceed URL lifetime
-**Solution:**
-```typescript
-async function downloadWithUrlRefresh(s3Source: S3DataSource, filePath: string) {
-  const startTime = Date.now();
-  const URL_LIFETIME_MS = 55 * 60 * 1000; // 55 minutes (buffer before 1 hour expiry)
-  try {
-    const content = await s3Source.downloadFile(filePath);
-    return content;
-  } catch (error) {
-    const elapsed = Date.now() - startTime;
-    if (error instanceof FileDiscoveryError &&
-        error.code === 'FILE_ACCESS_DENIED' &&
-        error.message?.includes('expired') &&
-        elapsed > URL_LIFETIME_MS) {
-      logger.warn('Presigned URL expired - recreating data source');
-      // Recreate S3DataSource (generates new presigned URLs)
-      const newS3Source = new S3DataSource(s3Source.getConfig(), logger);
-      // Retry with new URL
-      return await newS3Source.downloadFile(filePath);
-    }
-    throw error;
-  }
-}
-```
----
-## Parsing Errors
-### Error 1: Malformed XML
-**Symptoms:**
-```
-FileParsingError: Failed to parse XML: Unclosed tag 'customer-email'
-  fileName: "order.xml"
-  lineNumber: 5
-  code: PARSE_ERROR
-```
-**Example Malformed XML:**
-```xml
-<?xml version="1.0" encoding="UTF-8"?>
-<order>
-  <order-id>12345</order-id>
-  <customer>
-    <email>test@example.com
-    <!-- ❌ Missing closing tag for <email> and <customer> -->
-```
-**Solution:**
-```typescript
-import { XMLParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
-async function parseXMLWithValidation(xmlContent: string, fileName: string) {
-  const parser = new XMLParserService();
-  try {
-    const parsed = await parser.parse(xmlContent);
-    return parsed;
-  } catch (error) {
-    if (error instanceof FileParsingError &&
-        error.code === 'PARSE_ERROR') {
-      logger.error('XML parsing failed:', {
-        fileName: error.fileName || fileName,
-        lineNumber: error.lineNumber,
-        message: error.message,
-        // Show first 200 chars of XML for debugging
-        xmlPreview: xmlContent.substring(0, 200),
-      });
-      throw new Error(
-        `Invalid XML in ${fileName}:\n` +
-        `  Line ${error.lineNumber}: ${error.message}\n` +
-        `  Fix: Ensure all XML tags are properly closed`
-      );
-    }
-    throw error;
-  }
-}
-```
-**Common XML Errors:**
-| Error | Cause | Fix |
-|-------|-------|-----|
-| `Unclosed tag 'tagname'` | Missing `</tagname>` | Add closing tag |
-| `Unexpected closing tag` | Extra `</tagname>` | Remove duplicate closing tag |
-| `Invalid character` | Special chars not escaped | Use `&amp;`, `&lt;`, `&gt;`, `&quot;`, `&apos;` |
-| `Text not allowed here` | Text outside tags | Wrap text in tags |
-### Error 2: Invalid JSON
-**Symptoms:**
-```
-FileParsingError: Failed to parse JSON: Unexpected token a in JSON at position 45
-  fileName: "order.json"
-  code: PARSE_ERROR
-```
-**Example Invalid JSON:**
-```json
-{
-  "orderId": "12345",
-  "status": active,     // ❌ Unquoted string
-  "items": [
-    {
-      "sku": "ABC-123",
-      "qty": 5,         // ❌ Trailing comma
-    }
-  ],                   // ❌ Trailing comma
-}
-```
-**Solution:**
-```typescript
-import { JSONParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
-async function parseJSONWithValidation(jsonContent: string, fileName: string) {
-  const parser = new JSONParserService();
-  try {
-    const parsed = await parser.parse(jsonContent);
-    return parsed;
-  } catch (error) {
-    if (error instanceof FileParsingError &&
-        error.code === 'PARSE_ERROR') {
-      // Extract position from error message
-      const positionMatch = error.message.match(/position (\d+)/);
-      const position = positionMatch ? parseInt(positionMatch[1]) : 0;
-      // Show context around error
-      const start = Math.max(0, position - 50);
-      const end = Math.min(jsonContent.length, position + 50);
-      const context = jsonContent.substring(start, end);
-      logger.error('JSON parsing failed:', {
-        fileName: fileName,
-        position: position,
-        context: context,
-        message: error.message,
-      });
-      throw new Error(
-        `Invalid JSON in ${fileName}:\n` +
-        `  ${error.message}\n` +
-        `  Context: ...${context}...`
-      );
-    }
-    throw error;
-  }
-}
-```
-**Common JSON Errors:**
-| Error | Cause | Fix |
-|-------|-------|-----|
-| `Unexpected token` | Unquoted string, trailing comma | Quote strings, remove trailing commas |
-| `Unexpected end of JSON` | Missing closing `}` or `]` | Add closing bracket |
-| `Expected property name` | Missing quotes on key | Quote object keys: `"key": value` |
-### Error 3: CSV Column Mismatch
-**Symptoms:**
-```
-FileParsingError: CSV column count mismatch
-  fileName: "inventory.csv"
-  code: INVALID_CSV_FORMAT
-  expectedColumns: 5
-  actualColumns: 4
-```
-**Example Invalid CSV:**
-```csv
-sku,location,quantity,date,status
-SKU-001,LOC-01,100,2024-01-01,ACTIVE
-SKU-002,LOC-02,50,2024-01-02          ← Missing 'status' column
-SKU-003,LOC-03,75,2024-01-03,ACTIVE
-```
-**Solution:**
-```typescript
-import { CSVParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
-async function parseCSVWithValidation(
-  csvContent: string,
-  fileName: string,
-  expectedColumns?: string[]
-) {
-  const parser = new CSVParserService();
-  try {
-    const parsed = await parser.parse(csvContent);
-    // Optional: Validate column names
-    if (expectedColumns && parsed.length > 0) {
-      const actualColumns = Object.keys(parsed[0]);
-      const missingColumns = expectedColumns.filter(col => !actualColumns.includes(col));
-      if (missingColumns.length > 0) {
-        throw new FileParsingError(
-          `CSV missing required columns: ${missingColumns.join(', ')}`,
-          'INVALID_CSV_FORMAT',
-          { fileName, missingColumns, actualColumns }
-        );
-      }
-    }
-    return parsed;
-  } catch (error) {
-    if (error instanceof FileParsingError &&
-        error.code === 'INVALID_CSV_FORMAT') {
-      logger.error('CSV parsing failed:', {
-        fileName: fileName,
-        message: error.message,
-        context: error.context,
-      });
-      throw new Error(
-        `Invalid CSV in ${fileName}:\n` +
-        `  ${error.message}\n` +
-        `  Fix: Ensure all rows have the same number of columns`
-      );
-    }
-    throw error;
-  }
-}
-// Usage
-const expectedColumns = ['sku', 'location', 'quantity', 'date', 'status'];
-const parsed = await parseCSVWithValidation(csvContent, 'inventory.csv', expectedColumns);
-```
-### Error 4: Encoding Issues (Non-UTF-8)
-**Symptoms:**
-```
-FileParsingError: Invalid UTF-8 sequence
-  code: ENCODING_ERROR
-```
-**Cause:**
-- File encoded in ISO-8859-1, Windows-1252, or other non-UTF-8 encoding
-- Special characters corrupted
-**Solution:**
-```typescript
-import { XMLParserService } from '@fluentcommerce/fc-connect-sdk';
-import iconv from 'iconv-lite';
-async function parseWithEncodingDetection(
-  fileContent: Buffer | string,
-  fileName: string
-) {
-  const parser = new XMLParserService();
-  try {
-    // Try UTF-8 first (SDK default)
-    const content = typeof fileContent === 'string' ? fileContent : fileContent.toString('utf-8');
-    return await parser.parse(content);
-  } catch (error) {
-    if (error instanceof FileParsingError &&
-        error.code === 'ENCODING_ERROR') {
-      logger.warn('UTF-8 parsing failed - trying alternate encodings');
-      // Try common alternate encodings
-      const encodings = ['iso-8859-1', 'windows-1252', 'utf-16le'];
-      for (const encoding of encodings) {
-        try {
-          const buffer = typeof fileContent === 'string' ?
-            Buffer.from(fileContent, 'binary') : fileContent;
-          const converted = iconv.decode(buffer, encoding);
-          const parsed = await parser.parse(converted);
-          logger.info(`✅ Successfully parsed with ${encoding} encoding`);
-          return parsed;
-        } catch (e) {
-          continue; // Try next encoding
-        }
-      }
-      // All encodings failed
-      throw new Error(
-        `Could not parse ${fileName} with any encoding. ` +
-        `Tried: UTF-8, ${encodings.join(', ')}`
-      );
-    }
-    throw error;
-  }
-}
-```
-**Note:** The SDK automatically handles UTF-8 BOM (Byte Order Mark) removal, so BOM presence is not an error.
----
-## Complete Examples
-### Example 1: Production-Ready S3 CSV Pipeline
-```typescript
-import {
-  S3DataSource,
-  CSVParserService,
-  FileDiscoveryError,
-  FileParsingError,
-  IngestionError,
-  IngestionErrorCode,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-async function processS3InventoryFiles() {
-  let s3Source: S3DataSource | null = null;
-  try {
-    // ────────────────────────────────────────────────────────
-    // STAGE 1: Connection
-    // ────────────────────────────────────────────────────────
-    logger.info('Creating S3 data source...');
-    s3Source = new S3DataSource(
-      {
-        type: 'S3_CSV',
-        s3Config: {
-          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-          region: process.env.AWS_REGION!,
-          bucket: process.env.S3_BUCKET!,
-        },
-      },
-      logger
-    );
-    // Validate connection
-    const isValid = await s3Source.validateConnection();
-    if (!isValid) {
-      throw new Error('S3 connection validation failed');
-    }
-    logger.info('✅ S3 connection validated');
-    // ────────────────────────────────────────────────────────
-    // STAGE 2: Discovery
-    // ────────────────────────────────────────────────────────
-    logger.info('Discovering files...');
-    const files = await s3Source.listFiles({
-      prefix: 'inventory/',
-      pattern: /\.csv$/i,
-    });
-    if (files.length === 0) {
-      logger.warn('No CSV files found in inventory/ folder');
-      return;
-    }
-    logger.info(`Found ${files.length} CSV files`);
-    // ────────────────────────────────────────────────────────
-    // STAGE 3: Reading & Parsing
-    // ────────────────────────────────────────────────────────
-    const parser = new CSVParserService();
-    const results = [];
-    for (const file of files) {
-      try {
-        logger.info(`Processing ${file.name}...`);
-        // Download file with retry
-        const content = await downloadWithRetry(s3Source, file.path, 3);
-        // Parse CSV
-        const parsed = await parser.parse(content as string);
-        logger.info(`✅ Parsed ${parsed.length} records from ${file.name}`);
-        results.push({ file: file.name, records: parsed });
-      } catch (error) {
-        if (error instanceof FileParsingError) {
-          logger.error(`❌ Failed to parse ${file.name}:`, {
-            code: error.code,
-            message: error.message,
-            lineNumber: error.lineNumber,
-          });
-          // Continue with next file (don't fail entire batch)
-          continue;
-        }
-        throw error; // Re-throw other errors
-      }
-    }
-    logger.info(`✅ Successfully processed ${results.length}/${files.length} files`);
-    return results;
-  } catch (error) {
-    // ────────────────────────────────────────────────────────
-    // STAGE 4: Error Handling
-    // ────────────────────────────────────────────────────────
-    if (error instanceof FileDiscoveryError) {
-      logger.error('File discovery error:', {
-        code: error.code,
-        message: error.message,
-        path: error.path,
-      });
-      if (error.code === 'FILE_ACCESS_DENIED') {
-        console.error('\n❌ S3 Access Denied:');
-        console.error('  1. Check AWS credentials are correct');
-        console.error('  2. Verify IAM policy allows s3:ListBucket and s3:GetObject');
-        console.error('  3. Confirm bucket exists and region is correct');
-      }
-    } else if (error instanceof IngestionError) {
-      logger.error('Ingestion error:', error.toJSON());
-      if (error.isRetryable()) {
-        console.error('\n⚠️  Retryable error - consider running again');
-      }
-    } else {
-      logger.error('Unexpected error:', error);
-    }
-    throw error;
-  }
-}
-// Helper: Download with retry (from earlier example)
-async function downloadWithRetry(
-  s3Source: S3DataSource,
-  filePath: string,
-  maxRetries = 3
-) {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      return await s3Source.downloadFile(filePath);
-    } catch (error) {
-      if (error instanceof IngestionError &&
-          error.code === IngestionErrorCode.NETWORK_ERROR &&
-          error.isRetryable() &&
-          attempt < maxRetries) {
-        const delay = Math.pow(2, attempt) * 1000;
-        logger.warn(`Retry attempt ${attempt}/${maxRetries} in ${delay}ms`);
-        await new Promise(resolve => setTimeout(resolve, delay));
-        continue;
-      }
-      throw error;
-    }
-  }
-}
-// Run the pipeline
-processS3InventoryFiles()
-  .then(() => logger.info('Pipeline completed successfully'))
-  .catch(error => {
-    logger.error('Pipeline failed:', error);
-    process.exit(1);
-  });
-```
-### Example 2: Production-Ready SFTP XML Pipeline
-```typescript
-import {
-  SftpDataSource,
-  XMLParserService,
-  FileDiscoveryError,
-  FileParsingError,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-async function processSftpOrderFiles() {
-  let sftpSource: SftpDataSource | null = null;
-  try {
-    // ────────────────────────────────────────────────────────
-    // STAGE 1: Connection
-    // ────────────────────────────────────────────────────────
-    logger.info('Connecting to SFTP...');
-    sftpSource = new SftpDataSource(
-      {
-        type: 'SFTP_XML',
-        connectionId: 'vendor-sftp',
-        settings: {
-          host: process.env.SFTP_HOST!,
-          port: parseInt(process.env.SFTP_PORT || '22'),
-          username: process.env.SFTP_USERNAME!,
-          privateKey: process.env.SFTP_PRIVATE_KEY!,
-          passphrase: process.env.SFTP_PASSPHRASE,
-          remotePath: '/orders',
-        },
-      },
-      logger
-    );
-    // Validate connection
-    const isValid = await sftpSource.validateConnection();
-    if (!isValid) {
-      throw new Error('SFTP connection validation failed');
-    }
-    logger.info('✅ SFTP connection validated');
-    // ────────────────────────────────────────────────────────
-    // STAGE 2: Discovery
-    // ────────────────────────────────────────────────────────
-    logger.info('Listing XML files...');
-    const files = await sftpSource.listFiles({
-      pattern: /\.xml$/i,
-    });
-    if (files.length === 0) {
-      logger.info('No XML files found');
-      return;
-    }
-    logger.info(`Found ${files.length} XML files`);
-    // ────────────────────────────────────────────────────────
-    // STAGE 3: Reading & Parsing
-    // ────────────────────────────────────────────────────────
-    const parser = new XMLParserService();
-    const results = [];
-    for (const file of files) {
-      try {
-        logger.info(`Processing ${file.name}...`);
-        // Download file
-        const content = await sftpSource.downloadFile(file.path);
-        // Parse XML
-        const parsed = await parser.parse(content as string);
-        logger.info(`✅ Parsed XML from ${file.name}`);
-        results.push({ file: file.name, data: parsed });
-        // Archive processed file
-        await sftpSource.archiveFile(file.path, `/archive/${file.name}`);
-        logger.info(`Archived ${file.name}`);
-      } catch (error) {
-        if (error instanceof FileParsingError) {
-          logger.error(`❌ XML parsing failed for ${file.name}:`, {
-            code: error.code,
-            message: error.message,
-            lineNumber: error.lineNumber,
-          });
-          // Move to error folder
-          await sftpSource.archiveFile(file.path, `/errors/${file.name}`);
-          continue; // Process next file
-        }
-        throw error;
-      }
-    }
-    logger.info(`✅ Processed ${results.length}/${files.length} files`);
-    return results;
-  } catch (error) {
-    if (error instanceof FileDiscoveryError) {
-      logger.error('SFTP error:', {
-        code: error.code,
-        message: error.message,
-      });
-      if (error.code === 'FILE_ACCESS_DENIED') {
-        console.error('\n❌ SFTP Authentication Failed:');
-        console.error('  1. Verify SSH private key is correct');
-        console.error('  2. Check username is correct');
-        console.error('  3. Confirm passphrase (if key is encrypted)');
-        console.error('  4. Test connection: ssh username@host');
-      }
-      if (error.message?.includes('timeout')) {
-        console.error('\n❌ SFTP Connection Timeout:');
-        console.error('  1. Check SFTP host is reachable');
-        console.error('  2. Verify firewall allows outbound port 22');
-        console.error('  3. Test connectivity: telnet host 22');
-      }
-    }
-    throw error;
-  }
-}
-// Run the pipeline
-processSftpOrderFiles()
-  .then(() => logger.info('SFTP pipeline completed'))
-  .catch(error => {
-    logger.error('SFTP pipeline failed:', error);
-    process.exit(1);
-  });
-```
----
-## Retry Strategies
-### When to Retry
-```typescript
-function shouldRetry(error: unknown): boolean {
-  // 1. Network errors - ALWAYS retry
-  if (error instanceof IngestionError &&
-      error.code === IngestionErrorCode.NETWORK_ERROR) {
-    return true;
-  }
-  // 2. Timeout errors - ALWAYS retry
-  if (error instanceof IngestionError &&
-      error.code === IngestionErrorCode.TIMEOUT_ERROR) {
-    return true;
-  }
-  // 3. File discovery errors - SOMETIMES retry
-  if (error instanceof FileDiscoveryError) {
-    // Retry if transient (network, timeout)
-    if (error.message?.includes('timeout') ||
-        error.message?.includes('ECONNREFUSED')) {
-      return true;
-    }
-    // Don't retry permanent errors (access denied, not found)
-    return false;
-  }
-  // 4. Parsing errors - NEVER retry
-  if (error instanceof FileParsingError) {
-    return false;
-  }
-  // 5. Unknown errors - NEVER retry
-  return false;
-}
-```
-### Exponential Backoff with Jitter
-```typescript
-async function retryWithBackoff<T>(
-  operation: () => Promise<T>,
-  maxRetries = 3,
-  baseDelayMs = 1000
-): Promise<T> {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      return await operation(); // ✅ Success
-    } catch (error) {
-      const isLastAttempt = attempt === maxRetries;
-      const shouldRetryError = shouldRetry(error);
-      if (!shouldRetryError || isLastAttempt) {
-        throw error; // Give up
-      }
-      // Calculate delay with exponential backoff + jitter
-      const exponentialDelay = Math.pow(2, attempt) * baseDelayMs;
-      const jitter = Math.random() * 1000; // Random 0-1000ms
-      const delay = exponentialDelay + jitter;
-      logger.warn(`Attempt ${attempt}/${maxRetries} failed - retrying in ${Math.round(delay)}ms`, {
-        error: error instanceof Error ? error.message : String(error),
-      });
-      await new Promise(resolve => setTimeout(resolve, delay));
-    }
-  }
-  throw new Error('Unreachable'); // TypeScript exhaustiveness check
-}
-// Usage
-const content = await retryWithBackoff(
-  () => s3Source.downloadFile('inventory.csv'),
-  3, // max 3 retries
-  1000 // start with 1 second
-);
-```
----
-## Troubleshooting Flowchart
-```
-┌─────────────────────────────────────────────┐
-│ Error occurred during file operation        │
-└────────────────┬────────────────────────────┘
-                 │
-                 ▼
-         ┌───────────────┐
-         │ Error type?   │
-         └───┬───────────┘
-             │
-     ┌───────┼───────┬────────┬──────────┐
-     │       │       │        │          │
-     ▼       ▼       ▼        ▼          ▼
-┌─────────────────────────────────────────────┐
-│ FileDiscoveryError                          │
-├─────────────────────────────────────────────┤
-│ Code: FILE_ACCESS_DENIED                    │
-│ ├─ Check credentials                        │
-│ ├─ Verify IAM/SSH permissions               │
-│ └─ Confirm bucket/host exists               │
-│                                             │
-│ Code: FILE_NOT_FOUND                        │
-│ ├─ Verify file path is correct             │
-│ ├─ Check prefix/pattern                    │
-│ └─ File may have been moved/deleted        │
-│                                             │
-│ Code: FILE_DISCOVERY_FAILED                 │
-│ ├─ Check network connectivity               │
-│ ├─ Test with ping/telnet                   │
-│ └─ Retry with exponential backoff          │
-└─────────────────────────────────────────────┘
-┌─────────────────────────────────────────────┐
-│ IngestionError (NETWORK_ERROR)              │
-├─────────────────────────────────────────────┤
-│ Symptoms: Timeout, connection refused      │
-│ ├─ ✅ RETRYABLE: Yes                        │
-│ ├─ Check network connectivity               │
-│ ├─ Verify endpoint is reachable            │
-│ └─ Retry with exponential backoff          │
-└─────────────────────────────────────────────┘
-┌─────────────────────────────────────────────┐
-│ FileParsingError                            │
-├─────────────────────────────────────────────┤
-│ Code: PARSE_ERROR                           │
-│ ├─ ❌ NOT RETRYABLE                         │
-│ ├─ Check file format (XML/JSON/CSV valid?) │
-│ ├─ Look at lineNumber in error             │
-│ └─ Fix source data                         │
-│                                             │
-│ Code: ENCODING_ERROR                        │
-│ ├─ File may not be UTF-8                   │
-│ ├─ Try alternate encodings                 │
-│ └─ Convert file to UTF-8                   │
-└─────────────────────────────────────────────┘
-┌─────────────────────────────────────────────┐
-│ Memory Error (heap out of memory)           │
-├─────────────────────────────────────────────┤
-│ Symptoms: "JavaScript heap out of memory"  │
-│ ├─ File too large for available memory     │
-│ ├─ Solution 1: Increase heap size          │
-│ │   node --max-old-space-size=4096         │
-│ ├─ Solution 2: Use chunked processing      │
-│ │   Process file in 1MB chunks             │
-│ └─ Solution 3: Stream instead of buffer    │
-└─────────────────────────────────────────────┘
-```
----
-## Production Checklist
-### Pre-Deployment
-**Connection Setup:**
-- [ ] S3/SFTP credentials stored in environment variables (not hardcoded)
-- [ ] `validateConnection()` called before processing files
-- [ ] IAM policies grant minimum required permissions (least privilege)
-- [ ] SFTP uses SSH keys (not passwords)
-- [ ] Connection timeout configured appropriately
-**Error Handling:**
-- [ ] All file operations wrapped in try-catch blocks
-- [ ] Errors logged with structured data (`error.toJSON()`)
-- [ ] Retry logic implemented for transient errors
-- [ ] Exponential backoff with jitter configured
-- [ ] Max retry limit set (typically 3)
-- [ ] Circuit breaker pattern for repeated failures
-**File Processing:**
-- [ ] File existence checked before download
-- [ ] Large files processed in chunks (if > 100MB)
-- [ ] Memory limits configured (`--max-old-space-size`)
-- [ ] Parsed data validated before use
-- [ ] Failed files moved to error folder
-**Monitoring:**
-- [ ] Structured logging enabled
-- [ ] Error rates tracked and alerted
-- [ ] File processing metrics collected (count, size, duration)
-- [ ] Presigned URL expiry monitored (S3)
-### Post-Deployment
-**Validation:**
-- [ ] Successful file processing confirmed
-- [ ] No duplicate processing detected
-- [ ] Error handling working as expected
-- [ ] Retry logic functioning correctly
-- [ ] Memory usage stable
-**Monitoring:**
-- [ ] Logs being collected and searchable
-- [ ] Alerts triggering appropriately
-- [ ] Metrics dashboards created
-- [ ] On-call runbooks updated
----
-## Quick Reference
-### Error Type Decision Tree
-```
-Does file operation fail?
-  │
-  ├─ Connection/authentication fails
-  │  → FileDiscoveryError (FILE_ACCESS_DENIED)
-  │  → Check credentials, IAM policies, SSH keys
-  │
-  ├─ File not found
-  │  → FileDiscoveryError (FILE_NOT_FOUND)
-  │  → Verify path, check file exists
-  │
-  ├─ Network timeout
-  │  → IngestionError (NETWORK_ERROR)
-  │  → RETRY with exponential backoff
-  │
-  ├─ File downloaded but parsing fails
-  │  → FileParsingError (PARSE_ERROR)
-  │  → DON'T RETRY - fix source data
-  │
-  └─ Out of memory
-     → Memory Error
-     → Increase heap or use chunked processing
-```
-### Common Error Codes
-| Error Code | Retryable? | Action |
-|------------|------------|--------|
-| `FILE_ACCESS_DENIED` | ❌ No | Fix credentials/permissions |
-| `FILE_NOT_FOUND` | ❌ No | Verify file path |
-| `FILE_DISCOVERY_FAILED` | ✅ Yes | Retry with backoff |
-| `NETWORK_ERROR` | ✅ Yes | Retry with backoff |
-| `TIMEOUT_ERROR` | ✅ Yes | Retry with backoff |
-| `PARSE_ERROR` | ❌ No | Fix source data |
-| `ENCODING_ERROR` | ❌ No | Convert to UTF-8 |
----
-## Related Documentation
-- [Error Handling Guide](../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md) - Core error handling patterns
-- [Data Sources Guide](./data-sources-readme.md) - S3/SFTP configuration and usage
-- [Parsers Guide](../parsers/) - CSV/XML/JSON/Parquet parsing details
-- [Troubleshooting Quick Reference](../../00-START-HERE/troubleshooting-quick-reference.md) - General debugging
----
-**Need Help?**
-- 📖 Review [Complete Examples](#complete-examples)
-- 🔍 Check [Troubleshooting Flowchart](#troubleshooting-flowchart)
-- 📋 Use [Production Checklist](#production-checklist)
-- 🐛 Report issues at [GitHub](https://github.com/fluentcommerce/fc-connect-sdk/issues)
+# File Operations Error Handling Guide
+> **Complete guide to handling errors when reading and parsing files from S3 and SFTP**
+> Production-ready patterns for connection errors, file discovery, reading, and parsing failures
+## Table of Contents
+- [Overview](#overview)
+- [Error Flow Diagram](#error-flow-diagram)
+- [Connection Errors](#connection-errors)
+- [Discovery Errors](#discovery-errors)
+- [Reading Errors](#reading-errors)
+- [Parsing Errors](#parsing-errors)
+- [Complete Examples](#complete-examples)
+- [Retry Strategies](#retry-strategies)
+- [Troubleshooting Flowchart](#troubleshooting-flowchart)
+- [Production Checklist](#production-checklist)
+---
+## Overview
+When working with S3 or SFTP data sources, errors can occur at multiple stages:
+```
+┌─────────────────┐
+│ 1. Connection   │ ← Credentials, network, timeouts
+├─────────────────┤
+│ 2. Discovery    │ ← File listing, permissions, paths
+├─────────────────┤
+│ 3. Reading      │ ← Download, memory, streaming
+├─────────────────┤
+│ 4. Parsing      │ ← CSV/XML/JSON/Parquet format errors
+└─────────────────┘
+```
+This guide covers **all four stages** with production-ready error handling patterns.
+### Key Error Types
+| Error Class | Stage | Retryable? | Common Causes |
+|-------------|-------|------------|---------------|
+| `FileDiscoveryError` | Connection/Discovery | ✅ Usually | Invalid credentials, network timeout, permissions |
+| `FileParsingError` | Parsing | ❌ No | Malformed XML/CSV/JSON, encoding issues |
+| `IngestionError` (NETWORK_ERROR) | Connection/Reading | ✅ Yes | Network timeout, API unreachable |
+| Memory errors | Reading | ⚠️ Sometimes | File too large, insufficient heap |
+---
+## Error Flow Diagram
+```
+┌──────────────────────────────────────────────────────────────┐
+│ START: Read and Parse File from S3/SFTP                      │
+└────────────────────┬─────────────────────────────────────────┘
+                     │
+                     ▼
+         ┌──────────────────────┐
+         │ Create Data Source   │
+         │ (S3/SFTP)            │
+         └──────┬───────────────┘
+                │
+                ▼
+         ┌──────────────────────┐
+         │ Validate Connection  │
+         └──────┬───────────────┘
+                │
+                ├─────────── ❌ FileDiscoveryError
+                │             → Invalid credentials
+                │             → Network timeout
+                │             → Permissions denied
+                │             → RETRY? Yes (if transient)
+                │
+                ▼
+         ┌──────────────────────┐
+         │ List Files           │
+         │ (discovery)          │
+         └──────┬───────────────┘
+                │
+                ├─────────── ❌ FileDiscoveryError
+                │             → Bucket/path not found
+                │             → Access denied
+                │             → RETRY? Usually No (config error)
+                │
+                ▼
+         ┌──────────────────────┐
+         │ Download File        │
+         │ (read content)       │
+         └──────┬───────────────┘
+                │
+                ├─────────── ❌ IngestionError (NETWORK_ERROR)
+                │             → Network timeout
+                │             → Presigned URL expired (S3)
+                │             → SFTP connection lost
+                │             → RETRY? Yes
+                │
+                ├─────────── ❌ Memory Error
+                │             → File too large
+                │             → Insufficient heap
+                │             → RETRY? No (use streaming)
+                │
+                ▼
+         ┌──────────────────────┐
+         │ Parse Content        │
+         │ (CSV/XML/JSON)       │
+         └──────┬───────────────┘
+                │
+                ├─────────── ❌ FileParsingError
+                │             → Malformed XML (missing tags)
+                │             → Invalid JSON (syntax error)
+                │             → CSV column mismatch
+                │             → Encoding error (non-UTF-8)
+                │             → RETRY? No (fix source data)
+                │
+                ▼
+         ┌──────────────────────┐
+         │ ✅ SUCCESS           │
+         │ Parsed data ready    │
+         └──────────────────────┘
+```
+---
+## Connection Errors
+### S3 Connection Errors
+#### Error 1: Invalid Credentials
+**Symptoms:**
+```
+FileDiscoveryError: Access Denied
+  code: FILE_ACCESS_DENIED
+  message: "The AWS Access Key Id you provided does not exist in our records"
+```
+**Causes:**
+- Invalid `accessKeyId` or `secretAccessKey`
+- Credentials expired (if using temporary credentials)
+- Wrong AWS region specified
+**Solution:**
+```typescript
+import {
+  S3DataSource,
+  FileDiscoveryError,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+async function createS3Source() {
+  try {
+    const s3Source = new S3DataSource(
+      {
+        type: 'S3_CSV',
+        s3Config: {
+          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+          region: process.env.AWS_REGION || 'us-east-1',
+          bucket: process.env.S3_BUCKET,
+        },
+      },
+      logger
+    );
+    // ✅ IMPORTANT: Validate connection before proceeding
+    const isValid = await s3Source.validateConnection();
+    if (!isValid) {
+      throw new Error('S3 connection validation failed');
+    }
+    return s3Source;
+  } catch (error) {
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_ACCESS_DENIED') {
+      logger.error('S3 credentials invalid:', {
+        bucket: process.env.S3_BUCKET,
+        region: process.env.AWS_REGION,
+        // DO NOT log credentials!
+      });
+      // Don't retry - configuration error
+      throw new Error('S3 credentials are invalid. Check AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY');
+    }
+    throw error;
+  }
+}
+```
+**Debugging Checklist:**
+- [ ] Verify `AWS_ACCESS_KEY_ID` is correct
+- [ ] Verify `AWS_SECRET_ACCESS_KEY` is correct
+- [ ] Check if credentials are expired (temporary credentials)
+- [ ] Verify AWS region matches bucket region
+- [ ] Test credentials with AWS CLI: `aws s3 ls s3://your-bucket`
+#### Error 2: Wrong Region
+**Symptoms:**
+```
+FileDiscoveryError: The bucket is in this region: us-west-2.
+Please use this region to retry the request
+```
+**Solution:**
+```typescript
+// ❌ WRONG: Hardcoded region doesn't match bucket
+const s3Source = new S3DataSource({
+  type: 'S3_CSV',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: 'us-east-1', // ❌ Bucket is in us-west-2
+    bucket: 'my-bucket',
+  },
+}, logger);
+// ✅ CORRECT: Use environment variable or bucket's actual region
+const s3Source = new S3DataSource({
+  type: 'S3_CSV',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: process.env.AWS_REGION!, // ✅ Set to 'us-west-2'
+    bucket: 'my-bucket',
+  },
+}, logger);
+```
+#### Error 3: Network Timeout (S3)
+**Symptoms:**
+```
+IngestionError: Network request timed out
+  code: NETWORK_ERROR
+  isRetryable: true
+```
+**Solution with Retry:**
+```typescript
+import { IngestionError, IngestionErrorCode } from '@fluentcommerce/fc-connect-sdk';
+async function downloadWithRetry(
+  s3Source: S3DataSource,
+  filePath: string,
+  maxRetries = 3
+) {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      const content = await s3Source.downloadFile(filePath);
+      return content; // ✅ Success
+    } catch (error) {
+      if (error instanceof IngestionError &&
+          error.code === IngestionErrorCode.NETWORK_ERROR &&
+          error.isRetryable() &&
+          attempt < maxRetries) {
+        const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
+        logger.warn(`S3 network error - retrying in ${delay}ms (attempt ${attempt}/${maxRetries})`);
+        await new Promise(resolve => setTimeout(resolve, delay));
+        continue;
+      }
+      throw error; // Not retryable or max retries exceeded
+    }
+  }
+}
+```
+### SFTP Connection Errors
+#### Error 1: Authentication Failed
+**Symptoms:**
+```
+FileDiscoveryError: All configured authentication methods failed
+  code: FILE_ACCESS_DENIED
+```
+**Causes:**
+- Invalid SSH private key
+- Wrong username
+- Private key passphrase incorrect
+- Host key verification failed
+**Solution:**
+```typescript
+import { SftpDataSource, FileDiscoveryError } from '@fluentcommerce/fc-connect-sdk';
+async function createSftpSource() {
+  try {
+    const sftpSource = new SftpDataSource(
+      {
+        type: 'SFTP_CSV',
+        connectionId: 'vendor-sftp',
+        settings: {
+          host: process.env.SFTP_HOST!,
+          port: parseInt(process.env.SFTP_PORT || '22'),
+          username: process.env.SFTP_USERNAME!,
+          privateKey: process.env.SFTP_PRIVATE_KEY!, // ✅ Use SSH key (not password)
+          passphrase: process.env.SFTP_PASSPHRASE, // If key is encrypted
+          remotePath: process.env.SFTP_REMOTE_PATH || '/data',
+        },
+      },
+      logger
+    );
+    // ✅ Validate connection
+    const isValid = await sftpSource.validateConnection();
+    if (!isValid) {
+      throw new Error('SFTP connection validation failed');
+    }
+    return sftpSource;
+  } catch (error) {
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_ACCESS_DENIED') {
+      logger.error('SFTP authentication failed:', {
+        host: process.env.SFTP_HOST,
+        username: process.env.SFTP_USERNAME,
+        // DO NOT log private key or passphrase!
+      });
+      // Check common issues
+      throw new Error(
+        'SFTP authentication failed. Check:\n' +
+        '1. SSH private key is correct\n' +
+        '2. Username is correct\n' +
+        '3. Passphrase is correct (if key is encrypted)\n' +
+        '4. Host key is trusted'
+      );
+    }
+    throw error;
+  }
+}
+```
+#### Error 2: Connection Timeout (SFTP)
+**Symptoms:**
+```
+FileDiscoveryError: Connection timeout after 30000ms
+  code: FILE_DISCOVERY_FAILED
+```
+**Causes:**
+- SFTP server unreachable
+- Firewall blocking port 22
+- Network connectivity issues
+- DNS resolution failure
+**Solution:**
+```typescript
+async function testSftpConnectivity() {
+  const sftpConfig = {
+    type: 'SFTP_CSV' as const,
+    connectionId: 'test-sftp',
+    settings: {
+      host: process.env.SFTP_HOST!,
+      port: 22,
+      username: process.env.SFTP_USERNAME!,
+      privateKey: process.env.SFTP_PRIVATE_KEY!,
+      remotePath: '/data',
+      timeout: 30000, // 30 seconds
+    },
+  };
+  try {
+    const sftpSource = new SftpDataSource(sftpConfig, logger);
+    logger.info('Testing SFTP connectivity...');
+    const isValid = await sftpSource.validateConnection();
+    if (isValid) {
+      logger.info('✅ SFTP connection successful');
+      return sftpSource;
+    } else {
+      throw new Error('SFTP validation returned false');
+    }
+  } catch (error) {
+    if (error instanceof FileDiscoveryError) {
+      logger.error('SFTP connection failed:', {
+        host: sftpConfig.settings.host,
+        port: sftpConfig.settings.port,
+        error: error.message,
+      });
+      // Debugging steps
+      console.error('\n🔍 Debugging steps:');
+      console.error(`1. Test DNS: ping ${sftpConfig.settings.host}`);
+      console.error(`2. Test connectivity: telnet ${sftpConfig.settings.host} 22`);
+      console.error('3. Check firewall allows outbound port 22');
+      console.error('4. Verify SFTP server is running');
+    }
+    throw error;
+  }
+}
+```
+---
+## Discovery Errors
+### Error 1: File Not Found
+**Symptoms:**
+```
+FileDiscoveryError: File not found
+  code: FILE_NOT_FOUND
+  path: "s3://my-bucket/inventory/missing.csv"
+```
+**Solution:**
+```typescript
+async function safeFileDownload(
+  dataSource: S3DataSource | SftpDataSource,
+  filePath: string
+) {
+  try {
+    // Check if file exists first
+    const files = await dataSource.listFiles({ prefix: filePath });
+    if (files.length === 0) {
+      logger.warn(`File not found: ${filePath}`);
+      return null; // Return null instead of throwing
+    }
+    // File exists - download it
+    const content = await dataSource.downloadFile(filePath);
+    return content;
+  } catch (error) {
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_NOT_FOUND') {
+      logger.warn('File not found during download:', { path: filePath });
+      return null; // Graceful handling
+    }
+    throw error; // Re-throw other errors
+  }
+}
+```
+### Error 2: Access Denied (Permissions)
+**Symptoms:**
+```
+FileDiscoveryError: Access Denied
+  code: FILE_ACCESS_DENIED
+  message: "User is not authorized to perform: s3:GetObject"
+```
+**IAM Policy Required (S3):**
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:ListBucket",
+        "s3:GetObject",
+        "s3:GetObjectVersion"
+      ],
+      "Resource": [
+        "arn:aws:s3:::my-bucket",
+        "arn:aws:s3:::my-bucket/*"
+      ]
+    }
+  ]
+}
+```
+**Solution:**
+```typescript
+async function validateS3Permissions(s3Source: S3DataSource) {
+  try {
+    logger.info('Validating S3 permissions...');
+    // Test 1: Can list bucket?
+    try {
+      await s3Source.listFiles({ prefix: '' });
+      logger.info('✅ s3:ListBucket permission OK');
+    } catch (error) {
+      logger.error('❌ s3:ListBucket permission DENIED');
+      throw error;
+    }
+    // Test 2: Can read objects?
+    const files = await s3Source.listFiles({ prefix: '' });
+    if (files.length > 0) {
+      try {
+        await s3Source.downloadFile(files[0].path);
+        logger.info('✅ s3:GetObject permission OK');
+      } catch (error) {
+        logger.error('❌ s3:GetObject permission DENIED');
+        throw error;
+      }
+    }
+    logger.info('✅ All S3 permissions validated');
+  } catch (error) {
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_ACCESS_DENIED') {
+      console.error('\n❌ IAM Permissions Required:');
+      console.error('  - s3:ListBucket on arn:aws:s3:::my-bucket');
+      console.error('  - s3:GetObject on arn:aws:s3:::my-bucket/*');
+      console.error('\nAdd this IAM policy to your AWS user/role.');
+    }
+    throw error;
+  }
+}
+```
+---
+## Reading Errors
+### Error 1: Out of Memory (Large Files)
+**Symptoms:**
+```
+FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
+```
+**Causes:**
+- File too large to load in memory
+- Default Node.js heap size (512MB - 2GB)
+- Multiple large files processed concurrently
+**Solution 1: Increase Heap Size**
+```bash
+# Temporary fix - increase Node.js heap
+node --max-old-space-size=4096 your-script.js  # 4GB heap
+```
+**Solution 2: Stream Large Files (Best Practice)**
+```typescript
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+import { Readable } from 'stream';
+async function processLargeFile(s3Source: S3DataSource, filePath: string) {
+  try {
+    // Option 1: Download in binary mode (Buffer)
+    const content = await s3Source.downloadFile(filePath, { encoding: 'binary' });
+    // Process in chunks
+    const buffer = content as Buffer;
+    const chunkSize = 1024 * 1024; // 1MB chunks
+    for (let offset = 0; offset < buffer.length; offset += chunkSize) {
+      const chunk = buffer.slice(offset, offset + chunkSize);
+      await processChunk(chunk);
+      // Allow garbage collection
+      if (global.gc) global.gc();
+    }
+  } catch (error) {
+    if (error.message?.includes('heap out of memory')) {
+      logger.error('Out of memory processing large file:', {
+        file: filePath,
+        suggestion: 'Use chunked processing or increase heap size',
+      });
+      throw new Error(
+        'File too large for available memory. ' +
+        'Increase heap size with: node --max-old-space-size=4096'
+      );
+    }
+    throw error;
+  }
+}
+async function processChunk(chunk: Buffer) {
+  // Process chunk here
+  logger.debug(`Processing chunk: ${chunk.length} bytes`);
+}
+```
+### Error 2: Presigned URL Expired (S3)
+**Symptoms:**
+```
+FileDiscoveryError: Request has expired
+  code: FILE_ACCESS_DENIED
+  statusCode: 403
+```
+**Cause:**
+- Presigned URLs expire after 1 hour (SDK default)
+- Long-running operations exceed URL lifetime
+**Solution:**
+```typescript
+async function downloadWithUrlRefresh(s3Source: S3DataSource, filePath: string) {
+  const startTime = Date.now();
+  const URL_LIFETIME_MS = 55 * 60 * 1000; // 55 minutes (buffer before 1 hour expiry)
+  try {
+    const content = await s3Source.downloadFile(filePath);
+    return content;
+  } catch (error) {
+    const elapsed = Date.now() - startTime;
+    if (error instanceof FileDiscoveryError &&
+        error.code === 'FILE_ACCESS_DENIED' &&
+        error.message?.includes('expired') &&
+        elapsed > URL_LIFETIME_MS) {
+      logger.warn('Presigned URL expired - recreating data source');
+      // Recreate S3DataSource (generates new presigned URLs)
+      const newS3Source = new S3DataSource(s3Source.getConfig(), logger);
+      // Retry with new URL
+      return await newS3Source.downloadFile(filePath);
+    }
+    throw error;
+  }
+}
+```
+---
+## Parsing Errors
+### Error 1: Malformed XML
+**Symptoms:**
+```
+FileParsingError: Failed to parse XML: Unclosed tag 'customer-email'
+  fileName: "order.xml"
+  lineNumber: 5
+  code: PARSE_ERROR
+```
+**Example Malformed XML:**
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<order>
+  <order-id>12345</order-id>
+  <customer>
+    <email>test@example.com
+    <!-- ❌ Missing closing tag for <email> and <customer> -->
+```
+**Solution:**
+```typescript
+import { XMLParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
+async function parseXMLWithValidation(xmlContent: string, fileName: string) {
+  const parser = new XMLParserService();
+  try {
+    const parsed = await parser.parse(xmlContent);
+    return parsed;
+  } catch (error) {
+    if (error instanceof FileParsingError &&
+        error.code === 'PARSE_ERROR') {
+      logger.error('XML parsing failed:', {
+        fileName: error.fileName || fileName,
+        lineNumber: error.lineNumber,
+        message: error.message,
+        // Show first 200 chars of XML for debugging
+        xmlPreview: xmlContent.substring(0, 200),
+      });
+      throw new Error(
+        `Invalid XML in ${fileName}:\n` +
+        `  Line ${error.lineNumber}: ${error.message}\n` +
+        `  Fix: Ensure all XML tags are properly closed`
+      );
+    }
+    throw error;
+  }
+}
+```
+**Common XML Errors:**
+| Error | Cause | Fix |
+|-------|-------|-----|
+| `Unclosed tag 'tagname'` | Missing `</tagname>` | Add closing tag |
+| `Unexpected closing tag` | Extra `</tagname>` | Remove duplicate closing tag |
+| `Invalid character` | Special chars not escaped | Use `&amp;`, `&lt;`, `&gt;`, `&quot;`, `&apos;` |
+| `Text not allowed here` | Text outside tags | Wrap text in tags |
+### Error 2: Invalid JSON
+**Symptoms:**
+```
+FileParsingError: Failed to parse JSON: Unexpected token a in JSON at position 45
+  fileName: "order.json"
+  code: PARSE_ERROR
+```
+**Example Invalid JSON:**
+```json
+{
+  "orderId": "12345",
+  "status": active,     // ❌ Unquoted string
+  "items": [
+    {
+      "sku": "ABC-123",
+      "qty": 5,         // ❌ Trailing comma
+    }
+  ],                   // ❌ Trailing comma
+}
+```
+**Solution:**
+```typescript
+import { JSONParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
+async function parseJSONWithValidation(jsonContent: string, fileName: string) {
+  const parser = new JSONParserService();
+  try {
+    const parsed = await parser.parse(jsonContent);
+    return parsed;
+  } catch (error) {
+    if (error instanceof FileParsingError &&
+        error.code === 'PARSE_ERROR') {
+      // Extract position from error message
+      const positionMatch = error.message.match(/position (\d+)/);
+      const position = positionMatch ? parseInt(positionMatch[1]) : 0;
+      // Show context around error
+      const start = Math.max(0, position - 50);
+      const end = Math.min(jsonContent.length, position + 50);
+      const context = jsonContent.substring(start, end);
+      logger.error('JSON parsing failed:', {
+        fileName: fileName,
+        position: position,
+        context: context,
+        message: error.message,
+      });
+      throw new Error(
+        `Invalid JSON in ${fileName}:\n` +
+        `  ${error.message}\n` +
+        `  Context: ...${context}...`
+      );
+    }
+    throw error;
+  }
+}
+```
+**Common JSON Errors:**
+| Error | Cause | Fix |
+|-------|-------|-----|
+| `Unexpected token` | Unquoted string, trailing comma | Quote strings, remove trailing commas |
+| `Unexpected end of JSON` | Missing closing `}` or `]` | Add closing bracket |
+| `Expected property name` | Missing quotes on key | Quote object keys: `"key": value` |
+### Error 3: CSV Column Mismatch
+**Symptoms:**
+```
+FileParsingError: CSV column count mismatch
+  fileName: "inventory.csv"
+  code: INVALID_CSV_FORMAT
+  expectedColumns: 5
+  actualColumns: 4
+```
+**Example Invalid CSV:**
+```csv
+sku,location,quantity,date,status
+SKU-001,LOC-01,100,2024-01-01,ACTIVE
+SKU-002,LOC-02,50,2024-01-02          ← Missing 'status' column
+SKU-003,LOC-03,75,2024-01-03,ACTIVE
+```
+**Solution:**
+```typescript
+import { CSVParserService, FileParsingError } from '@fluentcommerce/fc-connect-sdk';
+async function parseCSVWithValidation(
+  csvContent: string,
+  fileName: string,
+  expectedColumns?: string[]
+) {
+  const parser = new CSVParserService();
+  try {
+    const parsed = await parser.parse(csvContent);
+    // Optional: Validate column names
+    if (expectedColumns && parsed.length > 0) {
+      const actualColumns = Object.keys(parsed[0]);
+      const missingColumns = expectedColumns.filter(col => !actualColumns.includes(col));
+      if (missingColumns.length > 0) {
+        throw new FileParsingError(
+          `CSV missing required columns: ${missingColumns.join(', ')}`,
+          'INVALID_CSV_FORMAT',
+          { fileName, missingColumns, actualColumns }
+        );
+      }
+    }
+    return parsed;
+  } catch (error) {
+    if (error instanceof FileParsingError &&
+        error.code === 'INVALID_CSV_FORMAT') {
+      logger.error('CSV parsing failed:', {
+        fileName: fileName,
+        message: error.message,
+        context: error.context,
+      });
+      throw new Error(
+        `Invalid CSV in ${fileName}:\n` +
+        `  ${error.message}\n` +
+        `  Fix: Ensure all rows have the same number of columns`
+      );
+    }
+    throw error;
+  }
+}
+// Usage
+const expectedColumns = ['sku', 'location', 'quantity', 'date', 'status'];
+const parsed = await parseCSVWithValidation(csvContent, 'inventory.csv', expectedColumns);
+```
+### Error 4: Encoding Issues (Non-UTF-8)
+**Symptoms:**
+```
+FileParsingError: Invalid UTF-8 sequence
+  code: ENCODING_ERROR
+```
+**Cause:**
+- File encoded in ISO-8859-1, Windows-1252, or other non-UTF-8 encoding
+- Special characters corrupted
+**Solution:**
+```typescript
+import { XMLParserService } from '@fluentcommerce/fc-connect-sdk';
+import iconv from 'iconv-lite';
+async function parseWithEncodingDetection(
+  fileContent: Buffer | string,
+  fileName: string
+) {
+  const parser = new XMLParserService();
+  try {
+    // Try UTF-8 first (SDK default)
+    const content = typeof fileContent === 'string' ? fileContent : fileContent.toString('utf-8');
+    return await parser.parse(content);
+  } catch (error) {
+    if (error instanceof FileParsingError &&
+        error.code === 'ENCODING_ERROR') {
+      logger.warn('UTF-8 parsing failed - trying alternate encodings');
+      // Try common alternate encodings
+      const encodings = ['iso-8859-1', 'windows-1252', 'utf-16le'];
+      for (const encoding of encodings) {
+        try {
+          const buffer = typeof fileContent === 'string' ?
+            Buffer.from(fileContent, 'binary') : fileContent;
+          const converted = iconv.decode(buffer, encoding);
+          const parsed = await parser.parse(converted);
+          logger.info(`✅ Successfully parsed with ${encoding} encoding`);
+          return parsed;
+        } catch (e) {
+          continue; // Try next encoding
+        }
+      }
+      // All encodings failed
+      throw new Error(
+        `Could not parse ${fileName} with any encoding. ` +
+        `Tried: UTF-8, ${encodings.join(', ')}`
+      );
+    }
+    throw error;
+  }
+}
+```
+**Note:** The SDK automatically handles UTF-8 BOM (Byte Order Mark) removal, so BOM presence is not an error.
+---
+## Complete Examples
+### Example 1: Production-Ready S3 CSV Pipeline
+```typescript
+import {
+  S3DataSource,
+  CSVParserService,
+  FileDiscoveryError,
+  FileParsingError,
+  IngestionError,
+  IngestionErrorCode,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+async function processS3InventoryFiles() {
+  let s3Source: S3DataSource | null = null;
+  try {
+    // ────────────────────────────────────────────────────────
+    // STAGE 1: Connection
+    // ────────────────────────────────────────────────────────
+    logger.info('Creating S3 data source...');
+    s3Source = new S3DataSource(
+      {
+        type: 'S3_CSV',
+        s3Config: {
+          accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+          region: process.env.AWS_REGION!,
+          bucket: process.env.S3_BUCKET!,
+        },
+      },
+      logger
+    );
+    // Validate connection
+    const isValid = await s3Source.validateConnection();
+    if (!isValid) {
+      throw new Error('S3 connection validation failed');
+    }
+    logger.info('✅ S3 connection validated');
+    // ────────────────────────────────────────────────────────
+    // STAGE 2: Discovery
+    // ────────────────────────────────────────────────────────
+    logger.info('Discovering files...');
+    const files = await s3Source.listFiles({
+      prefix: 'inventory/',
+      pattern: /\.csv$/i,
+    });
+    if (files.length === 0) {
+      logger.warn('No CSV files found in inventory/ folder');
+      return;
+    }
+    logger.info(`Found ${files.length} CSV files`);
+    // ────────────────────────────────────────────────────────
+    // STAGE 3: Reading & Parsing
+    // ────────────────────────────────────────────────────────
+    const parser = new CSVParserService();
+    const results = [];
+    for (const file of files) {
+      try {
+        logger.info(`Processing ${file.name}...`);
+        // Download file with retry
+        const content = await downloadWithRetry(s3Source, file.path, 3);
+        // Parse CSV
+        const parsed = await parser.parse(content as string);
+        logger.info(`✅ Parsed ${parsed.length} records from ${file.name}`);
+        results.push({ file: file.name, records: parsed });
+      } catch (error) {
+        if (error instanceof FileParsingError) {
+          logger.error(`❌ Failed to parse ${file.name}:`, {
+            code: error.code,
+            message: error.message,
+            lineNumber: error.lineNumber,
+          });
+          // Continue with next file (don't fail entire batch)
+          continue;
+        }
+        throw error; // Re-throw other errors
+      }
+    }
+    logger.info(`✅ Successfully processed ${results.length}/${files.length} files`);
+    return results;
+  } catch (error) {
+    // ────────────────────────────────────────────────────────
+    // STAGE 4: Error Handling
+    // ────────────────────────────────────────────────────────
+    if (error instanceof FileDiscoveryError) {
+      logger.error('File discovery error:', {
+        code: error.code,
+        message: error.message,
+        path: error.path,
+      });
+      if (error.code === 'FILE_ACCESS_DENIED') {
+        console.error('\n❌ S3 Access Denied:');
+        console.error('  1. Check AWS credentials are correct');
+        console.error('  2. Verify IAM policy allows s3:ListBucket and s3:GetObject');
+        console.error('  3. Confirm bucket exists and region is correct');
+      }
+    } else if (error instanceof IngestionError) {
+      logger.error('Ingestion error:', error.toJSON());
+      if (error.isRetryable()) {
+        console.error('\n⚠️  Retryable error - consider running again');
+      }
+    } else {
+      logger.error('Unexpected error:', error);
+    }
+    throw error;
+  }
+}
+// Helper: Download with retry (from earlier example)
+async function downloadWithRetry(
+  s3Source: S3DataSource,
+  filePath: string,
+  maxRetries = 3
+) {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await s3Source.downloadFile(filePath);
+    } catch (error) {
+      if (error instanceof IngestionError &&
+          error.code === IngestionErrorCode.NETWORK_ERROR &&
+          error.isRetryable() &&
+          attempt < maxRetries) {
+        const delay = Math.pow(2, attempt) * 1000;
+        logger.warn(`Retry attempt ${attempt}/${maxRetries} in ${delay}ms`);
+        await new Promise(resolve => setTimeout(resolve, delay));
+        continue;
+      }
+      throw error;
+    }
+  }
+}
+// Run the pipeline
+processS3InventoryFiles()
+  .then(() => logger.info('Pipeline completed successfully'))
+  .catch(error => {
+    logger.error('Pipeline failed:', error);
+    process.exit(1);
+  });
+```
+### Example 2: Production-Ready SFTP XML Pipeline
+```typescript
+import {
+  SftpDataSource,
+  XMLParserService,
+  FileDiscoveryError,
+  FileParsingError,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+async function processSftpOrderFiles() {
+  let sftpSource: SftpDataSource | null = null;
+  try {
+    // ────────────────────────────────────────────────────────
+    // STAGE 1: Connection
+    // ────────────────────────────────────────────────────────
+    logger.info('Connecting to SFTP...');
+    sftpSource = new SftpDataSource(
+      {
+        type: 'SFTP_XML',
+        connectionId: 'vendor-sftp',
+        settings: {
+          host: process.env.SFTP_HOST!,
+          port: parseInt(process.env.SFTP_PORT || '22'),
+          username: process.env.SFTP_USERNAME!,
+          privateKey: process.env.SFTP_PRIVATE_KEY!,
+          passphrase: process.env.SFTP_PASSPHRASE,
+          remotePath: '/orders',
+        },
+      },
+      logger
+    );
+    // Validate connection
+    const isValid = await sftpSource.validateConnection();
+    if (!isValid) {
+      throw new Error('SFTP connection validation failed');
+    }
+    logger.info('✅ SFTP connection validated');
+    // ────────────────────────────────────────────────────────
+    // STAGE 2: Discovery
+    // ────────────────────────────────────────────────────────
+    logger.info('Listing XML files...');
+    const files = await sftpSource.listFiles({
+      pattern: /\.xml$/i,
+    });
+    if (files.length === 0) {
+      logger.info('No XML files found');
+      return;
+    }
+    logger.info(`Found ${files.length} XML files`);
+    // ────────────────────────────────────────────────────────
+    // STAGE 3: Reading & Parsing
+    // ────────────────────────────────────────────────────────
+    const parser = new XMLParserService();
+    const results = [];
+    for (const file of files) {
+      try {
+        logger.info(`Processing ${file.name}...`);
+        // Download file
+        const content = await sftpSource.downloadFile(file.path);
+        // Parse XML
+        const parsed = await parser.parse(content as string);
+        logger.info(`✅ Parsed XML from ${file.name}`);
+        results.push({ file: file.name, data: parsed });
+        // Archive processed file
+        await sftpSource.archiveFile(file.path, `/archive/${file.name}`);
+        logger.info(`Archived ${file.name}`);
+      } catch (error) {
+        if (error instanceof FileParsingError) {
+          logger.error(`❌ XML parsing failed for ${file.name}:`, {
+            code: error.code,
+            message: error.message,
+            lineNumber: error.lineNumber,
+          });
+          // Move to error folder
+          await sftpSource.archiveFile(file.path, `/errors/${file.name}`);
+          continue; // Process next file
+        }
+        throw error;
+      }
+    }
+    logger.info(`✅ Processed ${results.length}/${files.length} files`);
+    return results;
+  } catch (error) {
+    if (error instanceof FileDiscoveryError) {
+      logger.error('SFTP error:', {
+        code: error.code,
+        message: error.message,
+      });
+      if (error.code === 'FILE_ACCESS_DENIED') {
+        console.error('\n❌ SFTP Authentication Failed:');
+        console.error('  1. Verify SSH private key is correct');
+        console.error('  2. Check username is correct');
+        console.error('  3. Confirm passphrase (if key is encrypted)');
+        console.error('  4. Test connection: ssh username@host');
+      }
+      if (error.message?.includes('timeout')) {
+        console.error('\n❌ SFTP Connection Timeout:');
+        console.error('  1. Check SFTP host is reachable');
+        console.error('  2. Verify firewall allows outbound port 22');
+        console.error('  3. Test connectivity: telnet host 22');
+      }
+    }
+    throw error;
+  }
+}
+// Run the pipeline
+processSftpOrderFiles()
+  .then(() => logger.info('SFTP pipeline completed'))
+  .catch(error => {
+    logger.error('SFTP pipeline failed:', error);
+    process.exit(1);
+  });
+```
+---
+## Retry Strategies
+### When to Retry
+```typescript
+function shouldRetry(error: unknown): boolean {
+  // 1. Network errors - ALWAYS retry
+  if (error instanceof IngestionError &&
+      error.code === IngestionErrorCode.NETWORK_ERROR) {
+    return true;
+  }
+  // 2. Timeout errors - ALWAYS retry
+  if (error instanceof IngestionError &&
+      error.code === IngestionErrorCode.TIMEOUT_ERROR) {
+    return true;
+  }
+  // 3. File discovery errors - SOMETIMES retry
+  if (error instanceof FileDiscoveryError) {
+    // Retry if transient (network, timeout)
+    if (error.message?.includes('timeout') ||
+        error.message?.includes('ECONNREFUSED')) {
+      return true;
+    }
+    // Don't retry permanent errors (access denied, not found)
+    return false;
+  }
+  // 4. Parsing errors - NEVER retry
+  if (error instanceof FileParsingError) {
+    return false;
+  }
+  // 5. Unknown errors - NEVER retry
+  return false;
+}
+```
+### Exponential Backoff with Jitter
+```typescript
+async function retryWithBackoff<T>(
+  operation: () => Promise<T>,
+  maxRetries = 3,
+  baseDelayMs = 1000
+): Promise<T> {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await operation(); // ✅ Success
+    } catch (error) {
+      const isLastAttempt = attempt === maxRetries;
+      const shouldRetryError = shouldRetry(error);
+      if (!shouldRetryError || isLastAttempt) {
+        throw error; // Give up
+      }
+      // Calculate delay with exponential backoff + jitter
+      const exponentialDelay = Math.pow(2, attempt) * baseDelayMs;
+      const jitter = Math.random() * 1000; // Random 0-1000ms
+      const delay = exponentialDelay + jitter;
+      logger.warn(`Attempt ${attempt}/${maxRetries} failed - retrying in ${Math.round(delay)}ms`, {
+        error: error instanceof Error ? error.message : String(error),
+      });
+      await new Promise(resolve => setTimeout(resolve, delay));
+    }
+  }
+  throw new Error('Unreachable'); // TypeScript exhaustiveness check
+}
+// Usage
+const content = await retryWithBackoff(
+  () => s3Source.downloadFile('inventory.csv'),
+  3, // max 3 retries
+  1000 // start with 1 second
+);
+```
+---
+## Troubleshooting Flowchart
+```
+┌─────────────────────────────────────────────┐
+│ Error occurred during file operation        │
+└────────────────┬────────────────────────────┘
+                 │
+                 ▼
+         ┌───────────────┐
+         │ Error type?   │
+         └───┬───────────┘
+             │
+     ┌───────┼───────┬────────┬──────────┐
+     │       │       │        │          │
+     ▼       ▼       ▼        ▼          ▼
+┌─────────────────────────────────────────────┐
+│ FileDiscoveryError                          │
+├─────────────────────────────────────────────┤
+│ Code: FILE_ACCESS_DENIED                    │
+│ ├─ Check credentials                        │
+│ ├─ Verify IAM/SSH permissions               │
+│ └─ Confirm bucket/host exists               │
+│                                             │
+│ Code: FILE_NOT_FOUND                        │
+│ ├─ Verify file path is correct             │
+│ ├─ Check prefix/pattern                    │
+│ └─ File may have been moved/deleted        │
+│                                             │
+│ Code: FILE_DISCOVERY_FAILED                 │
+│ ├─ Check network connectivity               │
+│ ├─ Test with ping/telnet                   │
+│ └─ Retry with exponential backoff          │
+└─────────────────────────────────────────────┘
+┌─────────────────────────────────────────────┐
+│ IngestionError (NETWORK_ERROR)              │
+├─────────────────────────────────────────────┤
+│ Symptoms: Timeout, connection refused      │
+│ ├─ ✅ RETRYABLE: Yes                        │
+│ ├─ Check network connectivity               │
+│ ├─ Verify endpoint is reachable            │
+│ └─ Retry with exponential backoff          │
+└─────────────────────────────────────────────┘
+┌─────────────────────────────────────────────┐
+│ FileParsingError                            │
+├─────────────────────────────────────────────┤
+│ Code: PARSE_ERROR                           │
+│ ├─ ❌ NOT RETRYABLE                         │
+│ ├─ Check file format (XML/JSON/CSV valid?) │
+│ ├─ Look at lineNumber in error             │
+│ └─ Fix source data                         │
+│                                             │
+│ Code: ENCODING_ERROR                        │
+│ ├─ File may not be UTF-8                   │
+│ ├─ Try alternate encodings                 │
+│ └─ Convert file to UTF-8                   │
+└─────────────────────────────────────────────┘
+┌─────────────────────────────────────────────┐
+│ Memory Error (heap out of memory)           │
+├─────────────────────────────────────────────┤
+│ Symptoms: "JavaScript heap out of memory"  │
+│ ├─ File too large for available memory     │
+│ ├─ Solution 1: Increase heap size          │
+│ │   node --max-old-space-size=4096         │
+│ ├─ Solution 2: Use chunked processing      │
+│ │   Process file in 1MB chunks             │
+│ └─ Solution 3: Stream instead of buffer    │
+└─────────────────────────────────────────────┘
+```
+---
+## Production Checklist
+### Pre-Deployment
+**Connection Setup:**
+- [ ] S3/SFTP credentials stored in environment variables (not hardcoded)
+- [ ] `validateConnection()` called before processing files
+- [ ] IAM policies grant minimum required permissions (least privilege)
+- [ ] SFTP uses SSH keys (not passwords)
+- [ ] Connection timeout configured appropriately
+**Error Handling:**
+- [ ] All file operations wrapped in try-catch blocks
+- [ ] Errors logged with structured data (`error.toJSON()`)
+- [ ] Retry logic implemented for transient errors
+- [ ] Exponential backoff with jitter configured
+- [ ] Max retry limit set (typically 3)
+- [ ] Circuit breaker pattern for repeated failures
+**File Processing:**
+- [ ] File existence checked before download
+- [ ] Large files processed in chunks (if > 100MB)
+- [ ] Memory limits configured (`--max-old-space-size`)
+- [ ] Parsed data validated before use
+- [ ] Failed files moved to error folder
+**Monitoring:**
+- [ ] Structured logging enabled
+- [ ] Error rates tracked and alerted
+- [ ] File processing metrics collected (count, size, duration)
+- [ ] Presigned URL expiry monitored (S3)
+### Post-Deployment
+**Validation:**
+- [ ] Successful file processing confirmed
+- [ ] No duplicate processing detected
+- [ ] Error handling working as expected
+- [ ] Retry logic functioning correctly
+- [ ] Memory usage stable
+**Monitoring:**
+- [ ] Logs being collected and searchable
+- [ ] Alerts triggering appropriately
+- [ ] Metrics dashboards created
+- [ ] On-call runbooks updated
+---
+## Quick Reference
+### Error Type Decision Tree
+```
+Does file operation fail?
+  │
+  ├─ Connection/authentication fails
+  │  → FileDiscoveryError (FILE_ACCESS_DENIED)
+  │  → Check credentials, IAM policies, SSH keys
+  │
+  ├─ File not found
+  │  → FileDiscoveryError (FILE_NOT_FOUND)
+  │  → Verify path, check file exists
+  │
+  ├─ Network timeout
+  │  → IngestionError (NETWORK_ERROR)
+  │  → RETRY with exponential backoff
+  │
+  ├─ File downloaded but parsing fails
+  │  → FileParsingError (PARSE_ERROR)
+  │  → DON'T RETRY - fix source data
+  │
+  └─ Out of memory
+     → Memory Error
+     → Increase heap or use chunked processing
+```
+### Common Error Codes
+| Error Code | Retryable? | Action |
+|------------|------------|--------|
+| `FILE_ACCESS_DENIED` | ❌ No | Fix credentials/permissions |
+| `FILE_NOT_FOUND` | ❌ No | Verify file path |
+| `FILE_DISCOVERY_FAILED` | ✅ Yes | Retry with backoff |
+| `NETWORK_ERROR` | ✅ Yes | Retry with backoff |
+| `TIMEOUT_ERROR` | ✅ Yes | Retry with backoff |
+| `PARSE_ERROR` | ❌ No | Fix source data |
+| `ENCODING_ERROR` | ❌ No | Convert to UTF-8 |
+---
+## Related Documentation
+- [Error Handling Guide](../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md) - Core error handling patterns
+- [Data Sources Guide](./data-sources-readme.md) - S3/SFTP configuration and usage
+- [Parsers Guide](../parsers/) - CSV/XML/JSON/Parquet parsing details
+- [Troubleshooting Quick Reference](../../00-START-HERE/troubleshooting-quick-reference.md) - General debugging
+---
+**Need Help?**
+- 📖 Review [Complete Examples](#complete-examples)
+- 🔍 Check [Troubleshooting Flowchart](#troubleshooting-flowchart)
+- 📋 Use [Production Checklist](#production-checklist)
+- 🐛 Report issues at [GitHub](https://github.com/fluentcommerce/fc-connect-sdk/issues)