@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-03-data-sources.md

@@ -1,1052 +1,1052 @@
-# Module 3: Data Sources & Parsers
-
-**Level:** Intermediate
-**Estimated Time:** 30 minutes
-
-## Overview
-
-This module teaches you how to read inventory data from different sources (S3, SFTP) and parse various file formats (CSV, Parquet, XML, JSON) using the SDK's data source services.
-
-**IMPORTANT:** Data sources in this SDK have integrated parsing capabilities. You don't need separate parser services - each data source handles its own file formats.
-
-## Learning Objectives
-
-By the end of this module, you will:
-- ✅ Understand the data source abstraction pattern
-- ✅ Configure S3DataSource for AWS S3 buckets with retry logic
-- ✅ Configure SftpDataSource with connection pooling and retry
-- ✅ Parse CSV, JSON, and Parquet files directly from data sources
-- ✅ Handle large files with streaming and batching
-- ✅ Choose the right data source configuration for your use case
-
-## Data Source Abstraction
-
-The SDK provides data sources that handle both file operations AND parsing:
-
-```typescript
-import { S3DataSource, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-// Each data source is a complete file + parsing solution
-// No need for separate parser services
-```
-
-### Benefits
-
-| Benefit | Description |
-|---------|-------------|
-| **Integrated** | File operations and parsing in one service |
-| **Retry Logic** | Automatic exponential backoff for transient failures |
-| **Connection Pooling** | (SFTP only) Reuse connections for performance |
-| **Format Support** | CSV, JSON, XML, Parquet, JSONL |
-| **Testability** | Mock data sources for unit testing |
-
-## S3 Data Source
-
-### Configuration
-
-```typescript
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-const logger = createConsoleLogger();
-
-const s3 = new S3DataSource(
-  {
-    type: 'S3_CSV',  // or 'S3_JSON', 'S3_XML', 'S3_PARQUET'
-    name: 'S3 Inventory Source',
-    connectionId: 's3-prod',
-    s3Config: {
-      bucket: 'inventory-bucket',
-      region: 'us-east-1',
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-
-      // Optional: Session token for temporary credentials
-      sessionToken: process.env.AWS_SESSION_TOKEN,
-
-      // Optional: Retry configuration
-      maxAttempts: 3  // Default: 3 attempts with exponential backoff
-    }
-  },
-  logger
-);
-```
-
-### Basic Operations
-
-```typescript
-// List files with filtering
-const files = await s3.listFiles({
-  prefix: 'data/',           // Filter by prefix
-  delimiter: '/',            // Group by "directory"
-  maxKeys: 1000,             // Max results per page
-  continuationToken: token   // For pagination
-});
-
-console.log(`Found ${files.length} files`);
-files.forEach(file => {
-  console.log(`  ${file.path} (${file.size} bytes, ${file.lastModified})`);
-});
-
-// Download file
-const content = await s3.downloadFile('data/inventory-2024-01-15.csv', {
-  encoding: 'utf8'  // or 'binary' for Buffer
-});
-
-// Download binary files (images, Parquet, etc.)
-const binaryData = await s3.downloadFile('data/inventory.parquet', {
-  encoding: 'binary'
-});
-```
-
-### Retry Configuration
-
-S3DataSource includes automatic retry with exponential backoff for transient failures:
-
-```typescript
-const s3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    name: 'S3 with Custom Retry',
-    s3Config: {
-      bucket: 'inventory-bucket',
-      region: 'us-east-1',
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-
-      // Retry configuration
-      maxAttempts: 5  // Try up to 5 times (default: 3)
-    }
-  },
-  logger
-);
-
-// Retry behavior:
-// - Attempt 1: Immediate
-// - Attempt 2: 500ms delay
-// - Attempt 3: 1000ms delay
-// - Attempt 4: 2000ms delay
-// - Attempt 5: 4000ms delay
-// - Max delay: 8000ms
-
-// Retries on: 429, 500, 502, 503, 504 status codes
-// Retries on: Network errors (ECONNRESET, ETIMEDOUT, etc.)
-```
-
-### Upload Operations
-
-```typescript
-// Upload text file
-await s3.uploadFile(
-  'processed/inventory-2024-01-15.csv',
-  csvContent,
-  {
-    contentType: 'text/csv',
-    metadata: {
-      processedAt: new Date().toISOString(),
-      source: 'inventory-sync'
-    }
-  }
-);
-
-// Upload binary file
-await s3.uploadFile(
-  'data/inventory.parquet',
-  parquetBuffer,
-  {
-    contentType: 'application/octet-stream'
-  }
-);
-```
-
-### File Management
-
-```typescript
-// Copy file
-await s3.copyFile(
-  'data/inventory.csv',
-  'backup/inventory.csv',
-  {
-    sourceBucket: 'source-bucket',  // Optional: cross-bucket copy
-    metadata: { backupDate: new Date().toISOString() }
-  }
-);
-
-// Move file within same bucket (copy + delete)
-await s3.moveFile(
-  'temp/inventory.csv',
-  'archive/inventory.csv'
-  // Optional: pass false to copy without deleting source
-  // await s3.moveFile('temp/inventory.csv', 'archive/inventory.csv', false);
-);
-
-// Archive file with automatic timestamping
-const archivePath = await s3.archiveFile(
-  'data/inventory.csv',
-  'archive/2024-01/'  // Archive prefix (timestamp + filename added automatically)
-);
-console.log(`File archived to: ${archivePath}`);
-// Returns: 'archive/2024-01/2025-10-29T12-30-45-123Z_inventory.csv'
-
-// For cross-bucket moves, use copyFile() + deleteFile() directly
-await s3.copyFile('inventory.csv', 'backup/inventory.csv', {
-  sourceBucket: 'source-bucket',
-  metadata: { archivedAt: new Date().toISOString() }
-});
-await s3.deleteFile('inventory.csv');
-
-// Delete file
-await s3.deleteFile('temp/old-file.csv');
-```
-
-### Parquet Support
-
-S3DataSource has built-in Parquet writing support:
-
-```typescript
-// Write records as Parquet file
-const records = [
-  { sku: 'ABC123', warehouse: 'WH1', quantity: 100 },
-  { sku: 'DEF456', warehouse: 'WH2', quantity: 50 }
-];
-
-const parquetBuffer = await s3.writeParquetContent(records, {
-  compression: 'GZIP',  // or 'SNAPPY', 'BROTLI', 'UNCOMPRESSED'
-  rowGroupSize: 5000,   // Records per row group
-  schema: {             // Optional: explicit schema
-    sku: { type: 'UTF8' },
-    warehouse: { type: 'UTF8' },
-    quantity: { type: 'INT64' }
-  }
-});
-
-// Upload Parquet file
-await s3.uploadFile(
-  'data/inventory.parquet',
-  parquetBuffer,
-  { contentType: 'application/octet-stream' }
-);
-```
-
-### Connection Validation
-
-```typescript
-// Validate S3 connection before processing
-const isValid = await s3.validateConnection();
-if (!isValid) {
-  throw new Error('Failed to connect to S3');
-}
-```
-
-### Complete S3 Example
-
-```typescript
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function processS3Files() {
-  const logger = createConsoleLogger();
-
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_CSV',
-      name: 'Inventory Source',
-      s3Config: {
-        bucket: 'inventory-bucket',
-        region: 'us-east-1',
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-        maxAttempts: 3
-      }
-    },
-    logger
-  );
-
-  try {
-    // Validate connection
-    await s3.validateConnection();
-
-    // List files
-    const files = await s3.listFiles({ prefix: 'incoming/' });
-
-    for (const file of files) {
-      logger.info(`Processing: ${file.path} (${file.size} bytes)`);
-
-      // Download file
-      const content = await s3.downloadFile(file.path);
-
-      // Process content...
-      // (Use CSVParserService or other parsers as needed)
-
-      // Archive processed file (timestamp added automatically)
-      const archivedPath = await s3.archiveFile(
-        file.path,
-        `archive/${new Date().toISOString().split('T')[0]}/`  // Just the prefix
-      );
-      logger.info(`Archived to: ${archivedPath}`);
-    }
-
-    logger.info('Processing complete');
-  } catch (error) {
-    logger.error('Processing failed', error);
-    throw error;
-  }
-}
-```
-
-## SFTP Data Source
-
-### Configuration
-
-```typescript
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-import * as fs from 'fs';
-
-const logger = createConsoleLogger();
-
-const sftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',  // or 'SFTP_JSON', 'SFTP_XML', 'SFTP_JSONL', 'SFTP_PARQUET'
-    name: 'SFTP Inventory Source',
-    connectionId: 'sftp-prod',
-    settings: {
-      host: 'sftp.example.com',
-      port: 22,
-      remotePath: '/data/inventory',
-      filePattern: '*.csv',
-
-      // Authentication: password OR private key
-      username: process.env.SFTP_USERNAME!,
-      password: process.env.SFTP_PASSWORD,
-
-      // OR use private key
-      privateKey: fs.readFileSync('/path/to/key.pem', 'utf8'),
-      passphrase: process.env.SFTP_KEY_PASSPHRASE,
-
-      // Connection pooling (NEW in v0.1.11+)
-      maxConnections: 5,  // Default: 5
-
-      // Retry configuration (NEW in v0.1.11+)
-      retry: {
-        maxAttempts: 3,      // Default: 3
-        baseDelayMs: 1000,   // Default: 1000ms
-        backoffFactor: 2,    // Default: 2 (exponential)
-        maxDelayMs: 8000,    // Default: 8000ms
-        jitter: 'full',      // 'full' or 'none'
-        reconnectOnFailure: true  // Reconnect on failure
-      },
-
-      // CSV parsing settings
-      csvDelimiter: ',',
-      csvHeaders: ['sku', 'warehouse', 'quantity'],
-      requiredCsvHeaders: ['sku', 'quantity'],
-      csvSkipEmptyLines: true,
-      csvTrimValues: true,
-
-      // JSON parsing settings
-      jsonFormat: 'json',  // or 'jsonl'
-      jsonValidate: true,
-      jsonMaxDepth: 10,
-
-      // Parquet settings
-      parquetCompression: 'gzip'
-    }
-  },
-  logger
-);
-```
-
-### Connection Pooling (NEW in v0.1.11+)
-
-SftpDataSource automatically manages a connection pool for performance:
-
-```typescript
-// Connection pool behavior:
-// 1. Reuses existing connections when available
-// 2. Creates new connections up to maxConnections limit
-// 3. Waits in queue when pool is full (30s timeout)
-// 4. Validates connection health before reuse
-// 5. Auto-reconnects on transient failures
-
-const sftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    name: 'SFTP with Pool',
-    settings: {
-      host: 'sftp.example.com',
-      username: process.env.SFTP_USERNAME!,
-      password: process.env.SFTP_PASSWORD!,
-      remotePath: '/data',
-      filePattern: '*.csv',
-
-      // Pool configuration
-      maxConnections: 10,  // Allow up to 10 concurrent connections
-      connectionTimeout: 10000,  // 10s connection timeout
-      keepAliveInterval: 10000   // Keep-alive every 10s
-    }
-  },
-  logger
-);
-
-// Connection pool is transparent - all operations use it automatically
-const files = await sftp.listFiles();  // Uses pool
-const content = await sftp.downloadFile(files[0].name);  // Uses pool
-
-// Get pool stats for monitoring
-const info = sftp.getConnectionInfo();
-console.log(`Pool size: ${info.connectionPoolSize}/${info.maxPoolSize}`);
-```
-
-### Enhanced Retry Logic (NEW in v0.1.11+)
-
-SFTP operations automatically retry with exponential backoff:
-
-```typescript
-const sftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    name: 'SFTP with Custom Retry',
-    settings: {
-      host: 'sftp.example.com',
-      username: process.env.SFTP_USERNAME!,
-      password: process.env.SFTP_PASSWORD!,
-      remotePath: '/data',
-      filePattern: '*.csv',
-
-      retry: {
-        maxAttempts: 5,        // Try up to 5 times
-        baseDelayMs: 2000,     // Start with 2s delay
-        backoffFactor: 2,      // Double delay each attempt
-        maxDelayMs: 16000,     // Cap at 16s
-        jitter: 'full',        // Random jitter (0 to maxDelay)
-        reconnectOnFailure: true,  // Clear pool and reconnect
-
-        // Custom retry predicate
-        isRetryable: (error: Error) => {
-          // Add custom logic for retryable errors
-          return error.message.includes('timeout') ||
-                 error.message.includes('connection');
-        }
-      }
-    }
-  },
-  logger
-);
-
-// Retry timeline with above config:
-// - Attempt 1: Immediate
-// - Attempt 2: Random(0-2000ms) delay
-// - Attempt 3: Random(0-4000ms) delay
-// - Attempt 4: Random(0-8000ms) delay
-// - Attempt 5: Random(0-16000ms) delay
-
-// Automatically retries on:
-// - ECONNRESET, ETIMEDOUT, ENOTFOUND, EHOSTUNREACH, ECONNREFUSED
-// - "connection lost", "handshake failed", "timeout", etc.
-//
-// Does NOT retry on:
-// - Authentication failures
-// - Permission denied
-// - File not found
-// - Algorithm/cipher/key exchange errors
-```
-
-### Basic Operations
-
-```typescript
-// List files with filtering
-const files = await sftp.listFiles({
-  remotePath: '/data/inventory',  // Optional: override config
-  filePattern: '*.csv',           // Optional: override config
-  lastProcessedTimestamp: '2024-01-15T00:00:00Z',  // Incremental sync
-  fileName: 'inventory'           // Filter by name substring
-});
-
-console.log(`Found ${files.length} files`);
-
-// Download file
-const content = await sftp.downloadFile('/data/inventory.csv', {
-  encoding: 'utf8',  // Default encoding
-  asBuffer: false,   // Return string
-  maxSize: 100 * 1024 * 1024  // 100MB limit
-});
-
-// Download as buffer
-const buffer = await sftp.downloadFile('/data/inventory.parquet', {
-  asBuffer: true
-});
-```
-
-### Integrated CSV Parsing
-
-SFTP data source has built-in CSV parsing:
-
-```typescript
-// Download and parse CSV in one step
-const csvContent = await sftp.downloadFile('/data/inventory.csv');
-
-const records = sftp.parseCsvContent(csvContent, {
-  name: 'inventory.csv',
-  path: '/data/inventory.csv',
-  size: csvContent.length,
-  source: 'sftp'
-});
-
-console.log(`Parsed ${records.length} records`);
-records.forEach(record => {
-  console.log(`SKU: ${record.sku}, Qty: ${record.quantity}`);
-});
-```
-
-### Integrated JSON Parsing
-
-```typescript
-// Download and parse JSON
-const jsonContent = await sftp.downloadFile('/data/inventory.json');
-
-const records = sftp.parseJsonContent(
-  jsonContent,
-  { name: 'inventory.json', path: '/data/inventory.json', source: 'sftp' },
-  {
-    format: 'json',    // or 'jsonl' for JSON Lines
-    validate: true,    // Validate structure
-    maxDepth: 10       // Limit nesting depth
-  }
-);
-
-console.log(`Parsed ${records.length} JSON records`);
-```
-
-### Writing Files
-
-```typescript
-// Write CSV file
-const records = [
-  { sku: 'ABC123', warehouse: 'WH1', quantity: 100 },
-  { sku: 'DEF456', warehouse: 'WH2', quantity: 50 }
-];
-
-const csvContent = sftp.writeCsvContent(records, {
-  delimiter: ',',
-  includeHeader: true,
-  quote: '"',
-  escape: '"'
-});
-
-await sftp.uploadFile('/data/output.csv', {
-  overwrite: true,
-  createDirectories: true
-}, csvContent);
-
-// Write JSON file
-const jsonContent = sftp.writeJsonContent(records, {
-  format: 'json',  // or 'jsonl'
-  indent: 2,
-  sortKeys: true
-});
-
-await sftp.uploadFile('/data/output.json', {
-  overwrite: true
-}, jsonContent);
-
-// Write Parquet file
-const parquetBuffer = await sftp.writeParquetContent(records, {
-  compression: 'GZIP',
-  rowGroupSize: 5000
-});
-
-await sftp.uploadFile('/data/output.parquet', {
-  overwrite: true
-}, parquetBuffer);
-```
-
-### File Management
-
-```typescript
-// Upload file from local path
-await sftp.uploadFile(
-  '/local/path/inventory.csv',
-  '/remote/path/inventory.csv',
-  {
-    overwrite: false,
-    createDirectories: true,
-    permissions: '0644'
-  }
-);
-
-// Create directory
-await sftp.createDirectory('/data/archive', true);  // recursive
-
-// Move file
-await sftp.moveFile(
-  '/data/inventory.csv',
-  '/data/processed/inventory.csv'
-);
-
-// Delete file
-await sftp.deleteFile('/data/old-file.csv');
-
-// Check if file exists
-const exists = await sftp.fileExists('/data/inventory.csv');
-if (exists) {
-  console.log('File exists');
-}
-
-// Check if directory exists
-const dirExists = await sftp.directoryExists('/data/archive');
-```
-
-### Connection Monitoring
-
-```typescript
-// Validate connection
-const isValid = await sftp.validateConnection();
-if (!isValid) {
-  throw new Error('SFTP connection failed');
-}
-
-// Get connection pool info
-const info = sftp.getConnectionInfo();
-console.log(`Connection Info:
-  Type: ${info.type}
-  Host: ${info.host}:${info.port}
-  Remote Path: ${info.remotePath}
-  File Pattern: ${info.filePattern}
-  Pool: ${info.connectionPoolSize}/${info.maxPoolSize} connections
-`);
-```
-
-### Complete SFTP Example
-
-```typescript
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function processSftpFiles() {
-  const logger = createConsoleLogger();
-
-  const sftp = new SftpDataSource(
-    {
-      type: 'SFTP_CSV',
-      name: 'SFTP Inventory',
-      settings: {
-        host: process.env.SFTP_HOST!,
-        username: process.env.SFTP_USERNAME!,
-        password: process.env.SFTP_PASSWORD!,
-        remotePath: '/data/inventory',
-        filePattern: '*.csv',
-
-        // Enable connection pooling and retry
-        maxConnections: 5,
-        retry: {
-          maxAttempts: 3,
-          reconnectOnFailure: true
-        },
-
-        // CSV parsing settings
-        csvDelimiter: ',',
-        requiredCsvHeaders: ['sku', 'quantity'],
-        csvSkipEmptyLines: true
-      }
-    },
-    logger
-  );
-
-  try {
-    // Validate connection
-    await sftp.validateConnection();
-
-    // List files
-    const files = await sftp.listFiles();
-    logger.info(`Found ${files.length} files`);
-
-    for (const file of files) {
-      logger.info(`Processing: ${file.name}`);
-
-      // Download file
-      const csvContent = await sftp.downloadFile(file.path);
-
-      // Parse CSV (integrated parsing)
-      const records = sftp.parseCsvContent(csvContent, file);
-      logger.info(`Parsed ${records.length} records`);
-
-      // Process records...
-      // (Send to Fluent Commerce, etc.)
-
-      // Move to processed folder
-      await sftp.moveFile(
-        file.path,
-        `/data/processed/${file.name}`
-      );
-    }
-
-    // Get final pool stats
-    const info = sftp.getConnectionInfo();
-    logger.info(`Pool: ${info.connectionPoolSize}/${info.maxPoolSize}`);
-
-  } catch (error) {
-    logger.error('Processing failed', error);
-    throw error;
-  }
-}
-```
-
-## Inventory Data Source (Base Interface)
-
-**Note:** `InventoryDataSource` is an abstract base interface for inventory-specific operations. Use the concrete implementations `S3DataSource` or `SftpDataSource` directly.
-
-```typescript
-import { InventoryDataSource, BaseInventoryDataSource } from '@fluentcommerce/fc-connect-sdk';
-
-// This is an interface, not a standalone data source
-interface InventoryDataSource {
-  getBucketName(): string;
-  getCsvFilePrefix(): string;
-  parseS3ListResponse(xmlResponse: string, prefix: string, lastModifiedAfter?: string): FileMetadata[];
-  getConfig(): DataSourceConfig;
-}
-
-// Base implementation
-class BaseInventoryDataSource implements InventoryDataSource {
-  constructor(protected config: DataSourceConfig) {}
-
-  getBucketName(): string {
-    return 'default-bucket';
-  }
-
-  getCsvFilePrefix(): string {
-    return 'inventory/';
-  }
-
-  parseS3ListResponse(xmlResponse: string, prefix: string, lastModifiedAfter?: string): FileMetadata[] {
-    // Simple XML parsing for S3 list response
-    // ...
-  }
-
-  getConfig(): DataSourceConfig {
-    return this.config;
-  }
-}
-```
-
-**Recommendation:** Use `S3DataSource` or `SftpDataSource` instead of implementing this interface.
-
-## Choosing the Right Data Source
-
-| Feature | S3DataSource | SftpDataSource |
-|---------|-------------|---------------|
-| **Connection Pooling** | ❌ Not needed (HTTP) | ✅ Yes (v0.1.11+) |
-| **Retry Logic** | ✅ Exponential backoff | ✅ Enhanced with reconnect |
-| **Formats** | CSV, JSON, XML, Parquet | CSV, JSON, XML, JSONL, Parquet |
-| **Integrated Parsing** | ❌ Use separate parsers | ✅ Built-in parsers |
-| **Streaming** | ✅ Download streams | ❌ Download then parse |
-| **File Operations** | List, Download, Upload, Copy, Move, Delete, Archive | List, Download, Upload, Move, Delete |
-| **Best For** | Cloud storage, large files, archiving | FTP servers, real-time feeds |
-| **Performance** | Excellent (parallel downloads) | Good (connection pool) |
-
-### Decision Tree
-
-```typescript
-function chooseDataSource(requirements: {
-  source: 'cloud' | 'sftp';
-  fileSize: 'small' | 'large';
-  frequency: 'realtime' | 'batch';
-  format: string;
-}): 'S3' | 'SFTP' {
-  if (requirements.source === 'cloud') {
-    return 'S3';  // S3 for cloud storage
-  }
-
-  if (requirements.source === 'sftp') {
-    if (requirements.frequency === 'realtime') {
-      // SFTP with connection pooling for frequent operations
-      return 'SFTP';
-    }
-
-    if (requirements.fileSize === 'large') {
-      // S3 better for very large files (streaming)
-      return 'S3';
-    }
-
-    return 'SFTP';  // SFTP for FTP servers
-  }
-
-  return 'S3';  // Default to S3
-}
-```
-
-## Performance Considerations
-
-### S3 Best Practices
-
-```typescript
-// 1. Use pagination for large lists
-let continuationToken: string | undefined;
-do {
-  const files = await s3.listFiles({
-    prefix: 'data/',
-    maxKeys: 1000,
-    continuationToken
-  });
-
-  // Process files...
-
-  // Get next page token (if available)
-  continuationToken = files.length === 1000 ? files[files.length - 1].path : undefined;
-} while (continuationToken);
-
-// 2. Use binary encoding for non-text files
-const parquetData = await s3.downloadFile('data/large.parquet', {
-  encoding: 'binary'
-});
-
-// 3. Enable retry for production
-const s3 = new S3DataSource(
-  {
-    type: 'S3_PARQUET',
-    s3Config: {
-      bucket: 'prod-bucket',
-      region: 'us-east-1',
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      maxAttempts: 5  // Production: 5 attempts
-    }
-  },
-  logger
-);
-```
-
-### SFTP Best Practices
-
-```typescript
-// 1. Configure connection pool for concurrent operations
-const sftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    settings: {
-      host: 'sftp.example.com',
-      username: process.env.SFTP_USERNAME!,
-      password: process.env.SFTP_PASSWORD!,
-      remotePath: '/data',
-      filePattern: '*.csv',
-
-      // Tune pool size based on workload
-      maxConnections: 10,  // Higher for concurrent operations
-
-      retry: {
-        maxAttempts: 5,      // Production: more retries
-        reconnectOnFailure: true
-      }
-    }
-  },
-  logger
-);
-
-// 2. Monitor pool usage
-const info = sftp.getConnectionInfo();
-if (info.connectionPoolSize === info.maxPoolSize) {
-  logger.warn('Connection pool at capacity - consider increasing maxConnections');
-}
-
-// 3. Use incremental sync for large datasets
-const lastSync = await getLastSyncTimestamp();
-const newFiles = await sftp.listFiles({
-  lastProcessedTimestamp: lastSync
-});
-logger.info(`Found ${newFiles.length} new files since ${lastSync}`);
-```
-
-## Error Handling
-
-### S3 Error Handling
-
-```typescript
-import { DataSourceError } from '@fluentcommerce/fc-connect-sdk';
-
-try {
-  const files = await s3.listFiles({ prefix: 'data/' });
-} catch (error) {
-  if (error instanceof DataSourceError) {
-    console.error(`Data source error: ${error.message}`);
-    console.error(`Type: ${error.sourceType}`);
-    console.error(`Operation: ${error.operation}`);
-    console.error(`Context:`, error.context);
-
-    // Handle specific errors
-    if (error.context?.statusCode === 403) {
-      throw new Error('Access denied - check AWS credentials');
-    }
-  }
-  throw error;
-}
-```
-
-### SFTP Error Handling
-
-```typescript
-try {
-  const files = await sftp.listFiles();
-} catch (error) {
-  if (error instanceof DataSourceError) {
-    console.error(`SFTP error: ${error.message}`);
-
-    // Retry logic already handled automatically
-    // This error means all retries exhausted
-
-    if (error.message.includes('authentication')) {
-      throw new Error('Authentication failed - check credentials');
-    }
-
-    if (error.message.includes('permission')) {
-      throw new Error('Permission denied - check file permissions');
-    }
-  }
-  throw error;
-}
-```
-
-## Practice Exercise
-
-Create an ingestion workflow that:
-1. Reads CSV files from SFTP with connection pooling
-2. Validates required columns
-3. Writes Parquet to S3 for archiving
-4. Moves processed files on SFTP
-
-<details>
-<summary>Click to see solution</summary>
-
-```typescript
-import { S3DataSource, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function exerciseSolution() {
-  const logger = createConsoleLogger();
-
-  // Configure SFTP with connection pooling
-  const sftp = new SftpDataSource(
-    {
-      type: 'SFTP_CSV',
-      name: 'SFTP Source',
-      settings: {
-        host: process.env.SFTP_HOST!,
-        username: process.env.SFTP_USERNAME!,
-        password: process.env.SFTP_PASSWORD!,
-        remotePath: '/incoming',
-        filePattern: '*.csv',
-
-        maxConnections: 5,
-        retry: {
-          maxAttempts: 3,
-          reconnectOnFailure: true
-        },
-
-        requiredCsvHeaders: ['sku', 'warehouse', 'quantity']
-      }
-    },
-    logger
-  );
-
-  // Configure S3 for archiving
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_PARQUET',
-      name: 'S3 Archive',
-      s3Config: {
-        bucket: 'archive-bucket',
-        region: 'us-east-1',
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-        maxAttempts: 3
-      }
-    },
-    logger
-  );
-
-  try {
-    // Validate connections
-    await sftp.validateConnection();
-    await s3.validateConnection();
-
-    // List files from SFTP
-    const files = await sftp.listFiles();
-    logger.info(`Found ${files.length} files to process`);
-
-    for (const file of files) {
-      try {
-        logger.info(`Processing: ${file.name}`);
-
-        // Download CSV from SFTP
-        const csvContent = await sftp.downloadFile(file.path);
-
-        // Parse CSV (validates required headers)
-        const records = sftp.parseCsvContent(csvContent, file);
-        logger.info(`Parsed ${records.length} records`);
-
-        // Convert to Parquet and write to S3
-        const parquetBuffer = await s3.writeParquetContent(records, {
-          compression: 'GZIP'
-        });
-
-        const archiveKey = `archive/${new Date().toISOString()}/${file.name}.parquet`;
-        await s3.uploadFile(archiveKey, parquetBuffer, {
-          contentType: 'application/octet-stream',
-          metadata: {
-            originalFile: file.name,
-            recordCount: records.length.toString()
-          }
-        });
-
-        logger.info(`Archived to S3: ${archiveKey}`);
-
-        // Move processed file on SFTP
-        await sftp.moveFile(
-          file.path,
-          `/processed/${file.name}`
-        );
-
-        logger.info(`Moved on SFTP: ${file.path} → /processed/${file.name}`);
-
-      } catch (fileError) {
-        logger.error(`Error processing ${file.name}:`, fileError);
-        // Continue with next file
-      }
-    }
-
-    // Log final stats
-    const sftpInfo = sftp.getConnectionInfo();
-    logger.info(`SFTP pool: ${sftpInfo.connectionPoolSize}/${sftpInfo.maxPoolSize}`);
-
-  } catch (error) {
-    logger.error('Workflow failed:', error);
-    throw error;
-  }
-}
-```
-
-</details>
-
-## Key Takeaways
-
-- 🎯 **S3DataSource** handles S3 operations with automatic retry and exponential backoff
-- 🎯 **SftpDataSource** includes connection pooling (v0.1.11+) and enhanced retry logic
-- 🎯 **Connection pooling** (SFTP only) improves performance for concurrent operations
-- 🎯 **Retry logic** handles transient failures automatically (both S3 and SFTP)
-- 🎯 **Integrated parsing** (SFTP only) - CSV, JSON, Parquet parsing built-in
-- 🎯 **File operations** include upload, download, move, delete, archive
-- 🎯 **Parquet support** - Both S3 and SFTP can write Parquet files
-- 🎯 **Error handling** - Use `DataSourceError` for detailed error context
-
-## Next Steps
-
-Continue to [Module 4: Field Mapping →](./02-core-guides-ingestion-04-field-mapping.md) to learn how to transform parsed data for Fluent Commerce.
-
----
-
-**Related Documentation:**
-- [S3DataSource API Reference](../../api-reference/modules/api-reference-06-data-sources.md#s3-data-source)
-- [SftpDataSource API Reference](../../api-reference/modules/api-reference-06-data-sources.md#sftp-data-source)
-- [SFTP Data Source Guide](../../data-sources/modules/data-sources-03-sftp-operations.md#connection-pooling)
-- [Error Handling Patterns](../../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md#retry-patterns)
+# Module 3: Data Sources & Parsers
+
+**Level:** Intermediate
+**Estimated Time:** 30 minutes
+
+## Overview
+
+This module teaches you how to read inventory data from different sources (S3, SFTP) and parse various file formats (CSV, Parquet, XML, JSON) using the SDK's data source services.
+
+**IMPORTANT:** Data sources in this SDK have integrated parsing capabilities. You don't need separate parser services - each data source handles its own file formats.
+
+## Learning Objectives
+
+By the end of this module, you will:
+- ✅ Understand the data source abstraction pattern
+- ✅ Configure S3DataSource for AWS S3 buckets with retry logic
+- ✅ Configure SftpDataSource with connection pooling and retry
+- ✅ Parse CSV, JSON, and Parquet files directly from data sources
+- ✅ Handle large files with streaming and batching
+- ✅ Choose the right data source configuration for your use case
+
+## Data Source Abstraction
+
+The SDK provides data sources that handle both file operations AND parsing:
+
+```typescript
+import { S3DataSource, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+// Each data source is a complete file + parsing solution
+// No need for separate parser services
+```
+
+### Benefits
+
+| Benefit | Description |
+|---------|-------------|
+| **Integrated** | File operations and parsing in one service |
+| **Retry Logic** | Automatic exponential backoff for transient failures |
+| **Connection Pooling** | (SFTP only) Reuse connections for performance |
+| **Format Support** | CSV, JSON, XML, Parquet, JSONL |
+| **Testability** | Mock data sources for unit testing |
+
+## S3 Data Source
+
+### Configuration
+
+```typescript
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+const logger = createConsoleLogger();
+
+const s3 = new S3DataSource(
+  {
+    type: 'S3_CSV',  // or 'S3_JSON', 'S3_XML', 'S3_PARQUET'
+    name: 'S3 Inventory Source',
+    connectionId: 's3-prod',
+    s3Config: {
+      bucket: 'inventory-bucket',
+      region: 'us-east-1',
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+
+      // Optional: Session token for temporary credentials
+      sessionToken: process.env.AWS_SESSION_TOKEN,
+
+      // Optional: Retry configuration
+      maxAttempts: 3  // Default: 3 attempts with exponential backoff
+    }
+  },
+  logger
+);
+```
+
+### Basic Operations
+
+```typescript
+// List files with filtering
+const files = await s3.listFiles({
+  prefix: 'data/',           // Filter by prefix
+  delimiter: '/',            // Group by "directory"
+  maxKeys: 1000,             // Max results per page
+  continuationToken: token   // For pagination
+});
+
+console.log(`Found ${files.length} files`);
+files.forEach(file => {
+  console.log(`  ${file.path} (${file.size} bytes, ${file.lastModified})`);
+});
+
+// Download file
+const content = await s3.downloadFile('data/inventory-2024-01-15.csv', {
+  encoding: 'utf8'  // or 'binary' for Buffer
+});
+
+// Download binary files (images, Parquet, etc.)
+const binaryData = await s3.downloadFile('data/inventory.parquet', {
+  encoding: 'binary'
+});
+```
+
+### Retry Configuration
+
+S3DataSource includes automatic retry with exponential backoff for transient failures:
+
+```typescript
+const s3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    name: 'S3 with Custom Retry',
+    s3Config: {
+      bucket: 'inventory-bucket',
+      region: 'us-east-1',
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+
+      // Retry configuration
+      maxAttempts: 5  // Try up to 5 times (default: 3)
+    }
+  },
+  logger
+);
+
+// Retry behavior:
+// - Attempt 1: Immediate
+// - Attempt 2: 500ms delay
+// - Attempt 3: 1000ms delay
+// - Attempt 4: 2000ms delay
+// - Attempt 5: 4000ms delay
+// - Max delay: 8000ms
+
+// Retries on: 429, 500, 502, 503, 504 status codes
+// Retries on: Network errors (ECONNRESET, ETIMEDOUT, etc.)
+```
+
+### Upload Operations
+
+```typescript
+// Upload text file
+await s3.uploadFile(
+  'processed/inventory-2024-01-15.csv',
+  csvContent,
+  {
+    contentType: 'text/csv',
+    metadata: {
+      processedAt: new Date().toISOString(),
+      source: 'inventory-sync'
+    }
+  }
+);
+
+// Upload binary file
+await s3.uploadFile(
+  'data/inventory.parquet',
+  parquetBuffer,
+  {
+    contentType: 'application/octet-stream'
+  }
+);
+```
+
+### File Management
+
+```typescript
+// Copy file
+await s3.copyFile(
+  'data/inventory.csv',
+  'backup/inventory.csv',
+  {
+    sourceBucket: 'source-bucket',  // Optional: cross-bucket copy
+    metadata: { backupDate: new Date().toISOString() }
+  }
+);
+
+// Move file within same bucket (copy + delete)
+await s3.moveFile(
+  'temp/inventory.csv',
+  'archive/inventory.csv'
+  // Optional: pass false to copy without deleting source
+  // await s3.moveFile('temp/inventory.csv', 'archive/inventory.csv', false);
+);
+
+// Archive file with automatic timestamping
+const archivePath = await s3.archiveFile(
+  'data/inventory.csv',
+  'archive/2024-01/'  // Archive prefix (timestamp + filename added automatically)
+);
+console.log(`File archived to: ${archivePath}`);
+// Returns: 'archive/2024-01/2025-10-29T12-30-45-123Z_inventory.csv'
+
+// For cross-bucket moves, use copyFile() + deleteFile() directly
+await s3.copyFile('inventory.csv', 'backup/inventory.csv', {
+  sourceBucket: 'source-bucket',
+  metadata: { archivedAt: new Date().toISOString() }
+});
+await s3.deleteFile('inventory.csv');
+
+// Delete file
+await s3.deleteFile('temp/old-file.csv');
+```
+
+### Parquet Support
+
+S3DataSource has built-in Parquet writing support:
+
+```typescript
+// Write records as Parquet file
+const records = [
+  { sku: 'ABC123', warehouse: 'WH1', quantity: 100 },
+  { sku: 'DEF456', warehouse: 'WH2', quantity: 50 }
+];
+
+const parquetBuffer = await s3.writeParquetContent(records, {
+  compression: 'GZIP',  // or 'SNAPPY', 'BROTLI', 'UNCOMPRESSED'
+  rowGroupSize: 5000,   // Records per row group
+  schema: {             // Optional: explicit schema
+    sku: { type: 'UTF8' },
+    warehouse: { type: 'UTF8' },
+    quantity: { type: 'INT64' }
+  }
+});
+
+// Upload Parquet file
+await s3.uploadFile(
+  'data/inventory.parquet',
+  parquetBuffer,
+  { contentType: 'application/octet-stream' }
+);
+```
+
+### Connection Validation
+
+```typescript
+// Validate S3 connection before processing
+const isValid = await s3.validateConnection();
+if (!isValid) {
+  throw new Error('Failed to connect to S3');
+}
+```
+
+### Complete S3 Example
+
+```typescript
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function processS3Files() {
+  const logger = createConsoleLogger();
+
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_CSV',
+      name: 'Inventory Source',
+      s3Config: {
+        bucket: 'inventory-bucket',
+        region: 'us-east-1',
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+        maxAttempts: 3
+      }
+    },
+    logger
+  );
+
+  try {
+    // Validate connection
+    await s3.validateConnection();
+
+    // List files
+    const files = await s3.listFiles({ prefix: 'incoming/' });
+
+    for (const file of files) {
+      logger.info(`Processing: ${file.path} (${file.size} bytes)`);
+
+      // Download file
+      const content = await s3.downloadFile(file.path);
+
+      // Process content...
+      // (Use CSVParserService or other parsers as needed)
+
+      // Archive processed file (timestamp added automatically)
+      const archivedPath = await s3.archiveFile(
+        file.path,
+        `archive/${new Date().toISOString().split('T')[0]}/`  // Just the prefix
+      );
+      logger.info(`Archived to: ${archivedPath}`);
+    }
+
+    logger.info('Processing complete');
+  } catch (error) {
+    logger.error('Processing failed', error);
+    throw error;
+  }
+}
+```
+
+## SFTP Data Source
+
+### Configuration
+
+```typescript
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+import * as fs from 'fs';
+
+const logger = createConsoleLogger();
+
+const sftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',  // or 'SFTP_JSON', 'SFTP_XML', 'SFTP_JSONL', 'SFTP_PARQUET'
+    name: 'SFTP Inventory Source',
+    connectionId: 'sftp-prod',
+    settings: {
+      host: 'sftp.example.com',
+      port: 22,
+      remotePath: '/data/inventory',
+      filePattern: '*.csv',
+
+      // Authentication: password OR private key
+      username: process.env.SFTP_USERNAME!,
+      password: process.env.SFTP_PASSWORD,
+
+      // OR use private key
+      privateKey: fs.readFileSync('/path/to/key.pem', 'utf8'),
+      passphrase: process.env.SFTP_KEY_PASSPHRASE,
+
+      // Connection pooling (NEW in v0.1.11+)
+      maxConnections: 5,  // Default: 5
+
+      // Retry configuration (NEW in v0.1.11+)
+      retry: {
+        maxAttempts: 3,      // Default: 3
+        baseDelayMs: 1000,   // Default: 1000ms
+        backoffFactor: 2,    // Default: 2 (exponential)
+        maxDelayMs: 8000,    // Default: 8000ms
+        jitter: 'full',      // 'full' or 'none'
+        reconnectOnFailure: true  // Reconnect on failure
+      },
+
+      // CSV parsing settings
+      csvDelimiter: ',',
+      csvHeaders: ['sku', 'warehouse', 'quantity'],
+      requiredCsvHeaders: ['sku', 'quantity'],
+      csvSkipEmptyLines: true,
+      csvTrimValues: true,
+
+      // JSON parsing settings
+      jsonFormat: 'json',  // or 'jsonl'
+      jsonValidate: true,
+      jsonMaxDepth: 10,
+
+      // Parquet settings
+      parquetCompression: 'gzip'
+    }
+  },
+  logger
+);
+```
+
+### Connection Pooling (NEW in v0.1.11+)
+
+SftpDataSource automatically manages a connection pool for performance:
+
+```typescript
+// Connection pool behavior:
+// 1. Reuses existing connections when available
+// 2. Creates new connections up to maxConnections limit
+// 3. Waits in queue when pool is full (30s timeout)
+// 4. Validates connection health before reuse
+// 5. Auto-reconnects on transient failures
+
+const sftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    name: 'SFTP with Pool',
+    settings: {
+      host: 'sftp.example.com',
+      username: process.env.SFTP_USERNAME!,
+      password: process.env.SFTP_PASSWORD!,
+      remotePath: '/data',
+      filePattern: '*.csv',
+
+      // Pool configuration
+      maxConnections: 10,  // Allow up to 10 concurrent connections
+      connectionTimeout: 10000,  // 10s connection timeout
+      keepAliveInterval: 10000   // Keep-alive every 10s
+    }
+  },
+  logger
+);
+
+// Connection pool is transparent - all operations use it automatically
+const files = await sftp.listFiles();  // Uses pool
+const content = await sftp.downloadFile(files[0].name);  // Uses pool
+
+// Get pool stats for monitoring
+const info = sftp.getConnectionInfo();
+console.log(`Pool size: ${info.connectionPoolSize}/${info.maxPoolSize}`);
+```
+
+### Enhanced Retry Logic (NEW in v0.1.11+)
+
+SFTP operations automatically retry with exponential backoff:
+
+```typescript
+const sftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    name: 'SFTP with Custom Retry',
+    settings: {
+      host: 'sftp.example.com',
+      username: process.env.SFTP_USERNAME!,
+      password: process.env.SFTP_PASSWORD!,
+      remotePath: '/data',
+      filePattern: '*.csv',
+
+      retry: {
+        maxAttempts: 5,        // Try up to 5 times
+        baseDelayMs: 2000,     // Start with 2s delay
+        backoffFactor: 2,      // Double delay each attempt
+        maxDelayMs: 16000,     // Cap at 16s
+        jitter: 'full',        // Random jitter (0 to maxDelay)
+        reconnectOnFailure: true,  // Clear pool and reconnect
+
+        // Custom retry predicate
+        isRetryable: (error: Error) => {
+          // Add custom logic for retryable errors
+          return error.message.includes('timeout') ||
+                 error.message.includes('connection');
+        }
+      }
+    }
+  },
+  logger
+);
+
+// Retry timeline with above config:
+// - Attempt 1: Immediate
+// - Attempt 2: Random(0-2000ms) delay
+// - Attempt 3: Random(0-4000ms) delay
+// - Attempt 4: Random(0-8000ms) delay
+// - Attempt 5: Random(0-16000ms) delay
+
+// Automatically retries on:
+// - ECONNRESET, ETIMEDOUT, ENOTFOUND, EHOSTUNREACH, ECONNREFUSED
+// - "connection lost", "handshake failed", "timeout", etc.
+//
+// Does NOT retry on:
+// - Authentication failures
+// - Permission denied
+// - File not found
+// - Algorithm/cipher/key exchange errors
+```
+
+### Basic Operations
+
+```typescript
+// List files with filtering
+const files = await sftp.listFiles({
+  remotePath: '/data/inventory',  // Optional: override config
+  filePattern: '*.csv',           // Optional: override config
+  lastProcessedTimestamp: '2024-01-15T00:00:00Z',  // Incremental sync
+  fileName: 'inventory'           // Filter by name substring
+});
+
+console.log(`Found ${files.length} files`);
+
+// Download file
+const content = await sftp.downloadFile('/data/inventory.csv', {
+  encoding: 'utf8',  // Default encoding
+  asBuffer: false,   // Return string
+  maxSize: 100 * 1024 * 1024  // 100MB limit
+});
+
+// Download as buffer
+const buffer = await sftp.downloadFile('/data/inventory.parquet', {
+  asBuffer: true
+});
+```
+
+### Integrated CSV Parsing
+
+SFTP data source has built-in CSV parsing:
+
+```typescript
+// Download and parse CSV in one step
+const csvContent = await sftp.downloadFile('/data/inventory.csv');
+
+const records = sftp.parseCsvContent(csvContent, {
+  name: 'inventory.csv',
+  path: '/data/inventory.csv',
+  size: csvContent.length,
+  source: 'sftp'
+});
+
+console.log(`Parsed ${records.length} records`);
+records.forEach(record => {
+  console.log(`SKU: ${record.sku}, Qty: ${record.quantity}`);
+});
+```
+
+### Integrated JSON Parsing
+
+```typescript
+// Download and parse JSON
+const jsonContent = await sftp.downloadFile('/data/inventory.json');
+
+const records = sftp.parseJsonContent(
+  jsonContent,
+  { name: 'inventory.json', path: '/data/inventory.json', source: 'sftp' },
+  {
+    format: 'json',    // or 'jsonl' for JSON Lines
+    validate: true,    // Validate structure
+    maxDepth: 10       // Limit nesting depth
+  }
+);
+
+console.log(`Parsed ${records.length} JSON records`);
+```
+
+### Writing Files
+
+```typescript
+// Write CSV file
+const records = [
+  { sku: 'ABC123', warehouse: 'WH1', quantity: 100 },
+  { sku: 'DEF456', warehouse: 'WH2', quantity: 50 }
+];
+
+const csvContent = sftp.writeCsvContent(records, {
+  delimiter: ',',
+  includeHeader: true,
+  quote: '"',
+  escape: '"'
+});
+
+await sftp.uploadFile('/data/output.csv', {
+  overwrite: true,
+  createDirectories: true
+}, csvContent);
+
+// Write JSON file
+const jsonContent = sftp.writeJsonContent(records, {
+  format: 'json',  // or 'jsonl'
+  indent: 2,
+  sortKeys: true
+});
+
+await sftp.uploadFile('/data/output.json', {
+  overwrite: true
+}, jsonContent);
+
+// Write Parquet file
+const parquetBuffer = await sftp.writeParquetContent(records, {
+  compression: 'GZIP',
+  rowGroupSize: 5000
+});
+
+await sftp.uploadFile('/data/output.parquet', {
+  overwrite: true
+}, parquetBuffer);
+```
+
+### File Management
+
+```typescript
+// Upload file from local path
+await sftp.uploadFile(
+  '/local/path/inventory.csv',
+  '/remote/path/inventory.csv',
+  {
+    overwrite: false,
+    createDirectories: true,
+    permissions: '0644'
+  }
+);
+
+// Create directory
+await sftp.createDirectory('/data/archive', true);  // recursive
+
+// Move file
+await sftp.moveFile(
+  '/data/inventory.csv',
+  '/data/processed/inventory.csv'
+);
+
+// Delete file
+await sftp.deleteFile('/data/old-file.csv');
+
+// Check if file exists
+const exists = await sftp.fileExists('/data/inventory.csv');
+if (exists) {
+  console.log('File exists');
+}
+
+// Check if directory exists
+const dirExists = await sftp.directoryExists('/data/archive');
+```
+
+### Connection Monitoring
+
+```typescript
+// Validate connection
+const isValid = await sftp.validateConnection();
+if (!isValid) {
+  throw new Error('SFTP connection failed');
+}
+
+// Get connection pool info
+const info = sftp.getConnectionInfo();
+console.log(`Connection Info:
+  Type: ${info.type}
+  Host: ${info.host}:${info.port}
+  Remote Path: ${info.remotePath}
+  File Pattern: ${info.filePattern}
+  Pool: ${info.connectionPoolSize}/${info.maxPoolSize} connections
+`);
+```
+
+### Complete SFTP Example
+
+```typescript
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function processSftpFiles() {
+  const logger = createConsoleLogger();
+
+  const sftp = new SftpDataSource(
+    {
+      type: 'SFTP_CSV',
+      name: 'SFTP Inventory',
+      settings: {
+        host: process.env.SFTP_HOST!,
+        username: process.env.SFTP_USERNAME!,
+        password: process.env.SFTP_PASSWORD!,
+        remotePath: '/data/inventory',
+        filePattern: '*.csv',
+
+        // Enable connection pooling and retry
+        maxConnections: 5,
+        retry: {
+          maxAttempts: 3,
+          reconnectOnFailure: true
+        },
+
+        // CSV parsing settings
+        csvDelimiter: ',',
+        requiredCsvHeaders: ['sku', 'quantity'],
+        csvSkipEmptyLines: true
+      }
+    },
+    logger
+  );
+
+  try {
+    // Validate connection
+    await sftp.validateConnection();
+
+    // List files
+    const files = await sftp.listFiles();
+    logger.info(`Found ${files.length} files`);
+
+    for (const file of files) {
+      logger.info(`Processing: ${file.name}`);
+
+      // Download file
+      const csvContent = await sftp.downloadFile(file.path);
+
+      // Parse CSV (integrated parsing)
+      const records = sftp.parseCsvContent(csvContent, file);
+      logger.info(`Parsed ${records.length} records`);
+
+      // Process records...
+      // (Send to Fluent Commerce, etc.)
+
+      // Move to processed folder
+      await sftp.moveFile(
+        file.path,
+        `/data/processed/${file.name}`
+      );
+    }
+
+    // Get final pool stats
+    const info = sftp.getConnectionInfo();
+    logger.info(`Pool: ${info.connectionPoolSize}/${info.maxPoolSize}`);
+
+  } catch (error) {
+    logger.error('Processing failed', error);
+    throw error;
+  }
+}
+```
+
+## Inventory Data Source (Base Interface)
+
+**Note:** `InventoryDataSource` is an abstract base interface for inventory-specific operations. Use the concrete implementations `S3DataSource` or `SftpDataSource` directly.
+
+```typescript
+import { InventoryDataSource, BaseInventoryDataSource } from '@fluentcommerce/fc-connect-sdk';
+
+// This is an interface, not a standalone data source
+interface InventoryDataSource {
+  getBucketName(): string;
+  getCsvFilePrefix(): string;
+  parseS3ListResponse(xmlResponse: string, prefix: string, lastModifiedAfter?: string): FileMetadata[];
+  getConfig(): DataSourceConfig;
+}
+
+// Base implementation
+class BaseInventoryDataSource implements InventoryDataSource {
+  constructor(protected config: DataSourceConfig) {}
+
+  getBucketName(): string {
+    return 'default-bucket';
+  }
+
+  getCsvFilePrefix(): string {
+    return 'inventory/';
+  }
+
+  parseS3ListResponse(xmlResponse: string, prefix: string, lastModifiedAfter?: string): FileMetadata[] {
+    // Simple XML parsing for S3 list response
+    // ...
+  }
+
+  getConfig(): DataSourceConfig {
+    return this.config;
+  }
+}
+```
+
+**Recommendation:** Use `S3DataSource` or `SftpDataSource` instead of implementing this interface.
+
+## Choosing the Right Data Source
+
+| Feature | S3DataSource | SftpDataSource |
+|---------|-------------|---------------|
+| **Connection Pooling** | ❌ Not needed (HTTP) | ✅ Yes (v0.1.11+) |
+| **Retry Logic** | ✅ Exponential backoff | ✅ Enhanced with reconnect |
+| **Formats** | CSV, JSON, XML, Parquet | CSV, JSON, XML, JSONL, Parquet |
+| **Integrated Parsing** | ❌ Use separate parsers | ✅ Built-in parsers |
+| **Streaming** | ✅ Download streams | ❌ Download then parse |
+| **File Operations** | List, Download, Upload, Copy, Move, Delete, Archive | List, Download, Upload, Move, Delete |
+| **Best For** | Cloud storage, large files, archiving | FTP servers, real-time feeds |
+| **Performance** | Excellent (parallel downloads) | Good (connection pool) |
+
+### Decision Tree
+
+```typescript
+function chooseDataSource(requirements: {
+  source: 'cloud' | 'sftp';
+  fileSize: 'small' | 'large';
+  frequency: 'realtime' | 'batch';
+  format: string;
+}): 'S3' | 'SFTP' {
+  if (requirements.source === 'cloud') {
+    return 'S3';  // S3 for cloud storage
+  }
+
+  if (requirements.source === 'sftp') {
+    if (requirements.frequency === 'realtime') {
+      // SFTP with connection pooling for frequent operations
+      return 'SFTP';
+    }
+
+    if (requirements.fileSize === 'large') {
+      // S3 better for very large files (streaming)
+      return 'S3';
+    }
+
+    return 'SFTP';  // SFTP for FTP servers
+  }
+
+  return 'S3';  // Default to S3
+}
+```
+
+## Performance Considerations
+
+### S3 Best Practices
+
+```typescript
+// 1. Use pagination for large lists
+let continuationToken: string | undefined;
+do {
+  const files = await s3.listFiles({
+    prefix: 'data/',
+    maxKeys: 1000,
+    continuationToken
+  });
+
+  // Process files...
+
+  // Get next page token (if available)
+  continuationToken = files.length === 1000 ? files[files.length - 1].path : undefined;
+} while (continuationToken);
+
+// 2. Use binary encoding for non-text files
+const parquetData = await s3.downloadFile('data/large.parquet', {
+  encoding: 'binary'
+});
+
+// 3. Enable retry for production
+const s3 = new S3DataSource(
+  {
+    type: 'S3_PARQUET',
+    s3Config: {
+      bucket: 'prod-bucket',
+      region: 'us-east-1',
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      maxAttempts: 5  // Production: 5 attempts
+    }
+  },
+  logger
+);
+```
+
+### SFTP Best Practices
+
+```typescript
+// 1. Configure connection pool for concurrent operations
+const sftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    settings: {
+      host: 'sftp.example.com',
+      username: process.env.SFTP_USERNAME!,
+      password: process.env.SFTP_PASSWORD!,
+      remotePath: '/data',
+      filePattern: '*.csv',
+
+      // Tune pool size based on workload
+      maxConnections: 10,  // Higher for concurrent operations
+
+      retry: {
+        maxAttempts: 5,      // Production: more retries
+        reconnectOnFailure: true
+      }
+    }
+  },
+  logger
+);
+
+// 2. Monitor pool usage
+const info = sftp.getConnectionInfo();
+if (info.connectionPoolSize === info.maxPoolSize) {
+  logger.warn('Connection pool at capacity - consider increasing maxConnections');
+}
+
+// 3. Use incremental sync for large datasets
+const lastSync = await getLastSyncTimestamp();
+const newFiles = await sftp.listFiles({
+  lastProcessedTimestamp: lastSync
+});
+logger.info(`Found ${newFiles.length} new files since ${lastSync}`);
+```
+
+## Error Handling
+
+### S3 Error Handling
+
+```typescript
+import { DataSourceError } from '@fluentcommerce/fc-connect-sdk';
+
+try {
+  const files = await s3.listFiles({ prefix: 'data/' });
+} catch (error) {
+  if (error instanceof DataSourceError) {
+    console.error(`Data source error: ${error.message}`);
+    console.error(`Type: ${error.sourceType}`);
+    console.error(`Operation: ${error.operation}`);
+    console.error(`Context:`, error.context);
+
+    // Handle specific errors
+    if (error.context?.statusCode === 403) {
+      throw new Error('Access denied - check AWS credentials');
+    }
+  }
+  throw error;
+}
+```
+
+### SFTP Error Handling
+
+```typescript
+try {
+  const files = await sftp.listFiles();
+} catch (error) {
+  if (error instanceof DataSourceError) {
+    console.error(`SFTP error: ${error.message}`);
+
+    // Retry logic already handled automatically
+    // This error means all retries exhausted
+
+    if (error.message.includes('authentication')) {
+      throw new Error('Authentication failed - check credentials');
+    }
+
+    if (error.message.includes('permission')) {
+      throw new Error('Permission denied - check file permissions');
+    }
+  }
+  throw error;
+}
+```
+
+## Practice Exercise
+
+Create an ingestion workflow that:
+1. Reads CSV files from SFTP with connection pooling
+2. Validates required columns
+3. Writes Parquet to S3 for archiving
+4. Moves processed files on SFTP
+
+<details>
+<summary>Click to see solution</summary>
+
+```typescript
+import { S3DataSource, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+import { createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function exerciseSolution() {
+  const logger = createConsoleLogger();
+
+  // Configure SFTP with connection pooling
+  const sftp = new SftpDataSource(
+    {
+      type: 'SFTP_CSV',
+      name: 'SFTP Source',
+      settings: {
+        host: process.env.SFTP_HOST!,
+        username: process.env.SFTP_USERNAME!,
+        password: process.env.SFTP_PASSWORD!,
+        remotePath: '/incoming',
+        filePattern: '*.csv',
+
+        maxConnections: 5,
+        retry: {
+          maxAttempts: 3,
+          reconnectOnFailure: true
+        },
+
+        requiredCsvHeaders: ['sku', 'warehouse', 'quantity']
+      }
+    },
+    logger
+  );
+
+  // Configure S3 for archiving
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_PARQUET',
+      name: 'S3 Archive',
+      s3Config: {
+        bucket: 'archive-bucket',
+        region: 'us-east-1',
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+        maxAttempts: 3
+      }
+    },
+    logger
+  );
+
+  try {
+    // Validate connections
+    await sftp.validateConnection();
+    await s3.validateConnection();
+
+    // List files from SFTP
+    const files = await sftp.listFiles();
+    logger.info(`Found ${files.length} files to process`);
+
+    for (const file of files) {
+      try {
+        logger.info(`Processing: ${file.name}`);
+
+        // Download CSV from SFTP
+        const csvContent = await sftp.downloadFile(file.path);
+
+        // Parse CSV (validates required headers)
+        const records = sftp.parseCsvContent(csvContent, file);
+        logger.info(`Parsed ${records.length} records`);
+
+        // Convert to Parquet and write to S3
+        const parquetBuffer = await s3.writeParquetContent(records, {
+          compression: 'GZIP'
+        });
+
+        const archiveKey = `archive/${new Date().toISOString()}/${file.name}.parquet`;
+        await s3.uploadFile(archiveKey, parquetBuffer, {
+          contentType: 'application/octet-stream',
+          metadata: {
+            originalFile: file.name,
+            recordCount: records.length.toString()
+          }
+        });
+
+        logger.info(`Archived to S3: ${archiveKey}`);
+
+        // Move processed file on SFTP
+        await sftp.moveFile(
+          file.path,
+          `/processed/${file.name}`
+        );
+
+        logger.info(`Moved on SFTP: ${file.path} → /processed/${file.name}`);
+
+      } catch (fileError) {
+        logger.error(`Error processing ${file.name}:`, fileError);
+        // Continue with next file
+      }
+    }
+
+    // Log final stats
+    const sftpInfo = sftp.getConnectionInfo();
+    logger.info(`SFTP pool: ${sftpInfo.connectionPoolSize}/${sftpInfo.maxPoolSize}`);
+
+  } catch (error) {
+    logger.error('Workflow failed:', error);
+    throw error;
+  }
+}
+```
+
+</details>
+
+## Key Takeaways
+
+- 🎯 **S3DataSource** handles S3 operations with automatic retry and exponential backoff
+- 🎯 **SftpDataSource** includes connection pooling (v0.1.11+) and enhanced retry logic
+- 🎯 **Connection pooling** (SFTP only) improves performance for concurrent operations
+- 🎯 **Retry logic** handles transient failures automatically (both S3 and SFTP)
+- 🎯 **Integrated parsing** (SFTP only) - CSV, JSON, Parquet parsing built-in
+- 🎯 **File operations** include upload, download, move, delete, archive
+- 🎯 **Parquet support** - Both S3 and SFTP can write Parquet files
+- 🎯 **Error handling** - Use `DataSourceError` for detailed error context
+
+## Next Steps
+
+Continue to [Module 4: Field Mapping →](./02-core-guides-ingestion-04-field-mapping.md) to learn how to transform parsed data for Fluent Commerce.
+
+---
+
+**Related Documentation:**
+- [S3DataSource API Reference](../../api-reference/modules/api-reference-06-data-sources.md#s3-data-source)
+- [SftpDataSource API Reference](../../api-reference/modules/api-reference-06-data-sources.md#sftp-data-source)
+- [SFTP Data Source Guide](../../data-sources/modules/data-sources-03-sftp-operations.md#connection-pooling)
+- [Error Handling Patterns](../../../03-PATTERN-GUIDES/error-handling/error-handling-readme.md#retry-patterns)