@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/02-CORE-GUIDES/data-sources/data-sources-quick-reference.md

@@ -1,836 +1,836 @@
-# Data Sources - Quick Reference
-
-> One-page cheat sheet for S3DataSource and SftpDataSource
-
-## Quick Decision: S3 vs SFTP
-
-```
-Need to choose between S3 and SFTP?
-
-Do you control the infrastructure?
-    ├─ YES → Is this cloud-native?
-    │         ├─ YES → Use S3
-    │         └─ NO  → Is volume high (>10K/hour)?
-    │                   ├─ YES → Use S3
-    │                   └─ NO  → Either (your preference)
-    └─ NO  → Use SFTP (vendor requirement)
-```
-
----
-
-## S3DataSource - Quick Start
-
-### Basic Configuration
-
-```typescript
-import {
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV', // S3_JSON, S3_XML, S3_PARQUET
-    connectionId: 's3-quick-ref',
-    name: 'S3 Quick Ref',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-      bucket: 'my-bucket', // Required
-    },
-  },
-  logger
-);
-```
-
-### Common Operations
-
-```typescript
-// List files
-const files = await s3Source.listFiles({ prefix: 'inventory/' });
-
-// Read file
-const content = await s3Source.downloadFile('file.csv');
-
-// Write file
-await s3Source.uploadFile('output.csv', csvData, {
-  contentType: 'text/csv',
-  metadata: { 'processed-date': new Date().toISOString() },
-});
-
-// Delete file
-await s3Source.deleteFile('old-file.csv');
-
-// Copy file
-await s3Source.copyFile('source.csv', 'archive/source.csv');
-
-// Move file (copy + delete)
-await s3Source.moveFile('incoming/file.csv', 'processed/file.csv');
-
-// Check existence
-// For S3 existence, list by prefix or handle 404 from download
-
-// Validate connection
-const isValid = await s3Source.validateConnection();
-```
-
-### Parquet Generation
-
-```typescript
-// Generate Parquet buffer
-const records = [
-  { id: 1, sku: 'PROD-001', quantity: 100, price: 29.99 },
-  { id: 2, sku: 'PROD-002', quantity: 200, price: 39.99 },
-];
-
-const parquetBuffer = await s3Source.writeParquetContent(records, {
-  compression: 'GZIP', // UNCOMPRESSED, GZIP, SNAPPY, BROTLI
-  rowGroupSize: 50000,
-});
-
-// Upload to S3
-await s3Source.writeFile('s3://bucket/data.parquet', parquetBuffer, {
-  contentType: 'application/octet-stream',
-});
-```
-
----
-
-## SftpDataSource - Quick Start
-
-### Basic Configuration
-
-```typescript
-import {
-  SftpDataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV', // SFTP_JSON, SFTP_JSONL, SFTP_PARQUET
-    connectionId: 'vendor-sftp',
-    name: 'Vendor SFTP Connection',
-    settings: {
-      host: 'sftp.vendor.com',
-      port: 22,
-      username: 'integration',
-      privateKey: process.env.SFTP_PRIVATE_KEY, // or password
-      remotePath: '/data/inventory',
-      filePattern: '*.csv',
-    },
-  },
-  logger
-);
-```
-
-### Common Operations
-
-```typescript
-// List files
-const files = await sftpSource.listFiles({
-  remotePath: '/data/current',
-  filePattern: '*.csv',
-});
-
-// Read file
-const content = await sftpSource.downloadFile('/data/inventory.csv');
-
-// Write file
-await sftpSource.writeFile('/exports/output.csv', csvData, {
-  createDirectories: true,
-  overwrite: false,
-  permissions: '0644',
-});
-
-// Delete file
-await sftpSource.deleteFile('/temp/old-file.csv');
-
-// Copy file (download + upload)
-await sftpSource.copyFile(
-  '/source/file.csv',
-  '/backup/file.csv',
-  true // overwrite
-);
-
-// Move file (atomic rename)
-await sftpSource.moveFile(
-  '/incoming/file.csv',
-  '/processed/file.csv',
-  false // don't overwrite
-);
-
-// Create directory
-await sftpSource.createDirectory('/exports/daily', true, '0755');
-
-// Check existence
-const fileExists = await sftpSource.fileExists('/data/file.csv');
-const dirExists = await sftpSource.directoryExists('/data/exports');
-
-// Validate connection
-const isValid = await sftpSource.validateConnection();
-```
-
----
-
-## Authentication
-
-### S3 (AWS Credentials)
-
-```bash
-# Environment variables
-AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
-AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
-AWS_REGION=us-east-1
-AWS_SESSION_TOKEN=optional-session-token  # For temporary credentials
-```
-
-### SFTP (SSH)
-
-**Standalone - Password Auth**:
-
-```typescript
-settings: {
-  host: 'sftp.vendor.com',
-  username: 'integration',
-  password: process.env.SFTP_PASSWORD
-}
-```
-
-**Standalone - Private Key Auth (Recommended)**:
-
-```typescript
-settings: {
-  host: 'sftp.vendor.com',
-  username: 'integration',
-  privateKey: process.env.SFTP_PRIVATE_KEY,
-  passphrase: process.env.SFTP_PASSPHRASE  // If key is encrypted
-}
-```
-
-```bash
-# Environment variable (PEM format, NOT file path)
-SFTP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
-MIIEpAIBAAKCAQEA...
------END RSA PRIVATE KEY-----"
-```
-
----
-
-## SFTP Credentials (Versori Platform)
-
-**For secure credential access in Versori connectors**, use one of these methods:
-
-### Method 1: activation.connections (Recommended)
-
-```typescript
-// Already decoded - no Buffer needed!
-const allConns = ctx.activation.connections || [];
-const sftpConn = allConns.find(c => c.name === 'versori_ftp_server');
-const { username, password } = sftpConn.credentials[0].credential.data.basicAuth;
-```
-
-### Method 2: credentials().get()
-
-```typescript
-import { Buffer } from 'node:buffer';
-
-// Requires base64 decoding
-const sftpCred = await ctx.credentials().getAccessToken('SFTP');
-const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
-const [username, password] = rawBasicAuth.split(':');
-```
-
-**See:** [sftp-credential-access-security.md](./data-sources-sftp-credential-access-security.md) for complete guide with error handling, validation, and security best practices.
-
----
-
-## Dual AWS Credentials (Source + Target)
-
-```typescript
-// Source S3 (Vendor - READ)
-const sourceS3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-      bucket: process.env.SOURCE_BUCKET,
-    },
-  },
-  logger
-);
-
-// Target S3 (Your Data Lake - WRITE)
-const targetS3 = new S3DataSource(
-  {
-    type: 'S3_PARQUET',
-    s3Config: {
-      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
-      region: 'us-west-2',
-      bucket: process.env.TARGET_BUCKET,
-    },
-  },
-  logger
-);
-
-// Read from vendor, write to your data lake
-const content = await sourceS3.downloadFile('inventory.csv');
-const processed = await processData(content);
-await targetS3.writeFile('s3://your-bucket/processed.parquet', processed);
-```
-
----
-
-## File Processing Patterns
-
-### Buffering (Small Files < 10MB)
-
-```typescript
-// Load entire file into memory
-const content = await dataSource.downloadFile(file.path);
-const records = await csvParser.parse(content as string);
-await sendToFluent(records);
-```
-
-### Chunked Processing (Large Files > 10MB)
-
-```typescript
-const content = await dataSource.downloadFile(file, { encoding: 'binary' });
-
-const chunkSize = 1024 * 1024; // 1MB chunks
-for (let offset = 0; offset < (content as Buffer).length; offset += chunkSize) {
-  const chunk = (content as Buffer).slice(offset, offset + chunkSize);
-  await processChunk(chunk);
-}
-```
-
-### Batch Processing
-
-```typescript
-const batchSize = 100;
-for (let i = 0; i < records.length; i += batchSize) {
-  const batch = records.slice(i, i + batchSize);
-  await client.sendBatch(jobId, { entities: batch });
-}
-```
-
----
-
-## State Management (Prevent Duplicates)
-
-```typescript
-import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-
-const stateService = new StateService(new VersoriKVAdapter(openKv()));
-
-for (const file of files) {
-  // Create unique state key
-  const stateKey = `processed:${file.path}:${file.lastModified.toISOString()}`;
-
-  // Check if already processed
-  if (await stateService.isFileProcessed(stateKey)) {
-    continue;
-  }
-
-  // Process file
-  const content = await dataSource.downloadFile(file.path);
-  await processContent(content);
-
-  // Mark as processed
-  await stateService.markFileProcessed(stateKey, {
-    fileName: file.name,
-    processedAt: new Date().toISOString(),
-  });
-}
-```
-
----
-
-## Format-Specific Processing
-
-### CSV
-
-```typescript
-import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
-
-const csvParser = new CSVParserService();
-const content = await dataSource.downloadFile(file);
-const records = await csvParser.parse(content as string, {
-  columns: true,
-  skip_empty_lines: true,
-  trim: true,
-});
-```
-
-### JSON
-
-```typescript
-const content = await dataSource.downloadFile(file);
-const data = JSON.parse(content);
-const records = Array.isArray(data) ? data : [data];
-```
-
-### JSON Lines (JSONL)
-
-```typescript
-const content = await dataSource.downloadFile(file);
-const lines = content.split('\n').filter(line => line.trim());
-const records = lines.map(line => JSON.parse(line));
-```
-
-### Parquet
-
-```typescript
-import { ParquetParserService } from '@fluentcommerce/fc-connect-sdk';
-
-const parquetParser = new ParquetParserService(logger);
-const buffer = await dataSource.downloadFile(file, { encoding: 'binary' });
-const records = await parquetParser.parse(buffer);
-```
-
----
-
-## Error Handling
-
-### Retry with Exponential Backoff
-
-```typescript
-async function processWithRetry<T>(fn: () => Promise<T>, maxRetries = 3): Promise<T> {
-  for (let attempt = 0; attempt <= maxRetries; attempt++) {
-    try {
-      return await fn();
-    } catch (error: any) {
-      // Don't retry client errors (4xx)
-      if (error.statusCode && error.statusCode >= 400 && error.statusCode < 500) {
-        throw error;
-      }
-
-      if (attempt < maxRetries) {
-        const delay = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
-        await sleep(delay);
-      } else {
-        throw error;
-      }
-    }
-  }
-}
-
-function sleep(ms: number): Promise<void> {
-  return new Promise(resolve => setTimeout(resolve, ms));
-}
-```
-
-### Partial Failure Handling
-
-```typescript
-const results = { successful: 0, failed: 0, errors: [] };
-
-for (let i = 0; i < records.length; i += batchSize) {
-  const batch = records.slice(i, i + batchSize);
-
-  try {
-    await client.sendBatch(jobId, { entities: batch });
-    results.successful += batch.length;
-  } catch (error) {
-    results.failed += batch.length;
-    results.errors.push({ batch: i / batchSize, error: error.message });
-    // Continue processing remaining batches
-  }
-}
-```
-
----
-
-## Monitoring
-
-### Structured Logging
-
-```typescript
-import {
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-// Versori Platform - Use native log
-const { log } = ctx;  // Native Versori log
-log.info('Processing file started', {
-  fileName: file.name,
-  fileSize: file.size,
-});
-
-// Standalone - Use function-based utilities
-const logger = toStructuredLogger(createConsoleLogger(), {
-  service: 'inventory-sync',
-  version: '1.0',
-  logLevel: 'info'
-});
-
-logger.info('Processing file started', {
-  fileName: file.name,
-  fileSize: file.size,
-});
-
-logger.error('Processing failed', error, {
-  fileName: file.name,
-  duration: Date.now() - startTime,
-});
-```
-
-### Health Checks
-
-```typescript
-async function healthCheck() {
-  const s3Valid = await s3Source.validateConnection();
-  const sftpValid = await sftpSource.validateConnection();
-
-  if (!s3Valid || !sftpValid) {
-    throw new Error('Data source connectivity check failed');
-  }
-}
-```
-
----
-
-## Common Patterns
-
-### Pattern 1: S3 Inventory Ingestion
-
-```typescript
-import {
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  StateService,
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function ingestInventory() {
-  const s3 = new S3DataSource(
-    {
-      /* config */
-    },
-    logger
-  );
-  const csvParser = new CSVParserService();
-  const mapper = new UniversalMapper(mappingConfig);
-  const stateService = new StateService(kvAdapter);
-
-  const files = await s3.listFiles({ prefix: 'inventory/' });
-
-  for (const file of files) {
-    const stateKey = `processed:${file.etag}`;
-    if (await stateService.isFileProcessed(stateKey)) continue;
-
-    const content = await s3.downloadFile(file.path);
-    const records = await csvParser.parse(content);
-    const result = await mapper.map(records);
-
-    if (result.success) {
-      await sendToFluent(result.data);
-      await stateService.markFileProcessed(stateKey, {
-        fileName: file.name,
-        recordCount: records.length,
-      });
-    }
-  }
-}
-```
-
-### Pattern 2: SFTP Vendor Sync
-
-```typescript
-import { SftpDataSource, CSVParserService, StateService } from '@fluentcommerce/fc-connect-sdk';
-
-async function syncVendorInventory() {
-  const sftp = new SftpDataSource(
-    {
-      /* config */
-    },
-    logger
-  );
-  const stateService = new StateService(kvAdapter);
-
-  const files = await sftp.listFiles({
-    remotePath: '/outbound/inventory',
-    filePattern: 'inventory_*.csv',
-  });
-
-  for (const file of files) {
-    const stateKey = `processed:${file.name}:${file.lastModified.toISOString()}`;
-    if (await stateService.isFileProcessed(stateKey)) continue;
-
-    const content = await sftp.downloadFile(file.path);
-    const records = await csvParser.parse(content);
-    await sendToFluent(records);
-
-    await stateService.markFileProcessed(stateKey);
-
-    // Move to processed folder
-    await sftp.moveFile(file.path, file.path.replace('/outbound/', '/processed/'));
-  }
-}
-```
-
-### Pattern 3: Dual AWS Accounts (Read + Write)
-
-```typescript
-const sourceS3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-    },
-  },
-  logger
-);
-
-const targetS3 = new S3DataSource(
-  {
-    type: 'S3_PARQUET',
-    s3Config: {
-      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
-      region: 'us-west-2',
-    },
-  },
-  logger
-);
-
-// Read from vendor, write to data lake
-const content = await sourceS3.downloadFile('inventory.csv');
-const processed = await processData(content);
-await targetS3.writeFile('s3://datalake/processed.parquet', processed);
-```
-
----
-
-## Configuration Reference
-
-### S3DataSource Config
-
-```typescript
-interface S3DataSourceConfig {
-  type: 'S3_CSV' | 'S3_JSON' | 'S3_XML' | 'S3_PARQUET';
-  s3Config: {
-    accessKeyId: string;
-    secretAccessKey: string;
-    region: string;
-    bucket: string; // Required
-    sessionToken?: string;
-    endpoint?: string; // For LocalStack
-    maxAttempts?: number; // Default: 3
-  };
-}
-```
-
-### SftpDataSource Config
-
-```typescript
-interface SftpDataSourceConfig {
-  type: 'SFTP_CSV' | 'SFTP_JSON' | 'SFTP_JSONL' | 'SFTP_PARQUET';
-  connectionId: string;
-  name?: string;
-  settings: {
-    host: string;
-    port?: number; // Default: 22
-    username: string;
-    password?: string;
-    privateKey?: string;
-    passphrase?: string;
-    remotePath?: string;
-    filePattern?: string;
-    maxConnections?: number; // Default: 5
-    // Note: connectionWaitTimeout is fixed at 30000ms (30 seconds), not user-configurable
-    connectionTimeout?: number; // Default: 30000ms
-    keepaliveInterval?: number; // Default: 5000ms
-    retry?: SftpRetryConfig; // Retry configuration
-    writeCreateDirectories?: boolean; // Default: true
-    writeOverwrite?: boolean; // Default: false
-    writePermissions?: string; // Default: '0644'
-  };
-}
-
-interface SftpRetryConfig {
-  maxAttempts?: number; // Default: 3
-  baseDelayMs?: number; // Default: 1000
-  backoffFactor?: number; // Default: 2 (exponential)
-  maxDelayMs?: number; // Default: 8000
-  jitter?: 'none' | 'full'; // Default: 'full'
-  reconnectOnFailure?: boolean; // Default: true
-  isRetryable?: (error: Error) => boolean; // Custom retry predicate
-}
-```
-
----
-
-## Built-in Features
-
-### S3 Automatic Retry
-
-**S3DataSource includes built-in retry logic** - no additional code needed!
-
-```typescript
-const s3Source = new S3DataSource({
-  type: 'S3_CSV',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: 'us-east-1',
-    bucket: 'my-bucket',
-    maxAttempts: 5  // Default: 3
-  }
-}, logger);
-
-// All operations retry automatically:
-await s3Source.downloadFile('file.csv');  // Retries on 429, 500, 502, 503, 504, network errors
-await s3Source.uploadFile('output.json', data);  // Retries automatically
-```
-
-**Retry Configuration:**
-- **Default attempts**: 3
-- **Retryable errors**: 429, 500, 502, 503, 504, network errors
-- **Backoff**: Exponential (500ms × 2^attempt, max 8s)
-- **Total wait time**: Up to 7.5s (with 5 attempts)
-
-### SFTP Enhanced Retry
-
-**SftpDataSource includes enhanced retry logic** - configurable with exponential backoff, jitter, and custom predicates!
-
-```typescript
-const sftpSource = new SftpDataSource({
-  type: 'SFTP_CSV',
-  connectionId: 'vendor-sftp',
-  settings: {
-    host: 'sftp.vendor.com',
-    username: 'integration',
-    privateKey: process.env.SFTP_PRIVATE_KEY,
-
-    // Enhanced retry configuration (optional)
-    retry: {
-      maxAttempts: 5,           // Default: 3
-      baseDelayMs: 1000,         // Default: 1000
-      backoffFactor: 2,          // Default: 2 (exponential)
-      maxDelayMs: 8000,          // Default: 8000
-      jitter: 'full',            // Default: 'full' (randomization)
-      reconnectOnFailure: true,  // Default: true
-
-      // Optional: Custom retry predicate
-      isRetryable: (error: Error) => {
-        // Return true to retry, false to fail immediately
-        return error.message.includes('ECONNRESET');
-      }
-    }
-  }
-}, logger);
-
-// All operations retry automatically with your configuration:
-await sftpSource.downloadFile('/data/file.csv');
-await sftpSource.uploadFile('/output/file.csv', data);
-```
-
-**Retry Configuration:**
-- **Default attempts**: 3
-- **Retryable errors**: Network errors (ECONNRESET, ETIMEDOUT, etc.), transient messages (connection lost, timeout, etc.)
-- **Non-retryable**: Authentication failures, file not found, configuration errors
-- **Backoff**: Exponential (baseDelayMs × 2^attempt, capped at maxDelayMs)
-- **Jitter**: Full jitter (randomization between 0 and calculated delay)
-- **Reconnection**: Automatic reconnection between retry attempts (if enabled)
-
-### SFTP Connection Pool Wait Queue
-
-**SftpDataSource includes connection wait queue** - handles burst concurrency automatically!
-
-```typescript
-// Scenario: 10 simultaneous operations
-const promises = files.map(file => sftpSource.downloadFile(file));
-await Promise.all(promises);
-// First 5: Get connections immediately
-// Remaining 5: Wait in queue (up to 30s timeout)
-// All complete successfully!
-```
-
-**Wait Queue Configuration:**
-- **Pool size**: 5 connections (fixed)
-- **Wait timeout**: 30 seconds (fixed)
-- **Queue**: FIFO (First-In-First-Out)
-- **Race protection**: Prevents over-allocation
-
-**Production Pattern:**
-
-```typescript
-// Use p-limit to stay below pool limit
-import pLimit from 'p-limit';
-
-const limit = pLimit(3); // Lower than pool max (5)
-const promises = files.map(file =>
-  limit(() => sftpSource.downloadFile(file))
-);
-await Promise.all(promises);
-// Never hits wait queue, optimal performance
-```
-
----
-
-## Troubleshooting
-
-| Issue                      | Solution                                        |
-| -------------------------- | ----------------------------------------------- |
-| S3 Access Denied (403)     | Check IAM policy, bucket policy, credentials    |
-| SFTP Connection Timeout    | Check firewall (port 22), verify host reachable |
-| File Already Exists (SFTP) | Use `overwrite: true` or check existence first  |
-| Presigned URL Expired      | Complete operations within 1 hour               |
-| Out of Memory              | Use chunked processing for large files          |
-
----
-
-## Performance Guidelines
-
-| File Size | Strategy           | Concurrency (S3) | Concurrency (SFTP) |
-| --------- | ------------------ | ---------------- | ------------------ |
-| < 10MB    | Buffer entire file | 10-20            | 1-3                |
-| 10-100MB  | Buffer or chunked  | 5-10             | 1-2                |
-| > 100MB   | Chunked processing | 3-5              | 1                  |
-
----
-
-## Best Practices
-
-1. ✅ Use SSH keys for SFTP (not passwords)
-2. ✅ Use dual AWS credentials for source/target separation
-3. ✅ Implement state management to prevent duplicates
-4. ✅ Use retry with exponential backoff for transient errors
-5. ✅ Use structured logging with rich context
-6. ✅ Validate files before processing (size, name pattern, age)
-7. ✅ Use appropriate batch sizes (100-250 records)
-8. ✅ Use chunked processing for large files (> 10MB)
-9. ✅ Set restrictive SFTP file permissions (0600) for sensitive data
-10. ✅ Monitor connection health with regular health checks
-
----
-
-## Additional Resources
-
-- [Module 1: Foundations](../auto-pagination/modules/auto-pagination-01-foundations.md) - S3 vs SFTP decision guide
-- [Module 2: S3 Operations](./modules/data-sources-02-s3-operations.md) - Complete S3DataSource reference
-- [Module 3: SFTP Operations](./modules/data-sources-03-sftp-operations.md) - Complete SftpDataSource reference
-- [Module 4: File Processing Patterns](./modules/data-sources-04-file-patterns.md) - Streaming, batching, validation
-- [Module 5: Advanced Topics](./modules/data-sources-05-advanced-topics.md) - Security, monitoring, testing
-
----
-
-[← Back to Main Guide](./data-sources-readme.md)
+# Data Sources - Quick Reference
+
+> One-page cheat sheet for S3DataSource and SftpDataSource
+
+## Quick Decision: S3 vs SFTP
+
+```
+Need to choose between S3 and SFTP?
+
+Do you control the infrastructure?
+    ├─ YES → Is this cloud-native?
+    │         ├─ YES → Use S3
+    │         └─ NO  → Is volume high (>10K/hour)?
+    │                   ├─ YES → Use S3
+    │                   └─ NO  → Either (your preference)
+    └─ NO  → Use SFTP (vendor requirement)
+```
+
+---
+
+## S3DataSource - Quick Start
+
+### Basic Configuration
+
+```typescript
+import {
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV', // S3_JSON, S3_XML, S3_PARQUET
+    connectionId: 's3-quick-ref',
+    name: 'S3 Quick Ref',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+      bucket: 'my-bucket', // Required
+    },
+  },
+  logger
+);
+```
+
+### Common Operations
+
+```typescript
+// List files
+const files = await s3Source.listFiles({ prefix: 'inventory/' });
+
+// Read file
+const content = await s3Source.downloadFile('file.csv');
+
+// Write file
+await s3Source.uploadFile('output.csv', csvData, {
+  contentType: 'text/csv',
+  metadata: { 'processed-date': new Date().toISOString() },
+});
+
+// Delete file
+await s3Source.deleteFile('old-file.csv');
+
+// Copy file
+await s3Source.copyFile('source.csv', 'archive/source.csv');
+
+// Move file (copy + delete)
+await s3Source.moveFile('incoming/file.csv', 'processed/file.csv');
+
+// Check existence
+// For S3 existence, list by prefix or handle 404 from download
+
+// Validate connection
+const isValid = await s3Source.validateConnection();
+```
+
+### Parquet Generation
+
+```typescript
+// Generate Parquet buffer
+const records = [
+  { id: 1, sku: 'PROD-001', quantity: 100, price: 29.99 },
+  { id: 2, sku: 'PROD-002', quantity: 200, price: 39.99 },
+];
+
+const parquetBuffer = await s3Source.writeParquetContent(records, {
+  compression: 'GZIP', // UNCOMPRESSED, GZIP, SNAPPY, BROTLI
+  rowGroupSize: 50000,
+});
+
+// Upload to S3
+await s3Source.writeFile('s3://bucket/data.parquet', parquetBuffer, {
+  contentType: 'application/octet-stream',
+});
+```
+
+---
+
+## SftpDataSource - Quick Start
+
+### Basic Configuration
+
+```typescript
+import {
+  SftpDataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV', // SFTP_JSON, SFTP_JSONL, SFTP_PARQUET
+    connectionId: 'vendor-sftp',
+    name: 'Vendor SFTP Connection',
+    settings: {
+      host: 'sftp.vendor.com',
+      port: 22,
+      username: 'integration',
+      privateKey: process.env.SFTP_PRIVATE_KEY, // or password
+      remotePath: '/data/inventory',
+      filePattern: '*.csv',
+    },
+  },
+  logger
+);
+```
+
+### Common Operations
+
+```typescript
+// List files
+const files = await sftpSource.listFiles({
+  remotePath: '/data/current',
+  filePattern: '*.csv',
+});
+
+// Read file
+const content = await sftpSource.downloadFile('/data/inventory.csv');
+
+// Write file
+await sftpSource.writeFile('/exports/output.csv', csvData, {
+  createDirectories: true,
+  overwrite: false,
+  permissions: '0644',
+});
+
+// Delete file
+await sftpSource.deleteFile('/temp/old-file.csv');
+
+// Copy file (download + upload)
+await sftpSource.copyFile(
+  '/source/file.csv',
+  '/backup/file.csv',
+  true // overwrite
+);
+
+// Move file (atomic rename)
+await sftpSource.moveFile(
+  '/incoming/file.csv',
+  '/processed/file.csv',
+  false // don't overwrite
+);
+
+// Create directory
+await sftpSource.createDirectory('/exports/daily', true, '0755');
+
+// Check existence
+const fileExists = await sftpSource.fileExists('/data/file.csv');
+const dirExists = await sftpSource.directoryExists('/data/exports');
+
+// Validate connection
+const isValid = await sftpSource.validateConnection();
+```
+
+---
+
+## Authentication
+
+### S3 (AWS Credentials)
+
+```bash
+# Environment variables
+AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
+AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
+AWS_REGION=us-east-1
+AWS_SESSION_TOKEN=optional-session-token  # For temporary credentials
+```
+
+### SFTP (SSH)
+
+**Standalone - Password Auth**:
+
+```typescript
+settings: {
+  host: 'sftp.vendor.com',
+  username: 'integration',
+  password: process.env.SFTP_PASSWORD
+}
+```
+
+**Standalone - Private Key Auth (Recommended)**:
+
+```typescript
+settings: {
+  host: 'sftp.vendor.com',
+  username: 'integration',
+  privateKey: process.env.SFTP_PRIVATE_KEY,
+  passphrase: process.env.SFTP_PASSPHRASE  // If key is encrypted
+}
+```
+
+```bash
+# Environment variable (PEM format, NOT file path)
+SFTP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
+MIIEpAIBAAKCAQEA...
+-----END RSA PRIVATE KEY-----"
+```
+
+---
+
+## SFTP Credentials (Versori Platform)
+
+**For secure credential access in Versori connectors**, use one of these methods:
+
+### Method 1: activation.connections (Recommended)
+
+```typescript
+// Already decoded - no Buffer needed!
+const allConns = ctx.activation.connections || [];
+const sftpConn = allConns.find(c => c.name === 'versori_ftp_server');
+const { username, password } = sftpConn.credentials[0].credential.data.basicAuth;
+```
+
+### Method 2: credentials().get()
+
+```typescript
+import { Buffer } from 'node:buffer';
+
+// Requires base64 decoding
+const sftpCred = await ctx.credentials().getAccessToken('SFTP');
+const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
+const [username, password] = rawBasicAuth.split(':');
+```
+
+**See:** [sftp-credential-access-security.md](./data-sources-sftp-credential-access-security.md) for complete guide with error handling, validation, and security best practices.
+
+---
+
+## Dual AWS Credentials (Source + Target)
+
+```typescript
+// Source S3 (Vendor - READ)
+const sourceS3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+      bucket: process.env.SOURCE_BUCKET,
+    },
+  },
+  logger
+);
+
+// Target S3 (Your Data Lake - WRITE)
+const targetS3 = new S3DataSource(
+  {
+    type: 'S3_PARQUET',
+    s3Config: {
+      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
+      region: 'us-west-2',
+      bucket: process.env.TARGET_BUCKET,
+    },
+  },
+  logger
+);
+
+// Read from vendor, write to your data lake
+const content = await sourceS3.downloadFile('inventory.csv');
+const processed = await processData(content);
+await targetS3.writeFile('s3://your-bucket/processed.parquet', processed);
+```
+
+---
+
+## File Processing Patterns
+
+### Buffering (Small Files < 10MB)
+
+```typescript
+// Load entire file into memory
+const content = await dataSource.downloadFile(file.path);
+const records = await csvParser.parse(content as string);
+await sendToFluent(records);
+```
+
+### Chunked Processing (Large Files > 10MB)
+
+```typescript
+const content = await dataSource.downloadFile(file, { encoding: 'binary' });
+
+const chunkSize = 1024 * 1024; // 1MB chunks
+for (let offset = 0; offset < (content as Buffer).length; offset += chunkSize) {
+  const chunk = (content as Buffer).slice(offset, offset + chunkSize);
+  await processChunk(chunk);
+}
+```
+
+### Batch Processing
+
+```typescript
+const batchSize = 100;
+for (let i = 0; i < records.length; i += batchSize) {
+  const batch = records.slice(i, i + batchSize);
+  await client.sendBatch(jobId, { entities: batch });
+}
+```
+
+---
+
+## State Management (Prevent Duplicates)
+
+```typescript
+import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+
+const stateService = new StateService(new VersoriKVAdapter(openKv()));
+
+for (const file of files) {
+  // Create unique state key
+  const stateKey = `processed:${file.path}:${file.lastModified.toISOString()}`;
+
+  // Check if already processed
+  if (await stateService.isFileProcessed(stateKey)) {
+    continue;
+  }
+
+  // Process file
+  const content = await dataSource.downloadFile(file.path);
+  await processContent(content);
+
+  // Mark as processed
+  await stateService.markFileProcessed(stateKey, {
+    fileName: file.name,
+    processedAt: new Date().toISOString(),
+  });
+}
+```
+
+---
+
+## Format-Specific Processing
+
+### CSV
+
+```typescript
+import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
+
+const csvParser = new CSVParserService();
+const content = await dataSource.downloadFile(file);
+const records = await csvParser.parse(content as string, {
+  columns: true,
+  skip_empty_lines: true,
+  trim: true,
+});
+```
+
+### JSON
+
+```typescript
+const content = await dataSource.downloadFile(file);
+const data = JSON.parse(content);
+const records = Array.isArray(data) ? data : [data];
+```
+
+### JSON Lines (JSONL)
+
+```typescript
+const content = await dataSource.downloadFile(file);
+const lines = content.split('\n').filter(line => line.trim());
+const records = lines.map(line => JSON.parse(line));
+```
+
+### Parquet
+
+```typescript
+import { ParquetParserService } from '@fluentcommerce/fc-connect-sdk';
+
+const parquetParser = new ParquetParserService(logger);
+const buffer = await dataSource.downloadFile(file, { encoding: 'binary' });
+const records = await parquetParser.parse(buffer);
+```
+
+---
+
+## Error Handling
+
+### Retry with Exponential Backoff
+
+```typescript
+async function processWithRetry<T>(fn: () => Promise<T>, maxRetries = 3): Promise<T> {
+  for (let attempt = 0; attempt <= maxRetries; attempt++) {
+    try {
+      return await fn();
+    } catch (error: any) {
+      // Don't retry client errors (4xx)
+      if (error.statusCode && error.statusCode >= 400 && error.statusCode < 500) {
+        throw error;
+      }
+
+      if (attempt < maxRetries) {
+        const delay = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
+        await sleep(delay);
+      } else {
+        throw error;
+      }
+    }
+  }
+}
+
+function sleep(ms: number): Promise<void> {
+  return new Promise(resolve => setTimeout(resolve, ms));
+}
+```
+
+### Partial Failure Handling
+
+```typescript
+const results = { successful: 0, failed: 0, errors: [] };
+
+for (let i = 0; i < records.length; i += batchSize) {
+  const batch = records.slice(i, i + batchSize);
+
+  try {
+    await client.sendBatch(jobId, { entities: batch });
+    results.successful += batch.length;
+  } catch (error) {
+    results.failed += batch.length;
+    results.errors.push({ batch: i / batchSize, error: error.message });
+    // Continue processing remaining batches
+  }
+}
+```
+
+---
+
+## Monitoring
+
+### Structured Logging
+
+```typescript
+import {
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+// Versori Platform - Use native log
+const { log } = ctx;  // Native Versori log
+log.info('Processing file started', {
+  fileName: file.name,
+  fileSize: file.size,
+});
+
+// Standalone - Use function-based utilities
+const logger = toStructuredLogger(createConsoleLogger(), {
+  service: 'inventory-sync',
+  version: '1.0',
+  logLevel: 'info'
+});
+
+logger.info('Processing file started', {
+  fileName: file.name,
+  fileSize: file.size,
+});
+
+logger.error('Processing failed', error, {
+  fileName: file.name,
+  duration: Date.now() - startTime,
+});
+```
+
+### Health Checks
+
+```typescript
+async function healthCheck() {
+  const s3Valid = await s3Source.validateConnection();
+  const sftpValid = await sftpSource.validateConnection();
+
+  if (!s3Valid || !sftpValid) {
+    throw new Error('Data source connectivity check failed');
+  }
+}
+```
+
+---
+
+## Common Patterns
+
+### Pattern 1: S3 Inventory Ingestion
+
+```typescript
+import {
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  StateService,
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function ingestInventory() {
+  const s3 = new S3DataSource(
+    {
+      /* config */
+    },
+    logger
+  );
+  const csvParser = new CSVParserService();
+  const mapper = new UniversalMapper(mappingConfig);
+  const stateService = new StateService(kvAdapter);
+
+  const files = await s3.listFiles({ prefix: 'inventory/' });
+
+  for (const file of files) {
+    const stateKey = `processed:${file.etag}`;
+    if (await stateService.isFileProcessed(stateKey)) continue;
+
+    const content = await s3.downloadFile(file.path);
+    const records = await csvParser.parse(content);
+    const result = await mapper.map(records);
+
+    if (result.success) {
+      await sendToFluent(result.data);
+      await stateService.markFileProcessed(stateKey, {
+        fileName: file.name,
+        recordCount: records.length,
+      });
+    }
+  }
+}
+```
+
+### Pattern 2: SFTP Vendor Sync
+
+```typescript
+import { SftpDataSource, CSVParserService, StateService } from '@fluentcommerce/fc-connect-sdk';
+
+async function syncVendorInventory() {
+  const sftp = new SftpDataSource(
+    {
+      /* config */
+    },
+    logger
+  );
+  const stateService = new StateService(kvAdapter);
+
+  const files = await sftp.listFiles({
+    remotePath: '/outbound/inventory',
+    filePattern: 'inventory_*.csv',
+  });
+
+  for (const file of files) {
+    const stateKey = `processed:${file.name}:${file.lastModified.toISOString()}`;
+    if (await stateService.isFileProcessed(stateKey)) continue;
+
+    const content = await sftp.downloadFile(file.path);
+    const records = await csvParser.parse(content);
+    await sendToFluent(records);
+
+    await stateService.markFileProcessed(stateKey);
+
+    // Move to processed folder
+    await sftp.moveFile(file.path, file.path.replace('/outbound/', '/processed/'));
+  }
+}
+```
+
+### Pattern 3: Dual AWS Accounts (Read + Write)
+
+```typescript
+const sourceS3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+    },
+  },
+  logger
+);
+
+const targetS3 = new S3DataSource(
+  {
+    type: 'S3_PARQUET',
+    s3Config: {
+      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
+      region: 'us-west-2',
+    },
+  },
+  logger
+);
+
+// Read from vendor, write to data lake
+const content = await sourceS3.downloadFile('inventory.csv');
+const processed = await processData(content);
+await targetS3.writeFile('s3://datalake/processed.parquet', processed);
+```
+
+---
+
+## Configuration Reference
+
+### S3DataSource Config
+
+```typescript
+interface S3DataSourceConfig {
+  type: 'S3_CSV' | 'S3_JSON' | 'S3_XML' | 'S3_PARQUET';
+  s3Config: {
+    accessKeyId: string;
+    secretAccessKey: string;
+    region: string;
+    bucket: string; // Required
+    sessionToken?: string;
+    endpoint?: string; // For LocalStack
+    maxAttempts?: number; // Default: 3
+  };
+}
+```
+
+### SftpDataSource Config
+
+```typescript
+interface SftpDataSourceConfig {
+  type: 'SFTP_CSV' | 'SFTP_JSON' | 'SFTP_JSONL' | 'SFTP_PARQUET';
+  connectionId: string;
+  name?: string;
+  settings: {
+    host: string;
+    port?: number; // Default: 22
+    username: string;
+    password?: string;
+    privateKey?: string;
+    passphrase?: string;
+    remotePath?: string;
+    filePattern?: string;
+    maxConnections?: number; // Default: 5
+    // Note: connectionWaitTimeout is fixed at 30000ms (30 seconds), not user-configurable
+    connectionTimeout?: number; // Default: 30000ms
+    keepaliveInterval?: number; // Default: 5000ms
+    retry?: SftpRetryConfig; // Retry configuration
+    writeCreateDirectories?: boolean; // Default: true
+    writeOverwrite?: boolean; // Default: false
+    writePermissions?: string; // Default: '0644'
+  };
+}
+
+interface SftpRetryConfig {
+  maxAttempts?: number; // Default: 3
+  baseDelayMs?: number; // Default: 1000
+  backoffFactor?: number; // Default: 2 (exponential)
+  maxDelayMs?: number; // Default: 8000
+  jitter?: 'none' | 'full'; // Default: 'full'
+  reconnectOnFailure?: boolean; // Default: true
+  isRetryable?: (error: Error) => boolean; // Custom retry predicate
+}
+```
+
+---
+
+## Built-in Features
+
+### S3 Automatic Retry
+
+**S3DataSource includes built-in retry logic** - no additional code needed!
+
+```typescript
+const s3Source = new S3DataSource({
+  type: 'S3_CSV',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: 'us-east-1',
+    bucket: 'my-bucket',
+    maxAttempts: 5  // Default: 3
+  }
+}, logger);
+
+// All operations retry automatically:
+await s3Source.downloadFile('file.csv');  // Retries on 429, 500, 502, 503, 504, network errors
+await s3Source.uploadFile('output.json', data);  // Retries automatically
+```
+
+**Retry Configuration:**
+- **Default attempts**: 3
+- **Retryable errors**: 429, 500, 502, 503, 504, network errors
+- **Backoff**: Exponential (500ms × 2^attempt, max 8s)
+- **Total wait time**: Up to 7.5s (with 5 attempts)
+
+### SFTP Enhanced Retry
+
+**SftpDataSource includes enhanced retry logic** - configurable with exponential backoff, jitter, and custom predicates!
+
+```typescript
+const sftpSource = new SftpDataSource({
+  type: 'SFTP_CSV',
+  connectionId: 'vendor-sftp',
+  settings: {
+    host: 'sftp.vendor.com',
+    username: 'integration',
+    privateKey: process.env.SFTP_PRIVATE_KEY,
+
+    // Enhanced retry configuration (optional)
+    retry: {
+      maxAttempts: 5,           // Default: 3
+      baseDelayMs: 1000,         // Default: 1000
+      backoffFactor: 2,          // Default: 2 (exponential)
+      maxDelayMs: 8000,          // Default: 8000
+      jitter: 'full',            // Default: 'full' (randomization)
+      reconnectOnFailure: true,  // Default: true
+
+      // Optional: Custom retry predicate
+      isRetryable: (error: Error) => {
+        // Return true to retry, false to fail immediately
+        return error.message.includes('ECONNRESET');
+      }
+    }
+  }
+}, logger);
+
+// All operations retry automatically with your configuration:
+await sftpSource.downloadFile('/data/file.csv');
+await sftpSource.uploadFile('/output/file.csv', data);
+```
+
+**Retry Configuration:**
+- **Default attempts**: 3
+- **Retryable errors**: Network errors (ECONNRESET, ETIMEDOUT, etc.), transient messages (connection lost, timeout, etc.)
+- **Non-retryable**: Authentication failures, file not found, configuration errors
+- **Backoff**: Exponential (baseDelayMs × 2^attempt, capped at maxDelayMs)
+- **Jitter**: Full jitter (randomization between 0 and calculated delay)
+- **Reconnection**: Automatic reconnection between retry attempts (if enabled)
+
+### SFTP Connection Pool Wait Queue
+
+**SftpDataSource includes connection wait queue** - handles burst concurrency automatically!
+
+```typescript
+// Scenario: 10 simultaneous operations
+const promises = files.map(file => sftpSource.downloadFile(file));
+await Promise.all(promises);
+// First 5: Get connections immediately
+// Remaining 5: Wait in queue (up to 30s timeout)
+// All complete successfully!
+```
+
+**Wait Queue Configuration:**
+- **Pool size**: 5 connections (fixed)
+- **Wait timeout**: 30 seconds (fixed)
+- **Queue**: FIFO (First-In-First-Out)
+- **Race protection**: Prevents over-allocation
+
+**Production Pattern:**
+
+```typescript
+// Use p-limit to stay below pool limit
+import pLimit from 'p-limit';
+
+const limit = pLimit(3); // Lower than pool max (5)
+const promises = files.map(file =>
+  limit(() => sftpSource.downloadFile(file))
+);
+await Promise.all(promises);
+// Never hits wait queue, optimal performance
+```
+
+---
+
+## Troubleshooting
+
+| Issue                      | Solution                                        |
+| -------------------------- | ----------------------------------------------- |
+| S3 Access Denied (403)     | Check IAM policy, bucket policy, credentials    |
+| SFTP Connection Timeout    | Check firewall (port 22), verify host reachable |
+| File Already Exists (SFTP) | Use `overwrite: true` or check existence first  |
+| Presigned URL Expired      | Complete operations within 1 hour               |
+| Out of Memory              | Use chunked processing for large files          |
+
+---
+
+## Performance Guidelines
+
+| File Size | Strategy           | Concurrency (S3) | Concurrency (SFTP) |
+| --------- | ------------------ | ---------------- | ------------------ |
+| < 10MB    | Buffer entire file | 10-20            | 1-3                |
+| 10-100MB  | Buffer or chunked  | 5-10             | 1-2                |
+| > 100MB   | Chunked processing | 3-5              | 1                  |
+
+---
+
+## Best Practices
+
+1. ✅ Use SSH keys for SFTP (not passwords)
+2. ✅ Use dual AWS credentials for source/target separation
+3. ✅ Implement state management to prevent duplicates
+4. ✅ Use retry with exponential backoff for transient errors
+5. ✅ Use structured logging with rich context
+6. ✅ Validate files before processing (size, name pattern, age)
+7. ✅ Use appropriate batch sizes (100-250 records)
+8. ✅ Use chunked processing for large files (> 10MB)
+9. ✅ Set restrictive SFTP file permissions (0600) for sensitive data
+10. ✅ Monitor connection health with regular health checks
+
+---
+
+## Additional Resources
+
+- [Module 1: Foundations](../auto-pagination/modules/auto-pagination-01-foundations.md) - S3 vs SFTP decision guide
+- [Module 2: S3 Operations](./modules/data-sources-02-s3-operations.md) - Complete S3DataSource reference
+- [Module 3: SFTP Operations](./modules/data-sources-03-sftp-operations.md) - Complete SftpDataSource reference
+- [Module 4: File Processing Patterns](./modules/data-sources-04-file-patterns.md) - Streaming, batching, validation
+- [Module 5: Advanced Topics](./modules/data-sources-05-advanced-topics.md) - Security, monitoring, testing
+
+---
+
+[← Back to Main Guide](./data-sources-readme.md)