@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-05-advanced-topics.md

@@ -1,813 +1,813 @@
-# Module 5: Advanced Topics
-
-Master advanced data source patterns - dual AWS credentials, security best practices, monitoring, testing strategies, and troubleshooting.
-
-## Table of Contents
-
-- [Dual AWS Credentials](#dual-aws-credentials)
-- [Security Best Practices](#security-best-practices)
-- [Monitoring and Observability](#monitoring-and-observability)
-- [Testing Strategies](#testing-strategies)
-- [Troubleshooting](#troubleshooting)
-- [Performance Tuning](#performance-tuning)
-- [Production Checklist](#production-checklist)
-
----
-
-## Dual AWS Credentials
-
-### Why Use Separate AWS Accounts?
-
-**Separation of Concerns**:
-
-```
-┌─────────────────────────────────────────────────────┐
-│ Vendor AWS Account (Source)                         │
-│ - Read-only access to vendor's S3 bucket            │
-│ - Limited IAM permissions (ListBucket, GetObject)   │
-│ - Vendor manages security and compliance            │
-└────────────────┬────────────────────────────────────┘
-                 │
-                 ▼ (Read inventory files)
-┌─────────────────────────────────────────────────────┐
-│ Your Application (Processing)                       │
-│ - Runs with vendor credentials to read              │
-│ - Runs with your credentials to write               │
-└────────────────┬────────────────────────────────────┘
-                 │
-                 ▼ (Write processed data)
-┌─────────────────────────────────────────────────────┐
-│ Your AWS Account (Target)                           │
-│ - Full write access to your data lake               │
-│ - Your IAM permissions (PutObject, lifecycle rules) │
-│ - You manage retention and compliance               │
-└─────────────────────────────────────────────────────┘
-```
-
-**Benefits**:
-
-- ✅ **Security**: Vendor credentials can't write to your buckets
-- ✅ **Compliance**: Clear data ownership and responsibility
-- ✅ **Cost control**: Separate billing for source vs target storage
-- ✅ **Flexibility**: Different retention policies per account
-
-### Implementation Pattern
-
-```typescript
-import {
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-
-// Source S3 (Vendor's AWS account - READ ONLY)
-const sourceS3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    connectionId: 's3-source-vendor',
-    name: 'S3 Source Vendor',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!, // Vendor credentials
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: process.env.AWS_REGION || 'us-east-1',
-      bucket: process.env.SOURCE_BUCKET, // Vendor's bucket
-    },
-  },
-  logger
-);
-
-// Target S3 (Your AWS account - WRITE)
-const targetS3 = new S3DataSource(
-  {
-    type: 'S3_PARQUET',
-    connectionId: 's3-target-own',
-    name: 'S3 Target Own',
-    s3Config: {
-      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!, // Your credentials
-      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
-      region: process.env.TARGET_AWS_REGION || 'us-west-2',
-      bucket: process.env.TARGET_BUCKET, // Your data lake
-    },
-  },
-  logger
-);
-
-// Usage: Read from vendor, write to your data lake
-async function processInventory() {
-  // Read from vendor's S3
-  const files = await sourceS3.listFiles({ prefix: 'inventory/' });
-
-  for (const file of files) {
-    // Download from vendor
-    const content = await sourceS3.downloadFile(file.path);
-
-    // Process data
-    const processed = await processData(content);
-
-    // Write to your data lake
-    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-    await targetS3.uploadFile(`s3://your-bucket/processed/${timestamp}/${file.name}`, processed);
-  }
-}
-```
-
-### Environment Variables for Dual Credentials
-
-```bash
-# .env file
-
-# Source AWS Account (Vendor)
-AWS_ACCESS_KEY_ID=AKIAVENDORKEY123
-AWS_SECRET_ACCESS_KEY=vendorSecretKey456
-AWS_REGION=us-east-1
-SOURCE_BUCKET=vendor-inventory-bucket
-
-# Target AWS Account (Your Data Lake)
-TARGET_AWS_ACCESS_KEY_ID=AKIAYOURKEY789
-TARGET_AWS_SECRET_ACCESS_KEY=yourSecretKey012
-TARGET_AWS_REGION=us-west-2
-TARGET_BUCKET=your-datalake-bucket
-```
-
-### IAM Policy Examples
-
-**Source (Vendor) - Read-Only Policy**:
-
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Effect": "Allow",
-      "Action": ["s3:ListBucket", "s3:GetObject", "s3:GetObjectVersion"],
-      "Resource": ["arn:aws:s3:::vendor-inventory-bucket", "arn:aws:s3:::vendor-inventory-bucket/*"]
-    }
-  ]
-}
-```
-
-**Target (Your) - Write + Lifecycle Policy**:
-
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Effect": "Allow",
-      "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
-      "Resource": ["arn:aws:s3:::your-datalake-bucket", "arn:aws:s3:::your-datalake-bucket/*"]
-    }
-  ]
-}
-```
-
----
-
-## Security Best Practices
-
-### SSH Key Management (SFTP)
-
-```typescript
-// 1. Use private key authentication (NOT passwords)
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor-sftp',
-    settings: {
-      host: 'sftp.vendor.com',
-      username: 'integration',
-      privateKey: process.env.SFTP_PRIVATE_KEY, // ✅ SSH key (recommended)
-      // password: process.env.SFTP_PASSWORD,    // ❌ Avoid passwords
-      remotePath: '/data/inventory',
-    },
-  },
-  logger
-);
-
-// 2. Use encrypted private keys with passphrase
-const secureConfig = {
-  privateKey: process.env.SFTP_PRIVATE_KEY, // Encrypted key
-  passphrase: process.env.SFTP_PASSPHRASE, // Key passphrase
-};
-
-// 3. Rotate keys regularly (every 90 days)
-// 4. Never commit keys to version control
-// 5. Use environment-specific keys (dev, staging, prod)
-```
-
-### Presigned URL Security (S3)
-
-```typescript
-// Presigned URL expiration (1 hour default)
-// URLs auto-expire after this period
-
-// Security checklist:
-// ✅ URLs expire after 1 hour (not customizable in current SDK)
-// ✅ Signature validates credentials at generation time
-// ✅ No credentials exposed in URL (only signature)
-// ❌ Anyone with URL can access during expiry window
-// ❌ URLs can be logged in proxies/browsers
-
-// For sensitive data, consider:
-// 1. Shorter processing windows (< 1 hour)
-// 2. Additional application-level encryption
-// 3. VPC endpoints (private S3 access)
-```
-
-### Credential Storage
-
-```bash
-# ✅ GOOD: Environment variables (never in code)
-AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
-AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
-
-# ✅ BETTER: Secret management service
-# - AWS Secrets Manager
-# - HashiCorp Vault
-# - Versori environment variables
-
-# ❌ NEVER: Hardcoded in source code
-const accessKey = "AKIAIOSFODNN7EXAMPLE"; // ❌ DON'T DO THIS
-```
-
-### File Permissions (SFTP)
-
-```typescript
-// Set restrictive file permissions
-await sftpSource.uploadFile('/exports/sensitive-data.csv', content, {
-  permissions: '0600', // Read/write owner only (rw-------)
-});
-
-// Permission codes (octal):
-// 0644 = rw-r--r-- (owner: rw, group: r, other: r) - default
-// 0600 = rw------- (owner: rw, group: -, other: -) - secure
-// 0755 = rwxr-xr-x (owner: rwx, group: rx, other: rx) - directories
-// 0700 = rwx------ (owner: rwx, group: -, other: -) - secure dirs
-```
-
----
-
-## Monitoring and Observability
-
-### Structured Logging
-
-```typescript
-import {
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = new LoggingService({
-  level: 'info', // debug, info, warn, error
-  context: { service: 'inventory-ingestion', version: '1.0' },
-});
-
-// File processing with rich context
-async function processFileWithLogging(dataSource: DataSource, file: FileMetadata) {
-  const startTime = Date.now();
-
-  logger.info('Processing file started', {
-    fileName: file.name,
-    fileSize: file.size,
-    lastModified: file.lastModified.toISOString(),
-  });
-
-  try {
-    const content = await dataSource.downloadFile(file.path);
-    const records = parseContent(content);
-
-    logger.info('File parsed successfully', {
-      fileName: file.name,
-      recordCount: records.length,
-      parseTime: Date.now() - startTime,
-    });
-
-    await sendToFluent(records);
-
-    const duration = Date.now() - startTime;
-    logger.info('File processed successfully', {
-      fileName: file.name,
-      recordCount: records.length,
-      duration,
-      throughput: (records.length / (duration / 1000)).toFixed(2) + ' records/sec',
-    });
-  } catch (error) {
-    logger.error('File processing failed', error as Error, {
-      fileName: file.name,
-      duration: Date.now() - startTime,
-    });
-    throw error;
-  }
-}
-```
-
-### Metrics Collection
-
-```typescript
-interface ProcessingMetrics {
-  filesProcessed: number;
-  filesFailed: number;
-  totalRecords: number;
-  totalDuration: number;
-  errors: Array<{ file: string; error: string }>;
-}
-
-async function processWithMetrics(
-  dataSource: DataSource,
-  files: FileMetadata[]
-): Promise<ProcessingMetrics> {
-  const metrics: ProcessingMetrics = {
-    filesProcessed: 0,
-    filesFailed: 0,
-    totalRecords: 0,
-    totalDuration: 0,
-    errors: [],
-  };
-
-  const startTime = Date.now();
-
-  for (const file of files) {
-    try {
-      const records = await processFile(dataSource, file.path);
-      metrics.filesProcessed++;
-      metrics.totalRecords += records.length;
-    } catch (error) {
-      metrics.filesFailed++;
-      metrics.errors.push({
-        file: file.name,
-        error: (error as Error).message,
-      });
-    }
-  }
-
-  metrics.totalDuration = Date.now() - startTime;
-
-  // Log summary metrics
-  logger.info('Processing complete', {
-    ...metrics,
-    averageRecordsPerFile: (metrics.totalRecords / metrics.filesProcessed).toFixed(2),
-    throughput: (metrics.totalRecords / (metrics.totalDuration / 1000)).toFixed(2) + ' records/sec',
-  });
-
-  return metrics;
-}
-```
-
-### Health Checks
-
-```typescript
-async function performHealthCheck(
-  s3Source?: S3DataSource,
-  sftpSource?: SftpDataSource
-): Promise<{ s3: boolean; sftp: boolean }> {
-  const health = { s3: false, sftp: false };
-
-  // Check S3 connectivity
-  if (s3Source) {
-    try {
-      health.s3 = await s3Source.validateConnection();
-      logger.info('S3 health check', { status: health.s3 ? 'OK' : 'FAIL' });
-    } catch (error) {
-      logger.error('S3 health check failed', error as Error);
-    }
-  }
-
-  // Check SFTP connectivity
-  if (sftpSource) {
-    try {
-      health.sftp = await sftpSource.validateConnection();
-      logger.info('SFTP health check', { status: health.sftp ? 'OK' : 'FAIL' });
-    } catch (error) {
-      logger.error('SFTP health check failed', error as Error);
-    }
-  }
-
-  return health;
-}
-
-// Use in startup or scheduled health checks
-async function startupHealthCheck() {
-  const health = await performHealthCheck(s3Source, sftpSource);
-
-  if (!health.s3 || !health.sftp) {
-    logger.error('Health check failed', { s3: health.s3, sftp: health.sftp });
-    throw new Error('Data source connectivity check failed');
-  }
-
-  logger.info('All health checks passed');
-}
-```
-
----
-
-## Testing Strategies
-
-### Unit Testing with Mocks
-
-```typescript
-// Mock data source for testing
-class MockS3DataSource {
-  private files = new Map<string, string>();
-
-  async listFiles() {
-    return Array.from(this.files.keys()).map(path => ({
-      name: path.split('/').pop()!,
-      path,
-      size: this.files.get(path)!.length,
-      lastModified: new Date(),
-      type: 'file' as const,
-      etag: 'mock-etag',
-    }));
-  }
-
-  async downloadFile(path: string) {
-    if (!this.files.has(path)) {
-      throw new Error(`File not found: ${path}`);
-    }
-    return this.files.get(path)!;
-  }
-
-  async writeFile(path: string, content: string) {
-    this.files.set(path, content);
-  }
-
-  // Add test files
-  addTestFile(path: string, content: string) {
-    this.files.set(path, content);
-  }
-}
-
-// Use in tests
-describe('Inventory Ingestion', () => {
-  let mockS3: MockS3DataSource;
-
-  beforeEach(() => {
-    mockS3 = new MockS3DataSource();
-    mockS3.addTestFile('inventory.csv', 'sku,quantity\nPROD-001,100');
-  });
-
-  it('should process CSV file', async () => {
-    const files = await mockS3.listFiles();
-    expect(files).toHaveLength(1);
-
-    const content = await mockS3.downloadFile('inventory.csv');
-    expect(content).toContain('PROD-001');
-  });
-});
-```
-
-### Integration Testing
-
-```typescript
-// Test with real S3 (LocalStack)
-describe('S3DataSource Integration', () => {
-  let s3Source: S3DataSource;
-
-  beforeAll(() => {
-    s3Source = new S3DataSource(
-      {
-        type: 'S3_CSV',
-        s3Config: {
-          accessKeyId: 'test',
-          secretAccessKey: 'test',
-          region: 'us-east-1',
-          endpoint: 'http://localhost:4566', // LocalStack
-          bucket: 'test-bucket',
-        },
-      },
-      logger
-    );
-  });
-
-  it('should upload and download file', async () => {
-    const testContent = 'sku,quantity\nTEST-001,50';
-
-    // Upload
-    await s3Source.uploadFile('s3://test-bucket/test.csv', testContent);
-
-    // Download
-    const downloaded = await s3Source.downloadFile('s3://test-bucket/test.csv');
-
-    expect(downloaded).toBe(testContent);
-  });
-});
-```
-
-### End-to-End Testing
-
-```typescript
-// E2E test: Upload to S3 → Process → Validate
-async function e2eTest() {
-  const testData = [
-    { sku: 'E2E-001', quantity: 100, location: 'DC-001' },
-    { sku: 'E2E-002', quantity: 200, location: 'DC-002' },
-  ];
-
-  // 1. Upload test file
-  const csvContent = testData.map(r => `${r.sku},${r.quantity},${r.location}`).join('\n');
-  await s3Source.uploadFile('s3://test-bucket/e2e-test.csv', 'sku,quantity,location\n' + csvContent);
-
-  // 2. Process file
-  const files = await s3Source.listFiles({ prefix: 'e2e-test' });
-  expect(files.length).toBe(1);
-
-  const content = await s3Source.downloadFile(files[0].path);
-  const records = await csvParser.parse(content as string);
-
-  // 3. Validate results
-  expect(records.length).toBe(2);
-  expect(records[0].sku).toBe('E2E-001');
-
-  // 4. Cleanup
-  await s3Source.deleteFile(files[0].path);
-}
-```
-
----
-
-## Troubleshooting
-
-### Common Issues and Solutions
-
-#### Issue 1: S3 Access Denied
-
-**Symptoms**:
-
-```
-Error: Access Denied (statusCode: 403)
-```
-
-**Diagnosis**:
-
-```typescript
-// Test S3 connectivity
-const isValid = await s3Source.validateConnection();
-console.log('S3 connection valid:', isValid);
-
-// Check bucket access
-const files = await s3Source.listFiles();
-// If this fails, check IAM permissions
-```
-
-**Solutions**:
-
-1. Verify IAM policy allows `s3:ListBucket`, `s3:GetObject`, `s3:PutObject`
-2. Check bucket policy allows your IAM user/role
-3. Verify credentials are correct (not expired)
-4. Ensure bucket exists in the specified region
-
-#### Issue 2: SFTP Connection Timeout
-
-**Symptoms**:
-
-```
-Error: Connection timeout after 30000ms
-```
-
-**Diagnosis**:
-
-```typescript
-// Test SFTP connectivity
-try {
-  const isValid = await sftpSource.validateConnection();
-  console.log('SFTP connection valid:', isValid);
-} catch (error) {
-  console.error('SFTP connection failed:', error.message);
-}
-```
-
-**Solutions**:
-
-1. Check firewall rules (allow outbound port 22)
-2. Verify SFTP host is reachable: `telnet sftp.vendor.com 22`
-3. Check SSH credentials (username + privateKey/password)
-4. Increase connection timeout in config
-5. Verify network connectivity to SFTP server
-
-#### Issue 3: File Already Exists (SFTP)
-
-**Symptoms**:
-
-```
-Error: File already exists (SFTP error code 11)
-```
-
-**Solutions**:
-
-```typescript
-// Option 1: Enable overwrite
-await sftpSource.writeFile(path, content, {
-  overwrite: true, // ✅ Allow overwriting
-});
-
-// Option 2: Check existence first
-if (await sftpSource.fileExists(path)) {
-  await sftpSource.deleteFile(path);
-}
-await sftpSource.writeFile(path, content);
-
-// Option 3: Use unique filenames
-const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-const uniquePath = `/exports/inventory-${timestamp}.csv`;
-await sftpSource.writeFile(uniquePath, content);
-```
-
-#### Issue 4: Presigned URL Expired
-
-**Symptoms**:
-
-```
-Error: Request has expired (statusCode: 403)
-```
-
-**Cause**: Presigned URLs expire after 1 hour (default)
-
-**Solutions**:
-
-1. Complete operations within 1 hour
-2. Regenerate URLs for long-running jobs
-3. Process files in smaller batches
-
-#### Issue 5: Out of Memory (Large Files)
-
-**Symptoms**:
-
-```
-Error: JavaScript heap out of memory
-```
-
-**Solutions**:
-
-```typescript
-// Option 1: Use chunked processing
-async function processLargeFile(dataSource: DataSource, file: string) {
-  const content = await dataSource.downloadFile(file, { encoding: 'binary' });
-
-  const chunkSize = 1024 * 1024; // 1MB chunks
-  for (let offset = 0; offset < (content as Buffer).length; offset += chunkSize) {
-    const chunk = (content as Buffer).slice(offset, offset + chunkSize);
-    await processChunk(chunk);
-  }
-}
-
-// Option 2: Increase Node.js heap size
-// node --max-old-space-size=4096 your-script.js (4GB heap)
-
-// Option 3: Split large files before processing
-```
-
-### Debugging Tips
-
-```typescript
-// Enable debug logging
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'debug'
-});
-
-// Log S3 presigned URLs
-logger.debug('Generated presigned URL', {
-  bucket: 'my-bucket',
-  key: 'file.csv',
-  operation: 'GET',
-  // URL is logged internally by S3DataSource
-});
-
-// Log SFTP connection details
-const connectionInfo = sftpSource.getConnectionInfo();
-logger.debug('SFTP connection info', connectionInfo);
-
-// Log file metadata
-const files = await dataSource.listFiles();
-for (const file of files) {
-  logger.debug('File metadata', {
-    name: file.name,
-    size: file.size,
-    lastModified: file.lastModified,
-  });
-}
-```
-
----
-
-## Performance Tuning
-
-### Optimization Strategies
-
-```typescript
-// 1. Parallel file processing (S3)
-const maxConcurrency = 10; // S3 scales well
-await Promise.all(files.slice(0, maxConcurrency).map(file => processFile(s3Source, file)));
-
-// 2. Connection pooling (SFTP)
-const sftpConfig = {
-  settings: {
-    maxConnections: 3, // Don't overwhelm vendor SFTP server
-    keepaliveInterval: 5000,
-  },
-};
-
-// 3. Batch size optimization
-const optimalBatchSize = Math.min(
-  Math.ceil(records.length / 10), // ~10 batches
-  500 // Max 500 per batch
-);
-
-// 4. Minimize data source round trips
-const files = await s3Source.listFiles({ prefix: 'inventory/' });
-// Get all file metadata in one call
-
-// 5. Use appropriate file formats
-// CSV: Human-readable, moderate size
-// Parquet: Columnar, best compression, fastest queries
-// JSON: Flexible schema, larger size
-```
-
-### Performance Benchmarks
-
-```
-File Size Benchmarks (approximate):
-┌───────────┬──────────────┬───────────────┬─────────────┐
-│ File Size │ Buffering    │ Chunked (1MB) │ Best Choice │
-├───────────┼──────────────┼───────────────┼─────────────┤
-│ < 1MB     │ 0.1s         │ 0.3s          │ Buffering   │
-│ 10MB      │ 0.8s         │ 1.2s          │ Buffering   │
-│ 50MB      │ 3.5s         │ 4.0s          │ Buffering   │
-│ 100MB     │ 7.0s         │ 7.5s          │ Either      │
-│ 500MB     │ OOM risk     │ 35s           │ Chunked     │
-│ 1GB+      │ Not possible │ 70s+          │ Chunked     │
-└───────────┴──────────────┴───────────────┴─────────────┘
-```
-
----
-
-## Production Checklist
-
-### Pre-Deployment
-
-- [ ] **Security**
-  - [ ] SSH keys stored in secure secret manager
-  - [ ] AWS credentials use least-privilege IAM policies
-  - [ ] No hardcoded credentials in source code
-  - [ ] SFTP file permissions set to 0600 for sensitive data
-
-- [ ] **Configuration**
-  - [ ] Environment variables documented
-  - [ ] Dual AWS credentials configured (source + target)
-  - [ ] Connection timeouts tuned for network conditions
-  - [ ] Batch sizes optimized for data volume
-
-- [ ] **Monitoring**
-  - [ ] Structured logging enabled
-  - [ ] Health checks implemented
-  - [ ] Metrics collection configured
-  - [ ] Alerting set up for failures
-
-- [ ] **Error Handling**
-  - [ ] Retry logic with exponential backoff
-  - [ ] Partial failure handling for batches
-  - [ ] Failed records logged for manual review
-  - [ ] Circuit breaker pattern for repeated failures
-
-- [ ] **State Management**
-  - [ ] State service configured (prevent duplicates)
-  - [ ] State keys use file metadata (etag or lastModified)
-  - [ ] State cleanup strategy defined
-
-- [ ] **Testing**
-  - [ ] Unit tests pass
-  - [ ] Integration tests with real services (or LocalStack)
-  - [ ] End-to-end test validates full pipeline
-  - [ ] Load testing completed for production volume
-
-### Post-Deployment
-
-- [ ] **Monitoring**
-  - [ ] Logs are being collected
-  - [ ] Metrics dashboards created
-  - [ ] Alerts are triggering correctly
-
-- [ ] **Performance**
-  - [ ] Processing throughput meets SLA
-  - [ ] Memory usage is stable
-  - [ ] Connection pool size is optimal
-
-- [ ] **Validation**
-  - [ ] Data quality checks passing
-  - [ ] No duplicate files processed
-  - [ ] Record counts match expected
-
----
-
-## Next Steps
-
-**Quick reference** → [Quick Reference](../../advanced-services/advanced-services-quick-reference.md)
-
-**Complete examples** → [Examples Directory](../examples/)
-
----
-
-[← Back to File Processing Patterns](./data-sources-04-file-patterns.md) | [Quick Reference →](../data-sources-quick-reference.md) | [↑ Back to Guide](../data-sources-readme.md)
+# Module 5: Advanced Topics
+
+Master advanced data source patterns - dual AWS credentials, security best practices, monitoring, testing strategies, and troubleshooting.
+
+## Table of Contents
+
+- [Dual AWS Credentials](#dual-aws-credentials)
+- [Security Best Practices](#security-best-practices)
+- [Monitoring and Observability](#monitoring-and-observability)
+- [Testing Strategies](#testing-strategies)
+- [Troubleshooting](#troubleshooting)
+- [Performance Tuning](#performance-tuning)
+- [Production Checklist](#production-checklist)
+
+---
+
+## Dual AWS Credentials
+
+### Why Use Separate AWS Accounts?
+
+**Separation of Concerns**:
+
+```
+┌─────────────────────────────────────────────────────┐
+│ Vendor AWS Account (Source)                         │
+│ - Read-only access to vendor's S3 bucket            │
+│ - Limited IAM permissions (ListBucket, GetObject)   │
+│ - Vendor manages security and compliance            │
+└────────────────┬────────────────────────────────────┘
+                 │
+                 ▼ (Read inventory files)
+┌─────────────────────────────────────────────────────┐
+│ Your Application (Processing)                       │
+│ - Runs with vendor credentials to read              │
+│ - Runs with your credentials to write               │
+└────────────────┬────────────────────────────────────┘
+                 │
+                 ▼ (Write processed data)
+┌─────────────────────────────────────────────────────┐
+│ Your AWS Account (Target)                           │
+│ - Full write access to your data lake               │
+│ - Your IAM permissions (PutObject, lifecycle rules) │
+│ - You manage retention and compliance               │
+└─────────────────────────────────────────────────────┘
+```
+
+**Benefits**:
+
+- ✅ **Security**: Vendor credentials can't write to your buckets
+- ✅ **Compliance**: Clear data ownership and responsibility
+- ✅ **Cost control**: Separate billing for source vs target storage
+- ✅ **Flexibility**: Different retention policies per account
+
+### Implementation Pattern
+
+```typescript
+import {
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+
+// Source S3 (Vendor's AWS account - READ ONLY)
+const sourceS3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    connectionId: 's3-source-vendor',
+    name: 'S3 Source Vendor',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!, // Vendor credentials
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: process.env.AWS_REGION || 'us-east-1',
+      bucket: process.env.SOURCE_BUCKET, // Vendor's bucket
+    },
+  },
+  logger
+);
+
+// Target S3 (Your AWS account - WRITE)
+const targetS3 = new S3DataSource(
+  {
+    type: 'S3_PARQUET',
+    connectionId: 's3-target-own',
+    name: 'S3 Target Own',
+    s3Config: {
+      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!, // Your credentials
+      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
+      region: process.env.TARGET_AWS_REGION || 'us-west-2',
+      bucket: process.env.TARGET_BUCKET, // Your data lake
+    },
+  },
+  logger
+);
+
+// Usage: Read from vendor, write to your data lake
+async function processInventory() {
+  // Read from vendor's S3
+  const files = await sourceS3.listFiles({ prefix: 'inventory/' });
+
+  for (const file of files) {
+    // Download from vendor
+    const content = await sourceS3.downloadFile(file.path);
+
+    // Process data
+    const processed = await processData(content);
+
+    // Write to your data lake
+    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+    await targetS3.uploadFile(`s3://your-bucket/processed/${timestamp}/${file.name}`, processed);
+  }
+}
+```
+
+### Environment Variables for Dual Credentials
+
+```bash
+# .env file
+
+# Source AWS Account (Vendor)
+AWS_ACCESS_KEY_ID=AKIAVENDORKEY123
+AWS_SECRET_ACCESS_KEY=vendorSecretKey456
+AWS_REGION=us-east-1
+SOURCE_BUCKET=vendor-inventory-bucket
+
+# Target AWS Account (Your Data Lake)
+TARGET_AWS_ACCESS_KEY_ID=AKIAYOURKEY789
+TARGET_AWS_SECRET_ACCESS_KEY=yourSecretKey012
+TARGET_AWS_REGION=us-west-2
+TARGET_BUCKET=your-datalake-bucket
+```
+
+### IAM Policy Examples
+
+**Source (Vendor) - Read-Only Policy**:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["s3:ListBucket", "s3:GetObject", "s3:GetObjectVersion"],
+      "Resource": ["arn:aws:s3:::vendor-inventory-bucket", "arn:aws:s3:::vendor-inventory-bucket/*"]
+    }
+  ]
+}
+```
+
+**Target (Your) - Write + Lifecycle Policy**:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
+      "Resource": ["arn:aws:s3:::your-datalake-bucket", "arn:aws:s3:::your-datalake-bucket/*"]
+    }
+  ]
+}
+```
+
+---
+
+## Security Best Practices
+
+### SSH Key Management (SFTP)
+
+```typescript
+// 1. Use private key authentication (NOT passwords)
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor-sftp',
+    settings: {
+      host: 'sftp.vendor.com',
+      username: 'integration',
+      privateKey: process.env.SFTP_PRIVATE_KEY, // ✅ SSH key (recommended)
+      // password: process.env.SFTP_PASSWORD,    // ❌ Avoid passwords
+      remotePath: '/data/inventory',
+    },
+  },
+  logger
+);
+
+// 2. Use encrypted private keys with passphrase
+const secureConfig = {
+  privateKey: process.env.SFTP_PRIVATE_KEY, // Encrypted key
+  passphrase: process.env.SFTP_PASSPHRASE, // Key passphrase
+};
+
+// 3. Rotate keys regularly (every 90 days)
+// 4. Never commit keys to version control
+// 5. Use environment-specific keys (dev, staging, prod)
+```
+
+### Presigned URL Security (S3)
+
+```typescript
+// Presigned URL expiration (1 hour default)
+// URLs auto-expire after this period
+
+// Security checklist:
+// ✅ URLs expire after 1 hour (not customizable in current SDK)
+// ✅ Signature validates credentials at generation time
+// ✅ No credentials exposed in URL (only signature)
+// ❌ Anyone with URL can access during expiry window
+// ❌ URLs can be logged in proxies/browsers
+
+// For sensitive data, consider:
+// 1. Shorter processing windows (< 1 hour)
+// 2. Additional application-level encryption
+// 3. VPC endpoints (private S3 access)
+```
+
+### Credential Storage
+
+```bash
+# ✅ GOOD: Environment variables (never in code)
+AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
+AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
+
+# ✅ BETTER: Secret management service
+# - AWS Secrets Manager
+# - HashiCorp Vault
+# - Versori environment variables
+
+# ❌ NEVER: Hardcoded in source code
+const accessKey = "AKIAIOSFODNN7EXAMPLE"; // ❌ DON'T DO THIS
+```
+
+### File Permissions (SFTP)
+
+```typescript
+// Set restrictive file permissions
+await sftpSource.uploadFile('/exports/sensitive-data.csv', content, {
+  permissions: '0600', // Read/write owner only (rw-------)
+});
+
+// Permission codes (octal):
+// 0644 = rw-r--r-- (owner: rw, group: r, other: r) - default
+// 0600 = rw------- (owner: rw, group: -, other: -) - secure
+// 0755 = rwxr-xr-x (owner: rwx, group: rx, other: rx) - directories
+// 0700 = rwx------ (owner: rwx, group: -, other: -) - secure dirs
+```
+
+---
+
+## Monitoring and Observability
+
+### Structured Logging
+
+```typescript
+import {
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = new LoggingService({
+  level: 'info', // debug, info, warn, error
+  context: { service: 'inventory-ingestion', version: '1.0' },
+});
+
+// File processing with rich context
+async function processFileWithLogging(dataSource: DataSource, file: FileMetadata) {
+  const startTime = Date.now();
+
+  logger.info('Processing file started', {
+    fileName: file.name,
+    fileSize: file.size,
+    lastModified: file.lastModified.toISOString(),
+  });
+
+  try {
+    const content = await dataSource.downloadFile(file.path);
+    const records = parseContent(content);
+
+    logger.info('File parsed successfully', {
+      fileName: file.name,
+      recordCount: records.length,
+      parseTime: Date.now() - startTime,
+    });
+
+    await sendToFluent(records);
+
+    const duration = Date.now() - startTime;
+    logger.info('File processed successfully', {
+      fileName: file.name,
+      recordCount: records.length,
+      duration,
+      throughput: (records.length / (duration / 1000)).toFixed(2) + ' records/sec',
+    });
+  } catch (error) {
+    logger.error('File processing failed', error as Error, {
+      fileName: file.name,
+      duration: Date.now() - startTime,
+    });
+    throw error;
+  }
+}
+```
+
+### Metrics Collection
+
+```typescript
+interface ProcessingMetrics {
+  filesProcessed: number;
+  filesFailed: number;
+  totalRecords: number;
+  totalDuration: number;
+  errors: Array<{ file: string; error: string }>;
+}
+
+async function processWithMetrics(
+  dataSource: DataSource,
+  files: FileMetadata[]
+): Promise<ProcessingMetrics> {
+  const metrics: ProcessingMetrics = {
+    filesProcessed: 0,
+    filesFailed: 0,
+    totalRecords: 0,
+    totalDuration: 0,
+    errors: [],
+  };
+
+  const startTime = Date.now();
+
+  for (const file of files) {
+    try {
+      const records = await processFile(dataSource, file.path);
+      metrics.filesProcessed++;
+      metrics.totalRecords += records.length;
+    } catch (error) {
+      metrics.filesFailed++;
+      metrics.errors.push({
+        file: file.name,
+        error: (error as Error).message,
+      });
+    }
+  }
+
+  metrics.totalDuration = Date.now() - startTime;
+
+  // Log summary metrics
+  logger.info('Processing complete', {
+    ...metrics,
+    averageRecordsPerFile: (metrics.totalRecords / metrics.filesProcessed).toFixed(2),
+    throughput: (metrics.totalRecords / (metrics.totalDuration / 1000)).toFixed(2) + ' records/sec',
+  });
+
+  return metrics;
+}
+```
+
+### Health Checks
+
+```typescript
+async function performHealthCheck(
+  s3Source?: S3DataSource,
+  sftpSource?: SftpDataSource
+): Promise<{ s3: boolean; sftp: boolean }> {
+  const health = { s3: false, sftp: false };
+
+  // Check S3 connectivity
+  if (s3Source) {
+    try {
+      health.s3 = await s3Source.validateConnection();
+      logger.info('S3 health check', { status: health.s3 ? 'OK' : 'FAIL' });
+    } catch (error) {
+      logger.error('S3 health check failed', error as Error);
+    }
+  }
+
+  // Check SFTP connectivity
+  if (sftpSource) {
+    try {
+      health.sftp = await sftpSource.validateConnection();
+      logger.info('SFTP health check', { status: health.sftp ? 'OK' : 'FAIL' });
+    } catch (error) {
+      logger.error('SFTP health check failed', error as Error);
+    }
+  }
+
+  return health;
+}
+
+// Use in startup or scheduled health checks
+async function startupHealthCheck() {
+  const health = await performHealthCheck(s3Source, sftpSource);
+
+  if (!health.s3 || !health.sftp) {
+    logger.error('Health check failed', { s3: health.s3, sftp: health.sftp });
+    throw new Error('Data source connectivity check failed');
+  }
+
+  logger.info('All health checks passed');
+}
+```
+
+---
+
+## Testing Strategies
+
+### Unit Testing with Mocks
+
+```typescript
+// Mock data source for testing
+class MockS3DataSource {
+  private files = new Map<string, string>();
+
+  async listFiles() {
+    return Array.from(this.files.keys()).map(path => ({
+      name: path.split('/').pop()!,
+      path,
+      size: this.files.get(path)!.length,
+      lastModified: new Date(),
+      type: 'file' as const,
+      etag: 'mock-etag',
+    }));
+  }
+
+  async downloadFile(path: string) {
+    if (!this.files.has(path)) {
+      throw new Error(`File not found: ${path}`);
+    }
+    return this.files.get(path)!;
+  }
+
+  async writeFile(path: string, content: string) {
+    this.files.set(path, content);
+  }
+
+  // Add test files
+  addTestFile(path: string, content: string) {
+    this.files.set(path, content);
+  }
+}
+
+// Use in tests
+describe('Inventory Ingestion', () => {
+  let mockS3: MockS3DataSource;
+
+  beforeEach(() => {
+    mockS3 = new MockS3DataSource();
+    mockS3.addTestFile('inventory.csv', 'sku,quantity\nPROD-001,100');
+  });
+
+  it('should process CSV file', async () => {
+    const files = await mockS3.listFiles();
+    expect(files).toHaveLength(1);
+
+    const content = await mockS3.downloadFile('inventory.csv');
+    expect(content).toContain('PROD-001');
+  });
+});
+```
+
+### Integration Testing
+
+```typescript
+// Test with real S3 (LocalStack)
+describe('S3DataSource Integration', () => {
+  let s3Source: S3DataSource;
+
+  beforeAll(() => {
+    s3Source = new S3DataSource(
+      {
+        type: 'S3_CSV',
+        s3Config: {
+          accessKeyId: 'test',
+          secretAccessKey: 'test',
+          region: 'us-east-1',
+          endpoint: 'http://localhost:4566', // LocalStack
+          bucket: 'test-bucket',
+        },
+      },
+      logger
+    );
+  });
+
+  it('should upload and download file', async () => {
+    const testContent = 'sku,quantity\nTEST-001,50';
+
+    // Upload
+    await s3Source.uploadFile('s3://test-bucket/test.csv', testContent);
+
+    // Download
+    const downloaded = await s3Source.downloadFile('s3://test-bucket/test.csv');
+
+    expect(downloaded).toBe(testContent);
+  });
+});
+```
+
+### End-to-End Testing
+
+```typescript
+// E2E test: Upload to S3 → Process → Validate
+async function e2eTest() {
+  const testData = [
+    { sku: 'E2E-001', quantity: 100, location: 'DC-001' },
+    { sku: 'E2E-002', quantity: 200, location: 'DC-002' },
+  ];
+
+  // 1. Upload test file
+  const csvContent = testData.map(r => `${r.sku},${r.quantity},${r.location}`).join('\n');
+  await s3Source.uploadFile('s3://test-bucket/e2e-test.csv', 'sku,quantity,location\n' + csvContent);
+
+  // 2. Process file
+  const files = await s3Source.listFiles({ prefix: 'e2e-test' });
+  expect(files.length).toBe(1);
+
+  const content = await s3Source.downloadFile(files[0].path);
+  const records = await csvParser.parse(content as string);
+
+  // 3. Validate results
+  expect(records.length).toBe(2);
+  expect(records[0].sku).toBe('E2E-001');
+
+  // 4. Cleanup
+  await s3Source.deleteFile(files[0].path);
+}
+```
+
+---
+
+## Troubleshooting
+
+### Common Issues and Solutions
+
+#### Issue 1: S3 Access Denied
+
+**Symptoms**:
+
+```
+Error: Access Denied (statusCode: 403)
+```
+
+**Diagnosis**:
+
+```typescript
+// Test S3 connectivity
+const isValid = await s3Source.validateConnection();
+console.log('S3 connection valid:', isValid);
+
+// Check bucket access
+const files = await s3Source.listFiles();
+// If this fails, check IAM permissions
+```
+
+**Solutions**:
+
+1. Verify IAM policy allows `s3:ListBucket`, `s3:GetObject`, `s3:PutObject`
+2. Check bucket policy allows your IAM user/role
+3. Verify credentials are correct (not expired)
+4. Ensure bucket exists in the specified region
+
+#### Issue 2: SFTP Connection Timeout
+
+**Symptoms**:
+
+```
+Error: Connection timeout after 30000ms
+```
+
+**Diagnosis**:
+
+```typescript
+// Test SFTP connectivity
+try {
+  const isValid = await sftpSource.validateConnection();
+  console.log('SFTP connection valid:', isValid);
+} catch (error) {
+  console.error('SFTP connection failed:', error.message);
+}
+```
+
+**Solutions**:
+
+1. Check firewall rules (allow outbound port 22)
+2. Verify SFTP host is reachable: `telnet sftp.vendor.com 22`
+3. Check SSH credentials (username + privateKey/password)
+4. Increase connection timeout in config
+5. Verify network connectivity to SFTP server
+
+#### Issue 3: File Already Exists (SFTP)
+
+**Symptoms**:
+
+```
+Error: File already exists (SFTP error code 11)
+```
+
+**Solutions**:
+
+```typescript
+// Option 1: Enable overwrite
+await sftpSource.writeFile(path, content, {
+  overwrite: true, // ✅ Allow overwriting
+});
+
+// Option 2: Check existence first
+if (await sftpSource.fileExists(path)) {
+  await sftpSource.deleteFile(path);
+}
+await sftpSource.writeFile(path, content);
+
+// Option 3: Use unique filenames
+const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+const uniquePath = `/exports/inventory-${timestamp}.csv`;
+await sftpSource.writeFile(uniquePath, content);
+```
+
+#### Issue 4: Presigned URL Expired
+
+**Symptoms**:
+
+```
+Error: Request has expired (statusCode: 403)
+```
+
+**Cause**: Presigned URLs expire after 1 hour (default)
+
+**Solutions**:
+
+1. Complete operations within 1 hour
+2. Regenerate URLs for long-running jobs
+3. Process files in smaller batches
+
+#### Issue 5: Out of Memory (Large Files)
+
+**Symptoms**:
+
+```
+Error: JavaScript heap out of memory
+```
+
+**Solutions**:
+
+```typescript
+// Option 1: Use chunked processing
+async function processLargeFile(dataSource: DataSource, file: string) {
+  const content = await dataSource.downloadFile(file, { encoding: 'binary' });
+
+  const chunkSize = 1024 * 1024; // 1MB chunks
+  for (let offset = 0; offset < (content as Buffer).length; offset += chunkSize) {
+    const chunk = (content as Buffer).slice(offset, offset + chunkSize);
+    await processChunk(chunk);
+  }
+}
+
+// Option 2: Increase Node.js heap size
+// node --max-old-space-size=4096 your-script.js (4GB heap)
+
+// Option 3: Split large files before processing
+```
+
+### Debugging Tips
+
+```typescript
+// Enable debug logging
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'debug'
+});
+
+// Log S3 presigned URLs
+logger.debug('Generated presigned URL', {
+  bucket: 'my-bucket',
+  key: 'file.csv',
+  operation: 'GET',
+  // URL is logged internally by S3DataSource
+});
+
+// Log SFTP connection details
+const connectionInfo = sftpSource.getConnectionInfo();
+logger.debug('SFTP connection info', connectionInfo);
+
+// Log file metadata
+const files = await dataSource.listFiles();
+for (const file of files) {
+  logger.debug('File metadata', {
+    name: file.name,
+    size: file.size,
+    lastModified: file.lastModified,
+  });
+}
+```
+
+---
+
+## Performance Tuning
+
+### Optimization Strategies
+
+```typescript
+// 1. Parallel file processing (S3)
+const maxConcurrency = 10; // S3 scales well
+await Promise.all(files.slice(0, maxConcurrency).map(file => processFile(s3Source, file)));
+
+// 2. Connection pooling (SFTP)
+const sftpConfig = {
+  settings: {
+    maxConnections: 3, // Don't overwhelm vendor SFTP server
+    keepaliveInterval: 5000,
+  },
+};
+
+// 3. Batch size optimization
+const optimalBatchSize = Math.min(
+  Math.ceil(records.length / 10), // ~10 batches
+  500 // Max 500 per batch
+);
+
+// 4. Minimize data source round trips
+const files = await s3Source.listFiles({ prefix: 'inventory/' });
+// Get all file metadata in one call
+
+// 5. Use appropriate file formats
+// CSV: Human-readable, moderate size
+// Parquet: Columnar, best compression, fastest queries
+// JSON: Flexible schema, larger size
+```
+
+### Performance Benchmarks
+
+```
+File Size Benchmarks (approximate):
+┌───────────┬──────────────┬───────────────┬─────────────┐
+│ File Size │ Buffering    │ Chunked (1MB) │ Best Choice │
+├───────────┼──────────────┼───────────────┼─────────────┤
+│ < 1MB     │ 0.1s         │ 0.3s          │ Buffering   │
+│ 10MB      │ 0.8s         │ 1.2s          │ Buffering   │
+│ 50MB      │ 3.5s         │ 4.0s          │ Buffering   │
+│ 100MB     │ 7.0s         │ 7.5s          │ Either      │
+│ 500MB     │ OOM risk     │ 35s           │ Chunked     │
+│ 1GB+      │ Not possible │ 70s+          │ Chunked     │
+└───────────┴──────────────┴───────────────┴─────────────┘
+```
+
+---
+
+## Production Checklist
+
+### Pre-Deployment
+
+- [ ] **Security**
+  - [ ] SSH keys stored in secure secret manager
+  - [ ] AWS credentials use least-privilege IAM policies
+  - [ ] No hardcoded credentials in source code
+  - [ ] SFTP file permissions set to 0600 for sensitive data
+
+- [ ] **Configuration**
+  - [ ] Environment variables documented
+  - [ ] Dual AWS credentials configured (source + target)
+  - [ ] Connection timeouts tuned for network conditions
+  - [ ] Batch sizes optimized for data volume
+
+- [ ] **Monitoring**
+  - [ ] Structured logging enabled
+  - [ ] Health checks implemented
+  - [ ] Metrics collection configured
+  - [ ] Alerting set up for failures
+
+- [ ] **Error Handling**
+  - [ ] Retry logic with exponential backoff
+  - [ ] Partial failure handling for batches
+  - [ ] Failed records logged for manual review
+  - [ ] Circuit breaker pattern for repeated failures
+
+- [ ] **State Management**
+  - [ ] State service configured (prevent duplicates)
+  - [ ] State keys use file metadata (etag or lastModified)
+  - [ ] State cleanup strategy defined
+
+- [ ] **Testing**
+  - [ ] Unit tests pass
+  - [ ] Integration tests with real services (or LocalStack)
+  - [ ] End-to-end test validates full pipeline
+  - [ ] Load testing completed for production volume
+
+### Post-Deployment
+
+- [ ] **Monitoring**
+  - [ ] Logs are being collected
+  - [ ] Metrics dashboards created
+  - [ ] Alerts are triggering correctly
+
+- [ ] **Performance**
+  - [ ] Processing throughput meets SLA
+  - [ ] Memory usage is stable
+  - [ ] Connection pool size is optimal
+
+- [ ] **Validation**
+  - [ ] Data quality checks passing
+  - [ ] No duplicate files processed
+  - [ ] Record counts match expected
+
+---
+
+## Next Steps
+
+**Quick reference** → [Quick Reference](../../advanced-services/advanced-services-quick-reference.md)
+
+**Complete examples** → [Examples Directory](../examples/)
+
+---
+
+[← Back to File Processing Patterns](./data-sources-04-file-patterns.md) | [Quick Reference →](../data-sources-quick-reference.md) | [↑ Back to Guide](../data-sources-readme.md)