@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/03-PATTERN-GUIDES/file-operations/modules/file-operations-07-testing-troubleshooting.md

@@ -1,567 +1,567 @@
-# Module 7: Testing & Troubleshooting
-
-**Level:** Intermediate
-**Estimated Time:** 20 minutes
-
-## Overview
-
-Learn about test coverage for file operations and how to troubleshoot common issues.
-
-## Learning Objectives
-
-By the end of this module, you will:
-- ✅ Understand test coverage for S3 and SFTP operations
-- ✅ Run and interpret test results
-- ✅ Troubleshoot common file operation errors
-- ✅ Debug connection and permission issues
-
-## Test Coverage
-
-### S3 Test Coverage
-
-**Unit Tests**: `tests/unit/s3-data-source.test.ts`
-
-```typescript
-describe('S3DataSource', () => {
-  describe('listFiles', () => {
-    it('should list all files in bucket', async () => { ... });
-    it('should filter by prefix', async () => { ... });
-    it('should filter by pattern', async () => { ... });
-  });
-
-  describe('downloadFile', () => {
-    it('should download file successfully', async () => { ... });
-    it('should handle file not found errors', async () => { ... });
-  });
-
-  describe('uploadFile', () => {
-    it('should upload string content', async () => { ... });
-    it('should upload buffer content', async () => { ... });
-  });
-
-  describe('moveFile', () => {
-    it('should move file (copy + delete)', async () => { ... });
-    it('should handle move errors', async () => { ... });
-  });
-
-  describe('copyFile', () => {
-    it('should copy file successfully', async () => { ... });
-  });
-
-  describe('deleteFile', () => {
-    it('should delete file successfully', async () => { ... });
-  });
-});
-```
-
-**Service Tests**: `tests/unit/services/s3/s3-service.test.ts`
-
-### SFTP Test Coverage
-
-**Unit Tests**: `tests/unit/data-sources/sftp-data-source.test.ts`
-
-```typescript
-describe('SftpDataSource', () => {
-  describe('listFiles', () => {
-    it('should list files matching pattern', async () => { ... });
-    it('should handle empty directory', async () => { ... });
-  });
-
-  describe('downloadFile', () => {
-    it('should download file successfully', async () => { ... });
-    it('should handle file not found errors', async () => { ... });
-  });
-
-  describe('uploadContent', () => {
-    it('should upload content successfully', async () => { ... });
-  });
-
-  describe('moveFile', () => {
-    it('should move file successfully', async () => { ... });
-    it('should create directories if specified', async () => { ... });
-    it('should handle move errors', async () => { ... });
-  });
-
-  describe('copyFile', () => {
-    it('should copy file successfully', async () => { ... });
-    it('should handle copy errors', async () => { ... });
-  });
-
-  describe('deleteFile', () => {
-    it('should delete file successfully', async () => { ... });
-    it('should handle file not found errors', async () => { ... });
-  });
-});
-```
-
-**Integration Tests**: `tests/integration/sftp-data-source.integration.test.ts`
-
-### Running Tests
-
-```bash
-cd fc-connect-sdk
-
-# Run all tests
-npm test
-
-# Run with coverage
-npm run test:coverage
-
-# Run S3 tests only
-npm test -- tests/unit/s3-data-source.test.ts
-npm test -- tests/unit/services/s3/
-
-# Run SFTP tests only
-npm test -- tests/unit/data-sources/sftp-data-source.test.ts
-
-# Run SFTP integration tests (requires .env)
-npm test -- tests/integration/sftp-data-source.integration.test.ts
-
-# Run specific test
-npm test -- --testNamePattern="should download file successfully"
-```
-
-### Test Coverage Thresholds
-
-```javascript
-// jest.config.js
-coverageThreshold: {
-  global: {
-    branches: 70,
-    functions: 70,
-    lines: 70,
-    statements: 70
-  }
-}
-```
-
-## Common Errors & Solutions
-
-### S3 Errors
-
-#### NoSuchKey Error
-
-**Error:**
-```
-NoSuchKey: The specified key does not exist.
-```
-
-**Causes:**
-- File doesn't exist at the specified key
-- Typo in file path
-- File was deleted
-
-**Solutions:**
-```typescript
-// Check if file exists before downloading
-if (await s3.fileExists('path/to/file.csv')) {
-  const content = await s3.downloadFile('path/to/file.csv');
-} else {
-  console.log('File does not exist');
-}
-
-// List files to verify path
-const files = await s3.listFiles({ prefix: 'path/to/' });
-files.forEach(f => console.log(f.name));
-```
-
-#### AccessDenied Error
-
-**Error:**
-```
-AccessDenied: Access Denied
-```
-
-**Causes:**
-- IAM policy doesn't allow operation
-- Bucket policy blocks access
-- Wrong AWS credentials
-
-**Solutions:**
-```typescript
-// Verify IAM policy has required permissions:
-// - s3:GetObject (download)
-// - s3:PutObject (upload)
-// - s3:DeleteObject (delete/move)
-// - s3:ListBucket (list)
-
-// Check credentials
-console.log('Access Key ID:', process.env.AWS_ACCESS_KEY_ID?.substring(0, 8) + '...');
-console.log('Region:', process.env.AWS_REGION);
-console.log('Bucket:', process.env.S3_BUCKET);
-```
-
-#### InvalidBucketName Error
-
-**Error:**
-```
-InvalidBucketName: The specified bucket is not valid.
-```
-
-**Causes:**
-- Bucket name contains invalid characters
-- Bucket name doesn't exist
-- Wrong region
-
-**Solutions:**
-```typescript
-// Verify bucket name
-const bucketName = process.env.S3_BUCKET;
-console.log('Bucket name:', bucketName);
-
-// Check bucket exists in correct region
-const s3 = new S3DataSource({
-  s3Config: {
-    region: 'us-east-1', // Must match bucket region!
-    credentials: {...}
-  },
-  bucketName
-}, logger);
-```
-
-### SFTP Errors
-
-#### Connection Timeout
-
-**Error:**
-```
-Error: Timed out while waiting for handshake
-```
-
-**Causes:**
-- Wrong host or port
-- Firewall blocking connection
-- SFTP server down
-
-**Solutions:**
-```typescript
-// Verify connection details
-console.log('Host:', process.env.SFTP_HOST);
-console.log('Port:', process.env.SFTP_PORT);
-
-// Test with increased timeout
-const sftp = new SftpDataSource({
-  settings: {
-    host: process.env.SFTP_HOST!,
-    port: parseInt(process.env.SFTP_PORT || '22'),
-    username: process.env.SFTP_USERNAME!,
-    password: process.env.SFTP_PASSWORD!,
-    connectTimeout: 60000 // 60 seconds
-  }
-}, logger);
-```
-
-#### Authentication Failed
-
-**Error:**
-```
-Error: All configured authentication methods failed
-```
-
-**Causes:**
-- Wrong username or password
-- Key-based auth misconfigured
-- Account locked
-
-**Solutions:**
-```typescript
-// Verify credentials
-console.log('Username:', process.env.SFTP_USERNAME);
-console.log('Password length:', process.env.SFTP_PASSWORD?.length);
-
-// Try key-based auth
-import * as fs from 'fs';
-const privateKey = fs.readFileSync('/path/to/key.pem', 'utf-8');
-
-const sftp = new SftpDataSource({
-  settings: {
-    host: process.env.SFTP_HOST!,
-    username: process.env.SFTP_USERNAME!,
-    privateKey: privateKey,
-    passphrase: process.env.KEY_PASSPHRASE
-  }
-}, logger);
-```
-
-#### No Such File
-
-**Error:**
-```
-Error: No such file
-```
-
-**Causes:**
-- File doesn't exist
-- Wrong path (relative vs absolute)
-- Path typo
-
-**Solutions:**
-```typescript
-// Use absolute paths with leading slash
-const content = await sftp.downloadFile('/data/file.csv'); // ✅
-// NOT: sftp.downloadFile('data/file.csv'); // ❌
-
-// List files to verify path
-const files = await sftp.listFiles();
-files.forEach(f => console.log('Path:', f.path));
-
-// Check existence before download
-if (await sftp.fileExists('/data/file.csv')) {
-  const content = await sftp.downloadFile('/data/file.csv');
-}
-```
-
-#### Permission Denied
-
-**Error:**
-```
-Error: Permission denied
-```
-
-**Causes:**
-- User doesn't have read/write permissions
-- Parent directory not writable
-- File owned by different user
-
-**Solutions:**
-```typescript
-// Verify permissions on SFTP server
-// Use createDirs flag when moving to new folders
-await sftp.moveFile(
-  '/source/file.csv',
-  '/archive/2025/01/file.csv',
-  true  // Create directories with proper permissions
-);
-
-// Check user permissions with SFTP admin
-```
-
-### General Debugging Tips
-
-#### Enable Debug Logging
-
-```typescript
-import {
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = createConsoleLogger();
-logger.setLevel('debug'); // Show all debug messages
-
-const s3 = new S3DataSource(config, logger);
-
-// Now all operations will log details
-const files = await s3.listFiles();
-```
-
-#### SFTP Connection Pooling
-
-The SDK automatically manages SFTP connection pooling for better performance:
-
-```typescript
-const sftp = new SftpDataSource({
-  type: 'SFTP_CSV',
-  connectionId: 'sftp-pooled',
-  name: 'SFTP with Connection Pool',
-  settings: {
-    host: 'sftp.example.com',
-    username: 'user',
-    password: 'pass',
-    maxConnections: 5,  // Pool size (default: 5)
-    connectionWaitTimeout: 30000  // Wait for available connection (default: 30s)
-  }
-}, logger);
-
-// Connections are automatically reused from pool
-for (let i = 0; i < 100; i++) {
-  await sftp.downloadFile(`/data/file${i}.csv`);  // Reuses connections
-}
-
-// IMPORTANT: Always dispose to close all pooled connections
-await sftp.dispose();
-```
-
-**Connection Pool Features:**
-- **Automatic Reuse**: Connections are returned to pool after each operation
-- **Health Checks**: Unhealthy connections are removed and replaced
-- **Wait Queue**: Requests wait for available connection when pool is exhausted
-- **Concurrent Operations**: Multiple operations can share pooled connections
-
-**Tuning Connection Pool:**
-```typescript
-// High-concurrency workflow (many parallel operations)
-settings: {
-  maxConnections: 10,  // Increase pool size
-  connectionWaitTimeout: 60000  // Longer wait for busy systems
-}
-
-// Low-concurrency workflow (sequential operations)
-settings: {
-  maxConnections: 2,  // Smaller pool
-  connectionWaitTimeout: 15000  // Shorter timeout
-}
-```
-
-#### Connection Leak Detection
-
-```typescript
-// ❌ BAD: Connection not disposed
-const sftp = new SftpDataSource(config, logger);
-await sftp.downloadFile('/file.csv');
-// Missing dispose() - ALL pooled connections leak!
-
-// ✅ GOOD: Always dispose
-const sftp = new SftpDataSource(config, logger);
-try {
-  await sftp.downloadFile('/file.csv');
-} finally {
-  await sftp.dispose(); // Closes ALL pooled connections
-}
-```
-
-#### Built-in Retry Logic
-
-**SFTP** and **S3** both have built-in retry logic with exponential backoff:
-
-**SFTP Automatic Retries:**
-```typescript
-const sftp = new SftpDataSource({
-  type: 'SFTP_CSV',
-  connectionId: 'sftp-retry',
-  name: 'SFTP with Retry',
-  settings: {
-    host: 'sftp.example.com',
-    username: 'user',
-    password: 'pass',
-    retry: {
-      maxAttempts: 3,        // Number of retry attempts (default: 3)
-      baseDelayMs: 1000,     // Initial delay (default: 1000ms)
-      backoffFactor: 2,      // Exponential multiplier (default: 2)
-      maxDelayMs: 8000,      // Max delay cap (default: 8000ms)
-      jitter: 'full',        // Adds randomness to prevent thundering herd
-      reconnectOnFailure: true  // Reconnect on connection errors
-    }
-  }
-}, logger);
-
-// All operations automatically retry on transient errors
-await sftp.downloadFile('/data/file.csv');  // Retries if network fails
-```
-
-**How Exponential Backoff Works:**
-- Attempt 1: Immediate (0ms delay)
-- Attempt 2: 1000ms delay (base)
-- Attempt 3: 2000ms delay (base × 2)
-- Attempt 4: 4000ms delay (base × 4)
-- Jitter: Adds random variance (±50%) to prevent multiple clients retrying simultaneously
-
-**S3 Automatic Retries:**
-```typescript
-const s3 = new S3DataSource({
-  type: 'S3_CSV',
-  connectionId: 's3-retry',
-  name: 'S3 with Retry',
-  s3Config: {
-    region: 'us-east-1',
-    bucket: 'my-bucket',
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    maxAttempts: 3  // AWS SDK handles retry logic
-  }
-}, logger);
-
-// S3 automatically retries on:
-// - 500, 502, 503, 504 HTTP errors
-// - Connection timeouts
-// - Network errors
-```
-
-**Retryable vs Non-Retryable Errors:**
-
-✅ **Automatically Retried:**
-- Network timeouts
-- Connection refused
-- DNS resolution failures
-- 500 Internal Server Error
-- 502 Bad Gateway
-- 503 Service Unavailable
-- 504 Gateway Timeout
-
-❌ **NOT Retried (fail immediately):**
-- File not found (404, NoSuchKey)
-- Permission denied (403, AccessDenied)
-- Invalid credentials (401)
-- File already exists (when overwrite=false)
-- Invalid path/filename
-
-#### Network Issues
-
-**Built-in retry handles most cases automatically**, but you can add custom logic:
-
-```typescript
-// Custom retry logic (only if needed beyond built-in retries)
-async function downloadWithRetry(
-  dataSource: S3DataSource | SftpDataSource,
-  path: string,
-  maxRetries = 3
-): Promise<string> {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      return await dataSource.downloadFile(path);
-    } catch (error: any) {
-      // Don't retry if file doesn't exist
-      if (error.name === 'NoSuchKey' || error.message.includes('No such file')) {
-        throw error;
-      }
-
-      if (attempt === maxRetries) throw error;
-
-      const delay = Math.pow(2, attempt) * 1000;
-      console.log(`Retry ${attempt}/${maxRetries} after ${delay}ms`);
-      await new Promise(resolve => setTimeout(resolve, delay));
-    }
-  }
-  throw new Error('Should not reach here');
-}
-```
-
-## Troubleshooting Checklist
-
-### S3 Issues
-
-- [ ] Verify AWS credentials are set
-- [ ] Check bucket name is correct
-- [ ] Verify region matches bucket region
-- [ ] Confirm IAM policy has required permissions
-- [ ] Check file path (no leading slash for S3)
-- [ ] Verify network connectivity to AWS
-
-### SFTP Issues
-
-- [ ] Verify host and port are correct
-- [ ] Check username and password/key
-- [ ] Confirm firewall allows SFTP traffic
-- [ ] Use absolute paths (with leading slash)
-- [ ] Always call `dispose()` in `finally` block
-- [ ] Check SFTP server is running
-
-## Key Takeaways
-
-- 🎯 SDK has comprehensive test coverage (≥70%)
-- 🎯 Run `npm test` to verify all operations work
-- 🎯 Use integration tests for SFTP with real credentials
-- 🎯 Enable debug logging to troubleshoot issues
-- 🎯 Always handle NoSuchKey/NoSuchFile errors
-- 🎯 Use retry logic for transient network errors
-- 🎯 Check permissions for AccessDenied errors
-
-## Next Steps
-
-Continue to [Module 8: API Reference](../../error-handling/modules/error-handling-08-api-reference.md) for complete method signatures.
-
----
-
-**Related Resources:**
-- [Quick Reference](../../../02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md) - Error handling examples
-- [Test Files](../../../../tests/) - Browse actual test implementations
+# Module 7: Testing & Troubleshooting
+
+**Level:** Intermediate
+**Estimated Time:** 20 minutes
+
+## Overview
+
+Learn about test coverage for file operations and how to troubleshoot common issues.
+
+## Learning Objectives
+
+By the end of this module, you will:
+- ✅ Understand test coverage for S3 and SFTP operations
+- ✅ Run and interpret test results
+- ✅ Troubleshoot common file operation errors
+- ✅ Debug connection and permission issues
+
+## Test Coverage
+
+### S3 Test Coverage
+
+**Unit Tests**: `tests/unit/s3-data-source.test.ts`
+
+```typescript
+describe('S3DataSource', () => {
+  describe('listFiles', () => {
+    it('should list all files in bucket', async () => { ... });
+    it('should filter by prefix', async () => { ... });
+    it('should filter by pattern', async () => { ... });
+  });
+
+  describe('downloadFile', () => {
+    it('should download file successfully', async () => { ... });
+    it('should handle file not found errors', async () => { ... });
+  });
+
+  describe('uploadFile', () => {
+    it('should upload string content', async () => { ... });
+    it('should upload buffer content', async () => { ... });
+  });
+
+  describe('moveFile', () => {
+    it('should move file (copy + delete)', async () => { ... });
+    it('should handle move errors', async () => { ... });
+  });
+
+  describe('copyFile', () => {
+    it('should copy file successfully', async () => { ... });
+  });
+
+  describe('deleteFile', () => {
+    it('should delete file successfully', async () => { ... });
+  });
+});
+```
+
+**Service Tests**: `tests/unit/services/s3/s3-service.test.ts`
+
+### SFTP Test Coverage
+
+**Unit Tests**: `tests/unit/data-sources/sftp-data-source.test.ts`
+
+```typescript
+describe('SftpDataSource', () => {
+  describe('listFiles', () => {
+    it('should list files matching pattern', async () => { ... });
+    it('should handle empty directory', async () => { ... });
+  });
+
+  describe('downloadFile', () => {
+    it('should download file successfully', async () => { ... });
+    it('should handle file not found errors', async () => { ... });
+  });
+
+  describe('uploadContent', () => {
+    it('should upload content successfully', async () => { ... });
+  });
+
+  describe('moveFile', () => {
+    it('should move file successfully', async () => { ... });
+    it('should create directories if specified', async () => { ... });
+    it('should handle move errors', async () => { ... });
+  });
+
+  describe('copyFile', () => {
+    it('should copy file successfully', async () => { ... });
+    it('should handle copy errors', async () => { ... });
+  });
+
+  describe('deleteFile', () => {
+    it('should delete file successfully', async () => { ... });
+    it('should handle file not found errors', async () => { ... });
+  });
+});
+```
+
+**Integration Tests**: `tests/integration/sftp-data-source.integration.test.ts`
+
+### Running Tests
+
+```bash
+cd fc-connect-sdk
+
+# Run all tests
+npm test
+
+# Run with coverage
+npm run test:coverage
+
+# Run S3 tests only
+npm test -- tests/unit/s3-data-source.test.ts
+npm test -- tests/unit/services/s3/
+
+# Run SFTP tests only
+npm test -- tests/unit/data-sources/sftp-data-source.test.ts
+
+# Run SFTP integration tests (requires .env)
+npm test -- tests/integration/sftp-data-source.integration.test.ts
+
+# Run specific test
+npm test -- --testNamePattern="should download file successfully"
+```
+
+### Test Coverage Thresholds
+
+```javascript
+// jest.config.js
+coverageThreshold: {
+  global: {
+    branches: 70,
+    functions: 70,
+    lines: 70,
+    statements: 70
+  }
+}
+```
+
+## Common Errors & Solutions
+
+### S3 Errors
+
+#### NoSuchKey Error
+
+**Error:**
+```
+NoSuchKey: The specified key does not exist.
+```
+
+**Causes:**
+- File doesn't exist at the specified key
+- Typo in file path
+- File was deleted
+
+**Solutions:**
+```typescript
+// Check if file exists before downloading
+if (await s3.fileExists('path/to/file.csv')) {
+  const content = await s3.downloadFile('path/to/file.csv');
+} else {
+  console.log('File does not exist');
+}
+
+// List files to verify path
+const files = await s3.listFiles({ prefix: 'path/to/' });
+files.forEach(f => console.log(f.name));
+```
+
+#### AccessDenied Error
+
+**Error:**
+```
+AccessDenied: Access Denied
+```
+
+**Causes:**
+- IAM policy doesn't allow operation
+- Bucket policy blocks access
+- Wrong AWS credentials
+
+**Solutions:**
+```typescript
+// Verify IAM policy has required permissions:
+// - s3:GetObject (download)
+// - s3:PutObject (upload)
+// - s3:DeleteObject (delete/move)
+// - s3:ListBucket (list)
+
+// Check credentials
+console.log('Access Key ID:', process.env.AWS_ACCESS_KEY_ID?.substring(0, 8) + '...');
+console.log('Region:', process.env.AWS_REGION);
+console.log('Bucket:', process.env.S3_BUCKET);
+```
+
+#### InvalidBucketName Error
+
+**Error:**
+```
+InvalidBucketName: The specified bucket is not valid.
+```
+
+**Causes:**
+- Bucket name contains invalid characters
+- Bucket name doesn't exist
+- Wrong region
+
+**Solutions:**
+```typescript
+// Verify bucket name
+const bucketName = process.env.S3_BUCKET;
+console.log('Bucket name:', bucketName);
+
+// Check bucket exists in correct region
+const s3 = new S3DataSource({
+  s3Config: {
+    region: 'us-east-1', // Must match bucket region!
+    credentials: {...}
+  },
+  bucketName
+}, logger);
+```
+
+### SFTP Errors
+
+#### Connection Timeout
+
+**Error:**
+```
+Error: Timed out while waiting for handshake
+```
+
+**Causes:**
+- Wrong host or port
+- Firewall blocking connection
+- SFTP server down
+
+**Solutions:**
+```typescript
+// Verify connection details
+console.log('Host:', process.env.SFTP_HOST);
+console.log('Port:', process.env.SFTP_PORT);
+
+// Test with increased timeout
+const sftp = new SftpDataSource({
+  settings: {
+    host: process.env.SFTP_HOST!,
+    port: parseInt(process.env.SFTP_PORT || '22'),
+    username: process.env.SFTP_USERNAME!,
+    password: process.env.SFTP_PASSWORD!,
+    connectTimeout: 60000 // 60 seconds
+  }
+}, logger);
+```
+
+#### Authentication Failed
+
+**Error:**
+```
+Error: All configured authentication methods failed
+```
+
+**Causes:**
+- Wrong username or password
+- Key-based auth misconfigured
+- Account locked
+
+**Solutions:**
+```typescript
+// Verify credentials
+console.log('Username:', process.env.SFTP_USERNAME);
+console.log('Password length:', process.env.SFTP_PASSWORD?.length);
+
+// Try key-based auth
+import * as fs from 'fs';
+const privateKey = fs.readFileSync('/path/to/key.pem', 'utf-8');
+
+const sftp = new SftpDataSource({
+  settings: {
+    host: process.env.SFTP_HOST!,
+    username: process.env.SFTP_USERNAME!,
+    privateKey: privateKey,
+    passphrase: process.env.KEY_PASSPHRASE
+  }
+}, logger);
+```
+
+#### No Such File
+
+**Error:**
+```
+Error: No such file
+```
+
+**Causes:**
+- File doesn't exist
+- Wrong path (relative vs absolute)
+- Path typo
+
+**Solutions:**
+```typescript
+// Use absolute paths with leading slash
+const content = await sftp.downloadFile('/data/file.csv'); // ✅
+// NOT: sftp.downloadFile('data/file.csv'); // ❌
+
+// List files to verify path
+const files = await sftp.listFiles();
+files.forEach(f => console.log('Path:', f.path));
+
+// Check existence before download
+if (await sftp.fileExists('/data/file.csv')) {
+  const content = await sftp.downloadFile('/data/file.csv');
+}
+```
+
+#### Permission Denied
+
+**Error:**
+```
+Error: Permission denied
+```
+
+**Causes:**
+- User doesn't have read/write permissions
+- Parent directory not writable
+- File owned by different user
+
+**Solutions:**
+```typescript
+// Verify permissions on SFTP server
+// Use createDirs flag when moving to new folders
+await sftp.moveFile(
+  '/source/file.csv',
+  '/archive/2025/01/file.csv',
+  true  // Create directories with proper permissions
+);
+
+// Check user permissions with SFTP admin
+```
+
+### General Debugging Tips
+
+#### Enable Debug Logging
+
+```typescript
+import {
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = createConsoleLogger();
+logger.setLevel('debug'); // Show all debug messages
+
+const s3 = new S3DataSource(config, logger);
+
+// Now all operations will log details
+const files = await s3.listFiles();
+```
+
+#### SFTP Connection Pooling
+
+The SDK automatically manages SFTP connection pooling for better performance:
+
+```typescript
+const sftp = new SftpDataSource({
+  type: 'SFTP_CSV',
+  connectionId: 'sftp-pooled',
+  name: 'SFTP with Connection Pool',
+  settings: {
+    host: 'sftp.example.com',
+    username: 'user',
+    password: 'pass',
+    maxConnections: 5,  // Pool size (default: 5)
+    connectionWaitTimeout: 30000  // Wait for available connection (default: 30s)
+  }
+}, logger);
+
+// Connections are automatically reused from pool
+for (let i = 0; i < 100; i++) {
+  await sftp.downloadFile(`/data/file${i}.csv`);  // Reuses connections
+}
+
+// IMPORTANT: Always dispose to close all pooled connections
+await sftp.dispose();
+```
+
+**Connection Pool Features:**
+- **Automatic Reuse**: Connections are returned to pool after each operation
+- **Health Checks**: Unhealthy connections are removed and replaced
+- **Wait Queue**: Requests wait for available connection when pool is exhausted
+- **Concurrent Operations**: Multiple operations can share pooled connections
+
+**Tuning Connection Pool:**
+```typescript
+// High-concurrency workflow (many parallel operations)
+settings: {
+  maxConnections: 10,  // Increase pool size
+  connectionWaitTimeout: 60000  // Longer wait for busy systems
+}
+
+// Low-concurrency workflow (sequential operations)
+settings: {
+  maxConnections: 2,  // Smaller pool
+  connectionWaitTimeout: 15000  // Shorter timeout
+}
+```
+
+#### Connection Leak Detection
+
+```typescript
+// ❌ BAD: Connection not disposed
+const sftp = new SftpDataSource(config, logger);
+await sftp.downloadFile('/file.csv');
+// Missing dispose() - ALL pooled connections leak!
+
+// ✅ GOOD: Always dispose
+const sftp = new SftpDataSource(config, logger);
+try {
+  await sftp.downloadFile('/file.csv');
+} finally {
+  await sftp.dispose(); // Closes ALL pooled connections
+}
+```
+
+#### Built-in Retry Logic
+
+**SFTP** and **S3** both have built-in retry logic with exponential backoff:
+
+**SFTP Automatic Retries:**
+```typescript
+const sftp = new SftpDataSource({
+  type: 'SFTP_CSV',
+  connectionId: 'sftp-retry',
+  name: 'SFTP with Retry',
+  settings: {
+    host: 'sftp.example.com',
+    username: 'user',
+    password: 'pass',
+    retry: {
+      maxAttempts: 3,        // Number of retry attempts (default: 3)
+      baseDelayMs: 1000,     // Initial delay (default: 1000ms)
+      backoffFactor: 2,      // Exponential multiplier (default: 2)
+      maxDelayMs: 8000,      // Max delay cap (default: 8000ms)
+      jitter: 'full',        // Adds randomness to prevent thundering herd
+      reconnectOnFailure: true  // Reconnect on connection errors
+    }
+  }
+}, logger);
+
+// All operations automatically retry on transient errors
+await sftp.downloadFile('/data/file.csv');  // Retries if network fails
+```
+
+**How Exponential Backoff Works:**
+- Attempt 1: Immediate (0ms delay)
+- Attempt 2: 1000ms delay (base)
+- Attempt 3: 2000ms delay (base × 2)
+- Attempt 4: 4000ms delay (base × 4)
+- Jitter: Adds random variance (±50%) to prevent multiple clients retrying simultaneously
+
+**S3 Automatic Retries:**
+```typescript
+const s3 = new S3DataSource({
+  type: 'S3_CSV',
+  connectionId: 's3-retry',
+  name: 'S3 with Retry',
+  s3Config: {
+    region: 'us-east-1',
+    bucket: 'my-bucket',
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    maxAttempts: 3  // AWS SDK handles retry logic
+  }
+}, logger);
+
+// S3 automatically retries on:
+// - 500, 502, 503, 504 HTTP errors
+// - Connection timeouts
+// - Network errors
+```
+
+**Retryable vs Non-Retryable Errors:**
+
+✅ **Automatically Retried:**
+- Network timeouts
+- Connection refused
+- DNS resolution failures
+- 500 Internal Server Error
+- 502 Bad Gateway
+- 503 Service Unavailable
+- 504 Gateway Timeout
+
+❌ **NOT Retried (fail immediately):**
+- File not found (404, NoSuchKey)
+- Permission denied (403, AccessDenied)
+- Invalid credentials (401)
+- File already exists (when overwrite=false)
+- Invalid path/filename
+
+#### Network Issues
+
+**Built-in retry handles most cases automatically**, but you can add custom logic:
+
+```typescript
+// Custom retry logic (only if needed beyond built-in retries)
+async function downloadWithRetry(
+  dataSource: S3DataSource | SftpDataSource,
+  path: string,
+  maxRetries = 3
+): Promise<string> {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await dataSource.downloadFile(path);
+    } catch (error: any) {
+      // Don't retry if file doesn't exist
+      if (error.name === 'NoSuchKey' || error.message.includes('No such file')) {
+        throw error;
+      }
+
+      if (attempt === maxRetries) throw error;
+
+      const delay = Math.pow(2, attempt) * 1000;
+      console.log(`Retry ${attempt}/${maxRetries} after ${delay}ms`);
+      await new Promise(resolve => setTimeout(resolve, delay));
+    }
+  }
+  throw new Error('Should not reach here');
+}
+```
+
+## Troubleshooting Checklist
+
+### S3 Issues
+
+- [ ] Verify AWS credentials are set
+- [ ] Check bucket name is correct
+- [ ] Verify region matches bucket region
+- [ ] Confirm IAM policy has required permissions
+- [ ] Check file path (no leading slash for S3)
+- [ ] Verify network connectivity to AWS
+
+### SFTP Issues
+
+- [ ] Verify host and port are correct
+- [ ] Check username and password/key
+- [ ] Confirm firewall allows SFTP traffic
+- [ ] Use absolute paths (with leading slash)
+- [ ] Always call `dispose()` in `finally` block
+- [ ] Check SFTP server is running
+
+## Key Takeaways
+
+- 🎯 SDK has comprehensive test coverage (≥70%)
+- 🎯 Run `npm test` to verify all operations work
+- 🎯 Use integration tests for SFTP with real credentials
+- 🎯 Enable debug logging to troubleshoot issues
+- 🎯 Always handle NoSuchKey/NoSuchFile errors
+- 🎯 Use retry logic for transient network errors
+- 🎯 Check permissions for AccessDenied errors
+
+## Next Steps
+
+Continue to [Module 8: API Reference](../../error-handling/modules/error-handling-08-api-reference.md) for complete method signatures.
+
+---
+
+**Related Resources:**
+- [Quick Reference](../../../02-CORE-GUIDES/advanced-services/advanced-services-quick-reference.md) - Error handling examples
+- [Test Files](../../../../tests/) - Browse actual test implementations