@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/03-PATTERN-GUIDES/file-operations/file-operations-quick-reference.md

@@ -1,462 +1,462 @@
-# File Operations Quick Reference
-
-> **Cheat sheet for all file operations in the SDK**
-
-## S3 Operations
-
-### S3DataSource Setup
-
-```typescript
-import {
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = createConsoleLogger();
-const s3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    connectionId: 's3-quick',
-    name: 'S3 Quick Reference',
-    s3Config: {
-      region: 'us-east-1',
-      bucket: 'my-bucket',
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    },
-  },
-  logger
-);
-```
-
-### S3 Methods
-
-| Method           | Signature                                                                         | Description           |
-| ---------------- | --------------------------------------------------------------------------------- | --------------------- |
-| `listFiles()`    | `listFiles(options?: { prefix?: string, delimiter?: string, maxKeys?: number, continuationToken?: string }): Promise<FileMetadata[]>` | List S3 objects       |
-| `downloadFile()` | `downloadFile(key: string, options?: { encoding?: 'utf8'\|'binary' }): Promise<string\|Buffer>`                                      | Download file content |
-| `uploadFile()`   | `uploadFile(key: string, content: string\|Buffer, options?: { contentType?: string; metadata?: Record<string, string> }): Promise<void>`                 | Upload to S3          |
-| `moveFile()`     | `moveFile(sourcePath: string, destPath: string, deleteSource: boolean = true): Promise<void>`                     | Move (copy + delete) ⚠️ **deleteSource defaults to true** |
-| `copyFile()`     | `copyFile(sourceKey: string, destKey: string, options?: { sourceBucket?: string; metadata?: Record<string, string> }): Promise<void>`                     | Copy object           |
-| `deleteFile()`   | `deleteFile(key: string): Promise<void>`                                          | Delete object         |
-
-### S3 Examples
-
-```typescript
-// List files with prefix
-const files = await s3.listFiles({ prefix: 'incoming/', maxKeys: 1000 });
-
-// Read file
-const content = await s3.downloadFile('incoming/data.csv');
-
-// Write file with metadata
-await s3.uploadFile('output/results.json', JSON.stringify(data), {
-  contentType: 'application/json',
-  metadata: { processedBy: 'sdk' }
-});
-
-// Move to archive (copy + delete)
-await s3.moveFile('incoming/data.csv', 'archive/data.csv', true);
-
-// Copy file
-await s3.copyFile('source/file.txt', 'backup/file.txt');
-
-// Delete file
-await s3.deleteFile('temp/old-file.csv');
-```
-
-## SFTP Operations
-
-### SftpDataSource Setup
-
-```typescript
-import {
-  SftpDataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = createConsoleLogger();
-const sftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'sftp-quick',
-    name: 'SFTP Quick Reference',
-    settings: {
-      host: process.env.SFTP_HOST!,
-      port: 22,
-      username: process.env.SFTP_USERNAME!,
-      password: process.env.SFTP_PASSWORD!,
-    },
-  },
-  logger
-);
-
-// Always dispose when done!
-try {
-  // operations...
-} finally {
-  await sftp.dispose();
-}
-```
-
-### SFTP Methods
-
-| Method            | Signature                                                                                                                                   | Description                 |
-| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
-| `listFiles()`     | `listFiles(params?: { lastProcessedTimestamp?: string; fileName?: string; remotePath?: string; filePattern?: string; }): Promise<FileMetadata[]>` | List files matching pattern |
-| `downloadFile()`  | `downloadFile(path: string, options?: { encoding?: string; asBuffer?: boolean; maxSize?: number; }): Promise<string \| Buffer>`            | Download file content       |
-| `uploadFile()` | `uploadFile(remotePath: string, content: string\|Buffer, options?: { overwrite?: boolean; createDirectories?: boolean; permissions?: string; encoding?: string }): Promise<void>` | Upload content              |
-| `moveFile()`      | `moveFile(oldPath: string, newPath: string, overwrite: boolean = false): Promise<void>`                                                    | Move/rename file            |
-| `copyFile()`      | `copyFile(sourcePath: string, destPath: string, overwrite: boolean = false): Promise<void>`                                                | Copy file                   |
-| `deleteFile()`    | `deleteFile(remotePath: string): Promise<void>`                                                                                             | Delete file                 |
-| `fileExists()`    | `fileExists(remotePath: string): Promise<boolean>`                                                                                          | Check existence             |
-
-**Note:** For `uploadFile()`, the options parameter is optional:
-- Defaults: `overwrite: false`, `createDirectories: false`, `permissions: server default`, `encoding: 'utf8'`
-- **Throws `DataSourceError`** if `overwrite: false` and file exists, or if `createDirectories: false` and parent directory missing
-- Supported encodings: `'utf8'`, `'ascii'`, `'base64'`, `'hex'`, `'utf16le'`, `'latin1'`
-- Common permissions: `'0644'` (rw-r--r--), `'0755'` (rwxr-xr-x), `'0600'` (rw-------)
-
-### SFTP Examples
-
-```typescript
-// List files
-const files = await sftp.listFiles();
-
-// Read file
-const content = await sftp.downloadFile('/data/incoming/orders.csv');
-
-// Write file
-await sftp.uploadFile('/data/outgoing/results.json', JSON.stringify(data));
-
-// Move to archive (create directories if needed)
-await sftp.moveFile('/data/incoming/orders.csv', '/archive/2025/orders.csv', true);
-
-// Copy file
-await sftp.copyFile('/data/file.txt', '/backup/file.txt');
-
-// Delete file
-await sftp.deleteFile('/temp/old-file.csv');
-
-// Check if exists
-if (await sftp.fileExists('/data/file.csv')) {
-  // process...
-}
-```
-
-## Archive Patterns
-
-### Simple Archive (Same Day)
-
-```typescript
-// S3
-await s3.moveFile('incoming/data.csv', 'archive/data.csv');
-
-// SFTP
-await sftp.moveFile('/incoming/data.csv', '/archive/data.csv');
-```
-
-### Date-Based Archive
-
-```typescript
-function getArchivePath(fileName: string, folder: string): string {
-  const date = new Date().toISOString().split('T')[0]; // YYYY-MM-DD
-  return `archive/${folder}/${date}/${fileName}`;
-}
-
-// S3
-await s3.moveFile('incoming/orders.csv', getArchivePath('orders.csv', 'orders'));
-
-// SFTP
-await sftp.moveFile('/incoming/orders.csv', '/' + getArchivePath('orders.csv', 'orders'));
-```
-
-### Year/Month/Day Archive
-
-```typescript
-function getYearMonthDayPath(fileName: string, type: string): string {
-  const now = new Date();
-  const year = now.getFullYear();
-  const month = String(now.getMonth() + 1).padStart(2, '0');
-  const day = String(now.getDate()).padStart(2, '0');
-  return `archive/${type}/${year}/${month}/${day}/${fileName}`;
-}
-
-// S3
-await s3.moveFile('incoming/inventory.csv', getYearMonthDayPath('inventory.csv', 'inventory'));
-
-// SFTP
-await sftp.moveFile(
-  '/incoming/inventory.csv',
-  '/' + getYearMonthDayPath('inventory.csv', 'inventory')
-);
-```
-
-## Multi-Folder Processing
-
-### S3 Multi-Folder
-
-```typescript
-const folders = ['orders', 'inventory', 'products'];
-
-for (const folder of folders) {
-  const files = await s3.listFiles({ prefix: `incoming/${folder}/` });
-
-  for (const file of files) {
-    const content = await s3.downloadFile(file.name);
-    await processData(folder, content);
-
-    await s3.moveFile(file.name, file.name.replace('incoming/', 'archive/'));
-  }
-}
-```
-
-### SFTP Multi-Folder
-
-```typescript
-const folders = ['orders', 'inventory', 'products'];
-
-for (const folder of folders) {
-  const sftp = new SftpDataSource(
-    {
-      type: 'SFTP_CSV',
-      connectionId: `sftp-${folder}`,
-      name: `SFTP ${folder}`,
-      settings: {
-        host: process.env.SFTP_HOST!,
-        username: process.env.SFTP_USERNAME!,
-        password: process.env.SFTP_PASSWORD!,
-      },
-    },
-    logger
-  );
-
-  try {
-    const files = await sftp.listFiles();
-
-    for (const file of files) {
-      const content = await sftp.downloadFile(file.path);
-      await processData(folder, content);
-
-      const archivePath = `/archive/${folder}/${file.name}`;
-      await sftp.moveFile(file.path, archivePath);
-    }
-  } finally {
-    await sftp.dispose();
-  }
-}
-```
-
-## Handling Large Files
-
-### S3 Large File Processing
-
-```typescript
-// For files >100MB: Download as Buffer and process in chunks
-import { Buffer } from 'node:buffer';
-
-const buffer = await s3.downloadFile('large-file.csv', { encoding: 'binary' });
-
-const chunkSize = 1024 * 1024; // 1MB chunks
-let offset = 0;
-
-while (offset < buffer.length) {
-  const chunk = buffer.slice(offset, offset + chunkSize);
-  processChunk(chunk.toString('utf8'));
-  offset += chunkSize;
-}
-```
-
-### SFTP Large File Processing
-
-```typescript
-// For files >100MB: Use maxSize option and download as Buffer
-import { Buffer } from 'node:buffer';
-
-const buffer = await sftp.downloadFile('/data/large-file.csv', {
-  asBuffer: true,
-  maxSize: 100 * 1024 * 1024  // 100 MB limit
-});
-
-const chunkSize = 1024 * 1024; // 1MB chunks
-let offset = 0;
-
-while (offset < buffer.length) {
-  const chunk = buffer.slice(offset, offset + chunkSize);
-  processChunk(chunk.toString('utf8'));
-  offset += chunkSize;
-}
-```
-
-**Note:** The SDK does not provide built-in streaming. For very large files (>500MB), consider splitting files at the source or using chunked processing strategies. See [Module 5: Streaming & Performance](./modules/file-operations-05-streaming-performance.md) for advanced patterns.
-
-## Error Handling
-
-### S3 Error Handling
-
-```typescript
-try {
-  const content = await s3.downloadFile('file.csv');
-} catch (error) {
-  if (error.name === 'NoSuchKey') {
-    console.error('File not found');
-  } else if (error.name === 'AccessDenied') {
-    console.error('Permission denied');
-  } else {
-    console.error('S3 error:', error);
-  }
-}
-```
-
-### SFTP Error Handling
-
-```typescript
-try {
-  const content = await sftp.downloadFile('/data/file.csv');
-} catch (error) {
-  if (error.message.includes('No such file')) {
-    console.error('File not found');
-  } else if (error.message.includes('Permission denied')) {
-    console.error('Permission denied');
-  } else {
-    console.error('SFTP error:', error);
-  }
-}
-```
-
-## Best Practices
-
-### Always Dispose SFTP Connections
-
-```typescript
-// ✅ CORRECT
-try {
-  const files = await sftp.listFiles();
-  // ... process files
-} finally {
-  await sftp.dispose(); // Always clean up!
-}
-
-// ❌ WRONG - connection leak!
-const files = await sftp.listFiles();
-// ... process files
-// No dispose() = connection stays open
-```
-
-### Use Buffer + Chunked Processing for Large Files
-
-```typescript
-import { Buffer } from 'node:buffer';
-
-// ❌ WRONG - loads entire 5GB file into memory as string
-const content = await s3.downloadFile('huge-file.csv'); // OOM!
-
-// ✅ CORRECT - download as Buffer and process in chunks (constant memory)
-const buffer = await s3.downloadFile('huge-file.csv', { encoding: 'binary' });
-
-const chunkSize = 1024 * 1024; // 1MB chunks
-for (let offset = 0; offset < buffer.length; offset += chunkSize) {
-  const chunk = buffer.slice(offset, offset + chunkSize);
-  processChunk(chunk.toString('utf8')); // Process 1MB at a time
-}
-```
-
-### Archive Before Processing
-
-```typescript
-// ❌ RISKY - delete source immediately
-await processData(content);
-await s3.deleteFile('source.csv'); // What if processing fails?
-
-// ✅ SAFE - archive for audit trail
-await processData(content);
-await s3.moveFile('source.csv', 'archive/source.csv'); // Can recover if needed
-```
-
-### Retry Network Operations
-
-```typescript
-async function downloadWithRetry(
-  dataSource: S3DataSource | SftpDataSource,
-  path: string,
-  maxRetries = 3
-): Promise<string> {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      return await dataSource.downloadFile(path);
-    } catch (error) {
-      if (attempt === maxRetries) throw error;
-
-      const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
-      console.log(`Retry ${attempt}/${maxRetries} after ${delay}ms`);
-      await new Promise(resolve => setTimeout(resolve, delay));
-    }
-  }
-  throw new Error('Should not reach here');
-}
-```
-
-## Testing
-
-### Run S3 Tests
-
-```bash
-# Unit tests
-npm test -- tests/unit/s3-data-source.test.ts
-
-# Service tests
-npm test -- tests/unit/services/s3/
-```
-
-### Run SFTP Tests
-
-```bash
-# Unit tests
-npm test -- tests/unit/data-sources/sftp-data-source.test.ts
-
-# Integration tests (requires .env)
-npm test -- tests/integration/sftp-data-source.integration.test.ts
-```
-
-## Environment Variables
-
-### S3 Configuration
-
-```bash
-AWS_ACCESS_KEY_ID=your-access-key
-AWS_SECRET_ACCESS_KEY=your-secret-key
-AWS_REGION=us-east-1
-S3_BUCKET=my-bucket
-```
-
-### SFTP Configuration
-
-```bash
-SFTP_HOST=sftp.example.com
-SFTP_PORT=22
-SFTP_USERNAME=your-username
-SFTP_PASSWORD=your-password
-SFTP_REMOTE_PATH=/data/incoming
-SFTP_FILE_PATTERN=*.csv
-```
-
-## Common Issues
-
-| Issue                   | Cause              | Solution                                       |
-| ----------------------- | ------------------ | ---------------------------------------------- |
-| SFTP connection timeout | Network/firewall   | Check host/port, verify network access         |
-| S3 AccessDenied         | IAM permissions    | Verify bucket policy and IAM role              |
-| SFTP "No such file"     | Wrong path         | Use absolute paths starting with `/`           |
-| Memory errors           | Large files        | Use `downloadFile()` with `encoding: 'binary'` (S3) or `asBuffer: true` (SFTP) and process in chunks |
-| SFTP connection leak    | No dispose         | Always call `dispose()` in `finally` block     |
-| S3 NoSuchKey            | File doesn't exist | Use `listFiles()` to verify file exists        |
-
-## See Also
-
-- [Module 3: S3 Operations](./modules/file-operations-03-s3-operations.md) - Complete S3 guide
-- [Module 4: SFTP Operations](./modules/file-operations-04-sftp-operations.md) - Complete SFTP guide
-- [Module 5: Streaming & Performance](./modules/file-operations-05-streaming-performance.md) - Large file handling
-- [Module 6: Archive Patterns](./modules/file-operations-06-archive-patterns.md) - Advanced archiving
-- [Examples Directory](./examples/) - Complete working examples
+# File Operations Quick Reference
+
+> **Cheat sheet for all file operations in the SDK**
+
+## S3 Operations
+
+### S3DataSource Setup
+
+```typescript
+import {
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = createConsoleLogger();
+const s3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    connectionId: 's3-quick',
+    name: 'S3 Quick Reference',
+    s3Config: {
+      region: 'us-east-1',
+      bucket: 'my-bucket',
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    },
+  },
+  logger
+);
+```
+
+### S3 Methods
+
+| Method           | Signature                                                                         | Description           |
+| ---------------- | --------------------------------------------------------------------------------- | --------------------- |
+| `listFiles()`    | `listFiles(options?: { prefix?: string, delimiter?: string, maxKeys?: number, continuationToken?: string }): Promise<FileMetadata[]>` | List S3 objects       |
+| `downloadFile()` | `downloadFile(key: string, options?: { encoding?: 'utf8'\|'binary' }): Promise<string\|Buffer>`                                      | Download file content |
+| `uploadFile()`   | `uploadFile(key: string, content: string\|Buffer, options?: { contentType?: string; metadata?: Record<string, string> }): Promise<void>`                 | Upload to S3          |
+| `moveFile()`     | `moveFile(sourcePath: string, destPath: string, deleteSource: boolean = true): Promise<void>`                     | Move (copy + delete) ⚠️ **deleteSource defaults to true** |
+| `copyFile()`     | `copyFile(sourceKey: string, destKey: string, options?: { sourceBucket?: string; metadata?: Record<string, string> }): Promise<void>`                     | Copy object           |
+| `deleteFile()`   | `deleteFile(key: string): Promise<void>`                                          | Delete object         |
+
+### S3 Examples
+
+```typescript
+// List files with prefix
+const files = await s3.listFiles({ prefix: 'incoming/', maxKeys: 1000 });
+
+// Read file
+const content = await s3.downloadFile('incoming/data.csv');
+
+// Write file with metadata
+await s3.uploadFile('output/results.json', JSON.stringify(data), {
+  contentType: 'application/json',
+  metadata: { processedBy: 'sdk' }
+});
+
+// Move to archive (copy + delete)
+await s3.moveFile('incoming/data.csv', 'archive/data.csv', true);
+
+// Copy file
+await s3.copyFile('source/file.txt', 'backup/file.txt');
+
+// Delete file
+await s3.deleteFile('temp/old-file.csv');
+```
+
+## SFTP Operations
+
+### SftpDataSource Setup
+
+```typescript
+import {
+  SftpDataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = createConsoleLogger();
+const sftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'sftp-quick',
+    name: 'SFTP Quick Reference',
+    settings: {
+      host: process.env.SFTP_HOST!,
+      port: 22,
+      username: process.env.SFTP_USERNAME!,
+      password: process.env.SFTP_PASSWORD!,
+    },
+  },
+  logger
+);
+
+// Always dispose when done!
+try {
+  // operations...
+} finally {
+  await sftp.dispose();
+}
+```
+
+### SFTP Methods
+
+| Method            | Signature                                                                                                                                   | Description                 |
+| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
+| `listFiles()`     | `listFiles(params?: { lastProcessedTimestamp?: string; fileName?: string; remotePath?: string; filePattern?: string; }): Promise<FileMetadata[]>` | List files matching pattern |
+| `downloadFile()`  | `downloadFile(path: string, options?: { encoding?: string; asBuffer?: boolean; maxSize?: number; }): Promise<string \| Buffer>`            | Download file content       |
+| `uploadFile()` | `uploadFile(remotePath: string, content: string\|Buffer, options?: { overwrite?: boolean; createDirectories?: boolean; permissions?: string; encoding?: string }): Promise<void>` | Upload content              |
+| `moveFile()`      | `moveFile(oldPath: string, newPath: string, overwrite: boolean = false): Promise<void>`                                                    | Move/rename file            |
+| `copyFile()`      | `copyFile(sourcePath: string, destPath: string, overwrite: boolean = false): Promise<void>`                                                | Copy file                   |
+| `deleteFile()`    | `deleteFile(remotePath: string): Promise<void>`                                                                                             | Delete file                 |
+| `fileExists()`    | `fileExists(remotePath: string): Promise<boolean>`                                                                                          | Check existence             |
+
+**Note:** For `uploadFile()`, the options parameter is optional:
+- Defaults: `overwrite: false`, `createDirectories: false`, `permissions: server default`, `encoding: 'utf8'`
+- **Throws `DataSourceError`** if `overwrite: false` and file exists, or if `createDirectories: false` and parent directory missing
+- Supported encodings: `'utf8'`, `'ascii'`, `'base64'`, `'hex'`, `'utf16le'`, `'latin1'`
+- Common permissions: `'0644'` (rw-r--r--), `'0755'` (rwxr-xr-x), `'0600'` (rw-------)
+
+### SFTP Examples
+
+```typescript
+// List files
+const files = await sftp.listFiles();
+
+// Read file
+const content = await sftp.downloadFile('/data/incoming/orders.csv');
+
+// Write file
+await sftp.uploadFile('/data/outgoing/results.json', JSON.stringify(data));
+
+// Move to archive (create directories if needed)
+await sftp.moveFile('/data/incoming/orders.csv', '/archive/2025/orders.csv', true);
+
+// Copy file
+await sftp.copyFile('/data/file.txt', '/backup/file.txt');
+
+// Delete file
+await sftp.deleteFile('/temp/old-file.csv');
+
+// Check if exists
+if (await sftp.fileExists('/data/file.csv')) {
+  // process...
+}
+```
+
+## Archive Patterns
+
+### Simple Archive (Same Day)
+
+```typescript
+// S3
+await s3.moveFile('incoming/data.csv', 'archive/data.csv');
+
+// SFTP
+await sftp.moveFile('/incoming/data.csv', '/archive/data.csv');
+```
+
+### Date-Based Archive
+
+```typescript
+function getArchivePath(fileName: string, folder: string): string {
+  const date = new Date().toISOString().split('T')[0]; // YYYY-MM-DD
+  return `archive/${folder}/${date}/${fileName}`;
+}
+
+// S3
+await s3.moveFile('incoming/orders.csv', getArchivePath('orders.csv', 'orders'));
+
+// SFTP
+await sftp.moveFile('/incoming/orders.csv', '/' + getArchivePath('orders.csv', 'orders'));
+```
+
+### Year/Month/Day Archive
+
+```typescript
+function getYearMonthDayPath(fileName: string, type: string): string {
+  const now = new Date();
+  const year = now.getFullYear();
+  const month = String(now.getMonth() + 1).padStart(2, '0');
+  const day = String(now.getDate()).padStart(2, '0');
+  return `archive/${type}/${year}/${month}/${day}/${fileName}`;
+}
+
+// S3
+await s3.moveFile('incoming/inventory.csv', getYearMonthDayPath('inventory.csv', 'inventory'));
+
+// SFTP
+await sftp.moveFile(
+  '/incoming/inventory.csv',
+  '/' + getYearMonthDayPath('inventory.csv', 'inventory')
+);
+```
+
+## Multi-Folder Processing
+
+### S3 Multi-Folder
+
+```typescript
+const folders = ['orders', 'inventory', 'products'];
+
+for (const folder of folders) {
+  const files = await s3.listFiles({ prefix: `incoming/${folder}/` });
+
+  for (const file of files) {
+    const content = await s3.downloadFile(file.name);
+    await processData(folder, content);
+
+    await s3.moveFile(file.name, file.name.replace('incoming/', 'archive/'));
+  }
+}
+```
+
+### SFTP Multi-Folder
+
+```typescript
+const folders = ['orders', 'inventory', 'products'];
+
+for (const folder of folders) {
+  const sftp = new SftpDataSource(
+    {
+      type: 'SFTP_CSV',
+      connectionId: `sftp-${folder}`,
+      name: `SFTP ${folder}`,
+      settings: {
+        host: process.env.SFTP_HOST!,
+        username: process.env.SFTP_USERNAME!,
+        password: process.env.SFTP_PASSWORD!,
+      },
+    },
+    logger
+  );
+
+  try {
+    const files = await sftp.listFiles();
+
+    for (const file of files) {
+      const content = await sftp.downloadFile(file.path);
+      await processData(folder, content);
+
+      const archivePath = `/archive/${folder}/${file.name}`;
+      await sftp.moveFile(file.path, archivePath);
+    }
+  } finally {
+    await sftp.dispose();
+  }
+}
+```
+
+## Handling Large Files
+
+### S3 Large File Processing
+
+```typescript
+// For files >100MB: Download as Buffer and process in chunks
+import { Buffer } from 'node:buffer';
+
+const buffer = await s3.downloadFile('large-file.csv', { encoding: 'binary' });
+
+const chunkSize = 1024 * 1024; // 1MB chunks
+let offset = 0;
+
+while (offset < buffer.length) {
+  const chunk = buffer.slice(offset, offset + chunkSize);
+  processChunk(chunk.toString('utf8'));
+  offset += chunkSize;
+}
+```
+
+### SFTP Large File Processing
+
+```typescript
+// For files >100MB: Use maxSize option and download as Buffer
+import { Buffer } from 'node:buffer';
+
+const buffer = await sftp.downloadFile('/data/large-file.csv', {
+  asBuffer: true,
+  maxSize: 100 * 1024 * 1024  // 100 MB limit
+});
+
+const chunkSize = 1024 * 1024; // 1MB chunks
+let offset = 0;
+
+while (offset < buffer.length) {
+  const chunk = buffer.slice(offset, offset + chunkSize);
+  processChunk(chunk.toString('utf8'));
+  offset += chunkSize;
+}
+```
+
+**Note:** The SDK does not provide built-in streaming. For very large files (>500MB), consider splitting files at the source or using chunked processing strategies. See [Module 5: Streaming & Performance](./modules/file-operations-05-streaming-performance.md) for advanced patterns.
+
+## Error Handling
+
+### S3 Error Handling
+
+```typescript
+try {
+  const content = await s3.downloadFile('file.csv');
+} catch (error) {
+  if (error.name === 'NoSuchKey') {
+    console.error('File not found');
+  } else if (error.name === 'AccessDenied') {
+    console.error('Permission denied');
+  } else {
+    console.error('S3 error:', error);
+  }
+}
+```
+
+### SFTP Error Handling
+
+```typescript
+try {
+  const content = await sftp.downloadFile('/data/file.csv');
+} catch (error) {
+  if (error.message.includes('No such file')) {
+    console.error('File not found');
+  } else if (error.message.includes('Permission denied')) {
+    console.error('Permission denied');
+  } else {
+    console.error('SFTP error:', error);
+  }
+}
+```
+
+## Best Practices
+
+### Always Dispose SFTP Connections
+
+```typescript
+// ✅ CORRECT
+try {
+  const files = await sftp.listFiles();
+  // ... process files
+} finally {
+  await sftp.dispose(); // Always clean up!
+}
+
+// ❌ WRONG - connection leak!
+const files = await sftp.listFiles();
+// ... process files
+// No dispose() = connection stays open
+```
+
+### Use Buffer + Chunked Processing for Large Files
+
+```typescript
+import { Buffer } from 'node:buffer';
+
+// ❌ WRONG - loads entire 5GB file into memory as string
+const content = await s3.downloadFile('huge-file.csv'); // OOM!
+
+// ✅ CORRECT - download as Buffer and process in chunks (constant memory)
+const buffer = await s3.downloadFile('huge-file.csv', { encoding: 'binary' });
+
+const chunkSize = 1024 * 1024; // 1MB chunks
+for (let offset = 0; offset < buffer.length; offset += chunkSize) {
+  const chunk = buffer.slice(offset, offset + chunkSize);
+  processChunk(chunk.toString('utf8')); // Process 1MB at a time
+}
+```
+
+### Archive Before Processing
+
+```typescript
+// ❌ RISKY - delete source immediately
+await processData(content);
+await s3.deleteFile('source.csv'); // What if processing fails?
+
+// ✅ SAFE - archive for audit trail
+await processData(content);
+await s3.moveFile('source.csv', 'archive/source.csv'); // Can recover if needed
+```
+
+### Retry Network Operations
+
+```typescript
+async function downloadWithRetry(
+  dataSource: S3DataSource | SftpDataSource,
+  path: string,
+  maxRetries = 3
+): Promise<string> {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await dataSource.downloadFile(path);
+    } catch (error) {
+      if (attempt === maxRetries) throw error;
+
+      const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
+      console.log(`Retry ${attempt}/${maxRetries} after ${delay}ms`);
+      await new Promise(resolve => setTimeout(resolve, delay));
+    }
+  }
+  throw new Error('Should not reach here');
+}
+```
+
+## Testing
+
+### Run S3 Tests
+
+```bash
+# Unit tests
+npm test -- tests/unit/s3-data-source.test.ts
+
+# Service tests
+npm test -- tests/unit/services/s3/
+```
+
+### Run SFTP Tests
+
+```bash
+# Unit tests
+npm test -- tests/unit/data-sources/sftp-data-source.test.ts
+
+# Integration tests (requires .env)
+npm test -- tests/integration/sftp-data-source.integration.test.ts
+```
+
+## Environment Variables
+
+### S3 Configuration
+
+```bash
+AWS_ACCESS_KEY_ID=your-access-key
+AWS_SECRET_ACCESS_KEY=your-secret-key
+AWS_REGION=us-east-1
+S3_BUCKET=my-bucket
+```
+
+### SFTP Configuration
+
+```bash
+SFTP_HOST=sftp.example.com
+SFTP_PORT=22
+SFTP_USERNAME=your-username
+SFTP_PASSWORD=your-password
+SFTP_REMOTE_PATH=/data/incoming
+SFTP_FILE_PATTERN=*.csv
+```
+
+## Common Issues
+
+| Issue                   | Cause              | Solution                                       |
+| ----------------------- | ------------------ | ---------------------------------------------- |
+| SFTP connection timeout | Network/firewall   | Check host/port, verify network access         |
+| S3 AccessDenied         | IAM permissions    | Verify bucket policy and IAM role              |
+| SFTP "No such file"     | Wrong path         | Use absolute paths starting with `/`           |
+| Memory errors           | Large files        | Use `downloadFile()` with `encoding: 'binary'` (S3) or `asBuffer: true` (SFTP) and process in chunks |
+| SFTP connection leak    | No dispose         | Always call `dispose()` in `finally` block     |
+| S3 NoSuchKey            | File doesn't exist | Use `listFiles()` to verify file exists        |
+
+## See Also
+
+- [Module 3: S3 Operations](./modules/file-operations-03-s3-operations.md) - Complete S3 guide
+- [Module 4: SFTP Operations](./modules/file-operations-04-sftp-operations.md) - Complete SFTP guide
+- [Module 5: Streaming & Performance](./modules/file-operations-05-streaming-performance.md) - Large file handling
+- [Module 6: Archive Patterns](./modules/file-operations-06-archive-patterns.md) - Advanced archiving
+- [Examples Directory](./examples/) - Complete working examples