@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/03-PATTERN-GUIDES/file-operations/file-operations-readme.md

@@ -1,379 +1,379 @@
-# File Operations Learning Path
-
-> **Complete guide to file operations in the Fluent Connect SDK**
-> Progressive learning modules with S3, SFTP, streaming, and archive patterns
-
-## 📚 What You'll Learn
-
-This guide teaches you how to work with files across different storage systems using the SDK's unified data source abstractions. By the end, you'll understand:
-
-- ✅ How to read, write, move, copy, and delete files on S3 and SFTP
-- ✅ How to stream large files efficiently without memory issues
-- ✅ How to implement archive workflows with date-based organization
-- ✅ How to process files from multiple folders and storage systems
-- ✅ How to test file operations with comprehensive test coverage
-
-## 🎯 Learning Path Overview
-
-This guide is organized into **8 progressive modules**. Follow them in order for the best learning experience:
-
-| Module | Topic | Level |
-|--------|-------|-------|
-| [Module 1](./modules/file-operations-01-foundations.md) | **Foundations** - Data sources, abstraction patterns, when to use S3 vs SFTP | Beginner |
-| [Module 2](./modules/file-operations-02-quick-start.md) | **Quick Start** - First working examples for S3 and SFTP | Beginner |
-| [Module 3](./modules/file-operations-03-s3-operations.md) | **S3 Operations** - Complete S3 file operations guide | Intermediate |
-| [Module 4](./modules/file-operations-04-sftp-operations.md) | **SFTP Operations** - Complete SFTP file operations guide | Intermediate |
-| [Module 5](./modules/file-operations-05-streaming-performance.md) | **Performance Optimization** - Handle large files efficiently | Advanced |
-| [Module 6](./modules/file-operations-06-archive-patterns.md) | **Archive Patterns** - Date-based archiving, multi-folder workflows | Intermediate |
-| [Module 7](./modules/file-operations-07-testing-troubleshooting.md) | **Testing & Troubleshooting** - Test coverage, common issues, debugging | Intermediate |
-| [Module 8](./modules/file-operations-08-api-reference.md) | **API Reference** - Complete method signatures and interfaces | Reference |
-
-## 🚀 Quick Navigation
-
-### By Use Case
-
-- **First time with file operations?** → Start with [Module 1: Foundations](./modules/file-operations-01-foundations.md)
-- **Need a working example now?** → Jump to [Module 2: Quick Start](./modules/file-operations-02-quick-start.md)
-- **Working with S3?** → See [Module 3: S3 Operations](./modules/file-operations-03-s3-operations.md)
-- **Working with SFTP?** → See [Module 4: SFTP Operations](./modules/file-operations-04-sftp-operations.md)
-- **Large file issues?** → Check [Module 5: Streaming & Performance](./modules/file-operations-05-streaming-performance.md)
-- **Need archive workflows?** → See [Module 6: Archive Patterns](./modules/file-operations-06-archive-patterns.md)
-- **Experiencing errors?** → Browse [Module 7: Troubleshooting](./modules/file-operations-07-testing-troubleshooting.md)
-- **Want code examples?** → Browse [Examples Directory](./examples/)
-
-### By Storage System
-
-| Storage | Complexity | Best For | See Module |
-|---------|------------|----------|---------|
-| **S3** | Low | Cloud storage, large datasets, serverless | [Module 3: S3 Operations](./modules/file-operations-03-s3-operations.md) |
-| **SFTP** | Medium | Enterprise systems, B2B integrations, secure transfers | [Module 4: SFTP Operations](./modules/file-operations-04-sftp-operations.md) |
-| **Local Files** | Low | Development, testing, small scripts | [Module 2: Quick Start](./modules/file-operations-02-quick-start.md) |
-
-### By Operation Type
-
-| Operation | S3 | SFTP | Complexity | See Module |
-|-----------|----|----|------------|---------|
-| **List Files** | ✅ | ✅ | Low | [Module 3](./modules/file-operations-03-s3-operations.md), [Module 4](./modules/file-operations-04-sftp-operations.md) |
-| **Read Files** | ✅ | ✅ | Low | [Module 2](./modules/file-operations-02-quick-start.md) |
-| **Write Files** | ✅ | ✅ | Low | [Module 2](./modules/file-operations-02-quick-start.md) |
-| **Move Files** | ✅ | ✅ | Medium | [Module 6](./modules/file-operations-06-archive-patterns.md) |
-| **Copy Files** | ✅ | ✅ | Medium | [Module 6](./modules/file-operations-06-archive-patterns.md) |
-| **Delete Files** | ✅ | ✅ | Low | [Module 3](./modules/file-operations-03-s3-operations.md), [Module 4](./modules/file-operations-04-sftp-operations.md) |
-| **Process Large Files** | ✅ | ✅ | Advanced | [Module 5](./modules/file-operations-05-streaming-performance.md) |
-| **Archive Files** | ✅ | ✅ | Medium | [Module 6](./modules/file-operations-06-archive-patterns.md) |
-
-## 📖 Quick Reference
-
-For quick lookups, see:
-- [Quick Reference Cheat Sheet](./file-operations-quick-reference.md) - All operations on one page
-- [Common Patterns](./examples/common-patterns.ts) - Copy-paste ready snippets
-- [Module 6: Archive Patterns](./modules/file-operations-06-archive-patterns.md) - Date-based archiving examples
-- [Troubleshooting Guide](./modules/file-operations-07-testing-troubleshooting.md) - Error messages and fixes
-
-## 🎓 Prerequisites
-
-Before starting this guide, you should:
-
-1. ✅ Have the SDK installed: `npm install @fluentcommerce/fc-connect-sdk`
-2. ✅ Have credentials for your storage system (S3 keys or SFTP credentials)
-3. ✅ Understand basic file system concepts (paths, directories, permissions)
-4. ✅ Be familiar with async/await in TypeScript/JavaScript
-
-**New to the SDK?** Start with the [Getting Started Guide](../../00-START-HERE/getting-started.md) first.
-
-## 🔍 Key Concepts at a Glance
-
-### Unified Data Source Interface
-
-```typescript
-// ❌ WITHOUT SDK: Different APIs for each storage system
-import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';
-import * as SftpClient from 'ssh2-sftp-client';
-
-// S3 - complex AWS SDK
-const s3 = new S3Client({});
-const s3Result = await s3.send(new GetObjectCommand({ Bucket: 'x', Key: 'y' }));
-
-// SFTP - completely different API
-const sftp = new SftpClient();
-await sftp.connect({ host: 'x', username: 'y', password: 'z' });
-const sftpResult = await sftp.get('/path/to/file');
-
-// ✅ WITH SDK: Same interface for all storage systems
-import { S3DataSource, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-
-const s3Source = new S3DataSource({
-  type: 'S3_CSV',
-  connectionId: 's3-readme',
-  name: 'S3 README Example',
-  s3Config: { region: 'us-east-1', bucket: 'my-bucket', accessKeyId: 'x', secretAccessKey: 'y' }
-}, logger);
-
-const sftpSource = new SftpDataSource({
-  type: 'SFTP_CSV',
-  connectionId: 'sftp-readme',
-  name: 'SFTP README Example',
-  settings: { host: 'x', username: 'y', password: 'z' }
-}, logger);
-
-// Same method signature!
-const s3File = await s3Source.downloadFile('path/to/file');
-const sftpFile = await sftpSource.downloadFile('path/to/file');
-```
-
-### Archive Workflow Pattern
-
-```typescript
-// Read → Process → Archive pattern
-const files = await dataSource.listFiles();
-
-for (const file of files) {
-  // 1. Read file
-  const content = await dataSource.downloadFile(file.path);
-
-  // 2. Process data
-  await processData(content);
-
-  // 3. Move to archive with date-based path
-  const archivePath = `/archive/${new Date().toISOString().split('T')[0]}/${file.name}`;
-  await dataSource.moveFile(file.path, archivePath, true);
-}
-```
-
-### Handling Large Files
-
-```typescript
-import { Buffer } from 'node:buffer';
-
-// ❌ WITHOUT chunked processing: Load entire file into memory (OOM risk!)
-const content = await dataSource.downloadFile('large-file.csv'); // 5GB loaded!
-
-// ✅ WITH chunked processing: Download as Buffer and process in chunks (constant memory)
-const buffer = await dataSource.downloadFile('large-file.csv', { asBuffer: true });
-
-const chunkSize = 1024 * 1024; // 1MB chunks
-for (let offset = 0; offset < buffer.length; offset += chunkSize) {
-  const chunk = buffer.slice(offset, offset + chunkSize);
-  processChunk(chunk.toString('utf8')); // Process 1MB at a time
-}
-```
-
-## 🛠️ Complete Example
-
-Here's a real-world multi-folder SFTP workflow with archiving:
-
-```typescript
-import { Buffer } from 'node:buffer';  // Required for Deno/Versori runtime!
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-
-const logger = createConsoleLogger();
-
-// Process multiple folders with archive
-const folders = ['orders', 'inventory', 'products'];
-
-for (const folder of folders) {
-  const sftp = new SftpDataSource({
-    type: 'SFTP_CSV',
-    connectionId: `sftp-${folder}`,
-    name: `SFTP ${folder} Folder`,
-    settings: {
-      host: process.env.SFTP_HOST!,
-      port: 22,
-      username: process.env.SFTP_USERNAME!,
-      password: process.env.SFTP_PASSWORD!
-    }
-  }, logger);
-
-  try {
-    // List all CSV files
-    const files = await sftp.listFiles();
-    logger.info(`Found ${files.length} files in ${folder}`);
-
-    for (const file of files) {
-      try {
-        // Download and process
-        const content = await sftp.downloadFile(file.path);
-        await processData(folder, content);
-
-        // Archive with date-based organization
-        const today = new Date().toISOString().split('T')[0];
-        const archivePath = `/archive/${folder}/${today}/${file.name}`;
-
-        await sftp.moveFile(file.path, archivePath, true);
-        logger.info(`Archived ${file.name} to ${archivePath}`);
-
-      } catch (error) {
-        logger.error(`Failed to process ${file.name}:`, error);
-        // Continue with next file
-      }
-    }
-  } finally {
-    // Clean up connection
-    await sftp.dispose();
-  }
-}
-
-async function processData(type: string, content: string): Promise<void> {
-  // Your business logic here
-  console.log(`Processing ${type} data: ${content.length} bytes`);
-}
-```
-
-## 📂 Repository Structure
-
-```
-docs/guides/file-operations/
-├── readme.md                           # This file - main index
-├── quick-reference.md                  # Cheat sheet for quick lookups
-│
-├── modules/                            # Learning modules
-│   ├── 01-foundations.md               # Data sources and patterns
-│   ├── 02-quick-start.md               # First working examples
-│   ├── 03-s3-operations.md             # Complete S3 guide
-│   ├── 04-sftp-operations.md           # Complete SFTP guide
-│   ├── 05-streaming-performance.md     # Large file handling
-│   ├── 06-archive-patterns.md          # Archive workflows
-│   ├── 07-testing-troubleshooting.md   # Testing and debugging
-│   └── 08-api-reference.md             # Complete API docs
-│
-└── examples/                           # Verified code examples
-    ├── readme.md                       # Examples index
-    ├── common-patterns.ts              # Copy-paste ready patterns
-    ├── s3-basic-operations.ts          # S3 basics
-    ├── sftp-basic-operations.ts        # SFTP basics
-    ├── list-files.ts                   # List files
-    ├── read-files.ts                   # Read operations
-    ├── write-files.ts                  # Write operations
-    ├── move-files.ts                   # Move operations
-    ├── copy-files.ts                   # Copy operations
-    ├── delete-files.ts                 # Delete operations
-    ├── stream-large-files.ts           # Streaming
-    ├── archive-workflow.ts             # Archive pattern
-    └── archive-patterns.ts             # Advanced archive patterns
-```
-
-## 🔗 Related Documentation
-
-- [Data Sources Architecture](../../04-REFERENCE/architecture/readme.md) - Understanding data source design
-- [Universal Mapping Guide](../../02-CORE-GUIDES/mapping/) - Transform file data
-- [Auto-Pagination Guide](../../02-CORE-GUIDES/auto-pagination/) - Fetch large datasets
-- [Integration Patterns](../integration-patterns/) - Common integration workflows
-
-## 🚀 Advanced Features
-
-### SFTP Connection Pooling
-
-The SDK automatically manages SFTP connection pooling for better performance:
-
-```typescript
-const sftp = new SftpDataSource({
-  type: 'SFTP_CSV',
-  connectionId: 'sftp-pooled',
-  name: 'SFTP with Connection Pool',
-  settings: {
-    host: 'sftp.example.com',
-    username: 'user',
-    password: 'pass',
-    maxConnections: 5,  // Pool size (default: 5)
-    connectionWaitTimeout: 30000  // Wait timeout in ms (default: 30000)
-  }
-}, logger);
-
-// Connections are automatically pooled and reused
-// Wait queue handles concurrent requests exceeding pool size
-```
-
-**Connection Pool Features:**
-- Automatic connection reuse for better performance
-- Configurable pool size via `maxConnections` (default: 5)
-- Wait queue for requests when pool is exhausted
-- Health checks before connection reuse
-- Automatic cleanup on `dispose()`
-
-### Retry Logic with Exponential Backoff
-
-Both S3 and SFTP support automatic retries with exponential backoff:
-
-**SFTP Retry Configuration:**
-```typescript
-const sftp = new SftpDataSource({
-  type: 'SFTP_CSV',
-  connectionId: 'sftp-retry',
-  name: 'SFTP with Retry',
-  settings: {
-    host: 'sftp.example.com',
-    username: 'user',
-    password: 'pass',
-    retry: {
-      maxAttempts: 3,        // Default: 3
-      baseDelayMs: 1000,     // Default: 1000ms
-      backoffFactor: 2,      // Default: 2 (exponential)
-      maxDelayMs: 8000,      // Default: 8000ms
-      jitter: 'full',        // Default: 'full' (adds randomness)
-      reconnectOnFailure: true  // Default: true
-    }
-  }
-}, logger);
-```
-
-**S3 Retry Configuration:**
-```typescript
-const s3 = new S3DataSource({
-  type: 'S3_CSV',
-  connectionId: 's3-retry',
-  name: 'S3 with Retry',
-  s3Config: {
-    region: 'us-east-1',
-    bucket: 'my-bucket',
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    maxAttempts: 3  // AWS SDK retry config
-  }
-}, logger);
-```
-
-**Retry Features:**
-- Exponential backoff: delay doubles on each retry (1s → 2s → 4s)
-- Jitter: adds randomness to prevent thundering herd
-- Automatic reconnection on connection failures (SFTP)
-- Retryable error detection (network errors, timeouts)
-- Detailed retry logging for debugging
-
-## 💡 Tips for Best Results
-
-1. **Always dispose connections** - Call `dispose()` in `finally` blocks for SFTP
-2. **Use chunked processing for large files** - Files >100MB should use `downloadFile()` with `encoding: 'binary'` (S3) or `asBuffer: true` (SFTP) and process in chunks
-3. **Configure retry logic** - Adjust retry settings based on your network reliability
-4. **Tune connection pool** - Increase `maxConnections` for high-concurrency workflows
-5. **Test with small datasets first** - Validate your workflow before processing thousands of files
-6. **Monitor memory usage** - Use streaming or batch processing for large-scale operations
-7. **Archive processed files** - Never delete source files immediately; archive for audit trails
-
-## ✅ Feature Support Matrix
-
-| Feature | S3 | SFTP | Notes |
-|---------|----|----|-------|
-| **Read Files** | ✅ | ✅ | Both tested |
-| **Write Files** | ✅ | ✅ | Both tested |
-| **Move Files** | ✅ | ✅ | S3 uses copy + delete |
-| **Copy Files** | ✅ | ✅ | Both tested |
-| **Delete Files** | ✅ | ✅ | Both tested |
-| **List Files** | ✅ | ✅ | Both support patterns |
-| **File Exists** | ✅ | ✅ | Both tested |
-| **Streaming** | ✅ | ✅ | For large files |
-| **Connection Pooling** | N/A | ✅ | SFTP only |
-| **Presigned URLs** | ✅ | N/A | S3 only |
-| **Unit Tests** | ✅ | ✅ | Comprehensive |
-| **Integration Tests** | ⚠️ | ✅ | SFTP has integration tests |
-
-## 🚦 Getting Started
-
-Ready to begin? Start with [Module 1: Foundations →](./modules/file-operations-01-foundations.md)
-
----
-
-**Need Help?**
-- 📖 Browse [Examples](./examples/)
-- 🔍 Check [Troubleshooting](./modules/file-operations-07-testing-troubleshooting.md)
-- 📧 Contact support@fluentcommerce.com
-- 🐛 Report issues at [GitHub](https://github.com/fluentcommerce/fc-connect-sdk/issues)
+# File Operations Learning Path
+
+> **Complete guide to file operations in the Fluent Connect SDK**
+> Progressive learning modules with S3, SFTP, streaming, and archive patterns
+
+## 📚 What You'll Learn
+
+This guide teaches you how to work with files across different storage systems using the SDK's unified data source abstractions. By the end, you'll understand:
+
+- ✅ How to read, write, move, copy, and delete files on S3 and SFTP
+- ✅ How to stream large files efficiently without memory issues
+- ✅ How to implement archive workflows with date-based organization
+- ✅ How to process files from multiple folders and storage systems
+- ✅ How to test file operations with comprehensive test coverage
+
+## 🎯 Learning Path Overview
+
+This guide is organized into **8 progressive modules**. Follow them in order for the best learning experience:
+
+| Module | Topic | Level |
+|--------|-------|-------|
+| [Module 1](./modules/file-operations-01-foundations.md) | **Foundations** - Data sources, abstraction patterns, when to use S3 vs SFTP | Beginner |
+| [Module 2](./modules/file-operations-02-quick-start.md) | **Quick Start** - First working examples for S3 and SFTP | Beginner |
+| [Module 3](./modules/file-operations-03-s3-operations.md) | **S3 Operations** - Complete S3 file operations guide | Intermediate |
+| [Module 4](./modules/file-operations-04-sftp-operations.md) | **SFTP Operations** - Complete SFTP file operations guide | Intermediate |
+| [Module 5](./modules/file-operations-05-streaming-performance.md) | **Performance Optimization** - Handle large files efficiently | Advanced |
+| [Module 6](./modules/file-operations-06-archive-patterns.md) | **Archive Patterns** - Date-based archiving, multi-folder workflows | Intermediate |
+| [Module 7](./modules/file-operations-07-testing-troubleshooting.md) | **Testing & Troubleshooting** - Test coverage, common issues, debugging | Intermediate |
+| [Module 8](./modules/file-operations-08-api-reference.md) | **API Reference** - Complete method signatures and interfaces | Reference |
+
+## 🚀 Quick Navigation
+
+### By Use Case
+
+- **First time with file operations?** → Start with [Module 1: Foundations](./modules/file-operations-01-foundations.md)
+- **Need a working example now?** → Jump to [Module 2: Quick Start](./modules/file-operations-02-quick-start.md)
+- **Working with S3?** → See [Module 3: S3 Operations](./modules/file-operations-03-s3-operations.md)
+- **Working with SFTP?** → See [Module 4: SFTP Operations](./modules/file-operations-04-sftp-operations.md)
+- **Large file issues?** → Check [Module 5: Streaming & Performance](./modules/file-operations-05-streaming-performance.md)
+- **Need archive workflows?** → See [Module 6: Archive Patterns](./modules/file-operations-06-archive-patterns.md)
+- **Experiencing errors?** → Browse [Module 7: Troubleshooting](./modules/file-operations-07-testing-troubleshooting.md)
+- **Want code examples?** → Browse [Examples Directory](./examples/)
+
+### By Storage System
+
+| Storage | Complexity | Best For | See Module |
+|---------|------------|----------|---------|
+| **S3** | Low | Cloud storage, large datasets, serverless | [Module 3: S3 Operations](./modules/file-operations-03-s3-operations.md) |
+| **SFTP** | Medium | Enterprise systems, B2B integrations, secure transfers | [Module 4: SFTP Operations](./modules/file-operations-04-sftp-operations.md) |
+| **Local Files** | Low | Development, testing, small scripts | [Module 2: Quick Start](./modules/file-operations-02-quick-start.md) |
+
+### By Operation Type
+
+| Operation | S3 | SFTP | Complexity | See Module |
+|-----------|----|----|------------|---------|
+| **List Files** | ✅ | ✅ | Low | [Module 3](./modules/file-operations-03-s3-operations.md), [Module 4](./modules/file-operations-04-sftp-operations.md) |
+| **Read Files** | ✅ | ✅ | Low | [Module 2](./modules/file-operations-02-quick-start.md) |
+| **Write Files** | ✅ | ✅ | Low | [Module 2](./modules/file-operations-02-quick-start.md) |
+| **Move Files** | ✅ | ✅ | Medium | [Module 6](./modules/file-operations-06-archive-patterns.md) |
+| **Copy Files** | ✅ | ✅ | Medium | [Module 6](./modules/file-operations-06-archive-patterns.md) |
+| **Delete Files** | ✅ | ✅ | Low | [Module 3](./modules/file-operations-03-s3-operations.md), [Module 4](./modules/file-operations-04-sftp-operations.md) |
+| **Process Large Files** | ✅ | ✅ | Advanced | [Module 5](./modules/file-operations-05-streaming-performance.md) |
+| **Archive Files** | ✅ | ✅ | Medium | [Module 6](./modules/file-operations-06-archive-patterns.md) |
+
+## 📖 Quick Reference
+
+For quick lookups, see:
+- [Quick Reference Cheat Sheet](./file-operations-quick-reference.md) - All operations on one page
+- [Common Patterns](./examples/common-patterns.ts) - Copy-paste ready snippets
+- [Module 6: Archive Patterns](./modules/file-operations-06-archive-patterns.md) - Date-based archiving examples
+- [Troubleshooting Guide](./modules/file-operations-07-testing-troubleshooting.md) - Error messages and fixes
+
+## 🎓 Prerequisites
+
+Before starting this guide, you should:
+
+1. ✅ Have the SDK installed: `npm install @fluentcommerce/fc-connect-sdk`
+2. ✅ Have credentials for your storage system (S3 keys or SFTP credentials)
+3. ✅ Understand basic file system concepts (paths, directories, permissions)
+4. ✅ Be familiar with async/await in TypeScript/JavaScript
+
+**New to the SDK?** Start with the [Getting Started Guide](../../00-START-HERE/getting-started.md) first.
+
+## 🔍 Key Concepts at a Glance
+
+### Unified Data Source Interface
+
+```typescript
+// ❌ WITHOUT SDK: Different APIs for each storage system
+import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';
+import * as SftpClient from 'ssh2-sftp-client';
+
+// S3 - complex AWS SDK
+const s3 = new S3Client({});
+const s3Result = await s3.send(new GetObjectCommand({ Bucket: 'x', Key: 'y' }));
+
+// SFTP - completely different API
+const sftp = new SftpClient();
+await sftp.connect({ host: 'x', username: 'y', password: 'z' });
+const sftpResult = await sftp.get('/path/to/file');
+
+// ✅ WITH SDK: Same interface for all storage systems
+import { S3DataSource, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+
+const s3Source = new S3DataSource({
+  type: 'S3_CSV',
+  connectionId: 's3-readme',
+  name: 'S3 README Example',
+  s3Config: { region: 'us-east-1', bucket: 'my-bucket', accessKeyId: 'x', secretAccessKey: 'y' }
+}, logger);
+
+const sftpSource = new SftpDataSource({
+  type: 'SFTP_CSV',
+  connectionId: 'sftp-readme',
+  name: 'SFTP README Example',
+  settings: { host: 'x', username: 'y', password: 'z' }
+}, logger);
+
+// Same method signature!
+const s3File = await s3Source.downloadFile('path/to/file');
+const sftpFile = await sftpSource.downloadFile('path/to/file');
+```
+
+### Archive Workflow Pattern
+
+```typescript
+// Read → Process → Archive pattern
+const files = await dataSource.listFiles();
+
+for (const file of files) {
+  // 1. Read file
+  const content = await dataSource.downloadFile(file.path);
+
+  // 2. Process data
+  await processData(content);
+
+  // 3. Move to archive with date-based path
+  const archivePath = `/archive/${new Date().toISOString().split('T')[0]}/${file.name}`;
+  await dataSource.moveFile(file.path, archivePath, true);
+}
+```
+
+### Handling Large Files
+
+```typescript
+import { Buffer } from 'node:buffer';
+
+// ❌ WITHOUT chunked processing: Load entire file into memory (OOM risk!)
+const content = await dataSource.downloadFile('large-file.csv'); // 5GB loaded!
+
+// ✅ WITH chunked processing: Download as Buffer and process in chunks (constant memory)
+const buffer = await dataSource.downloadFile('large-file.csv', { asBuffer: true });
+
+const chunkSize = 1024 * 1024; // 1MB chunks
+for (let offset = 0; offset < buffer.length; offset += chunkSize) {
+  const chunk = buffer.slice(offset, offset + chunkSize);
+  processChunk(chunk.toString('utf8')); // Process 1MB at a time
+}
+```
+
+## 🛠️ Complete Example
+
+Here's a real-world multi-folder SFTP workflow with archiving:
+
+```typescript
+import { Buffer } from 'node:buffer';  // Required for Deno/Versori runtime!
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+
+const logger = createConsoleLogger();
+
+// Process multiple folders with archive
+const folders = ['orders', 'inventory', 'products'];
+
+for (const folder of folders) {
+  const sftp = new SftpDataSource({
+    type: 'SFTP_CSV',
+    connectionId: `sftp-${folder}`,
+    name: `SFTP ${folder} Folder`,
+    settings: {
+      host: process.env.SFTP_HOST!,
+      port: 22,
+      username: process.env.SFTP_USERNAME!,
+      password: process.env.SFTP_PASSWORD!
+    }
+  }, logger);
+
+  try {
+    // List all CSV files
+    const files = await sftp.listFiles();
+    logger.info(`Found ${files.length} files in ${folder}`);
+
+    for (const file of files) {
+      try {
+        // Download and process
+        const content = await sftp.downloadFile(file.path);
+        await processData(folder, content);
+
+        // Archive with date-based organization
+        const today = new Date().toISOString().split('T')[0];
+        const archivePath = `/archive/${folder}/${today}/${file.name}`;
+
+        await sftp.moveFile(file.path, archivePath, true);
+        logger.info(`Archived ${file.name} to ${archivePath}`);
+
+      } catch (error) {
+        logger.error(`Failed to process ${file.name}:`, error);
+        // Continue with next file
+      }
+    }
+  } finally {
+    // Clean up connection
+    await sftp.dispose();
+  }
+}
+
+async function processData(type: string, content: string): Promise<void> {
+  // Your business logic here
+  console.log(`Processing ${type} data: ${content.length} bytes`);
+}
+```
+
+## 📂 Repository Structure
+
+```
+docs/guides/file-operations/
+├── readme.md                           # This file - main index
+├── quick-reference.md                  # Cheat sheet for quick lookups
+│
+├── modules/                            # Learning modules
+│   ├── 01-foundations.md               # Data sources and patterns
+│   ├── 02-quick-start.md               # First working examples
+│   ├── 03-s3-operations.md             # Complete S3 guide
+│   ├── 04-sftp-operations.md           # Complete SFTP guide
+│   ├── 05-streaming-performance.md     # Large file handling
+│   ├── 06-archive-patterns.md          # Archive workflows
+│   ├── 07-testing-troubleshooting.md   # Testing and debugging
+│   └── 08-api-reference.md             # Complete API docs
+│
+└── examples/                           # Verified code examples
+    ├── readme.md                       # Examples index
+    ├── common-patterns.ts              # Copy-paste ready patterns
+    ├── s3-basic-operations.ts          # S3 basics
+    ├── sftp-basic-operations.ts        # SFTP basics
+    ├── list-files.ts                   # List files
+    ├── read-files.ts                   # Read operations
+    ├── write-files.ts                  # Write operations
+    ├── move-files.ts                   # Move operations
+    ├── copy-files.ts                   # Copy operations
+    ├── delete-files.ts                 # Delete operations
+    ├── stream-large-files.ts           # Streaming
+    ├── archive-workflow.ts             # Archive pattern
+    └── archive-patterns.ts             # Advanced archive patterns
+```
+
+## 🔗 Related Documentation
+
+- [Data Sources Architecture](../../04-REFERENCE/architecture/readme.md) - Understanding data source design
+- [Universal Mapping Guide](../../02-CORE-GUIDES/mapping/) - Transform file data
+- [Auto-Pagination Guide](../../02-CORE-GUIDES/auto-pagination/) - Fetch large datasets
+- [Integration Patterns](../integration-patterns/) - Common integration workflows
+
+## 🚀 Advanced Features
+
+### SFTP Connection Pooling
+
+The SDK automatically manages SFTP connection pooling for better performance:
+
+```typescript
+const sftp = new SftpDataSource({
+  type: 'SFTP_CSV',
+  connectionId: 'sftp-pooled',
+  name: 'SFTP with Connection Pool',
+  settings: {
+    host: 'sftp.example.com',
+    username: 'user',
+    password: 'pass',
+    maxConnections: 5,  // Pool size (default: 5)
+    connectionWaitTimeout: 30000  // Wait timeout in ms (default: 30000)
+  }
+}, logger);
+
+// Connections are automatically pooled and reused
+// Wait queue handles concurrent requests exceeding pool size
+```
+
+**Connection Pool Features:**
+- Automatic connection reuse for better performance
+- Configurable pool size via `maxConnections` (default: 5)
+- Wait queue for requests when pool is exhausted
+- Health checks before connection reuse
+- Automatic cleanup on `dispose()`
+
+### Retry Logic with Exponential Backoff
+
+Both S3 and SFTP support automatic retries with exponential backoff:
+
+**SFTP Retry Configuration:**
+```typescript
+const sftp = new SftpDataSource({
+  type: 'SFTP_CSV',
+  connectionId: 'sftp-retry',
+  name: 'SFTP with Retry',
+  settings: {
+    host: 'sftp.example.com',
+    username: 'user',
+    password: 'pass',
+    retry: {
+      maxAttempts: 3,        // Default: 3
+      baseDelayMs: 1000,     // Default: 1000ms
+      backoffFactor: 2,      // Default: 2 (exponential)
+      maxDelayMs: 8000,      // Default: 8000ms
+      jitter: 'full',        // Default: 'full' (adds randomness)
+      reconnectOnFailure: true  // Default: true
+    }
+  }
+}, logger);
+```
+
+**S3 Retry Configuration:**
+```typescript
+const s3 = new S3DataSource({
+  type: 'S3_CSV',
+  connectionId: 's3-retry',
+  name: 'S3 with Retry',
+  s3Config: {
+    region: 'us-east-1',
+    bucket: 'my-bucket',
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    maxAttempts: 3  // AWS SDK retry config
+  }
+}, logger);
+```
+
+**Retry Features:**
+- Exponential backoff: delay doubles on each retry (1s → 2s → 4s)
+- Jitter: adds randomness to prevent thundering herd
+- Automatic reconnection on connection failures (SFTP)
+- Retryable error detection (network errors, timeouts)
+- Detailed retry logging for debugging
+
+## 💡 Tips for Best Results
+
+1. **Always dispose connections** - Call `dispose()` in `finally` blocks for SFTP
+2. **Use chunked processing for large files** - Files >100MB should use `downloadFile()` with `encoding: 'binary'` (S3) or `asBuffer: true` (SFTP) and process in chunks
+3. **Configure retry logic** - Adjust retry settings based on your network reliability
+4. **Tune connection pool** - Increase `maxConnections` for high-concurrency workflows
+5. **Test with small datasets first** - Validate your workflow before processing thousands of files
+6. **Monitor memory usage** - Use streaming or batch processing for large-scale operations
+7. **Archive processed files** - Never delete source files immediately; archive for audit trails
+
+## ✅ Feature Support Matrix
+
+| Feature | S3 | SFTP | Notes |
+|---------|----|----|-------|
+| **Read Files** | ✅ | ✅ | Both tested |
+| **Write Files** | ✅ | ✅ | Both tested |
+| **Move Files** | ✅ | ✅ | S3 uses copy + delete |
+| **Copy Files** | ✅ | ✅ | Both tested |
+| **Delete Files** | ✅ | ✅ | Both tested |
+| **List Files** | ✅ | ✅ | Both support patterns |
+| **File Exists** | ✅ | ✅ | Both tested |
+| **Streaming** | ✅ | ✅ | For large files |
+| **Connection Pooling** | N/A | ✅ | SFTP only |
+| **Presigned URLs** | ✅ | N/A | S3 only |
+| **Unit Tests** | ✅ | ✅ | Comprehensive |
+| **Integration Tests** | ⚠️ | ✅ | SFTP has integration tests |
+
+## 🚦 Getting Started
+
+Ready to begin? Start with [Module 1: Foundations →](./modules/file-operations-01-foundations.md)
+
+---
+
+**Need Help?**
+- 📖 Browse [Examples](./examples/)
+- 🔍 Check [Troubleshooting](./modules/file-operations-07-testing-troubleshooting.md)
+- 📧 Contact support@fluentcommerce.com
+- 🐛 Report issues at [GitHub](https://github.com/fluentcommerce/fc-connect-sdk/issues)