@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/02-CORE-GUIDES/data-sources/data-sources-readme.md

@@ -1,276 +1,276 @@
-# Data Sources Guide
-
-**Unified File Access for S3 and SFTP**
-
-The FC Connect SDK provides powerful data source abstractions for accessing files from S3 and SFTP servers. Whether you're ingesting inventory from a vendor's SFTP server or extracting data to S3, the SDK handles connection management, streaming, error recovery, and security for you.
-
-## Quick Start
-
-```typescript
-import { S3DataSource, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-
-// S3 data source (uses presigned URLs - works everywhere)
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    connectionId: 's3-source',
-    name: 'S3 Source',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-      bucket: 'my-inventory-bucket',
-    },
-  },
-  logger
-);
-
-// SFTP data source (direct SSH connection)
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor-sftp',
-    settings: {
-      host: 'sftp.vendor.com',
-      username: 'integration',
-      privateKey: process.env.SFTP_KEY,
-    },
-  },
-  logger
-);
-```
-
-**That's it.** Both data sources provide the same interface for file operations, making it easy to switch between S3 and SFTP.
-
----
-
-## Learning Path
-
-This guide is organized into progressive modules. Follow the learning path below or jump to specific topics.
-
-| Module                                      | Topic                                                                 | Level        | Estimated Time |
-| ------------------------------------------- | --------------------------------------------------------------------- | ------------ | -------------- |
-| [Module 1](../auto-pagination/modules/auto-pagination-01-foundations.md)     | **Foundations** - S3 vs SFTP, when to use each, architecture          | Beginner     | 20 minutes     |
-| [Module 2](./modules/data-sources-02-s3-operations.md)   | **S3 Operations** - S3DataSource complete guide, presigned URLs       | Intermediate | 45 minutes     |
-| [Module 3](./modules/data-sources-03-sftp-operations.md) | **SFTP Operations** - SftpDataSource complete guide, SSH auth         | Intermediate | 45 minutes     |
-| [Module 4](./modules/data-sources-04-file-patterns.md)   | **File Processing** - Streaming, batching, error handling, validation | Advanced     | 40 minutes     |
-| [Module 5](./modules/data-sources-05-advanced-topics.md) | **Advanced Topics** - Dual credentials, security, monitoring, testing | Advanced     | 30 minutes     |
-
-**Total learning time:** ~3 hours for complete mastery
-
----
-
-## Quick Reference
-
-Need a cheat sheet? See [Quick Reference](../advanced-services/advanced-services-quick-reference.md) for one-page summary of all data source patterns.
-
----
-
-## Module Descriptions
-
-### Module 1: Foundations
-
-Learn the fundamental concepts:
-
-- Understanding S3 vs SFTP (when to use each)
-- Data source architecture in the SDK
-- Configuration patterns
-- Universal interface design
-
-**Prerequisites:** Basic TypeScript knowledge
-**Outcomes:** Understand when to use S3 vs SFTP, basic configuration
-
-### Module 2: S3 Operations
-
-Master S3 data source:
-
-- S3DataSource complete API reference
-- Presigned URL architecture
-- Reading and writing files
-- Streaming large files
-- Dual AWS credentials (source + target)
-- Performance optimization
-
-**Prerequisites:** Module 1
-**Outcomes:** Build production S3 integrations
-
-### Module 3: SFTP Operations
-
-Master SFTP data source:
-
-- SftpDataSource complete API reference
-- SSH authentication (password and key-based)
-- Connection pooling and management
-- File operations (upload, download, list, delete, move)
-- Error recovery patterns
-- Security best practices
-
-**Prerequisites:** Module 1
-**Outcomes:** Build production SFTP integrations
-
-### Module 4: File Processing Patterns
-
-Process files efficiently:
-
-- Streaming vs buffering strategies
-- Batch processing patterns
-- CSV, JSON, JSONL, Parquet support
-- State management (prevent duplicates)
-- Error handling and recovery
-- Validation strategies
-
-**Prerequisites:** Modules 2-3
-**Outcomes:** Handle any file processing scenario
-
-### Module 5: Advanced Topics
-
-Master advanced patterns:
-
-- Dual AWS credentials (source + target accounts)
-- Presigned URL security
-- Connection pooling optimization
-- Monitoring and observability
-- Testing strategies
-- Troubleshooting common issues
-
-**Prerequisites:** Modules 1-4
-**Outcomes:** Production-ready implementations with security and monitoring
-
----
-
-## Common Use Cases - Quick Links
-
-**I want to...**
-
-- **Read inventory CSV from S3** → [Module 2: Reading Files](./modules/data-sources-02-s3-operations.md#reading-files)
-- **Upload extraction results to S3** → [Module 2: Writing Files](./modules/data-sources-02-s3-operations.md#writing-files)
-- **Connect to vendor SFTP server** → [Module 3: SSH Authentication](./modules/data-sources-03-sftp-operations.md#authentication)
-- **Stream large Parquet files** → [Module 4: Streaming Patterns](./modules/data-sources-04-file-patterns.md#streaming-large-files)
-- **Prevent duplicate file processing** → [Module 4: State Management](./modules/data-sources-04-file-patterns.md#state-management)
-- **Handle connection failures** → [Module 4: Error Recovery](./modules/data-sources-04-file-patterns.md#error-recovery)
-- **Use separate AWS accounts** → [Module 5: Dual Credentials](./modules/data-sources-05-advanced-topics.md#dual-aws-credentials)
-- **Test SFTP connections** → [Module 5: Testing](./modules/data-sources-05-advanced-topics.md#testing-strategies)
-
----
-
-## Key Concepts
-
-### Universal Interface
-
-Both `S3DataSource` and `SftpDataSource` implement the same base interface, making it easy to switch between storage backends:
-
-```typescript
-interface DataSource {
-  downloadFile(key: string): Promise<string | Buffer>;
-  writeFile(key: string, content: string): Promise<void>;
-  listFiles(prefix?: string): Promise<string[]>;
-  fileExists(key: string): Promise<boolean>;
-  deleteFile(key: string): Promise<void>;
-  streamFile(key: string): Promise<ReadableStream>;
-}
-```
-
-### Presigned URL Architecture (S3)
-
-`S3DataSource` uses **presigned URLs exclusively** for universal compatibility:
-
-- Works in Node.js, Deno, and Versori
-- No AWS SDK bundling required
-- Simple `fetch()` API for all operations
-- Automatic signature generation via `S3PresignService`
-
-### Direct SSH Connection (SFTP)
-
-`SftpDataSource` uses **direct SSH connections** via `ssh2-sftp-client`:
-
-- No platform dependencies
-- Password or private key authentication
-- Connection pooling for efficiency
-- Automatic retry with exponential backoff
-
----
-
-## Examples
-
-Complete working examples are available in the [`examples/`](./examples/) directory:
-
-- `s3-csv-ingestion.ts` - Read CSV from S3, process, send to Fluent
-- `s3-parquet-extraction.ts` - Extract Fluent data to S3 as Parquet
-- `sftp-inventory-sync.ts` - Read inventory from SFTP, sync to Fluent
-- `dual-credentials.ts` - Use separate AWS accounts for source/target
-- `streaming-large-files.ts` - Stream processing for large datasets
-
----
-
-## Best Practices
-
-1. **Use streaming for large files** (>10MB) to avoid memory issues (Module 4)
-2. **Implement state management** to prevent duplicate processing (Module 4)
-3. **Use separate AWS credentials** for source/target accounts (Module 5)
-4. **Enable connection pooling** for SFTP to reduce overhead (Module 3)
-5. **Validate files before processing** to catch errors early (Module 4)
-6. **Use private key auth** for SFTP instead of passwords (Module 3)
-7. **Monitor presigned URL expiration** for long-running jobs (Module 5)
-
----
-
-## Architecture Comparison
-
-| Feature                | S3DataSource                   | SftpDataSource                      |
-| ---------------------- | ------------------------------ | ----------------------------------- |
-| **Transport**          | HTTPS (presigned URLs)         | SSH (sftp-client)                   |
-| **Authentication**     | AWS credentials                | Password or private key             |
-| **Platform Support**   | Node.js, Deno, Versori         | Node.js, Deno, Versori              |
-| **Dependencies**       | None (uses fetch)              | ssh2-sftp-client                    |
-| **Connection Pooling** | N/A (stateless HTTP)           | Yes (configurable)                  |
-| **Streaming**          | Yes (native fetch)             | Yes (sftp streams)                  |
-| **Best For**           | Cloud-native, scalable storage | Vendor integrations, enterprise systems |
-
----
-
-## When to Use Each Data Source
-
-### Use S3 When:
-
-- ✅ Building cloud-native integrations
-- ✅ Need scalable, durable storage
-- ✅ Working with large datasets (GB+)
-- ✅ Deploying to Versori or serverless platforms
-- ✅ Need global distribution (CloudFront)
-
-### Use SFTP When:
-
-- ✅ Integrating with vendor/partner systems
-- ✅ Working with enterprise systems
-- ✅ Required by compliance/security policies
-- ✅ Vendor provides SFTP-only access
-- ✅ Need real-time file monitoring
-
-See [Module 1: Decision Guide](../auto-pagination/modules/auto-pagination-01-foundations.md#decision-guide) for detailed comparison.
-
----
-
-## Getting Help
-
-- **Quick answers:** [Quick Reference](../advanced-services/advanced-services-quick-reference.md)
-- **Error Handling:** [File Operations Error Handling Guide](./data-sources-file-operations-error-handling.md) - Comprehensive guide for connection, discovery, reading, and parsing errors
-- **Troubleshooting:** [Module 5: Troubleshooting](./modules/data-sources-05-advanced-topics.md#troubleshooting)
-- **Examples:** [`examples/`](./examples/)
-- **API Reference:** [Module 2: S3 API](./modules/data-sources-02-s3-operations.md#api-reference), [Module 3: SFTP API](./modules/data-sources-03-sftp-operations.md#api-reference)
-
----
-
-## Next Steps
-
-**New to data sources?** Start with [Module 1: Foundations](../auto-pagination/modules/auto-pagination-01-foundations.md)
-
-**Already familiar?** Jump to specific topics:
-
-- [S3 Operations](./modules/data-sources-02-s3-operations.md)
-- [SFTP Operations](./modules/data-sources-03-sftp-operations.md)
-- [File Processing Patterns](./modules/data-sources-04-file-patterns.md)
-- [Advanced Topics](./modules/data-sources-05-advanced-topics.md)
-
----
-
-**Remember:** ONE interface for BOTH S3 and SFTP. Learn the pattern once, use it everywhere.
+# Data Sources Guide
+
+**Unified File Access for S3 and SFTP**
+
+The FC Connect SDK provides powerful data source abstractions for accessing files from S3 and SFTP servers. Whether you're ingesting inventory from a vendor's SFTP server or extracting data to S3, the SDK handles connection management, streaming, error recovery, and security for you.
+
+## Quick Start
+
+```typescript
+import { S3DataSource, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+
+// S3 data source (uses presigned URLs - works everywhere)
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    connectionId: 's3-source',
+    name: 'S3 Source',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+      bucket: 'my-inventory-bucket',
+    },
+  },
+  logger
+);
+
+// SFTP data source (direct SSH connection)
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor-sftp',
+    settings: {
+      host: 'sftp.vendor.com',
+      username: 'integration',
+      privateKey: process.env.SFTP_KEY,
+    },
+  },
+  logger
+);
+```
+
+**That's it.** Both data sources provide the same interface for file operations, making it easy to switch between S3 and SFTP.
+
+---
+
+## Learning Path
+
+This guide is organized into progressive modules. Follow the learning path below or jump to specific topics.
+
+| Module                                      | Topic                                                                 | Level        | Estimated Time |
+| ------------------------------------------- | --------------------------------------------------------------------- | ------------ | -------------- |
+| [Module 1](../auto-pagination/modules/auto-pagination-01-foundations.md)     | **Foundations** - S3 vs SFTP, when to use each, architecture          | Beginner     | 20 minutes     |
+| [Module 2](./modules/data-sources-02-s3-operations.md)   | **S3 Operations** - S3DataSource complete guide, presigned URLs       | Intermediate | 45 minutes     |
+| [Module 3](./modules/data-sources-03-sftp-operations.md) | **SFTP Operations** - SftpDataSource complete guide, SSH auth         | Intermediate | 45 minutes     |
+| [Module 4](./modules/data-sources-04-file-patterns.md)   | **File Processing** - Streaming, batching, error handling, validation | Advanced     | 40 minutes     |
+| [Module 5](./modules/data-sources-05-advanced-topics.md) | **Advanced Topics** - Dual credentials, security, monitoring, testing | Advanced     | 30 minutes     |
+
+**Total learning time:** ~3 hours for complete mastery
+
+---
+
+## Quick Reference
+
+Need a cheat sheet? See [Quick Reference](../advanced-services/advanced-services-quick-reference.md) for one-page summary of all data source patterns.
+
+---
+
+## Module Descriptions
+
+### Module 1: Foundations
+
+Learn the fundamental concepts:
+
+- Understanding S3 vs SFTP (when to use each)
+- Data source architecture in the SDK
+- Configuration patterns
+- Universal interface design
+
+**Prerequisites:** Basic TypeScript knowledge
+**Outcomes:** Understand when to use S3 vs SFTP, basic configuration
+
+### Module 2: S3 Operations
+
+Master S3 data source:
+
+- S3DataSource complete API reference
+- Presigned URL architecture
+- Reading and writing files
+- Streaming large files
+- Dual AWS credentials (source + target)
+- Performance optimization
+
+**Prerequisites:** Module 1
+**Outcomes:** Build production S3 integrations
+
+### Module 3: SFTP Operations
+
+Master SFTP data source:
+
+- SftpDataSource complete API reference
+- SSH authentication (password and key-based)
+- Connection pooling and management
+- File operations (upload, download, list, delete, move)
+- Error recovery patterns
+- Security best practices
+
+**Prerequisites:** Module 1
+**Outcomes:** Build production SFTP integrations
+
+### Module 4: File Processing Patterns
+
+Process files efficiently:
+
+- Streaming vs buffering strategies
+- Batch processing patterns
+- CSV, JSON, JSONL, Parquet support
+- State management (prevent duplicates)
+- Error handling and recovery
+- Validation strategies
+
+**Prerequisites:** Modules 2-3
+**Outcomes:** Handle any file processing scenario
+
+### Module 5: Advanced Topics
+
+Master advanced patterns:
+
+- Dual AWS credentials (source + target accounts)
+- Presigned URL security
+- Connection pooling optimization
+- Monitoring and observability
+- Testing strategies
+- Troubleshooting common issues
+
+**Prerequisites:** Modules 1-4
+**Outcomes:** Production-ready implementations with security and monitoring
+
+---
+
+## Common Use Cases - Quick Links
+
+**I want to...**
+
+- **Read inventory CSV from S3** → [Module 2: Reading Files](./modules/data-sources-02-s3-operations.md#reading-files)
+- **Upload extraction results to S3** → [Module 2: Writing Files](./modules/data-sources-02-s3-operations.md#writing-files)
+- **Connect to vendor SFTP server** → [Module 3: SSH Authentication](./modules/data-sources-03-sftp-operations.md#authentication)
+- **Stream large Parquet files** → [Module 4: Streaming Patterns](./modules/data-sources-04-file-patterns.md#streaming-large-files)
+- **Prevent duplicate file processing** → [Module 4: State Management](./modules/data-sources-04-file-patterns.md#state-management)
+- **Handle connection failures** → [Module 4: Error Recovery](./modules/data-sources-04-file-patterns.md#error-recovery)
+- **Use separate AWS accounts** → [Module 5: Dual Credentials](./modules/data-sources-05-advanced-topics.md#dual-aws-credentials)
+- **Test SFTP connections** → [Module 5: Testing](./modules/data-sources-05-advanced-topics.md#testing-strategies)
+
+---
+
+## Key Concepts
+
+### Universal Interface
+
+Both `S3DataSource` and `SftpDataSource` implement the same base interface, making it easy to switch between storage backends:
+
+```typescript
+interface DataSource {
+  downloadFile(key: string): Promise<string | Buffer>;
+  writeFile(key: string, content: string): Promise<void>;
+  listFiles(prefix?: string): Promise<string[]>;
+  fileExists(key: string): Promise<boolean>;
+  deleteFile(key: string): Promise<void>;
+  streamFile(key: string): Promise<ReadableStream>;
+}
+```
+
+### Presigned URL Architecture (S3)
+
+`S3DataSource` uses **presigned URLs exclusively** for universal compatibility:
+
+- Works in Node.js, Deno, and Versori
+- No AWS SDK bundling required
+- Simple `fetch()` API for all operations
+- Automatic signature generation via `S3PresignService`
+
+### Direct SSH Connection (SFTP)
+
+`SftpDataSource` uses **direct SSH connections** via `ssh2-sftp-client`:
+
+- No platform dependencies
+- Password or private key authentication
+- Connection pooling for efficiency
+- Automatic retry with exponential backoff
+
+---
+
+## Examples
+
+Complete working examples are available in the [`examples/`](./examples/) directory:
+
+- `s3-csv-ingestion.ts` - Read CSV from S3, process, send to Fluent
+- `s3-parquet-extraction.ts` - Extract Fluent data to S3 as Parquet
+- `sftp-inventory-sync.ts` - Read inventory from SFTP, sync to Fluent
+- `dual-credentials.ts` - Use separate AWS accounts for source/target
+- `streaming-large-files.ts` - Stream processing for large datasets
+
+---
+
+## Best Practices
+
+1. **Use streaming for large files** (>10MB) to avoid memory issues (Module 4)
+2. **Implement state management** to prevent duplicate processing (Module 4)
+3. **Use separate AWS credentials** for source/target accounts (Module 5)
+4. **Enable connection pooling** for SFTP to reduce overhead (Module 3)
+5. **Validate files before processing** to catch errors early (Module 4)
+6. **Use private key auth** for SFTP instead of passwords (Module 3)
+7. **Monitor presigned URL expiration** for long-running jobs (Module 5)
+
+---
+
+## Architecture Comparison
+
+| Feature                | S3DataSource                   | SftpDataSource                      |
+| ---------------------- | ------------------------------ | ----------------------------------- |
+| **Transport**          | HTTPS (presigned URLs)         | SSH (sftp-client)                   |
+| **Authentication**     | AWS credentials                | Password or private key             |
+| **Platform Support**   | Node.js, Deno, Versori         | Node.js, Deno, Versori              |
+| **Dependencies**       | None (uses fetch)              | ssh2-sftp-client                    |
+| **Connection Pooling** | N/A (stateless HTTP)           | Yes (configurable)                  |
+| **Streaming**          | Yes (native fetch)             | Yes (sftp streams)                  |
+| **Best For**           | Cloud-native, scalable storage | Vendor integrations, enterprise systems |
+
+---
+
+## When to Use Each Data Source
+
+### Use S3 When:
+
+- ✅ Building cloud-native integrations
+- ✅ Need scalable, durable storage
+- ✅ Working with large datasets (GB+)
+- ✅ Deploying to Versori or serverless platforms
+- ✅ Need global distribution (CloudFront)
+
+### Use SFTP When:
+
+- ✅ Integrating with vendor/partner systems
+- ✅ Working with enterprise systems
+- ✅ Required by compliance/security policies
+- ✅ Vendor provides SFTP-only access
+- ✅ Need real-time file monitoring
+
+See [Module 1: Decision Guide](../auto-pagination/modules/auto-pagination-01-foundations.md#decision-guide) for detailed comparison.
+
+---
+
+## Getting Help
+
+- **Quick answers:** [Quick Reference](../advanced-services/advanced-services-quick-reference.md)
+- **Error Handling:** [File Operations Error Handling Guide](./data-sources-file-operations-error-handling.md) - Comprehensive guide for connection, discovery, reading, and parsing errors
+- **Troubleshooting:** [Module 5: Troubleshooting](./modules/data-sources-05-advanced-topics.md#troubleshooting)
+- **Examples:** [`examples/`](./examples/)
+- **API Reference:** [Module 2: S3 API](./modules/data-sources-02-s3-operations.md#api-reference), [Module 3: SFTP API](./modules/data-sources-03-sftp-operations.md#api-reference)
+
+---
+
+## Next Steps
+
+**New to data sources?** Start with [Module 1: Foundations](../auto-pagination/modules/auto-pagination-01-foundations.md)
+
+**Already familiar?** Jump to specific topics:
+
+- [S3 Operations](./modules/data-sources-02-s3-operations.md)
+- [SFTP Operations](./modules/data-sources-03-sftp-operations.md)
+- [File Processing Patterns](./modules/data-sources-04-file-patterns.md)
+- [Advanced Topics](./modules/data-sources-05-advanced-topics.md)
+
+---
+
+**Remember:** ONE interface for BOTH S3 and SFTP. Learn the pattern once, use it everywhere.